Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Here's something that's guaranteed to make Jared crazy: 10 Web Form Design Guidelines Based on Eyetracking http://www.smileycat.com/miaow/archives/001750.php :-) Not only were these guidelines around before eye tracking, one of them is even wrong. (I am speaking of the one about left-aligning labels; and the reason is that it depends on the purpose of the form and the lengths of the labels.) Elizabeth -- Elizabeth Buie Luminanze Consulting, LLC tel: +1.301.943.4168 www.luminanze.com @ebuie Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
eye tracking provides much needed razzmatazz to impress clueless people.. who don%u2019t understand usability. - Jakob Neilsen @will. That's cool. I think Jakob nails it here: it can be used as a visual to all the good stuff you already eek out with proven usability methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Those following this thread might be interested in a study just published in the JUS that used eyetracking to compare newspaper websites and tv news websites. http://www.upassoc.org/upa_publications/jus/2009august/gibbs1.html To me, this is interesting not so much for its conclusions (which, frankly, amount to very little) as for the myriad citations to scholarly research going back many years regarding various issues touched on in this discussion. Anyone with the time or inclination might like to peruse some of this previous research. At the same, the authors of this study concede that there has been insufficient work that links usability with eyetracking results. As previously, the situation is as clear as mud. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Jared - After reviewing the video, I was hoping you would expand your thoughts on the validity of eye tracking data and ouija boards and correlation to thought processes. You've suggested that fixations don't offer any evidence/insight/it's false etc. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 26, 2009, at 1:33 AM, Guy Redwood wrote: Jared - After reviewing the video, I was hoping you would expand your thoughts on the validity of eye tracking data and ouija boards and correlation to thought processes. You've suggested that fixations don't offer any evidence/insight/it's false etc. And what in the video was supposed to show me that it offered any insights? I missed them. I just saw a repeat of findings that I knew without having to watch the eye tracking display, but just working with users in a traditional usability test. Where was the thing I'd only discover watching eye movements? Maybe I'm just too dense to understand this eye tracking stuff, despite the hundreds of hours I've sat in studies watching people interact with them? It's clear that I'm not smart enough for this new technology. I think that's the message we can take from this seemingly-endless thread. Jared Jared M. Spool User Interface Engineering 510 Turnpike St., Suite 102, North Andover, MA 01845 e: jsp...@uie.com p: +1 978 327 5561 http://uie.com Blog: http://uie.com/brainsparks Twitter: @jmspool Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 25, 2009, at 6:18 PM, Nick Gould wrote: Jared, you are selling Caroline's point short... how about this scenario: We report that test participants asked to locate the search box looked in the upper right corner for it. They told us that this is where they expected it to be and the eyetracking confirms that this is where they looked for it. But that's my point! Who cares if they looked in the upper right for the search box? The real question that will make a difference in the performance is, Did the user find the search box? Our studies show that, while users expect the search to be in the upper right, they are not hampered in any meaningful way if it's anywhere else prominent on the page. (See http://is.gd/2zZM1.) Now, maybe you reached for the wrong example here. If you want to try again, I'm all for it. But there is something we can learn from this example. Because you were focusing on the granularity of where does the user look and not what does the user need, you were focusing on the wrong problem to solve. Moving the search box from whereever it is to the upper right would not solve any problem. The data you collected forced you to focus on the wrong problem, which would commandeer resources to generate a result that probably won't make the design any better. I'm not sure which point of Caroline's you think I'm selling short. Was it this one? Jared seems to be focusing strongly on point 1. I somewhat sympathise with his point of view, in that I've not found that the eye-tracking stuff adds greatly to what I can find out from an ordinary observational test without eye-tracking. Because I don't think I sold that point short at all! So, yes, the ET lent further support to a talk aloud finding. For some clients - rightly or wrongly - this strengthens their confidence in the results. That's not razzle dazzle, it's just additional, consistent feedback. They said this and they did this. Why is it any different from reporting where they clicked?* It's only consistent feedback because you decided to make it consistent. You could just as easily say that those eye movements are involuntary and not really indicative of how someone succeeds or fails with the design. As you (or someone in this thread) stated, eye tracking records unconscious behaviors. Applying meaning to unconscious behavior is a difficult road to go down, because the odds of applying the wrong meanings are very high. Sure, it's optional - I don't think anyone claims ET replaces talk aloud or that it's even necessary for a good study. But it can be a valid, additional tool (in the right hands) for helping clients to feel comfortable about the research results. Yah, you keep repeating this. Still waiting to see what that additional value is. If ET doesn't replace traditional stuff, why bother with it? The traditional stuff is easier, cheaper, better understood, proven to work, and more reliable. Jared The Proof is in The Pudding Spool Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 25, 2009, at 5:44 AM, Guy Redwood wrote: That's why the in-depth understanding comes from using the eyetracking data in a retrospective review session with the users. When you play back eye tracking to the user, they tell you why they did things. We don't sit there looking at heatmaps, making things up. No. You make things up first, then badger the user into admitting that's what they were thinking by showing them irrelevant eye tracking footage. That's my opinion of retrospectives with eye tracking. (PEEP method and others. PEEP claims its innovative, but we were doing retrospectives with eye tracking in 1993.) Don't get me wrong. Retrospectives are a fabulous tool and I love them. It's the introduction of the eye tracking data into the retrospective process that distorts it past meaning. Jared Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Jared, we don't badger and we don't make things up. Users are very relaxed and chat freely in our retrospective sessions. That's the point. Are you assuming that we need to badger users because that's what you do in your think aloud sessions? Is that because the user is trying to do something, then trying to think about why they are doing something, then trying to verbalise what they are thinking - oh and also trying to remember what they had set out to do in the first place. Here's a video of some retail testing. http://www.youtube.com/watch?v=38w95lKFIWc What do you find irrelevant about the eye tracking? I've said it before but I'll repeat it here - You have an open invitation from us to come and see how we do things and hopefully take a fresh look at eye tracking. You've clearly had some bad experiences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Wow, just as I was thinking that this thread was getting a little tiresome and unproductive, I discovered that it actually started in 2005! http://www.ixda.org/search.php?tag=eyetracking Jared - it's really not fair as you have had way more practice attacking eyetracking than I have had defending it. I need to go train some more... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 25, 2009, at 7:21 AM, Guy Redwood wrote: Here's a video of some retail testing. http://www.youtube.com/watch?v=38w95lKFIWc What do you find irrelevant about the eye tracking? I didn't see anything in the video that surprised me or informed me about users interacting with the Amazon site. When I read the full commentary on http://is.gd/2ytGQ, I didn't see anything that was shown by the eye tracking that wouldn't have been shown through a traditional test. (I know this because I've watched about 400 shoppers on Amazon over the last 13 years, so I'm well aware of what we can and can't learn from traditional tests. Oh, by the way, I've done eye tracking studies on Amazon a few years back. Nothing you show here is any different that what we saw then.) Where was the retrospective interview? I'd like to see (and hear) that. You're going to have to do better than this to convince me that eye tracking adds any value. Jared Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Jared, you are selling Caroline's point short... how about this scenario: We report that test participants asked to locate the search box looked in the upper right corner for it. They told us that this is where they expected it to be and the eyetracking confirms that this is where they looked for it. So, yes, the ET lent further support to a talk aloud finding. For some clients - rightly or wrongly - this strengthens their confidence in the results. That's not razzle dazzle, it's just additional, consistent feedback. They said this and they did this. Why is it any different from reporting where they clicked?* Sure, it's optional - I don't think anyone claims ET replaces talk aloud or that it's even necessary for a good study. But it can be a valid, additional tool (in the right hands) for helping clients to feel comfortable about the research results. NG *As an aside, I think it's interesting that many of your arguments against eyetracking could also be leveled against clickstream analysis / clickmaps, etc... I am amazed at how willing clients are to believe that this data is meaningful on its own. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
I'm starting too see that both camps are not open to persuasion @ritchielee Have to disagree. This thread has changed the way I think about eye tracking. I'm not saying I'm now for or against it with any fervor, but am thinking about it more critically. And isn't that the point of these discussions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On another list recently, Carolyn Snyder pointed out that there are two purposes to user research: 1. Finding out about the product 2. Changing the product (I paraphrase). Jared seems to be focusing strongly on point 1. I somewhat sympathise with his point of view, in that I've not found that the eye-tracking stuff adds greatly to what I can find out from an ordinary observational test without eye-tracking. I'd prefer to focus somewhat on point 2, which to me is about influencing stakeholders to make them want to make changes based on what I've found. I have found that eye-tracking stuff can be rather helpful, in some circumstances, in helping stakeholders to understand what we've found and persuade them to act on it. If your work consists solely of the finding out aspect, then probably I'd agree that you don't need eye-tracking. If you are stakeholder/decision-maker, then I'd also probably agree that you don't need eye-tracking. For the rest of us, who want to make changes but need to influence other people to do so: it can be helpful. It's another tool in the toolbox, and I don't see why we shouldn't use it just because some people don't feel the need to use it. And I declare an interest: I have used data from eye-tracking in my talks on forms. I find that the illustrations help attendees to see what I've seen. Best Caroline Jarrett www.formsthatwork.com Forms that work: Designing web forms for usability. Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
I need to brush up on my 'dealing with phobics' skills. A lot of the positions against eye tracking are unreasonable. There is a clear correlation between what people look at and what they comprehend. Why would anybody deny it? I've never been very good at reading things I've not looked at! At the last eye tracking conference in Frankfurt we discussed the issues with what poor research was doing to the reputation of the eye tracking industry. I didn't realise that Mr Spool had played a part in that over the last 15 years. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from ixda.org (via iPhone) http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Fri, Aug 21, 2009 at 7:28 PM, g...@simpleusability.com wrote: I need to brush up on my 'dealing with phobics' skills. A lot of the positions against eye tracking are unreasonable. There is a clear correlation between what people look at and what they comprehend. A 1:1 correlation? What sort of correlation? I'm not sure I've heard anyone on this thread deny that comprehension is related to seeing which is related to gazing, but the devil is in the details, no? Why would anybody deny it? I've never been very good at reading things I've not looked at! This sounds like a straw-man argument to me. At the last eye tracking conference in Frankfurt we discussed the issues with what poor research was doing to the reputation of the eye tracking industry. I didn't realise that Mr Spool had played a part in that over the last 15 years. I'm sort of amazed there are eyetracking conferences and an eyetracking industry. Is there a cardsorting industry? -x- -- Christian Crumlish I'm writing a book so please forgive any lag http://designingsocialinterfaces.com Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 22, 2009, at 2:28 AM, g...@simpleusability.com wrote: There is a clear correlation between what people look at and what they comprehend. No. No there isn't. If there's a correlation, it's definitive unclear. Please, clear it up for all of us. At the last eye tracking conference in Frankfurt we discussed the issues with what poor research was doing to the reputation of the eye tracking industry. Produce the good research Guy. I'd love to see it. Jared Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 21, 2009, at 6:22 PM, Nick Gould wrote: Seems that, given your professional impermeability relating to this issue, you could just leave well enough alone; give your opinion when asked but otherwise respect the right of others to run their businesses as they see fit. Anyway... Nick, I don't have a problem with someone running their business as they see fit as long as it doesn't impact the field I work in. And herein lies the crux of the problem with your statement. You see, as a designer and UX professional, I'm part consultant and part educator to my clients and this field. As a consultant, my role is to provide services to my client that have a measurable impact on their business. As an educator, my duty is to educate them ethically about what our field provides. Why is honesty, integrity, and ethics so hard to come by? Perhaps the shiny color of that gold coin is more inviting that the value of doing real and meaningful work. I take pride in my field, my work, the service this field can provide to the world, what we can contribute, and the legacy we can leave behind. This is why I personally take issue with things like this. Eyetracking doesn't really provide any value other than to show some fancy visualization heat maps on screen. That's all it does. Yeah, it's impressive to see those heat maps. I love looking at them. But that's the only true value—visual aesthetics. It doesn't really tell you anything about why anyone does anything. Making that inference is a HUGE unsubstantiated leap. The claims I typically see made through ET in my view are unethical and unsupportable. Instead of trying to find a solution that ET solves, which to date and in 15 years in this field, I've not seen one, we should be focusing our efforts on existing research methods, or developing new ones, that actually do provide value, provide quality data, and from which we can make reliable inferences with integrity. When I've pressed ET advocates on the reliability of the data they produce and the reliability of the inferences they're making based solely on ET data, they buckle like a house of cards. Call a spade a spade. It's about as scientifically valid as snake oil. Cheers! Todd Zaki Warfel Principal Design Researcher Messagefirst | Designing Information. Beautifully. -- Contact Info Voice: (215) 825-7423 Email: t...@messagefirst.com AIM:twar...@mac.com Blog: http://toddwarfel.com Twitter:zakiwarfel -- In theory, theory and practice are the same. In practice, they are not. Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 22, 2009, at 9:51 AM, Guy wrote: At the last eye tracking conference in Frankfurt we discussed the issues with what poor research was doing to the reputation of the eye tracking industry. Perhaps you should've been discussing the harm that eye tracking does to real research. Cheers! Todd Zaki Warfel Principal Design Researcher Messagefirst | Designing Information. Beautifully. -- Contact Info Voice: (215) 825-7423 Email: t...@messagefirst.com AIM:twar...@mac.com Blog: http://toddwarfel.com Twitter:zakiwarfel -- In theory, theory and practice are the same. In practice, they are not. Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Hi! I have an SMI system in our facility in downtown Montreal. I'm very interested in the discussion. The pros and cons of using ET for usability testing seem pretty well described above. At the same time, I dislike what I understood as the suggestion that some practitioners are using ET to con clients. NO methodology or tool should be offered (honestly) without being clear about its deliverables, benefits and limitations. As a firm, we bear the cost of purchasing the equipment and the learning curve in order to be able to offer ET to clients. I don't see where this costs our clients unnecessarily and I will NEVER impose the use of ET on a client project just because it cost me to buy it. My job is very clearly to design and execute research that uses the best mix of methodologies that will really serve my clients' needs. ET is one of them. I also agree that expertise in the area of ET needs to be developed and that best practices can be shared. Feel free to get in touch with me by e-mail or by phone to discuss. Have a great day! Kate kcaldw...@ux-research.com 1 514 502-5862 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 20, 2009, at 9:16 PM, Elizabeth Buie wrote: I do have a question for you, Jared, to help me understand your point: Are you saying that we don't need to know how much time people spend with their eyes off the road while trying to text, or that we can get those data without doing eye tracking? I do understand what you're saying regarding regular usability testing, and I'm asking for clarification on what you're saying here regarding studies of texting while driving. Thanks for anything you can add to clear up this point for me. As I've stated before, eye tracking is a great research tool. If your goal is to study human physiology and cognitive psychology, looking for links between physiometric results and their cognitive counterparts, I think the eye tracking systems of today are invaluable. I also think that eye tracking is an interesting input and control device. It's certainly as interesting as the technology embedded in Microsoft's Big Ass Table, er, Surface device. There's a lot we could be doing with this, especially in the area of assistive devices. But I don't think eye tracking is useful in a production design process, where the goal of user research activities is to effectively inform the design for making decisions on how to improve a specific product. We don't know enough about how to translate the raw data emitted from the eye tracking device into the information we need to make decisions. (As everyone else who has tried to defend these silly devices has done, the going thinking is, well, you can't use eye tracking alone. I agree. However, my position is that it doesn't add value as an additive input. We don't need to go down this road again, in my opinion.) So, to answer your question, if you're talking about research from a scientific standpoint, I think the eye tracking equipment is a great idea. Add it to a quality driving simulator and you can learn a ton. (A shout out to my friends at George Mason who are doing some kickass studies in driving simulation.) If you're talking about designing some sort of product or tool to help you communicate while driving, I'm not sure I see the value of adding eye tracking into your research tools. Knowing the exact milliseconds someone is distracted probably won't help you decide on design requirements or solutions. Jared Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 20, 2009, at 9:43 AM, Kate Caldwell wrote: I have an SMI system in our facility in downtown Montreal. I'm very interested in the discussion. The pros and cons of using ET for usability testing seem pretty well described above. At the same time, I dislike what I understood as the suggestion that some practitioners are using ET to con clients. NO methodology or tool should be offered (honestly) without being clear about its deliverables, benefits and limitations. Kate, I agree. Given that I'm the one making the suggestion, (and I think it was more of an accusation than a suggestion,) I'd like to say that I also think that we need to be honest about what we do, especially to ourselves. I'd be interested in hearing the disclaimers you give your clients before presenting inferences from eye tracking data. Jared Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
At 9:37 AM -0400 8/21/09, Jared Spool wrote: Thanks for the clear statement, Jared. So, to answer your question, if you're talking about research from a scientific standpoint, I think the eye tracking equipment is a great idea. Add it to a quality driving simulator and you can learn a ton. Yes, this is exactly what I'm talking about. I'm sorry if it wasn't clear in my first post on this topic. If you're talking about designing some sort of product or tool to help you communicate while driving, Nope. In fact, I am not at all convinced that that is even possible: It's a problem of the attention required by the communication, not an issue of the means used. Elizabeth -- Elizabeth Buie Luminanze Consulting, LLC tel: +1.301.943.4168 www.luminanze.com @ebuie Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 20, 2009, at 5:04 PM, Andrei Herasimchuk wrote: To toss this little nugget into the mix as some valid if minor counterpoint to Jared's stated opposition to eye-tracking is really a disservice to how much Jared actually knows about this topic, and how much experience and expertise he has in research and technology. Apparently, I've hit a nerve. :) I knew my views on eye tracking would piss some people off. As I said earlier, I'm good with that. Jared Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 21, 2009, at 9:59 AM, Elizabeth Buie wrote: Thanks for the clear statement, Jared. So, to answer your question, if you're talking about research from a scientific standpoint, I think the eye tracking equipment is a great idea. Add it to a quality driving simulator and you can learn a ton. Yes, this is exactly what I'm talking about. I'm sorry if it wasn't clear in my first post on this topic. If you're talking about designing some sort of product or tool to help you communicate while driving, Nope. In fact, I am not at all convinced that that is even possible: It's a problem of the attention required by the communication, not an issue of the means used. To (possibly unnecessarily) clarify further: I have nothing against the hardware or its use. My issue has to do with the claims we make about what we can learn by using it in specific contexts, particularly when identifying issues about a screen's design. Jared Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
At 10:04 AM -0400 8/21/09, Jared Spool wrote: Apparently, I've hit a nerve. :) You stole my line. :-) Elizabeth -- Elizabeth Buie Luminanze Consulting, LLC tel: +1.301.943.4168 www.luminanze.com @ebuie Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Hi Jared, How are you? It didn't seem you were alone in the accusing (your word;-)) camp. I ALWAYS explain to clients that: - ET does not equal measuring seeing (because seeing is a cognitive action), it's the CORRELATION between seeing and point of regard fixations and saccades we're measuring. - ET measures foveal point of regard and NOT peripheral vision which is ALSO used by people to gather information about the stimulus whether it's a screen, a room or anything else, so yes, you can see things off fovea (whether they're actually there or not is another question;-)) - Calibration quality in ET is key if we are to reduce error margins to acceptable levels - error margins basically translate into a drifting of correspondances across the X and Y coords. We need to take this into account when defining Areas of Interest for analysis. - ET sampling rate is another - different machines have different rates, so yes there can be missed data. - Look out for how methodology can change behaviour (the think aloud vs silent task issue) - ET results should be addressed ONLY within the scope and context of the tasks that were given to the respondents, i.e. don't use results from one task to imply something else. - Usual caveats about sample sizes (qual vs quant) and statistical projection Where I'm finding ET really interesting is with larger sample sizes. We're looking right now at examples coming from the 100-person study about online surveys (a whole 'nother controversy;-)) we did earlier this year. What's interesting about that is we have ET data AND survey answers - which themselves infer that respondents read questions and saw labels because they selected items and input text answers too. Seeing how these line up - or not - is really providing some interesting learnings. Have a great day! Kate kcaldw...@ux-research.com +1 514 502-5862 From: Jared Spool jsp...@uie.com To: Kate Caldwell caldwell_k...@yahoo.ca Cc: disc...@ixda.org Sent: Friday, August 21, 2009 9:40:42 AM Subject: Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations On Aug 20, 2009, at 9:43 AM, Kate Caldwell wrote: I have an SMI system in our facility in downtown Montreal. I'm very interested in the discussion. The pros and cons of using ET for usability testing seem pretty well described above. At the same time, I dislike what I understood as the suggestion that some practitioners are using ET to con clients. NO methodology or tool should be offered (honestly) without being clear about its deliverables, benefits and limitations. Kate, I agree. Given that I'm the one making the suggestion, (and I think it was more of an accusation than a suggestion,) I'd like to say that I also think that we need to be honest about what we do, especially to ourselves. I'd be interested in hearing the disclaimers you give your clients before presenting inferences from eye tracking data. Jared __ Be smarter than spam. See how smart SpamGuard is at giving junk email the boot with the All-new Yahoo! Mail. Click on Options in Mail and switch to New Mail today or register for free at http://mail.yahoo.ca Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Oh, I'd love to know this. On Aug 21, 2009, at 9:40 AM, Jared Spool wrote: I'd be interested in hearing the disclaimers you give your clients before presenting inferences from eye tracking data. Cheers! Todd Zaki Warfel Principal Design Researcher Messagefirst | Designing Information. Beautifully. -- Contact Info Voice: (215) 825-7423 Email: t...@messagefirst.com AIM:twar...@mac.com Blog: http://toddwarfel.com Twitter:zakiwarfel -- In theory, theory and practice are the same. In practice, they are not. Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 21, 2009, at 12:14 PM, Kate Caldwell wrote: I ALWAYS explain to clients that[...] Well, based on these disclaimers, I really don't see any value in ET at all. Instead, it leaves me wondering why I should use ET at all. I won't claim to be an ET expert, but I have used it in the past. I've never really been a big fan, as I think the leap from what is gathered to inferences that are made leaves a pretty large gap. Okay, HUGE gap, actually. It's one of those solutions looking for a problem in my book. Yeah, it looks cool, but as a researcher, I just don't see good quality research data coming out of it. Cheers! Todd Zaki Warfel Principal Design Researcher Messagefirst | Designing Information. Beautifully. -- Contact Info Voice: (215) 825-7423 Email: t...@messagefirst.com AIM:twar...@mac.com Blog: http://toddwarfel.com Twitter:zakiwarfel -- In theory, theory and practice are the same. In practice, they are not. Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Hi Todd! Interesting answer;-) Was there a caveat in particular that spoke to you more? Or maybe it was the imagined magnitude of possible errors? I don't know of a single methodology that doesn't have limitations so I feel quite at ease pointing them out when it comes to ET - as a researcher you also need to know what they are so you take them into account in your analysis. The things to determine are, given the limitations and the benefits of ET (also discussed above) towards what research goals can it contribute? There have been a few suggestions above. The other big issue that we keep running into with ET discussions is the cost of the equipment and the learning curve. It does preclude an everyone can/should do it approach which seems to be proned by lots of folks. I actually don't think that everyone can (although everyone can learn to) design research or facilitate a test or conduct a home visit or do observational fieldwork or write a screener or conduct analysis but that will be another great conversation to have on IXDA;-) Survey-monkey users and home-based recruiters unite! I especially liked the idea Jared mentioned of assistive technologies and new ET-based forms of interaction (I read there's currently an ET system out that folks with reduced motor skills can use for gaming and Second Life, will try to find it and post back.) Great discussion! Hopefully this is helping Kristen. Have a great evening! Kate From: Todd Zaki Warfel li...@toddwarfel.com To: Kate Caldwell caldwell_k...@yahoo.ca Cc: Jared Spool jsp...@uie.com; disc...@ixda.org Sent: Friday, August 21, 2009 4:55:09 PM Subject: Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations On Aug 21, 2009, at 12:14 PM, Kate Caldwell wrote: I ALWAYS explain to clients that[...] Well, based on these disclaimers, I really don't see any value in ET at all. Instead, it leaves me wondering why I should use ET at all. I won't claim to be an ET expert, but I have used it in the past. I've never really been a big fan, as I think the leap from what is gathered to inferences that are made leaves a pretty large gap. Okay, HUGE gap, actually. It's one of those solutions looking for a problem in my book. Yeah, it looks cool, but as a researcher, I just don't see good quality research data coming out of it. Cheers! Todd Zaki Warfel Principal Design Researcher Messagefirst | Designing Information. Beautifully. -- Contact Info Voice:(215) 825-7423 Email:t...@messagefirst.com AIM:twar...@mac.com Blog:http://toddwarfel.com Twitter:zakiwarfel -- In theory, theory and practice are the same. In practice, they are not. __ Get the name you've always wanted @ymail.com or @rocketmail.com! Go to http://ca.promos.yahoo.com/jacko/ Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Issues aside, this is an amazing discussion! Great points being made on both sides. So, for the hopper, a statement and a question for @jmspool. First, the suggestion that Jared's position on eyetracking is a result of his anxiety about what the technology will mean for his methodologies / reputation / business is not sensible. Jared is a longtime recognized leader in the usability field and could just as easily embrace eyetracking as reject it with no particular impact on his career. No, I believe that Jared's opinions are based on the merits of the technology (or lack thereof) as he perceives them. I won't say the same about the barnacles who cling to his position to advance their own credibility - but that's another discussion. However, I can't quite make out why eyetracking proponents are the specific target of affirmative attacks by you, Jared. You seem unwilling to admit the possibility that those who find value in the technology are anything but thieves and charlatans (or children playing with toys). Seems that, given your professional impermeability relating to this issue, you could just leave well enough alone; give your opinion when asked but otherwise respect the right of others to run their businesses as they see fit. Anyway... My question for you, Jared: Do you place NNG / Jakob Nielsen among the phonies? I understand that they use eyetracking quite regularly and are about to release a book about it. con molto rispetto Nick Gould CEO Catalyst Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 21, 2009, at 6:22 PM, Nick Gould wrote: You seem unwilling to admit the possibility that those who find value in the technology are anything but thieves and charlatans (or children playing with toys). I'm willing to accept that anyone who finds value in it, does indeed find value in it. I'm just not willing to accept when they make false claims about what it can do. I find value in a good chocolate chip cookie, but I don't claim it's the solution to world hunger. If you want to play with an eye tracker that's fine. I don't have any real interest in what you do with your free time. If you want to sell your eye tracker to your clients as something that makes your work special, that's fine too. I don't see how it's any different than a fancy conference room or nicely formatted reports. If it brings a special flare to your work and makes you look smart, then I'm all for it. But, don't try to sell me (or anyone who is the least bit intelligent) that the eye tracker somehow tells you something about how people see a design. I'm not buying it. Don't try to tell me that it gives us information we can't glean any other way. I'm not buying that either. Don't try to tell me that knowing where the user has gazed has any relevance on anything useful. I'm not buying that. But, we've been through this already. So, I don't see any value in continuing this, until someone has something interesting to say. What would that be? Well, it could be someone who isn't a consultant that makes money off eye tracking services saying how they found a way to use the technology within their design practice to produce results that they wouldn't have found otherwise. It could be someone who isn't from an eye tracking consultancy sharing how the device continues to deliver insights they wouldn't gain any other way. That's what I'm waiting for. And have been waiting for since I first started playing with eye tracking equipment in 1994. I've got 15 years of experience with these suckers and have put them through their paces. I think I know a little something about them. Seems that, given your professional impermeability relating to this issue, you could just leave well enough alone; give your opinion when asked but otherwise respect the right of others to run their businesses as they see fit. Anyway... This has nothing to do with how you want to run your business. This has to do with being honest about your work. My son's a professional magician. As a result, I've spent a lot of time around magicians. When magicians perform for laymen, they often talk about the mysterious forces in the universe. They work hard to let their audience believe in something big -- magic. A good magician plays on the wonder of the audience and, if they do their job well, those audience members leave believing they've witnessed something miraculous. If you want to use eye tracking to let your clients walk out the door believing in something miraculous, I'm all for it. The interesting thing is this: When magicians gather at their professional conferences or groups to talk about their craft, they don't pretend magic really exists. They are honest with themselves. They talk about illusions, mechanics, and stage craft. I believe that an eye tracking system can produce great stage craft. I've seen it. It is stunning what people will pay attention to when they are shown the devices in action. But let's be honest with ourselves about what it really can and can't do. That's all I'm asking for - a little professional integrity. My question for you, Jared: Do you place NNG / Jakob Nielsen among the phonies? I understand that they use eyetracking quite regularly and are about to release a book about it. Funny you should mention that. First, the book, which you say they are about to release was first supposed to be released in 8/2007. It's been delayed for more than 2 years. Not heard why. Second, in my keynote at the UPA conference, with Jakob in the audience, I showed examples from his eye tracking research and called out the same issues that we've discussed here. I didn't call anyone a phony and I haven't called anyone a phony. All I've said is that eye tracking is a tool for consultants to differentiate their work and manipulate data to support their points. You're the one who has assumed that means you're a phony. (Of course, if the shoe fits...) Is Jakob a phony because he might someday release a book that talks about using eye tracking in usability work? No. Will I be impressed by what he has to say? We'll see. I haven't been impressed with what he's published so far on this topic (his F pattern stuff is very amusing - http://is.gd/2sUNW), but if he says something interesting, I'll be right there. Jakob I don't agree on everything. Nobody seems to have a problem with that. You I don't have to agree on everything
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
What sort of ROI are you wanting? A 60% uplift in sales because we understood how users were subconsciously making decisions? 22% uplift in sales because we saw the relationship between the use of language and the task? All 'stuff' that think-aloud wouldn't show you. The way I see it, the world falls into 4 camps. Camp 1 Those that hate eye tracking because they have only experienced poor research. Camp 2 Those that view eye tracking as a threat to their career and just follow and bleet negative comments with no reference. Camp 3 Those that use eye tracking to occasionally supplement research and publish heatmaps and inferences, to fuel the paranoia of camps 1 and 2. Camp 4 Those that use it as a core tool for user experience research and possibly don't worry too much about the flat-earthers that are missing a huge opportunity to get inside the users' heads. Which are you? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 19, 2009, at 7:17 AM, Guy Redwood wrote: Camp 4 Those that use it as a core tool for user experience research and possibly don't worry too much about the flat-earthers that are missing a huge opportunity to get inside the users' heads. Which are you? How about Camp 5? Those who have used it for going on 15+ years, since the early days of the first systems, and conducted much of the ground-breaking research that was the basis of what we know about how people interact with systems, and after hundreds of sessions has thoughtfully decided the equipment doesn't offer any real added value to other established practices. That's where I am. This is not about being a flat-earther. This is about actual, substantive, useful value. If y'all want to buy your toys and play with them, please feel free to do so with your heart's content. I have no problem with that. In fact, I encourage it. Play time is important. However, let's keep clear on what the actual data from eye tracking tells us. It can't tell us what the user sees. It can't tell us what the user doesn't see. It only tells us what they gaze at, which from my experience of working with the technology, isn't really that useful. There is no huge opportunity to get into the users' head. That's a myth propagated by people who spent a lot of money on equipment that doesn't do anything but confirm whatever version of the world they want it to confirm. Remember, if you torture any data hard enough, it will confess to anything you want. Eye tracking is the waterboarding of usability data. Feel free to dismiss my experience if you want, but that's where I'm at. Jared Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
At 4:30 PM -0400 8/20/09, Jared Spool wrote: However, let's keep clear on what the actual data from eye tracking tells us. It can't tell us what the user sees. It can't tell us what the user doesn't see. It only tells us what they gaze at, which from my experience of working with the technology, isn't really that useful. Except if you're studying driver behavior while texting or something. I saw a video today in which they described the use of eye tracking to learn just how long drivers trying to text (in simulated conditions) took their eyes off the road. I know that's not what you're talking about Jared, so I'm not really arguing against you. I'm just pointing out that there are some cases where it *is* helpful -- even necessary -- to know where people are looking, and for how long. Or, more specifically, where they are NOT looking. Elizabeth -- Elizabeth Buie Luminanze Consulting, LLC www.luminanze.com @ebuie Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Good point both of you. Really, what it comes down to to is the understanding of the difference between seeing and looking. And knowing which data you are attempting to gather, will probably help you decide what methodology to use. On Aug 20, 2009, at 1:41 PM, Elizabeth Buie wrote: At 4:30 PM -0400 8/20/09, Jared Spool wrote: However, let's keep clear on what the actual data from eye tracking tells us. It can't tell us what the user sees. It can't tell us what the user doesn't see. It only tells us what they gaze at, which from my experience of working with the technology, isn't really that useful. Except if you're studying driver behavior while texting or something. I saw a video today in which they described the use of eye tracking to learn just how long drivers trying to text (in simulated conditions) took their eyes off the road. I know that's not what you're talking about Jared, so I'm not really arguing against you. I'm just pointing out that there are some cases where it *is* helpful -- even necessary -- to know where people are looking, and for how long. Or, more specifically, where they are NOT looking. Elizabeth -- Elizabeth Buie Luminanze Consulting, LLC www.luminanze.com @ebuie Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 20, 2009, at 1:41 PM, Elizabeth Buie wrote: I know that's not what you're talking about Jared, so I'm not really arguing against you. I'm just pointing out that there are some cases where it *is* helpful -- even necessary -- to know where people are looking, and for how long. Or, more specifically, where they are NOT looking. It doesn't take expensive eye-tracking devices to determine that if someone is texting while driving, they are probably a good candidate for a Darwin Award. To toss this little nugget into the mix as some valid if minor counterpoint to Jared's stated opposition to eye-tracking is really a disservice to how much Jared actually knows about this topic, and how much experience and expertise he has in research and technology. -- Andrei Herasimchuk Chief Design Officer, Involution Studios innovating the digital world e. and...@involutionstudios.com c. +1 408 306 6422 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
And I think that calling a willing research participant in a simulated environment, a Darwin Award candidate is a disservice to all those that take part in our research! On Aug 20, 2009, at 2:04 PM, Andrei Herasimchuk wrote: It doesn't take expensive eye-tracking devices to determine that if someone is texting while driving, they are probably a good candidate for a Darwin Award. To toss this little nugget into the mix as some valid if minor counterpoint to Jared's stated opposition to eye-tracking is really a disservice to how much Jared actually knows about this topic, and how much experience and expertise he has in research and technology. On Aug 20, 2009, at 1:41 PM, Elizabeth Buie wrote: I know that's not what you're talking about Jared, so I'm not really arguing against you. I'm just pointing out that there are some cases where it *is* helpful -- even necessary -- to know where people are looking, and for how long. Or, more specifically, where they are NOT looking. Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
That simulation does not even come close to accurately simulating driving while texting. I don't want to admit how I know that. Maybe if you were trying to send a text while competing in a NASCAR event or something. In the ET-texting-while-driving case, the data could be used to test whether design changes incrementally reduce the time drivers are distracted. It is theoretically possible that several of these changes may be combined to create a miracle device that is safe to use for texting while driving. No think-aloud test is going to parse out those milisecond differences, and an experiment where you compare (simulated, I hope) accident rates would need such a high N it would cost WAY more than an eye-tracking device. jz . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
At 5:17 PM -0400 8/20/09, Jared Spool wrote: [regarding driver studies] Using eye tracking in this research doesn't add any value to any of the data we already have. You may well be right, and certainly you know a lot more about eye tracking than I do. Let me say a little more about what I'm thinking, if I may. As a life-or-death issue, texting while driving may need more extreme measures than we would ordinarily apply. Not to convince those of us who know research, but to convince people (e.g., legislators) who are in a position to do something with the findings. The cell-phone simulation that you posted is useful for convincing me, as an individual, that I personally cannot do both at the same time (not that I need convincing; I don't even use the phone while driving). But there are plenty of people who would not even try the simulation, let alone be remotely persuaded by it, because they believe in their own invincibility. I am not actually fully convinced that eye tracking is necessary even in the study of texting while driving. But I *would* argue that we should consider it carefully, because the circumstances and issues are special and the risks are so huge. I do have a question for you, Jared, to help me understand your point: Are you saying that we don't need to know how much time people spend with their eyes off the road while trying to text, or that we can get those data without doing eye tracking? I do understand what you're saying regarding regular usability testing, and I'm asking for clarification on what you're saying here regarding studies of texting while driving. Thanks for anything you can add to clear up this point for me. Elizabeth -- Elizabeth Buie Luminanze Consulting, LLC www.luminanze.com @ebuie Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
I'm starting too see that both camps are not open to persuasion; and I'll admit I'm still rediculously far from accepting any worthwhile ROI. @jay. We know users cannot verbalise their eye movements; and we know they scan everywhere at break-neck speed looking for something to click. We can design for that without any eye tracking interpretations. A statement like: 'Rather than waiting many weeks or months to see live site click-thru behavior, you can incorporate eye tracking earlier in your product cycle to make the design process more efficient.' - really is overkill for: Make sure your navigation is obvious. Following a design lifecycle with observation, interview, heuristics and iterative evaluation is all that is necessary to uncover any issues. As Darth said 'Search your feelings' 'you know it to be true' . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
When used together with other user experience research, Eye tracking offers tremendous value for improving products. Eye tracking measures unconscious behavior - and provides data that people simply cannot verbalize in other common user research methods, especially think aloud usability testing protocols. Decades of psychology research show that much human behavior occurs at an unconscious level. The human eye, for example, can make up to 5 fixations per second and this occurs below people's level of conscious awareness. So in a 30 second scan of a typical homepage, the customer may be looking at up to 150 items on the page. Your customers (or research participants) simply cannot verbally tell you where their eyes are going and this is exactly the value that good eye tracking data provides. Knowing what people are looking at - and in what order - is essential information for improving interaction and visual design on websites. Creating a design that guides eye flow in a way that meets both business and user needs is an essential part of an easy-to-use design. Our experience is that visual attention data IS correlated with behavioral performance metrics. If people don't see something, then they are less likely to click it. Rather than waiting many weeks or months to see live site click-thru behavior, you can incorporate eye tracking earlier in your product cycle to make the design process more efficient. For example, if you are working on several iterations of your homepage in a redesign effort (common scenario for most internet companies!), knowing which design better captures visual attention and how it is distributed across the page elements - is a critical input to iterative design work. And as we all know, sometimes its much harder (or impossible) to change things later, so there's a huge advantage to obtaining this data as early as possible in the product design cycle and not waiting until later... disclaimer - yes, we provide eye tracking! http://www.customerexperiencelabs.com/services/eye-tracking-lab/ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Jared: When a consultant looks at eye tracking results and says, The user clearly sees X but they don't see Y, they are making **it up. And using their tools badly. What's with the hate campaign on eye-trackers, Jared? This reminds me of the olden days when we first had video. It was an expensive technology. It took time to learn to use, and to learn how to use it properly. Agencies used it as a differentiator (we have video!!!) when they weren't strong enough to differentiate on the quality of their thinking. Clients liked it. It was a (relative) waste of money. Then, I found that the real benefit was that it allowed me to show clients, quite easily, things that I'd had to learn how to see in many, many sessions. Eventually, video got cheap and easy. Now I'd routinely record stuff because the overhead is minimal and sometimes, but by no means always, it's helpful to show clients selections from the records and even (gasp) sometimes to watch them myself to remind myself of something. Now - fast forward to eye-trackers. An expensive technology, etc etc, right up to (relative) waste of money. It's just technology! It's not a magic bullet that will help the hard-of-thinking to do a better job. And coming to the substantive point: I've used TOBII eye-trackers. They are indeed expensive, the software is expensive, and it's expensive to keep up with the upgrades. I looked quite seriously at getting one for a client who had a short-term surplus money problem, but it was easy to decide to hire one on the occasions that we considered it might be useful. They are relatively easy to use, but they don't infallibly track everything. Best Caroline Jarrett Forms that work: Designing web forms for usability www.formsthatwork.com Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 15, 2009, at 2:52 PM, Jared Spool wrote: User research, when done well, isn't a science at all. It's an engineering tool. If you have to demonstrate its scientific validity (and deal with the fact that the people you're working with perceive it as a soft science), then you've already lost the game, in my opinion. This is more a sign of an internally broken corporate culture than anything else. Cheers! Todd Zaki Warfel Principal Design Researcher Messagefirst | Designing Information. Beautifully. -- Contact Info Voice: (215) 825-7423 Email: t...@messagefirst.com AIM:twar...@mac.com Blog: http://toddwarfel.com Twitter:zakiwarfel -- In theory, theory and practice are the same. In practice, they are not. Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 15, 2009, at 9:28 PM, Jared Spool wrote: When a consultant looks at eye tracking results and says, The user clearly sees X but they don't see Y, they are making shit up. What eye tracking doesn't tell you is why they were focusing on X. Okay, so, yeah, their eyes were gazing at this object in the center left of the page of .08 microns of a second more than the object 40 pixels to the right of it. Uh, huh... and so what? This is the Web. It's about moving, interacting, finding, exploring. Fixation doesn't really measure anything other than how long they looked at what. As a designer, I don't care about fixation, I care about discovery, interaction, transactions. Fixation doesn't tell me that, it doesn't show me that. Watching someone use a system and watching what they interact with does. Inferring anything from fixation is sketchy at best. FYI, I've used eye-tracking systems in the past and even the people who are ET advocates will tell you that by itself, it's pretty much just a good marketing tool. Personally, any study that only uses ET, I wouldn't put an ounce of faith in. Just give me a person I can watch and talk to. Cheers! Todd Zaki Warfel Principal Design Researcher Messagefirst | Designing Information. Beautifully. -- Contact Info Voice: (215) 825-7423 Email: t...@messagefirst.com AIM:twar...@mac.com Blog: http://toddwarfel.com Twitter:zakiwarfel -- In theory, theory and practice are the same. In practice, they are not. Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 17, 2009, at 4:23 AM, Caroline Jarrett wrote: Jared: When a consultant looks at eye tracking results and says, The user clearly sees X but they don't see Y, they are making **it up. And using their tools badly. Yet, that's what they do. Remember Spool's First Law of Competency: It takes no skill to do something poorly. So, if you don't have skills, you'll use the tools badly. What's with the hate campaign on eye-trackers, Jared? I don't hate eye-trackers. I think, as a piece of hardware, it's very cool. It's got lots of great applications. Hell, I even worked on projects for the Navy, Army, and NASA that made very cool use of eye tracking. In fact, I'm surprised that the IxD world hasn't jumped all over these devices. They bring a level of interaction (using eye movement to control the device) that you can't get otherwise. Imagine popping up menus by staring at a specific button, then selecting the right object by just fixating on a control handle, for starters. Combine it with touch and voice, and you have a really huge increase in multi-modal interaction. Lots of interesting possibilities here. This reminds me of the olden days when we first had video. It was an expensive technology. It took time to learn to use, and to learn how to use it properly. Agencies used it as a differentiator (we have video!!!) when they weren't strong enough to differentiate on the quality of their thinking. Clients liked it. It was a (relative) waste of money. Then, I found that the real benefit was that it allowed me to show clients, quite easily, things that I'd had to learn how to see in many, many sessions. Eventually, video got cheap and easy. Now I'd routinely record stuff because the overhead is minimal and sometimes, but by no means always, it's helpful to show clients selections from the records and even (gasp) sometimes to watch them myself to remind myself of something. Maybe I don't get this analogy because every lab I've ever worked in, starting back when we built the first software usability testing lab had video. Costs have definitely come down, but that's not what we're talking about. The value of video has always been understood. Now - fast forward to eye-trackers. An expensive technology, etc etc, right up to (relative) waste of money. It's just technology! It's not a magic bullet that will help the hard-of-thinking to do a better job. And coming to the substantive point: I've used TOBII eye-trackers. They are indeed expensive, the software is expensive, and it's expensive to keep up with the upgrades. I looked quite seriously at getting one for a client who had a short-term surplus money problem, but it was easy to decide to hire one on the occasions that we considered it might be useful. They are relatively easy to use, but they don't infallibly track everything. This isn't about making eye trackers more cost effective. It's about whether they add *any* value at all. I contend they don't. I contend, at best, they are theatrical devices to demonstrate a theory of use gleaned elsewhere. If we just declared them as that -- as, like the video you talked about, a tool to show clients things that you'd learned -- I'd be ok. We state clearly that we use them purely for demonstration purposes. I could get behind that. Where I start on my little rants is when people contend they'll learn something from an eye tracker that they can't learn by just watching users. It's not true. There's no evidence to support that statement. And, what they claim they learn is often wrong. Wrong learnings lead to wrong decisions. That's bad. And, when people tell me that a nice TOBII system costs $30,000+, I look at that money and see 60 non-eye-tracking usability sessions. That's getting the team in front of 60 people, where they'll learn far more than by watching the red dot bounce around the screen for a small handful of folks with the budget they have left after they bought a device that will inevitably sit in a corner months later. If we can all agree that eye trackers are great tools for demonstration purposes only, then I'll stop my hate campaign, as you called it. Jared Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Cards on the table. I love eye tracking. It%u2019s the sharpest tool in the box for user experience research. It%u2019s the best way to observe natural behaviour. Think-aloud in usability testing is unnatural and can create false data. More cards on the table. I%u2019m appalled at some of the work carried out by some of the eye tracking companies. If you ever observe a user being encouraged to think aloud during an eye tracking session, just walk out and find a better research company. In simple terms, there are two ways in which you can use eye tracking equipment - Quantitatively and Qualitatively. Modern eye tracking equipment is simple to use. Modern eye tracking equipment takes 30 seconds to set-up. Modern eye tracking equipment is non-invasive. Modern eye tracking equipment is very accurate. Don%u2019t listen to the flat-Earthers, the earth is round and eye tracking is not a Ouija Board. In quantitative analysis you can show people stimulus and measure how they visually engage with it. Ultimately allowing you to benchmark the effectiveness of designs against the original brief. The user should not talk during the sessions as users will look at what they are talking about and produce false data (such as f-patterns?). There are varying degrees of inference with this type of research - but you keep this in mind when using the evidence to make decisions. Who doesn%u2019t want to see where people look when reviewing design options? In qualitative analysis, you sit people in front of an eye tracking monitor and get people to engage with stimulus in a natural way. As an example, if it%u2019s a website you%u2019re researching, you%u2019d start the participant off at google and ask them to go buy that thing they%u2019d just been talking about. Once the user has completed their tasks, the practitioner would play back the eye tracking to the user. This is where the really cool stuff happens. Giving a participant the visual cue of where they looked, allows them to recall conscious and subconscious strategies. If they looked at something and didn%u2019t see it, they will tell you they didn%u2019t see it. In fact, they will tell you why they didn%u2019t see it. The ketchup in the fridge analogy is very important and demonstrates why it%u2019s important to use a retrospective methodology in testing. Inference has no place in qualitative analysis in eye tracking studies. Here%u2019s the real deal. Asking users to think-out-aloud adds a huge cognitive load to the user. So much extra cognitive load, that users are more likely to fail a task because of the thinking out aloud. Over 60% of your behaviour is automatic and we don%u2019t know why/how we do things. We just do stuff. So, when a user is thinking-out-aloud in research - just what are usability practitioners listening to? A projected persona? Google sums this up in a nice quote from 2008: %u201Cpeople are masters of saying one thing and doing another%u201D Eye tracking allows us to see what they do and retrospective review gives us deep insight into why. Nobody should be making inferences from heatmaps. Please move away from the basic red dot bouncing on a screen argument, and think about what you could do with a tool that allowed you to really understand user strategies, indepth. Guy Redwood founder of SimpleUsability http://www.simpleusability.com . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Hello, Kristen In our lab we have a Tobii eye tracker, using Tobii Studio. An eyetrack study is not enough for analyzing usability, but it can be very usefull to collect specific data about screens layouts and interface elements positioning. If you have budget enough, I do recomend an eyetracker hardware. But before buying one, be sure that you have good cameras and hardware/software for video edition. Best regards, Robson Santos, D.Sc. Senior Usability and User Experience Researcher [+55 92] 8407 0523 [+55 92] 2126 1101 http://www.robsonsantos.com http://interfaceando.blogspot.com On Thu, Aug 13, 2009 at 6:23 AM, Kristenkrist...@pmgintelligence.com wrote: I am currently setting up a user research lab and am looking into purchasing eye-tracker software/hardware. I'm wondering what other labs use and the pros/cons of those systems. Thanks! Reply to this thread at ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
I have limited experience with eye-tracking but, for me, you haven't covered the most important reasons to use it. 1. Big bosses love it... it's a persuader it's science-y but funner. 2. Talk Aloud, when a participant is watching their video, becomes Post Talk Aloud and they can tell you what they were thinking and doing. This approach is MUCH more informative and inclusive / collaborative rather than talk aloud where the participant is centre stage and being tested... this shift is subtle but hugely important. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
I think a solid example is needed to vouch any true benefits beyond agency differentiation. Monitoring attention in such pinpoint detail seems a distraction for all parties, from what are probably fundamental design issues. Observation, interview and heuristics are much stronger methods; which should be exploited fully before donning headgear. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Great discussion!! Really enjoying all the thought and insights (on both sides). In general, I have to side on the pro-ET side here. The %u201Cmale refrigerator blindness%u201D and peripheral vision problems are important, and must be considered. But don't these problems have corollaries in talk-aloud (e.g., unnatural behavior or telling the interviewer what you think they want to hear) and every other method? I must say that I think the characterization of ET data interpretation as %u201Cmaking shit up%u201D is unfair for two reasons. First, making shit up is the basis of data interpretation. Couldn't giving several %u201Cexperts%u201D the same talk aloud protocol and test interface also lead to more than one story? Claiming that ET interpretation is more susceptible to variance might be more fair (and less abrasive). Secondly, the reliability of analysis is only as good at the analyst. If the experts surveyed could not tell that the data with different thresholds were the same, or even worse did not ask what the thresholds were, I think the problem may lie with their approach to analysis (or level of expertise). A true expert knows the limitations of ET (and any method they use, including talk-aloud and task-based protocols, both of which have plenty of limitations) as well as important ways to expose and deal with those limitations. My take is that ET work should be reserved for specific situations, and ET is best used to complement other methods. Handled with care, I think ET can provide quite a bit of insight. Jesse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Hi Kristen, Apologies from coming late. I've had some experience with eye tracking during my scientific work and to be honest, it probably wasn't worth the effort. Eye tracking measures immediate visual focus (and not necessarily attention - it is possible that the two can be split on occasions) and it's difficult to infer from this up to the higher levels of cognition that usability work is most often concerned with. Well used (i.e., experienced analysts, good participants, an appropriate task, and suitable conditions), it can shed light on the low level features of something, but miss any of these out, and the data are probably misleading. I used this system a while ago and cannot remember what its name was sorry, but I remember about 30% of participants being unmeasurable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Booyah Jared and spot on, of course. marianne mswe...@speakeasy.net -Original Message- From: discuss-boun...@lists.interactiondesigners.com [mailto:discuss-boun...@lists.interactiondesigners.com] On Behalf Of Jared Spool Sent: Friday, August 14, 2009 11:24 AM To: Kristen; Joshua Porter; IxDA Subject: Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations On Aug 14, 2009, at 7:48 AM, Joshua Porter wrote: Interesting take from Google on their use of eye trackers: In addition to search research, we also use eye-tracking to study the usability of other products, such as Google News and Image Search. For these products, eye-tracking helps us answer questions, such as Is the 'Top Stories' link discoverable on the left of the Google News page? or How do the users typically scan the image results - in rows, in columns or in some other way? Eye-tracking gives us valuable information about our users' focus of attention - information that would be very hard to come by any other way and that we can use to improve the design of our products. However, in our ongoing quest to make our products more useful, usable, and enjoyable, we always complement our eye-tracking studies with other methods, such as interviews, field studies and live experiments. More here: http://googleblog.blogspot.com/2009/02/eye-tracking-studies-more-than -meets.html Ok, may this warrants a more serious response. The problem with Google everyone else's use of eye tracking is that it requires a leap of faith from observation to inference. We can see the observation clearly and most of the time, we can agree on them. An observation is that the user's gaze was recorded on a specific x/y coordinate for a specific time period. Another observation might be that the device didn't record any gaze fixations on a different x/y coordinate. We might also observe that the first x/y coordinate matches up with a link to a news story. The second x/y coordinate matches up with an advertisement. So, we could conclude that the fixation of the user was on the news because they were interested in it. And that they didn't look at the ad because they weren't interested. But that conclusion could be very flawed. Assuming we can account for any calibration errors in the device (where the x/y coordinates didn't actually match the news link or ad -- a frequent occurrence in state- of-the-art eye tracking systems), we still don't know the brain activity behind the gaze fixations. Maybe they stared at the news link because they were completely baffled by the headline? Maybe they didn't realize it was a news headline and thought it was something else? Maybe they actually saw the ad in a quick, transitive glance that was too fast for the eye tracker to pick up? Maybe they registered the ad out of there peripheral vision, beyond that of the foveal focus region? (Many eye trackers won't show an experienced user's eyes moving to scroll bar even though they move their mouse there to scroll. It seems they acquire the scroll bar with peripheral vision, keeping their focus on the items of interest on the screen.) Jumping too quickly from observation to inference is the #1 cause of design problems. We assume things without eliminating other possibilities, make assumptions, and run with them. Spending a little more time to test our inferences, to ensure we've properly qualified them and eliminated alternative explanations can save a lot of energy and downstream problems. (I've written about this in an article called The Road to Recommendation: http://www.uie.com/events/roadshow/articles/recommendation/) So, here's the problem with eye trackers: Every inference must be tested without the eye tracker. As the folks from Google say: we always complement our eye-tracking studies with other methods, such as interviews, field studies and live experiments. Fact is, had they started with the other methods, they wouldn't have discovered anything new in the eye tracker. And the other methods are cheaper, more efficient, and more beneficial. There is one advantage to eye tracking hardware. On a recent visit to the Googleplex, I asked about their usage their and this observation/ inference problem. They agreed with me, but told me about the real reason they use the devices. It turns out, the engineers and developers are more likely to attend usability tests when the eye tracker is in use. In the few labs they have that aren't outfitted with the devices, the engineers and developers rarely attend. They line up to watch eye tracking tests. For that purpose, the device may have some value. But so does good chinese food. I've found that a quality catering job is much more cost effective than mucking with the toys. (At Google, that might not work so well, since they have four-star chefs in their cafeteria -- hard to top that with catering.) That's my more serious response. It pisses off eye-tracking aficionados world-wide
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Eye-tracking is just one of many techniques, and should never be a replacement for observation and exploration of real users' experiences and motivations. It does require inferring what the user was thinking about as their eye moved, while the eye movement itself could have been caused by any number of things unrelated to the design. Are they tired, did something in the room distract them, did they see a word that had particular meaning to them personally? All these could give you false readings of the effectiveness of the design. Eye-tracking can be useful if you are trying to make a point with technology management, who often are impressed by what gadgets do and think of usability testing as soft science. If you have some findings you are having difficultly communicating, eye-tracking may be a means of proving it to skeptical engineers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 15, 2009, at 8:11 AM, Will Hacker wrote: Eye-tracking is just one of many techniques, and should never be a replacement for observation and exploration of real users' experiences and motivations. I hear that. However, its cost is so much larger than the other techniques you have to do anyways *and* it doesn't tell you anything you can't learn from those other techniques. So, why spend the resources to get info you have to verify with cheaper methods anyways? Eye-tracking can be useful if you are trying to make a point with technology management, who often are impressed by what gadgets do and think of usability testing as soft science. If you have some findings you are having difficultly communicating, eye-tracking may be a means of proving it to skeptical engineers. Maybe it's just me, but I never have this issue. I don't need to trick anyone into believing that the little dot on the eye-tracker screen or red splotch on the heat map means something it probably doesn't mean, just because I say it means that. User research, when done well, isn't a science at all. It's an engineering tool. If you have to demonstrate its scientific validity (and deal with the fact that the people you're working with perceive it as a soft science), then you've already lost the game, in my opinion. They should understand why its valuable before you've invested any resources in doing it. Otherwise, you're stuck making crap up to support your point of view. That's my opinion. It's worth what you paid for it. Jared Jared M. Spool User Interface Engineering 510 Turnpike St., Suite 102, North Andover, MA 01845 e: jsp...@uie.com p: +1 978 327 5561 http://uie.com Blog: http://uie.com/brainsparks Twitter: @jmspool Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Thank you for your responses so far! I have read several forums that have debated the pro's and con's of eye-tracking in general. Personally, I am a proponent of eye-tracking, and I am looking for others who hold the same opinion. I would like to know which systems labs are currently using and a review of that system. I apologize if this was not clear! Thanks everyone!! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Look, nobody said eye-tracking is a substitute for talk aloud, and nobody said it was a perfect methodology. We use it in conjunction with talk aloud for many of our tests and we find it actually does provide additional value. There are certainly issues: 1) the technology is still buggy, and 2) the analysis can be difficult and time consuming. Issue #1 is being resolved and will ultimately work itself out. For issue #2 it is the responsibility of the user (us) to deploy the eye-tracker in a way that is sensible, given its limitations and to exercise restraint in our interpretation of the results -- i.e. don't read-into the results, just report what the users saw and didn't see. We NEVER claim to know what the user was thinking (i.e. we don't infer from these observations), unless the ET data stimulates a discussion during talk aloud and the user reports their impressions or thoughts directly. I agree with Jared that user research is not a science. Clients are not confused about this fact (normally) where talk aloud usability is concerned. However, they are quite tempted to view eye-tracking as a somehow more scientific and therefore more valid methodology. Again, it's our responsibility to be straightforward about the tool's capabilities and shortcomings. I also agree with Jared that the hardware, software, and training investment associated with ET are significant given the current state of the technology. And, yes, you can get GREAT results as a user researcher without spending this money. Kristen, you will need to decide whether you and your clients will value the results enough to justify the cost. However, do not think that just buying the machine is enough. There is no user manual and the learning curve is steep. We have heard of many firms that bought the eye tracker and it sits collecting dust in the corner... Where I part company, respectfully, with Jared is in his assertion (made here and elsewhere, forcefully) that ET provides no information that can't be learned through traditional means. That's just factually false. Eye tracking tells you where users look on the page; where attention clusters and the paths they take as they explore. Users can't tell you this information. And when the question you are asking is do they see X? the eye tracker can give you your answer. It's that simple. We, and our clients, have found these answers to be valuable. Moreover we feel that thinking through these issues has broadened our understanding of how users interact with designs and how to produce the most actionable results for our clients. Anyway - that's my opinion and my firm's experience. Nick Gould CEO Catalyst Group www.catalystnyc.com . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 15, 2009, at 1:27 PM, Nick Gould wrote: Where I part company, respectfully, with Jared is in his assertion (made here and elsewhere, forcefully) that ET provides no information that can't be learned through traditional means. That's just factually false. Eye tracking tells you where users look on the page; where attention clusters and the paths they take as they explore. Users can't tell you this information. And when the question you are asking is do they see X? the eye tracker can give you your answer. It's that simple. We, and our clients, have found these answers to be valuable. Moreover we feel that thinking through these issues has broadened our understanding of how users interact with designs and how to produce the most actionable results for our clients. I contend 2 things: 1) A trained observer can get much of this information through what you call traditional means. 2) You can't tell from an eye tracker what the users sees. All you can tell from the eye tracking system is what the users focuses their gaze on. What the user sees requires cognitive effort the eye tracker doesn't measure. (Anyone who's had the experience of not seeing the ketchup bottle that is clearly on the shelf in front of them in the fridge has had the experience of gazing at something without seeing it. My late wife called this phenomena male refrigerator blindness. I'm quite afflicted myself.) When a consultant looks at eye tracking results and says, The user clearly sees X but they don't see Y, they are making shit up. I know because I've tested this theory many times. Just hand eye tracking results to three or more experts in eye tracking and they will each report radically different interpretations of the data. (Bonus test: change the thresholds on the eye tracking so that the heatmaps and gaze paths all shift around, and it gets even more entertaining as they try to tell you that the same user on the same results did radically different things.) With all due respect to Nick and his team at Catalyst, in my experience, eye tracking is a tool that consultants use to differentiate their services from all the consultants that don't have eye tracking. (Hire us because we have that eye tracking gizmo and they don't!) When companies buy eye trackers for internal use, very few continue to use it after a few months. As Nick said, it's a great expense and takes real skill and expertise to operate. Plus, there's no common understanding or best practices on how to use it. Minor adjustments to the device, such as setting the capture thresholds, will report radically different results, as it captures more or less gaze data which can be very noisy. You control the amount of noise by adjusting the thresholds, but that also can miss important gaze data. There's no standards or common understanding as to what the ideal settings are. (In fact, they are very specific to local lighting conditions, physiology of the subject, and other local contextual conditions. So, from one day to the next, the device reports different results.) The lifetime cost of buying, installing, training, using, and maintaining an eye tracker in a internal corporate setting can be equivalent to as many as 40 additional usability sessions a year. Personally, I'd rather get the data from the 40 additional users than spend it diddling with an ineffective piece of hardware. That's my opinion. I'm a researcher who doesn't (any more) use an eye tracker. Jared Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Oof, it's late, but hell, I'll take another swing... :-) Jared, everything you say is true regarding the limitations of the tool. I stated as much myself in the earlier post. Perhaps see was a loaded term to use as I didn't mean to imply that we understood the user's cognitive process - just what their gaze fixated on. I thought I made that clear, but I guess not. Jared, I would really appreciate it if you would share the data - or even the methodology for how you tested your theory. I would conduct that same test tomorrow to settle this once and for all. If there is a demonstrable basis to completely invalidate eye tracking then I would really want to know about it! Contrary to your assertion that we make shit up or bamboozle our clients with gizmos we actually do give a crap about the quality of our work (and our reputation, with our clients and peers) and we wouldn't be using the method if we didn't think it had value. As effective as this forum is as a platform for debate (at least we can use more than 140 characters), how about we continue it in person - over a beer -- next time you're in NYC? An eye tracking beer summit, if you will. At a minimum, I'd like to try to persuade you that not everyone who uses an eye tracker is a snake-oil salesman... NG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Aw, sorry Kristen in all the fuss I missed your real question in there... We use the Tobii system. Get in touch directly if you are interested in discussing our experiences in more detail. Best, NG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=44684 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Kristen - I attended a pro vs cons debate on eye-tracking a few months ago in London. On the 'cons' side was Kara Pernice who is the MD at NNG. Her main thrust was that eye-tracking was largely irrelevant for most usability work, particularly given the expense. I have since had a couple of conversations on this with heavy eye-tracking users. Both confessed that it really was overkill for most usability studies and that you did have to be experienced in order to interpret the results correctly (so it wasn't something you would use casually). On the plus side, it does some to be a good way to engage technology-oriented clients! Regards, William Hudson Syntagm Ltd Design for Usability UK 01235-522859 World +44-1235-522859 US Toll Free 1-866-SYNTAGM mailto:william.hud...@syntagm.co.uk http://www.syntagm.co.uk skype:williamhudsonskype Syntagm is a limited company registered in England and Wales (1985). Registered number: 1895345. Registered office: 10 Oxford Road, Abingdon OX14 2DS. Confused about dates in interaction design? See our new study (free): http://www.syntagm.co.uk/design/datesstudy.htm 12 UK mobile phone e-commerce sites compared! Buy the report: http://www.syntagm.co.uk/design/uxbench.shtml Courses in card sorting and Ajax interaction design. London, Las Vegas and Berlin: http://www.syntagm.co.uk/design/csadvances.shtml http://www.syntagm.co.uk/design/ajaxdesign.shtml -Original Message- From: new-boun...@ixda.org [mailto:new-boun...@ixda.org] On Behalf Of Kristen Sent: 13 August 2009 11:23 AM To: disc...@ixda.org Subject: [IxDA Discuss] Eye-Tracker software/hardware recommendations ... Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 13, 2009, at 10:23 AM, Kristen wrote: I am currently setting up a user research lab and am looking into purchasing eye-tracker software/hardware. I'm wondering what other labs use and the pros/cons of those systems. I'm with William. I suggest you get a Oiuja Board instead of an eye- tracker. It will produce exactly the same useful results, but at a significantly less cost. New Orleans-style Voodoo Dolls and Tarot Cards are also a good substitute. Again, the accuracy is identical between all of them. Jared Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
Interesting take from Google on their use of eye trackers: In addition to search research, we also use eye-tracking to study the usability of other products, such as Google News and Image Search. For these products, eye-tracking helps us answer questions, such as Is the 'Top Stories' link discoverable on the left of the Google News page? or How do the users typically scan the image results — in rows, in columns or in some other way? Eye-tracking gives us valuable information about our users' focus of attention — information that would be very hard to come by any other way and that we can use to improve the design of our products. However, in our ongoing quest to make our products more useful, usable, and enjoyable, we always complement our eye-tracking studies with other methods, such as interviews, field studies and live experiments. More here: http://googleblog.blogspot.com/2009/02/eye-tracking-studies-more-than-meets.html On Aug 14, 2009, at 8:32 AM, Jared Spool wrote: On Aug 13, 2009, at 10:23 AM, Kristen wrote: I am currently setting up a user research lab and am looking into purchasing eye-tracker software/hardware. I'm wondering what other labs use and the pros/cons of those systems. I'm with William. I suggest you get a Oiuja Board instead of an eye- tracker. It will produce exactly the same useful results, but at a significantly less cost. New Orleans-style Voodoo Dolls and Tarot Cards are also a good substitute. Again, the accuracy is identical between all of them. Jared Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Eye-Tracker software/hardware recommendations
On Aug 14, 2009, at 7:48 AM, Joshua Porter wrote: Interesting take from Google on their use of eye trackers: In addition to search research, we also use eye-tracking to study the usability of other products, such as Google News and Image Search. For these products, eye-tracking helps us answer questions, such as Is the 'Top Stories' link discoverable on the left of the Google News page? or How do the users typically scan the image results — in rows, in columns or in some other way? Eye-tracking gives us valuable information about our users' focus of attention — information that would be very hard to come by any other way and that we can use to improve the design of our products. However, in our ongoing quest to make our products more useful, usable, and enjoyable, we always complement our eye-tracking studies with other methods, such as interviews, field studies and live experiments. More here: http://googleblog.blogspot.com/2009/02/eye-tracking-studies-more-than-meets.html Ok, may this warrants a more serious response. The problem with Google everyone else's use of eye tracking is that it requires a leap of faith from observation to inference. We can see the observation clearly and most of the time, we can agree on them. An observation is that the user's gaze was recorded on a specific x/y coordinate for a specific time period. Another observation might be that the device didn't record any gaze fixations on a different x/y coordinate. We might also observe that the first x/y coordinate matches up with a link to a news story. The second x/y coordinate matches up with an advertisement. So, we could conclude that the fixation of the user was on the news because they were interested in it. And that they didn't look at the ad because they weren't interested. But that conclusion could be very flawed. Assuming we can account for any calibration errors in the device (where the x/y coordinates didn't actually match the news link or ad -- a frequent occurrence in state- of-the-art eye tracking systems), we still don't know the brain activity behind the gaze fixations. Maybe they stared at the news link because they were completely baffled by the headline? Maybe they didn't realize it was a news headline and thought it was something else? Maybe they actually saw the ad in a quick, transitive glance that was too fast for the eye tracker to pick up? Maybe they registered the ad out of there peripheral vision, beyond that of the foveal focus region? (Many eye trackers won't show an experienced user's eyes moving to scroll bar even though they move their mouse there to scroll. It seems they acquire the scroll bar with peripheral vision, keeping their focus on the items of interest on the screen.) Jumping too quickly from observation to inference is the #1 cause of design problems. We assume things without eliminating other possibilities, make assumptions, and run with them. Spending a little more time to test our inferences, to ensure we've properly qualified them and eliminated alternative explanations can save a lot of energy and downstream problems. (I've written about this in an article called The Road to Recommendation: http://www.uie.com/events/roadshow/articles/recommendation/) So, here's the problem with eye trackers: Every inference must be tested without the eye tracker. As the folks from Google say: we always complement our eye-tracking studies with other methods, such as interviews, field studies and live experiments. Fact is, had they started with the other methods, they wouldn't have discovered anything new in the eye tracker. And the other methods are cheaper, more efficient, and more beneficial. There is one advantage to eye tracking hardware. On a recent visit to the Googleplex, I asked about their usage their and this observation/ inference problem. They agreed with me, but told me about the real reason they use the devices. It turns out, the engineers and developers are more likely to attend usability tests when the eye tracker is in use. In the few labs they have that aren't outfitted with the devices, the engineers and developers rarely attend. They line up to watch eye tracking tests. For that purpose, the device may have some value. But so does good chinese food. I've found that a quality catering job is much more cost effective than mucking with the toys. (At Google, that might not work so well, since they have four-star chefs in their cafeteria -- hard to top that with catering.) That's my more serious response. It pisses off eye-tracking aficionados world-wide. I'm good with that. Jared Jared M. Spool User Interface Engineering 510 Turnpike St., Suite 102, North Andover, MA 01845 e: jsp...@uie.com p: +1 978 327 5561 http://uie.com Blog: http://uie.com/brainsparks Twitter: @jmspool
