Re: [IxDA Discuss] Article on Number of Usability Test Participants
Thomas, In my experience, it depends. Obviously, issues stemming from visual design decisions made after the wireframe stage will not be caught in wireframe tests. At the opposite end of the spectrum, misfits between the designed workflow and the habits/expectations of the target audience can be caught very early, often in storyboard tests. A final product test can mislead as much as a mockup test if, for example, the browser in the lab reveals more of a web page than the browser the customer uses, or the network connection is much faster in the lab than in the field, or customers have a far different attitude about made-up data than real-life data. I wouldn't recommend uncritically extrapolating a lab study of a product at any stage to the eventual experience of the users. But testing at various stages of development helps to catch important issues that get harder to remedy as time goes on. Larry On Oct 1, 2009, at 9:51 PM, Thomas Petersen wrote: If we are talking wireframes or any other replacements for the real thing whatever you will find have very little if anything to do with what you find in the end. The real issues arise after the launch not before and the real question is not how many participants but at what point participants should be used. Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Article on Number of Usability Test Participants
On Oct 8, 2009, at 9:26 AM, James Page wrote: Have a hypothesis before you conduct your test, and don't fish. But surely this applies to traditional lab testing as well. And is the reason why Web Stats are great as Hypothesis generating tool, but not a tool to test a hypothesis. James, I believe it can work the other way as well. A lab test of a deployed product may expose a source of friction that does not completely prevent the observed users from completing their tasks. That observation can lead to a hypothesis that the rate of abandons from a particular page in a particular situation could be high. Analysis of log data, or an A-B test against a revised design, can verify or refute the hypothesis. I like to say, somewhat simplistically, Web stats tell you what, usability studies tell you why. No matter which you discover first, the quantitative what or the qualitative why, you will sometimes benefit from seeking to discover the other. Larry Tesler Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Article on Number of Usability Test Participants
I've been running usability studies since 1974. In 1980-81, the strategy I settled upon was to continue adding testers (qualified subjects) until I had learned nothing new from two people in a row. Depending on the circumstances, that point was usually reached after 4-8 tests, in the same range that other usability professionals have discovered. Often, an early tester--sometimes the very first--would encounter a problem such as ambiguous or misleading copy that would obviously affect a lot of users. We'd fix such problems immediately, often before the next tester arrived. Since that time, I haven't seen many situations where most serious problems weren't exposed by a well-designed and carefully observed study of 4-8 qualified subjects, people who would be likely users of the software with typical levels of prior experience and knowledge. Problems encountered too rarely to detect with a small number of testers can still be serious. But most such problems are more effectively discovered in other ways, such as analysis of customer service and usage logs. Errors with profound consequences, such as physical injury or substantial economic loss, require different approaches, including careful walkthroughs with security experts, large beta trials in which adverse consequences are artificially limited, and exposure of test subjects to intentionally risky scenarios within simulated environments. Larry Tesler Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Bowman leaves Google
Kevin, Your post is both balanced and wise. I, on the other hand, am having trouble communicating my position on this issue. Shades of blue and widths of borders could well be inhibiting the crossing of a valley if undue time and resources are devoted to such 0.1% improvements to the exclusion of 10% leaps. Larry On Mar 23, 2009, at 2:35 PM, Kevin Fox wrote: Larry, I agree with the dangers of hill-climbing to the exclusion of finding new hills, but I don't believe this is the case at Google. Data-driven design is used when performing incremental design changes, and is more like QA or Usability testing. Neither preclude the utilization of blue-sky or revolutionary design, but all are important in optimizing design. Even Doug's examples of 41 blues and 3, 4, or 5 pixel borders aren't cases where data is inhibiting the crossing of a design valley, unless you consider a hue or a pixel width to be revolutionary. For another former Google designer's take on Doug's departure, and Google UX in general, I submit the post I wrote this morning: http://fury.com/2009/03/google-design-the-kids-are-alright/ Thanks, Kevin Fox Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Bowman leaves Google
Jenifer, Thank you for providing your insider-informed and well-reasoned perspective. I'd like to respond to this remark you made about my earlier post: On Mar 22, 2009, at 7:10 AM, Jenifer Tidwell wrote: The point about hill-climbing with data-driven incremental changes is well taken, but honestly, don't you think that It Would Be Bad to accidentally send Google Web Search into a design valley while you blundered about looking for a higher hill? By my comments (reproduced below), I didn't mean that a company should leap blindly off their current hill in hopes of landing on a higher one. My point was that over-reliance on data-driven incremental changes would be ill-advised, as would choices made solely on the basis of performance data. I advocated a smart combination of analytical thinking and design thinking to better climb the current hill and also search for taller mountains. The post by my former colleague Tom Chi that dave malouf cited makes that same point and more. It's at http://www.ok-cancel.com/comic/177.html . Also, like several others who have commented on this topic, I was not referring specifically to Google, but rather to the practice of web design wherever it takes place. Larry Tesler On Sat, Mar 21, 2009 at 5:59 PM, Larry Tesler wrote: Yes, over-reliance on data-driven incremental design (DDID) is ill- advised. - Customers who use more than one of a company's products tend to be the most valuable customers in the long run. DDID usually optimizes one product at a time. The resulting inconsistencies may make each product a bit more profitable but can make it less likely for a heavy user of one to become a casual user of another. - DDID is an effective way to climb a little higher on a profit hill. It will never get you off the current hill onto a taller mountain. - Changing shades of blue and line widths can nudge a product higher on its current hill. But an organization that makes choices based solely on the basis of performance data won't learn why a certain shade or width works better, and is unlikely to apply the lesson to the next project. Revenue is foregone, costs mount and precious resources are tied up while each new product is gradually optimized. But many managers love DDID. It a systematic, replicable, and inherently measurable. Delight in the experience and passion for the product line are much harder to measure. The non-mathematical way that designers go about evoking such emotions isn't something that the staffing and training departments can reliably replicate. These days, great success usually emerges from a smart combination of analytical thinking and design thinking, a combination that requires mutual respect and cooperation as equals among the various practitioners. Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Bowman leaves Google
Yes, over-reliance on data-driven incremental design (DDID) is ill- advised. - Customers who use more than one of a company's products tend to be the most valuable customers in the long run. DDID usually optimizes one product at a time. The resulting inconsistencies may make each product a bit more profitable but can make it less likely for a heavy user of one to become a casual user of another. - DDID is an effective way to climb a little higher on a profit hill. It will never get you off the current hill onto a taller mountain. - Changing shades of blue and line widths can nudge a product higher on its current hill. But an organization that makes choices based solely on the basis of performance data won't learn why a certain shade or width works better, and is unlikely to apply the lesson to the next project. Revenue is foregone, costs mount and precious resources are tied up while each new product is gradually optimized. But many managers love DDID. It a systematic, replicable, and inherently measurable. Delight in the experience and passion for the product line are much harder to measure. The non-mathematical way that designers go about evoking such emotions isn't something that the staffing and training departments can reliably replicate. These days, great success usually emerges from a smart combination of analytical thinking and design thinking, a combination that requires mutual respect and cooperation as equals among the various practitioners. Larry Tesler When a company is filled with engineers, it turns to engineering to solve problems. Reduce each decision to a simple logic problem. Remove all subjectivity and just look at the data. Data in your favor? Ok, launch it. Data shows negative effects? Back to the drawing board. And that data eventually becomes a crutch for every decision, paralyzing the company and preventing it from making any daring design decisions. Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] The Save Icon rut
Many printers do not resemble whatever print icon a particular app may provide. Apple's Mail app uses a paper airplane to mean Send. It is a loose metaphor to be sure--and lacks any association whatsoever to the accompanying sound effect. But it is understandable and probably more recognizable and memorable than a sidewalk mailbox or a more accurate SMTP server. A spyglass would be a suboptimal tool for searching a room full of files or the Library of Congress for all documents containing a particular phrase. But it is recognizable and memorable, so that's what most apps use to represent search. A paint brush for copying styles? A single sheet of paper with a folded corner to represent what could be a 400-page book, a 500-row spreadsheet or a two-hour movie? A stagecoach to represent a bank? Once an icon becomes established, it is easier to teach it to new users than to re-teach a new icon to habituated users. Storage media today range from disk drives encased in usually rectangular boxes to circular optical discs to flash memory chips. None are as distinctive and charming as an old-fashioned floppy. Using any one of them might be taken to exclude the rest. For example, an optical disc might be assumed to mean Burn. I think it would take a spectacularly clever icon to displace the floppy for Save. Larry Tesler On Mar 19, 2009, at 9:09 AM, Jake Trimble wrote: Has anyone seen any attempts to replace the standard floppy disk Save icon? Seeing as most people haven't touched a 3.5 floppy in a decade, is anyone addressing this archaic icon and how we can replace the current mental model associated with it? Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] People are Used to it
Jared, This digital download is now the bestseller on Amazon. I suspect that it became so after your comment yesterday. Larry On Dec 27, 2008, at 11:00 AM, Jared Spool wrote: If you want to understand how consumers view features versus usability, I'd start with the Harvard Business Review article, Defeating Feature Fatigue. http://tinyurl.com/88phdp Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
[IxDA Discuss] Yahoo! to 23andMe (was Re: People are Used to it)
Jared, I was one of the personnel changes. After transitioning my responsibilities to some great people, I left Yahoo! November 7 and started at 23andMe four days later. Larry Tesler P.S.: 23andMe has openings for designers. Multiple design skills preferred (visual/interaction/visualization). Interest in personal genomics required. Candidates are invited to contact me at la...@23andme.com . On Dec 28, 2008, at 9:13 AM, Jared Spool wrote: How are you doing? I hope the recent Yahoo! personnel changes worked out the way you'd hope. Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] How to cue people for drag and drop
Alan, Surprisingly, there is still no standard pattern or even a frequently used convention for indicating what can be dragged. (Or there is one but you and I have never heard of it, which means it isn't a very successful standard.) Still, some things can sometimes be dragged, e.g., an icon, a member of a thing-bar (whether or not the things are icons), an object within a drawing or layout, a border line, selected text, a selected group of things, or a drag-handle and whatever the handle drags. Experienced users probably consider the possibility that such things can be dragged. You might be able to take advantage of that suspicion. For example, while the pointer is hovering over a table row, or (to be less disruptive) when a row is selected, surround it by a drop shadow that makes it appear to float, or display a drag handle alongside it. Either cue may tempt the user to try dragging the row or its handle, as the case may be. Or, if row-dragging can only happen in an edit mode (I hate modes but when a document is shared, especially on the web, an edit mode can reduce the incidence of inadvertent corruption), display a drag handle to the left of each row whenever it is in edit mode. Larry Tesler On Dec 1, 2008, at 7:59 PM, Alan Wexelblat wrote: I have a screen with a slightly unusual UI. There is a table of rows that can be edited in place, but you can also drag and drop the rows, somewhat in the style of Excel. You can also drag the rows onto targets adjacent to the grid. The question is - how do I give people reasonable visual cues that drag-and-drop are acceptable actions here? Given that there are a large number of drop targets, I can't think of a good design pattern other than drag-drop for this but I'm open to suggestions on that as well. Any thoughts? --Alan Welcome to the Interaction Design Association (IxDA)! To post to this list ... [EMAIL PROTECTED] Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] [Off-topic] Alarm Clock Recommendations
My July 2006 wake up call about this alarming topic was documented at: http://blog.360.yahoo.com/nomodes Here's the text: The devolution of a clock Some time during the 1990's, I purchased a Braun travel alarm that was a delight to own until it eventually died last year. I replaced it by a similar-looking Braun travel alarm that I found on the web site of a Canadian retailer. (Braun no longer sells clocks in the United States.) The new model (which is now out of the lineup) has a face light controlled by a single easy-to-reach bar. The old model had no light. Advantage new model. The new model features radio control. If you are lucky, it will receive a radio signal occasionally that automatically sets the time accurately, taking seasonal time adjustments into account. More likely, in my experience, you will have to set it manually by holding down the Light Bar for about 10 seconds, then keeping it down for up to a minute until it reaches the current time. If you overshoot, you need to do it again. On the old model, you simply opened the back and turned a small wheel in either direction. Advantage old model. On the new model, when the alarm sounds, you can hit the Light Bar to snooze, i.e., to silence the alarm for a few minutes. On the old model, a proximity sensor allowed you to simply wave your hand near the clock to make it snooze. The coupling of snooze and light is clever and useful, but if you'd rather keep your eyes shut when you need more sleep, you'd prefer the old model. Both the old and the new models have a knurled wheel on the right side that is easy to reach when you want to change the alarm time. On the old model, you could turn the alarm-set wheel either way to adjust the wake-up time. On the new model, you can turn it only one way. To wake up ten minutes earlier, you have to nudge it more than twenty times. If you turn it too far (an easy mistake to make when you're sleepy), you have to do that again. Major advantage old model. On the old model, the alarm-set wheel turned silently. On the new model, it clicks 300 times as you go through the process just described. Someone at Braun forgot that roommates don't always retire at the same time. An alarm clock should be perfectly quiet, or at least emit a soft, constant sound, except at the time you wanted to wake up. Major advantage old model. I can't understand why companies that have mastered the art of friendly design often remove the most user-friendly features of their products in later models. I can only guess that they aren't conducting customer research or that they are ignoring the results. Braun is a great product company but the devolution of their alarm clock series is a disappointment. Larry Welcome to the Interaction Design Association (IxDA)! To post to this list ... [EMAIL PROTECTED] Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] Is Eye Tracking too expensive or complicated?
Jared, If most readers think this has gone on too long, we should wind it down. On Apr 22, 2008, at 5:59 AM, Jared M. Spool wrote: On Apr 22, 2008, at 3:08 AM, Larry Tesler wrote: The fact that different observers see different things in the same raw eye tracking data is of no more concern to me than the fact that different players count a different number of words on the same Boggle board. Some people see words that are hidden in plain sight; some do not. But noticed or not, the words are there. In the tea leaves, there are no hidden words. Larry, I have no doubt that the observations are of interest. My point is that the inferences drawn from those observations have little-to-no validity, thus the tea leaf analogy. I consider them valid if they inspire us to make design changes that lead to improvements in objective metrics. In any study, with or without an eye tracker, I look for: - a well-designed experiment - clean data - appropriate, error-free analysis - perceptive observation (which may require several observers to point things out to each other and reach consensus, as radiologists often do when faced with difficult images) - generation of hypotheses consistent with the study observations and any other available observations - prioritization of those hypotheses - generation of design solutions that respond to the most likely hypothesis - implementation - bucket testing - if no improvement is seen, iterate with alternate hypotheses If someone fixates on a link for a unusually large time, does that mean they are confused by it? Or they aren't confused, but are trying to decide if its what they want? Or they know whether they want it or not but are considering something else? If the user's mental state matters to you, ask the user what it was. They may know. If they do not know, devise a more clever experiment. But sometimes, the user's mental state doesn't matter. We may have run the test because too few people were clicking on the link. We thought perhaps they didn't even look at the area of the page that contained the link. The tracker has refuted our hypothesis. We know that some people look straight at the link and still do not click it. Other data may be needed if we want to find out why. But the study was a success. It achieved its goal. Different inferences will lead to completely different design solutions. Are you saying it doesn't matter which inference (and therefore, which design solution) the observers choose? If the radiologists call your malignant tumor benign, or vice versa, you may receive the wrong treatment, which could be a costly mistake. But design changes that eye tracking studies inspire often entail simple modifications to layout, color, size, typeface, etc., that help to steer attention. They are often cheap to implement. If there are two competing inferences, you can often try both implied solutions. Of course, if you had both designs in mind (and particularly the one that ultimately proved to be best) before you ran the eye tracking study, then the study was a waste of time. If that is the situation you are in, and you never want to run a study that may simply confirm that you were right, then your point is valid. But it is not always the situation. There may be no unrefuted theory about why users are not clicking the link. There may be no considered designs that would increase clicks. There may be too many credible designs--more than one has the time and staff to implement and test. Or you may simply want to confirm other data or hunches. When you back an eye-tracking supporter into a corner about this, they all say, Well, you should only use eye tracking in conjunction with other data collection tools and techniques to verify your inferences. In almost all cases, the other data collection tools and techniques would yield just as much value without the eye tracking as with it, so what's the benefit? The eye tracking test may have been the source of the first clue as to what ails your interface. Without it, you may never have thought to try those other tools and techniques. Or the eye tracking test may help to rule out hypotheses that one has generated from the use of other tools and techniques. Second, in almost all uses of eye tracking I've seen in the last 5 years, it's in the form of twisting the meaning of the heatmap/plot diagram/tea leaf reading into supporting whatever wacky inference the specialist wants to support. See that big red spot there. That means the users are confused v. See that big red spot there, that means we fixed the design. I agree that features of the heat map don't tell you mental states. But if they are inconsistent with hypotheses about mental states, they may call those hypotheses into question. See that big red spot over there. Maybe they were looking at the link after all. A big red
