"If your client website has 1 million visitors a year, a usability issue that effects 10% of the users would be unlikely to be discovered on a test of only 5 to 10 users, but would give 100,000 people a bad experience when they visit the site."
Actually, that's not true. You'd be fairly likely to discover it with only 5-10 users - in the 65%+ range of 'likely'. Manufacturing quality control systems and product quality testing have been using such statistical methods since the 20's and they went through heavy refinement and sophistication in the 60's, 70's and 80's. It's also worth repeating the message both Jakob & Jared Spool are constantly talking about: test iteratively with a group of 5-10 participants. You'll find that 65%+ figure above rises to 99%+ in that case. Again, doesn't change your basic points about cultural diversity and behaviour affecting the test parameters, but your above point is not entirely accurate. Cheers Steve 2009/10/2 James Page <[email protected]> > It is dependent on how many issues there are, the cultural variance of your > user base, and the margin of error you are happy with. Five users or even > 10 > is not enough on a modern well designed web site. > > The easy way to think of a Usability Test is a treasure hunt. If the > treasure is very obvious then you will need fewer people, if less obvious > then you will need more people. If you increase the area of the hunt then > you will need more people. Most of the advocates of only testing 5 to 10 > users, experience comes from one country. Behaviour changes significantly > country by country, even in Western Europe. See my blog post here : > http://blog.feralabs.com/2009/01/does-culture-effect-online-behaviour/ > > If your client website has 1 million visitors a year, a usability issue > that > effects 10% of the users would be unlikely to be discovered on a test of > only 5 to 10 users, but would give 100,000 people a bad experience when > they > visit the site. > > Can you find treasure with only five or ten users. Of course you can. But > how sure can you be that you have found even significant issues. > > A very good argument in why 10 is not enough is Woolrych and Cockton 2001. > They point out an issue in Nielsen formula in that he does not take into > account the visibility of an issue. They show using only 5 users can > significantly under count even significant usability issues. > > The following powerpoint from an eyetracking study demonstrates the issue > with only using a few users. > http://docs.realeyes.it/why50.ppt > > You may also want to look at the margin of error for the test that you are > doing. > > All the best > > James > blog.feralabs.com > > 2009/10/1 Will Hacker <[email protected]> > > > Chris, > > > > There is not any statistical formula or method that will tell you the > > correct number of people to test. In my experience it depends on the > > functions you are testing, how many test scenarios you want to run > > and how many of those can be done by one participant in one session, > > and how many different levels of expertise you need (e.g. novice, > > intermediate, and/or expert) to really exercise your application. > > > > I have gotten valuable insight from testing 6-10 people for ecommerce > > sites with fairly common functionality that people are generally > > familiar with but have used more for more complex applications where > > there are different levels of features that some users rely on > > heavily and others never use. > > > > I do believe that any testing is better than none, and realize you > > are likely limited by time and budget. I think you can usually get > > fairly effective results with 10 or fewer people. > > > > Will > > > > > > . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . > > Posted from the new ixda.org > > http://www.ixda.org/discuss?post=46278 > > > > > > ________________________________________________________________ > > Welcome to the Interaction Design Association (IxDA)! > > To post to this list ....... [email protected] > > Unsubscribe ................ http://www.ixda.org/unsubscribe > > List Guidelines ............ http://www.ixda.org/guidelines > > List Help .................. http://www.ixda.org/help > > > ________________________________________________________________ > Welcome to the Interaction Design Association (IxDA)! > To post to this list ....... [email protected] > Unsubscribe ................ http://www.ixda.org/unsubscribe > List Guidelines ............ http://www.ixda.org/guidelines > List Help .................. http://www.ixda.org/help > -- Steve 'Doc' Baty | Principal | Meld Consulting | P: +61 417 061 292 | E: [email protected] | Twitter: docbaty | Skype: steve_baty | LinkedIn: www.linkedin.com/in/stevebaty ________________________________________________________________ Welcome to the Interaction Design Association (IxDA)! To post to this list ....... [email protected] Unsubscribe ................ http://www.ixda.org/unsubscribe List Guidelines ............ http://www.ixda.org/guidelines List Help .................. http://www.ixda.org/help
