On Oct 4, 2009, at 5:40 AM, James Page wrote:
The issue I have with testing with just a few users is that it can
exclude a significant issue.
James,
I think that's the major flaw in your thinking. You're trying to use
usability testing primarily for issue detection and it's a very
inefficient tool for that.
Nielsen makes a claim that his useit site might look awful, but that
it is readable, which is is not the case for me. I am Dyslexic, and
I find Nielsen's useit website hard going, because he uses very wide
column widths.
I too am dyslexic, but the column widths aren't the big issue I have
with Jakob's site. The big issue issue I have is his content.
By only using a few people for user research in one location, are
you not excluding a significant number of your site's audience?
Yes.
Which is why using usability testing as a sole source for issue
detection will inevitably fail.
There's no way you could put together a cost-effective study (even
with super-duper remote testing applications) that would participants
at chance for every possible variance found in humans.
By trying to use usability testing in this way, you're creating a
final inspection mentality, which Demming and the world of statistical
quality control has taught us (since the 40s) is the most expensive
and least reliable way of ensuring high quality. Issues will be missed
and users will be less satisfied using this approach.
Instead, a better approach is to prevent the usability problems from
being built into the design in the first place. Jakob shouldn't need
to conduct usability tests to discover that longer column widths could
be a problem with people with reading disabilities. In fact, those of
us who've paid attention to the research on effective publishing
practices have known for a long time that shorter columns are better.
Larger sample sizes, even when the testing is dirt cheap, is too
expensive for finding problems like this. We need to shift away from
the mentality that usability testing is a quality control technique.
Because of this, we've found in our research that teams get the most
value from usability testing (along the other user research
techniques) when they use it to inform their design process. By
getting exposure to the users, the teams can make informed decisions
about their design. The more exposure, the better the outcomes of the
designs.
To research this, we studied teams building a variety of online
experiences. We looked for correlations between those teams' user
research practices and how effective the team was at producing great
designs. We looked at the range of techniques they employed, whether
they hired experienced researchers, how many studies they ran, how
frequently the studies were, and about 15 other related variables.
We found that many of the variables, including the nature of the
studies (lab versus field, for example) or number of study
participants did not correlate to better designs.
More importantly, we found that 2 key variables did correlate
substantially to better designs: the % of hours of exposure each team
member had to primary observation and the frequency of primary
observation.
This led us to start recommending that teams try to get every team
member exposed to as many hours of observing users throughout the
design process. The minimum we're recommending is 2 hours of
observation every 6 weeks. The best teams have their team members
observing users for several hours every week or so.
Based on our research, we can confidently predict that having each
team member watch two users for two hours every 3 weeks will result in
a substantially better design than hiring the world's most experienced
user researchers to conduct a 90-participant study that none of the
team members observe.
So, number of participants in the study is a red herring. The real
value is number of hours each team member is exposed to users.
That's my opinion, and it's worth what you paid for it.
Jared
p.s. Is Webnographer an unmoderated remote usability testing tool? It
occurred to me this morning that it would be great to combine
unmoderated remote usability testing with eye tracking. Then we could
throw out all the data in a single step, instead of having to ignore
it piecemeal. A huge step forward in efficiency, I would think.
Jared M. Spool
User Interface Engineering
510 Turnpike St., Suite 102, North Andover, MA 01845
e: [email protected] p: +1 978 327 5561
http://uie.com Blog: http://uie.com/brainsparks Twitter: @jmspool
________________________________________________________________
Welcome to the Interaction Design Association (IxDA)!
To post to this list ....... [email protected]
Unsubscribe ................ http://www.ixda.org/unsubscribe
List Guidelines ............ http://www.ixda.org/guidelines
List Help .................. http://www.ixda.org/help
