This is an excellent topic and a critical one for our profession. It
would make for a good workshop or presentation at UPA, or CHI, or
IxDA. There are some good issues with taking notes of usability
sessions:
1. Many note taking sessions focus on point problems -- the person
chooses the wrong menu item, but problems that occur over a longer
interval may be harder to extract from notes. For example, a person
might do something early in a test that results in an incorrect result
later, but the initial action that started the chain of events may not
be obvious to anyone.
2. Watching videos is time consuming, but whenever I watch (or skim)
the videos, I see more problems and also note actions that were't
coded as problems (like a person spending 5 seconds looking for
something) but that might indicate a usability issue. In one test I
tried about 6 years ago, I had watched a tape of a session and listed
problems and compared them with the problems recorded by multiple
notetakers and I found 50% more problems.
3. I think that asking observers to record problems can be
problematic :-). I like to do some training to cue observers about
what things might constitute a "problem".
4. Most of us track issues for a single study, but sometimes the more
important issues might only show up in a meta-analysis across multiple
studies (using different methods).
5. What is the difference in noting problems if you see the person's
face versus just seeing the interaction with the software? I think
that there is some research that seeing a person's face will result in
problems being rated as more serious on average. I have noticed that
people who aren't watching the person note more details than observers
who actually see the person's face.
6. How you do deal with repeated examples of the same problem within
a session and across sessions. What is someone has the same problem 4
times in one session .... is that one problem or 4 instances of the
same problem?
7. What levels of granularity are represented in problem reports.
This is a big issue in notetaking. Some people might report problems
at a high-level of granularity ("Person has problem with dialog box
X") while others may report something much more specific ("Person has
problem with filtering UI in dialog box X").
I think that this is an area that should receive more attention in our
field. Thanks for this posting and responses.
Chauncey
On Tue, Jul 1, 2008 at 3:30 PM, Sarah Kampman <[EMAIL PROTECTED]> wrote:
> Though I take notes, I rely more heavily on quantitative measures when
> assessing the results of a usability test. All of the tasks I have
> participants complete have degrees of success, and often a time
> component as well. These measures lend themselves to comparison and
> analysis in a way that quotes do not. This is important for me, as
> usability testing only part of what I do, and I don't have the time to
> write out transcripts. The easier & faster I can make analysis, the
> better.
>
> As for the comments that I record, they fall into two categories:
> marketing and feedback. The "marketing" quotes are used to make a point
> internally, often to help position a change/enhancement as meaningful to
> a particular persona. The "feedback" quotes I use to fix whatever was
> problematic in the usability test, and I'll typically have my mockups up
> in Dreamweaver as I go through the feedback notes so that I can make the
> needed changes immediately.
>
> -Sarah Kampman
>
>
> -----Original Message-----
> I'm trying to find a better way to do usability test analysis.
>
> My current approach is: after i finish a usability study, with 8 or 10
> users, and collected my own and all observers' notes, I usually read all
> notes and then immediately write down the issues I feel area appearing
> more often (assuming my brain will remember issues that are repeated in
> many notes more than issues than appear only once).
>
> Now, how do YOU approach analyzing those notes? Reading and re-writing
> by heart? Putting all notes on a wall and eye-balling? Tagging the text
> with some piece of software?
> ________________________________________________________________
> Welcome to the Interaction Design Association (IxDA)!
> To post to this list ....... [EMAIL PROTECTED]
> Unsubscribe ................ http://www.ixda.org/unsubscribe
> List Guidelines ............ http://www.ixda.org/guidelines
> List Help .................. http://www.ixda.org/help
>
________________________________________________________________
Welcome to the Interaction Design Association (IxDA)!
To post to this list ....... [EMAIL PROTECTED]
Unsubscribe ................ http://www.ixda.org/unsubscribe
List Guidelines ............ http://www.ixda.org/guidelines
List Help .................. http://www.ixda.org/help
