One of the issues that should be considered in discount usability engineering is that these discount methods still need some rigor. Take the discount method of "think-aloud usability testing" (TAUT), one of the prominent discount methods. Doing a good TAUT requires careful attention to the choice of tasks that you use (or if you let the user define the task, a careful interview to understand what he/she would like to do). Choosing tasks and deciding on the order of tasks is a single line in a checklist, but it requires some thought about biases of stakeholders, the appropriate granularity of the task, the degree to which the task reflects the real world in terms of it complexity, data inputs, etc., whether one tasks will affect the next task, etc. What are the criteria for choosing tasks? You can choose tasks by frequency of use, criticality (rare, but devastating), new features that marketing will advertise about heavily, and many other criteria. The choice of tasks is even an ethical issue in that you could choose tasks that make the team look good (I've seen this when I was a lab director and rented out lab out to colleagues who wanted to please their managers or vice presidents).
Oh, and each task should be tied to both a business goal and user experience goal which is often not done by those with limited experience. Thinking-aloud (which can be used for a variety of methods) in a usability test can be affected greatly by the verbal and non-verbal prompting and interview style of the facilitator. Facilitating a think-aloud usability test is often portrayed as a simple discount method, but the reality is that it is a complex interaction that requires some rigor and constant self-reflection. Studies at the University of Washington have shown that the variability in how facilitators conduct sessions is extreme. Analyzing the results of think-aloud usability testing seems easy, just list the problems that you hear and see, but extracting problems is difficult. If I show a 10-minute tape of a typical session of something that is a bit complex the results will vary considerably. There are few places where people are trained in the art and science of taking good notes (or in the case of automated logging - marking the places where you then have to take good notes). Note-taking is one of the dirty secrets of our field. People often see the same event in different ways and inter-rater reliability is often low. One of the issues here is that it isn't always clear when a problem starts. Theoretically, a person might do something that does not prompt a "problem alert", but later something happens because of this unnoticed thing and a visible problem occurs. Another subtle issue that I've seen is that some people take notes on what they hear (the thinking aloud) while others note what they see on the screen. Sometimes a subtle hesitation or oscillation of the mouse between a few links or menu items might be a problem, but in my experience, few people will write that down. Since the 1980s, when nearly everyone watched hours of videotape, there has been a general trashing of videotape viewing because it takes too long. Well, yes, verbatim videotaping is probably not cost effective, a quick scan of tapes, especially with complex products will reveal things that people missed, sometimes serious issues that are not obvious when a person is focused intently - sometimes the issues might show up across task, but there is often not a good way to connect that quickly in the notetaking. A colleaue and I did a little experiment where he took notes during a session that last an hour on a complex network performance product. I watched a tape of the session and listed the problems that I saw and the difference between our two lists was significant. I saw about 50% more problems - including some serious ones. The issue of what constitutes a "usability problem" is hard because things that might be a problem for one user group might be a positive thing for another group so even defining a problem requires some reflection and consideration of the participants. Oh, and then there is the severity rating issue - ratings of severity by different people of the same thing are often quite unreliable. One reason for the unreliability is that there is often limited training on rating severity with examples and discussions of context. Much of the research on this topic shoves a 1-4 or 1-5 or 1-10 scale at participants with no real training or background and those who use scales in the real-world fail to work on a convergence of understanding in some semi-structured way. So, even a "discount" or "guerilla" requries some rigor and training and reflection (like having usability teams review a tape together to help each other understand why they listed something as a problem or not), and practice (there are very few HCI or Usability programs where people get extensive training on the basic types of interviews - structured, semi-structured, and unstructured interviews). Moreover, a good HCI/UUX/IxDA plan should consider a mix of discount and full-featured methods (for example, conducting field interviews for the next version while conducting simplified thinking-aloud and user interface inspections on the current product). Discount methods still need rigor, practive, review, and reflection. Chauncey On 10/31/07, Peter Boersma <[EMAIL PROTECTED]> wrote: > Alan asked: > > [..] However after talking to them for a while it becomes clear that > > they really don't know what Usability is [..] > > So I ask you: Is this the price we pay for growing popularity of > > Usability and User-Centered Design? > > Yes, and we should fight it with (drumroll...) discount usability methods. > Jakob Nielsen wrote an article on Guerrilla HCI in 1994: > > Guerrilla HCI: Using Discount Usability Engineering to Penetrate the > Intimidation Barrier > "I used the term "guerrilla HCI" in the title [..] because I believe that > simplified usability methods can be a way for a company to gradually build up > its reliance on systematic usability methods, starting with the bare minimum > and gradually progressing to a more refined lifecycle approach." > (http://www.useit.com/papers/guerrilla_hci.html) > > If these simplified methods come to the people with less knowledge in a > well-documented and illustrated way, there is hope that they won't mess it up > and bring our practice down again. > > And, since this is IxDA and not CHI-Web, I believe each field under the UX > umbrella should have its own set of simplified methods, as I explained for > IAs on my blog in "A piece of IA pie: little, micro, lite or guerrilla?" > (http://www.peterboersma.com/blog/2005/02/piece-of-ia-pie-little-micro-lite-or.html > or http://tinyurl.com/c7tw4). > IxDA.org's Resource Library (http://resources.ixda.org/) and the IA > Institute's Learning IA section (http://www.iainstitute.org/en/learn/) should > be examples of collections of these well-documented, simplified methods. > > Peter > -- > Peter Boersma | Senior Interaction Designer | Info.nl > http://www.peterboersma.com/blog | http://www.info.nl > ________________________________________________________________ > *Come to IxDA Interaction08 | Savannah* > February 8-10, 2008 in Savannah, GA, USA > Register today: http://interaction08.ixda.org/ > > ________________________________________________________________ > Welcome to the Interaction Design Association (IxDA)! > To post to this list ....... [EMAIL PROTECTED] > Unsubscribe ................ http://gamma.ixda.org/unsubscribe > List Guidelines ............ http://gamma.ixda.org/guidelines > List Help .................. http://gamma.ixda.org/help > ________________________________________________________________ *Come to IxDA Interaction08 | Savannah* February 8-10, 2008 in Savannah, GA, USA Register today: http://interaction08.ixda.org/ ________________________________________________________________ Welcome to the Interaction Design Association (IxDA)! To post to this list ....... [EMAIL PROTECTED] Unsubscribe ................ http://gamma.ixda.org/unsubscribe List Guidelines ............ http://gamma.ixda.org/guidelines List Help .................. http://gamma.ixda.org/help
