Frank, An interesting study.
I think the issue with using heuristic evaluations is the well known issue with the evaluator effect. For example see: http://akira.ruc.dk/~mhz/Research/Publ/IJHCI2001_preprint.pdf<http://akira.ruc.dk/%7Emhz/Research/Publ/IJHCI2001_preprint.pdf> On to the issue of the brief. Was the market research given to the teams both Qualitative and Quantitative? All the best James http://blog.feralabs.com 2009/5/29 frank long <[email protected]> > Oh, man. This is hardly solid research. Now you're just begging for a > debate. ; ) > > 1. They tested the effectiveness of personas by performing heuristic > evaluations. That's like testing the happiness of a cat by > determining the ground speed of a duck. Usability and the > effectiveness of personas have little to do with each other. Any > decent designer can put something together that does well in a > heuristic evaluation %u2014 it doesn't mean the app meets the needs > of its audience. Not even remotely. > > 2. They say they set up the teams to be of relatively similar > strength, but how was that done, exactly? How do you make sure each > team has a relatively equal level of experience, skill, talent, > knowledge, and an ability to persuade? How do you set up three teams > in a way that no one person on any one team is able to talk the rest > of that team into a bad idea based on faulty knowledge? It may seem > like I'm nitpicking on this one, but I think this is a pretty > important point. Human beings are slippery. > > 3. The paper doesn't indicate what the control group used instead of > personas beyond "image boards" (which are meaningless as research > tools). All three of these teams were comprised of people from the > same class %u2014 meaning they all had received education on personas > by the same instructor. What else did the instructor teach these > people that they could put to use to come up with a good design > without personas? > > The only thing this study shows is that 2 out of the 3 teams created > a more usable design as measured against heuristics (this assumes, of > course (and it's a big assumption), that the evaluators did good > evaluations), and that they happened to be the same groups that used > personas in the project. At the absolute best, this is a loose > correlation. It's absolutely not proof of genuine causation. I could > have fared as well as any of them without personas and without a team. > > 4. Even if you throw out arguments # 2 and # 3 above, # 1 still makes > it all a moot point. > > All that said, I still love you Jared. : ) > > -r- > > Hi Robert, > Thanks for taking the time to read the paper %u2013 In answer to the > 3 points that you made regarding the validity of the paper. > > Point 1 %u2013 happiness of cats V ground speed of ducks. > > Everyone knows that cats like to chase ducks. The happiness of the > domestic feline is therefore inversely proportional to the ground > speed of said duck. > But seriously, the research did not use heuristics to evaluate the > effectiveness of personas. I used heuristics to evaluate the > usability of the resulting designs %u2013 allowing me to compare all > of the diverse design solutions consistently with each other. (user > testing would have been preferable but not feasible). The measure of > effectiveness was based on a number of factors including the > usability of the solution, interviews with students, observations > from their tutors and group discussions after the project. My initial > research question was to see if using personas made any difference > %u2013 and based on the research I found that they did. > > Point 2 %u2013 teams of equal strength. > > The students had completed 2.5 years of industrial design course when > I conducted the experiment. The course is entirely project based. The > same tutors work with the students day-in, day-out. After 2.5 years > you have a pretty accurate idea of each student%u2019s ability. > The fact that all students were working in close proximity was an > acknowledged weakness of the study. To work around this unavoidable > problem we informed all students of the experimental nature of the > project %u2013 not the specific goals, but the fact that each group > would be using a different design tool to solve the brief. The > importance of secrecy between the groups was stressed and we simply > asked for the student%u2019s cooperation in this regard. At the end > of the research we gathered feedback on the level of > %u2018information leaks%u2019 between groups. While a small degree > of information did pass between groups, it was not significant. > > Point 3 %u2013 The control group > > All teams received a set of briefing documents outlining the product > specifications, the manufacturing constraints, and a market research > file outlining the target user demographics. The brief stated that > the user-friendliness of the product was of paramount importance > %u2013 especially the set-up task as this was seen as a barrier to > product sales. The market research outlined the user profile and the > personas were created from this same user profile. > The image boards were given to the control group covered 3 areas, > Product environment, lifestyle and brand landscape. The items shown > on the image boards were also mentioned in the personas so that the > information provided to all teams were as uniform as possible. > > Point 4 - conclusion > The conclusions of the study find that using personas is an effective > tool and did produce more user-focused solutions. It also acknowledges > that personas are only one of many design tools %u2013 and does not > claim that it is the most effective method. As the saying > goes%u2026there is more than one way to skin a cat, or to make a duck > happy. > > :) > > Regards > Frank > > > > > . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . > Posted from the new ixda.org > http://www.ixda.org/discuss?post=42315 > > > ________________________________________________________________ > Welcome to the Interaction Design Association (IxDA)! > To post to this list ....... [email protected] > Unsubscribe ................ http://www.ixda.org/unsubscribe > List Guidelines ............ http://www.ixda.org/guidelines > List Help .................. http://www.ixda.org/help > ________________________________________________________________ Welcome to the Interaction Design Association (IxDA)! To post to this list ....... [email protected] Unsubscribe ................ http://www.ixda.org/unsubscribe List Guidelines ............ http://www.ixda.org/guidelines List Help .................. http://www.ixda.org/help
