On May 28, 2009, at 5:55 PM, Robert Hoekman Jr wrote:
Oh, man. This is hardly solid research. Now you're just begging for
a debate. ;)
Trust me. Compared to many of the hundreds of research papers I review
each year, this one is pretty solid.
And maybe, just maybe, I was begging for debate.
Oh, and debate is for women. (Wait, no, I was thinking of something
else. Nevermind,)
1. ...
2. ...
3. ...
I actually think Frank did a nice job of defending his study, so I
won't do that here.
The only thing this study shows is that 2 out of the 3 teams created
a more usable design as measured against heuristics (this assumes,
of course (and it's a big assumption), that the evaluators did good
evaluations), and that they happened to be the same groups that used
personas in the project. At the absolute best, this is a loose
correlation. It's absolutely not proof of genuine causation. I could
have fared as well as any of them without personas and without a team.
4. Even if you throw out arguments #2 and #3 above, #1 still makes
it all a moot point.
No, no, no. You're looking at this wrong.
Scientific research studies like this are little building blocks. You
disassemble the problem into little problems, evaluate each problem,
then reassemble them to build your case.
While the bigger problem is, "Do teams that employ personas produce
effective designs for their audiences?" that, as you've correctly
pointed out, is hard to prove in a study. So, you break it down.
What this study does (in a very sweet, nice way) tackle one small
aspect of the problem: if you take a group of designers, break them
up, give some personas and others not, do you see different results.
The null hypothesis is, if personas don't make a differences, then the
control group (the folks w/o personas) will not produce
distinguishably different results from those that do.
This study contradicts the null hypothesis, because the teams with
personas produced different results based on the criteria (heuristic
evaluation).
Now, as you rightly point out, we can question the criteria (and
should!). That would be a different study. What criteria would you
like Frank or other researchers to measure against? That's the next
building block.
Also, we could perform the same study with different user research
tools. What tools would you like to see studied?
Of course, this study can't stand alone. Good research, like this,
needs to be duplicated elsewhere before you can really stand behind
it. Other researchers should try to replicate the experiment to see if
they get similar results. Then, and only then, will we empirically
know that the results were great. Frank did a good job of explaining
to another team how to duplicate the experiment.
You have to take this type of research for what it is and not expect a
single study to prove everything. I found it fascinating because it
duplicated what we've seen in our research in a controlled setting:
teams that use personas have a different dynamic than teams that don't
-- a dynamic that, in my opinion, leads to better design.
All that said, I still love you Jared. :)
Oh, Robert, you certainly know how to make a guy swoon. Hugs & Kisses.
Jared
________________________________________________________________
Welcome to the Interaction Design Association (IxDA)!
To post to this list ....... [email protected]
Unsubscribe ................ http://www.ixda.org/unsubscribe
List Guidelines ............ http://www.ixda.org/guidelines
List Help .................. http://www.ixda.org/help
