One tool that fits in this conversation is Usability benchmarking. It is one quantitative way to track the total impact of a user interface effort. See - http://www.google.com/search?q=usability+benchmark. It is a baseline measurement of the overall experience and can be compared against periodically.
At the corporate level, UX folks are always at a large disadvantage because they are rarely a primary function in the company, and treated accordingly. >From the CEO view, why shouldn't the minority be evaluated in the same way as the majority? Metrics can not fix that alone. The good news is that you don't necessarily need metrics to be taken seriously anyway. What you do need is the support of the senior engineers and business folks the executives already respect. If they applaud and cheer when you ask for more budget, and support your requests whatever they are, I doubt anyone will ask to see your metrics. And if they did demand numbers ROI type arguments, where the cost of a UI designer is translated into a 1.5x impact on the bottom line, is a better line of data than tracking how many prototypes were made. But even if metrics were the only way to get executive interest, it's critical to separate out a) the things you do as a manager to jockey for executive support, from b) what things actually improve the quality of the user experience. You don't want your team confusing a with b. -Scott -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Russell Wilson Sent: Tuesday, August 19, 2008 1:24 PM To: Katie Albers Cc: [EMAIL PROTECTED] Subject: Re: [IxDA Discuss] 6 Metrics for Managing UI Design Katie - not sure I agree with the fact that UI cannot be measured. I think it's more difficult to come up with valid metrics, but I think it's possible. And I would also argue that if you ever want to be taken seriously at the corporate level, you *better* come up with some sort of quantitative indicator of the value UI brings to the table... however fragile that indicator is. And to your point, maybe "tracking the reverse" is a method worth exploring? Russell Wilson Vice President of Product Design, NetQoS Blog: http://www.dexodesign.com On Tue, Aug 19, 2008 at 2:14 PM, Katie Albers <[EMAIL PROTECTED]>wrote: > Everything Scott said and one more point: UI, Design, UX, IA and all > the associated fields are qualitative fields. They cannot -- BY > DEFINITION -- be measured. The closest you can come to measuring how > well you do your job is to measure clients'/customers' > dissatisfaction...does your help desk get fewer calls on this problem > than they used to? Are complaints lower than a similar app's are (and > good luck trying to get *that* data). Users seldom laud our work -- > the closer we are to "perfect", the less they notice that it's been done at all -- so really, you're stuck tracking the reverse. > > Katie > > " > At 11:33 AM -0800 8/19/08, <[EMAIL PROTECTED]> wrote: > >> One quick test of any metric is to spend 5 minutes trying to hack it >> - What if you were your evil twin: how could you make evil happen >> while still scoring well on these metrics? Better metrics make life >> harder for your evil twin. Lousy metrics make it easy. >> >> 1) Number of layouts delivered >>> 2) Number of interactive prototypes created >>> 3) Percentage of product design requests completed by commit date >>> 4) Number of users tested >>> 5) Number of product improvements made >>> 6) Number of products insights documented >>> >> >> One big assumption you're making is higher numbers means better results. >> One >> excellent prototype might do the work of 5 mediocre ones, but the >> designer who tends to need 5 mediocre ones will score better here. >> Same for # of users tested (you're rewarding people with sloppy study >> designs, or who can't win basic arguments without going to the lab), >> etc. Volume is a very poor measure for quality. But since measuring >> volume is easy and popular it explains the dozens of organizations >> proud of their fancy metrics, but somehow in denial of their lousy >> products. I'm really not a fan of systematic metrics - it's a >> favorite fuel for micromanagers. >> >> You should also note there is nothing wrong with subjective metrics. >> Why cant your team score itself 1 to 10 on team performance every >> month, or even better, ask your clients & stakeholders to rate your >> performance. Then at least you have a metric that is very difficult >> to manipulate. So what if it's not scientific: science is not a >> panacea. If the goal is to get a sense of how you're doing and focus >> team energy, qualitative measures can be just as effective as >> quantitative ones. RMPT can work fine with subjective measures. >> >> Lastly thinking like a general manager, which I was most of my >> career, the only metric I'd ever evaluate you on if I were your boss would be #5: >> number >> of product improvements made. That's the *only* metric that earns >> your team its salary. A favorite scheme I've seen used for usability >> engineers is simply this: # of usability issues found, # of >> recommendations made, # of recommendations approved. You might need a >> different set for designers, but you get the idea. >> >> If you discover more layouts, more prototypes, more magic spells, >> lead to more approved recommendations, you'll be rewarded for it. And >> if those things (layouts, protos, etc.) turn out to be a waste of >> time, you wont have a team of people doing those things anyway just >> because there is a metric that rewards it. (But do note that this is >> pretty much the only way to get people to respect metrics: they must >> be tied to rewards). >> >> And finally, I'd guess NetQos is a metric happy place give the >> business you're in, which is fine. But Creative work doesn't fit >> metric schemes as well as, say, performance testing does - creative >> work is inherently sloppy, messy and wasteful - I'd seek out other >> creative groups, PR, Marketing, Advertising, etc. and see how they're >> handling fitting their creative work into metrics. I suspect you'll >> get better ideas from them than from the engineering and Q&A orgs. >> >> -Scott >> >> Scott Berkun >> www.scottberkun.com >> > > > -- > > ------------------ > Katie Albers > User Experience Strategy & Project Management [EMAIL PROTECTED] > > ________________________________________________________________ > Welcome to the Interaction Design Association (IxDA)! > To post to this list ....... [EMAIL PROTECTED] Unsubscribe > ................ http://www.ixda.org/unsubscribe List Guidelines > ............ http://www.ixda.org/guidelines List Help > .................. http://www.ixda.org/help > ________________________________________________________________ Welcome to the Interaction Design Association (IxDA)! To post to this list ....... [EMAIL PROTECTED] Unsubscribe ................ http://www.ixda.org/unsubscribe List Guidelines ............ http://www.ixda.org/guidelines List Help .................. http://www.ixda.org/help ________________________________________________________________ Welcome to the Interaction Design Association (IxDA)! To post to this list ....... [EMAIL PROTECTED] Unsubscribe ................ http://www.ixda.org/unsubscribe List Guidelines ............ http://www.ixda.org/guidelines List Help .................. http://www.ixda.org/help
