There are several flavors of these sorts of things, but ironically setting
metrics to measure is often done without a well formed goal. Are you trying
to measure the value of your group? The value of people on your team? To win
arguments with other groups? It's usually a mistake to create metrics unless
you're clear on what you want to do with it, or more cynically, how others
might use them against you.
The most common reason this stuff gets generated is because everyone else is
- the CEO or VP is mandating it. In which case you should quickly decide
what parts of this process are done for show, and what parts you care about
and will find useful. Politically speaking, it can be best to align your
metrics with the peer team that has the strongest standing and most affinity
for your group. It's certainly a consideration to keep in mind (in other
words, at a minimum your metrics speaks a similar language to their
metrics).
I studied this stuff years ago, so here's a rusty recollection of an opinion
on this:
The usability/analytic side is easier:
1) Ratio of usability recommendations to implemented changes - This is the
most effective metric of how much value a usability group is adding. Running
studies is one thing, but if a study results in zero changes than either the
study was unnecessary or the results were ignored.
Often this ratio points out usability teams are better at generating data
than they are at getting anyone to do anything with it. Which suggests their
growth will come most from developing persuasion, storytelling,
communication and political skills more than learning new methodologies.
2) Number of requests for consultations and usability studies - This is a
reflection of how valued the usability team is perceived to be. If no one is
asking for your input, perception of value is low. If everyone is asking for
your input, and you can't meet demand, your perception is high. Should also
track, per group, a) when in their project cycle their request for help came
b) if they used your advice or not - more indicators of perceived value.
3) How often usability goals appear in the goals of project managers, team
leaders and even executives. Ideally a UX goal is simply one of several
project goals that the entire project team is expected to defend. If the
only organization with a UX goal is the UX team, something is wrong - the UX
team is set up to fail.
4) Work produced. This is easy to measure but has questionable value. # of
Reports written, # of studies done, etc. But it captures zero about the
impact or value of the work. Popular things like usability scorecards or
heuristic evaluations are effectively a kind of recommendation generator
(see #1 above) and are best measured in terms of their impact rather than
their quantity.
For design/creative it's harder:
The way designers are used varies so much it's harder to give one generic
answer. Managers and team leads always have highly subjective measures for
their own performance - so don't be afraid of having subjective measures for
designers (There is a good philosophical argument that all metrics are
subjective simply because someone has to pick which things to measure :).
1) Recommendations vs. implementations is always a good measure. However for
design it's more subjective, as what constitutes a design recommendation vs.
a prototype or a conversation is something you have to sort out. Still, the
balance should be on impact and effect on what goes out the door to
customers.
2) Initiatives vs. results. Designers in a proactive role should be
initiating feature, project and process designs into projects (e.g. The
drafting of UX guidelines, or a new metaphor for a new website). Did anyone
use them? How well were they used? Etc.) Even a subjective measure, by you
and other designers, of the impact of designer driven initiatives has value.
For example, for every quarter there should one design initiative per
designer, and your job as a team is to meet at the end of every quarter and
evaluate the results. Even subjective measures ("score from 1 to 5 on how
successful this was on the following attributes..." etc.) can be useful.
3) Requests vs. results. In more service oriented roles, how did the
engineer or manager requesting services feel their needs were met. Basic
customer satisfaction data can be collected here in much the same way you do
for actual (external) customers.
If you tell me more about the design work you're doing, and the nature of
the relationship (proactive/responsive) with the clients, and I'll have
better advice on the design side.
References:
I haven't done work on this stuff in awhile, but here's some working links
from an old pile of bookmarks. Sadly googling for "ux goals" brings up very
litte:
http://www.scottberkun.com/essays/27-the-art-of-usability-benchmarking/ -
This is a good way to make UX a team goal. The project leaders should have
the goal of raising benchmark scores with every release.
http://www.nigelbevan.com/papers/Classifying%20and%20selecting%20UX%20and%20
usability%20measures.pdf
-Scott
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Alan
Cox
Sent: Friday, March 06, 2009 1:25 PM
To: [email protected]
Subject: [IxDA Discuss] Design/UX goals in your company
I'm curious: what type of goals and metrics exist in your company that are
related to good user experience and good design? Do you have goals &
metrics that are company-wide, team-wide and individual?
________________________________________________________________
Welcome to the Interaction Design Association (IxDA)!
To post to this list ....... [email protected]
Unsubscribe ................ http://www.ixda.org/unsubscribe
List Guidelines ............ http://www.ixda.org/guidelines
List Help .................. http://www.ixda.org/help
