David A. Golden writes:
> Andy Lester wrote:
>
> > Why is there a scoreboard? Why do we care about rankings? Why is
> > it necessary to compare one measure to another? What purpose is
> > being served?
>
> Why is there XP on perlmonks? Or Karma on Slashdot?
Indeed, and those also have odd effects: rather than being pure measures
of users' abilities/reputations/whatever, their very existence changes
how some users behave, where they do things specifically to increase XP
rather than because they have intrinsic value.
In the UK the government a few years ago introduced league tables for
hosptials, taking things such as length of waiting lists into account.
This has had ridiculous consequences where a hospital will choose to
postpone a complex operation in favour of several simpler ones, because
the latter will remove more people from the waiting list and improve
their ranking in the league tables!
I know clothes shop workers who were given weekly sales targets, with a
bonus for meeting them. If they were close to the target towards the
end of the week they might buy a few garments themselves to trigger the
bonus, then bring those clothes back for a refund early the following
week. Or if they'd already met the target for one week, they might
start hoarding some sales tags and keeping them to enter into the
computer system the following week, to increase the chance of getting
that bonus too.
The fact that measurements are taking place changes how people behave.
Sometimes this only results in gaming the measurement system
("kwalitee") and actually having a negative affect on the thing that the
measurers really want improving ("quality").
So I largely agree with Andy's point. Except that I don't really mind:
anybody who's getting worked up about the kwalitee score of their
modules is taking kwalitee too seriously, and they deserve what they
get.
> I think the advantage of a scoreboard system is that metrics like this
> are a motivator. Rather than defining a qualitative standard of "good
> module style", CPANTS defines a quantitative standard and measures
> against it, Many programmers may well be motivated to improve their
> metrics, either for personal improvement or through competitive spirit.
Yup, but all the above apply even without a public chart.
> The advantage of a scoreboard is that it provides a peer benchmark,
Depends what "peer" means. For example, here's a journal entry
complaining that is_prereq is worthless because modules with high-level
functionality specifically aimed at being used in scripts rather than by
other modules will never achieve it:
http://use.perl.org/~ethan/journal/23949
I reckon that doesn't matter, and when considering which of several
modules to use, all the candidates are going to be of one type or the
other, so their relative kwalitees are comparable and useful. But the
leaderboard encourages apples-and-oranges comparisions between different
sorts of modules.
The above author is also frustrated that he's getting a non-perfect
score for is_prereq yet there isn't anything he can directly do about
this. That's true, but it shouldn't matter: kwalitee is trying to
approximate the quality of a module, and if a kwalitee-measured factor
has a correlation with quality then the causation of that factor is
obviously irrelevant.
Yet it apparently does matter to some people. The fact that there's a
public chart on which they appear with a lower ranking than others
through something that they can't change probably doesn't help here.
> We can/should debate the metrics, but not the philosophy of
> measurement.
It isn't the measurement that Andy was objecting to, but the
presentation of it as a per-user chart. As far as possible it should be
ensured that the measuring itself doesn't have negative consequences ...
Smylers
