On 09/12/13 14:03, Clint Byrum wrote:
Excerpts from Zane Bitter's message of 2013-12-09 09:52:25 -0800:
On 09/12/13 06:31, Steven Hardy wrote:
Hi all,

So I've been getting concerned about $subject recently, and based on some
recent discussions so have some other heat-core folks, so I wanted to start
a discussion where we can agree and communicate our expectations related to
nomination for heat-core membership (becuase we do need more core
reviewers):

The issues I have are:
- Russell's stats (while very useful) are being used by some projects as
    the principal metric related to -core membership (ref TripleO's monthly
    cull/name&shame, which I am opposed to btw).  This is in some cases
    encouraging some stats-seeking in our review process, IMO.

- Review quality can't be measured mechanically - we have some folks who
    contribute fewer, but very high quality reviews, and are also very active
    contributors (so knowledge of the codebase is not stale).  I'd like to
    see these people do more reviews, but removing people from core just
    because they drop below some arbitrary threshold makes no sense to me.

+1

Fun fact: due to the quirks of how Gerrit produces the JSON data dump,
it's not actually possible for the reviewstats tools to count +0
reviews. So, for example, one can juice one's review stats by actively
obstructing someone else's work (voting -1) when a friendly comment
would have sufficed. This is one of many ways in which metrics offer
perverse incentives.

Statistics can be useful. They can be particularly useful *in the
aggregate*. But as soon as you add a closed feedback loop you're no
longer measuring what you originally thought - mostly you're just
measuring the gain of the feedback loop.


I think I understand the psychology of stats and incentives, and I know
that this _may_ happen.

However, can we please be more careful about how this is referenced?
Your message above is suggesting the absolute _worst_ behavior from our
community. That is not what I expect, and I think anybody who was doing
that would be dealt with _swiftly_.

Sorry for the confusion, I wasn't trying to suggest that at all. FWIW I haven't noticed anyone gaming the stats (maybe I haven't been looking at enough reviews ;). What I have noticed is that every time I leave a +0 comment on a patch, I catch myself thinking "this won't look good on the stats" - and then I continue on regardless. If somebody who wasn't core but wanted to be were to -1 the patch instead in similar circumstances, then I wouldn't blame them in the least for responding to that incentive.

My point, and I think Steve's, is that we should be careful how we *use* the stats, so that folks won't feel this pressure. It's not at all about calling anybody out, but I apologise for not making that clearer.

cheers,
Zane.

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to