Re: [openstack-dev] [nova] gerrit based statistics

2014-10-09 Thread Daniel P. Berrange
On Wed, Oct 08, 2014 at 04:30:16PM -0700, Joe Gordon wrote:
 Recently there has been a lot of discussion around the development growing
 pains in nova. Instead of guessing about how bad some of the issues are, I
 tried to answer a few questions that may help us better understand the
 issues.

What time period did you calculate these stats over ? One whole dev
cycle ?

 Q: How many revisions does it take to merge a patch?
 
 Average: 6.76 revisions
 median: 4.0 revisions
 
 
 Q: How many rechecks/verifies does it take to merge a patch (ignoring
 rechecks where the same job failed before and after)?
 
 Average: 0.749 rechecks per patch revision
 median: 0.4285  rechecks per patch revision

These feel really low to me, but I guess that is a consequence of
aggregating over a long time period.

 For comparison here are the same results for tempest, which has a lot more
 gating tests:
 
 Average: 1.01591525738
 median: 0.6
 
 
 Q: How long does it take for a patch to get approved?
 
 Average: 28 days
 median: 11 days
 
 
 Q: How long does it take for a patch to get approved that touches
 'nova/virt/'?
 
 Average: 34 days
 median: 18 days
 
 
 When looking at these numbers two things stick out out:
 
 * We successfully use recheck an awful lot. More then I expected
 * Patches that touch 'nova/virt' take about 20% more time to land or about
 6 days. While that is definitely a difference, its smaller then I expected
 
 
 Dataset: last 800 patches in nova
 Code: https://github.com/jogo/gerrit-fun

Ah ok, so I think what would be more informative is to caculate the
stats on weekly blocks and graph it over 2 cycles. There are certainly
times when the problems are much worse than others and a long term
average will mask that to a large extent. It is the periods when it is
particularly bad that are leading to the high frustration amongst
contributors IMHO, so we must look at the worst cases.

For example look at the stats I produced for patch review comments per
core team reviewer:

  https://www.berrange.com/~dan/novacodeandspecsreviewrate.png

The gap betweeen good weeks and bad weeks is enourmous.


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] gerrit based statistics

2014-10-09 Thread Russell Bryant
On 10/08/2014 07:30 PM, Joe Gordon wrote:
 Recently there has been a lot of discussion around the development
 growing pains in nova. Instead of guessing about how bad some of the
 issues are, I tried to answer a few questions that may help us better
 understand the issues.
 
 
 Q: How many revisions does it take to merge a patch?
 
 Average: 6.76 revisions
 median: 4.0 revisions
 
 
 Q: How many rechecks/verifies does it take to merge a patch (ignoring
 rechecks where the same job failed before and after)?
 
 Average: 0.749 rechecks per patch revision
 median: 0.4285  rechecks per patch revision
 
 For comparison here are the same results for tempest, which has a lot
 more gating tests:
 
 Average: 1.01591525738
 median: 0.6
 
 
 Q: How long does it take for a patch to get approved?
 
 Average: 28 days
 median: 11 days
 
 
 Q: How long does it take for a patch to get approved that touches
 'nova/virt/'?
 
 Average: 34 days
 median: 18 days
 
 
 When looking at these numbers two things stick out out:
 
 * We successfully use recheck an awful lot. More then I expected
 * Patches that touch 'nova/virt' take about 20% more time to land or
 about 6 days. While that is definitely a difference, its smaller then I
 expected
 
 
 Dataset: last 800 patches in nova
 Code: https://github.com/jogo/gerrit-fun

Some related stats on open code reviews:

http://russellbryant.net/openstack-stats/nova-openreviews.html

I don't have historical data, which would be really useful.  However,
based on my memory and an old ML post [1], these numbers have tripled
for Nova since mid 2013.

[1] http://lists.openstack.org/pipermail/openstack-dev/2013-June/011043.html

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] gerrit based statistics

2014-10-09 Thread Matt Riedemann



On 10/8/2014 9:21 PM, Joe Gordon wrote:



On Wed, Oct 8, 2014 at 6:12 PM, Michael Still mi...@stillhq.com
mailto:mi...@stillhq.com wrote:

On Thu, Oct 9, 2014 at 11:58 AM, Joe Gordon joe.gord...@gmail.com
mailto:joe.gord...@gmail.com wrote:
  On Wed, Oct 8, 2014 at 4:30 PM, Joe Gordon joe.gord...@gmail.com
mailto:joe.gord...@gmail.com wrote:
 
  Recently there has been a lot of discussion around the
development growing
  pains in nova. Instead of guessing about how bad some of the
issues are, I
  tried to answer a few questions that may help us better
understand the
  issues.
 
 
  Q: How many revisions does it take to merge a patch?
 
  Average: 6.76 revisions
  median: 4.0 revisions
 
 
  Q: How many rechecks/verifies does it take to merge a patch
(ignoring
  rechecks where the same job failed before and after)?
 
  Average: 0.749 rechecks per patch revision
  median: 0.4285  rechecks per patch revision
 
  For comparison here are the same results for tempest, which has
a lot more
  gating tests:
 
  Average: 1.01591525738
  median: 0.6
 
 
  Q: How long does it take for a patch to get approved?
 
  Average: 28 days
  median: 11 days
 
 
  Q: How long does it take for a patch to get approved that touches
  'nova/virt/'?
 
  Average: 34 days
  median: 18 days
 
 
  To expand on these numbers, same results for last 6 months of
commits:
 
  all of nova (1723 patches):
  Average: 28.8
  median: 11.0
 
  nova/virt (476 patches):
   Average: 34.5

I think it would be interesting to break this up by driver
directory... Are there drivers which take longer to land code for than
others?


Like this?

subtree: None (1724 patches):
Average: 28.7
median: 11.0
subtree: nova/virt/ (476 patches):
Average: 34.5
median: 18.0
subtree: nova/virt/hyperv/ (38 patches):
Average: 46.8
median: 33.0
subtree: nova/virt/libvirt/ (224 patches):
Average: 35.9
median: 18.0
subtree: nova/virt/xenapi/ (72 patches):
Average: 39.5
median: 20.0
subtree: nova/virt/vmwareapi/ (134 patches):
Average: 38.7
median: 26.0

 When looking at these numbers two things stick out out:

 * We successfully use recheck an awful lot. More then I expected
 * Patches that touch 'nova/virt' take about 20% more time to land or 
about
 6 days. While that is definitely a difference, its smaller then I 
expected


 Dataset: last 800 patches in nova
 Code:https://github.com/jogo/gerrit-fun



  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
mailto:OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 

Michael

--
Rackspace Australia

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
mailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



That's the distribution, percentage-wise, I'd expect to see for the 4 
in-tree virt drivers.


--

Thanks,

Matt Riedemann


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] gerrit based statistics

2014-10-08 Thread Joe Gordon
On Wed, Oct 8, 2014 at 4:30 PM, Joe Gordon joe.gord...@gmail.com wrote:

 Recently there has been a lot of discussion around the development growing
 pains in nova. Instead of guessing about how bad some of the issues are, I
 tried to answer a few questions that may help us better understand the
 issues.


 Q: How many revisions does it take to merge a patch?

 Average: 6.76 revisions
 median: 4.0 revisions


 Q: How many rechecks/verifies does it take to merge a patch (ignoring
 rechecks where the same job failed before and after)?

 Average: 0.749 rechecks per patch revision
 median: 0.4285  rechecks per patch revision

 For comparison here are the same results for tempest, which has a lot more
 gating tests:

 Average: 1.01591525738
 median: 0.6


 Q: How long does it take for a patch to get approved?

 Average: 28 days
 median: 11 days


 Q: How long does it take for a patch to get approved that touches
 'nova/virt/'?

 Average: 34 days
 median: 18 days


To expand on these numbers, same results for last 6 months of commits:

all of nova (1723 patches):
Average: 28.8
median: 11.0

nova/virt (476 patches):
 Average: 34.5




 When looking at these numbers two things stick out out:

 * We successfully use recheck an awful lot. More then I expected
 * Patches that touch 'nova/virt' take about 20% more time to land or about
 6 days. While that is definitely a difference, its smaller then I expected


 Dataset: last 800 patches in nova
 Code: https://github.com/jogo/gerrit-fun

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] gerrit based statistics

2014-10-08 Thread Michael Still
On Thu, Oct 9, 2014 at 11:58 AM, Joe Gordon joe.gord...@gmail.com wrote:
 On Wed, Oct 8, 2014 at 4:30 PM, Joe Gordon joe.gord...@gmail.com wrote:

 Recently there has been a lot of discussion around the development growing
 pains in nova. Instead of guessing about how bad some of the issues are, I
 tried to answer a few questions that may help us better understand the
 issues.


 Q: How many revisions does it take to merge a patch?

 Average: 6.76 revisions
 median: 4.0 revisions


 Q: How many rechecks/verifies does it take to merge a patch (ignoring
 rechecks where the same job failed before and after)?

 Average: 0.749 rechecks per patch revision
 median: 0.4285  rechecks per patch revision

 For comparison here are the same results for tempest, which has a lot more
 gating tests:

 Average: 1.01591525738
 median: 0.6


 Q: How long does it take for a patch to get approved?

 Average: 28 days
 median: 11 days


 Q: How long does it take for a patch to get approved that touches
 'nova/virt/'?

 Average: 34 days
 median: 18 days


 To expand on these numbers, same results for last 6 months of commits:

 all of nova (1723 patches):
 Average: 28.8
 median: 11.0

 nova/virt (476 patches):
  Average: 34.5

I think it would be interesting to break this up by driver
directory... Are there drivers which take longer to land code for than
others?

 When looking at these numbers two things stick out out:

 * We successfully use recheck an awful lot. More then I expected
 * Patches that touch 'nova/virt' take about 20% more time to land or about
 6 days. While that is definitely a difference, its smaller then I expected


 Dataset: last 800 patches in nova
 Code: https://github.com/jogo/gerrit-fun



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Michael

-- 
Rackspace Australia

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] gerrit based statistics

2014-10-08 Thread Joe Gordon
On Wed, Oct 8, 2014 at 6:12 PM, Michael Still mi...@stillhq.com wrote:

 On Thu, Oct 9, 2014 at 11:58 AM, Joe Gordon joe.gord...@gmail.com wrote:
  On Wed, Oct 8, 2014 at 4:30 PM, Joe Gordon joe.gord...@gmail.com
 wrote:
 
  Recently there has been a lot of discussion around the development
 growing
  pains in nova. Instead of guessing about how bad some of the issues
 are, I
  tried to answer a few questions that may help us better understand the
  issues.
 
 
  Q: How many revisions does it take to merge a patch?
 
  Average: 6.76 revisions
  median: 4.0 revisions
 
 
  Q: How many rechecks/verifies does it take to merge a patch (ignoring
  rechecks where the same job failed before and after)?
 
  Average: 0.749 rechecks per patch revision
  median: 0.4285  rechecks per patch revision
 
  For comparison here are the same results for tempest, which has a lot
 more
  gating tests:
 
  Average: 1.01591525738
  median: 0.6
 
 
  Q: How long does it take for a patch to get approved?
 
  Average: 28 days
  median: 11 days
 
 
  Q: How long does it take for a patch to get approved that touches
  'nova/virt/'?
 
  Average: 34 days
  median: 18 days
 
 
  To expand on these numbers, same results for last 6 months of commits:
 
  all of nova (1723 patches):
  Average: 28.8
  median: 11.0
 
  nova/virt (476 patches):
   Average: 34.5

 I think it would be interesting to break this up by driver
 directory... Are there drivers which take longer to land code for than
 others?


Like this?

subtree: None (1724 patches):
Average: 28.7
median: 11.0
subtree: nova/virt/ (476 patches):
Average: 34.5
median: 18.0
subtree: nova/virt/hyperv/ (38 patches):
Average: 46.8
median: 33.0
subtree: nova/virt/libvirt/ (224 patches):
Average: 35.9
median: 18.0
subtree: nova/virt/xenapi/ (72 patches):
Average: 39.5
median: 20.0
subtree: nova/virt/vmwareapi/ (134 patches):
Average: 38.7
median: 26.0



  When looking at these numbers two things stick out out:
 
  * We successfully use recheck an awful lot. More then I expected
  * Patches that touch 'nova/virt' take about 20% more time to land or
 about
  6 days. While that is definitely a difference, its smaller then I
 expected
 
 
  Dataset: last 800 patches in nova
  Code: https://github.com/jogo/gerrit-fun
 
 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 

 Michael

 --
 Rackspace Australia

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev