Re: [openstack-dev] [cinder] The Absurdity of the Milestone-1 Deadline for Drivers

[Ben Swartzlander]

On 09/30/2015 12:11 PM, Mike Perez wrote:

On 13:29 Sep 28, Ben Swartzlander wrote:

I've always thought it was a bit strange to require new drivers to
merge by milestone 1. I think I understand the motivations of the
policy. The main motivation was to free up reviewers to review "other
things" and this policy guarantees that for 75% of the release
reviewers don't have to review new drivers. The other motivation was
to prevent vendors from turning up at the last minute with crappy
drivers that needed a ton of work, by encouraging them to get started
earlier, or forcing them to wait until the next cycle.

I believe that the deadline actually does more harm than good.

First of all, to those that don't want to spend time on driver
reviews, there are other solutions to that problem. Some people do
want to review the drivers, and those who don't can simply ignore
them and spend time on what they care about. I've heard people who
spend time on driver reviews say that the milestone-1 deadline
doesn't mean they spend less time reviewing drivers overall, it just
all gets crammed into the beginning of each release. It should be
obvious that setting a deadline doesn't actually affect the amount of
reviewer effort, it just concentrates that effort.

Some bad assumptions here:

* Nobody said they didn't want to review drivers.

* "Crammed" is completely an incorrect word here. An example with last release,
   we only had 3/17 drivers trying to get in during the last week of the
   milestone [1]. I don't think you're very active in Cinder to really judge how
   well the team has worked together to get these drivers in a timely way with
   vendors.


There are fair points. No argument. I think I managed to obscure my main 
point with too much assumptions and rhetoric though.


Let me restate my argument as simply as possible.

Drivers are relatively low risk to the project. They're a lot of work to 
review due to the size, but the risk of missing bugs is small because 
those bugs will affect only the users who choose to deploy the given 
driver. Also drivers are well understood, so the process of reviewing 
them is straightforward.


New features are high risk. Even a small change to the manager or API 
code can have dramatic impact on all users of Cinder. Larger changes 
that touch multiple modules in different areas must be reviewed by 
people who understand all of Cinder just to get basic assurance that 
they do what they say. Finding bugs in these kinds of changes is tricky. 
Reading the code only gets you so far, and automated testing only 
scratches the surface. You have to run the code and try it out. These 
things take time and core team time is a limited and precious resource.


Now, if you have some high risk changes and some low risk changes, which 
do you think it makes sense to work on early in the release, and which 
do you think is safe to merge at the last minute? I asked myself that 
question and decided that I'd rather to high risk stuff early and low 
risk stuff later. Based on that belief, I'm making a suggestion to move 
the deadlines around.




The argument about crappy code is also a lot weaker now that there
are CI requirements which force vendors to spend much more time up
front and clear a much higher quality bar before the driver is even
considered for merging. Drivers that aren't ready for merge can
always be deferred to a later release, but it seems weird to defer
drivers that are high quality just because they're submitted during
milestones 2 or 3.

"Crappy code" ... I don't know where that's coming from. If anything, CI has
helped get the drivers in faster to get rid of what you call "cramming".



That's good. If that's true, then I would think it supports an argument 
that the deadlines are unnecessary because the underlying problem 
(limited reviewer time) has been solved.




All the the above is just my opinion though, and you shouldn't care
about my opinions, as I don't do much coding and reviewing in Cinder.
There is a real reason I'm writing this email...

In Manila we added some major new features during Liberty. All of the
new features merged in the last week of L-3. It was a nightmare of
merge conflicts and angry core reviewers, and many contributors
worked through a holiday weekend to bring the release together. While
asking myself how we can avoid such a situation in the future, it
became clear to me that bigger features need to merge earlier -- the
earlier the better.

When I look at the release timeline, and ask myself when is the best
time to merge new major features, and when is the best time to merge
new drivers, it seems obvious that *features* need to happen early
and drivers should come *later*. New major features require FAR more
review time than new drivers, and they require testing, and even
after they merge they cause merge conflicts that everyone else has to
deal with. Better that that works happens in milestones 1 and 2 than
right before feature freeze. New 

Re: [openstack-dev] [cinder] The Absurdity of the Milestone-1 Deadline for Drivers

[Mike Perez]

On 13:29 Sep 28, Ben Swartzlander wrote:
> I've always thought it was a bit strange to require new drivers to
> merge by milestone 1. I think I understand the motivations of the
> policy. The main motivation was to free up reviewers to review "other
> things" and this policy guarantees that for 75% of the release
> reviewers don't have to review new drivers. The other motivation was
> to prevent vendors from turning up at the last minute with crappy
> drivers that needed a ton of work, by encouraging them to get started
> earlier, or forcing them to wait until the next cycle.
> 
> I believe that the deadline actually does more harm than good.
> 
> First of all, to those that don't want to spend time on driver
> reviews, there are other solutions to that problem. Some people do
> want to review the drivers, and those who don't can simply ignore
> them and spend time on what they care about. I've heard people who
> spend time on driver reviews say that the milestone-1 deadline
> doesn't mean they spend less time reviewing drivers overall, it just
> all gets crammed into the beginning of each release. It should be
> obvious that setting a deadline doesn't actually affect the amount of
> reviewer effort, it just concentrates that effort.

Some bad assumptions here:

* Nobody said they didn't want to review drivers.

* "Crammed" is completely an incorrect word here. An example with last release,
  we only had 3/17 drivers trying to get in during the last week of the
  milestone [1]. I don't think you're very active in Cinder to really judge how
  well the team has worked together to get these drivers in a timely way with
  vendors.

> The argument about crappy code is also a lot weaker now that there
> are CI requirements which force vendors to spend much more time up
> front and clear a much higher quality bar before the driver is even
> considered for merging. Drivers that aren't ready for merge can
> always be deferred to a later release, but it seems weird to defer
> drivers that are high quality just because they're submitted during
> milestones 2 or 3.

"Crappy code" ... I don't know where that's coming from. If anything, CI has
helped get the drivers in faster to get rid of what you call "cramming".

> All the the above is just my opinion though, and you shouldn't care
> about my opinions, as I don't do much coding and reviewing in Cinder.
> There is a real reason I'm writing this email...
>
> In Manila we added some major new features during Liberty. All of the
> new features merged in the last week of L-3. It was a nightmare of
> merge conflicts and angry core reviewers, and many contributors
> worked through a holiday weekend to bring the release together. While
> asking myself how we can avoid such a situation in the future, it
> became clear to me that bigger features need to merge earlier -- the
> earlier the better.
> 
> When I look at the release timeline, and ask myself when is the best
> time to merge new major features, and when is the best time to merge
> new drivers, it seems obvious that *features* need to happen early
> and drivers should come *later*. New major features require FAR more
> review time than new drivers, and they require testing, and even
> after they merge they cause merge conflicts that everyone else has to
> deal with. Better that that works happens in milestones 1 and 2 than
> right before feature freeze. New drivers can come in right before
> feature freeze as far as I'm concerned. Drivers don't cause merge
> conflicts, and drivers don't need huge amounts of testing (presumably
> the CI system ensure some level of quality).
> 
> It also occurs to me that new features which require driver
> implementation (hello replication!) *really* should go in during the
> first milestone so that drivers have time to implement the feature
> during the same release.

I disagree. You're under the assumption that there is an intention of getting
a feature being worked in Liberty, to be ready for Liberty.

No.

I've expressed this numerous times at the Cinder midcycle sprint you attended
that I did not want to see drivers working on replication in their driver.

> So I'm asking the Cinder core team to reconsider the milestone-1
> deadline for drivers, and to change it to a deadline for new major
> features (in milestone-1 or milestone-2), and to allow drivers to
> merge whenever*. This is the same pitch I'll be making to the Manila
> core team. I've been considering this idea for a few weeks now but I
> wanted to wait until after PTL elections to suggest it here.

During the release, a feature can be worked on, but adoption in drivers can be
difficult. Since just about every driver is behind on potential features, I'd
rather see driver maintainers focused on those features that have been ready
for sometime, not what we just merged a week ago.

There is no good reason to rush and suffer quality for the sake of vendors
wanting the latest and greatest feature. We're more mature than that.

[openstack-dev] [cinder] The Absurdity of the Milestone-1 Deadline for Drivers

[Ben Swartzlander]

I've always thought it was a bit strange to require new drivers to merge 
by milestone 1. I think I understand the motivations of the policy. The 
main motivation was to free up reviewers to review "other things" and 
this policy guarantees that for 75% of the release reviewers don't have 
to review new drivers. The other motivation was to prevent vendors from 
turning up at the last minute with crappy drivers that needed a ton of 
work, by encouraging them to get started earlier, or forcing them to 
wait until the next cycle.


I believe that the deadline actually does more harm than good.

First of all, to those that don't want to spend time on driver reviews, 
there are other solutions to that problem. Some people do want to review 
the drivers, and those who don't can simply ignore them and spend time 
on what they care about. I've heard people who spend time on driver 
reviews say that the milestone-1 deadline doesn't mean they spend less 
time reviewing drivers overall, it just all gets crammed into the 
beginning of each release. It should be obvious that setting a deadline 
doesn't actually affect the amount of reviewer effort, it just 
concentrates that effort.


The argument about crappy code is also a lot weaker now that there are 
CI requirements which force vendors to spend much more time up front and 
clear a much higher quality bar before the driver is even considered for 
merging. Drivers that aren't ready for merge can always be deferred to a 
later release, but it seems weird to defer drivers that are high quality 
just because they're submitted during milestones 2 or 3.


All the the above is just my opinion though, and you shouldn't care 
about my opinions, as I don't do much coding and reviewing in Cinder. 
There is a real reason I'm writing this email...


In Manila we added some major new features during Liberty. All of the 
new features merged in the last week of L-3. It was a nightmare of merge 
conflicts and angry core reviewers, and many contributors worked through 
a holiday weekend to bring the release together. While asking myself how 
we can avoid such a situation in the future, it became clear to me that 
bigger features need to merge earlier -- the earlier the better.


When I look at the release timeline, and ask myself when is the best 
time to merge new major features, and when is the best time to merge new 
drivers, it seems obvious that *features* need to happen early and 
drivers should come *later*. New major features require FAR more review 
time than new drivers, and they require testing, and even after they 
merge they cause merge conflicts that everyone else has to deal with. 
Better that that works happens in milestones 1 and 2 than right before 
feature freeze. New drivers can come in right before feature freeze as 
far as I'm concerned. Drivers don't cause merge conflicts, and drivers 
don't need huge amounts of testing (presumably the CI system ensure some 
level of quality).


It also occurs to me that new features which require driver 
implementation (hello replication!) *really* should go in during the 
first milestone so that drivers have time to implement the feature 
during the same release.


So I'm asking the Cinder core team to reconsider the milestone-1 
deadline for drivers, and to change it to a deadline for new major 
features (in milestone-1 or milestone-2), and to allow drivers to merge 
whenever*. This is the same pitch I'll be making to the Manila core 
team. I've been considering this idea for a few weeks now but I wanted 
to wait until after PTL elections to suggest it here.


-Ben Swartzlander


* I don't actually care if/when there is a driver deadline, what I care 
about is that reviewers are free during M-1 to work on reviewing/testing 
of features. The easiest way to achieve that seems to be moving the 
driver deadline.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] The Absurdity of the Milestone-1 Deadline for Drivers

[Duncan Thomas]

I can definitely see your logic, but we've a history in cinder of vendors
trying to cram drivers in at the last minute which we very much wanted to
stop dead. I might suggest that the second milestone, rather than the first
might be a better one to dedicate to driver reviews...

An interesting thing is that, from the point of view of cinder core, we
don't need more drivers. We've drivers coming out of our ears. Sure it is
incrementally better to get more drivers, but we've got far more to gain
from harnessing the developers of companies who want to merge new drivers
for core work (or ci improvements, or translation or docs improvements or
whatever) than we do from yet another driver. Increasing the bar for enter
slightly does us no major harm that I can tell - there's a clear benefit
from having your driver in tree, so if there's a small 'tax' to get added
then I suspect we can benefit from it substantially.

Definitely worth reviewing deadlines and other bureaucracy occasionally and
working out if it is still serving us well.

Cheers
On 28 Sep 2015 20:32, "Ben Swartzlander"  wrote:

> I've always thought it was a bit strange to require new drivers to merge
> by milestone 1. I think I understand the motivations of the policy. The
> main motivation was to free up reviewers to review "other things" and this
> policy guarantees that for 75% of the release reviewers don't have to
> review new drivers. The other motivation was to prevent vendors from
> turning up at the last minute with crappy drivers that needed a ton of
> work, by encouraging them to get started earlier, or forcing them to wait
> until the next cycle.
>
> I believe that the deadline actually does more harm than good.
>
> First of all, to those that don't want to spend time on driver reviews,
> there are other solutions to that problem. Some people do want to review
> the drivers, and those who don't can simply ignore them and spend time on
> what they care about. I've heard people who spend time on driver reviews
> say that the milestone-1 deadline doesn't mean they spend less time
> reviewing drivers overall, it just all gets crammed into the beginning of
> each release. It should be obvious that setting a deadline doesn't actually
> affect the amount of reviewer effort, it just concentrates that effort.
>
> The argument about crappy code is also a lot weaker now that there are CI
> requirements which force vendors to spend much more time up front and clear
> a much higher quality bar before the driver is even considered for merging.
> Drivers that aren't ready for merge can always be deferred to a later
> release, but it seems weird to defer drivers that are high quality just
> because they're submitted during milestones 2 or 3.
>
> All the the above is just my opinion though, and you shouldn't care about
> my opinions, as I don't do much coding and reviewing in Cinder. There is a
> real reason I'm writing this email...
>
> In Manila we added some major new features during Liberty. All of the new
> features merged in the last week of L-3. It was a nightmare of merge
> conflicts and angry core reviewers, and many contributors worked through a
> holiday weekend to bring the release together. While asking myself how we
> can avoid such a situation in the future, it became clear to me that bigger
> features need to merge earlier -- the earlier the better.
>
> When I look at the release timeline, and ask myself when is the best time
> to merge new major features, and when is the best time to merge new
> drivers, it seems obvious that *features* need to happen early and drivers
> should come *later*. New major features require FAR more review time than
> new drivers, and they require testing, and even after they merge they cause
> merge conflicts that everyone else has to deal with. Better that that works
> happens in milestones 1 and 2 than right before feature freeze. New drivers
> can come in right before feature freeze as far as I'm concerned. Drivers
> don't cause merge conflicts, and drivers don't need huge amounts of testing
> (presumably the CI system ensure some level of quality).
>
> It also occurs to me that new features which require driver implementation
> (hello replication!) *really* should go in during the first milestone so
> that drivers have time to implement the feature during the same release.
>
> So I'm asking the Cinder core team to reconsider the milestone-1 deadline
> for drivers, and to change it to a deadline for new major features (in
> milestone-1 or milestone-2), and to allow drivers to merge whenever*. This
> is the same pitch I'll be making to the Manila core team. I've been
> considering this idea for a few weeks now but I wanted to wait until after
> PTL elections to suggest it here.
>
> -Ben Swartzlander
>
>
> * I don't actually care if/when there is a driver deadline, what I care
> about is that reviewers are free during M-1 to work on reviewing/testing of
> features. The easiest way to 

Re: [openstack-dev] [cinder] The Absurdity of the Milestone-1 Deadline for Drivers

[Sean McGinnis]

On Mon, Sep 28, 2015 at 12:13:04PM -0600, John Griffith wrote:
> On Mon, Sep 28, 2015 at 11:29 AM, Ben Swartzlander 
> wrote:
> 
> > I've always thought it was a bit strange to require new drivers to merge
> > by milestone 1. I think I understand the motivations of the policy. The
> > main motivation was to free up reviewers to review "other things" and this
> > policy guarantees that for 75% of the release reviewers don't have to
> > review new drivers. The other motivation was to prevent vendors from
> > turning up at the last minute with crappy drivers that needed a ton of
> > work, by encouraging them to get started earlier, or forcing them to wait
> > until the next cycle.
> >
> 
> ​Yep, these were some of the ideas behind it but the first milestone did
> for sure create some consequences.​
> 
> 
> >
> > I believe that the deadline actually does more harm than good.
> >
> 
> ​In retrospect I'd agree with you on this.  We ended up spending our major
> focus for the first milestone on nothing but drivers which I think looking
> back wasn't so good.  But to be fair, we try things, see how they work,
> revisit and move on.  Which is the plan last I checked (there's a proposal
> to talk about some of this at the summit in Tokyo).​
> 

We will have more discussion on this for sure, but I figure I should
chime in with some of my thoughts.

I definitely do want to reconsider our deadlines. There are going to
be challenges no matter what point in the cycle we set for things like
driver submissions, but as John said, we need to try things and see how
it works. I saw a lot of logic in moving new drivers to the first
milestone, but I don't think it worked out as well as we had hoped it
would.

The biggest problem I see is it made the drivers a major focus for the
first part of the cycle. It seemed to me that that distracted a lot of
focus from core functionality. It got that part out of the way (mostly)
but it sort of disrupted the momentum from things discussed at the
Summit.

> 
> >
> > First of all, to those that don't want to spend time on driver reviews,
> > there are other solutions to that problem. Some people do want to review
> > the drivers, and those who don't can simply ignore them and spend time on
> > what they care about. I've heard people who spend time on driver reviews
> > say that the milestone-1 deadline doesn't mean they spend less time
> > reviewing drivers overall, it just all gets crammed into the beginning of
> > each release. It should be obvious that setting a deadline doesn't actually
> > affect the amount of reviewer effort, it just concentrates that effort.
> >
> 

I do think this is a very valid point.

I certainly don't have all the answers. I'm looking forward to the
discussions around this to get ideas of how to make things better. But I
do think we need to try something else to find a better way. Maybe we'll
end up back to where we are, but I do think it warrants more discussion.

Sean

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] The Absurdity of the Milestone-1 Deadline for Drivers

[Ben Swartzlander]

On 09/28/2015 02:42 PM, Walter A. Boring IV wrote:

On 09/28/2015 10:29 AM, Ben Swartzlander wrote:
I've always thought it was a bit strange to require new drivers to 
merge by milestone 1. I think I understand the motivations of the 
policy. The main motivation was to free up reviewers to review "other 
things" and this policy guarantees that for 75% of the release 
reviewers don't have to review new drivers. The other motivation was 
to prevent vendors from turning up at the last minute with crappy 
drivers that needed a ton of work, by encouraging them to get started 
earlier, or forcing them to wait until the next cycle.


I believe that the deadline actually does more harm than good.
But harm to whom?   It certainly puts the pressure on driver 
developers to make sure they get involved in the Cinder community and 
get aware of when the deadlines are.
I believe it simply shifts the time in which drivers get into tree. My 
$0.02 of opinion is, that if a new driver developer misses the 
milestone, then they have the rest of the release to work on getting 
CI up and running and ready to go for the next release.   I'm not sure 
I see the harm to the Cinder community or the project.   It's a 
deadline that a driver developer has to be aware of and compensate 
for.  We've had how many drivers land in the last 2 releases using 
this requirement?  I believe it's somewhere of 20+ drivers?




Walt I think you missed the point of my suggestion. I don't actually 
care when drivers go in. What I care about is that features land early 
so they can get appropriate review time and testing time, and so that 
merge conflicts can be dealt with. It seems to me that many people 
review nothing but drivers during milestone-1 so I suggest moving the 
deadline so those reviews can happen later. The 3rd milestone is when 
there should be no big architectural changes going on and is IMO the 
safest time to merge drivers. So the harm of the milestone-1 deadline is 
that is sucks all the oxygen out of the room, delaying features to 
milestone-2.


-Ben




First of all, to those that don't want to spend time on driver 
reviews, there are other solutions to that problem. Some people do 
want to review the drivers, and those who don't can simply ignore 
them and spend time on what they care about. I've heard people who 
spend time on driver reviews say that the milestone-1 deadline 
doesn't mean they spend less time reviewing drivers overall, it just 
all gets crammed into the beginning of each release. It should be 
obvious that setting a deadline doesn't actually affect the amount of 
reviewer effort, it just concentrates that effort.


The argument about crappy code is also a lot weaker now that there 
are CI requirements which force vendors to spend much more time up 
front and clear a much higher quality bar before the driver is even 
considered for merging. Drivers that aren't ready for merge can 
always be deferred to a later release, but it seems weird to defer 
drivers that are high quality just because they're submitted during 
milestones 2 or 3.
I disagree here.  CI doesn't prevent you from having a crappy driver.  
Your driver just needs to pass CI tests.  CI ensures that your driver 
works, but doesn't ensure that it
really meats the core reviewers standards for code.  Do we care? I 
think we do.  Having drivers talk directly to the db, or FC drivers 
missing the FCZM decorators for auto zoning, etc.




All the the above is just my opinion though, and you shouldn't care 
about my opinions, as I don't do much coding and reviewing in Cinder. 
There is a real reason I'm writing this email...


In Manila we added some major new features during Liberty. All of the 
new features merged in the last week of L-3. It was a nightmare of 
merge conflicts and angry core reviewers, and many contributors 
worked through a holiday weekend to bring the release together. While 
asking myself how we can avoid such a situation in the future, it 
became clear to me that bigger features need to merge earlier -- the 
earlier the better.


When I look at the release timeline, and ask myself when is the best 
time to merge new major features, and when is the best time to merge 
new drivers, it seems obvious that *features* need to happen early 
and drivers should come *later*. New major features require FAR more 
review time than new drivers, and they require testing, and even 
after they merge they cause merge conflicts that everyone else has to 
deal with. Better that that works happens in milestones 1 and 2 than 
right before feature freeze. New drivers can come in right before 
feature freeze as far as I'm concerned. Drivers don't cause merge 
conflicts, and drivers don't need huge amounts of testing (presumably 
the CI system ensure some level of quality).


It also occurs to me that new features which require driver 
implementation (hello replication!) *really* should go in during the 
first milestone so that drivers have time to 

Re: [openstack-dev] [cinder] The Absurdity of the Milestone-1 Deadline for Drivers

[John Griffith]

On Mon, Sep 28, 2015 at 12:11 PM, Duncan Thomas 
wrote:

> I can definitely see your logic, but we've a history in cinder of vendors
> trying to cram drivers in at the last minute which we very much wanted to
> stop dead. I might suggest that the second milestone, rather than the first
> might be a better one to dedicate to driver reviews...
>
​I guess we're not waiting until the summit :)  So honestly I think the
whole driver freeze thing should just be a part of the regular
feature-freeze rules and requirements.  If the code gets submitted late;
well the submitter runs the risk of it not getting reviewed and not making
it.  That's life, and no amount of PM's on IRC or email pleading for a
review really sway me in those cases.

We've given this weird expectation to folks that "if you submit X by date
Y, we guarantee it'll merge" which is not only inaccurate, but completely
unfair.  It needs to be clear that there is a 6 month (actually more like 4
or 5) cycle to get your code in.  The longer you wait, the less likely
everything will get reviewed and merged.​  Sorry, but that's how it goes; I
personally stopped treating drivers as 'special' submissions a while ago
and have no intention of going back to pretending they're something
"special".  They're just another feature add IMHO.


> An interesting thing is that, from the point of view of cinder core, we
> don't need more drivers. We've drivers coming out of our ears.
>

​True that!!!  What are we at like, 80 or something now?​

​  Anybody interested in this topic come chat with me at the summit
(shameless plug for Griffith's agendas on removing drivers from
cinder/volume/drivers..., or at least coming up with a different method of
implementing them)  :)

It's a hard problem, no easy answer.  I also argue that "good" drivers do
make Cinder better; but we shouldn't make calls on what's good and bad
other than code review.

Sure it is incrementally better to get more drivers, but we've got far more
> to gain from harnessing the developers of companies who want to merge new
> drivers for core work (or ci improvements, or translation or docs
> improvements or whatever) than we do from yet another driver. Increasing
> the bar for enter slightly does us no major harm that I can tell - there's
> a clear benefit from having your driver in tree, so if there's a small
> 'tax' to get added then I suspect we can benefit from it substantially.
>
> Definitely worth reviewing deadlines and other bureaucracy occasionally
> and working out if it is still serving us well.
> ​
>
>
>
Cheers
> On 28 Sep 2015 20:32, "Ben Swartzlander"  wrote:
>
>> I've always thought it was a bit strange to require new drivers to merge
>> by milestone 1. I think I understand the motivations of the policy. The
>> main motivation was to free up reviewers to review "other things" and this
>> policy guarantees that for 75% of the release reviewers don't have to
>> review new drivers. The other motivation was to prevent vendors from
>> turning up at the last minute with crappy drivers that needed a ton of
>> work, by encouraging them to get started earlier, or forcing them to wait
>> until the next cycle.
>>
>> I believe that the deadline actually does more harm than good.
>>
>> First of all, to those that don't want to spend time on driver reviews,
>> there are other solutions to that problem. Some people do want to review
>> the drivers, and those who don't can simply ignore them and spend time on
>> what they care about. I've heard people who spend time on driver reviews
>> say that the milestone-1 deadline doesn't mean they spend less time
>> reviewing drivers overall, it just all gets crammed into the beginning of
>> each release. It should be obvious that setting a deadline doesn't actually
>> affect the amount of reviewer effort, it just concentrates that effort.
>>
>> The argument about crappy code is also a lot weaker now that there are CI
>> requirements which force vendors to spend much more time up front and clear
>> a much higher quality bar before the driver is even considered for merging.
>> Drivers that aren't ready for merge can always be deferred to a later
>> release, but it seems weird to defer drivers that are high quality just
>> because they're submitted during milestones 2 or 3.
>>
>> All the the above is just my opinion though, and you shouldn't care about
>> my opinions, as I don't do much coding and reviewing in Cinder. There is a
>> real reason I'm writing this email...
>>
>> In Manila we added some major new features during Liberty. All of the new
>> features merged in the last week of L-3. It was a nightmare of merge
>> conflicts and angry core reviewers, and many contributors worked through a
>> holiday weekend to bring the release together. While asking myself how we
>> can avoid such a situation in the future, it became clear to me that bigger
>> features need to merge earlier -- the earlier the better.
>>
>> When I 

Re: [openstack-dev] [cinder] The Absurdity of the Milestone-1 Deadline for Drivers

[John Griffith]

On Mon, Sep 28, 2015 at 11:29 AM, Ben Swartzlander 
wrote:

> I've always thought it was a bit strange to require new drivers to merge
> by milestone 1. I think I understand the motivations of the policy. The
> main motivation was to free up reviewers to review "other things" and this
> policy guarantees that for 75% of the release reviewers don't have to
> review new drivers. The other motivation was to prevent vendors from
> turning up at the last minute with crappy drivers that needed a ton of
> work, by encouraging them to get started earlier, or forcing them to wait
> until the next cycle.
>

​Yep, these were some of the ideas behind it but the first milestone did
for sure create some consequences.​


>
> I believe that the deadline actually does more harm than good.
>

​In retrospect I'd agree with you on this.  We ended up spending our major
focus for the first milestone on nothing but drivers which I think looking
back wasn't so good.  But to be fair, we try things, see how they work,
revisit and move on.  Which is the plan last I checked (there's a proposal
to talk about some of this at the summit in Tokyo).​


>
> First of all, to those that don't want to spend time on driver reviews,
> there are other solutions to that problem. Some people do want to review
> the drivers, and those who don't can simply ignore them and spend time on
> what they care about. I've heard people who spend time on driver reviews
> say that the milestone-1 deadline doesn't mean they spend less time
> reviewing drivers overall, it just all gets crammed into the beginning of
> each release. It should be obvious that setting a deadline doesn't actually
> affect the amount of reviewer effort, it just concentrates that effort.
>

​True statement, although there was an effort to have core reviewer 'sign
up' and take ownership/responsibility specifically to review various
drivers.​


>
> The argument about crappy code is also a lot weaker now that there are CI
> requirements which force vendors to spend much more time up front and clear
> a much higher quality bar before the driver is even considered for merging.
> Drivers that aren't ready for merge can always be deferred to a later
> release, but it seems weird to defer drivers that are high quality just
> because they're submitted during milestones 2 or 3.
>
> All the the above is just my opinion though, and you shouldn't care about
> my opinions, as I don't do much coding and reviewing in Cinder. There is a
> real reason I'm writing this email...
>
> In Manila we added some major new features during Liberty. All of the new
> features merged in the last week of L-3. It was a nightmare of merge
> conflicts and angry core reviewers, and many contributors worked through a
> holiday weekend to bring the release together. While asking myself how we
> can avoid such a situation in the future, it became clear to me that bigger
> features need to merge earlier -- the earlier the better.
>

​So this is most certainly an issue that we've been having in Cinder as
well.  It's a bad problem to have in my opinion and also I personally took
a pretty hard stance against some new features, reworking of core code
because it wasn't ready until the last week or so of the milestone and
frankly they were HUGE changes.
​


>
> When I look at the release timeline, and ask myself when is the best time
> to merge new major features, and when is the best time to merge new
> drivers, it seems obvious that *features* need to happen early and drivers
> should come *later*. New major features require FAR more review time than
> new drivers, and they require testing, and even after they merge they cause
> merge conflicts that everyone else has to deal with. Better that that works
> happens in milestones 1 and 2 than right before feature freeze. New drivers
> can come in right before feature freeze as far as I'm concerned. Drivers
> don't cause merge conflicts, and drivers don't need huge amounts of testing
> (presumably the CI system ensure some level of quality).
>

​For the most part I think I completely agree with everything you've said
above.​


>
> It also occurs to me that new features which require driver implementation
> (hello replication!) *really* should go in during the first milestone so
> that drivers have time to implement the feature during the same release.
>

​Yep, I most certainly should have pushed this in to the core code MUCH
earlier.  But to be honest, if you look at the life-cycle of the spec and
the patch that implemented it; it was proposed very early, BUT there was
very little useful input until after the mid-cycle'ish meetup in FTC. Was
it a matter of review focus, bike-shedding, or me being lazy... maybe a
little of all three.​


>
> So I'm asking the Cinder core team to reconsider the milestone-1 deadline
> for drivers, and to change it to a deadline for new major features (in
> milestone-1 or milestone-2), and to allow drivers to merge whenever*. This

Re: [openstack-dev] [cinder] The Absurdity of the Milestone-1 Deadline for Drivers

[Walter A. Boring IV]

On 09/28/2015 10:29 AM, Ben Swartzlander wrote:
I've always thought it was a bit strange to require new drivers to 
merge by milestone 1. I think I understand the motivations of the 
policy. The main motivation was to free up reviewers to review "other 
things" and this policy guarantees that for 75% of the release 
reviewers don't have to review new drivers. The other motivation was 
to prevent vendors from turning up at the last minute with crappy 
drivers that needed a ton of work, by encouraging them to get started 
earlier, or forcing them to wait until the next cycle.


I believe that the deadline actually does more harm than good.
But harm to whom?   It certainly puts the pressure on driver developers 
to make sure they get involved in the Cinder community and get aware of 
when the deadlines are.
I believe it simply shifts the time in which drivers get into tree. My 
$0.02 of opinion is, that if a new driver developer misses the 
milestone, then they have the rest of the release to work on getting CI 
up and running and ready to go for the next release.   I'm not sure I 
see the harm to the Cinder community or the project.   It's a deadline 
that a driver developer has to be aware of and compensate for.  We've 
had how many drivers land in the last 2 releases using this 
requirement?  I believe it's somewhere of 20+ drivers?




First of all, to those that don't want to spend time on driver 
reviews, there are other solutions to that problem. Some people do 
want to review the drivers, and those who don't can simply ignore them 
and spend time on what they care about. I've heard people who spend 
time on driver reviews say that the milestone-1 deadline doesn't mean 
they spend less time reviewing drivers overall, it just all gets 
crammed into the beginning of each release. It should be obvious that 
setting a deadline doesn't actually affect the amount of reviewer 
effort, it just concentrates that effort.


The argument about crappy code is also a lot weaker now that there are 
CI requirements which force vendors to spend much more time up front 
and clear a much higher quality bar before the driver is even 
considered for merging. Drivers that aren't ready for merge can always 
be deferred to a later release, but it seems weird to defer drivers 
that are high quality just because they're submitted during milestones 
2 or 3.
I disagree here.  CI doesn't prevent you from having a crappy driver.  
Your driver just needs to pass CI tests.  CI ensures that your driver 
works, but doesn't ensure that it
really meats the core reviewers standards for code.  Do we care?  I 
think we do.  Having drivers talk directly to the db, or FC drivers 
missing the FCZM decorators for auto zoning, etc.




All the the above is just my opinion though, and you shouldn't care 
about my opinions, as I don't do much coding and reviewing in Cinder. 
There is a real reason I'm writing this email...


In Manila we added some major new features during Liberty. All of the 
new features merged in the last week of L-3. It was a nightmare of 
merge conflicts and angry core reviewers, and many contributors worked 
through a holiday weekend to bring the release together. While asking 
myself how we can avoid such a situation in the future, it became 
clear to me that bigger features need to merge earlier -- the earlier 
the better.


When I look at the release timeline, and ask myself when is the best 
time to merge new major features, and when is the best time to merge 
new drivers, it seems obvious that *features* need to happen early and 
drivers should come *later*. New major features require FAR more 
review time than new drivers, and they require testing, and even after 
they merge they cause merge conflicts that everyone else has to deal 
with. Better that that works happens in milestones 1 and 2 than right 
before feature freeze. New drivers can come in right before feature 
freeze as far as I'm concerned. Drivers don't cause merge conflicts, 
and drivers don't need huge amounts of testing (presumably the CI 
system ensure some level of quality).


It also occurs to me that new features which require driver 
implementation (hello replication!) *really* should go in during the 
first milestone so that drivers have time to implement the feature 
during the same release.


So I'm asking the Cinder core team to reconsider the milestone-1 
deadline for drivers, and to change it to a deadline for new major 
features (in milestone-1 or milestone-2), and to allow drivers to 
merge whenever*. This is the same pitch I'll be making to the Manila 
core team. I've been considering this idea for a few weeks now but I 
wanted to wait until after PTL elections to suggest it here.


-Ben Swartzlander


* I don't actually care if/when there is a driver deadline, what I 
care about is that reviewers are free during M-1 to work on 
reviewing/testing of features. The easiest way to achieve that seems 
to be moving the driver deadline.
I'm