Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-23 Thread Doug Hellmann
On Wed, Jan 22, 2014 at 4:45 PM, Joe Gordon joe.gord...@gmail.com wrote:




 On Wed, Jan 22, 2014 at 5:19 AM, Julien Danjou jul...@danjou.info wrote:

 On Tue, Jan 21 2014, Joe Gordon wrote:

  I would like to propose having a integration test job in Oslo incubator
  that syncs in the code, similar to how we do global requirements.

 I don't think that would be possible as a voting job, since the point of
 oslo-incubator is to be able to break the API compatibility.


 If oslo-incubator can break APIs whenever it wants how can a downstream
 project stay in sync with oslo-incubator?


Usually the API maintainer or those making significant changes to an API
take responsibility for syncing that specific module into the projects
which use it [1]

[1] https://wiki.openstack.org/wiki/Oslo#Syncing_Code_from_Incubator

That happens, AFAICT, when a known incompatibility is introduced. It does
not always happen when a compatible change is made. I'm really not sure of
the best way to improve that, other than to remind contributors to oslo of
the expectations.

Doug






 --
 Julien Danjou
 ;; Free Software hacker ; independent consultant
 ;; http://julien.danjou.info



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-23 Thread Doug Hellmann
On Tue, Jan 21, 2014 at 1:28 PM, Sean Dague s...@dague.net wrote:

 On 01/21/2014 01:14 PM, Joe Gordon wrote:
 
  On Jan 17, 2014 12:24 AM, Flavio Percoco fla...@redhat.com
  mailto:fla...@redhat.com wrote:
 
  On 16/01/14 17:32 -0500, Doug Hellmann wrote:
 
  On Thu, Jan 16, 2014 at 3:19 PM, Ben Nemec openst...@nemebean.com
  mailto:openst...@nemebean.com wrote:
 
 On 2014-01-16 13:48, John Griffith wrote:
 
 Hey Everyone,
 
 A review came up today that cherry-picked a specific commit to
  OSLO
 Incubator, without updating the rest of the files in the
  module.  I
 rejected that patch, because my philosophy has been that when
 you
 update/pull from oslo-incubator it should be done as a full
  sync of
 the entire module, not a cherry pick of the bits and pieces
  that you
 may or may not be interested in.
 
 As it turns out I've received a bit of push back on this, so
  it seems
 maybe I'm being unreasonable, or that I'm mistaken in my
  understanding
 of the process here.  To me it seems like a complete and total
  waste
 to have an oslo-incubator and common libs if you're going to
 turn
 around and just cherry pick changes, but maybe I'm completely
  out of
 line.
 
 Thoughts??
 
 
 I suppose there might be exceptions, but in general I'm with you.
   For one
 thing, if someone tries to pull out a specific change in the Oslo
  code,
 there's no guarantee that code even works.  Depending on how the
  sync was
 done it's possible the code they're syncing never passed the Oslo
 unit
 tests in the form being synced, and since unit tests aren't synced
  to the
 target projects it's conceivable that completely broken code could
 get
 through Jenkins.
 
 Obviously it's possible to do a successful partial sync, but for
  the sake
 of reviewer sanity I'm -1 on partial syncs without a _very_ good
  reason
 (like it's blocking the gate and there's some reason the full
  module can't
 be synced).
 
 
  I agree. Cherry picking a single (or even partial) commit really
  should be
  avoided.
 
  The update tool does allow syncing just a single module, but that
  should be
  used very VERY carefully, especially because some of the changes
  we're making
  as we work on graduating some more libraries will include
 cross-dependent
  changes between oslo modules.
 
 
  Agrred. Syncing on master should be complete synchornization from Oslo
  incubator. IMHO, the only case where cherry-picking from oslo should
  be allowed is when backporting patches to stable branches. Master
  branches should try to keep up-to-date with Oslo and sync everything
  every time.
 
  When we started Oslo incubator, we treated that code as trusted. But
  since then there have been occasional issues when syncing the code. So
  Oslo incubator code has lost *my* trust. Therefore I am always a
  hesitant to do a full Oslo sync because I am not an expert on the Oslo
  code and I risk breaking something when doing it (the issue may not
  appear 100% of the time too). Syncing code in becomes the first time
  that code is run against tempest, which scares me.
 
  I would like to propose having a integration test job in Oslo incubator
  that syncs in the code, similar to how we do global requirements.

 That would be great, especially given that oslo currently doesn't run
 it's own unit tests in parallel, which means it's a lot easier for non
 parallel safe code to get in, and we don't bring the unit tests over to
 the projects on sync.

 Getting preemptive and doing a forward sync for validation would be
 really valuable.


Forcing the code in the incubator to pass the tests when synced to
projects eliminates its usefulness as a place to improve APIs. The
incubator does not make backwards compatibility claims, which is why most
of the time we want the author of code in the incubator to make the merge
and associated application code changes at the same time. When there aren't
any API changes, a straight sync should pass the tests in the project.

We should look into why we're not running the unit tests in parallel. That
seems like something easy to change, but I'm not sure why it's like that to
begin with.

Doug




 -Sean

 --
 Sean Dague
 Samsung Research America
 s...@dague.net / sean.da...@samsung.com
 http://dague.net


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-23 Thread Doug Hellmann
On Wed, Jan 22, 2014 at 11:59 AM, Ben Nemec openst...@nemebean.com wrote:

 On 2014-01-22 06:32, Sean Dague wrote:

 I think we need to graduate things to stable interfaces a lot faster.
 Realizing that stable just means have to deprecate to change it. So
 the interface is still changeable, just requires standard deprecation
 techniques. Which we are trying to get more python libraries to do
 anyway, so it would be good if we built up a bunch of best practices here.

 -Sean


 Big +1 to this.  Eliminating the sync process is going to be the cleanest
 solution for the code that is stable enough to be usable with things like
 automatic syncs.  The less code that is left in incubator, the easier the
 syncs will be.

 That said, I think there's only a few people (Doug, Mark, and Thierry?)
 who have done the promote to library thing, and I will admit I don't have a
 good handle on what is involved.  It may be that we need better
 documentation of that process so more people can help out with it.  I know
 Michael Still mentioned he was planning to graduate lockutils but didn't
 know exactly how.


I've outlined what I see as the likely set of libraries for the current set
of incubated code already, as well as some implementation changes I would
like to see before approving graduation in a separate thread. Documenting
that process is next on my list. 

Doug





 -Ben


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-23 Thread Sean Dague
On 01/23/2014 09:58 AM, Doug Hellmann wrote:
 
 
 
 On Tue, Jan 21, 2014 at 1:28 PM, Sean Dague s...@dague.net
 mailto:s...@dague.net wrote:
 
 On 01/21/2014 01:14 PM, Joe Gordon wrote:
 
  On Jan 17, 2014 12:24 AM, Flavio Percoco fla...@redhat.com
 mailto:fla...@redhat.com
  mailto:fla...@redhat.com mailto:fla...@redhat.com wrote:
 
  On 16/01/14 17:32 -0500, Doug Hellmann wrote:
 
  On Thu, Jan 16, 2014 at 3:19 PM, Ben Nemec
 openst...@nemebean.com mailto:openst...@nemebean.com
  mailto:openst...@nemebean.com mailto:openst...@nemebean.com
 wrote:
 
 On 2014-01-16 13:48, John Griffith wrote:
 
 Hey Everyone,
 
 A review came up today that cherry-picked a specific
 commit to
  OSLO
 Incubator, without updating the rest of the files in the
  module.  I
 rejected that patch, because my philosophy has been that
 when you
 update/pull from oslo-incubator it should be done as a full
  sync of
 the entire module, not a cherry pick of the bits and pieces
  that you
 may or may not be interested in.
 
 As it turns out I've received a bit of push back on this, so
  it seems
 maybe I'm being unreasonable, or that I'm mistaken in my
  understanding
 of the process here.  To me it seems like a complete and
 total
  waste
 to have an oslo-incubator and common libs if you're going
 to turn
 around and just cherry pick changes, but maybe I'm completely
  out of
 line.
 
 Thoughts??
 
 
 I suppose there might be exceptions, but in general I'm with you.
   For one
 thing, if someone tries to pull out a specific change in the Oslo
  code,
 there's no guarantee that code even works.  Depending on how the
  sync was
 done it's possible the code they're syncing never passed the
 Oslo unit
 tests in the form being synced, and since unit tests aren't
 synced
  to the
 target projects it's conceivable that completely broken code
 could get
 through Jenkins.
 
 Obviously it's possible to do a successful partial sync, but for
  the sake
 of reviewer sanity I'm -1 on partial syncs without a _very_ good
  reason
 (like it's blocking the gate and there's some reason the full
  module can't
 be synced).
 
 
  I agree. Cherry picking a single (or even partial) commit really
  should be
  avoided.
 
  The update tool does allow syncing just a single module, but that
  should be
  used very VERY carefully, especially because some of the changes
  we're making
  as we work on graduating some more libraries will include
 cross-dependent
  changes between oslo modules.
 
 
  Agrred. Syncing on master should be complete synchornization from
 Oslo
  incubator. IMHO, the only case where cherry-picking from oslo should
  be allowed is when backporting patches to stable branches. Master
  branches should try to keep up-to-date with Oslo and sync everything
  every time.
 
  When we started Oslo incubator, we treated that code as trusted. But
  since then there have been occasional issues when syncing the code. So
  Oslo incubator code has lost *my* trust. Therefore I am always a
  hesitant to do a full Oslo sync because I am not an expert on the Oslo
  code and I risk breaking something when doing it (the issue may not
  appear 100% of the time too). Syncing code in becomes the first time
  that code is run against tempest, which scares me.
 
  I would like to propose having a integration test job in Oslo
 incubator
  that syncs in the code, similar to how we do global requirements.
 
 That would be great, especially given that oslo currently doesn't run
 it's own unit tests in parallel, which means it's a lot easier for non
 parallel safe code to get in, and we don't bring the unit tests over to
 the projects on sync.
 
 Getting preemptive and doing a forward sync for validation would be
 really valuable.
 
 
 Forcing the code in the incubator to pass the tests when synced to
 projects eliminates its usefulness as a place to improve APIs. The
 incubator does not make backwards compatibility claims, which is why
 most of the time we want the author of code in the incubator to make the
 merge and associated application code changes at the same time. When
 there aren't any API changes, a straight sync should pass the tests in
 the project.

So that's kind of the question at hand.

If oslo-incubator is a place where we keep guarantees so that projects
don't have to rework their code immediately (i.e. have a deprecation /
compat period), so we can forward sync 

Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-23 Thread Doug Hellmann
On Thu, Jan 23, 2014 at 10:44 AM, Sean Dague s...@dague.net wrote:

 On 01/23/2014 09:58 AM, Doug Hellmann wrote:
 
 
 
  On Tue, Jan 21, 2014 at 1:28 PM, Sean Dague s...@dague.net
  mailto:s...@dague.net wrote:
 
  On 01/21/2014 01:14 PM, Joe Gordon wrote:
  
   On Jan 17, 2014 12:24 AM, Flavio Percoco fla...@redhat.com
  mailto:fla...@redhat.com
   mailto:fla...@redhat.com mailto:fla...@redhat.com wrote:
  
   On 16/01/14 17:32 -0500, Doug Hellmann wrote:
  
   On Thu, Jan 16, 2014 at 3:19 PM, Ben Nemec
  openst...@nemebean.com mailto:openst...@nemebean.com
   mailto:openst...@nemebean.com mailto:openst...@nemebean.com
  wrote:
  
  On 2014-01-16 13:48, John Griffith wrote:
  
  Hey Everyone,
  
  A review came up today that cherry-picked a specific
  commit to
   OSLO
  Incubator, without updating the rest of the files in the
   module.  I
  rejected that patch, because my philosophy has been that
  when you
  update/pull from oslo-incubator it should be done as a
 full
   sync of
  the entire module, not a cherry pick of the bits and
 pieces
   that you
  may or may not be interested in.
  
  As it turns out I've received a bit of push back on this,
 so
   it seems
  maybe I'm being unreasonable, or that I'm mistaken in my
   understanding
  of the process here.  To me it seems like a complete and
  total
   waste
  to have an oslo-incubator and common libs if you're going
  to turn
  around and just cherry pick changes, but maybe I'm
 completely
   out of
  line.
  
  Thoughts??
  
  
  I suppose there might be exceptions, but in general I'm with
 you.
For one
  thing, if someone tries to pull out a specific change in the
 Oslo
   code,
  there's no guarantee that code even works.  Depending on how
 the
   sync was
  done it's possible the code they're syncing never passed the
  Oslo unit
  tests in the form being synced, and since unit tests aren't
  synced
   to the
  target projects it's conceivable that completely broken code
  could get
  through Jenkins.
  
  Obviously it's possible to do a successful partial sync, but
 for
   the sake
  of reviewer sanity I'm -1 on partial syncs without a _very_
 good
   reason
  (like it's blocking the gate and there's some reason the full
   module can't
  be synced).
  
  
   I agree. Cherry picking a single (or even partial) commit really
   should be
   avoided.
  
   The update tool does allow syncing just a single module, but that
   should be
   used very VERY carefully, especially because some of the changes
   we're making
   as we work on graduating some more libraries will include
  cross-dependent
   changes between oslo modules.
  
  
   Agrred. Syncing on master should be complete synchornization from
  Oslo
   incubator. IMHO, the only case where cherry-picking from oslo
 should
   be allowed is when backporting patches to stable branches. Master
   branches should try to keep up-to-date with Oslo and sync
 everything
   every time.
  
   When we started Oslo incubator, we treated that code as trusted.
 But
   since then there have been occasional issues when syncing the
 code. So
   Oslo incubator code has lost *my* trust. Therefore I am always a
   hesitant to do a full Oslo sync because I am not an expert on the
 Oslo
   code and I risk breaking something when doing it (the issue may not
   appear 100% of the time too). Syncing code in becomes the first
 time
   that code is run against tempest, which scares me.
  
   I would like to propose having a integration test job in Oslo
  incubator
   that syncs in the code, similar to how we do global requirements.
 
  That would be great, especially given that oslo currently doesn't run
  it's own unit tests in parallel, which means it's a lot easier for
 non
  parallel safe code to get in, and we don't bring the unit tests over
 to
  the projects on sync.
 
  Getting preemptive and doing a forward sync for validation would be
  really valuable.
 
 
  Forcing the code in the incubator to pass the tests when synced to
  projects eliminates its usefulness as a place to improve APIs. The
  incubator does not make backwards compatibility claims, which is why
  most of the time we want the author of code in the incubator to make the
  merge and associated application code changes at the same time. When
  there aren't any API changes, a straight sync should pass the tests in
  the project.

 So that's kind of the question 

Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-22 Thread Flavio Percoco

On 21/01/14 13:14 -0500, Joe Gordon wrote:


On Jan 17, 2014 12:24 AM, Flavio Percoco fla...@redhat.com wrote:


On 16/01/14 17:32 -0500, Doug Hellmann wrote:


On Thu, Jan 16, 2014 at 3:19 PM, Ben Nemec openst...@nemebean.com wrote:

   On 2014-01-16 13:48, John Griffith wrote:

   Hey Everyone,

   A review came up today that cherry-picked a specific commit to OSLO
   Incubator, without updating the rest of the files in the module.  I
   rejected that patch, because my philosophy has been that when you
   update/pull from oslo-incubator it should be done as a full sync of
   the entire module, not a cherry pick of the bits and pieces that you
   may or may not be interested in.

   As it turns out I've received a bit of push back on this, so it seems
   maybe I'm being unreasonable, or that I'm mistaken in my

understanding

   of the process here.  To me it seems like a complete and total waste
   to have an oslo-incubator and common libs if you're going to turn
   around and just cherry pick changes, but maybe I'm completely out of
   line.

   Thoughts??


   I suppose there might be exceptions, but in general I'm with you.  For

one

   thing, if someone tries to pull out a specific change in the Oslo code,
   there's no guarantee that code even works.  Depending on how the sync was
   done it's possible the code they're syncing never passed the Oslo unit
   tests in the form being synced, and since unit tests aren't synced to the
   target projects it's conceivable that completely broken code could get
   through Jenkins.

   Obviously it's possible to do a successful partial sync, but for the sake
   of reviewer sanity I'm -1 on partial syncs without a _very_ good reason
   (like it's blocking the gate and there's some reason the full module

can't

   be synced).


I agree. Cherry picking a single (or even partial) commit really should be
avoided.

The update tool does allow syncing just a single module, but that should be
used very VERY carefully, especially because some of the changes we're

making

as we work on graduating some more libraries will include cross-dependent
changes between oslo modules.



Agrred. Syncing on master should be complete synchornization from Oslo
incubator. IMHO, the only case where cherry-picking from oslo should
be allowed is when backporting patches to stable branches. Master
branches should try to keep up-to-date with Oslo and sync everything
every time.


When we started Oslo incubator, we treated that code as trusted. But since then
there have been occasional issues when syncing the code. So Oslo incubator code
has lost *my* trust. Therefore I am always a hesitant to do a full Oslo sync
because I am not an expert on the Oslo code and I risk breaking something when
doing it (the issue may not appear 100% of the time too). Syncing code in
becomes the first time that code is run against tempest, which scares me.


While this might be true in some cases, I think we should address it
differently. Just dropping the trust on the project won't help much.


I would like to propose having a integration test job in Oslo incubator that
syncs in the code, similar to how we do global requirements.


But isn't this what other gates are for? I mean, when proposing an
oslo sync, each project has it's own gate plus integrated tests that
do this exact job.


Additionally, what about a periodic jenkins job that does the Oslo syncs and is
managed by the Oslo team itself?


This would be awesome. It would take the burden of doing the sync from
the project maintainers. Before doing this, though, we need to improve
the `update` script. Currently, there's no good way to generate useful
commit messages out of the sync.

Cheers,
FF

--
@flaper87
Flavio Percoco


pgpfQZZGqrqIv.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-22 Thread Julien Danjou
On Tue, Jan 21 2014, Joe Gordon wrote:

 I would like to propose having a integration test job in Oslo incubator
 that syncs in the code, similar to how we do global requirements.

I don't think that would be possible as a voting job, since the point of
oslo-incubator is to be able to break the API compatibility.

-- 
Julien Danjou
;; Free Software hacker ; independent consultant
;; http://julien.danjou.info


signature.asc
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-22 Thread Sean Dague
On 01/22/2014 05:19 AM, Julien Danjou wrote:
 On Tue, Jan 21 2014, Joe Gordon wrote:
 
 I would like to propose having a integration test job in Oslo incubator
 that syncs in the code, similar to how we do global requirements.
 
 I don't think that would be possible as a voting job, since the point of
 oslo-incubator is to be able to break the API compatibility.

I'm starting to feel like we need to revisit that point. Because what
happens now is a chunk of code gets worked off in a corner, possibly
randomly changing interfaces, not running unit tests in a way that we
know it's multi process safe.

So there ends up being a ton of blind trust in the sync right now. Which
is why the syncs are coming slower, and you'll have nova 4 - 6 months
behind on many modules, missing a critical bug fix that's buried some
where inside a bunch of other interface changes that are expensive. (Not
theoretical, I just tripped over this in Dec).

I think we need to graduate things to stable interfaces a lot faster.
Realizing that stable just means have to deprecate to change it. So
the interface is still changeable, just requires standard deprecation
techniques. Which we are trying to get more python libraries to do
anyway, so it would be good if we built up a bunch of best practices here.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-22 Thread Flavio Percoco

On 22/01/14 07:32 -0500, Sean Dague wrote:

On 01/22/2014 05:19 AM, Julien Danjou wrote:

On Tue, Jan 21 2014, Joe Gordon wrote:


I would like to propose having a integration test job in Oslo incubator
that syncs in the code, similar to how we do global requirements.


I don't think that would be possible as a voting job, since the point of
oslo-incubator is to be able to break the API compatibility.


I'm starting to feel like we need to revisit that point. Because what
happens now is a chunk of code gets worked off in a corner, possibly
randomly changing interfaces, not running unit tests in a way that we
know it's multi process safe.


This is not true. If there have been abrupt changes on the interfaces
then we failed at keeping backwards compatibility. However, that
doesn't mean the API is not considered when reviewing nor that it
doesn't matter because the library is incubated.

This kind of changes usually get filed on all projects using a
specific functionality. For example, 
https://bugs.launchpad.net/oslo/+bug/1266962

Again, if there have been cases where an API has been changed without
even notifying others - either with a good commit message, m-l thread
or bug report - then there's definitely something wrong in the process
and it should be fixed. Also, I'd expect these errors to be raised as
soon as they're noted because they may also affect other projects as
well.

The above is very different than just saying oslo-incubator is not
trustworthy because things get copied around and changes to the
libraries are made randomly.


So there ends up being a ton of blind trust in the sync right now. Which
is why the syncs are coming slower, and you'll have nova 4 - 6 months
behind on many modules, missing a critical bug fix that's buried some
where inside a bunch of other interface changes that are expensive. (Not
theoretical, I just tripped over this in Dec).


I'm sorry but this is not an excuse to avoid syncing from
oslo-incubator. Actually, if things like this can happen, the bigger
the gap is the harder it'll be to sync from oslo. My suggestion has
always been to do periodic syncs from oslo and keep up to day.
Interface changes that *just* break other projects without a good way
forward have to be raised here.

I know we're talking about incubated libraries that are suppose to
change but as mentioned above, we always consider backwards
compatibility even on incubated libs because they're on its way to
stability and breaking other projects is not fun.


I think we need to graduate things to stable interfaces a lot faster.
Realizing that stable just means have to deprecate to change it. So
the interface is still changeable, just requires standard deprecation
techniques. Which we are trying to get more python libraries to do
anyway, so it would be good if we built up a bunch of best practices here.


Agreed. This is something that we've been working on during Icehouse.
We should probably define more clear what's the incubation path of
modules that land in oslo-incubator. For example, determine where
would they fit, how long should they be around based on their
functionality and/or complexity etc.

We talked about having a meeting on this matter after I-2. Not sure
when it'll happen but it'll be a perfect time to discuss this further.

Cheers,
FF

--
@flaper87
Flavio Percoco


pgpv4mSdRVkCi.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-22 Thread Ben Nemec

On 2014-01-22 06:32, Sean Dague wrote:

I think we need to graduate things to stable interfaces a lot faster.
Realizing that stable just means have to deprecate to change it. So
the interface is still changeable, just requires standard deprecation
techniques. Which we are trying to get more python libraries to do
anyway, so it would be good if we built up a bunch of best practices 
here.


-Sean


Big +1 to this.  Eliminating the sync process is going to be the 
cleanest solution for the code that is stable enough to be usable with 
things like automatic syncs.  The less code that is left in incubator, 
the easier the syncs will be.


That said, I think there's only a few people (Doug, Mark, and Thierry?) 
who have done the promote to library thing, and I will admit I don't 
have a good handle on what is involved.  It may be that we need better 
documentation of that process so more people can help out with it.  I 
know Michael Still mentioned he was planning to graduate lockutils but 
didn't know exactly how.


-Ben

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-22 Thread Flavio Percoco

On 22/01/14 10:59 -0600, Ben Nemec wrote:

On 2014-01-22 06:32, Sean Dague wrote:

I think we need to graduate things to stable interfaces a lot faster.
Realizing that stable just means have to deprecate to change it. So
the interface is still changeable, just requires standard deprecation
techniques. Which we are trying to get more python libraries to do
anyway, so it would be good if we built up a bunch of best practices 
here.


-Sean


Big +1 to this.  Eliminating the sync process is going to be the 
cleanest solution for the code that is stable enough to be usable with 
things like automatic syncs.  The less code that is left in incubator, 
the easier the syncs will be.


That said, I think there's only a few people (Doug, Mark, and 
Thierry?) who have done the promote to library thing, and I will admit 
I don't have a good handle on what is involved.  It may be that we 
need better documentation of that process so more people can help out 
with it.  I know Michael Still mentioned he was planning to graduate 
lockutils but didn't know exactly how.


We're in the process of grouping independent modules into modules that
actually make sense to avoid having 1 python package per module on
pypi.

Some of the graduation status is being tracked here[0] and here's[1] a
graph of the current dependencies.

As mentioned in my last email, I fully agree with this and we should
definitely establish what the process is. oslo.config was the first
package that graduated from the incubator. Other packages will come
out of there during Icehouse.

Cheers,
FF

[0] https://wiki.openstack.org/wiki/Oslo/GraduationStatus
[1] https://wiki.openstack.org/wiki/Oslo/Dependencies



-Ben

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
@flaper87
Flavio Percoco


pgpCMoxOfF9LI.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-22 Thread Joe Gordon
On Wed, Jan 22, 2014 at 3:22 AM, Flavio Percoco fla...@redhat.com wrote:

 On 21/01/14 13:14 -0500, Joe Gordon wrote:


 On Jan 17, 2014 12:24 AM, Flavio Percoco fla...@redhat.com wrote:


 On 16/01/14 17:32 -0500, Doug Hellmann wrote:


 On Thu, Jan 16, 2014 at 3:19 PM, Ben Nemec openst...@nemebean.com
 wrote:

On 2014-01-16 13:48, John Griffith wrote:

Hey Everyone,

A review came up today that cherry-picked a specific commit to
 OSLO
Incubator, without updating the rest of the files in the module.
  I
rejected that patch, because my philosophy has been that when you
update/pull from oslo-incubator it should be done as a full sync
 of
the entire module, not a cherry pick of the bits and pieces that
 you
may or may not be interested in.

As it turns out I've received a bit of push back on this, so it
 seems
maybe I'm being unreasonable, or that I'm mistaken in my

 understanding

of the process here.  To me it seems like a complete and total
 waste
to have an oslo-incubator and common libs if you're going to turn
around and just cherry pick changes, but maybe I'm completely
 out of
line.

Thoughts??


I suppose there might be exceptions, but in general I'm with you.
  For

 one

thing, if someone tries to pull out a specific change in the Oslo
 code,
there's no guarantee that code even works.  Depending on how the
 sync was
done it's possible the code they're syncing never passed the Oslo
 unit
tests in the form being synced, and since unit tests aren't synced
 to the
target projects it's conceivable that completely broken code could
 get
through Jenkins.

Obviously it's possible to do a successful partial sync, but for the
 sake
of reviewer sanity I'm -1 on partial syncs without a _very_ good
 reason
(like it's blocking the gate and there's some reason the full module

 can't

be synced).


 I agree. Cherry picking a single (or even partial) commit really should
 be
 avoided.

 The update tool does allow syncing just a single module, but that
 should be
 used very VERY carefully, especially because some of the changes we're

 making

 as we work on graduating some more libraries will include cross-dependent
 changes between oslo modules.



 Agrred. Syncing on master should be complete synchornization from Oslo
 incubator. IMHO, the only case where cherry-picking from oslo should
 be allowed is when backporting patches to stable branches. Master
 branches should try to keep up-to-date with Oslo and sync everything
 every time.


 When we started Oslo incubator, we treated that code as trusted. But
 since then
 there have been occasional issues when syncing the code. So Oslo
 incubator code
 has lost *my* trust. Therefore I am always a hesitant to do a full Oslo
 sync
 because I am not an expert on the Oslo code and I risk breaking something
 when
 doing it (the issue may not appear 100% of the time too). Syncing code in
 becomes the first time that code is run against tempest, which scares me.


 While this might be true in some cases, I think we should address it
 differently. Just dropping the trust on the project won't help much.


How else would you address it?




  I would like to propose having a integration test job in Oslo incubator
 that
 syncs in the code, similar to how we do global requirements.


 But isn't this what other gates are for? I mean, when proposing an
 oslo sync, each project has it's own gate plus integrated tests that
 do this exact job.


Sort of.  There are two possible failure modes here:

1) oslo-incubator sync is attempted by Alice and the patch fails
integration tests in the check queue. Alice doesn't know why it failed and
has to go and resolve issue with oslo folks. Alice now thinks doing
oslo-incubator syncs are a hassle and stops doing them and moves on to
something else.

2) oslo-incubator sync is merged, but introduces non-deterministic bug.
This wasn't caught in oslo-incubator because there are no integration tests
there. The risk here is that the oslo-incubator code isn't run enough to
detect non-deterministic bugs. In fact we just found one yesterday (
https://review.openstack.org/#/c/68275/).




  Additionally, what about a periodic jenkins job that does the Oslo syncs
 and is
 managed by the Oslo team itself?


 This would be awesome. It would take the burden of doing the sync from
 the project maintainers. Before doing this, though, we need to improve
 the `update` script. Currently, there's no good way to generate useful
 commit messages out of the sync.


 Cheers,
 FF

 --
 @flaper87
 Flavio Percoco

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org

Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-22 Thread Joe Gordon
On Wed, Jan 22, 2014 at 5:19 AM, Julien Danjou jul...@danjou.info wrote:

 On Tue, Jan 21 2014, Joe Gordon wrote:

  I would like to propose having a integration test job in Oslo incubator
  that syncs in the code, similar to how we do global requirements.

 I don't think that would be possible as a voting job, since the point of
 oslo-incubator is to be able to break the API compatibility.


If oslo-incubator can break APIs whenever it wants how can a downstream
project stay in sync with oslo-incubator?



 --
 Julien Danjou
 ;; Free Software hacker ; independent consultant
 ;; http://julien.danjou.info

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-21 Thread Joe Gordon
On Jan 17, 2014 12:24 AM, Flavio Percoco fla...@redhat.com wrote:

 On 16/01/14 17:32 -0500, Doug Hellmann wrote:

 On Thu, Jan 16, 2014 at 3:19 PM, Ben Nemec openst...@nemebean.com
wrote:

On 2014-01-16 13:48, John Griffith wrote:

Hey Everyone,

A review came up today that cherry-picked a specific commit to
OSLO
Incubator, without updating the rest of the files in the module.
 I
rejected that patch, because my philosophy has been that when you
update/pull from oslo-incubator it should be done as a full sync
of
the entire module, not a cherry pick of the bits and pieces that
you
may or may not be interested in.

As it turns out I've received a bit of push back on this, so it
seems
maybe I'm being unreasonable, or that I'm mistaken in my
understanding
of the process here.  To me it seems like a complete and total
waste
to have an oslo-incubator and common libs if you're going to turn
around and just cherry pick changes, but maybe I'm completely out
of
line.

Thoughts??


I suppose there might be exceptions, but in general I'm with you.
 For one
thing, if someone tries to pull out a specific change in the Oslo
code,
there's no guarantee that code even works.  Depending on how the sync
was
done it's possible the code they're syncing never passed the Oslo unit
tests in the form being synced, and since unit tests aren't synced to
the
target projects it's conceivable that completely broken code could get
through Jenkins.

Obviously it's possible to do a successful partial sync, but for the
sake
of reviewer sanity I'm -1 on partial syncs without a _very_ good
reason
(like it's blocking the gate and there's some reason the full module
can't
be synced).


 I agree. Cherry picking a single (or even partial) commit really should
be
 avoided.

 The update tool does allow syncing just a single module, but that should
be
 used very VERY carefully, especially because some of the changes we're
making
 as we work on graduating some more libraries will include cross-dependent
 changes between oslo modules.


 Agrred. Syncing on master should be complete synchornization from Oslo
 incubator. IMHO, the only case where cherry-picking from oslo should
 be allowed is when backporting patches to stable branches. Master
 branches should try to keep up-to-date with Oslo and sync everything
 every time.

When we started Oslo incubator, we treated that code as trusted. But since
then there have been occasional issues when syncing the code. So Oslo
incubator code has lost *my* trust. Therefore I am always a hesitant to do
a full Oslo sync because I am not an expert on the Oslo code and I risk
breaking something when doing it (the issue may not appear 100% of the time
too). Syncing code in becomes the first time that code is run against
tempest, which scares me.

I would like to propose having a integration test job in Oslo incubator
that syncs in the code, similar to how we do global requirements.

Additionally, what about a periodic jenkins job that does the Oslo syncs
and is managed by the Oslo team itself?


 With that in mind, I'd like to request project's members to do
 periodic syncs from Oslo incubator. Yes, it is tedious, painful and
 sometimes requires more than just syncing, but we should all try to
 keep up-to-date with Oslo. The main reason why I'm asking this is
 precisely stable branches. If the project stays way behind the
 oslo-incubator, it'll be really painful to backport patches to stable
 branches in case of failures.

 Unfortunately, there are projects that are quite behind from
 oslo-incubator master.

 One last comment. FWIW, backwards compatibility is always considered
 in all Oslo reviews and if there's a crazy-breaking change, it's
 always notified.

 Thankfully, this all will be alleviated with the libs that are being
 pulled out from the incubator. The syncs will contain fewer modules
 and will be smaller.


 I'm happy you brought this up now. I was meaning to do it.

 Cheers,
 FF


 --
 @flaper87
 Flavio Percoco

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-21 Thread Sean Dague
On 01/21/2014 01:14 PM, Joe Gordon wrote:
 
 On Jan 17, 2014 12:24 AM, Flavio Percoco fla...@redhat.com
 mailto:fla...@redhat.com wrote:

 On 16/01/14 17:32 -0500, Doug Hellmann wrote:

 On Thu, Jan 16, 2014 at 3:19 PM, Ben Nemec openst...@nemebean.com
 mailto:openst...@nemebean.com wrote:

On 2014-01-16 13:48, John Griffith wrote:

Hey Everyone,

A review came up today that cherry-picked a specific commit to
 OSLO
Incubator, without updating the rest of the files in the
 module.  I
rejected that patch, because my philosophy has been that when you
update/pull from oslo-incubator it should be done as a full
 sync of
the entire module, not a cherry pick of the bits and pieces
 that you
may or may not be interested in.

As it turns out I've received a bit of push back on this, so
 it seems
maybe I'm being unreasonable, or that I'm mistaken in my
 understanding
of the process here.  To me it seems like a complete and total
 waste
to have an oslo-incubator and common libs if you're going to turn
around and just cherry pick changes, but maybe I'm completely
 out of
line.

Thoughts??


I suppose there might be exceptions, but in general I'm with you.
  For one
thing, if someone tries to pull out a specific change in the Oslo
 code,
there's no guarantee that code even works.  Depending on how the
 sync was
done it's possible the code they're syncing never passed the Oslo unit
tests in the form being synced, and since unit tests aren't synced
 to the
target projects it's conceivable that completely broken code could get
through Jenkins.

Obviously it's possible to do a successful partial sync, but for
 the sake
of reviewer sanity I'm -1 on partial syncs without a _very_ good
 reason
(like it's blocking the gate and there's some reason the full
 module can't
be synced).


 I agree. Cherry picking a single (or even partial) commit really
 should be
 avoided.

 The update tool does allow syncing just a single module, but that
 should be
 used very VERY carefully, especially because some of the changes
 we're making
 as we work on graduating some more libraries will include cross-dependent
 changes between oslo modules.


 Agrred. Syncing on master should be complete synchornization from Oslo
 incubator. IMHO, the only case where cherry-picking from oslo should
 be allowed is when backporting patches to stable branches. Master
 branches should try to keep up-to-date with Oslo and sync everything
 every time.
 
 When we started Oslo incubator, we treated that code as trusted. But
 since then there have been occasional issues when syncing the code. So
 Oslo incubator code has lost *my* trust. Therefore I am always a
 hesitant to do a full Oslo sync because I am not an expert on the Oslo
 code and I risk breaking something when doing it (the issue may not
 appear 100% of the time too). Syncing code in becomes the first time
 that code is run against tempest, which scares me.
 
 I would like to propose having a integration test job in Oslo incubator
 that syncs in the code, similar to how we do global requirements.

That would be great, especially given that oslo currently doesn't run
it's own unit tests in parallel, which means it's a lot easier for non
parallel safe code to get in, and we don't bring the unit tests over to
the projects on sync.

Getting preemptive and doing a forward sync for validation would be
really valuable.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-21 Thread John Griffith
On Tue, Jan 21, 2014 at 11:14 AM, Joe Gordon joe.gord...@gmail.com wrote:

 On Jan 17, 2014 12:24 AM, Flavio Percoco fla...@redhat.com wrote:

 On 16/01/14 17:32 -0500, Doug Hellmann wrote:

 On Thu, Jan 16, 2014 at 3:19 PM, Ben Nemec openst...@nemebean.com
 wrote:

On 2014-01-16 13:48, John Griffith wrote:

Hey Everyone,

A review came up today that cherry-picked a specific commit to
 OSLO
Incubator, without updating the rest of the files in the module.
 I
rejected that patch, because my philosophy has been that when you
update/pull from oslo-incubator it should be done as a full sync
 of
the entire module, not a cherry pick of the bits and pieces that
 you
may or may not be interested in.

As it turns out I've received a bit of push back on this, so it
 seems
maybe I'm being unreasonable, or that I'm mistaken in my
 understanding
of the process here.  To me it seems like a complete and total
 waste
to have an oslo-incubator and common libs if you're going to turn
around and just cherry pick changes, but maybe I'm completely out
 of
line.

Thoughts??


I suppose there might be exceptions, but in general I'm with you.  For
 one
thing, if someone tries to pull out a specific change in the Oslo
 code,
there's no guarantee that code even works.  Depending on how the sync
 was
done it's possible the code they're syncing never passed the Oslo unit
tests in the form being synced, and since unit tests aren't synced to
 the
target projects it's conceivable that completely broken code could get
through Jenkins.

Obviously it's possible to do a successful partial sync, but for the
 sake
of reviewer sanity I'm -1 on partial syncs without a _very_ good
 reason
(like it's blocking the gate and there's some reason the full module
 can't
be synced).


 I agree. Cherry picking a single (or even partial) commit really should
 be
 avoided.

 The update tool does allow syncing just a single module, but that should
 be
 used very VERY carefully, especially because some of the changes we're
 making
 as we work on graduating some more libraries will include cross-dependent
 changes between oslo modules.


 Agrred. Syncing on master should be complete synchornization from Oslo
 incubator. IMHO, the only case where cherry-picking from oslo should
 be allowed is when backporting patches to stable branches. Master
 branches should try to keep up-to-date with Oslo and sync everything
 every time.

 When we started Oslo incubator, we treated that code as trusted. But since
 then there have been occasional issues when syncing the code. So Oslo
 incubator code has lost *my* trust. Therefore I am always a hesitant to do a
 full Oslo sync because I am not an expert on the Oslo code and I risk
 breaking something when doing it (the issue may not appear 100% of the time
 too). Syncing code in becomes the first time that code is run against
 tempest, which scares me.

Understood and agreed, but frankly this defeats the intended purpose
IMO.  If we're going to go this route and never make them true libs
commonly shared/used then we're not gaining anything at all.

We might as well go back to maintaining our own versions in each
project and copying/pasting fixes around.  In essence that's exactly
what you end up with in this situation.

 I would like to propose having a integration test job in Oslo incubator that
 syncs in the code, similar to how we do global requirements.

 Additionally, what about a periodic jenkins job that does the Oslo syncs and
 is managed by the Oslo team itself?

Sure, that's a cool idea... once we get through an initial sync I
think it's doable.  As you've pointed out however there's some
challenges with projects that have been doing cherry picks or just
ignoring updates for any length of time.



 With that in mind, I'd like to request project's members to do
 periodic syncs from Oslo incubator. Yes, it is tedious, painful and
 sometimes requires more than just syncing, but we should all try to
 keep up-to-date with Oslo. The main reason why I'm asking this is
 precisely stable branches. If the project stays way behind the
 oslo-incubator, it'll be really painful to backport patches to stable
 branches in case of failures.

I'd agree with this for sure


 Unfortunately, there are projects that are quite behind from
 oslo-incubator master.

 One last comment. FWIW, backwards compatibility is always considered
 in all Oslo reviews and if there's a crazy-breaking change, it's
 always notified.

I agree there are challenges here, some due to being too far out of
date etc.  I also agree that a lot goes in to backward compat,
however, I think the emphasis on this and what drives a significant
number of the changes is Nova specific.  I also think that's where
most people's focus stops, I don't think there's significant checking
or testing across other projects.  

Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-21 Thread Ben Nemec

On 2014-01-21 12:28, Sean Dague wrote:

On 01/21/2014 01:14 PM, Joe Gordon wrote:


On Jan 17, 2014 12:24 AM, Flavio Percoco fla...@redhat.com
mailto:fla...@redhat.com wrote:


On 16/01/14 17:32 -0500, Doug Hellmann wrote:


On Thu, Jan 16, 2014 at 3:19 PM, Ben Nemec openst...@nemebean.com

mailto:openst...@nemebean.com wrote:


   On 2014-01-16 13:48, John Griffith wrote:

   Hey Everyone,

   A review came up today that cherry-picked a specific commit 
to

OSLO

   Incubator, without updating the rest of the files in the

module.  I
   rejected that patch, because my philosophy has been that when 
you

   update/pull from oslo-incubator it should be done as a full

sync of

   the entire module, not a cherry pick of the bits and pieces

that you

   may or may not be interested in.

   As it turns out I've received a bit of push back on this, so

it seems

   maybe I'm being unreasonable, or that I'm mistaken in my

understanding
   of the process here.  To me it seems like a complete and 
total

waste
   to have an oslo-incubator and common libs if you're going to 
turn

   around and just cherry pick changes, but maybe I'm completely

out of

   line.

   Thoughts??


   I suppose there might be exceptions, but in general I'm with you.

 For one

   thing, if someone tries to pull out a specific change in the Oslo

code,

   there's no guarantee that code even works.  Depending on how the

sync was
   done it's possible the code they're syncing never passed the Oslo 
unit
   tests in the form being synced, and since unit tests aren't 
synced

to the
   target projects it's conceivable that completely broken code 
could get

   through Jenkins.

   Obviously it's possible to do a successful partial sync, but for

the sake

   of reviewer sanity I'm -1 on partial syncs without a _very_ good

reason

   (like it's blocking the gate and there's some reason the full

module can't

   be synced).


I agree. Cherry picking a single (or even partial) commit really

should be

avoided.

The update tool does allow syncing just a single module, but that

should be

used very VERY carefully, especially because some of the changes

we're making
as we work on graduating some more libraries will include 
cross-dependent

changes between oslo modules.



Agrred. Syncing on master should be complete synchornization from 
Oslo

incubator. IMHO, the only case where cherry-picking from oslo should
be allowed is when backporting patches to stable branches. Master
branches should try to keep up-to-date with Oslo and sync everything
every time.


When we started Oslo incubator, we treated that code as trusted. But
since then there have been occasional issues when syncing the code. So
Oslo incubator code has lost *my* trust. Therefore I am always a
hesitant to do a full Oslo sync because I am not an expert on the Oslo
code and I risk breaking something when doing it (the issue may not
appear 100% of the time too). Syncing code in becomes the first time
that code is run against tempest, which scares me.

I would like to propose having a integration test job in Oslo 
incubator

that syncs in the code, similar to how we do global requirements.


That would be great, especially given that oslo currently doesn't run
it's own unit tests in parallel, which means it's a lot easier for non
parallel safe code to get in, and we don't bring the unit tests over to
the projects on sync.

Getting preemptive and doing a forward sync for validation would be
really valuable.

-Sean


That would be nice, but if we could do that then oslo wouldn't be an 
incubating project.  A fair number of the commits to oslo are 
incompatible with the current state of the consuming projects and 
require changes to work.  A sync job would have to be non-voting at 
best, and would be broken most, if not all, of the time while we wait to 
get the necessary changes into all of the consuming projects.


-Ben

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-17 Thread Flavio Percoco

On 16/01/14 17:32 -0500, Doug Hellmann wrote:

On Thu, Jan 16, 2014 at 3:19 PM, Ben Nemec openst...@nemebean.com wrote:

   On 2014-01-16 13:48, John Griffith wrote:

   Hey Everyone,

   A review came up today that cherry-picked a specific commit to OSLO
   Incubator, without updating the rest of the files in the module.  I
   rejected that patch, because my philosophy has been that when you
   update/pull from oslo-incubator it should be done as a full sync of
   the entire module, not a cherry pick of the bits and pieces that you
   may or may not be interested in.

   As it turns out I've received a bit of push back on this, so it seems
   maybe I'm being unreasonable, or that I'm mistaken in my understanding
   of the process here.  To me it seems like a complete and total waste
   to have an oslo-incubator and common libs if you're going to turn
   around and just cherry pick changes, but maybe I'm completely out of
   line.

   Thoughts??


   I suppose there might be exceptions, but in general I'm with you.  For one
   thing, if someone tries to pull out a specific change in the Oslo code,
   there's no guarantee that code even works.  Depending on how the sync was
   done it's possible the code they're syncing never passed the Oslo unit
   tests in the form being synced, and since unit tests aren't synced to the
   target projects it's conceivable that completely broken code could get
   through Jenkins.

   Obviously it's possible to do a successful partial sync, but for the sake
   of reviewer sanity I'm -1 on partial syncs without a _very_ good reason
   (like it's blocking the gate and there's some reason the full module can't
   be synced).


I agree. Cherry picking a single (or even partial) commit really should be
avoided.

The update tool does allow syncing just a single module, but that should be
used very VERY carefully, especially because some of the changes we're making
as we work on graduating some more libraries will include cross-dependent
changes between oslo modules.


Agrred. Syncing on master should be complete synchornization from Oslo
incubator. IMHO, the only case where cherry-picking from oslo should
be allowed is when backporting patches to stable branches. Master
branches should try to keep up-to-date with Oslo and sync everything
every time.

With that in mind, I'd like to request project's members to do
periodic syncs from Oslo incubator. Yes, it is tedious, painful and
sometimes requires more than just syncing, but we should all try to
keep up-to-date with Oslo. The main reason why I'm asking this is
precisely stable branches. If the project stays way behind the
oslo-incubator, it'll be really painful to backport patches to stable
branches in case of failures.

Unfortunately, there are projects that are quite behind from
oslo-incubator master.

One last comment. FWIW, backwards compatibility is always considered
in all Oslo reviews and if there's a crazy-breaking change, it's
always notified.

Thankfully, this all will be alleviated with the libs that are being
pulled out from the incubator. The syncs will contain fewer modules
and will be smaller.


I'm happy you brought this up now. I was meaning to do it.

Cheers,
FF


--
@flaper87
Flavio Percoco


pgpCcGdi9afUn.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-17 Thread Roman Podoliaka
Hi all,

Huge +1 for periodic syncs for two reasons:
1) it makes syncs smaller and thus easier
2) code in oslo-incubator often contains important bug fixes (e.g.
incorrect usage of eventlet TLS we found in Nova a few months ago)

Thanks,
Roman

On Fri, Jan 17, 2014 at 10:15 AM, Flavio Percoco fla...@redhat.com wrote:
 On 16/01/14 17:32 -0500, Doug Hellmann wrote:

 On Thu, Jan 16, 2014 at 3:19 PM, Ben Nemec openst...@nemebean.com wrote:

On 2014-01-16 13:48, John Griffith wrote:

Hey Everyone,

A review came up today that cherry-picked a specific commit to OSLO
Incubator, without updating the rest of the files in the module.  I
rejected that patch, because my philosophy has been that when you
update/pull from oslo-incubator it should be done as a full sync of
the entire module, not a cherry pick of the bits and pieces that
 you
may or may not be interested in.

As it turns out I've received a bit of push back on this, so it
 seems
maybe I'm being unreasonable, or that I'm mistaken in my
 understanding
of the process here.  To me it seems like a complete and total
 waste
to have an oslo-incubator and common libs if you're going to turn
around and just cherry pick changes, but maybe I'm completely out
 of
line.

Thoughts??


I suppose there might be exceptions, but in general I'm with you.  For
 one
thing, if someone tries to pull out a specific change in the Oslo code,
there's no guarantee that code even works.  Depending on how the sync
 was
done it's possible the code they're syncing never passed the Oslo unit
tests in the form being synced, and since unit tests aren't synced to
 the
target projects it's conceivable that completely broken code could get
through Jenkins.

Obviously it's possible to do a successful partial sync, but for the
 sake
of reviewer sanity I'm -1 on partial syncs without a _very_ good reason
(like it's blocking the gate and there's some reason the full module
 can't
be synced).


 I agree. Cherry picking a single (or even partial) commit really should be
 avoided.

 The update tool does allow syncing just a single module, but that should
 be
 used very VERY carefully, especially because some of the changes we're
 making
 as we work on graduating some more libraries will include cross-dependent
 changes between oslo modules.


 Agrred. Syncing on master should be complete synchornization from Oslo
 incubator. IMHO, the only case where cherry-picking from oslo should
 be allowed is when backporting patches to stable branches. Master
 branches should try to keep up-to-date with Oslo and sync everything
 every time.

 With that in mind, I'd like to request project's members to do
 periodic syncs from Oslo incubator. Yes, it is tedious, painful and
 sometimes requires more than just syncing, but we should all try to
 keep up-to-date with Oslo. The main reason why I'm asking this is
 precisely stable branches. If the project stays way behind the
 oslo-incubator, it'll be really painful to backport patches to stable
 branches in case of failures.

 Unfortunately, there are projects that are quite behind from
 oslo-incubator master.

 One last comment. FWIW, backwards compatibility is always considered
 in all Oslo reviews and if there's a crazy-breaking change, it's
 always notified.

 Thankfully, this all will be alleviated with the libs that are being
 pulled out from the incubator. The syncs will contain fewer modules
 and will be smaller.


 I'm happy you brought this up now. I was meaning to do it.

 Cheers,
 FF


 --
 @flaper87
 Flavio Percoco

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-17 Thread Julien Danjou
On Thu, Jan 16 2014, John Griffith wrote:

 As it turns out I've received a bit of push back on this, so it seems
 maybe I'm being unreasonable, or that I'm mistaken in my understanding
 of the process here.  To me it seems like a complete and total waste
 to have an oslo-incubator and common libs if you're going to turn
 around and just cherry pick changes, but maybe I'm completely out of
 line.

 Thoughts??

You're not, as others have already stated, I definitely think this
approach is better too.

-- 
Julien Danjou
# Free Software hacker # independent consultant
# http://julien.danjou.info


signature.asc
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-17 Thread John Griffith
On Fri, Jan 17, 2014 at 1:15 AM, Flavio Percoco fla...@redhat.com wrote:
 On 16/01/14 17:32 -0500, Doug Hellmann wrote:

 On Thu, Jan 16, 2014 at 3:19 PM, Ben Nemec openst...@nemebean.com wrote:

On 2014-01-16 13:48, John Griffith wrote:

Hey Everyone,

A review came up today that cherry-picked a specific commit to OSLO
Incubator, without updating the rest of the files in the module.  I
rejected that patch, because my philosophy has been that when you
update/pull from oslo-incubator it should be done as a full sync of
the entire module, not a cherry pick of the bits and pieces that
 you
may or may not be interested in.

As it turns out I've received a bit of push back on this, so it
 seems
maybe I'm being unreasonable, or that I'm mistaken in my
 understanding
of the process here.  To me it seems like a complete and total
 waste
to have an oslo-incubator and common libs if you're going to turn
around and just cherry pick changes, but maybe I'm completely out
 of
line.

Thoughts??


I suppose there might be exceptions, but in general I'm with you.  For
 one
thing, if someone tries to pull out a specific change in the Oslo code,
there's no guarantee that code even works.  Depending on how the sync
 was
done it's possible the code they're syncing never passed the Oslo unit
tests in the form being synced, and since unit tests aren't synced to
 the
target projects it's conceivable that completely broken code could get
through Jenkins.

Obviously it's possible to do a successful partial sync, but for the
 sake
of reviewer sanity I'm -1 on partial syncs without a _very_ good reason
(like it's blocking the gate and there's some reason the full module
 can't
be synced).


 I agree. Cherry picking a single (or even partial) commit really should be
 avoided.

 The update tool does allow syncing just a single module, but that should
 be
 used very VERY carefully, especially because some of the changes we're
 making
 as we work on graduating some more libraries will include cross-dependent
 changes between oslo modules.


 Agrred. Syncing on master should be complete synchornization from Oslo
 incubator. IMHO, the only case where cherry-picking from oslo should
 be allowed is when backporting patches to stable branches. Master
 branches should try to keep up-to-date with Oslo and sync everything
 every time.

 With that in mind, I'd like to request project's members to do
 periodic syncs from Oslo incubator. Yes, it is tedious, painful and
 sometimes requires more than just syncing, but we should all try to
 keep up-to-date with Oslo. The main reason why I'm asking this is
 precisely stable branches. If the project stays way behind the

Fully agree here, it's something we started in Cinder but sort of died
off and met some push-back (some of that admittedly was from myself at
the beginning).  It is something that we need to look at again though,
if nothing else to prevent falling so far behind that when we do need
a fix/update it's not a monumental undertaking to make it happen.

 oslo-incubator, it'll be really painful to backport patches to stable
 branches in case of failures.

 Unfortunately, there are projects that are quite behind from
 oslo-incubator master.

 One last comment. FWIW, backwards compatibility is always considered
 in all Oslo reviews and if there's a crazy-breaking change, it's
 always notified.

 Thankfully, this all will be alleviated with the libs that are being
 pulled out from the incubator. The syncs will contain fewer modules
 and will be smaller.


 I'm happy you brought this up now. I was meaning to do it.

 Cheers,
 FF


 --
 @flaper87
 Flavio Percoco

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-16 Thread Alex Meade
I think I agree with you John. Micromanaging the common library just makes more 
work. Is there any argument to not syncing everything? And do you mean 
cherry-picking only some of the changes in a single module?!? if that's the 
case then future syncing could get weird, really weird.

-Alex

-Original Message-
From: John Griffith john.griff...@solidfire.com
Sent: Thursday, January 16, 2014 2:48pm
To: OpenStack Development Mailing List openstack-dev@lists.openstack.org
Subject: [openstack-dev] [OpenStack-Dev] Cherry picking commit from 
oslo-incubator

Hey Everyone,

A review came up today that cherry-picked a specific commit to OSLO
Incubator, without updating the rest of the files in the module.  I
rejected that patch, because my philosophy has been that when you
update/pull from oslo-incubator it should be done as a full sync of
the entire module, not a cherry pick of the bits and pieces that you
may or may not be interested in.

As it turns out I've received a bit of push back on this, so it seems
maybe I'm being unreasonable, or that I'm mistaken in my understanding
of the process here.  To me it seems like a complete and total waste
to have an oslo-incubator and common libs if you're going to turn
around and just cherry pick changes, but maybe I'm completely out of
line.

Thoughts??

John

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-16 Thread Ben Nemec

On 2014-01-16 13:48, John Griffith wrote:

Hey Everyone,

A review came up today that cherry-picked a specific commit to OSLO
Incubator, without updating the rest of the files in the module.  I
rejected that patch, because my philosophy has been that when you
update/pull from oslo-incubator it should be done as a full sync of
the entire module, not a cherry pick of the bits and pieces that you
may or may not be interested in.

As it turns out I've received a bit of push back on this, so it seems
maybe I'm being unreasonable, or that I'm mistaken in my understanding
of the process here.  To me it seems like a complete and total waste
to have an oslo-incubator and common libs if you're going to turn
around and just cherry pick changes, but maybe I'm completely out of
line.

Thoughts??


I suppose there might be exceptions, but in general I'm with you.  For 
one thing, if someone tries to pull out a specific change in the Oslo 
code, there's no guarantee that code even works.  Depending on how the 
sync was done it's possible the code they're syncing never passed the 
Oslo unit tests in the form being synced, and since unit tests aren't 
synced to the target projects it's conceivable that completely broken 
code could get through Jenkins.


Obviously it's possible to do a successful partial sync, but for the 
sake of reviewer sanity I'm -1 on partial syncs without a _very_ good 
reason (like it's blocking the gate and there's some reason the full 
module can't be synced).


-Ben

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] Cherry picking commit from oslo-incubator

2014-01-16 Thread Doug Hellmann
On Thu, Jan 16, 2014 at 3:19 PM, Ben Nemec openst...@nemebean.com wrote:

 On 2014-01-16 13:48, John Griffith wrote:

 Hey Everyone,

 A review came up today that cherry-picked a specific commit to OSLO
 Incubator, without updating the rest of the files in the module.  I
 rejected that patch, because my philosophy has been that when you
 update/pull from oslo-incubator it should be done as a full sync of
 the entire module, not a cherry pick of the bits and pieces that you
 may or may not be interested in.

 As it turns out I've received a bit of push back on this, so it seems
 maybe I'm being unreasonable, or that I'm mistaken in my understanding
 of the process here.  To me it seems like a complete and total waste
 to have an oslo-incubator and common libs if you're going to turn
 around and just cherry pick changes, but maybe I'm completely out of
 line.

 Thoughts??


 I suppose there might be exceptions, but in general I'm with you.  For one
 thing, if someone tries to pull out a specific change in the Oslo code,
 there's no guarantee that code even works.  Depending on how the sync was
 done it's possible the code they're syncing never passed the Oslo unit
 tests in the form being synced, and since unit tests aren't synced to the
 target projects it's conceivable that completely broken code could get
 through Jenkins.

 Obviously it's possible to do a successful partial sync, but for the sake
 of reviewer sanity I'm -1 on partial syncs without a _very_ good reason
 (like it's blocking the gate and there's some reason the full module can't
 be synced).


I agree. Cherry picking a single (or even partial) commit really should be
avoided.

The update tool does allow syncing just a single module, but that should be
used very VERY carefully, especially because some of the changes we're
making as we work on graduating some more libraries will include
cross-dependent changes between oslo modules.

Doug





 -Ben


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev