Re: [Openstack] best practices for merging common into specific projects

[Thierry Carrez]

Mark McLoughlin wrote:
 On Thu, 2012-08-02 at 15:47 -0700, Vishvananda Ishaya wrote:
 On Aug 2, 2012, at 1:05 PM, Eric Windisch e...@cloudscaling.com wrote:

 The scope of common is expanding. I believe it is time to seriously
 consider a proper PTL. Preferably, before the PTL elections.

 +1
 
 So, I guess I've been doing this unofficially. I'm happy for that to be
 official until the next round elections.

The only difference between a regular team lead and a PTL is the
assignment of one seat on the Technical Committee. So the question is
not whether openstack-common should have a formal lead, the question is
whether the openstack-common lead should have an assigned seat on the
future Technical Committee (TC).

As you may know, I was advocating for a lean, directly-elected committee
of 9 members, with no reserved seats. The current PPB (responsible for
bootstrapping the TC) preferred another option: (elected) PTLs + 5
directly-elected members. My main opposition to that latter model was
that it was difficult to separate between projects worth an automatic
seat and projects not worth an automatic seat, and that it would
ultimately result in a bloated board as we continue to add new projects.
With the all-directly-elected model we wouldn't even be having this
discussion.

That said, I certainly think openstack-common is worth an automatic
seat, and that with 13 members we'd not yet cross a critical bloat
threshold. And Mark (or whoever else would get elected to common PTL)
would certainly get elected in a direct election anyway. So I support
that move, and will prepare a resolution to adjust the TC charter to
accommodate this change and propose it at the next PPB meeting.

I just think that at some point in a not-so-distant future we'll have to
reconsider the PTLs+5 setup to support further project growth while
keeping the TC dedicated and efficient.

Regards,

-- 
Thierry Carrez (ttx)
Release Manager, OpenStack

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Thierry Carrez]

Thierry Carrez wrote:
 So I support
 that move, and will prepare a resolution to adjust the TC charter to
 accommodate this change and propose it at the next PPB meeting.

Pushed to the PPB's next meeting agenda as:
http://wiki.openstack.org/Governance/Proposed/OpenStackCommonException

Cheers,

-- 
Thierry Carrez (ttx)
Release Manager, OpenStack

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Jay Pipes]

On 08/02/2012 08:52 PM, Eric Windisch wrote:

 What do you mean by membership services?
 See the email today from Yun Mao.  This is a proposal to have a pluggable 
 framework for integration services that maintain memberships. This was 
 originally desiged to replace the MySQL heartbeats in Nova, although there 
 will be a mysql-heartbeat backend by default as a drop-in replacement. There 
 is a zookeeper backend in the works, and we've discussed the possibility of 
 building a backend that can poll RabbitMQ's list_consumers.

Ah, yes. I've urged the team to use the term ServiceGroup instead of the
Zookeeper membership terminology -- as membership has other
connotations in Glance and Nova -- for instance, membership in a
project/tenant really has nothing to do with the concept of service
groups that can monitor response/hearbeat of service daemons.

Best,
-jay

 This is useful for more than just Nova's heartbeats, however.  This will 
 largely supplant the requirement for the matchmaker to build these backends 
 in itself, which had been my original plan (the matchmaker is already in 
 openstack-common).  As such, it had already been my intent to have a 
 MySQL-backed matchmaker.  The only thing new is that someone has actually 
 written the code.
 
 In the first pass, the intention is to leave the matchmaker in and introduce 
 the membership modules.  Then, the matchmaker would either use the new 
 membership modules as a backend, or even replaced entirely.
 
 Regards,
 Eric Windisch
 
 
 

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Eric Windisch]

 
 Ah, yes. I've urged the team to use the term ServiceGroup instead of the
 Zookeeper membership terminology -- as membership has other
 connotations in Glance and Nova -- for instance, membership in a
 project/tenant really has nothing to do with the concept of service
 groups that can monitor response/hearbeat of service daemons.
 
I see.  For some additional context, I'm looking to use this managing consumers 
of round-robin and fanout queues with the ZeroMQ driver, instead of the static 
hashmap that is used currently.

Regards,
Eric Windisch




___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Eric Windisch]

On Monday, July 23, 2012 at 12:04 PM, Doug Hellmann wrote:

   Sorry if this rekindles old arguments, but could someone summarize the
   reasons for an openstack-common PTL without voting rights? I would
   have defaulted to giving them a vote *especially* because the code in
   common is, well, common to all of the projects.
  
  
  So far, the PPB considered openstack-common to be driven by all PTLs,
  so it didn't have a specific PTL.
  
  As far as future governance is concerned (technical committee of the
  Foundation), openstack-common would technically be considered a
  supporting library (rather than a core project) -- those can have leads,
  but those do not get granted an automatic TC seat.
 
 
 OK, I can see the distinction there. I think the project needs an official 
 leader, even if we don't call them a PTL in the sense meant for other 
 projects. And I would expect anyone willing to take on the PTL role for 
 common to be qualified to run for one of the open positions on the new TC, if 
 they wanted to participate there.
The scope of common is expanding. I believe it is time to seriously consider a 
proper PTL. Preferably, before the PTL elections.

The RPC code is there now. We're talking about putting the membership services 
there too, for the sake of RPC, and even the low-level SQLAlchemy/MySQL access 
code for the sake of membership services. A wrapper around pyopenssl is likely 
to land there too, for the sake of RPC. These are just some of the changes that 
have already landed, or are expected to land within Folsom.

Common contains essential pieces to the success of OpenStack which are 
currently lacking (official) leadership. Everyone's problem is nobody's problem.

Consider this my +1 on assigning a PTL for common.

Regards,
Eric Windisch




___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Christopher B Ferris]

+1Cheers,Christopher FerrisIBM Distinguished Engineer, CTO Industry and Cloud StandardsMember, IBM Academy of TechnologyIBM Software Group, Standards Strategyemail: chris...@us.ibm.comTwitter: christo4ferrisphone: +1 508 234 2986-openstack-bounces+chrisfer=us.ibm@lists.launchpad.net wrote: -To: Doug Hellmann doug.hellm...@dreamhost.comFrom: Eric Windisch Sent by: openstack-bounces+chrisfer=us.ibm@lists.launchpad.netDate: 08/02/2012 04:59PMCc: Thierry Carrez thie...@openstack.org, openstack@lists.launchpad.netSubject: Re: [Openstack] best practices for merging common into specific projectsOn Monday, July 23, 2012 at 12:04 PM, Doug Hellmann wrote:   Sorry if this rekindles old arguments, but could someone summarize the   reasons for an openstack-common "PTL" without voting rights? I would   have defaulted to giving them a vote *especially* because the code in   common is, well, common to all of the projects.  So far, the PPB considered openstack-common to be driven by "all PTLs",  so it didn't have a specific PTL.As far as future governance is concerned (technical committee of the  Foundation), openstack-common would technically be considered a  supporting library (rather than a core project) -- those can have leads,  but those do not get granted an automatic TC seat.   OK, I can see the distinction there. I think the project needs an official leader, even if we don't call them a PTL in the sense meant for other projects. And I would expect anyone willing to take on the PTL role for common to be qualified to run for one of the open positions on the new TC, if they wanted to participate there.The scope of common is expanding. I believe it is time to seriously consider a proper PTL. Preferably, before the PTL elections.The RPC code is there now. We're talking about putting the membership services there too, for the sake of RPC, and even the low-level SQLAlchemy/MySQL access code for the sake of membership services. A wrapper around pyopenssl is likely to land there too, for the sake of RPC. These are just some of the changes that have already landed, or are expected to land within Folsom.Common contains essential pieces to the success of OpenStack which are currently lacking (official) leadership. Everyone's problem is nobody's problem.Consider this my +1 on assigning a PTL for common.Regards,Eric Windisch___Mailing list: https://launchpad.net/~openstackPost to   : openstack@lists.launchpad.netUnsubscribe : https://launchpad.net/~openstackMore help  : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Vishvananda Ishaya]

On Aug 2, 2012, at 1:05 PM, Eric Windisch e...@cloudscaling.com wrote:

 The scope of common is expanding. I believe it is time to seriously consider 
 a proper PTL. Preferably, before the PTL elections.

+1
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Jay Pipes]

On 08/02/2012 04:05 PM, Eric Windisch wrote:
 On Monday, July 23, 2012 at 12:04 PM, Doug Hellmann wrote:
 
 Sorry if this rekindles old arguments, but could someone summarize the
 reasons for an openstack-common PTL without voting rights? I would
 have defaulted to giving them a vote *especially* because the code in
 common is, well, common to all of the projects.


 So far, the PPB considered openstack-common to be driven by all PTLs,
 so it didn't have a specific PTL.

 As far as future governance is concerned (technical committee of the
 Foundation), openstack-common would technically be considered a
 supporting library (rather than a core project) -- those can have leads,
 but those do not get granted an automatic TC seat.


 OK, I can see the distinction there. I think the project needs an official 
 leader, even if we don't call them a PTL in the sense meant for other 
 projects. And I would expect anyone willing to take on the PTL role for 
 common to be qualified to run for one of the open positions on the new TC, 
 if they wanted to participate there.
 The scope of common is expanding. I believe it is time to seriously consider 
 a proper PTL. Preferably, before the PTL elections.

No disagreement from me.

 The RPC code is there now. We're talking about putting the membership 
 services there too, for the sake of RPC, and even the low-level 
 SQLAlchemy/MySQL access code for the sake of membership services. A wrapper 
 around pyopenssl is likely to land there too, for the sake of RPC. These are 
 just some of the changes that have already landed, or are expected to land 
 within Folsom.

What do you mean by membership services?

 Common contains essential pieces to the success of OpenStack which are 
 currently lacking (official) leadership. Everyone's problem is nobody's 
 problem.
 
 Consider this my +1 on assigning a PTL for common.

Sure, me too.

-jay

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Zhongyue Luo]

+1

On Fri, Aug 3, 2012 at 6:47 AM, Vishvananda Ishaya vishvana...@gmail.comwrote:


 On Aug 2, 2012, at 1:05 PM, Eric Windisch e...@cloudscaling.com wrote:

  The scope of common is expanding. I believe it is time to seriously
 consider a proper PTL. Preferably, before the PTL elections.

 +1
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp




-- 
*Intel SSG/SSD/SOTC/PRC/CITT*
880 Zixing Road, Zizhu Science Park, Minhang District, Shanghai, 200241,
China
+862161166500
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Eric Windisch]

 
 What do you mean by membership services?
See the email today from Yun Mao.  This is a proposal to have a pluggable 
framework for integration services that maintain memberships. This was 
originally desiged to replace the MySQL heartbeats in Nova, although there will 
be a mysql-heartbeat backend by default as a drop-in replacement. There is a 
zookeeper backend in the works, and we've discussed the possibility of building 
a backend that can poll RabbitMQ's list_consumers.

This is useful for more than just Nova's heartbeats, however.  This will 
largely supplant the requirement for the matchmaker to build these backends in 
itself, which had been my original plan (the matchmaker is already in 
openstack-common).  As such, it had already been my intent to have a 
MySQL-backed matchmaker.  The only thing new is that someone has actually 
written the code.

In the first pass, the intention is to leave the matchmaker in and introduce 
the membership modules.  Then, the matchmaker would either use the new 
membership modules as a backend, or even replaced entirely.

Regards,
Eric Windisch




___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Mark McLoughlin]

On Thu, 2012-08-02 at 15:47 -0700, Vishvananda Ishaya wrote:
 On Aug 2, 2012, at 1:05 PM, Eric Windisch e...@cloudscaling.com wrote:
 
  The scope of common is expanding. I believe it is time to seriously
  consider a proper PTL. Preferably, before the PTL elections.
 
 +1

So, I guess I've been doing this unofficially. I'm happy for that to be
official until the next round elections.

Cheers,
Mark.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Doug Hellmann]

On Wed, Jul 18, 2012 at 7:00 PM, Thierry Carrez thie...@openstack.orgwrote:

 Mark McLoughlin wrote:
  Making our multiple projects converge onto consolidated and
  well-accepted APIs is a bit painful work, but it is a prerequisite to
  turning openstack-common into a proper library (or set of libraries).
 
  I'd say the whole thing suffers from not having a proper
  team/leader/coordinator dedicated to it: relying on existing,
  overstretched PTLs to lead that effort might not be the fastest path.
 
  While I was on vacation, I read in the weekly newsletter:
 
It developed into a request for leadership for openstack-common
 
  and was like WTF do you call the work that e.g. I, Jason, Russell and
  Doug have been doing?
 
  But I see your point is a little different - you feel there should be an
  elected/appointed PTL without a PPB vote or whatever to represent the
  project. I guess that could help clarify things since it's what folks
  are used to with other projects.

 Right. So far we said that openstack-common was driven by all the
 PTLs, but that didn't prove particularly fast and efficient. Having a
 clear face associated with it, someone specific taking the lead on
 this project, will, I think, help a bit in getting to the next step.


Sorry if this rekindles old arguments, but could someone summarize the
reasons for an openstack-common PTL without voting rights? I would have
defaulted to giving them a vote *especially* because the code in common is,
well, common to all of the projects.

Doug
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Thierry Carrez]

Doug Hellmann wrote:
 
 On Wed, Jul 18, 2012 at 7:00 PM, Thierry Carrez thie...@openstack.org
 mailto:thie...@openstack.org wrote:
 
 Mark McLoughlin wrote:
  Making our multiple projects converge onto consolidated and
  well-accepted APIs is a bit painful work, but it is a prerequisite to
  turning openstack-common into a proper library (or set of libraries).
 
  I'd say the whole thing suffers from not having a proper
  team/leader/coordinator dedicated to it: relying on existing,
  overstretched PTLs to lead that effort might not be the fastest path.
 
  While I was on vacation, I read in the weekly newsletter:
 
It developed into a request for leadership for openstack-common
 
  and was like WTF do you call the work that e.g. I, Jason, Russell and
  Doug have been doing?
 
  But I see your point is a little different - you feel there should
 be an
  elected/appointed PTL without a PPB vote or whatever to
 represent the
  project. I guess that could help clarify things since it's what folks
  are used to with other projects.
 
 Right. So far we said that openstack-common was driven by all the
 PTLs, but that didn't prove particularly fast and efficient. Having a
 clear face associated with it, someone specific taking the lead on
 this project, will, I think, help a bit in getting to the next step.
 
 
 Sorry if this rekindles old arguments, but could someone summarize the
 reasons for an openstack-common PTL without voting rights? I would
 have defaulted to giving them a vote *especially* because the code in
 common is, well, common to all of the projects.

So far, the PPB considered openstack-common to be driven by all PTLs,
so it didn't have a specific PTL.

As far as future governance is concerned (technical committee of the
Foundation), openstack-common would technically be considered a
supporting library (rather than a core project) -- those can have leads,
but those do not get granted an automatic TC seat.

[ Avoiding the need to distinguish between worthy and unworthy
projects leads was one of the many reasons why I preferred the TC to be
completely directly-elected. ]

-- 
Thierry Carrez (ttx)
Release Manager, OpenStack

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Doug Hellmann]

On Mon, Jul 23, 2012 at 12:00 PM, Thierry Carrez thie...@openstack.orgwrote:

 Doug Hellmann wrote:
 
  On Wed, Jul 18, 2012 at 7:00 PM, Thierry Carrez thie...@openstack.org
  mailto:thie...@openstack.org wrote:
 
  Mark McLoughlin wrote:
   Making our multiple projects converge onto consolidated and
   well-accepted APIs is a bit painful work, but it is a
 prerequisite to
   turning openstack-common into a proper library (or set of
 libraries).
  
   I'd say the whole thing suffers from not having a proper
   team/leader/coordinator dedicated to it: relying on existing,
   overstretched PTLs to lead that effort might not be the fastest
 path.
  
   While I was on vacation, I read in the weekly newsletter:
  
 It developed into a request for leadership for openstack-common
  
   and was like WTF do you call the work that e.g. I, Jason, Russell
 and
   Doug have been doing?
  
   But I see your point is a little different - you feel there should
  be an
   elected/appointed PTL without a PPB vote or whatever to
  represent the
   project. I guess that could help clarify things since it's what
 folks
   are used to with other projects.
 
  Right. So far we said that openstack-common was driven by all the
  PTLs, but that didn't prove particularly fast and efficient. Having
 a
  clear face associated with it, someone specific taking the lead on
  this project, will, I think, help a bit in getting to the next step.
 
 
  Sorry if this rekindles old arguments, but could someone summarize the
  reasons for an openstack-common PTL without voting rights? I would
  have defaulted to giving them a vote *especially* because the code in
  common is, well, common to all of the projects.

 So far, the PPB considered openstack-common to be driven by all PTLs,
 so it didn't have a specific PTL.

 As far as future governance is concerned (technical committee of the
 Foundation), openstack-common would technically be considered a
 supporting library (rather than a core project) -- those can have leads,
 but those do not get granted an automatic TC seat.


OK, I can see the distinction there. I think the project needs an official
leader, even if we don't call them a PTL in the sense meant for other
projects. And I would expect anyone willing to take on the PTL role for
common to be qualified to run for one of the open positions on the new TC,
if they wanted to participate there.



 [ Avoiding the need to distinguish between worthy and unworthy
 projects leads was one of the many reasons why I preferred the TC to be
 completely directly-elected. ]


That does make sense.

Doug
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Mark McLoughlin]

(Sorry, was away for a couple of weeks)

On Mon, 2012-07-02 at 15:26 -0400, Russell Bryant wrote:
 On 07/02/2012 03:16 PM, Andrew Bogott wrote:
  Background:
  
  The openstack-common project is subject to a standard code-review
  process (and, soon, will also have Jenkins testing gates.)  Sadly,
  patches that are merged into openstack-common are essentially orphans. 
  Bringing those changes into actual use requires yet another step, a
  'merge from common' patch where the code changes in common are copied
  into a specific project (e.g. nova.)
  Merge-from-common patches are generated via an automated process. 
  Specific projects express dependencies on specific common components via
  a config file, e.g. 'nova/openstack-common.conf'.  The actual file copy
  is performed by 'openstack-common/update.py,' and its behavior is
  governed by the appropriate openstack-common.conf file.
 
 More background:
 
 http://wiki.openstack.org/CommonLibrary
 
  Questions:
  
  When should changes from common be merged into other projects?
  What should a 'merge-from-common' patch look like?
  What code-review standards should core committers observe when
  reviewing merge-from-common patches?
  
  Proposals:
  
  I.  As soon as a patch drops into common, the patch author should
  submit merge-from-common patches to all affected projects.
  A.  (This should really be done by a bot, but that's not going to
  happen overnight)
 
 All of the APIs in openstack-common right now are considered to be in
 incubation, meaning that breaking changes could be made.  I don't think
 automated merges are good for anything in incubation.
 
 Automation would be suitable for stable APIs.  Once an API is no longer
 in incubation, we should be looking at how to make releases and treat it
 as a proper library.  The copy/paste madness should be limited to APIs
 still in incubation.
 
 There are multiple APIs close or at the point where I think we should be
 able to commit to them.  I'll leave the specifics for a separate
 discussion, but I think moving on this front is key to reducing the pain
 we are seeing with code copying.

Well said.

  II. In the event that I. is not observed, merge-from-common patches
  will contain bits from multiple precursor patches.  That is not only OK,
  but encouraged.
  A.  Keeping projects in sync with common is important!
  B.  Asking producers of merge-from-common patches to understand the
  full diff will discourage the generation of such merges.
 
 I don't see this as much different as any other patches to nova (or
 whatever project is using common).  It should be a proper patch series.
  If the person pulling it in doesn't understand the merge well enough to
 produce the patch series with proper commit messages, then they are the
 wrong person to be doing the merge in the first place.

It sounds like there was some confusion as to what you meant here. All
people need to do to understand is look at how you have been syncing
stuff into Nova e.g.

  https://github.com/openstack/nova/commit/af387b9

or one of my syncs:

https://github.com/openstack/nova/commit/c1bf456

Cheers,
Mark.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Mark McLoughlin]

On Tue, 2012-07-03 at 18:59 +, Gabriel Hurley wrote:
 The notion that copying code is any protection against APIs that may
 change is a red herring. It's the exact same effect as pegging a
 version of a dependency (whether it's a commit hash or a real version
 number), except now you have code duplication. An unstable upgrade
 path is an unstable upgrade path, and copying the code into the
 project doesn't alleviate the pain for the project if the upstream
 library decides to change its APIs.

A library with an unstable upgrade path isn't a library.

Projects pegging to different versions of the same library is evil for
distributions.

Example:

  horizon pegs to openstack-common=0.2
  nova pegs to openstack-common=0.6
  glance pegs to openstack-common=0.5

We release Folsom and tell distributors that they need to package
versions 0.2, 0.5 and 0.6 of openstack-common?

No - the openstack-common library, when we start releasing it, will not
have an unstable upgrade path because it will not have unstable APIs.

 Also, we're really calling something used by more or less all the core
 projects incubated? ;-) Seems like it's past the proof-of-concept
 phase now, at least for many parts of common. Questions of API
 stability are an issue unto themselves.

If an individual APIs isn't stable, it's referred to as being in
incubation. That's all the term means in this context.

 Anyhow, I'm +1 on turning it into a real library of its own, as a
 couple people suggested already.

Superb. No-one is -1 on releasing it as a library. It's been the
documented plan from the beginning.

Cynically, I find people getting indignant about openstack-common's
update.py process quite hilarious.

Within OpenStack, there's been an ongoing culture of copying-and-pasting
code willy nilly between projects without any heed to the long-term
maintenance consequences. The openstack-common project is about paying
back some of that technical debt.

While we're preparing to do a first release of an openstack-common
library with stable APIs, we have this managed copy-and-paste process.
Yes, it sucks. But it sucks far less than the copy-and-pasting the
project has been doing all along.

Next time someone flames this managed copy-and-paste thing, I'm going
to dig through the git history and find examples of them copying and
pasting code between projects :-)

Oh, and lest anyone claim that the project is maturing and moving away
from the bad old days of copy-and-paste ... take a look at what we've
just done with Cinder. That's our most epic copy-and-paste yet!

Mark.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Mark McLoughlin]

On Tue, 2012-07-03 at 14:47 -0500, Andrew Bogott wrote:

 Like most people in this thread, I too long for an end to the weird 
 double-commit process that we're using now.  So I'm happy to set aside 
 my original Best Practices proposal until there's some consensus 
 regarding how much longer we're going to use that process.  Presumably 
 opinions about how to handle merge-from-common commits will vary in the 
 meantime, but that's something we can live with.
 
 In terms of promoting common into a real project, though, I want to 
 raise another option that's guaranteed to be unpopular:  We make 
 openstack-common a git-submodule that is automatically checked out 
 within the directory tree of each other project.

Several people have suggested using submodules and I've always said my
instinct says ick! but feel free to try it out and we'll consider it.

I've had bad experiences with git submodules. They confuse the heck out
of people.

And since this managed copy-and-paste process is for incubating APIs
only and we'll soon have library releases for stable APIs, I just don't
think it's worth the hassle of submodules.

Again, though, if someone shows how it can work and not be confusing for
people, I could easily be persuaded.

Cheers,
Mark.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Mark McLoughlin]

On Wed, 2012-07-04 at 11:57 +0200, Thierry Carrez wrote:
 Monty Taylor wrote:
  However, with a versioned library model, the projects can consume things
  pinned to specific versions, and then they can submit a change that
  updates the version depend and the code which needs to be updated to
  support the version change, and that change can be atomic.
  
  So honestly, I'd say the real key is getting us closer to the point
  where openstack-common is a proper library, because all of the rest of
  the complexity is stuff we're inventing to make life harder on
  ourselves, when the standard library with api-contract and a version
  model of the world works pretty fine without needing coordinated changes
  across multiple repositories.
 
 Yes, that's the end goal. And IMHO we are not very far away. I think the
 main reason we are not there yet is that while a lot of people enjoy
 giving their opinions about how openstack-common should be done and
 consumed by projects, not so many people follow up and actually do the work.

This is true :)

 Making our multiple projects converge onto consolidated and
 well-accepted APIs is a bit painful work, but it is a prerequisite to
 turning openstack-common into a proper library (or set of libraries).
 
 I'd say the whole thing suffers from not having a proper
 team/leader/coordinator dedicated to it: relying on existing,
 overstretched PTLs to lead that effort might not be the fastest path.

While I was on vacation, I read in the weekly newsletter:

  It developed into a request for leadership for openstack-common

and was like WTF do you call the work that e.g. I, Jason, Russell and
Doug have been doing?

But I see your point is a little different - you feel there should be an
elected/appointed PTL without a PPB vote or whatever to represent the
project. I guess that could help clarify things since it's what folks
are used to with other projects.

It would be a heavy crown, but I guess one of us would be fine with
wearing it :-)

Honestly, I think we would have done this before now if it wasn't going
to inevitably involve everyone getting their knickers in a twist about
whether the openstack-common PTL should have a vote. Meh.

Cheers,
Mark.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Thierry Carrez]

Mark McLoughlin wrote:
 Making our multiple projects converge onto consolidated and
 well-accepted APIs is a bit painful work, but it is a prerequisite to
 turning openstack-common into a proper library (or set of libraries).

 I'd say the whole thing suffers from not having a proper
 team/leader/coordinator dedicated to it: relying on existing,
 overstretched PTLs to lead that effort might not be the fastest path.
 
 While I was on vacation, I read in the weekly newsletter:
 
   It developed into a request for leadership for openstack-common
 
 and was like WTF do you call the work that e.g. I, Jason, Russell and
 Doug have been doing?
 
 But I see your point is a little different - you feel there should be an
 elected/appointed PTL without a PPB vote or whatever to represent the
 project. I guess that could help clarify things since it's what folks
 are used to with other projects.

Right. So far we said that openstack-common was driven by all the
PTLs, but that didn't prove particularly fast and efficient. Having a
clear face associated with it, someone specific taking the lead on
this project, will, I think, help a bit in getting to the next step.

-- 
Thierry Carrez (ttx)
Release Manager, OpenStack


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Christopher B Ferris]

-openstack-bounces+chrisfer=us.ibm@lists.launchpad.net wrote: -

To: openstack@lists.launchpad.net
From: Monty Taylor 

[snip]

So honestly, I'd say the real key is getting us closer to the point
where openstack-common is a proper library, because all of the rest of
the complexity is stuff we're inventing to make life harder on
ourselves, when the standard library with api-contract and a version
model of the world works pretty fine without needing coordinated changes
across multiple repositories.

+1

Cheers,

Christopher Ferris
IBM Distinguished Engineer, CTO Industry and Cloud Standards
Member, IBM Academy of Technology
IBM Software Group, Standards Strategy
email: chris...@us.ibm.com
Twitter: christo4ferris
phone: +1 508 234 2986


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Sean Dague]

On 07/03/2012 02:00 PM, Dan Prince wrote:

I don't see this as much different as any other patches to nova (or
whatever project is using common).  It should be a proper patch
series.
  If the person pulling it in doesn't understand the merge well enough
  to
produce the patch series with proper commit messages, then they are
the
wrong person to be doing the merge in the first place.


I went on a bit of a rant about this on IRC yesterday. While I agree a patch 
series is appropriate for many new features and bug fixes I don't think it 
should be required for keeping openstack-common in sync. Especially since we 
don't merge tests from openstack-common which would help verify that the person 
doing the merges doesn't mess up the order of the patchset. If we were to 
include the tests from openstack-common in each project this could change my 
mind.

If someone wants to split openstack-common changes into patchsets that might be 
Okay in small numbers. If you are merging say 5-10 changes from openstack 
common into all the various openstack projects that could translate into a 
rather large number of reviews (25+) for things that have been already reviewed 
once.  For me using patchsets to keep openstack-common in sync just causes 
thrashing of Jenkins, SmokeStack, etc. for things that have already been gated. 
Seems like an awful waste of review/CI time. In my opinion patchsets are the 
way to go with getting things into openstack-common... but not when syncing to 
projects.

Hopefully this situation is short lived however and we start using a proper 
library sooner rather than later.


+1. I think the bisectability isn't a clear win without the unit tests 
coming over, and the rest is just more reviews (which are already in 
scarce supply), and more jenkins time which just slows down everything else.


-Sean

--
Sean Dague
IBM Linux Technology Center
email: sda...@linux.vnet.ibm.com
alt-email: slda...@us.ibm.com



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Doug Hellmann]

On Tue, Jul 3, 2012 at 2:59 PM, Gabriel Hurley gabriel.hur...@nebula.comwrote:

 The notion that copying code is any protection against APIs that may
 change is a red herring. It's the exact same effect as pegging a version of
 a dependency (whether it's a commit hash or a real version number), except
 now you have code duplication. An unstable upgrade path is an unstable
 upgrade path, and copying the code into the project doesn't alleviate the
 pain for the project if the upstream library decides to change its APIs.


It does if the API being changed is the one that was copied because it
means the project doesn't have to respond to the change immediately. For
example, if we add something to the rpc library that ceilometer needs, nova
doesn't have to stop what they're doing to handle any potential changes. If
openstack-common was installed as a separate library, there wouldn't be a
(good) way to have both versions installed so ceilometer and nova couldn't
run on the same host (a requirement for the ceilometer agent).

So, I'm +1 on turning common into *several* libraries with their own
versioning scheme, when the components are ready. I don't see a need to
rush that as part of Folsom, though.

Doug
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Joshua Harlow]

Ya, let me know, I'll jump in and see what I can do also...

On 7/4/12 3:18 PM, Gabriel Hurley gabriel.hur...@nebula.com wrote:

Having a team/leader in that arena would definitely help. I'd contribute to 
common more if I knew what needed contributing, who to talk to about it, etc... 
Same goes for helping in terms of packaging, etc. to make it a proper common 
library.

- Gabriel

 -Original Message-
 From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net
 [mailto:openstack-
 bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of
 Thierry Carrez
 Sent: Wednesday, July 04, 2012 2:57 AM
 To: openstack@lists.launchpad.net
 Subject: Re: [Openstack] best practices for merging common into specific
 projects

 Monty Taylor wrote:
  However, with a versioned library model, the projects can consume
  things pinned to specific versions, and then they can submit a change
  that updates the version depend and the code which needs to be updated
  to support the version change, and that change can be atomic.
 
  So honestly, I'd say the real key is getting us closer to the point
  where openstack-common is a proper library, because all of the rest of
  the complexity is stuff we're inventing to make life harder on
  ourselves, when the standard library with api-contract and a version
  model of the world works pretty fine without needing coordinated
  changes across multiple repositories.

 Yes, that's the end goal. And IMHO we are not very far away. I think the main
 reason we are not there yet is that while a lot of people enjoy giving their
 opinions about how openstack-common should be done and consumed by
 projects, not so many people follow up and actually do the work.

 Making our multiple projects converge onto consolidated and well-accepted
 APIs is a bit painful work, but it is a prerequisite to turning openstack-
 common into a proper library (or set of libraries).

 I'd say the whole thing suffers from not having a proper
 team/leader/coordinator dedicated to it: relying on existing, overstretched
 PTLs to lead that effort might not be the fastest path.

 Regards,

 --
 Thierry Carrez (ttx)
 Release Manager, OpenStack


 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Thierry Carrez]

Monty Taylor wrote:
 However, with a versioned library model, the projects can consume things
 pinned to specific versions, and then they can submit a change that
 updates the version depend and the code which needs to be updated to
 support the version change, and that change can be atomic.
 
 So honestly, I'd say the real key is getting us closer to the point
 where openstack-common is a proper library, because all of the rest of
 the complexity is stuff we're inventing to make life harder on
 ourselves, when the standard library with api-contract and a version
 model of the world works pretty fine without needing coordinated changes
 across multiple repositories.

Yes, that's the end goal. And IMHO we are not very far away. I think the
main reason we are not there yet is that while a lot of people enjoy
giving their opinions about how openstack-common should be done and
consumed by projects, not so many people follow up and actually do the work.

Making our multiple projects converge onto consolidated and
well-accepted APIs is a bit painful work, but it is a prerequisite to
turning openstack-common into a proper library (or set of libraries).

I'd say the whole thing suffers from not having a proper
team/leader/coordinator dedicated to it: relying on existing,
overstretched PTLs to lead that effort might not be the fastest path.

Regards,

-- 
Thierry Carrez (ttx)
Release Manager, OpenStack


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Gabriel Hurley]

Having a team/leader in that arena would definitely help. I'd contribute to 
common more if I knew what needed contributing, who to talk to about it, etc... 
Same goes for helping in terms of packaging, etc. to make it a proper common 
library.

- Gabriel

 -Original Message-
 From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net
 [mailto:openstack-
 bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of
 Thierry Carrez
 Sent: Wednesday, July 04, 2012 2:57 AM
 To: openstack@lists.launchpad.net
 Subject: Re: [Openstack] best practices for merging common into specific
 projects
 
 Monty Taylor wrote:
  However, with a versioned library model, the projects can consume
  things pinned to specific versions, and then they can submit a change
  that updates the version depend and the code which needs to be updated
  to support the version change, and that change can be atomic.
 
  So honestly, I'd say the real key is getting us closer to the point
  where openstack-common is a proper library, because all of the rest of
  the complexity is stuff we're inventing to make life harder on
  ourselves, when the standard library with api-contract and a version
  model of the world works pretty fine without needing coordinated
  changes across multiple repositories.
 
 Yes, that's the end goal. And IMHO we are not very far away. I think the main
 reason we are not there yet is that while a lot of people enjoy giving their
 opinions about how openstack-common should be done and consumed by
 projects, not so many people follow up and actually do the work.
 
 Making our multiple projects converge onto consolidated and well-accepted
 APIs is a bit painful work, but it is a prerequisite to turning openstack-
 common into a proper library (or set of libraries).
 
 I'd say the whole thing suffers from not having a proper
 team/leader/coordinator dedicated to it: relying on existing, overstretched
 PTLs to lead that effort might not be the fastest path.
 
 Regards,
 
 --
 Thierry Carrez (ttx)
 Release Manager, OpenStack
 
 
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Eric Windisch]

On Tuesday, July 3, 2012 at 21:17 PM, Monty Taylor wrote:
 Interestingly enough - gerrit supports submodules pretty well... and it
 does exactly what Eric said below ... if both the project and
 superproject are in gerrit, and a change is made to the project, gerrit
 can automatically update the superproject reference.



Actually, I may not have been clear enough. I meant to say that we do NOT need 
to update the project reference automatically, this can be done manually, more 
reflective of the current development pattern. Yet, even if done manually, it 
would greatly minimize the double-commit scenario we have today. Someone could 
manually update the reference, which is a single-line change in a single 
commit, rather than merging and reintegrating many patches to many files.

 At least as an external library you can freeze a version requirement until 
 such time as you see fit to properly updated that code and *ensure* 
 compatibility in your project
Isn't this what we get with git submodules? Sure, that version is just a 
commit-id (or tag), but it isn't tracking HEAD, either. For stable releases, we 
can tag and update the reference to point to that tag.

-- 
Eric Windisch


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Thierry Carrez]

Gabriel Hurley wrote:
 On a more fundamental level, did I miss some tremendous reason why we have 
 this merge from common pattern instead of making OpenStack Common a 
 standard python dependency just like anything else? Especially with the work 
 Monty has recently done on versioning and packaging the client libs from 
 Jenkins, I can't see a reason to keep following this update common and merge 
 to everything else pattern at all...

This discussion should probably wait for markmc to come back, since he
set up most of this framework in the first place. He would certainly
produce a more compelling rationale than I can :)

IIRC the idea was to have openstack-common APIs in incubation until
some of them are stable enough that we can apply backward compatibility
for them at the level expected from any other decent Python library.
When we reach this point, those stable modules would be out of
incubation and released in a real openstack-common library. Unstable
APIs would stay in incubation and still use the merge model.

My understanding is that we are not yet at this point, especially as we
tweak/enrich openstack-common modules to make them acceptable by all the
projects...

-- 
Thierry Carrez (ttx)
Release Manager, OpenStack

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Thierry Carrez]

Thierry Carrez wrote:
 Gabriel Hurley wrote:
 On a more fundamental level, did I miss some tremendous reason why we have 
 this merge from common pattern instead of making OpenStack Common a 
 standard python dependency just like anything else? Especially with the work 
 Monty has recently done on versioning and packaging the client libs from 
 Jenkins, I can't see a reason to keep following this update common and 
 merge to everything else pattern at all...
 
 This discussion should probably wait for markmc to come back, since he
 set up most of this framework in the first place. He would certainly
 produce a more compelling rationale than I can :)

Actually http://wiki.openstack.org/CommonLibrary explains it quite well.
In particular:

openstack-common also provides a process for incubating APIs which,
while they are shared between multiple OpenStack projects, have not yet
matured to meet the [library inclusion] criteria described above.

Incubation shouldn't be seen as a long term option for any API - it is
merely a stepping stone to inclusion into the openstack-common library
proper.

-- 
Thierry Carrez (ttx)
Release Manager, OpenStack

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Doug Hellmann]

On Jul 3, 2012, at 5:31 AM, Thierry Carrez thie...@openstack.org wrote:

 Gabriel Hurley wrote:
 On a more fundamental level, did I miss some tremendous reason why we have 
 this merge from common pattern instead of making OpenStack Common a 
 standard python dependency just like anything else? Especially with the work 
 Monty has recently done on versioning and packaging the client libs from 
 Jenkins, I can't see a reason to keep following this update common and 
 merge to everything else pattern at all...
 
 This discussion should probably wait for markmc to come back, since he
 set up most of this framework in the first place. He would certainly
 produce a more compelling rationale than I can :)
 
 IIRC the idea was to have openstack-common APIs in incubation until
 some of them are stable enough that we can apply backward compatibility
 for them at the level expected from any other decent Python library.
 When we reach this point, those stable modules would be out of
 incubation and released in a real openstack-common library. Unstable
 APIs would stay in incubation and still use the merge model.
 
 My understanding is that we are not yet at this point, especially as we
 tweak/enrich openstack-common modules to make them acceptable by all the
 projects...
 

Ideally when we reach that point the libraries will be released as individual 
components instead of a monolithic shared library, too. 

Doug


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[James E. Blair]

Andrew Bogott abog...@wikimedia.org writes:

 I.  As soon as a patch drops into common, the patch author should
 submit merge-from-common patches to all affected projects.
 A.  (This should really be done by a bot, but that's not going to
 happen overnight)

Actually, I think with our current level of tooling, we can have Jenkins
do this (run by Zuul as a post-merge job on openstack-common).

I very much believe that the long-term goal should be to make
openstack-common a library -- so nothing I say here should be construed
against that.  But as long as it's in an incubation phase, if doing
something like this would help move things along, we can certainly
implement it, and fairly easily.

Note that a naive implementation might generate quite a bit of review
spam if several small changes land to openstack-common (there would then
be changes*projects number of reviews in gerrit).  We have some code
laying around which might be useful here that looks for an existing open
change and amends it; at least that would let us have at most only one
open merge-from-common-change per-project.

Okay, that's all on that; I don't want to derail the main conversation,
and I'd much rather it just be a library if we're close to being ready
for that.

-Jim

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Joshua Harlow]

I think that's a good little explanation as to why we have openstack-common, 
but when did it become a good reason to copy code around via an inclusion 
mechanism?

Lots of code is in packages (outside of openstack, in pypi and elsewhere) that 
is also in 'incubation' (in fact, what code isn't in perpetual incubation), 
that's why u still have version numbers?

I just worry about inclusion of code that isn't versioned into other projects, 
and I don't see the benefit of that when u can just have a package that has 
that code as well.

On 7/3/12 2:35 AM, Thierry Carrez thie...@openstack.org wrote:

Thierry Carrez wrote:
 Gabriel Hurley wrote:
 On a more fundamental level, did I miss some tremendous reason why we have 
 this merge from common pattern instead of making OpenStack Common a 
 standard python dependency just like anything else? Especially with the work 
 Monty has recently done on versioning and packaging the client libs from 
 Jenkins, I can't see a reason to keep following this update common and 
 merge to everything else pattern at all...

 This discussion should probably wait for markmc to come back, since he
 set up most of this framework in the first place. He would certainly
 produce a more compelling rationale than I can :)

Actually http://wiki.openstack.org/CommonLibrary explains it quite well.
In particular:

openstack-common also provides a process for incubating APIs which,
while they are shared between multiple OpenStack projects, have not yet
matured to meet the [library inclusion] criteria described above.

Incubation shouldn't be seen as a long term option for any API - it is
merely a stepping stone to inclusion into the openstack-common library
proper.

--
Thierry Carrez (ttx)
Release Manager, OpenStack

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Dan Prince]

- Original Message -
 From: Russell Bryant rbry...@redhat.com
 To: andrewbog...@gmail.com
 Cc: Andrew Bogott abog...@wikimedia.org, openstack@lists.launchpad.net
 Sent: Monday, July 2, 2012 3:26:56 PM
 Subject: Re: [Openstack] best practices for merging common into specific  
 projects
 
 On 07/02/2012 03:16 PM, Andrew Bogott wrote:
  Background:
  
  The openstack-common project is subject to a standard
  code-review
  process (and, soon, will also have Jenkins testing gates.)  Sadly,
  patches that are merged into openstack-common are essentially
  orphans.
  Bringing those changes into actual use requires yet another step, a
  'merge from common' patch where the code changes in common are
  copied
  into a specific project (e.g. nova.)
  Merge-from-common patches are generated via an automated
  process.
  Specific projects express dependencies on specific common
  components via
  a config file, e.g. 'nova/openstack-common.conf'.  The actual file
  copy
  is performed by 'openstack-common/update.py,' and its behavior is
  governed by the appropriate openstack-common.conf file.
 
 More background:
 
 http://wiki.openstack.org/CommonLibrary
 
  Questions:
  
  When should changes from common be merged into other projects?
  What should a 'merge-from-common' patch look like?
  What code-review standards should core committers observe when
  reviewing merge-from-common patches?
  
  Proposals:
  
  I.  As soon as a patch drops into common, the patch author
  should
  submit merge-from-common patches to all affected projects.
  A.  (This should really be done by a bot, but that's not going
  to
  happen overnight)
 
 All of the APIs in openstack-common right now are considered to be in
 incubation, meaning that breaking changes could be made.  I don't
 think
 automated merges are good for anything in incubation.
 
 Automation would be suitable for stable APIs.  Once an API is no
 longer
 in incubation, we should be looking at how to make releases and treat
 it
 as a proper library.  The copy/paste madness should be limited to
 APIs
 still in incubation.
 
 There are multiple APIs close or at the point where I think we should
 be
 able to commit to them.  I'll leave the specifics for a separate
 discussion, but I think moving on this front is key to reducing the
 pain
 we are seeing with code copying.
 
  II. In the event that I. is not observed, merge-from-common
  patches
  will contain bits from multiple precursor patches.  That is not
  only OK,
  but encouraged.
  A.  Keeping projects in sync with common is important!
  B.  Asking producers of merge-from-common patches to understand
  the
  full diff will discourage the generation of such merges.
 
 I don't see this as much different as any other patches to nova (or
 whatever project is using common).  It should be a proper patch
 series.
  If the person pulling it in doesn't understand the merge well enough
  to
 produce the patch series with proper commit messages, then they are
 the
 wrong person to be doing the merge in the first place.

I went on a bit of a rant about this on IRC yesterday. While I agree a patch 
series is appropriate for many new features and bug fixes I don't think it 
should be required for keeping openstack-common in sync. Especially since we 
don't merge tests from openstack-common which would help verify that the person 
doing the merges doesn't mess up the order of the patchset. If we were to 
include the tests from openstack-common in each project this could change my 
mind.

If someone wants to split openstack-common changes into patchsets that might be 
Okay in small numbers. If you are merging say 5-10 changes from openstack 
common into all the various openstack projects that could translate into a 
rather large number of reviews (25+) for things that have been already reviewed 
once.  For me using patchsets to keep openstack-common in sync just causes 
thrashing of Jenkins, SmokeStack, etc. for things that have already been gated. 
Seems like an awful waste of review/CI time. In my opinion patchsets are the 
way to go with getting things into openstack-common... but not when syncing to 
projects.

Hopefully this situation is short lived however and we start using a proper 
library sooner rather than later.


 
  III.Merge-from-common patches should be the product of a single
  unedited run of update.py.
 
 Disagree, see above.
 
  A.  If a merge-from-common patch breaks pep8 or a test in nova,
  don't fix the patch; fix the code in common.
 
 Agreed.
 
  IV.Merge-from-common patches are 'presumed justified'.  That
  means:
  A. Reviewers of merge-from-common patches should consider test
  failures and pep8 breakages, and obvious functional problems.
  B. Reviewers of merge-from-common patches should not consider
  the
  usefulness, design, etc. of merge-from-common patches.  Such
  concerns
  need to be handled within

Re: [Openstack] best practices for merging common into specific projects

[Gabriel Hurley]

The notion that copying code is any protection against APIs that may change is 
a red herring. It's the exact same effect as pegging a version of a dependency 
(whether it's a commit hash or a real version number), except now you have code 
duplication. An unstable upgrade path is an unstable upgrade path, and copying 
the code into the project doesn't alleviate the pain for the project if the 
upstream library decides to change its APIs.

Also, we're really calling something used by more or less all the core projects 
incubated? ;-) Seems like it's past the proof-of-concept phase now, at least 
for many parts of common. Questions of API stability are an issue unto 
themselves.

Anyhow, I'm +1 on turning it into a real library of its own, as a couple people 
suggested already.

- Gabriel

 -Original Message-
 From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net
 [mailto:openstack-
 bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of
 James E. Blair
 Sent: Tuesday, July 03, 2012 6:56 AM
 To: andrewbog...@gmail.com
 Cc: openstack@lists.launchpad.net
 Subject: Re: [Openstack] best practices for merging common into specific
 projects
 
 Andrew Bogott abog...@wikimedia.org writes:
 
  I.  As soon as a patch drops into common, the patch author should
  submit merge-from-common patches to all affected projects.
  A.  (This should really be done by a bot, but that's not going to
  happen overnight)
 
 Actually, I think with our current level of tooling, we can have Jenkins do 
 this
 (run by Zuul as a post-merge job on openstack-common).
 
 I very much believe that the long-term goal should be to make openstack-
 common a library -- so nothing I say here should be construed against that.
 But as long as it's in an incubation phase, if doing something like this would
 help move things along, we can certainly implement it, and fairly easily.
 
 Note that a naive implementation might generate quite a bit of review spam
 if several small changes land to openstack-common (there would then be
 changes*projects number of reviews in gerrit).  We have some code laying
 around which might be useful here that looks for an existing open change
 and amends it; at least that would let us have at most only one open merge-
 from-common-change per-project.
 
 Okay, that's all on that; I don't want to derail the main conversation, and 
 I'd
 much rather it just be a library if we're close to being ready for that.
 
 -Jim
 
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Joshua Harlow]

I 150% agree that is a red-herring, that's why I wonder what it really offers 
besides a 'façade' and/or the feeling that what u are using isn't a package, 
when in concept it really is, except now u have lost all the benefits of using 
version numbers, having dependency versions (with history) and all that, so the 
'façade' seems pretty weak imho.

+1 for library that follows the normal packaging methodology

On 7/3/12 11:59 AM, Gabriel Hurley gabriel.hur...@nebula.com wrote:

The notion that copying code is any protection against APIs that may change is 
a red herring. It's the exact same effect as pegging a version of a dependency 
(whether it's a commit hash or a real version number), except now you have code 
duplication. An unstable upgrade path is an unstable upgrade path, and copying 
the code into the project doesn't alleviate the pain for the project if the 
upstream library decides to change its APIs.

Also, we're really calling something used by more or less all the core projects 
incubated? ;-) Seems like it's past the proof-of-concept phase now, at least 
for many parts of common. Questions of API stability are an issue unto 
themselves.

Anyhow, I'm +1 on turning it into a real library of its own, as a couple people 
suggested already.

- Gabriel

 -Original Message-
 From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net
 [mailto:openstack-
 bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of
 James E. Blair
 Sent: Tuesday, July 03, 2012 6:56 AM
 To: andrewbog...@gmail.com
 Cc: openstack@lists.launchpad.net
 Subject: Re: [Openstack] best practices for merging common into specific
 projects

 Andrew Bogott abog...@wikimedia.org writes:

  I.  As soon as a patch drops into common, the patch author should
  submit merge-from-common patches to all affected projects.
  A.  (This should really be done by a bot, but that's not going to
  happen overnight)

 Actually, I think with our current level of tooling, we can have Jenkins do 
 this
 (run by Zuul as a post-merge job on openstack-common).

 I very much believe that the long-term goal should be to make openstack-
 common a library -- so nothing I say here should be construed against that.
 But as long as it's in an incubation phase, if doing something like this would
 help move things along, we can certainly implement it, and fairly easily.

 Note that a naive implementation might generate quite a bit of review spam
 if several small changes land to openstack-common (there would then be
 changes*projects number of reviews in gerrit).  We have some code laying
 around which might be useful here that looks for an existing open change
 and amends it; at least that would let us have at most only one open merge-
 from-common-change per-project.

 Okay, that's all on that; I don't want to derail the main conversation, and 
 I'd
 much rather it just be a library if we're close to being ready for that.

 -Jim

 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[James E. Blair]

Dan Prince dpri...@redhat.com writes:

 If someone wants to split openstack-common changes into patchsets that
 might be Okay in small numbers. If you are merging say 5-10 changes
 from openstack common into all the various openstack projects that
 could translate into a rather large number of reviews (25+) for things
 that have been already reviewed once.  For me using patchsets to keep
 openstack-common in sync just causes thrashing of Jenkins, SmokeStack,
 etc. for things that have already been gated. Seems like an awful
 waste of review/CI time. In my opinion patchsets are the way to go
 with getting things into openstack-common... but not when syncing to
 projects.

 Hopefully this situation is short lived however and we start using a
 proper library sooner rather than later.

Indeed, as a real library, it could be incorporated into the devstack
gate tests like the rest of the libraries, which would simultaneously
gate changes to it using all of the projects that reference it.

-Jim

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Eric Windisch]

git submodules don't have to be linked to the head of a branch. Instead of 
double-commiting (for every commit), we can do a single commit in each project 
to change the git reference of the submodule. This would not be too far from 
the existing behavior, except that it would minimize the double commits. 

-- 
Eric Windisch


On Tuesday, July 3, 2012 at 15:47 PM, Andrew Bogott wrote:

 On 7/3/12 1:59 PM, Gabriel Hurley wrote:
  The notion that copying code is any protection against APIs that may change 
  is a red herring. It's the exact same effect as pegging a version of a 
  dependency (whether it's a commit hash or a real version number), except 
  now you have code duplication. An unstable upgrade path is an unstable 
  upgrade path, and copying the code into the project doesn't alleviate the 
  pain for the project if the upstream library decides to change its APIs.
  
  Also, we're really calling something used by more or less all the core 
  projects incubated? ;-) Seems like it's past the proof-of-concept phase 
  now, at least for many parts of common. Questions of API stability are an 
  issue unto themselves.
  
  Anyhow, I'm +1 on turning it into a real library of its own, as a couple 
  people suggested already.
  
  - Gabriel
 
 I feel like I should speak up since I started this fight in the first 
 place :)
 
 Like most people in this thread, I too long for an end to the weird 
 double-commit process that we're using now. So I'm happy to set aside 
 my original Best Practices proposal until there's some consensus 
 regarding how much longer we're going to use that process. Presumably 
 opinions about how to handle merge-from-common commits will vary in the 
 meantime, but that's something we can live with.
 
 In terms of promoting common into a real project, though, I want to 
 raise another option that's guaranteed to be unpopular: We make 
 openstack-common a git-submodule that is automatically checked out 
 within the directory tree of each other project. Then each commit to 
 common would need to be gated by the full set of tests on every project 
 that includes common.
 
 I haven't thought deeply about the pros and cons of code submodule vs. 
 python project, but I want to bring up the option because it's the 
 system that I'm the most familiar with, and one that's been discussed a 
 bit off and on.
 
 -Andrew
 
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net (mailto:openstack@lists.launchpad.net)
 Unsubscribe : https://launchpad.net/~openstack
 More help : https://help.launchpad.net/ListHelp
 
 


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Andrew Bogott]

On 7/3/12 5:47 PM, Eric Windisch wrote:
git submodules don't have to be linked to the head of a branch. 
Instead of double-commiting (for every commit), we can do a single 
commit in each project to change the git reference of the submodule. 
This would not be too far from the existing behavior, except that it 
would minimize the double commits.


Oh, I guess I left out an important part of my vision, which is that 
there would be a commit hook in common which moves the submodule 
reference in the parent projects anytime a patch is merged in common.  
So, in short: once a patch passed review for inclusion in common, that 
patch would automatically go live in all other project heads simultaneously.



--
Eric Windisch

On Tuesday, July 3, 2012 at 15:47 PM, Andrew Bogott wrote:


On 7/3/12 1:59 PM, Gabriel Hurley wrote:
The notion that copying code is any protection against APIs that may 
change is a red herring. It's the exact same effect as pegging a 
version of a dependency (whether it's a commit hash or a real 
version number), except now you have code duplication. An unstable 
upgrade path is an unstable upgrade path, and copying the code into 
the project doesn't alleviate the pain for the project if the 
upstream library decides to change its APIs.


Also, we're really calling something used by more or less all the 
core projects incubated? ;-) Seems like it's past the 
proof-of-concept phase now, at least for many parts of common. 
Questions of API stability are an issue unto themselves.


Anyhow, I'm +1 on turning it into a real library of its own, as a 
couple people suggested already.


- Gabriel


I feel like I should speak up since I started this fight in the first
place :)

Like most people in this thread, I too long for an end to the weird
double-commit process that we're using now. So I'm happy to set aside
my original Best Practices proposal until there's some consensus
regarding how much longer we're going to use that process. Presumably
opinions about how to handle merge-from-common commits will vary in the
meantime, but that's something we can live with.

In terms of promoting common into a real project, though, I want to
raise another option that's guaranteed to be unpopular: We make
openstack-common a git-submodule that is automatically checked out
within the directory tree of each other project. Then each commit to
common would need to be gated by the full set of tests on every project
that includes common.

I haven't thought deeply about the pros and cons of code submodule vs.
python project, but I want to bring up the option because it's the
system that I'm the most familiar with, and one that's been discussed a
bit off and on.

-Andrew

___
Mailing list: https://launchpad.net/~openstack 
https://launchpad.net/%7Eopenstack
Post to : openstack@lists.launchpad.net 
mailto:openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack 
https://launchpad.net/%7Eopenstack

More help : https://help.launchpad.net/ListHelp




___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Gabriel Hurley]

I’m pretty -1 on triggering changes in other projects from common. That’s gonna 
result in a broken code (whether subtle or obvious) no matter how good your 
gates are.

At least as an external library you can freeze a version requirement until such 
time as you see fit to properly updated that code and *ensure* compatibility in 
your project.

Or if your project likes ridin’ trunk, then don’t pin a version and you’ve got 
the same effect as an automatic trigger.


-  Gabriel

From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net 
[mailto:openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net] On 
Behalf Of Andrew Bogott
Sent: Tuesday, July 03, 2012 3:54 PM
To: Eric Windisch; openstack@lists.launchpad.net
Subject: Re: [Openstack] best practices for merging common into specific 
projects

On 7/3/12 5:47 PM, Eric Windisch wrote:
git submodules don't have to be linked to the head of a branch. Instead of 
double-commiting (for every commit), we can do a single commit in each project 
to change the git reference of the submodule. This would not be too far from 
the existing behavior, except that it would minimize the double commits.

Oh, I guess I left out an important part of my vision, which is that there 
would be a commit hook in common which moves the submodule reference in the 
parent projects anytime a patch is merged in common.  So, in short: once a 
patch passed review for inclusion in common, that patch would automatically go 
live in all other project heads simultaneously.


--
Eric Windisch


On Tuesday, July 3, 2012 at 15:47 PM, Andrew Bogott wrote:
On 7/3/12 1:59 PM, Gabriel Hurley wrote:
The notion that copying code is any protection against APIs that may change is 
a red herring. It's the exact same effect as pegging a version of a dependency 
(whether it's a commit hash or a real version number), except now you have code 
duplication. An unstable upgrade path is an unstable upgrade path, and copying 
the code into the project doesn't alleviate the pain for the project if the 
upstream library decides to change its APIs.

Also, we're really calling something used by more or less all the core projects 
incubated? ;-) Seems like it's past the proof-of-concept phase now, at least 
for many parts of common. Questions of API stability are an issue unto 
themselves.

Anyhow, I'm +1 on turning it into a real library of its own, as a couple people 
suggested already.

- Gabriel

I feel like I should speak up since I started this fight in the first
place :)

Like most people in this thread, I too long for an end to the weird
double-commit process that we're using now. So I'm happy to set aside
my original Best Practices proposal until there's some consensus
regarding how much longer we're going to use that process. Presumably
opinions about how to handle merge-from-common commits will vary in the
meantime, but that's something we can live with.

In terms of promoting common into a real project, though, I want to
raise another option that's guaranteed to be unpopular: We make
openstack-common a git-submodule that is automatically checked out
within the directory tree of each other project. Then each commit to
common would need to be gated by the full set of tests on every project
that includes common.

I haven't thought deeply about the pros and cons of code submodule vs.
python project, but I want to bring up the option because it's the
system that I'm the most familiar with, and one that's been discussed a
bit off and on.

-Andrew

___
Mailing list: 
https://launchpad.net/~openstackhttps://launchpad.net/%7Eopenstack
Post to : openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net
Unsubscribe : 
https://launchpad.net/~openstackhttps://launchpad.net/%7Eopenstack
More help : https://help.launchpad.net/ListHelp


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Timothy Daly]

I've been following along at home a bit.  I can totally see where it's 
desirable to have well thought out APIs that you can commit to supporting and 
encourage other people to use.  And that you sometimes have expedient code that 
you aren't as comfortable with.  

What I don't get is how using a different mechanism to make libraries makes the 
code any less of a library.  Just make it a library using normal packaging 
methodology, call it version 0.1, and put a README that says we're not real 
comfortable with this API yet.  That accomplishes the same thing, but it's a 
lot less hairy.


Cheers,
Tim


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Monty Taylor]

Interestingly enough - gerrit supports submodules pretty well... and it
does exactly what Eric said below ... if both the project and
superproject are in gerrit, and a change is made to the project, gerrit
can automatically update the superproject reference.

Here's the thing though (and one of the reasons we haven't actually
earnestly suggested using this for anything yet) ... testing

If you propose a change to openstack-common and we were going to use the
auto-update feature, we'd need to test that that change doesn't break
ALL of the projects that use it. Now, what if nova is going to need a
patch to use the new version of the code. Oops. Complexity. Overload.
Everyone dies.

However, with a versioned library model, the projects can consume things
pinned to specific versions, and then they can submit a change that
updates the version depend and the code which needs to be updated to
support the version change, and that change can be atomic.

So honestly, I'd say the real key is getting us closer to the point
where openstack-common is a proper library, because all of the rest of
the complexity is stuff we're inventing to make life harder on
ourselves, when the standard library with api-contract and a version
model of the world works pretty fine without needing coordinated changes
across multiple repositories.

On 07/03/2012 06:54 PM, Gabriel Hurley wrote:
 I’m pretty -1 on triggering changes in other projects from common.
 That’s gonna result in a broken code (whether subtle or obvious) no
 matter how good your gates are.
 
  
 
 At least as an external library you can freeze a version requirement
 until such time as you see fit to properly updated that code and
 **ensure** compatibility in your project.
 
  
 
 Or if your project likes ridin’ trunk, then don’t pin a version and
 you’ve got the same effect as an automatic trigger.
 
  
 
 -  Gabriel
 
  
 
 *From:*openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net
 [mailto:openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net]
 *On Behalf Of *Andrew Bogott
 *Sent:* Tuesday, July 03, 2012 3:54 PM
 *To:* Eric Windisch; openstack@lists.launchpad.net
 *Subject:* Re: [Openstack] best practices for merging common into
 specific projects
 
  
 
 On 7/3/12 5:47 PM, Eric Windisch wrote:
 
 git submodules don't have to be linked to the head of a branch. Instead
 of double-commiting (for every commit), we can do a single commit in
 each project to change the git reference of the submodule. This would
 not be too far from the existing behavior, except that it would minimize
 the double commits.
 
  
 
 Oh, I guess I left out an important part of my vision, which is that
 there would be a commit hook in common which moves the submodule
 reference in the parent projects anytime a patch is merged in common. 
 So, in short: once a patch passed review for inclusion in common, that
 patch would automatically go live in all other project heads simultaneously.
 
 
 -- 
 Eric Windisch
 
  
 
 On Tuesday, July 3, 2012 at 15:47 PM, Andrew Bogott wrote:
 
 On 7/3/12 1:59 PM, Gabriel Hurley wrote:
 
 The notion that copying code is any protection against APIs that
 may change is a red herring. It's the exact same effect as
 pegging a version of a dependency (whether it's a commit hash or
 a real version number), except now you have code duplication. An
 unstable upgrade path is an unstable upgrade path, and copying
 the code into the project doesn't alleviate the pain for the
 project if the upstream library decides to change its APIs.
 
  
 
 Also, we're really calling something used by more or less all
 the core projects incubated? ;-) Seems like it's past the
 proof-of-concept phase now, at least for many parts of common.
 Questions of API stability are an issue unto themselves.
 
  
 
 Anyhow, I'm +1 on turning it into a real library of its own, as
 a couple people suggested already.
 
  
 
 - Gabriel
 
  
 
 I feel like I should speak up since I started this fight in the first
 
 place :)
 
  
 
 Like most people in this thread, I too long for an end to the weird
 
 double-commit process that we're using now. So I'm happy to set aside
 
 my original Best Practices proposal until there's some consensus
 
 regarding how much longer we're going to use that process. Presumably
 
 opinions about how to handle merge-from-common commits will vary in the
 
 meantime, but that's something we can live with.
 
  
 
 In terms of promoting common into a real project, though, I want to
 
 raise another option that's guaranteed to be unpopular: We make
 
 openstack-common a git-submodule that is automatically checked out
 
 within the directory tree of each other project. Then each commit to
 
 common would need to be gated by the full set of tests on every

[Openstack] best practices for merging common into specific projects

[Andrew Bogott]

Having spent some time last week writing code for openstack-common, 
and having spent yet more time trying to get those changes into Nova, I 
think it would be useful to define some best practices when crossing the 
boundary between common and other openstack projects.


Background:

The openstack-common project is subject to a standard code-review 
process (and, soon, will also have Jenkins testing gates.)  Sadly, 
patches that are merged into openstack-common are essentially orphans.  
Bringing those changes into actual use requires yet another step, a 
'merge from common' patch where the code changes in common are copied 
into a specific project (e.g. nova.)
Merge-from-common patches are generated via an automated process.  
Specific projects express dependencies on specific common components via 
a config file, e.g. 'nova/openstack-common.conf'.  The actual file copy 
is performed by 'openstack-common/update.py,' and its behavior is 
governed by the appropriate openstack-common.conf file.


Questions:

When should changes from common be merged into other projects?
What should a 'merge-from-common' patch look like?
What code-review standards should core committers observe when 
reviewing merge-from-common patches?


Proposals:

I.  As soon as a patch drops into common, the patch author should 
submit merge-from-common patches to all affected projects.
A.  (This should really be done by a bot, but that's not going to 
happen overnight)


II. In the event that I. is not observed, merge-from-common patches 
will contain bits from multiple precursor patches.  That is not only OK, 
but encouraged.

A.  Keeping projects in sync with common is important!
B.  Asking producers of merge-from-common patches to understand the 
full diff will discourage the generation of such merges.


III.Merge-from-common patches should be the product of a single 
unedited run of update.py.
A.  If a merge-from-common patch breaks pep8 or a test in nova, 
don't fix the patch; fix the code in common.


IV.Merge-from-common patches are 'presumed justified'.  That means:
A. Reviewers of merge-from-common patches should consider test 
failures and pep8 breakages, and obvious functional problems.
B. Reviewers of merge-from-common patches should not consider the 
usefulness, design, etc. of merge-from-common patches.  Such concerns 
need to be handled within the common review process.
C. Merges from common should get reviewed early and approved 
readily in order to avoid project divergence


V. Changes to openstack-common.conf are a special case.
A. Patches with openstack-common.conf changes should include the 
relevant merge-from-common changes.
B. Such patches should /not/ include any other merge-from-common 
changes.
C. Such patches should not only include the common merges, but 
should also include whatever code changes make use of the new merge.
D. Such patches require the same justification and scrutiny as any 
other standard project patch.


Please discuss!  If we're able to reach general agreement about this 
process, I will document the process in the openstack-common HACKING 
guidelines.


-Andrew




On 7/2/12 11:38 AM, Russell Bryant (Code Review) wrote:

I did that before seeing this patch.  However, I think I prefer what I pushed since it's 
individual updates explaining what they update instead of a blanket update 
everything commit.





___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Russell Bryant]

On 07/02/2012 03:16 PM, Andrew Bogott wrote:
 Background:
 
 The openstack-common project is subject to a standard code-review
 process (and, soon, will also have Jenkins testing gates.)  Sadly,
 patches that are merged into openstack-common are essentially orphans. 
 Bringing those changes into actual use requires yet another step, a
 'merge from common' patch where the code changes in common are copied
 into a specific project (e.g. nova.)
 Merge-from-common patches are generated via an automated process. 
 Specific projects express dependencies on specific common components via
 a config file, e.g. 'nova/openstack-common.conf'.  The actual file copy
 is performed by 'openstack-common/update.py,' and its behavior is
 governed by the appropriate openstack-common.conf file.

More background:

http://wiki.openstack.org/CommonLibrary

 Questions:
 
 When should changes from common be merged into other projects?
 What should a 'merge-from-common' patch look like?
 What code-review standards should core committers observe when
 reviewing merge-from-common patches?
 
 Proposals:
 
 I.  As soon as a patch drops into common, the patch author should
 submit merge-from-common patches to all affected projects.
 A.  (This should really be done by a bot, but that's not going to
 happen overnight)

All of the APIs in openstack-common right now are considered to be in
incubation, meaning that breaking changes could be made.  I don't think
automated merges are good for anything in incubation.

Automation would be suitable for stable APIs.  Once an API is no longer
in incubation, we should be looking at how to make releases and treat it
as a proper library.  The copy/paste madness should be limited to APIs
still in incubation.

There are multiple APIs close or at the point where I think we should be
able to commit to them.  I'll leave the specifics for a separate
discussion, but I think moving on this front is key to reducing the pain
we are seeing with code copying.

 II. In the event that I. is not observed, merge-from-common patches
 will contain bits from multiple precursor patches.  That is not only OK,
 but encouraged.
 A.  Keeping projects in sync with common is important!
 B.  Asking producers of merge-from-common patches to understand the
 full diff will discourage the generation of such merges.

I don't see this as much different as any other patches to nova (or
whatever project is using common).  It should be a proper patch series.
 If the person pulling it in doesn't understand the merge well enough to
produce the patch series with proper commit messages, then they are the
wrong person to be doing the merge in the first place.

 III.Merge-from-common patches should be the product of a single
 unedited run of update.py.

Disagree, see above.

 A.  If a merge-from-common patch breaks pep8 or a test in nova,
 don't fix the patch; fix the code in common.

Agreed.

 IV.Merge-from-common patches are 'presumed justified'.  That means:
 A. Reviewers of merge-from-common patches should consider test
 failures and pep8 breakages, and obvious functional problems.
 B. Reviewers of merge-from-common patches should not consider the
 usefulness, design, etc. of merge-from-common patches.  Such concerns
 need to be handled within the common review process.
 C. Merges from common should get reviewed early and approved readily
 in order to avoid project divergence

I think core reviewers of projects using openstack-common are justified
in doing critical review of new code being used by their project.  A
core reviewer of nova for examp[le may have a very good reason why the
code is not suitable for consumption in nova that openstack-common
reviewers didn't think of.

I don't think blind approvals are ever a good idea.

 V. Changes to openstack-common.conf are a special case.
 A. Patches with openstack-common.conf changes should include the
 relevant merge-from-common changes.
 B. Such patches should /not/ include any other merge-from-common
 changes.
 C. Such patches should not only include the common merges, but
 should also include whatever code changes make use of the new merge.
 D. Such patches require the same justification and scrutiny as any
 other standard project patch.

This all seems fine to me.

 Please discuss!  If we're able to reach general agreement about this
 process, I will document the process in the openstack-common HACKING
 guidelines.


-- 
Russell Bryant



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Joshua Harlow]

What about using openstack-common as a library instead of a preprocessor 
'inclusion' system/copy code around system??

Maybe its time for that to happen?

It always seemed sort of silly to me that files are being copied around to 
different projects like this, instead of referring to code in a common library 
package.

On 7/2/12 12:16 PM, Andrew Bogott abog...@wikimedia.org wrote:

Having spent some time last week writing code for openstack-common,
and having spent yet more time trying to get those changes into Nova, I
think it would be useful to define some best practices when crossing the
boundary between common and other openstack projects.

Background:

 The openstack-common project is subject to a standard code-review
process (and, soon, will also have Jenkins testing gates.)  Sadly,
patches that are merged into openstack-common are essentially orphans.
Bringing those changes into actual use requires yet another step, a
'merge from common' patch where the code changes in common are copied
into a specific project (e.g. nova.)
 Merge-from-common patches are generated via an automated process.
Specific projects express dependencies on specific common components via
a config file, e.g. 'nova/openstack-common.conf'.  The actual file copy
is performed by 'openstack-common/update.py,' and its behavior is
governed by the appropriate openstack-common.conf file.

Questions:

 When should changes from common be merged into other projects?
 What should a 'merge-from-common' patch look like?
 What code-review standards should core committers observe when
reviewing merge-from-common patches?

Proposals:

I.  As soon as a patch drops into common, the patch author should
submit merge-from-common patches to all affected projects.
 A.  (This should really be done by a bot, but that's not going to
happen overnight)

II. In the event that I. is not observed, merge-from-common patches
will contain bits from multiple precursor patches.  That is not only OK,
but encouraged.
 A.  Keeping projects in sync with common is important!
 B.  Asking producers of merge-from-common patches to understand the
full diff will discourage the generation of such merges.

III.Merge-from-common patches should be the product of a single
unedited run of update.py.
 A.  If a merge-from-common patch breaks pep8 or a test in nova,
don't fix the patch; fix the code in common.

IV.Merge-from-common patches are 'presumed justified'.  That means:
 A. Reviewers of merge-from-common patches should consider test
failures and pep8 breakages, and obvious functional problems.
 B. Reviewers of merge-from-common patches should not consider the
usefulness, design, etc. of merge-from-common patches.  Such concerns
need to be handled within the common review process.
 C. Merges from common should get reviewed early and approved
readily in order to avoid project divergence

V. Changes to openstack-common.conf are a special case.
 A. Patches with openstack-common.conf changes should include the
relevant merge-from-common changes.
 B. Such patches should /not/ include any other merge-from-common
changes.
 C. Such patches should not only include the common merges, but
should also include whatever code changes make use of the new merge.
 D. Such patches require the same justification and scrutiny as any
other standard project patch.

Please discuss!  If we're able to reach general agreement about this
process, I will document the process in the openstack-common HACKING
guidelines.

-Andrew




On 7/2/12 11:38 AM, Russell Bryant (Code Review) wrote:
 I did that before seeing this patch.  However, I think I prefer what I pushed 
 since it's individual updates explaining what they update instead of a 
 blanket update everything commit.




___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[Joshua Harlow]

Maybe its time to break out of that incubation??

Seems like if most projects are depending on these config files for code 
copying that that break out has really already occurred, but it just hasn't 
been written down or made official?

I don't quite understand how a project can be in incubation, but has components 
that are required for other projects to work?

Isn't this the same as having library versions where a project would pull in 
X.Y.Z openstack-common library version (with the features it wants) while the 
openstack-common people work on X.Y.Z + 1?

Then when X.Y.Z + 1 is approved 'stable' they can move to X.Y.Z+1.

On 7/2/12 12:26 PM, Russell Bryant rbry...@redhat.com wrote:

On 07/02/2012 03:16 PM, Andrew Bogott wrote:
 Background:

 The openstack-common project is subject to a standard code-review
 process (and, soon, will also have Jenkins testing gates.)  Sadly,
 patches that are merged into openstack-common are essentially orphans.
 Bringing those changes into actual use requires yet another step, a
 'merge from common' patch where the code changes in common are copied
 into a specific project (e.g. nova.)
 Merge-from-common patches are generated via an automated process.
 Specific projects express dependencies on specific common components via
 a config file, e.g. 'nova/openstack-common.conf'.  The actual file copy
 is performed by 'openstack-common/update.py,' and its behavior is
 governed by the appropriate openstack-common.conf file.

More background:

http://wiki.openstack.org/CommonLibrary

 Questions:

 When should changes from common be merged into other projects?
 What should a 'merge-from-common' patch look like?
 What code-review standards should core committers observe when
 reviewing merge-from-common patches?

 Proposals:

 I.  As soon as a patch drops into common, the patch author should
 submit merge-from-common patches to all affected projects.
 A.  (This should really be done by a bot, but that's not going to
 happen overnight)

All of the APIs in openstack-common right now are considered to be in
incubation, meaning that breaking changes could be made.  I don't think
automated merges are good for anything in incubation.

Automation would be suitable for stable APIs.  Once an API is no longer
in incubation, we should be looking at how to make releases and treat it
as a proper library.  The copy/paste madness should be limited to APIs
still in incubation.

There are multiple APIs close or at the point where I think we should be
able to commit to them.  I'll leave the specifics for a separate
discussion, but I think moving on this front is key to reducing the pain
we are seeing with code copying.

 II. In the event that I. is not observed, merge-from-common patches
 will contain bits from multiple precursor patches.  That is not only OK,
 but encouraged.
 A.  Keeping projects in sync with common is important!
 B.  Asking producers of merge-from-common patches to understand the
 full diff will discourage the generation of such merges.

I don't see this as much different as any other patches to nova (or
whatever project is using common).  It should be a proper patch series.
 If the person pulling it in doesn't understand the merge well enough to
produce the patch series with proper commit messages, then they are the
wrong person to be doing the merge in the first place.

 III.Merge-from-common patches should be the product of a single
 unedited run of update.py.

Disagree, see above.

 A.  If a merge-from-common patch breaks pep8 or a test in nova,
 don't fix the patch; fix the code in common.

Agreed.

 IV.Merge-from-common patches are 'presumed justified'.  That means:
 A. Reviewers of merge-from-common patches should consider test
 failures and pep8 breakages, and obvious functional problems.
 B. Reviewers of merge-from-common patches should not consider the
 usefulness, design, etc. of merge-from-common patches.  Such concerns
 need to be handled within the common review process.
 C. Merges from common should get reviewed early and approved readily
 in order to avoid project divergence

I think core reviewers of projects using openstack-common are justified
in doing critical review of new code being used by their project.  A
core reviewer of nova for examp[le may have a very good reason why the
code is not suitable for consumption in nova that openstack-common
reviewers didn't think of.

I don't think blind approvals are ever a good idea.

 V. Changes to openstack-common.conf are a special case.
 A. Patches with openstack-common.conf changes should include the
 relevant merge-from-common changes.
 B. Such patches should /not/ include any other merge-from-common
 changes.
 C. Such patches should not only include the common merges, but
 should also include whatever code changes make use of the new merge.
 D. Such patches require the same justification and scrutiny as any
 other standard 

Re: [Openstack] best practices for merging common into specific projects

[Gabriel Hurley]

On a more fundamental level, did I miss some tremendous reason why we have this 
merge from common pattern instead of making OpenStack Common a standard 
python dependency just like anything else? Especially with the work Monty has 
recently done on versioning and packaging the client libs from Jenkins, I can't 
see a reason to keep following this update common and merge to everything 
else pattern at all...

- Gabriel

 -Original Message-
 From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net
 [mailto:openstack-
 bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of
 Andrew Bogott
 Sent: Monday, July 02, 2012 12:17 PM
 To: openstack@lists.launchpad.net
 Subject: [Openstack] best practices for merging common into specific
 projects
 
  Having spent some time last week writing code for openstack-common,
 and having spent yet more time trying to get those changes into Nova, I think
 it would be useful to define some best practices when crossing the boundary
 between common and other openstack projects.
 
 Background:
 
  The openstack-common project is subject to a standard code-review
 process (and, soon, will also have Jenkins testing gates.)  Sadly, patches 
 that
 are merged into openstack-common are essentially orphans.
 Bringing those changes into actual use requires yet another step, a 'merge
 from common' patch where the code changes in common are copied into a
 specific project (e.g. nova.)
  Merge-from-common patches are generated via an automated process.
 Specific projects express dependencies on specific common components via
 a config file, e.g. 'nova/openstack-common.conf'.  The actual file copy is
 performed by 'openstack-common/update.py,' and its behavior is governed
 by the appropriate openstack-common.conf file.
 
 Questions:
 
  When should changes from common be merged into other projects?
  What should a 'merge-from-common' patch look like?
  What code-review standards should core committers observe when
 reviewing merge-from-common patches?
 
 Proposals:
 
 I.  As soon as a patch drops into common, the patch author should
 submit merge-from-common patches to all affected projects.
  A.  (This should really be done by a bot, but that's not going to happen
 overnight)
 
 II. In the event that I. is not observed, merge-from-common patches
 will contain bits from multiple precursor patches.  That is not only OK, but
 encouraged.
  A.  Keeping projects in sync with common is important!
  B.  Asking producers of merge-from-common patches to understand the
 full diff will discourage the generation of such merges.
 
 III.Merge-from-common patches should be the product of a single
 unedited run of update.py.
  A.  If a merge-from-common patch breaks pep8 or a test in nova, don't fix
 the patch; fix the code in common.
 
 IV.Merge-from-common patches are 'presumed justified'.  That means:
  A. Reviewers of merge-from-common patches should consider test
 failures and pep8 breakages, and obvious functional problems.
  B. Reviewers of merge-from-common patches should not consider the
 usefulness, design, etc. of merge-from-common patches.  Such concerns
 need to be handled within the common review process.
  C. Merges from common should get reviewed early and approved readily
 in order to avoid project divergence
 
 V. Changes to openstack-common.conf are a special case.
  A. Patches with openstack-common.conf changes should include the
 relevant merge-from-common changes.
  B. Such patches should /not/ include any other merge-from-common
 changes.
  C. Such patches should not only include the common merges, but should
 also include whatever code changes make use of the new merge.
  D. Such patches require the same justification and scrutiny as any other
 standard project patch.
 
 Please discuss!  If we're able to reach general agreement about this process,
 I will document the process in the openstack-common HACKING guidelines.
 
 -Andrew
 
 
 
 
 On 7/2/12 11:38 AM, Russell Bryant (Code Review) wrote:
  I did that before seeing this patch.  However, I think I prefer what I 
  pushed
 since it's individual updates explaining what they update instead of a blanket
 update everything commit.
 
 
 
 
 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] best practices for merging common into specific projects

[John Postlethwait]

I agree, I think moving them into one of the package/dependency managers the 
Open Stack projects already use is the proper way to do this.

The current process of copying the latest seemingly un-versionable code 
around into the projects really bothered me when it just happened to Horizon 
for the JSON parser a few weeks ago, I wash;t too keen on approving that review 
and I'd like to get away from it if possible, even if done by a bot of some 
kind...  


John Postlethwait
Nebula, Inc.
206-999-4492


On Monday, July 2, 2012 at 2:41 PM, Joshua Harlow wrote:

 Re: [Openstack] best practices for merging common into specific projects What 
 about using openstack-common as a library instead of a preprocessor 
 ‘inclusion’ system/copy code around system??
  
 Maybe its time for that to happen?  
  
 It always seemed sort of silly to me that files are being copied around to 
 different projects like this, instead of referring to code in a common 
 library package.
  
 On 7/2/12 12:16 PM, Andrew Bogott abog...@wikimedia.org wrote:
  
  Having spent some time last week writing code for openstack-common,
  and having spent yet more time trying to get those changes into Nova, I
  think it would be useful to define some best practices when crossing the
  boundary between common and other openstack projects.
   
  Background:
   
   The openstack-common project is subject to a standard code-review
  process (and, soon, will also have Jenkins testing gates.)  Sadly,
  patches that are merged into openstack-common are essentially orphans.  
  Bringing those changes into actual use requires yet another step, a
  'merge from common' patch where the code changes in common are copied
  into a specific project (e.g. nova.)
   Merge-from-common patches are generated via an automated process.  
  Specific projects express dependencies on specific common components via
  a config file, e.g. 'nova/openstack-common.conf'.  The actual file copy
  is performed by 'openstack-common/update.py,' and its behavior is
  governed by the appropriate openstack-common.conf file.
   
  Questions:
   
   When should changes from common be merged into other projects?
   What should a 'merge-from-common' patch look like?
   What code-review standards should core committers observe when
  reviewing merge-from-common patches?
   
  Proposals:
   
  I.  As soon as a patch drops into common, the patch author should
  submit merge-from-common patches to all affected projects.
   A.  (This should really be done by a bot, but that's not going to
  happen overnight)
   
  II. In the event that I. is not observed, merge-from-common patches
  will contain bits from multiple precursor patches.  That is not only OK,
  but encouraged.
   A.  Keeping projects in sync with common is important!
   B.  Asking producers of merge-from-common patches to understand the
  full diff will discourage the generation of such merges.
   
  III.Merge-from-common patches should be the product of a single
  unedited run of update.py.
   A.  If a merge-from-common patch breaks pep8 or a test in nova,
  don't fix the patch; fix the code in common.
   
  IV.Merge-from-common patches are 'presumed justified'.  That means:
   A. Reviewers of merge-from-common patches should consider test
  failures and pep8 breakages, and obvious functional problems.
   B. Reviewers of merge-from-common patches should not consider the
  usefulness, design, etc. of merge-from-common patches.  Such concerns
  need to be handled within the common review process.
   C. Merges from common should get reviewed early and approved
  readily in order to avoid project divergence
   
  V. Changes to openstack-common.conf are a special case.
   A. Patches with openstack-common.conf changes should include the
  relevant merge-from-common changes.
   B. Such patches should /not/ include any other merge-from-common
  changes.
   C. Such patches should not only include the common merges, but
  should also include whatever code changes make use of the new merge.
   D. Such patches require the same justification and scrutiny as any
  other standard project patch.
   
  Please discuss!  If we're able to reach general agreement about this
  process, I will document the process in the openstack-common HACKING
  guidelines.
   
  -Andrew
   
   
   
   
  On 7/2/12 11:38 AM, Russell Bryant (Code Review) wrote:
   I did that before seeing this patch.  However, I think I prefer what I 
   pushed since it's individual updates explaining what they update instead 
   of a blanket update everything commit.
  
  
   
   
  ___
  Mailing list: https://launchpad.net/~openstack
  Post to : openstack@lists.launchpad.net
  Unsubscribe : https://launchpad.net/~openstack
  More help   : https://help.launchpad.net/ListHelp
   
 ___
 Mailing list: https

Re: [Openstack] best practices for merging common into specific projects

[Christopher B Ferris]

-openstack-bounces+chrisfer=us.ibm@lists.launchpad.net wrote: -

To: andrewbog...@gmail.com andrewbog...@gmail.com, Andrew Bogott
abog...@wikimedia.org, openstack openstack@lists.launchpad.net
From: Joshua Harlow 
Sent by: openstack-bounces+chrisfer=us.ibm@lists.launchpad.net
Date: 07/02/2012 07:21PM
Subject: Re: [Openstack] best practices for merging common into
specific projects



Re: [Openstack] best practices for merging common into specific
projects


What about using openstack-common as a library instead of a
preprocessor #8216;inclusion#8217; system/copy code around system??



Maybe its time for that to happen? 



It always seemed sort of silly to me that files are being copied around
to different projects like this, instead of referring to code in a
common library package.

+1

Cheers,

Christopher Ferris
IBM Distinguished Engineer, CTO Industry and Cloud Standards
Member, IBM Academy of Technology
IBM Software Group, Standards Strategy
email: chris...@us.ibm.com
Twitter: christo4ferris
phone: +1 508 234 2986


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp