Re: [Openstack] best practices for merging common into specific projects
Mark McLoughlin wrote: On Thu, 2012-08-02 at 15:47 -0700, Vishvananda Ishaya wrote: On Aug 2, 2012, at 1:05 PM, Eric Windisch e...@cloudscaling.com wrote: The scope of common is expanding. I believe it is time to seriously consider a proper PTL. Preferably, before the PTL elections. +1 So, I guess I've been doing this unofficially. I'm happy for that to be official until the next round elections. The only difference between a regular team lead and a PTL is the assignment of one seat on the Technical Committee. So the question is not whether openstack-common should have a formal lead, the question is whether the openstack-common lead should have an assigned seat on the future Technical Committee (TC). As you may know, I was advocating for a lean, directly-elected committee of 9 members, with no reserved seats. The current PPB (responsible for bootstrapping the TC) preferred another option: (elected) PTLs + 5 directly-elected members. My main opposition to that latter model was that it was difficult to separate between projects worth an automatic seat and projects not worth an automatic seat, and that it would ultimately result in a bloated board as we continue to add new projects. With the all-directly-elected model we wouldn't even be having this discussion. That said, I certainly think openstack-common is worth an automatic seat, and that with 13 members we'd not yet cross a critical bloat threshold. And Mark (or whoever else would get elected to common PTL) would certainly get elected in a direct election anyway. So I support that move, and will prepare a resolution to adjust the TC charter to accommodate this change and propose it at the next PPB meeting. I just think that at some point in a not-so-distant future we'll have to reconsider the PTLs+5 setup to support further project growth while keeping the TC dedicated and efficient. Regards, -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
Thierry Carrez wrote: So I support that move, and will prepare a resolution to adjust the TC charter to accommodate this change and propose it at the next PPB meeting. Pushed to the PPB's next meeting agenda as: http://wiki.openstack.org/Governance/Proposed/OpenStackCommonException Cheers, -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
On 08/02/2012 08:52 PM, Eric Windisch wrote: What do you mean by membership services? See the email today from Yun Mao. This is a proposal to have a pluggable framework for integration services that maintain memberships. This was originally desiged to replace the MySQL heartbeats in Nova, although there will be a mysql-heartbeat backend by default as a drop-in replacement. There is a zookeeper backend in the works, and we've discussed the possibility of building a backend that can poll RabbitMQ's list_consumers. Ah, yes. I've urged the team to use the term ServiceGroup instead of the Zookeeper membership terminology -- as membership has other connotations in Glance and Nova -- for instance, membership in a project/tenant really has nothing to do with the concept of service groups that can monitor response/hearbeat of service daemons. Best, -jay This is useful for more than just Nova's heartbeats, however. This will largely supplant the requirement for the matchmaker to build these backends in itself, which had been my original plan (the matchmaker is already in openstack-common). As such, it had already been my intent to have a MySQL-backed matchmaker. The only thing new is that someone has actually written the code. In the first pass, the intention is to leave the matchmaker in and introduce the membership modules. Then, the matchmaker would either use the new membership modules as a backend, or even replaced entirely. Regards, Eric Windisch ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
Ah, yes. I've urged the team to use the term ServiceGroup instead of the Zookeeper membership terminology -- as membership has other connotations in Glance and Nova -- for instance, membership in a project/tenant really has nothing to do with the concept of service groups that can monitor response/hearbeat of service daemons. I see. For some additional context, I'm looking to use this managing consumers of round-robin and fanout queues with the ZeroMQ driver, instead of the static hashmap that is used currently. Regards, Eric Windisch ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
On Monday, July 23, 2012 at 12:04 PM, Doug Hellmann wrote: Sorry if this rekindles old arguments, but could someone summarize the reasons for an openstack-common PTL without voting rights? I would have defaulted to giving them a vote *especially* because the code in common is, well, common to all of the projects. So far, the PPB considered openstack-common to be driven by all PTLs, so it didn't have a specific PTL. As far as future governance is concerned (technical committee of the Foundation), openstack-common would technically be considered a supporting library (rather than a core project) -- those can have leads, but those do not get granted an automatic TC seat. OK, I can see the distinction there. I think the project needs an official leader, even if we don't call them a PTL in the sense meant for other projects. And I would expect anyone willing to take on the PTL role for common to be qualified to run for one of the open positions on the new TC, if they wanted to participate there. The scope of common is expanding. I believe it is time to seriously consider a proper PTL. Preferably, before the PTL elections. The RPC code is there now. We're talking about putting the membership services there too, for the sake of RPC, and even the low-level SQLAlchemy/MySQL access code for the sake of membership services. A wrapper around pyopenssl is likely to land there too, for the sake of RPC. These are just some of the changes that have already landed, or are expected to land within Folsom. Common contains essential pieces to the success of OpenStack which are currently lacking (official) leadership. Everyone's problem is nobody's problem. Consider this my +1 on assigning a PTL for common. Regards, Eric Windisch ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
+1Cheers,Christopher FerrisIBM Distinguished Engineer, CTO Industry and Cloud StandardsMember, IBM Academy of TechnologyIBM Software Group, Standards Strategyemail: chris...@us.ibm.comTwitter: christo4ferrisphone: +1 508 234 2986-openstack-bounces+chrisfer=us.ibm@lists.launchpad.net wrote: -To: Doug Hellmann doug.hellm...@dreamhost.comFrom: Eric WindischSent by: openstack-bounces+chrisfer=us.ibm@lists.launchpad.netDate: 08/02/2012 04:59PMCc: Thierry Carrez thie...@openstack.org, openstack@lists.launchpad.netSubject: Re: [Openstack] best practices for merging common into specific projectsOn Monday, July 23, 2012 at 12:04 PM, Doug Hellmann wrote: Sorry if this rekindles old arguments, but could someone summarize the reasons for an openstack-common "PTL" without voting rights? I would have defaulted to giving them a vote *especially* because the code in common is, well, common to all of the projects. So far, the PPB considered openstack-common to be driven by "all PTLs", so it didn't have a specific PTL.As far as future governance is concerned (technical committee of the Foundation), openstack-common would technically be considered a supporting library (rather than a core project) -- those can have leads, but those do not get granted an automatic TC seat. OK, I can see the distinction there. I think the project needs an official leader, even if we don't call them a PTL in the sense meant for other projects. And I would expect anyone willing to take on the PTL role for common to be qualified to run for one of the open positions on the new TC, if they wanted to participate there.The scope of common is expanding. I believe it is time to seriously consider a proper PTL. Preferably, before the PTL elections.The RPC code is there now. We're talking about putting the membership services there too, for the sake of RPC, and even the low-level SQLAlchemy/MySQL access code for the sake of membership services. A wrapper around pyopenssl is likely to land there too, for the sake of RPC. These are just some of the changes that have already landed, or are expected to land within Folsom.Common contains essential pieces to the success of OpenStack which are currently lacking (official) leadership. Everyone's problem is nobody's problem.Consider this my +1 on assigning a PTL for common.Regards,Eric Windisch___Mailing list: https://launchpad.net/~openstackPost to : openstack@lists.launchpad.netUnsubscribe : https://launchpad.net/~openstackMore help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
On Aug 2, 2012, at 1:05 PM, Eric Windisch e...@cloudscaling.com wrote: The scope of common is expanding. I believe it is time to seriously consider a proper PTL. Preferably, before the PTL elections. +1 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
On 08/02/2012 04:05 PM, Eric Windisch wrote: On Monday, July 23, 2012 at 12:04 PM, Doug Hellmann wrote: Sorry if this rekindles old arguments, but could someone summarize the reasons for an openstack-common PTL without voting rights? I would have defaulted to giving them a vote *especially* because the code in common is, well, common to all of the projects. So far, the PPB considered openstack-common to be driven by all PTLs, so it didn't have a specific PTL. As far as future governance is concerned (technical committee of the Foundation), openstack-common would technically be considered a supporting library (rather than a core project) -- those can have leads, but those do not get granted an automatic TC seat. OK, I can see the distinction there. I think the project needs an official leader, even if we don't call them a PTL in the sense meant for other projects. And I would expect anyone willing to take on the PTL role for common to be qualified to run for one of the open positions on the new TC, if they wanted to participate there. The scope of common is expanding. I believe it is time to seriously consider a proper PTL. Preferably, before the PTL elections. No disagreement from me. The RPC code is there now. We're talking about putting the membership services there too, for the sake of RPC, and even the low-level SQLAlchemy/MySQL access code for the sake of membership services. A wrapper around pyopenssl is likely to land there too, for the sake of RPC. These are just some of the changes that have already landed, or are expected to land within Folsom. What do you mean by membership services? Common contains essential pieces to the success of OpenStack which are currently lacking (official) leadership. Everyone's problem is nobody's problem. Consider this my +1 on assigning a PTL for common. Sure, me too. -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
+1 On Fri, Aug 3, 2012 at 6:47 AM, Vishvananda Ishaya vishvana...@gmail.comwrote: On Aug 2, 2012, at 1:05 PM, Eric Windisch e...@cloudscaling.com wrote: The scope of common is expanding. I believe it is time to seriously consider a proper PTL. Preferably, before the PTL elections. +1 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- *Intel SSG/SSD/SOTC/PRC/CITT* 880 Zixing Road, Zizhu Science Park, Minhang District, Shanghai, 200241, China +862161166500 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
What do you mean by membership services? See the email today from Yun Mao. This is a proposal to have a pluggable framework for integration services that maintain memberships. This was originally desiged to replace the MySQL heartbeats in Nova, although there will be a mysql-heartbeat backend by default as a drop-in replacement. There is a zookeeper backend in the works, and we've discussed the possibility of building a backend that can poll RabbitMQ's list_consumers. This is useful for more than just Nova's heartbeats, however. This will largely supplant the requirement for the matchmaker to build these backends in itself, which had been my original plan (the matchmaker is already in openstack-common). As such, it had already been my intent to have a MySQL-backed matchmaker. The only thing new is that someone has actually written the code. In the first pass, the intention is to leave the matchmaker in and introduce the membership modules. Then, the matchmaker would either use the new membership modules as a backend, or even replaced entirely. Regards, Eric Windisch ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
On Thu, 2012-08-02 at 15:47 -0700, Vishvananda Ishaya wrote: On Aug 2, 2012, at 1:05 PM, Eric Windisch e...@cloudscaling.com wrote: The scope of common is expanding. I believe it is time to seriously consider a proper PTL. Preferably, before the PTL elections. +1 So, I guess I've been doing this unofficially. I'm happy for that to be official until the next round elections. Cheers, Mark. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
On Wed, Jul 18, 2012 at 7:00 PM, Thierry Carrez thie...@openstack.orgwrote: Mark McLoughlin wrote: Making our multiple projects converge onto consolidated and well-accepted APIs is a bit painful work, but it is a prerequisite to turning openstack-common into a proper library (or set of libraries). I'd say the whole thing suffers from not having a proper team/leader/coordinator dedicated to it: relying on existing, overstretched PTLs to lead that effort might not be the fastest path. While I was on vacation, I read in the weekly newsletter: It developed into a request for leadership for openstack-common and was like WTF do you call the work that e.g. I, Jason, Russell and Doug have been doing? But I see your point is a little different - you feel there should be an elected/appointed PTL without a PPB vote or whatever to represent the project. I guess that could help clarify things since it's what folks are used to with other projects. Right. So far we said that openstack-common was driven by all the PTLs, but that didn't prove particularly fast and efficient. Having a clear face associated with it, someone specific taking the lead on this project, will, I think, help a bit in getting to the next step. Sorry if this rekindles old arguments, but could someone summarize the reasons for an openstack-common PTL without voting rights? I would have defaulted to giving them a vote *especially* because the code in common is, well, common to all of the projects. Doug ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
Doug Hellmann wrote: On Wed, Jul 18, 2012 at 7:00 PM, Thierry Carrez thie...@openstack.org mailto:thie...@openstack.org wrote: Mark McLoughlin wrote: Making our multiple projects converge onto consolidated and well-accepted APIs is a bit painful work, but it is a prerequisite to turning openstack-common into a proper library (or set of libraries). I'd say the whole thing suffers from not having a proper team/leader/coordinator dedicated to it: relying on existing, overstretched PTLs to lead that effort might not be the fastest path. While I was on vacation, I read in the weekly newsletter: It developed into a request for leadership for openstack-common and was like WTF do you call the work that e.g. I, Jason, Russell and Doug have been doing? But I see your point is a little different - you feel there should be an elected/appointed PTL without a PPB vote or whatever to represent the project. I guess that could help clarify things since it's what folks are used to with other projects. Right. So far we said that openstack-common was driven by all the PTLs, but that didn't prove particularly fast and efficient. Having a clear face associated with it, someone specific taking the lead on this project, will, I think, help a bit in getting to the next step. Sorry if this rekindles old arguments, but could someone summarize the reasons for an openstack-common PTL without voting rights? I would have defaulted to giving them a vote *especially* because the code in common is, well, common to all of the projects. So far, the PPB considered openstack-common to be driven by all PTLs, so it didn't have a specific PTL. As far as future governance is concerned (technical committee of the Foundation), openstack-common would technically be considered a supporting library (rather than a core project) -- those can have leads, but those do not get granted an automatic TC seat. [ Avoiding the need to distinguish between worthy and unworthy projects leads was one of the many reasons why I preferred the TC to be completely directly-elected. ] -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
On Mon, Jul 23, 2012 at 12:00 PM, Thierry Carrez thie...@openstack.orgwrote: Doug Hellmann wrote: On Wed, Jul 18, 2012 at 7:00 PM, Thierry Carrez thie...@openstack.org mailto:thie...@openstack.org wrote: Mark McLoughlin wrote: Making our multiple projects converge onto consolidated and well-accepted APIs is a bit painful work, but it is a prerequisite to turning openstack-common into a proper library (or set of libraries). I'd say the whole thing suffers from not having a proper team/leader/coordinator dedicated to it: relying on existing, overstretched PTLs to lead that effort might not be the fastest path. While I was on vacation, I read in the weekly newsletter: It developed into a request for leadership for openstack-common and was like WTF do you call the work that e.g. I, Jason, Russell and Doug have been doing? But I see your point is a little different - you feel there should be an elected/appointed PTL without a PPB vote or whatever to represent the project. I guess that could help clarify things since it's what folks are used to with other projects. Right. So far we said that openstack-common was driven by all the PTLs, but that didn't prove particularly fast and efficient. Having a clear face associated with it, someone specific taking the lead on this project, will, I think, help a bit in getting to the next step. Sorry if this rekindles old arguments, but could someone summarize the reasons for an openstack-common PTL without voting rights? I would have defaulted to giving them a vote *especially* because the code in common is, well, common to all of the projects. So far, the PPB considered openstack-common to be driven by all PTLs, so it didn't have a specific PTL. As far as future governance is concerned (technical committee of the Foundation), openstack-common would technically be considered a supporting library (rather than a core project) -- those can have leads, but those do not get granted an automatic TC seat. OK, I can see the distinction there. I think the project needs an official leader, even if we don't call them a PTL in the sense meant for other projects. And I would expect anyone willing to take on the PTL role for common to be qualified to run for one of the open positions on the new TC, if they wanted to participate there. [ Avoiding the need to distinguish between worthy and unworthy projects leads was one of the many reasons why I preferred the TC to be completely directly-elected. ] That does make sense. Doug ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
(Sorry, was away for a couple of weeks) On Mon, 2012-07-02 at 15:26 -0400, Russell Bryant wrote: On 07/02/2012 03:16 PM, Andrew Bogott wrote: Background: The openstack-common project is subject to a standard code-review process (and, soon, will also have Jenkins testing gates.) Sadly, patches that are merged into openstack-common are essentially orphans. Bringing those changes into actual use requires yet another step, a 'merge from common' patch where the code changes in common are copied into a specific project (e.g. nova.) Merge-from-common patches are generated via an automated process. Specific projects express dependencies on specific common components via a config file, e.g. 'nova/openstack-common.conf'. The actual file copy is performed by 'openstack-common/update.py,' and its behavior is governed by the appropriate openstack-common.conf file. More background: http://wiki.openstack.org/CommonLibrary Questions: When should changes from common be merged into other projects? What should a 'merge-from-common' patch look like? What code-review standards should core committers observe when reviewing merge-from-common patches? Proposals: I. As soon as a patch drops into common, the patch author should submit merge-from-common patches to all affected projects. A. (This should really be done by a bot, but that's not going to happen overnight) All of the APIs in openstack-common right now are considered to be in incubation, meaning that breaking changes could be made. I don't think automated merges are good for anything in incubation. Automation would be suitable for stable APIs. Once an API is no longer in incubation, we should be looking at how to make releases and treat it as a proper library. The copy/paste madness should be limited to APIs still in incubation. There are multiple APIs close or at the point where I think we should be able to commit to them. I'll leave the specifics for a separate discussion, but I think moving on this front is key to reducing the pain we are seeing with code copying. Well said. II. In the event that I. is not observed, merge-from-common patches will contain bits from multiple precursor patches. That is not only OK, but encouraged. A. Keeping projects in sync with common is important! B. Asking producers of merge-from-common patches to understand the full diff will discourage the generation of such merges. I don't see this as much different as any other patches to nova (or whatever project is using common). It should be a proper patch series. If the person pulling it in doesn't understand the merge well enough to produce the patch series with proper commit messages, then they are the wrong person to be doing the merge in the first place. It sounds like there was some confusion as to what you meant here. All people need to do to understand is look at how you have been syncing stuff into Nova e.g. https://github.com/openstack/nova/commit/af387b9 or one of my syncs: https://github.com/openstack/nova/commit/c1bf456 Cheers, Mark. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
On Tue, 2012-07-03 at 18:59 +, Gabriel Hurley wrote: The notion that copying code is any protection against APIs that may change is a red herring. It's the exact same effect as pegging a version of a dependency (whether it's a commit hash or a real version number), except now you have code duplication. An unstable upgrade path is an unstable upgrade path, and copying the code into the project doesn't alleviate the pain for the project if the upstream library decides to change its APIs. A library with an unstable upgrade path isn't a library. Projects pegging to different versions of the same library is evil for distributions. Example: horizon pegs to openstack-common=0.2 nova pegs to openstack-common=0.6 glance pegs to openstack-common=0.5 We release Folsom and tell distributors that they need to package versions 0.2, 0.5 and 0.6 of openstack-common? No - the openstack-common library, when we start releasing it, will not have an unstable upgrade path because it will not have unstable APIs. Also, we're really calling something used by more or less all the core projects incubated? ;-) Seems like it's past the proof-of-concept phase now, at least for many parts of common. Questions of API stability are an issue unto themselves. If an individual APIs isn't stable, it's referred to as being in incubation. That's all the term means in this context. Anyhow, I'm +1 on turning it into a real library of its own, as a couple people suggested already. Superb. No-one is -1 on releasing it as a library. It's been the documented plan from the beginning. Cynically, I find people getting indignant about openstack-common's update.py process quite hilarious. Within OpenStack, there's been an ongoing culture of copying-and-pasting code willy nilly between projects without any heed to the long-term maintenance consequences. The openstack-common project is about paying back some of that technical debt. While we're preparing to do a first release of an openstack-common library with stable APIs, we have this managed copy-and-paste process. Yes, it sucks. But it sucks far less than the copy-and-pasting the project has been doing all along. Next time someone flames this managed copy-and-paste thing, I'm going to dig through the git history and find examples of them copying and pasting code between projects :-) Oh, and lest anyone claim that the project is maturing and moving away from the bad old days of copy-and-paste ... take a look at what we've just done with Cinder. That's our most epic copy-and-paste yet! Mark. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
On Tue, 2012-07-03 at 14:47 -0500, Andrew Bogott wrote: Like most people in this thread, I too long for an end to the weird double-commit process that we're using now. So I'm happy to set aside my original Best Practices proposal until there's some consensus regarding how much longer we're going to use that process. Presumably opinions about how to handle merge-from-common commits will vary in the meantime, but that's something we can live with. In terms of promoting common into a real project, though, I want to raise another option that's guaranteed to be unpopular: We make openstack-common a git-submodule that is automatically checked out within the directory tree of each other project. Several people have suggested using submodules and I've always said my instinct says ick! but feel free to try it out and we'll consider it. I've had bad experiences with git submodules. They confuse the heck out of people. And since this managed copy-and-paste process is for incubating APIs only and we'll soon have library releases for stable APIs, I just don't think it's worth the hassle of submodules. Again, though, if someone shows how it can work and not be confusing for people, I could easily be persuaded. Cheers, Mark. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
On Wed, 2012-07-04 at 11:57 +0200, Thierry Carrez wrote: Monty Taylor wrote: However, with a versioned library model, the projects can consume things pinned to specific versions, and then they can submit a change that updates the version depend and the code which needs to be updated to support the version change, and that change can be atomic. So honestly, I'd say the real key is getting us closer to the point where openstack-common is a proper library, because all of the rest of the complexity is stuff we're inventing to make life harder on ourselves, when the standard library with api-contract and a version model of the world works pretty fine without needing coordinated changes across multiple repositories. Yes, that's the end goal. And IMHO we are not very far away. I think the main reason we are not there yet is that while a lot of people enjoy giving their opinions about how openstack-common should be done and consumed by projects, not so many people follow up and actually do the work. This is true :) Making our multiple projects converge onto consolidated and well-accepted APIs is a bit painful work, but it is a prerequisite to turning openstack-common into a proper library (or set of libraries). I'd say the whole thing suffers from not having a proper team/leader/coordinator dedicated to it: relying on existing, overstretched PTLs to lead that effort might not be the fastest path. While I was on vacation, I read in the weekly newsletter: It developed into a request for leadership for openstack-common and was like WTF do you call the work that e.g. I, Jason, Russell and Doug have been doing? But I see your point is a little different - you feel there should be an elected/appointed PTL without a PPB vote or whatever to represent the project. I guess that could help clarify things since it's what folks are used to with other projects. It would be a heavy crown, but I guess one of us would be fine with wearing it :-) Honestly, I think we would have done this before now if it wasn't going to inevitably involve everyone getting their knickers in a twist about whether the openstack-common PTL should have a vote. Meh. Cheers, Mark. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
Mark McLoughlin wrote: Making our multiple projects converge onto consolidated and well-accepted APIs is a bit painful work, but it is a prerequisite to turning openstack-common into a proper library (or set of libraries). I'd say the whole thing suffers from not having a proper team/leader/coordinator dedicated to it: relying on existing, overstretched PTLs to lead that effort might not be the fastest path. While I was on vacation, I read in the weekly newsletter: It developed into a request for leadership for openstack-common and was like WTF do you call the work that e.g. I, Jason, Russell and Doug have been doing? But I see your point is a little different - you feel there should be an elected/appointed PTL without a PPB vote or whatever to represent the project. I guess that could help clarify things since it's what folks are used to with other projects. Right. So far we said that openstack-common was driven by all the PTLs, but that didn't prove particularly fast and efficient. Having a clear face associated with it, someone specific taking the lead on this project, will, I think, help a bit in getting to the next step. -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
-openstack-bounces+chrisfer=us.ibm@lists.launchpad.net wrote: - To: openstack@lists.launchpad.net From: Monty Taylor [snip] So honestly, I'd say the real key is getting us closer to the point where openstack-common is a proper library, because all of the rest of the complexity is stuff we're inventing to make life harder on ourselves, when the standard library with api-contract and a version model of the world works pretty fine without needing coordinated changes across multiple repositories. +1 Cheers, Christopher Ferris IBM Distinguished Engineer, CTO Industry and Cloud Standards Member, IBM Academy of Technology IBM Software Group, Standards Strategy email: chris...@us.ibm.com Twitter: christo4ferris phone: +1 508 234 2986 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
On 07/03/2012 02:00 PM, Dan Prince wrote: I don't see this as much different as any other patches to nova (or whatever project is using common). It should be a proper patch series. If the person pulling it in doesn't understand the merge well enough to produce the patch series with proper commit messages, then they are the wrong person to be doing the merge in the first place. I went on a bit of a rant about this on IRC yesterday. While I agree a patch series is appropriate for many new features and bug fixes I don't think it should be required for keeping openstack-common in sync. Especially since we don't merge tests from openstack-common which would help verify that the person doing the merges doesn't mess up the order of the patchset. If we were to include the tests from openstack-common in each project this could change my mind. If someone wants to split openstack-common changes into patchsets that might be Okay in small numbers. If you are merging say 5-10 changes from openstack common into all the various openstack projects that could translate into a rather large number of reviews (25+) for things that have been already reviewed once. For me using patchsets to keep openstack-common in sync just causes thrashing of Jenkins, SmokeStack, etc. for things that have already been gated. Seems like an awful waste of review/CI time. In my opinion patchsets are the way to go with getting things into openstack-common... but not when syncing to projects. Hopefully this situation is short lived however and we start using a proper library sooner rather than later. +1. I think the bisectability isn't a clear win without the unit tests coming over, and the rest is just more reviews (which are already in scarce supply), and more jenkins time which just slows down everything else. -Sean -- Sean Dague IBM Linux Technology Center email: sda...@linux.vnet.ibm.com alt-email: slda...@us.ibm.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
On Tue, Jul 3, 2012 at 2:59 PM, Gabriel Hurley gabriel.hur...@nebula.comwrote: The notion that copying code is any protection against APIs that may change is a red herring. It's the exact same effect as pegging a version of a dependency (whether it's a commit hash or a real version number), except now you have code duplication. An unstable upgrade path is an unstable upgrade path, and copying the code into the project doesn't alleviate the pain for the project if the upstream library decides to change its APIs. It does if the API being changed is the one that was copied because it means the project doesn't have to respond to the change immediately. For example, if we add something to the rpc library that ceilometer needs, nova doesn't have to stop what they're doing to handle any potential changes. If openstack-common was installed as a separate library, there wouldn't be a (good) way to have both versions installed so ceilometer and nova couldn't run on the same host (a requirement for the ceilometer agent). So, I'm +1 on turning common into *several* libraries with their own versioning scheme, when the components are ready. I don't see a need to rush that as part of Folsom, though. Doug ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
Ya, let me know, I'll jump in and see what I can do also... On 7/4/12 3:18 PM, Gabriel Hurley gabriel.hur...@nebula.com wrote: Having a team/leader in that arena would definitely help. I'd contribute to common more if I knew what needed contributing, who to talk to about it, etc... Same goes for helping in terms of packaging, etc. to make it a proper common library. - Gabriel -Original Message- From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of Thierry Carrez Sent: Wednesday, July 04, 2012 2:57 AM To: openstack@lists.launchpad.net Subject: Re: [Openstack] best practices for merging common into specific projects Monty Taylor wrote: However, with a versioned library model, the projects can consume things pinned to specific versions, and then they can submit a change that updates the version depend and the code which needs to be updated to support the version change, and that change can be atomic. So honestly, I'd say the real key is getting us closer to the point where openstack-common is a proper library, because all of the rest of the complexity is stuff we're inventing to make life harder on ourselves, when the standard library with api-contract and a version model of the world works pretty fine without needing coordinated changes across multiple repositories. Yes, that's the end goal. And IMHO we are not very far away. I think the main reason we are not there yet is that while a lot of people enjoy giving their opinions about how openstack-common should be done and consumed by projects, not so many people follow up and actually do the work. Making our multiple projects converge onto consolidated and well-accepted APIs is a bit painful work, but it is a prerequisite to turning openstack- common into a proper library (or set of libraries). I'd say the whole thing suffers from not having a proper team/leader/coordinator dedicated to it: relying on existing, overstretched PTLs to lead that effort might not be the fastest path. Regards, -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
Monty Taylor wrote: However, with a versioned library model, the projects can consume things pinned to specific versions, and then they can submit a change that updates the version depend and the code which needs to be updated to support the version change, and that change can be atomic. So honestly, I'd say the real key is getting us closer to the point where openstack-common is a proper library, because all of the rest of the complexity is stuff we're inventing to make life harder on ourselves, when the standard library with api-contract and a version model of the world works pretty fine without needing coordinated changes across multiple repositories. Yes, that's the end goal. And IMHO we are not very far away. I think the main reason we are not there yet is that while a lot of people enjoy giving their opinions about how openstack-common should be done and consumed by projects, not so many people follow up and actually do the work. Making our multiple projects converge onto consolidated and well-accepted APIs is a bit painful work, but it is a prerequisite to turning openstack-common into a proper library (or set of libraries). I'd say the whole thing suffers from not having a proper team/leader/coordinator dedicated to it: relying on existing, overstretched PTLs to lead that effort might not be the fastest path. Regards, -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
Having a team/leader in that arena would definitely help. I'd contribute to common more if I knew what needed contributing, who to talk to about it, etc... Same goes for helping in terms of packaging, etc. to make it a proper common library. - Gabriel -Original Message- From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of Thierry Carrez Sent: Wednesday, July 04, 2012 2:57 AM To: openstack@lists.launchpad.net Subject: Re: [Openstack] best practices for merging common into specific projects Monty Taylor wrote: However, with a versioned library model, the projects can consume things pinned to specific versions, and then they can submit a change that updates the version depend and the code which needs to be updated to support the version change, and that change can be atomic. So honestly, I'd say the real key is getting us closer to the point where openstack-common is a proper library, because all of the rest of the complexity is stuff we're inventing to make life harder on ourselves, when the standard library with api-contract and a version model of the world works pretty fine without needing coordinated changes across multiple repositories. Yes, that's the end goal. And IMHO we are not very far away. I think the main reason we are not there yet is that while a lot of people enjoy giving their opinions about how openstack-common should be done and consumed by projects, not so many people follow up and actually do the work. Making our multiple projects converge onto consolidated and well-accepted APIs is a bit painful work, but it is a prerequisite to turning openstack- common into a proper library (or set of libraries). I'd say the whole thing suffers from not having a proper team/leader/coordinator dedicated to it: relying on existing, overstretched PTLs to lead that effort might not be the fastest path. Regards, -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
On Tuesday, July 3, 2012 at 21:17 PM, Monty Taylor wrote: Interestingly enough - gerrit supports submodules pretty well... and it does exactly what Eric said below ... if both the project and superproject are in gerrit, and a change is made to the project, gerrit can automatically update the superproject reference. Actually, I may not have been clear enough. I meant to say that we do NOT need to update the project reference automatically, this can be done manually, more reflective of the current development pattern. Yet, even if done manually, it would greatly minimize the double-commit scenario we have today. Someone could manually update the reference, which is a single-line change in a single commit, rather than merging and reintegrating many patches to many files. At least as an external library you can freeze a version requirement until such time as you see fit to properly updated that code and *ensure* compatibility in your project Isn't this what we get with git submodules? Sure, that version is just a commit-id (or tag), but it isn't tracking HEAD, either. For stable releases, we can tag and update the reference to point to that tag. -- Eric Windisch ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
Gabriel Hurley wrote: On a more fundamental level, did I miss some tremendous reason why we have this merge from common pattern instead of making OpenStack Common a standard python dependency just like anything else? Especially with the work Monty has recently done on versioning and packaging the client libs from Jenkins, I can't see a reason to keep following this update common and merge to everything else pattern at all... This discussion should probably wait for markmc to come back, since he set up most of this framework in the first place. He would certainly produce a more compelling rationale than I can :) IIRC the idea was to have openstack-common APIs in incubation until some of them are stable enough that we can apply backward compatibility for them at the level expected from any other decent Python library. When we reach this point, those stable modules would be out of incubation and released in a real openstack-common library. Unstable APIs would stay in incubation and still use the merge model. My understanding is that we are not yet at this point, especially as we tweak/enrich openstack-common modules to make them acceptable by all the projects... -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
Thierry Carrez wrote: Gabriel Hurley wrote: On a more fundamental level, did I miss some tremendous reason why we have this merge from common pattern instead of making OpenStack Common a standard python dependency just like anything else? Especially with the work Monty has recently done on versioning and packaging the client libs from Jenkins, I can't see a reason to keep following this update common and merge to everything else pattern at all... This discussion should probably wait for markmc to come back, since he set up most of this framework in the first place. He would certainly produce a more compelling rationale than I can :) Actually http://wiki.openstack.org/CommonLibrary explains it quite well. In particular: openstack-common also provides a process for incubating APIs which, while they are shared between multiple OpenStack projects, have not yet matured to meet the [library inclusion] criteria described above. Incubation shouldn't be seen as a long term option for any API - it is merely a stepping stone to inclusion into the openstack-common library proper. -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
On Jul 3, 2012, at 5:31 AM, Thierry Carrez thie...@openstack.org wrote: Gabriel Hurley wrote: On a more fundamental level, did I miss some tremendous reason why we have this merge from common pattern instead of making OpenStack Common a standard python dependency just like anything else? Especially with the work Monty has recently done on versioning and packaging the client libs from Jenkins, I can't see a reason to keep following this update common and merge to everything else pattern at all... This discussion should probably wait for markmc to come back, since he set up most of this framework in the first place. He would certainly produce a more compelling rationale than I can :) IIRC the idea was to have openstack-common APIs in incubation until some of them are stable enough that we can apply backward compatibility for them at the level expected from any other decent Python library. When we reach this point, those stable modules would be out of incubation and released in a real openstack-common library. Unstable APIs would stay in incubation and still use the merge model. My understanding is that we are not yet at this point, especially as we tweak/enrich openstack-common modules to make them acceptable by all the projects... Ideally when we reach that point the libraries will be released as individual components instead of a monolithic shared library, too. Doug ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
Andrew Bogott abog...@wikimedia.org writes: I. As soon as a patch drops into common, the patch author should submit merge-from-common patches to all affected projects. A. (This should really be done by a bot, but that's not going to happen overnight) Actually, I think with our current level of tooling, we can have Jenkins do this (run by Zuul as a post-merge job on openstack-common). I very much believe that the long-term goal should be to make openstack-common a library -- so nothing I say here should be construed against that. But as long as it's in an incubation phase, if doing something like this would help move things along, we can certainly implement it, and fairly easily. Note that a naive implementation might generate quite a bit of review spam if several small changes land to openstack-common (there would then be changes*projects number of reviews in gerrit). We have some code laying around which might be useful here that looks for an existing open change and amends it; at least that would let us have at most only one open merge-from-common-change per-project. Okay, that's all on that; I don't want to derail the main conversation, and I'd much rather it just be a library if we're close to being ready for that. -Jim ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
I think that's a good little explanation as to why we have openstack-common, but when did it become a good reason to copy code around via an inclusion mechanism? Lots of code is in packages (outside of openstack, in pypi and elsewhere) that is also in 'incubation' (in fact, what code isn't in perpetual incubation), that's why u still have version numbers? I just worry about inclusion of code that isn't versioned into other projects, and I don't see the benefit of that when u can just have a package that has that code as well. On 7/3/12 2:35 AM, Thierry Carrez thie...@openstack.org wrote: Thierry Carrez wrote: Gabriel Hurley wrote: On a more fundamental level, did I miss some tremendous reason why we have this merge from common pattern instead of making OpenStack Common a standard python dependency just like anything else? Especially with the work Monty has recently done on versioning and packaging the client libs from Jenkins, I can't see a reason to keep following this update common and merge to everything else pattern at all... This discussion should probably wait for markmc to come back, since he set up most of this framework in the first place. He would certainly produce a more compelling rationale than I can :) Actually http://wiki.openstack.org/CommonLibrary explains it quite well. In particular: openstack-common also provides a process for incubating APIs which, while they are shared between multiple OpenStack projects, have not yet matured to meet the [library inclusion] criteria described above. Incubation shouldn't be seen as a long term option for any API - it is merely a stepping stone to inclusion into the openstack-common library proper. -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
- Original Message - From: Russell Bryant rbry...@redhat.com To: andrewbog...@gmail.com Cc: Andrew Bogott abog...@wikimedia.org, openstack@lists.launchpad.net Sent: Monday, July 2, 2012 3:26:56 PM Subject: Re: [Openstack] best practices for merging common into specific projects On 07/02/2012 03:16 PM, Andrew Bogott wrote: Background: The openstack-common project is subject to a standard code-review process (and, soon, will also have Jenkins testing gates.) Sadly, patches that are merged into openstack-common are essentially orphans. Bringing those changes into actual use requires yet another step, a 'merge from common' patch where the code changes in common are copied into a specific project (e.g. nova.) Merge-from-common patches are generated via an automated process. Specific projects express dependencies on specific common components via a config file, e.g. 'nova/openstack-common.conf'. The actual file copy is performed by 'openstack-common/update.py,' and its behavior is governed by the appropriate openstack-common.conf file. More background: http://wiki.openstack.org/CommonLibrary Questions: When should changes from common be merged into other projects? What should a 'merge-from-common' patch look like? What code-review standards should core committers observe when reviewing merge-from-common patches? Proposals: I. As soon as a patch drops into common, the patch author should submit merge-from-common patches to all affected projects. A. (This should really be done by a bot, but that's not going to happen overnight) All of the APIs in openstack-common right now are considered to be in incubation, meaning that breaking changes could be made. I don't think automated merges are good for anything in incubation. Automation would be suitable for stable APIs. Once an API is no longer in incubation, we should be looking at how to make releases and treat it as a proper library. The copy/paste madness should be limited to APIs still in incubation. There are multiple APIs close or at the point where I think we should be able to commit to them. I'll leave the specifics for a separate discussion, but I think moving on this front is key to reducing the pain we are seeing with code copying. II. In the event that I. is not observed, merge-from-common patches will contain bits from multiple precursor patches. That is not only OK, but encouraged. A. Keeping projects in sync with common is important! B. Asking producers of merge-from-common patches to understand the full diff will discourage the generation of such merges. I don't see this as much different as any other patches to nova (or whatever project is using common). It should be a proper patch series. If the person pulling it in doesn't understand the merge well enough to produce the patch series with proper commit messages, then they are the wrong person to be doing the merge in the first place. I went on a bit of a rant about this on IRC yesterday. While I agree a patch series is appropriate for many new features and bug fixes I don't think it should be required for keeping openstack-common in sync. Especially since we don't merge tests from openstack-common which would help verify that the person doing the merges doesn't mess up the order of the patchset. If we were to include the tests from openstack-common in each project this could change my mind. If someone wants to split openstack-common changes into patchsets that might be Okay in small numbers. If you are merging say 5-10 changes from openstack common into all the various openstack projects that could translate into a rather large number of reviews (25+) for things that have been already reviewed once. For me using patchsets to keep openstack-common in sync just causes thrashing of Jenkins, SmokeStack, etc. for things that have already been gated. Seems like an awful waste of review/CI time. In my opinion patchsets are the way to go with getting things into openstack-common... but not when syncing to projects. Hopefully this situation is short lived however and we start using a proper library sooner rather than later. III.Merge-from-common patches should be the product of a single unedited run of update.py. Disagree, see above. A. If a merge-from-common patch breaks pep8 or a test in nova, don't fix the patch; fix the code in common. Agreed. IV.Merge-from-common patches are 'presumed justified'. That means: A. Reviewers of merge-from-common patches should consider test failures and pep8 breakages, and obvious functional problems. B. Reviewers of merge-from-common patches should not consider the usefulness, design, etc. of merge-from-common patches. Such concerns need to be handled within
Re: [Openstack] best practices for merging common into specific projects
The notion that copying code is any protection against APIs that may change is a red herring. It's the exact same effect as pegging a version of a dependency (whether it's a commit hash or a real version number), except now you have code duplication. An unstable upgrade path is an unstable upgrade path, and copying the code into the project doesn't alleviate the pain for the project if the upstream library decides to change its APIs. Also, we're really calling something used by more or less all the core projects incubated? ;-) Seems like it's past the proof-of-concept phase now, at least for many parts of common. Questions of API stability are an issue unto themselves. Anyhow, I'm +1 on turning it into a real library of its own, as a couple people suggested already. - Gabriel -Original Message- From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of James E. Blair Sent: Tuesday, July 03, 2012 6:56 AM To: andrewbog...@gmail.com Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] best practices for merging common into specific projects Andrew Bogott abog...@wikimedia.org writes: I. As soon as a patch drops into common, the patch author should submit merge-from-common patches to all affected projects. A. (This should really be done by a bot, but that's not going to happen overnight) Actually, I think with our current level of tooling, we can have Jenkins do this (run by Zuul as a post-merge job on openstack-common). I very much believe that the long-term goal should be to make openstack- common a library -- so nothing I say here should be construed against that. But as long as it's in an incubation phase, if doing something like this would help move things along, we can certainly implement it, and fairly easily. Note that a naive implementation might generate quite a bit of review spam if several small changes land to openstack-common (there would then be changes*projects number of reviews in gerrit). We have some code laying around which might be useful here that looks for an existing open change and amends it; at least that would let us have at most only one open merge- from-common-change per-project. Okay, that's all on that; I don't want to derail the main conversation, and I'd much rather it just be a library if we're close to being ready for that. -Jim ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
I 150% agree that is a red-herring, that's why I wonder what it really offers besides a 'façade' and/or the feeling that what u are using isn't a package, when in concept it really is, except now u have lost all the benefits of using version numbers, having dependency versions (with history) and all that, so the 'façade' seems pretty weak imho. +1 for library that follows the normal packaging methodology On 7/3/12 11:59 AM, Gabriel Hurley gabriel.hur...@nebula.com wrote: The notion that copying code is any protection against APIs that may change is a red herring. It's the exact same effect as pegging a version of a dependency (whether it's a commit hash or a real version number), except now you have code duplication. An unstable upgrade path is an unstable upgrade path, and copying the code into the project doesn't alleviate the pain for the project if the upstream library decides to change its APIs. Also, we're really calling something used by more or less all the core projects incubated? ;-) Seems like it's past the proof-of-concept phase now, at least for many parts of common. Questions of API stability are an issue unto themselves. Anyhow, I'm +1 on turning it into a real library of its own, as a couple people suggested already. - Gabriel -Original Message- From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of James E. Blair Sent: Tuesday, July 03, 2012 6:56 AM To: andrewbog...@gmail.com Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] best practices for merging common into specific projects Andrew Bogott abog...@wikimedia.org writes: I. As soon as a patch drops into common, the patch author should submit merge-from-common patches to all affected projects. A. (This should really be done by a bot, but that's not going to happen overnight) Actually, I think with our current level of tooling, we can have Jenkins do this (run by Zuul as a post-merge job on openstack-common). I very much believe that the long-term goal should be to make openstack- common a library -- so nothing I say here should be construed against that. But as long as it's in an incubation phase, if doing something like this would help move things along, we can certainly implement it, and fairly easily. Note that a naive implementation might generate quite a bit of review spam if several small changes land to openstack-common (there would then be changes*projects number of reviews in gerrit). We have some code laying around which might be useful here that looks for an existing open change and amends it; at least that would let us have at most only one open merge- from-common-change per-project. Okay, that's all on that; I don't want to derail the main conversation, and I'd much rather it just be a library if we're close to being ready for that. -Jim ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
Dan Prince dpri...@redhat.com writes: If someone wants to split openstack-common changes into patchsets that might be Okay in small numbers. If you are merging say 5-10 changes from openstack common into all the various openstack projects that could translate into a rather large number of reviews (25+) for things that have been already reviewed once. For me using patchsets to keep openstack-common in sync just causes thrashing of Jenkins, SmokeStack, etc. for things that have already been gated. Seems like an awful waste of review/CI time. In my opinion patchsets are the way to go with getting things into openstack-common... but not when syncing to projects. Hopefully this situation is short lived however and we start using a proper library sooner rather than later. Indeed, as a real library, it could be incorporated into the devstack gate tests like the rest of the libraries, which would simultaneously gate changes to it using all of the projects that reference it. -Jim ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
git submodules don't have to be linked to the head of a branch. Instead of double-commiting (for every commit), we can do a single commit in each project to change the git reference of the submodule. This would not be too far from the existing behavior, except that it would minimize the double commits. -- Eric Windisch On Tuesday, July 3, 2012 at 15:47 PM, Andrew Bogott wrote: On 7/3/12 1:59 PM, Gabriel Hurley wrote: The notion that copying code is any protection against APIs that may change is a red herring. It's the exact same effect as pegging a version of a dependency (whether it's a commit hash or a real version number), except now you have code duplication. An unstable upgrade path is an unstable upgrade path, and copying the code into the project doesn't alleviate the pain for the project if the upstream library decides to change its APIs. Also, we're really calling something used by more or less all the core projects incubated? ;-) Seems like it's past the proof-of-concept phase now, at least for many parts of common. Questions of API stability are an issue unto themselves. Anyhow, I'm +1 on turning it into a real library of its own, as a couple people suggested already. - Gabriel I feel like I should speak up since I started this fight in the first place :) Like most people in this thread, I too long for an end to the weird double-commit process that we're using now. So I'm happy to set aside my original Best Practices proposal until there's some consensus regarding how much longer we're going to use that process. Presumably opinions about how to handle merge-from-common commits will vary in the meantime, but that's something we can live with. In terms of promoting common into a real project, though, I want to raise another option that's guaranteed to be unpopular: We make openstack-common a git-submodule that is automatically checked out within the directory tree of each other project. Then each commit to common would need to be gated by the full set of tests on every project that includes common. I haven't thought deeply about the pros and cons of code submodule vs. python project, but I want to bring up the option because it's the system that I'm the most familiar with, and one that's been discussed a bit off and on. -Andrew ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net (mailto:openstack@lists.launchpad.net) Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
On 7/3/12 5:47 PM, Eric Windisch wrote: git submodules don't have to be linked to the head of a branch. Instead of double-commiting (for every commit), we can do a single commit in each project to change the git reference of the submodule. This would not be too far from the existing behavior, except that it would minimize the double commits. Oh, I guess I left out an important part of my vision, which is that there would be a commit hook in common which moves the submodule reference in the parent projects anytime a patch is merged in common. So, in short: once a patch passed review for inclusion in common, that patch would automatically go live in all other project heads simultaneously. -- Eric Windisch On Tuesday, July 3, 2012 at 15:47 PM, Andrew Bogott wrote: On 7/3/12 1:59 PM, Gabriel Hurley wrote: The notion that copying code is any protection against APIs that may change is a red herring. It's the exact same effect as pegging a version of a dependency (whether it's a commit hash or a real version number), except now you have code duplication. An unstable upgrade path is an unstable upgrade path, and copying the code into the project doesn't alleviate the pain for the project if the upstream library decides to change its APIs. Also, we're really calling something used by more or less all the core projects incubated? ;-) Seems like it's past the proof-of-concept phase now, at least for many parts of common. Questions of API stability are an issue unto themselves. Anyhow, I'm +1 on turning it into a real library of its own, as a couple people suggested already. - Gabriel I feel like I should speak up since I started this fight in the first place :) Like most people in this thread, I too long for an end to the weird double-commit process that we're using now. So I'm happy to set aside my original Best Practices proposal until there's some consensus regarding how much longer we're going to use that process. Presumably opinions about how to handle merge-from-common commits will vary in the meantime, but that's something we can live with. In terms of promoting common into a real project, though, I want to raise another option that's guaranteed to be unpopular: We make openstack-common a git-submodule that is automatically checked out within the directory tree of each other project. Then each commit to common would need to be gated by the full set of tests on every project that includes common. I haven't thought deeply about the pros and cons of code submodule vs. python project, but I want to bring up the option because it's the system that I'm the most familiar with, and one that's been discussed a bit off and on. -Andrew ___ Mailing list: https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
I’m pretty -1 on triggering changes in other projects from common. That’s gonna result in a broken code (whether subtle or obvious) no matter how good your gates are. At least as an external library you can freeze a version requirement until such time as you see fit to properly updated that code and *ensure* compatibility in your project. Or if your project likes ridin’ trunk, then don’t pin a version and you’ve got the same effect as an automatic trigger. - Gabriel From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of Andrew Bogott Sent: Tuesday, July 03, 2012 3:54 PM To: Eric Windisch; openstack@lists.launchpad.net Subject: Re: [Openstack] best practices for merging common into specific projects On 7/3/12 5:47 PM, Eric Windisch wrote: git submodules don't have to be linked to the head of a branch. Instead of double-commiting (for every commit), we can do a single commit in each project to change the git reference of the submodule. This would not be too far from the existing behavior, except that it would minimize the double commits. Oh, I guess I left out an important part of my vision, which is that there would be a commit hook in common which moves the submodule reference in the parent projects anytime a patch is merged in common. So, in short: once a patch passed review for inclusion in common, that patch would automatically go live in all other project heads simultaneously. -- Eric Windisch On Tuesday, July 3, 2012 at 15:47 PM, Andrew Bogott wrote: On 7/3/12 1:59 PM, Gabriel Hurley wrote: The notion that copying code is any protection against APIs that may change is a red herring. It's the exact same effect as pegging a version of a dependency (whether it's a commit hash or a real version number), except now you have code duplication. An unstable upgrade path is an unstable upgrade path, and copying the code into the project doesn't alleviate the pain for the project if the upstream library decides to change its APIs. Also, we're really calling something used by more or less all the core projects incubated? ;-) Seems like it's past the proof-of-concept phase now, at least for many parts of common. Questions of API stability are an issue unto themselves. Anyhow, I'm +1 on turning it into a real library of its own, as a couple people suggested already. - Gabriel I feel like I should speak up since I started this fight in the first place :) Like most people in this thread, I too long for an end to the weird double-commit process that we're using now. So I'm happy to set aside my original Best Practices proposal until there's some consensus regarding how much longer we're going to use that process. Presumably opinions about how to handle merge-from-common commits will vary in the meantime, but that's something we can live with. In terms of promoting common into a real project, though, I want to raise another option that's guaranteed to be unpopular: We make openstack-common a git-submodule that is automatically checked out within the directory tree of each other project. Then each commit to common would need to be gated by the full set of tests on every project that includes common. I haven't thought deeply about the pros and cons of code submodule vs. python project, but I want to bring up the option because it's the system that I'm the most familiar with, and one that's been discussed a bit off and on. -Andrew ___ Mailing list: https://launchpad.net/~openstackhttps://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstackhttps://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
I've been following along at home a bit. I can totally see where it's desirable to have well thought out APIs that you can commit to supporting and encourage other people to use. And that you sometimes have expedient code that you aren't as comfortable with. What I don't get is how using a different mechanism to make libraries makes the code any less of a library. Just make it a library using normal packaging methodology, call it version 0.1, and put a README that says we're not real comfortable with this API yet. That accomplishes the same thing, but it's a lot less hairy. Cheers, Tim ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
Interestingly enough - gerrit supports submodules pretty well... and it does exactly what Eric said below ... if both the project and superproject are in gerrit, and a change is made to the project, gerrit can automatically update the superproject reference. Here's the thing though (and one of the reasons we haven't actually earnestly suggested using this for anything yet) ... testing If you propose a change to openstack-common and we were going to use the auto-update feature, we'd need to test that that change doesn't break ALL of the projects that use it. Now, what if nova is going to need a patch to use the new version of the code. Oops. Complexity. Overload. Everyone dies. However, with a versioned library model, the projects can consume things pinned to specific versions, and then they can submit a change that updates the version depend and the code which needs to be updated to support the version change, and that change can be atomic. So honestly, I'd say the real key is getting us closer to the point where openstack-common is a proper library, because all of the rest of the complexity is stuff we're inventing to make life harder on ourselves, when the standard library with api-contract and a version model of the world works pretty fine without needing coordinated changes across multiple repositories. On 07/03/2012 06:54 PM, Gabriel Hurley wrote: I’m pretty -1 on triggering changes in other projects from common. That’s gonna result in a broken code (whether subtle or obvious) no matter how good your gates are. At least as an external library you can freeze a version requirement until such time as you see fit to properly updated that code and **ensure** compatibility in your project. Or if your project likes ridin’ trunk, then don’t pin a version and you’ve got the same effect as an automatic trigger. - Gabriel *From:*openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net] *On Behalf Of *Andrew Bogott *Sent:* Tuesday, July 03, 2012 3:54 PM *To:* Eric Windisch; openstack@lists.launchpad.net *Subject:* Re: [Openstack] best practices for merging common into specific projects On 7/3/12 5:47 PM, Eric Windisch wrote: git submodules don't have to be linked to the head of a branch. Instead of double-commiting (for every commit), we can do a single commit in each project to change the git reference of the submodule. This would not be too far from the existing behavior, except that it would minimize the double commits. Oh, I guess I left out an important part of my vision, which is that there would be a commit hook in common which moves the submodule reference in the parent projects anytime a patch is merged in common. So, in short: once a patch passed review for inclusion in common, that patch would automatically go live in all other project heads simultaneously. -- Eric Windisch On Tuesday, July 3, 2012 at 15:47 PM, Andrew Bogott wrote: On 7/3/12 1:59 PM, Gabriel Hurley wrote: The notion that copying code is any protection against APIs that may change is a red herring. It's the exact same effect as pegging a version of a dependency (whether it's a commit hash or a real version number), except now you have code duplication. An unstable upgrade path is an unstable upgrade path, and copying the code into the project doesn't alleviate the pain for the project if the upstream library decides to change its APIs. Also, we're really calling something used by more or less all the core projects incubated? ;-) Seems like it's past the proof-of-concept phase now, at least for many parts of common. Questions of API stability are an issue unto themselves. Anyhow, I'm +1 on turning it into a real library of its own, as a couple people suggested already. - Gabriel I feel like I should speak up since I started this fight in the first place :) Like most people in this thread, I too long for an end to the weird double-commit process that we're using now. So I'm happy to set aside my original Best Practices proposal until there's some consensus regarding how much longer we're going to use that process. Presumably opinions about how to handle merge-from-common commits will vary in the meantime, but that's something we can live with. In terms of promoting common into a real project, though, I want to raise another option that's guaranteed to be unpopular: We make openstack-common a git-submodule that is automatically checked out within the directory tree of each other project. Then each commit to common would need to be gated by the full set of tests on every
[Openstack] best practices for merging common into specific projects
Having spent some time last week writing code for openstack-common, and having spent yet more time trying to get those changes into Nova, I think it would be useful to define some best practices when crossing the boundary between common and other openstack projects. Background: The openstack-common project is subject to a standard code-review process (and, soon, will also have Jenkins testing gates.) Sadly, patches that are merged into openstack-common are essentially orphans. Bringing those changes into actual use requires yet another step, a 'merge from common' patch where the code changes in common are copied into a specific project (e.g. nova.) Merge-from-common patches are generated via an automated process. Specific projects express dependencies on specific common components via a config file, e.g. 'nova/openstack-common.conf'. The actual file copy is performed by 'openstack-common/update.py,' and its behavior is governed by the appropriate openstack-common.conf file. Questions: When should changes from common be merged into other projects? What should a 'merge-from-common' patch look like? What code-review standards should core committers observe when reviewing merge-from-common patches? Proposals: I. As soon as a patch drops into common, the patch author should submit merge-from-common patches to all affected projects. A. (This should really be done by a bot, but that's not going to happen overnight) II. In the event that I. is not observed, merge-from-common patches will contain bits from multiple precursor patches. That is not only OK, but encouraged. A. Keeping projects in sync with common is important! B. Asking producers of merge-from-common patches to understand the full diff will discourage the generation of such merges. III.Merge-from-common patches should be the product of a single unedited run of update.py. A. If a merge-from-common patch breaks pep8 or a test in nova, don't fix the patch; fix the code in common. IV.Merge-from-common patches are 'presumed justified'. That means: A. Reviewers of merge-from-common patches should consider test failures and pep8 breakages, and obvious functional problems. B. Reviewers of merge-from-common patches should not consider the usefulness, design, etc. of merge-from-common patches. Such concerns need to be handled within the common review process. C. Merges from common should get reviewed early and approved readily in order to avoid project divergence V. Changes to openstack-common.conf are a special case. A. Patches with openstack-common.conf changes should include the relevant merge-from-common changes. B. Such patches should /not/ include any other merge-from-common changes. C. Such patches should not only include the common merges, but should also include whatever code changes make use of the new merge. D. Such patches require the same justification and scrutiny as any other standard project patch. Please discuss! If we're able to reach general agreement about this process, I will document the process in the openstack-common HACKING guidelines. -Andrew On 7/2/12 11:38 AM, Russell Bryant (Code Review) wrote: I did that before seeing this patch. However, I think I prefer what I pushed since it's individual updates explaining what they update instead of a blanket update everything commit. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
On 07/02/2012 03:16 PM, Andrew Bogott wrote: Background: The openstack-common project is subject to a standard code-review process (and, soon, will also have Jenkins testing gates.) Sadly, patches that are merged into openstack-common are essentially orphans. Bringing those changes into actual use requires yet another step, a 'merge from common' patch where the code changes in common are copied into a specific project (e.g. nova.) Merge-from-common patches are generated via an automated process. Specific projects express dependencies on specific common components via a config file, e.g. 'nova/openstack-common.conf'. The actual file copy is performed by 'openstack-common/update.py,' and its behavior is governed by the appropriate openstack-common.conf file. More background: http://wiki.openstack.org/CommonLibrary Questions: When should changes from common be merged into other projects? What should a 'merge-from-common' patch look like? What code-review standards should core committers observe when reviewing merge-from-common patches? Proposals: I. As soon as a patch drops into common, the patch author should submit merge-from-common patches to all affected projects. A. (This should really be done by a bot, but that's not going to happen overnight) All of the APIs in openstack-common right now are considered to be in incubation, meaning that breaking changes could be made. I don't think automated merges are good for anything in incubation. Automation would be suitable for stable APIs. Once an API is no longer in incubation, we should be looking at how to make releases and treat it as a proper library. The copy/paste madness should be limited to APIs still in incubation. There are multiple APIs close or at the point where I think we should be able to commit to them. I'll leave the specifics for a separate discussion, but I think moving on this front is key to reducing the pain we are seeing with code copying. II. In the event that I. is not observed, merge-from-common patches will contain bits from multiple precursor patches. That is not only OK, but encouraged. A. Keeping projects in sync with common is important! B. Asking producers of merge-from-common patches to understand the full diff will discourage the generation of such merges. I don't see this as much different as any other patches to nova (or whatever project is using common). It should be a proper patch series. If the person pulling it in doesn't understand the merge well enough to produce the patch series with proper commit messages, then they are the wrong person to be doing the merge in the first place. III.Merge-from-common patches should be the product of a single unedited run of update.py. Disagree, see above. A. If a merge-from-common patch breaks pep8 or a test in nova, don't fix the patch; fix the code in common. Agreed. IV.Merge-from-common patches are 'presumed justified'. That means: A. Reviewers of merge-from-common patches should consider test failures and pep8 breakages, and obvious functional problems. B. Reviewers of merge-from-common patches should not consider the usefulness, design, etc. of merge-from-common patches. Such concerns need to be handled within the common review process. C. Merges from common should get reviewed early and approved readily in order to avoid project divergence I think core reviewers of projects using openstack-common are justified in doing critical review of new code being used by their project. A core reviewer of nova for examp[le may have a very good reason why the code is not suitable for consumption in nova that openstack-common reviewers didn't think of. I don't think blind approvals are ever a good idea. V. Changes to openstack-common.conf are a special case. A. Patches with openstack-common.conf changes should include the relevant merge-from-common changes. B. Such patches should /not/ include any other merge-from-common changes. C. Such patches should not only include the common merges, but should also include whatever code changes make use of the new merge. D. Such patches require the same justification and scrutiny as any other standard project patch. This all seems fine to me. Please discuss! If we're able to reach general agreement about this process, I will document the process in the openstack-common HACKING guidelines. -- Russell Bryant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
What about using openstack-common as a library instead of a preprocessor 'inclusion' system/copy code around system?? Maybe its time for that to happen? It always seemed sort of silly to me that files are being copied around to different projects like this, instead of referring to code in a common library package. On 7/2/12 12:16 PM, Andrew Bogott abog...@wikimedia.org wrote: Having spent some time last week writing code for openstack-common, and having spent yet more time trying to get those changes into Nova, I think it would be useful to define some best practices when crossing the boundary between common and other openstack projects. Background: The openstack-common project is subject to a standard code-review process (and, soon, will also have Jenkins testing gates.) Sadly, patches that are merged into openstack-common are essentially orphans. Bringing those changes into actual use requires yet another step, a 'merge from common' patch where the code changes in common are copied into a specific project (e.g. nova.) Merge-from-common patches are generated via an automated process. Specific projects express dependencies on specific common components via a config file, e.g. 'nova/openstack-common.conf'. The actual file copy is performed by 'openstack-common/update.py,' and its behavior is governed by the appropriate openstack-common.conf file. Questions: When should changes from common be merged into other projects? What should a 'merge-from-common' patch look like? What code-review standards should core committers observe when reviewing merge-from-common patches? Proposals: I. As soon as a patch drops into common, the patch author should submit merge-from-common patches to all affected projects. A. (This should really be done by a bot, but that's not going to happen overnight) II. In the event that I. is not observed, merge-from-common patches will contain bits from multiple precursor patches. That is not only OK, but encouraged. A. Keeping projects in sync with common is important! B. Asking producers of merge-from-common patches to understand the full diff will discourage the generation of such merges. III.Merge-from-common patches should be the product of a single unedited run of update.py. A. If a merge-from-common patch breaks pep8 or a test in nova, don't fix the patch; fix the code in common. IV.Merge-from-common patches are 'presumed justified'. That means: A. Reviewers of merge-from-common patches should consider test failures and pep8 breakages, and obvious functional problems. B. Reviewers of merge-from-common patches should not consider the usefulness, design, etc. of merge-from-common patches. Such concerns need to be handled within the common review process. C. Merges from common should get reviewed early and approved readily in order to avoid project divergence V. Changes to openstack-common.conf are a special case. A. Patches with openstack-common.conf changes should include the relevant merge-from-common changes. B. Such patches should /not/ include any other merge-from-common changes. C. Such patches should not only include the common merges, but should also include whatever code changes make use of the new merge. D. Such patches require the same justification and scrutiny as any other standard project patch. Please discuss! If we're able to reach general agreement about this process, I will document the process in the openstack-common HACKING guidelines. -Andrew On 7/2/12 11:38 AM, Russell Bryant (Code Review) wrote: I did that before seeing this patch. However, I think I prefer what I pushed since it's individual updates explaining what they update instead of a blanket update everything commit. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
Maybe its time to break out of that incubation?? Seems like if most projects are depending on these config files for code copying that that break out has really already occurred, but it just hasn't been written down or made official? I don't quite understand how a project can be in incubation, but has components that are required for other projects to work? Isn't this the same as having library versions where a project would pull in X.Y.Z openstack-common library version (with the features it wants) while the openstack-common people work on X.Y.Z + 1? Then when X.Y.Z + 1 is approved 'stable' they can move to X.Y.Z+1. On 7/2/12 12:26 PM, Russell Bryant rbry...@redhat.com wrote: On 07/02/2012 03:16 PM, Andrew Bogott wrote: Background: The openstack-common project is subject to a standard code-review process (and, soon, will also have Jenkins testing gates.) Sadly, patches that are merged into openstack-common are essentially orphans. Bringing those changes into actual use requires yet another step, a 'merge from common' patch where the code changes in common are copied into a specific project (e.g. nova.) Merge-from-common patches are generated via an automated process. Specific projects express dependencies on specific common components via a config file, e.g. 'nova/openstack-common.conf'. The actual file copy is performed by 'openstack-common/update.py,' and its behavior is governed by the appropriate openstack-common.conf file. More background: http://wiki.openstack.org/CommonLibrary Questions: When should changes from common be merged into other projects? What should a 'merge-from-common' patch look like? What code-review standards should core committers observe when reviewing merge-from-common patches? Proposals: I. As soon as a patch drops into common, the patch author should submit merge-from-common patches to all affected projects. A. (This should really be done by a bot, but that's not going to happen overnight) All of the APIs in openstack-common right now are considered to be in incubation, meaning that breaking changes could be made. I don't think automated merges are good for anything in incubation. Automation would be suitable for stable APIs. Once an API is no longer in incubation, we should be looking at how to make releases and treat it as a proper library. The copy/paste madness should be limited to APIs still in incubation. There are multiple APIs close or at the point where I think we should be able to commit to them. I'll leave the specifics for a separate discussion, but I think moving on this front is key to reducing the pain we are seeing with code copying. II. In the event that I. is not observed, merge-from-common patches will contain bits from multiple precursor patches. That is not only OK, but encouraged. A. Keeping projects in sync with common is important! B. Asking producers of merge-from-common patches to understand the full diff will discourage the generation of such merges. I don't see this as much different as any other patches to nova (or whatever project is using common). It should be a proper patch series. If the person pulling it in doesn't understand the merge well enough to produce the patch series with proper commit messages, then they are the wrong person to be doing the merge in the first place. III.Merge-from-common patches should be the product of a single unedited run of update.py. Disagree, see above. A. If a merge-from-common patch breaks pep8 or a test in nova, don't fix the patch; fix the code in common. Agreed. IV.Merge-from-common patches are 'presumed justified'. That means: A. Reviewers of merge-from-common patches should consider test failures and pep8 breakages, and obvious functional problems. B. Reviewers of merge-from-common patches should not consider the usefulness, design, etc. of merge-from-common patches. Such concerns need to be handled within the common review process. C. Merges from common should get reviewed early and approved readily in order to avoid project divergence I think core reviewers of projects using openstack-common are justified in doing critical review of new code being used by their project. A core reviewer of nova for examp[le may have a very good reason why the code is not suitable for consumption in nova that openstack-common reviewers didn't think of. I don't think blind approvals are ever a good idea. V. Changes to openstack-common.conf are a special case. A. Patches with openstack-common.conf changes should include the relevant merge-from-common changes. B. Such patches should /not/ include any other merge-from-common changes. C. Such patches should not only include the common merges, but should also include whatever code changes make use of the new merge. D. Such patches require the same justification and scrutiny as any other standard
Re: [Openstack] best practices for merging common into specific projects
On a more fundamental level, did I miss some tremendous reason why we have this merge from common pattern instead of making OpenStack Common a standard python dependency just like anything else? Especially with the work Monty has recently done on versioning and packaging the client libs from Jenkins, I can't see a reason to keep following this update common and merge to everything else pattern at all... - Gabriel -Original Message- From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of Andrew Bogott Sent: Monday, July 02, 2012 12:17 PM To: openstack@lists.launchpad.net Subject: [Openstack] best practices for merging common into specific projects Having spent some time last week writing code for openstack-common, and having spent yet more time trying to get those changes into Nova, I think it would be useful to define some best practices when crossing the boundary between common and other openstack projects. Background: The openstack-common project is subject to a standard code-review process (and, soon, will also have Jenkins testing gates.) Sadly, patches that are merged into openstack-common are essentially orphans. Bringing those changes into actual use requires yet another step, a 'merge from common' patch where the code changes in common are copied into a specific project (e.g. nova.) Merge-from-common patches are generated via an automated process. Specific projects express dependencies on specific common components via a config file, e.g. 'nova/openstack-common.conf'. The actual file copy is performed by 'openstack-common/update.py,' and its behavior is governed by the appropriate openstack-common.conf file. Questions: When should changes from common be merged into other projects? What should a 'merge-from-common' patch look like? What code-review standards should core committers observe when reviewing merge-from-common patches? Proposals: I. As soon as a patch drops into common, the patch author should submit merge-from-common patches to all affected projects. A. (This should really be done by a bot, but that's not going to happen overnight) II. In the event that I. is not observed, merge-from-common patches will contain bits from multiple precursor patches. That is not only OK, but encouraged. A. Keeping projects in sync with common is important! B. Asking producers of merge-from-common patches to understand the full diff will discourage the generation of such merges. III.Merge-from-common patches should be the product of a single unedited run of update.py. A. If a merge-from-common patch breaks pep8 or a test in nova, don't fix the patch; fix the code in common. IV.Merge-from-common patches are 'presumed justified'. That means: A. Reviewers of merge-from-common patches should consider test failures and pep8 breakages, and obvious functional problems. B. Reviewers of merge-from-common patches should not consider the usefulness, design, etc. of merge-from-common patches. Such concerns need to be handled within the common review process. C. Merges from common should get reviewed early and approved readily in order to avoid project divergence V. Changes to openstack-common.conf are a special case. A. Patches with openstack-common.conf changes should include the relevant merge-from-common changes. B. Such patches should /not/ include any other merge-from-common changes. C. Such patches should not only include the common merges, but should also include whatever code changes make use of the new merge. D. Such patches require the same justification and scrutiny as any other standard project patch. Please discuss! If we're able to reach general agreement about this process, I will document the process in the openstack-common HACKING guidelines. -Andrew On 7/2/12 11:38 AM, Russell Bryant (Code Review) wrote: I did that before seeing this patch. However, I think I prefer what I pushed since it's individual updates explaining what they update instead of a blanket update everything commit. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] best practices for merging common into specific projects
I agree, I think moving them into one of the package/dependency managers the Open Stack projects already use is the proper way to do this. The current process of copying the latest seemingly un-versionable code around into the projects really bothered me when it just happened to Horizon for the JSON parser a few weeks ago, I wash;t too keen on approving that review and I'd like to get away from it if possible, even if done by a bot of some kind... John Postlethwait Nebula, Inc. 206-999-4492 On Monday, July 2, 2012 at 2:41 PM, Joshua Harlow wrote: Re: [Openstack] best practices for merging common into specific projects What about using openstack-common as a library instead of a preprocessor ‘inclusion’ system/copy code around system?? Maybe its time for that to happen? It always seemed sort of silly to me that files are being copied around to different projects like this, instead of referring to code in a common library package. On 7/2/12 12:16 PM, Andrew Bogott abog...@wikimedia.org wrote: Having spent some time last week writing code for openstack-common, and having spent yet more time trying to get those changes into Nova, I think it would be useful to define some best practices when crossing the boundary between common and other openstack projects. Background: The openstack-common project is subject to a standard code-review process (and, soon, will also have Jenkins testing gates.) Sadly, patches that are merged into openstack-common are essentially orphans. Bringing those changes into actual use requires yet another step, a 'merge from common' patch where the code changes in common are copied into a specific project (e.g. nova.) Merge-from-common patches are generated via an automated process. Specific projects express dependencies on specific common components via a config file, e.g. 'nova/openstack-common.conf'. The actual file copy is performed by 'openstack-common/update.py,' and its behavior is governed by the appropriate openstack-common.conf file. Questions: When should changes from common be merged into other projects? What should a 'merge-from-common' patch look like? What code-review standards should core committers observe when reviewing merge-from-common patches? Proposals: I. As soon as a patch drops into common, the patch author should submit merge-from-common patches to all affected projects. A. (This should really be done by a bot, but that's not going to happen overnight) II. In the event that I. is not observed, merge-from-common patches will contain bits from multiple precursor patches. That is not only OK, but encouraged. A. Keeping projects in sync with common is important! B. Asking producers of merge-from-common patches to understand the full diff will discourage the generation of such merges. III.Merge-from-common patches should be the product of a single unedited run of update.py. A. If a merge-from-common patch breaks pep8 or a test in nova, don't fix the patch; fix the code in common. IV.Merge-from-common patches are 'presumed justified'. That means: A. Reviewers of merge-from-common patches should consider test failures and pep8 breakages, and obvious functional problems. B. Reviewers of merge-from-common patches should not consider the usefulness, design, etc. of merge-from-common patches. Such concerns need to be handled within the common review process. C. Merges from common should get reviewed early and approved readily in order to avoid project divergence V. Changes to openstack-common.conf are a special case. A. Patches with openstack-common.conf changes should include the relevant merge-from-common changes. B. Such patches should /not/ include any other merge-from-common changes. C. Such patches should not only include the common merges, but should also include whatever code changes make use of the new merge. D. Such patches require the same justification and scrutiny as any other standard project patch. Please discuss! If we're able to reach general agreement about this process, I will document the process in the openstack-common HACKING guidelines. -Andrew On 7/2/12 11:38 AM, Russell Bryant (Code Review) wrote: I did that before seeing this patch. However, I think I prefer what I pushed since it's individual updates explaining what they update instead of a blanket update everything commit. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https
Re: [Openstack] best practices for merging common into specific projects
-openstack-bounces+chrisfer=us.ibm@lists.launchpad.net wrote: - To: andrewbog...@gmail.com andrewbog...@gmail.com, Andrew Bogott abog...@wikimedia.org, openstack openstack@lists.launchpad.net From: Joshua Harlow Sent by: openstack-bounces+chrisfer=us.ibm@lists.launchpad.net Date: 07/02/2012 07:21PM Subject: Re: [Openstack] best practices for merging common into specific projects Re: [Openstack] best practices for merging common into specific projects What about using openstack-common as a library instead of a preprocessor #8216;inclusion#8217; system/copy code around system?? Maybe its time for that to happen? It always seemed sort of silly to me that files are being copied around to different projects like this, instead of referring to code in a common library package. +1 Cheers, Christopher Ferris IBM Distinguished Engineer, CTO Industry and Cloud Standards Member, IBM Academy of Technology IBM Software Group, Standards Strategy email: chris...@us.ibm.com Twitter: christo4ferris phone: +1 508 234 2986 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp