Re: [openstack-dev] [puppet] [fuel] more collaboration request
Zane, Backing the rules with incentives is very important, thanks for bringing it up! This is exactly why I'm trying to be conservative about how much upstream integration overhead Fuel team can commit to: make it too burdensome and it will fizzle out. On the other hand, unintrusive and incremental process changes have a better chance of long-term adoption. Earlier today, Bogdan has updated Fuel wiki with a recommendation to create a Fuel plugin with a vanilla upstream module instead of adding a forked version of the new module into Fuel core: https://wiki.openstack.org/w/index.php?title=Fuel/How_to_contributediff=nextoldid=81244 This, together with the way you're describing handling upstream modules in RDO, got me thinking: this can work (after all, it has no immediate impact on the code that's already in fuel-library), and it will protect us against adding more forked modules to the gap between Fuel and upstream, but what can we do to start actually closing the gap instead of simply stopping it from growing? What I came up with is this. We currently have 3 levels of upstream integration for different modules in fuel-library: 1) Fuel-specific modules that are useless outside of Fuel (e.g. osnailyfacter). 2) Forked modules that have never been synced with upstream since their initial inclusion (e.g. puppet-openstack). 3) Forked modules that have been synced with upstream using our new policy (e.g. puppet-nova), but remain forked. Restricting new modules to plugins introduces the next, fourth level of integration: vanilla upstream modules that exactly match a specified known-good tag or git commit id in the upstream repository. The problem is, if you freeze it to a fixed version, only one side of the gap is fixed: as upstream moves ahead and you stay behind the gap is still growing. To truly call it the next level of integration, we have to come up with a way to keep up with upstream. And at the same time, we have to prevent that from breaking Fuel: having to constantly deal with regressions caused by upstream changes is exactly the kind of incentive that is going to work against the whole idea of upstream integration. And this brings us back to the reason why I said making Fuel CI vote on upstream commits is a pre-requisite for using upstream modules directly. If we can detect changes that would introduce regressions into Fuel early (ideally before they are merged), tracking every upstream commit becomes a reality. And when time lag behind upstream is eliminated altogether, committing directly to upstream becomes less work than maintaining a local fork. The best part is, we don't have to wait for the Fuel pluggable architecture to mature to the level where we can start moving Puppet modules from core to plugins (currently this won't work for some key modules such as keystone, mysql, etc). All we need is to package fuel-library core into a set of binary packages (rpm or deb), one per Puppet module, as Bogdan has proposed. Start with a single git repo used as source package for all such binary packages, with primary fuel-library package depending on module packages. Then, one at a time, extract a module into its own git-buildpackage [0] repo and use gbp-pq [1] to convert Fuel specific patches into a quilt patch queue (tell me if there are better ways to do that for RPMs). [0] http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.html [1] http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/man.gbp.pq.html After that: 1. Fix fuel-library dependency to the first post-split revision of the module (which matches the pre-split code). 2. Set up CI to track upstream master branch and post non-voting Fuel CI test results to upstream gerrit. 3. Commit changes to Fuel core and upstream to get the Fuel test results to pass. 4. Turn on voting for Fuel CI tests in upstream. 5. Relax fuel-library dependency to unversioned, so that it takes the latest available revision of the module. At that point, transition to full upstream integration is completed, for that particular module. What are we missing that's needed to make this work? Aside from splitting fuel-library package, and making Fuel CI work on openstack-infra? -- Dmitry Borodaenko On Fri, Jun 12, 2015 at 05:26:11PM -0400, Zane Bitter wrote: This thread kind of deteriorated a bit (though it looks like it's hopefully recovering), so I'd just like to add some observations. What we have here is a classic case of a long-running fork, with all that that entails. In this case the fork is a public one, but that actually makes very little difference to the fundamentals. (I think it's great that Mirantis have chosen to develop Fuel in the open though! Kudos.) The fact is that maintaining a fork is very expensive. And while it's expensive for the upstream community in terms of lost opportunities for bug fixes, it's far, far more expensive for the maintainers of the downstream fork. IMHO that's one of
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On 06/12/2015 07:58 AM, Bogdan Dobrelya wrote: I'm actually happy to hear from you, since we were discussing together about that over the last 2 summits, without real plan between both groups. I believe as a first steep, the contribution policy to Fuel library should be clear and *prevent new forks of upstream modules* to be accepted in future. This will prevent the technical dept and fork maintain cost to be increased in the future even more. I suggested few changes to the following wiki section [0], see Case B. Adding a new module. Let's please discuss this change as I took initiative and edited the current version just in-place (good for me, there is a history in wiki). The main point of the change is to allow only pulling in changes to existing forks, and prohibit adding of new forks, like [1] or [2] (related revert [3]). There was also a solution suggested for new upstream modules should be added to Fuel as plugins [4] and distributed as packages. Any emergency custom patches may be added as usual patches for packages. Submodules could be also an option. [0] https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Adding_new_puppet_modules_to_fuel-library [1] https://review.openstack.org/#/c/190612/ [2] https://review.openstack.org/#/c/128575/ [3] https://review.openstack.org/#/c/191769/ [4] https://wiki.openstack.org/wiki/Fuel/Plugins Good feedback from the patch's author. Sounds like another plan here, which sounds great. Can you clarify what must be done by upstream manifests? The OpenStack should be deployed from upstream packages with the help of upstream puppet modules instead of forks in Fuel library, we should go a bit further in acceptance criteria, that is that I mean. -- Best regards, Bogdan Dobrelya, Skype #bogdando_at_yahoo.com Irc #bogdando __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On 15.06.2015 13:59, Bogdan Dobrelya wrote: I believe as a first steep, the contribution policy to Fuel library Sorry, the step, it is not so steep. should be clear and *prevent new forks of upstream modules* to be accepted in future. This will prevent the technical dept and fork maintain cost to be increased in the future even more. I suggested few changes to the following wiki section [0], see Case B. Adding a new module. Let's please discuss this change as I took initiative and edited the current version just in-place (good for me, there is a history in wiki). The main point of the change is to allow only pulling in changes to existing forks, and prohibit adding of new forks, like [1] or [2] (related revert [3]). There was also a solution suggested for new upstream modules should be added to Fuel as plugins [4] and distributed as packages. Any emergency custom patches may be added as usual patches for packages. Submodules could be also an option. [0] https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Adding_new_puppet_modules_to_fuel-library [1] https://review.openstack.org/#/c/190612/ [2] https://review.openstack.org/#/c/128575/ [3] https://review.openstack.org/#/c/191769/ [4] https://wiki.openstack.org/wiki/Fuel/Plugins And the second step, as I can see it, is for Fuel build system to be switched on upstream puppet modules plus custom patches - as it was suggested above in this mail-thread. The similar way we do for the fuel-library6.1 package by related bp [0]. Fuel library components should be bundled as packages, like fuel-puppet-nova, fuel-puppet-neutron and so on. These packages should be build from upstream repositories of corresponding release branches. All custom changes in Fuel library should be applied atop of these builds and be maintained (rebase, if build failed) by Fuel dev team, until got contributed upstream and removed from custom patches. Here is related blueprint for this step [1] [0] https://blueprints.launchpad.net/fuel/+spec/package-fuel-components [1] https://blueprints.launchpad.net/fuel/+spec/build-fuel-library-from-upstream The OpenStack should be deployed from upstream packages with the help of upstream puppet modules instead of forks in Fuel library, we should go a bit further in acceptance criteria, that is that I mean. -- Best regards, Bogdan Dobrelya, Skype #bogdando_at_yahoo.com Irc #bogdando __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
Emilien, Thanks for re-raising this. This is a sore subject on the Fuel's side, and I will sulk in my own shame for not being better here and continue to encourage my colleagues to be better in this regard. On Wed, Jun 10, 2015 at 10:45 PM Emilien Macchi emil...@redhat.com wrote: Hi, Before reading this e-mail, please keep in mind: * I have a lot of admiration for Fuel and since I'm working on OpenStack Installers (at eNovance and now Red Hat), Fuel is something I always consider a good product. * This e-mail is about Fuel and Puppet, nothing about Mirantis. * I'm writing on behalf of my thoughts, and not on our group. * I'm using open mailing-list for open discussion. There is not bad spirit in this e-mail and I want to have a productive thread. I have some concerns I would like to share with you and hopefully find some solutions together. Since I've been working on Puppet OpenStack (2 years now), I see some situations that happen - according to me - too often: * A bug is reported in both Fuel Library and the Puppet module having trouble. A patch is provided in Fuel Library (your fork of Puppet OpenStack modules) but not in Puppet upstream module. That means you fix the bug for Fuel, and not for Puppet OpenStack community. It does not happen all the time but quite often. We had started doing this for puppet-openstack and openstack upstreams, there where some projects that pushed back on this so the team was asked to stop cross adding projects. Lets identify if this is desired by the puppet project or otherwise work out a solution. Obviously adding another project to the same bug is the least work for some of us but it may end up with spam from improperly handled bugs, there are easily 20-50 bug modifications a day from the fuel guys so this could become a fire hose if we are not careful. * A patch is submitted in a Puppet module and quite often does not land because there is no activity, no tests or is abandonned later because fixed in Fuel Library. I've noticed the patch is fixed in Fuel Library though. I think this, at least in part, due to the nature in the differences in our CI systems, since we don't yet rely on rspec the patches that can land in fuel-library could lack these and the for encourage some one to only propose to fuel. This is wrong and we need to cover the gap here so the patch has an equal cost to propose so we can eventually show that maintenance is lower in upstream. * RAW copy/paste between upstream modules code and your forks. In term of Licensing, I'm even not sure you have the right to do that (I'm not a CLA expert though) but well... in term of authorship and statistics on code, I'm not sure it's fair. Using submodules with custom patches would have been great to respect the authors who created the original code and you could have personalize the manifests. (Is possible for us to maintain this data through gerrit with out creating more than one patch set?) I don't think anyone on the fuel teams prefers to do this. While it's unfair, its a technical issue that we are more than open to a better solution for. Also, if you want to feel better about it (maybe) it costs us a lot to maintain the forks as it is (Technically and Socially as you noted). When we started fuel, we had initially stared with submodules, and they where a burden to maintain with our tiny team at the time when we had only 50 modules. Now there are 70 in the directory. Some are ours, some our others. With the numerous upstreams, base commits and effective requirement to move fast some times the forks won. Now that we are larger and have more support (people and tools) It would be a great time to step back and evaluate a better solution. For the time being, we have the perceived requirement that we are able patch something quickly if the business need arises. So a solution would require this functionality at least until we have the trust and support to do so directly upstream if necessary. Note: you can see that I don't give any example because I'm not here to blame people or judge anyone. So the goal of my e-mail is to open the discussion and have a *real* collaboration between Fuel team which seems to have a lot of good Puppet engineers and Puppet OpenStack team. We had this kind of discussion at the Summit (in Vancouver and also Paris, and even before). Now I would like to officialy know if you are interested or not to be more involved. I'm also open at any feedback about Puppet OpenStack group and if something blocks you to contribute more. We are very much interested, and have been making progress (even if you cant see it) towards becoming a better member in the community here. * We have opened externally visible CI for nearly all component testing and build processes. Once we get run ci from upstream trunk working you will be able to see everything * There has been work to pull us up to recent revisions of the modules, this is necessary for us
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On 11/06/15 17:36 +0300, Matthew Mosesohn wrote: [..] Secondly, I'd like to point out that Fuel is not so different from what other teams are doing. At the Summit, I heard from others who all maintain internal Gerrits and internal forks of the modules. The difference is that Fuel is being worked on in the open in StackForge. Anyone is free to contribute to Fuel as he or she wishes, take our patches, or review changesets. TBH, I really dislike the fact that there are internal forks but as long as they are kept internal, I don't really care. It's not correct to just copy/paste code, sure, but at least they are not making it publicly consumable with the wrong attributions. I do prefer (and I believe Emiliem does as well) to have Fuel in the open, which is - I'd guess - why this thread was started in the first place. Finding a solution to the issues stated in Emiliem's email would certainly help. None of this is meant as an accusation, I'm just making a point in favor of improving the collaboration between both teams because you're all doing an amazing job. Cheers, Flavio -- @flaper87 Flavio Percoco pgp2tfXMHfQhu.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On Thu, Jun 11, 2015 at 11:43:09PM -0400, Emilien Macchi wrote: What about code history and respect of commit ownership? I'm personally wondering if it's fair to copy/paste several thousands of lines of code from another Open-Source project without asking to the community or notifying the authors before. I know it's Open-Source and Apache 2.0 but well... :-) Being able to copy code without having to ask for permission is exactly what Free Software (and more recently, Open Source) is for. You can't rely on commit history and even changelog to track attribution and licensing, source tree itself should contain all appropriate copyright notices and licenses, and we keep all of those intact: https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/cinder/Modulefile#L3-L4 https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/cinder/LICENSE Besides, there's a historic precedent that stripping commit history is acceptable even with GPL: https://lwn.net/Articles/432012/ Should not it be the way around? Puppet OpenStack modules provide the original code. If there is a bug, it has to be fixed in the modules. Puppet OpenStack developers don't have time/bandwidth and moreover don't want to periodically have a look at Fuel git history. I'm not sure this is the best solution for the community. (...) The reality (and again I won't blame any patch, you can find them on Gerrit) is that most of patches are not merged and in staled status. If I can suggest something, the policy should be more upstream first where you submit a patch upstream, you backport it downstream, and in the until it's merged you should make sure it land upstream after community review process. The last step is I think the problem I'm mentioning here and part of the root cause of this topic. Yes, this right here is the biggest point of contention in the whole discussion. The most problematic implication of what you're asking for is the additional effort that it would require from Fuel developers. When you say that Puppet OpenStack developers don't have time to look at Fuel git history for bugfixes, and then observe that actually Fuel developers do propose their patches to upstream, but those patches are stalled in the community review process, this indicates that you don't consider taking over and landing these patches a priority: We don't consider taking the patches? Why do you misinterpret my words this way here, and then few paragraphs later you demonstrate that you clearly understand the difference between taking patches and taking over patches? Please go on Gerrit, look at the patches and tell me if there is no review from Puppet OpenStack community. Most of the patchs are -1 or not passing unit testing which means the code can't be merged. Let me give you examples so you can see Puppet OpenStack folks is doing reviews on patchs from Fuel team: https://review.openstack.org/#/c/170485/ https://review.openstack.org/#/c/157004/ https://review.openstack.org/#/c/176924/ https://review.openstack.org/#/c/168848/ https://review.openstack.org/#/c/130937/ https://review.openstack.org/#/c/131710/ https://review.openstack.org/#/c/174811/ And this is only 'in progress' patches. A lot of fixed have been abandoned upstream. You can easily query them on Gerrit. Once again, this only disproves your concern that Puppet OpenStack developers would have to waste time digging through Fuel commit history to track down bugfixes. Fuel developers are taking care of this for you. The fact that you have started this thread means that you actually do care to get these bugfixes into Puppet OpenStack. If that's true, maybe you can consider a middleground approach: Fuel team agrees to propose all our changes to upstream (i.e. do a better job at something we've already committed to unilaterally), and you help us land the patches we propose, and take over those that get stalled when the submitter from Fuel team has moved on to other tasks? If I understand correctly, you're asking for Puppet OpenStack group to take over patches that are sent from Fuel team but have negative reviews (not passing unit tests, not compliant with Puppet best practices), just because they have to switch to another task and they can't take time to finish the upstream work? Yes, except for the judgemental parts of your wording: I'm asking for Puppet OpenStack developers *to consider it an option* to take over patches (including those that are sent from Fuel team) blocked by negative reviews (even for objective reasons such as failing unit tests), when the original patch submitter has decided that landing that patch in upstream is no longer as high in their priorities as other activities. I fully understand that this would require more time from Puppet OpenStack developers, but I suspect that it would take less time for them to land such a patch than it would for
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On 06/12/2015 04:31 AM, Dmitry Borodaenko wrote: A better alternative would be to make all upstream Puppet OpenStack directly usable in Fuel, but even if we figure out a way to make that work, it will take a long journey to get there. On the upstream side, Fuel core reviewers would have to also be upstream core reviewers, and Fuel CI would have to be allowed to vote on upstream commits. On Fuel side, we'd have to complete the reconciliation and modularization of all our forked modules, and move all Fuel CI to openstack-infra. The potential benefits for both sides are tremendous, but only after we essentially merge the two projects. Even if that's achievable, is that something whole Puppet OpenStack community is interested in? I'd really love to see this happen. That, plus the fact that we're trying to push for packages in upstream infra, could make it possible to test a Fuel deployment on a commit on a server project upstream. It'd be awesome! Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
Greetings, On 11/06/15 19:31 -0700, Dmitry Borodaenko wrote: On Thu, Jun 11, 2015 at 05:39:28PM -0400, Emilien Macchi wrote: On 06/11/2015 10:36 AM, Matthew Mosesohn wrote: I'm not saying it's the most community-oriented approach, but Fuel would have never evolved and matured without it. The attribution in commits is lost because our directory namespace differs such that it can't just be merged cleanly. Revisiting submodules is an option, but it led to maintenance problems in the past. What kind of problems? It was before I got involved with Fuel, but I can offer a guess. Managing submodules introduces operational overhead and with it more opportunities for failures and mishaps than a single flat git repo. Flat repo makes it more difficult to reconcile with upstream modules, but, when using the process I have described in my previous email to this thread, not as difficult as one could think. We also reconcile an average module with upstream much less frequently (no more than once per Fuel release) than we make commits to that module (many times per release), which also tilts the overhead balance if favor of using a flat repo. Matthew, Dmitry, I'm sure you both, and the Fuel team, are acting on good faith but I believe, in this case, there's no problem that makes copy/pasting code, and therefore loosing commits attribution, acceptable. The fact that you are developing Fuel in the open is awesome and I really hope you never change your mind on that but please, do find a solution for this issue. As you can see from this thread, it creates lots of frustration and it makes other project's work more difficult. [..] The difference is that Fuel is being worked on in the open in StackForge. Anyone is free to contribute to Fuel as he or she wishes, take our patches, or review changesets. Should not it be the way around? Puppet OpenStack modules provide the original code. If there is a bug, it has to be fixed in the modules. Puppet OpenStack developers don't have time/bandwidth and moreover don't want to periodically have a look at Fuel git history. I'm not sure this is the best solution for the community. +1 (...) The reality (and again I won't blame any patch, you can find them on Gerrit) is that most of patches are not merged and in staled status. If I can suggest something, the policy should be more upstream first where you submit a patch upstream, you backport it downstream, and in the until it's merged you should make sure it land upstream after community review process. The last step is I think the problem I'm mentioning here and part of the root cause of this topic. Yes, this right here is the biggest point of contention in the whole discussion. The most problematic implication of what you're asking for is the additional effort that it would require from Fuel developers. When you say that Puppet OpenStack developers don't have time to look at Fuel git history for bugfixes, and then observe that actually Fuel developers do propose their patches to upstream, but those patches are stalled in the community review process, this indicates that you don't consider taking over and landing these patches a priority: http://lifehacker.com/5892948/instead-of-saying-i-dont-have-time-say-its-not-a-priority The fact that you have started this thread means that you actually do care to get these bugfixes into Puppet OpenStack. If that's true, maybe you can consider a middleground approach: Fuel team agrees to propose all our changes to upstream (i.e. do a better job at something we've already committed to unilaterally), and you help us land the patches we propose, and take over those that get stalled when the submitter from Fuel team has moved on to other tasks? Assuming good faith, I'd guess you meant something: Please, help us prioritize patches comming from Fuel. How many members of the Fuel team are also part of Puppet OpenStack ? It'd be awesome if more members of the Fuel team could collaborate with reviews on Puppet OpenStack. That'd certainly increase the team's bandwidth. (I'd guess/hope you guys are already doing it). [..] Flavio -- @flaper87 Flavio Percoco pgpqds8qhOnSP.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On Thu, Jun 11, 2015 at 11:01:19PM -0400, Emilien Macchi wrote: 1) If you are adding a module that is the work of another project and is already tracked in separate repo (...) review should also contain the commit hash from the upstream repo in the commit message. Using this reference to upstream commit, you can cleanly and automatically isolate Fuel specific changes to that module into a patch series with two useful properties: a) You can associate each deviant line with a commit in fuel-library repository that would refer to specific LP bug or blueprint and otherwise explain the change. b) The whole patch series can be cleanly applied on top of the specified commit in upstream git. This means you can automatically graft a branch made out of the patch series onto upstream git, and then use git rebase to make that branch mergeable with the current upstream git head. I spent some time to read Fuel Library, specially its custom manifests [1] and I don't see any example of these commits. I'm pretty sure I've missed them, can you give some example? [1] https://github.com/stackforge/fuel-library/commits/master/deployment/puppet/openstack/manifests/ puppet-openstack is the module that has the most Fuel-specific customizations, so it's probably going to be the last to be synced. I see that by the time you finished writing your email you've found an example of how synced our nova module with puppet-nova, here's a another example of how it's supposed to look: One commit to import a specific revision from upstream: https://review.openstack.org/101166 Second commit to make the module compatible with Fuel: https://review.openstack.org/101471 2) submit patch to upstream module first, and once merged or +2 recieved from a core reviewer, the patch should be backported to Fuel library as well. Aside from the obvious benefit of immediate contribution to upstream, this rule helps to keep Fuel specific changes confined within Fuel-specific modules, and discourages arbitrary deviations of forked modules from upstream. Please let me show you an example of a bug that has been fixed in Fuel, but not in upstream (puppet-nova here), *after* the new Fuel policy (mentioned by Matthew, October 2014). The bug: https://bugs.launchpad.net/fuel/+bug/1378962 Node Libvirt Unique UUID Not Generated On Deployment Patch in Fuel, merged released: https://review.openstack.org/#/c/128640/ Patch in puppet-nova, with negative feedback, without any +2 and staled for October 2014: https://review.openstack.org/#/c/131710 I can see how this example shows a less than ideal experience for everyone involved (including Fuel developers). It does however confirm that we do submit our bugfixes to upstream, and, at least in this particular case, we don't do a fire and forget and actually make an effort to comply with upstream reviewers' comments. It also illustrates the reason why holding a merge into Fuel back until there's +2 from upstream is something we can't yet afford: it took 1 patch set and 5 days for the patch to land in Fuel, and it took 6 patch sets and 62 days for the Fuel developer to give up on trying to land it in upstream. This is an example and again, I don't blame anyone here. I respect people who did the patches and I'm happy it's fixed in Fuel. Thanks for staying positive, greatly appreciated! I understand that even with these rules in place the process isn't perfect, even if it were followed flawlessly. If you have ideas on what else we can tweak to make upstream's life easier, please share, we're always looking for ways to improve our processes. Like puppet-murano? Someone from Fuel team created first the module in Fuel, 6 months ago [1] and 3 months later someone from Fuel team created an empty repository in Stackforge [2]. By the way, Puppet OpenStack community does not have core permissions on this module and it's own by Murano team. [1] https://github.com/stackforge/fuel-library/tree/master/deployment/puppet/murano/ [2] https://review.openstack.org/#/c/155688/ Again another example, where I'm very happy to see the module in Fuel but nothing in Stackforge leveraged by Puppet OpenStack community. Thanks for reporting this! Serg, can you comment? The situation with puppet-murano repository on Stackforge doesn't look right to me. On the lemons to lemonade side: we can leverage the fact that Murano team owns both the murano module in fuel-library and puppet-murano repository on Stackforge, and try to use this module as a proving ground to experiment with different collaboration models. I've been working on OpenStack installers for 2 years now [1] [2], I understand what is a product deadline. [1] http://spinalstack.enovance.com [2] https://www.rdoproject.org/RDO-Manager Can you share more about your experience with these projects? Were/are you using vanilla Puppet modules from Puppet OpenStack and
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On Fri, Jun 12, 2015 at 09:24:33AM +0200, Flavio Percoco wrote: I'm sure you both, and the Fuel team, are acting on good faith but I believe, in this case, there's no problem that makes copy/pasting code, and therefore loosing commits attribution, acceptable. To sum up my previous emails, you're wrong on all accounts: commits and attribution are different things; we're not losing attribution; losing commit history is acceptable. The fact that you are developing Fuel in the open is awesome and I really hope you never change your mind on that but please, do find a solution for this issue. As you can see from this thread, it creates lots of frustration and it makes other project's work more difficult. I have already explained in the thread how we address the problem of tracking down and managing the Fuel specific changes in forked modules. With that problem addressed, I don't see any other objective reason for frustration. Does anybody's bonus depend on the number of lines of code in stackforge repositories such as fuel-library that git blame attributes to their name? The most problematic implication of what you're asking for is the additional effort that it would require from Fuel developers. When you say that Puppet OpenStack developers don't have time to look at Fuel git history for bugfixes, and then observe that actually Fuel developers do propose their patches to upstream, but those patches are stalled in the community review process, this indicates that you don't consider taking over and landing these patches a priority: http://lifehacker.com/5892948/instead-of-saying-i-dont-have-time-say-its-not-a-priority The fact that you have started this thread means that you actually do care to get these bugfixes into Puppet OpenStack. If that's true, maybe you can consider a middleground approach: Fuel team agrees to propose all our changes to upstream (i.e. do a better job at something we've already committed to unilaterally), and you help us land the patches we propose, and take over those that get stalled when the submitter from Fuel team has moved on to other tasks? Assuming good faith, I'd guess you meant something: Please, help us prioritize patches comming from Fuel. Please do not assume that what I actually meant (as I explained in previous reply to Emilien) is incompatible with good faith. I am a strong believer in Free Software, and I want to improve collaboration between Puppet OpenStack and Fuel. I also know that unless we come up with collaboration improvements that are mutually beneficial to both projects, nothing will change and this discussion will remain, as Emilien has put it, just words. -- Dmitry Borodaenko __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On 12/06/15 03:28 -0700, Dmitry Borodaenko wrote: On Fri, Jun 12, 2015 at 09:24:33AM +0200, Flavio Percoco wrote: I'm sure you both, and the Fuel team, are acting on good faith but I believe, in this case, there's no problem that makes copy/pasting code, and therefore loosing commits attribution, acceptable. To sum up my previous emails, you're wrong on all accounts: commits and attribution are different things; we're not losing attribution; losing commit history is acceptable. Do you keep the author of the code you copied? The fact that you are developing Fuel in the open is awesome and I really hope you never change your mind on that but please, do find a solution for this issue. As you can see from this thread, it creates lots of frustration and it makes other project's work more difficult. I have already explained in the thread how we address the problem of tracking down and managing the Fuel specific changes in forked modules. With that problem addressed, I don't see any other objective reason for frustration. Does anybody's bonus depend on the number of lines of code in stackforge repositories such as fuel-library that git blame attributes to their name? I don't think anyone here is talking about bonuses or worrying about salaries. The fact that you mention it offends the purposes of this thread and, as much as you don't care, I'm really sad to read that. The whole thing this thread is trying to achieve is improving collaboration and you are derailing the conversation with completely unfriendly/unhelpful comments like the one above. It does cause frustration because, as you can read from Emiliem's original email, it not just adds some extra burden to people in the puppet team but it also defeates the purposes of the team itself, which is creating OpenStack puppet manifests that are consumable by everyone. Help them be better and let them help you improve your workflow/app. That's all. The most problematic implication of what you're asking for is the additional effort that it would require from Fuel developers. When you say that Puppet OpenStack developers don't have time to look at Fuel git history for bugfixes, and then observe that actually Fuel developers do propose their patches to upstream, but those patches are stalled in the community review process, this indicates that you don't consider taking over and landing these patches a priority: http://lifehacker.com/5892948/instead-of-saying-i-dont-have-time-say-its-not-a-priority The fact that you have started this thread means that you actually do care to get these bugfixes into Puppet OpenStack. If that's true, maybe you can consider a middleground approach: Fuel team agrees to propose all our changes to upstream (i.e. do a better job at something we've already committed to unilaterally), and you help us land the patches we propose, and take over those that get stalled when the submitter from Fuel team has moved on to other tasks? Assuming good faith, I'd guess you meant something: Please, help us prioritize patches comming from Fuel. Please do not assume that what I actually meant (as I explained in previous reply to Emilien) is incompatible with good faith. I am a strong believer in Free Software, and I want to improve collaboration between Puppet OpenStack and Fuel. I also know that unless we come up with collaboration improvements that are mutually beneficial to both projects, nothing will change and this discussion will remain, as Emilien has put it, just words. Please, do come up with something that works for both. It'll be of great benefit for both projects. Just to be clear, I believe in yours and both teams good faith. The reason I mentioned that is because, as a non-native English speaker, I could've read that paragraph in a different way. Just trying to be explicit to avoid others to read it the same way I did. Thanks a lot, Flavio -- @flaper87 Flavio Percoco pgpU9ndjm2aq4.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On Fri, Jun 12, 2015 at 09:31:45AM +0200, Flavio Percoco wrote: On 11/06/15 17:36 +0300, Matthew Mosesohn wrote: Secondly, I'd like to point out that Fuel is not so different from what other teams are doing. At the Summit, I heard from others who all maintain internal Gerrits and internal forks of the modules. The difference is that Fuel is being worked on in the open in StackForge. Anyone is free to contribute to Fuel as he or she wishes, take our patches, or review changesets. TBH, I really dislike the fact that there are internal forks but as long as they are kept internal, I don't really care. Internal may apply to other projects Matt is referring to, but it does not apply to Fuel. Fuel's forks of upstream puppet modules are not internal, they're embedded into the fuel-library repository, which, along with the rest of Fuel source code, is fully public. It's not correct to just copy/paste code, sure, but at least they are not making it publicly consumable with the wrong attributions. We are making Fuel publicly consumable, and, as I've pointed out in previous email, we're keeping all attributions in the source code intact. I do prefer (and I believe Emiliem does as well) to have Fuel in the open, And yet in your previous statements you say that publishing Fuel source code is somehow worse than keeping one's modifications of open source code unavailable to public. Which one is it? -- Dmitry Borodaenko __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On 12/06/15 03:04 -0700, Dmitry Borodaenko wrote: On Fri, Jun 12, 2015 at 09:31:45AM +0200, Flavio Percoco wrote: On 11/06/15 17:36 +0300, Matthew Mosesohn wrote: Secondly, I'd like to point out that Fuel is not so different from what other teams are doing. At the Summit, I heard from others who all maintain internal Gerrits and internal forks of the modules. The difference is that Fuel is being worked on in the open in StackForge. Anyone is free to contribute to Fuel as he or she wishes, take our patches, or review changesets. TBH, I really dislike the fact that there are internal forks but as long as they are kept internal, I don't really care. Internal may apply to other projects Matt is referring to, but it does not apply to Fuel. Fuel's forks of upstream puppet modules are not internal, they're embedded into the fuel-library repository, which, along with the rest of Fuel source code, is fully public. Yup, I was referring to other projects too. I should've been more explicit but thanks for clarifying. It's not correct to just copy/paste code, sure, but at least they are not making it publicly consumable with the wrong attributions. We are making Fuel publicly consumable, and, as I've pointed out in previous email, we're keeping all attributions in the source code intact. I do prefer (and I believe Emiliem does as well) to have Fuel in the open, And yet in your previous statements you say that publishing Fuel source code is somehow worse than keeping one's modifications of open source code unavailable to public. Which one is it? I was referring to other projects :) I like Fuel open, I like every project open but I'd very much want them to do it right. Cheers, Flavio -- @flaper87 Flavio Percoco pgpkkTKmBPIw7.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On Fri, Jun 12, 2015 at 08:33:56AM -0700, James Bottomley wrote: On Fri, 2015-06-12 at 02:43 -0700, Dmitry Borodaenko wrote: On Thu, Jun 11, 2015 at 11:43:09PM -0400, Emilien Macchi wrote: What about code history and respect of commit ownership? I'm personally wondering if it's fair to copy/paste several thousands of lines of code from another Open-Source project without asking to the community or notifying the authors before. I know it's Open-Source and Apache 2.0 but well... :-) Being able to copy code without having to ask for permission is exactly what Free Software (and more recently, Open Source) is for. Copy and Paste forking of code into compatibly licenced code (without asking permission) is legally fine as long as you observe the licence, but it comes with a huge technical penalty: 1. Copy and Paste forks can't share patches: a patch for one has to be manually applied to the other. The amount of manual intervention required grows as the forks move out of sync. 2. Usually one fork gets more attention than the other, so the patch problem of point 1 eventually causes the less attended fork to become unmaintainable (or if not patched, eventually unusable). 3. In the odd chance both forks receive equal attention, you're still expending way over 2x the effort you would have on a single code base. There's no rule of thumb for this: we all paste snippets (pieces of code of up to around 10 lines or so). Sometimes these snippets contain errors and suddenly hundreds of places need fixing. The way around this problem is to share code, either by inclusion, modularity or linking. The reason we paste snippets is because sharing for them is enormous effort. However, as the size of the paste grows, so does the fork penalty and it becomes advantageous to avoid it and the effort of sharing the code looks a lot less problematic. Even in the case where the fork is simply patch the original for bug fixes and some new functionality, the fork penalty rules apply. The data that supports all of this came from Red Hat and SUSE. The end of the 2.4 kernel release cycle for them was a disaster with patch sets larger than the actual kernel itself. Sorting through the resulting rubble is where the upstream first policy actually came from. Thanks for the excellent summary of the technical penalties incurred by straying too far from upstream. It's funny how after years of trying to convince Fuel developers to put more effort into collaboration with upstream, in this thread I managed to come across as if I were arguing the opposite. To reiterate, I understand and support the practical reasons to reduce the gap between Fuel and Puppet OpenStack, and I believe that practical reasons are a much better way to motivate Fuel developers to collaborate than arguing whether what Fuel team has done in the past was fair or wrong. -- Dmitry Borodaenko __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On Fri, Jun 12, 2015 at 02:25:31PM +0200, Flavio Percoco wrote: I have already explained in the thread how we address the problem of tracking down and managing the Fuel specific changes in forked modules. With that problem addressed, I don't see any other objective reason for frustration. Does anybody's bonus depend on the number of lines of code in stackforge repositories such as fuel-library that git blame attributes to their name? I don't think anyone here is talking about bonuses or worrying about salaries. The fact that you mention it offends the purposes of this thread and, as much as you don't care, I'm really sad to read that. The whole thing this thread is trying to achieve is improving collaboration and you are derailing the conversation with completely unfriendly/unhelpful comments like the one above. I am really sorry that I made you feel bad about what I wrote, I didn't mean to do that. I actually completely agree with you that this aspect of the thread was derailing the conversation, and I tried to use reductio ad absurdum to demonstrate how ridiculous it can get if we focus on perfecting author attribution instead of discussing collaboration. I should have been more explicit in indicating that I didn't actually mean this as a serious question. Lets write it off as a bad joke that didn't make it across the language barrier. It does cause frustration because, as you can read from Emiliem's original email, it not just adds some extra burden to people in the puppet team but it also defeates the purposes of the team itself, which is creating OpenStack puppet manifests that are consumable by everyone. Now I see that we're on the same page. I agree that it does add extra burden, and even though we've done what we could to reduce that burden in the process I've described earlier, the only way to eliminate it completely is to use upstream Puppet modules in Fuel directly and without Fuel specific modifications. I see a broad consensus on this thread in favor of setting this as the end goal, and I gladly join that sentiment. To prove that I'm not merely trying to placate you, here's what I had to say about this to the Fuel team back in March 2014 when we first came up with our current process for tracking upsteam: https://lists.launchpad.net/fuel-dev/msg00727.html Peace? -- Dmitry Borodaenko __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On Fri, Jun 12, 2015 at 01:23:28PM -0700, James Bottomley wrote: However, the commit history is vital to obtaining the provenance of the code. If there's ever a question about who authored what part of the code (or worse, who copied it wrongly from a different project, as in the SCO suit against Linux) you need the commit history to establish the chain of conveyance into the code. If we lose this, the protection of the OpenStack CLA and ICLA will be lost as well (along with any patent grants that may have been captured) because they rely on knowing where the code came from. So in legal hygiene and governance terms, you're not free to flush the commit history without setting up the project for provenance problems on down the road. This kind of provenance is currently provided by including sha1 id of the upstream commit from which the module was imported. That gives you enough information to a) confirm that the imported version of the code exactly matches the referenced version in upstream git, and b) use upstream git commit history to further track down origin of any imported line of code. Yes, a hassle, but at least the track is not lost. -- Dmitry Borodaenko __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On Fri, Jun 12, 2015 at 12:14:52PM -0400, Emilien Macchi wrote: On 06/12/2015 11:41 AM, Sergii Golovatiuk wrote: IMO, it's a communication issue and related more to Puppet OpenStack community that to Fuel Library folks. In Fuel Library when patch from external contributor has some problems we cherry-pick it, update a patchset to succeed our expectations. Meanwhile we contact the contributor over IRC or email and explain why we did that. That's very important as contributor may not know all architectural details. He may not know the details of CI tests or details how we test. That's the good attitude to help newcomers. So they will be naturally involved to community. Yes, it takes time for Fuel Library folks, but we continue doing that way as we think that communication with contributor is a key of success. Adding someone by using Gerrit is not enough. Communication on IRC with Puppet OpenStack group would be good on the right channel, like it's done in other OpenStack projects quite often. +1 I have looked over patches in progress. I may be wrong but I didn't find that Puppet OpenStack community updated patch to pass CI. It's not complex to cherry-pick and fix failed tests. It's also not complex to contact person over IRC or in email to explain what needs to be done. Trust me, usually it takes once. Smart creatives are clever enough not to make same mistakes twice. +1, and I also agree with Emilien that Fuel developers should join #puppet-openstack for such discussions, instead of waiting for Puppet OpenStack developers to find them on #fuel-dev. https://bugs.launchpad.net/puppet-openstack/+bugs is for puppet-openstack, which is deprecated in Juno. You should look https://launchpad.net/openstack-puppet-modules which contains mostly triaged bugs. Looks like you need to update the links here: https://wiki.openstack.org/wiki/Puppet It still sends bug reporters to https://launchpad.net/puppet-openstack/ Honestly, if you submit a good patch now, it will land in maximum one week or so. Yes, one week is a timeframe we can work with. If Fuel team could also participate in upstream reviews that would be awesome: * they would be involved in the community * they would get experience from other patches and provide better patches in the future, and get reviews merged faster. Agreed. Even something as small as one review per week would be a good start. Do you have a gerrit review dashboard like the one we use in Fuel: https://wiki.openstack.org/wiki/Fuel#Development_related_links or something else to track reviews backlog? From Fuel side I see that some engineers will be involved to review process. They will participate in weekly meetings. They also be active in communication asking people for help in review or asking why CI failed. Good. I'm wondering if we could set up something like an upstream liaison duty roster, so that there's always a couple of engineers in the Fuel team who make sure that communication with upstream is not falling through the cracks. -- Dmitry Borodaenko __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
This thread kind of deteriorated a bit (though it looks like it's hopefully recovering), so I'd just like to add some observations. What we have here is a classic case of a long-running fork, with all that that entails. In this case the fork is a public one, but that actually makes very little difference to the fundamentals. (I think it's great that Mirantis have chosen to develop Fuel in the open though! Kudos.) The fact is that maintaining a fork is very expensive. And while it's expensive for the upstream community in terms of lost opportunities for bug fixes, it's far, far more expensive for the maintainers of the downstream fork. IMHO that's one of the reasons that permissive licenses like ASL2 have gained so much ground over the GPL - it didn't take very long for almost everyone to realise that there were more compelling reasons to contribute your code upstream than that you were compelled to by the license. I don't think a project like OpenStack could exist if they hadn't. It's simply better business, even if you consider the other upstream users to be competitors. So I think both projects would benefit from more co-operation, but Fuel has by far the most to gain. I see from the thread that a lot of well-intentioned policies have been put in place to try to improve co-operation, and it's actually not that surprising to see them not working that well because the incentives are wrong. When you set up a conflict between incentives and rules... incentives tend to win. (I probably don't need to try to prove this, because it was IMHO one of the great lessons of the communist experiment, and looking at the names in this thread I suspect that most of y'all at least know someone with direct experience of that.) So at the moment committing a patch to Fuel is easy for a Fuel developer, whereas getting that same patch into upstream is hard. So it is much more likely that the downstream patch lands while the upstream patch languishes, despite the hidden cost that another Fuel developer will need to reconcile the two later. To get this to work, you need to make upstream the default (and therefore easiest) path to get changes included in Fuel. Of course you will need a way to make urgent changes to your product without waiting for upstream. As an example, we do this in RDO Manager by maintaining patches on top of an upstream snapshot. (We do actually use Git - in a non-traditional way - as a tool to aid this process, but it's not really the point and there are many ways to tackle the problem.) The snapshot gets updated regularly, so changes that are committed upstream just show up without any extra work. If we need something urgently, we have to option to apply it as a patch, but our enthusiasm to do so is always tempered by the knowledge that if a change that is at least extremely similar does not land upstream then we are creating extra work for ourselves in the very near future. That's how we keep the incentives and policies aligned. (In this way, building a project around a library like this is very similar to building a downstream distribution around an upstream project. We use essentially the same techniques.) And of course once the upstream becomes the default place to land patches, you'll very quickly stop thinking of upstream as 'them' and start thinking of them as 'us'. You'll start assimilating the ideas of what are and are not good coding standards so that you won't have to rework them nearly as much before they can be merged, and once you get involved in the community you'll have the opportunity to influence those ideas as well. Once everyone is up to speed I'm sure you'd see a lot of folks get added to core. Instead of upstream co-operation appearing to consume time that you don't have (which appears to be the problem at the moment), I'm quite sure that same people will be able to get a *lot* more done. Tinkering with the current model by putting in place more policies or trying to offload work to the upstream openstack-puppet team will not work, and more importantly would not realise the same benefits to the Fuel team even if it did work. The problem, of course, is that once you are on a long-running fork it takes a big up-front investment to get off it. (Ask anyone still running an OpenStack Folsom cloud ;) That can be hard to make a case for, especially when you have other priorities and the dividends take some time to appear. I think in this case it would be totally worth the investment, and I hope the Fuel team will consider making that investment. As a bonus, it'll be more polite to the original authors of the code, it'll help everyone who is deploying OpenStack with Puppet (which is most people in the community), and it'll help Fuel users join a bigger critical mass of users so they can get better support from channels like ask.openstack.org. cheers, Zane. On 11/06/15 10:36, Matthew Mosesohn wrote: Hi
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On 06/12/2015 03:33 PM, Dmitry Borodaenko wrote: On Fri, Jun 12, 2015 at 02:25:31PM +0200, Flavio Percoco wrote: I have already explained in the thread how we address the problem of tracking down and managing the Fuel specific changes in forked modules. With that problem addressed, I don't see any other objective reason for frustration. Does anybody's bonus depend on the number of lines of code in stackforge repositories such as fuel-library that git blame attributes to their name? I don't think anyone here is talking about bonuses or worrying about salaries. The fact that you mention it offends the purposes of this thread and, as much as you don't care, I'm really sad to read that. The whole thing this thread is trying to achieve is improving collaboration and you are derailing the conversation with completely unfriendly/unhelpful comments like the one above. I am really sorry that I made you feel bad about what I wrote, I didn't mean to do that. I actually completely agree with you that this aspect of the thread was derailing the conversation, and I tried to use reductio ad absurdum to demonstrate how ridiculous it can get if we focus on perfecting author attribution instead of discussing collaboration. I should have been more explicit in indicating that I didn't actually mean this as a serious question. Lets write it off as a bad joke that didn't make it across the language barrier. It does cause frustration because, as you can read from Emiliem's original email, it not just adds some extra burden to people in the puppet team but it also defeates the purposes of the team itself, which is creating OpenStack puppet manifests that are consumable by everyone. Now I see that we're on the same page. I agree that it does add extra burden, and even though we've done what we could to reduce that burden in the process I've described earlier, the only way to eliminate it completely is to use upstream Puppet modules in Fuel directly and without Fuel specific modifications. I see a broad consensus on this thread in favor of setting this as the end goal, and I gladly join that sentiment. To prove that I'm not merely trying to placate you, here's what I had to say about this to the Fuel team back in March 2014 when we first came up with our current process for tracking upsteam: https://lists.launchpad.net/fuel-dev/msg00727.html Peace? It seems we finally broke the ice and found some agreements here, I'm quite happy. Thanks for your help and your involvement in this topic, it's really appreciated. -- Emilien Macchi signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On Fri, 2015-06-12 at 13:05 -0700, Dmitry Borodaenko wrote: On Fri, Jun 12, 2015 at 08:33:56AM -0700, James Bottomley wrote: On Fri, 2015-06-12 at 02:43 -0700, Dmitry Borodaenko wrote: On Thu, Jun 11, 2015 at 11:43:09PM -0400, Emilien Macchi wrote: What about code history and respect of commit ownership? I'm personally wondering if it's fair to copy/paste several thousands of lines of code from another Open-Source project without asking to the community or notifying the authors before. I know it's Open-Source and Apache 2.0 but well... :-) Being able to copy code without having to ask for permission is exactly what Free Software (and more recently, Open Source) is for. Copy and Paste forking of code into compatibly licenced code (without asking permission) is legally fine as long as you observe the licence, but it comes with a huge technical penalty: 1. Copy and Paste forks can't share patches: a patch for one has to be manually applied to the other. The amount of manual intervention required grows as the forks move out of sync. 2. Usually one fork gets more attention than the other, so the patch problem of point 1 eventually causes the less attended fork to become unmaintainable (or if not patched, eventually unusable). 3. In the odd chance both forks receive equal attention, you're still expending way over 2x the effort you would have on a single code base. There's no rule of thumb for this: we all paste snippets (pieces of code of up to around 10 lines or so). Sometimes these snippets contain errors and suddenly hundreds of places need fixing. The way around this problem is to share code, either by inclusion, modularity or linking. The reason we paste snippets is because sharing for them is enormous effort. However, as the size of the paste grows, so does the fork penalty and it becomes advantageous to avoid it and the effort of sharing the code looks a lot less problematic. Even in the case where the fork is simply patch the original for bug fixes and some new functionality, the fork penalty rules apply. The data that supports all of this came from Red Hat and SUSE. The end of the 2.4 kernel release cycle for them was a disaster with patch sets larger than the actual kernel itself. Sorting through the resulting rubble is where the upstream first policy actually came from. Thanks for the excellent summary of the technical penalties incurred by straying too far from upstream. You're welcome. It's funny how after years of trying to convince Fuel developers to put more effort into collaboration with upstream, in this thread I managed to come across as if I were arguing the opposite. To reiterate, I understand and support the practical reasons to reduce the gap between Fuel and Puppet OpenStack, and I believe that practical reasons are a much better way to motivate Fuel developers to collaborate than arguing whether what Fuel team has done in the past was fair or wrong. I agree; recriminations never solve anything but just to close out on the topic of authorship and commit history, since I think there's been some misunderstandings there as well: The licence is the ultimate arbiter of what you absolutely *have* to do to remain in compliance. The licence governs only the code, not the commit history, so under the licence, you're free to flush all the commit history with no legal consequence from the terms of the licence. However, the commit history is vital to obtaining the provenance of the code. If there's ever a question about who authored what part of the code (or worse, who copied it wrongly from a different project, as in the SCO suit against Linux) you need the commit history to establish the chain of conveyance into the code. If we lose this, the protection of the OpenStack CLA and ICLA will be lost as well (along with any patent grants that may have been captured) because they rely on knowing where the code came from. So in legal hygiene and governance terms, you're not free to flush the commit history without setting up the project for provenance problems on down the road. James __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On 06/12/2015 11:41 AM, Sergii Golovatiuk wrote: Hi, I have read all this thread trying to understand what's going on. It has many emotions but very few logical proposals. Let me try to sum up and make some proposals. * A bug is reported in both Fuel Library and the Puppet module having trouble. A patch is provided in Fuel Library (your fork of Puppet OpenStack modules) but not in Puppet upstream module. That means you fix the bug for Fuel, and not for Puppet OpenStack community. It does not happen all the time but quite often. I agree, Fuel Library had such issues in the past. As Dmitry Borodaenko noted we changed the policy and follow it. Fuel Core reviewers don't accept the patch if it's not submitted to upstream first. Fuel Library does all its best to have less divergence with community. * A patch is submitted in a Puppet module and quite often does not land because there is no activity, no tests or is abandonned later because fixed in Fuel Library. I've noticed the patch is fixed in Fuel Library though. John F. Kennedy said: Ask not what your country can do for you — ask what you can do for your country IMO, it's a communication issue and related more to Puppet OpenStack community that to Fuel Library folks. In Fuel Library when patch from external contributor has some problems we cherry-pick it, update a patchset to succeed our expectations. Meanwhile we contact the contributor over IRC or email and explain why we did that. That's very important as contributor may not know all architectural details. He may not know the details of CI tests or details how we test. That's the good attitude to help newcomers. So they will be naturally involved to community. Yes, it takes time for Fuel Library folks, but we continue doing that way as we think that communication with contributor is a key of success. Adding someone by using Gerrit is not enough. Communication on IRC with Puppet OpenStack group would be good on the right channel, like it's done in other OpenStack projects quite often. I have looked over patches in progress. I may be wrong but I didn't find that Puppet OpenStack community updated patch to pass CI. It's not complex to cherry-pick and fix failed tests. It's also not complex to contact person over IRC or in email to explain what needs to be done. Trust me, usually it takes once. Smart creatives are clever enough not to make same mistakes twice. RAW copy/paste between upstream modules code and your forks. In term of Licensing, I'm even not sure you have the right to do that (I'm not a CLA expert though) but well... in term of authorship and statistics on code, I'm not sure it's fair. Using submodules with custom patches would have been great to respect the authors who created the original code and you could have personalize the manifests. This happens. People fork projects when they have some communication issues or different architectural views. However, I like Compiz and Beryl story. After some time both communities negotiated the issues and merged the projects. Merging projects and make some concession is more respectful in terms of attitude between smart creatives. We as a community don't do a great job watching bugs, so personally I'd prefer that fuel developers just push patches, filing a bug too if you want. (Note: we do need to improve our bug tracking!) Thank you Matt for bringing this up. That's definitely needs improvement. I asked people in Fuel Library in person if they know how to open a bug in Puppet OpenStack community. They also said We usually create a review and add someone's from community to review. Dmitry Borodaenko already pointed to How To contribute guide and policy that everyone in Fuel follows. It helps a lot for our external contributors. Also we do strictly require Closes-Bug or Implements: blueprint in every review. There only few bugs on https://bugs.launchpad.net/puppet-openstack/+bugs Though, there is no assignee or milestones. Some bugs can be invalidated. Some bugs require additional info but nobody asked to provide that info. https://bugs.launchpad.net/puppet-openstack/+bugs is for puppet-openstack, which is deprecated in Juno. You should look https://launchpad.net/openstack-puppet-modules which contains mostly triaged bugs. Velocity of review is also a big problem fur Fuel. Sometimes it takes 3-6 months even when all tests are written and tested. That happened when we worked on Juno where Fuel library submitted many patches before official packages were released. This should be improved also. This should be improved also sorry but we can't do magic here. Puppet OpenStack folks is already working a lot and we do our best to clean the upstream patches backlog. 3-6 months rarely happens and when it does, it's quite often because nobody takes care of the patch (both from
Re: [openstack-dev] [puppet] [fuel] more collaboration request
Hi, Before you read me, please remember I know almost nothing about puppet. :) On 06/11/2015 11:03 PM, Matt Fischer wrote: Matt, I appreciate a lot who you are, and all the help you've given me so far, but what you are asking here is wrong. You shouldn't ask Emilien to track the work of the Fuel team, and ping them on IRC to contribute back. It should be up to them to directly fix upstream *first*, and *then* fix back in Fuel. This is what we should do, indeed, as a Fuel library team. First, always get the patch merged upstream, and only next - backport this to Fuel fork. Ideally, we will next switch to upstream manifests, eventually, so there would be no need for forks anymore. Sadly, this *never* worked for us and doesn't work yet as we're, it seems, not ready for this *quite a long path* of changes landing So there was a lazy compromise and shortcuts found, which I personally don't like. I strongly believe that someday this will start to work for us. And this is not just a words of hope. Before we did the first get-closer-to-upstream effort, our fork's code base diverge was ~97% and 0 patches contributed upstream by changes in Fuel library. An initial sync with upstream modules was the very first step on the right way. And we're keep doing the best to reduce the code diverge to be ready to switch upstream modules one day. It shouldn't be the way either. The team working on fuel-library should be pro-active and doing the contributions, Emilien shouldn't have to Nothing to add, you are completely right. discuss a specific bug of commits. I believe also that Emilien's reasoning goes beyond just one or 2 commits, it's a general thinking. On 06/11/2015 04:36 PM, Matthew Mosesohn wrote: This isn't the only place where we have a huge git repository doing everything. This IMO is a big mistake, which give us more work because we have to duplicate what's upstream and constantly rebase. This is yet another technical dept... This only works because we have a lot of Agree, this makes the technical dept only to grow uncontrollable. Mirantis employee doing the work, so the inefficiency is counter balanced by the work force. But as you know, we're always pushing to the very limit of everyone to release a new version of MOS and Fuel, so maybe now is the time to rethink the way we work. To move forward, I really believe we (as in: Mirantis) should be: 1/ Rework fuel-library to use multiple git for puppet, and maybe work out a way to package these individually. 2/ Using unmodified version of upstream puppet as much as possible 3/ Work *only* on upstream puppet and not on a separate fork I'm all for this option. We have a backlog item to deploy OpenStack from upstream packages with Fuel. I'd say this must be done by upstream puppet manifests as well. As a lot of the changes that I propose, this would be a one-off painful effort to kill this technical dept, but on the long run, we would really benefits from such reorganization. If we don't do the above, it's going to be business as usual, no mater how much efforts Mirantis engineer will put on: the pressure we have to deliver Fuel/MOS should shift from the fork to what's upstream. Cheers, Thomas Goirand (zigo) -- Best regards, Bogdan Dobrelya, Skype #bogdando_at_yahoo.com Irc #bogdando __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
Folks As Dmitry already said, we are open towards merging with upstream and would like to make our code divergence no more than several percents of lines, but there are historical reasons for this divergence with which we are not happy either. So let's just point out that both sides look into the same direction and we just need to figure out workflows and technical details on how to make it comfortable both for Puppet OpenStack and Fuel developers, taking into consideration that we all need to meet our deadlines and goals. Let's close this dicussion with positive results that we both want to merge the codebases as much possible and stop all the blame and judgement game here. Let's do a good friendly meeting and come up with a list of action items for both sides. I do not think that moving towards a quarell is in any way productive. Let's do IRC or at least voice-based meeting and figure out all the details. On Fri, Jun 12, 2015 at 3:29 PM, Flavio Percoco fla...@redhat.com wrote: On 12/06/15 03:04 -0700, Dmitry Borodaenko wrote: On Fri, Jun 12, 2015 at 09:31:45AM +0200, Flavio Percoco wrote: On 11/06/15 17:36 +0300, Matthew Mosesohn wrote: Secondly, I'd like to point out that Fuel is not so different from what other teams are doing. At the Summit, I heard from others who all maintain internal Gerrits and internal forks of the modules. The difference is that Fuel is being worked on in the open in StackForge. Anyone is free to contribute to Fuel as he or she wishes, take our patches, or review changesets. TBH, I really dislike the fact that there are internal forks but as long as they are kept internal, I don't really care. Internal may apply to other projects Matt is referring to, but it does not apply to Fuel. Fuel's forks of upstream puppet modules are not internal, they're embedded into the fuel-library repository, which, along with the rest of Fuel source code, is fully public. Yup, I was referring to other projects too. I should've been more explicit but thanks for clarifying. It's not correct to just copy/paste code, sure, but at least they are not making it publicly consumable with the wrong attributions. We are making Fuel publicly consumable, and, as I've pointed out in previous email, we're keeping all attributions in the source code intact. I do prefer (and I believe Emiliem does as well) to have Fuel in the open, And yet in your previous statements you say that publishing Fuel source code is somehow worse than keeping one's modifications of open source code unavailable to public. Which one is it? I was referring to other projects :) I like Fuel open, I like every project open but I'd very much want them to do it right. Cheers, Flavio -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Yours Faithfully, Vladimir Kuklin, Fuel Library Tech Lead, Mirantis, Inc. +7 (495) 640-49-04 +7 (926) 702-39-68 Skype kuklinvv 35bk3, Vorontsovskaya Str. Moscow, Russia, www.mirantis.com http://www.mirantis.ru/ www.mirantis.ru vkuk...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On 06/12/2015 07:58 AM, Bogdan Dobrelya wrote: Hi, Before you read me, please remember I know almost nothing about puppet. :) On 06/11/2015 11:03 PM, Matt Fischer wrote: Matt, I appreciate a lot who you are, and all the help you've given me so far, but what you are asking here is wrong. You shouldn't ask Emilien to track the work of the Fuel team, and ping them on IRC to contribute back. It should be up to them to directly fix upstream *first*, and *then* fix back in Fuel. This is what we should do, indeed, as a Fuel library team. First, always get the patch merged upstream, and only next - backport this to Fuel fork. Ideally, we will next switch to upstream manifests, eventually, so there would be no need for forks anymore. Sadly, this *never* worked for us and doesn't work yet as we're, it seems, not ready for this *quite a long path* of changes landing So there was a lazy compromise and shortcuts found, which I personally don't like. I strongly believe that someday this will start to work for us. And this is not just a words of hope. Before we did the first get-closer-to-upstream effort, our fork's code base diverge was ~97% and 0 patches contributed upstream by changes in Fuel library. An initial sync with upstream modules was the very first step on the right way. And we're keep doing the best to reduce the code diverge to be ready to switch upstream modules one day. I'm actually happy to hear from you, since we were discussing together about that over the last 2 summits, without real plan between both groups. It shouldn't be the way either. The team working on fuel-library should be pro-active and doing the contributions, Emilien shouldn't have to Nothing to add, you are completely right. discuss a specific bug of commits. I believe also that Emilien's reasoning goes beyond just one or 2 commits, it's a general thinking. On 06/11/2015 04:36 PM, Matthew Mosesohn wrote: This isn't the only place where we have a huge git repository doing everything. This IMO is a big mistake, which give us more work because we have to duplicate what's upstream and constantly rebase. This is yet another technical dept... This only works because we have a lot of Agree, this makes the technical dept only to grow uncontrollable. Good feedback from the patch's author. Mirantis employee doing the work, so the inefficiency is counter balanced by the work force. But as you know, we're always pushing to the very limit of everyone to release a new version of MOS and Fuel, so maybe now is the time to rethink the way we work. To move forward, I really believe we (as in: Mirantis) should be: 1/ Rework fuel-library to use multiple git for puppet, and maybe work out a way to package these individually. 2/ Using unmodified version of upstream puppet as much as possible 3/ Work *only* on upstream puppet and not on a separate fork I'm all for this option. We have a backlog item to deploy Sounds like another plan here, which sounds great. OpenStack from upstream packages with Fuel. I'd say this must be done by upstream puppet manifests as well. Can you clarify what must be done by upstream manifests? As a lot of the changes that I propose, this would be a one-off painful effort to kill this technical dept, but on the long run, we would really benefits from such reorganization. If we don't do the above, it's going to be business as usual, no mater how much efforts Mirantis engineer will put on: the pressure we have to deliver Fuel/MOS should shift from the fork to what's upstream. Cheers, Thomas Goirand (zigo) -- Emilien Macchi signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
-1. You are correct in that it is allowed in free source, and done in practice. There has to be a very good reason for it for it though. It feels like you are using the nuclear option when there are other ways to solve the problem that are more friendly to the attribution of the authors. One of the reasons free software developers do what they do is for recognition, and by this type of copying, that gets lost. Not a good thing. It discourages developers from committing. The way other distro's deal with this is use an upstream release with a set of vendor patches that fix the bugs that haven't made it upstream yet. Its a bit of a hassle to maintain the patches this way, but it does allow the bugs to be fixed relatively quickly if upstream is slow and creates incentive for the patches to get back upstream so they don't have to be carried. The patches should be able to be relatively easily dumped from gerrit as well, so you can submit the patch for review, and then dump it to a patch while its waiting and stick it in the fuel repo until it lands. Once it lands you can simply delete the patch out of fuel's repo. Would this work? Thanks, Kevin From: Dmitry Borodaenko [dborodae...@mirantis.com] Sent: Friday, June 12, 2015 2:43 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [puppet] [fuel] more collaboration request On Thu, Jun 11, 2015 at 11:43:09PM -0400, Emilien Macchi wrote: What about code history and respect of commit ownership? I'm personally wondering if it's fair to copy/paste several thousands of lines of code from another Open-Source project without asking to the community or notifying the authors before. I know it's Open-Source and Apache 2.0 but well... :-) Being able to copy code without having to ask for permission is exactly what Free Software (and more recently, Open Source) is for. You can't rely on commit history and even changelog to track attribution and licensing, source tree itself should contain all appropriate copyright notices and licenses, and we keep all of those intact: https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/cinder/Modulefile#L3-L4 https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/cinder/LICENSE Besides, there's a historic precedent that stripping commit history is acceptable even with GPL: https://lwn.net/Articles/432012/ Should not it be the way around? Puppet OpenStack modules provide the original code. If there is a bug, it has to be fixed in the modules. Puppet OpenStack developers don't have time/bandwidth and moreover don't want to periodically have a look at Fuel git history. I'm not sure this is the best solution for the community. (...) The reality (and again I won't blame any patch, you can find them on Gerrit) is that most of patches are not merged and in staled status. If I can suggest something, the policy should be more upstream first where you submit a patch upstream, you backport it downstream, and in the until it's merged you should make sure it land upstream after community review process. The last step is I think the problem I'm mentioning here and part of the root cause of this topic. Yes, this right here is the biggest point of contention in the whole discussion. The most problematic implication of what you're asking for is the additional effort that it would require from Fuel developers. When you say that Puppet OpenStack developers don't have time to look at Fuel git history for bugfixes, and then observe that actually Fuel developers do propose their patches to upstream, but those patches are stalled in the community review process, this indicates that you don't consider taking over and landing these patches a priority: We don't consider taking the patches? Why do you misinterpret my words this way here, and then few paragraphs later you demonstrate that you clearly understand the difference between taking patches and taking over patches? Please go on Gerrit, look at the patches and tell me if there is no review from Puppet OpenStack community. Most of the patchs are -1 or not passing unit testing which means the code can't be merged. Let me give you examples so you can see Puppet OpenStack folks is doing reviews on patchs from Fuel team: https://review.openstack.org/#/c/170485/ https://review.openstack.org/#/c/157004/ https://review.openstack.org/#/c/176924/ https://review.openstack.org/#/c/168848/ https://review.openstack.org/#/c/130937/ https://review.openstack.org/#/c/131710/ https://review.openstack.org/#/c/174811/ And this is only 'in progress' patches. A lot of fixed have been abandoned upstream. You can easily query them on Gerrit. Once again, this only disproves your concern that Puppet OpenStack developers would have to waste time digging through Fuel commit history to track down bugfixes. Fuel developers are taking care
Re: [openstack-dev] [puppet] [fuel] more collaboration request
Hi, I have read all this thread trying to understand what's going on. It has many emotions but very few logical proposals. Let me try to sum up and make some proposals. * A bug is reported in both Fuel Library and the Puppet module having trouble. A patch is provided in Fuel Library (your fork of Puppet OpenStack modules) but not in Puppet upstream module. That means you fix the bug for Fuel, and not for Puppet OpenStack community. It does not happen all the time but quite often. I agree, Fuel Library had such issues in the past. As Dmitry Borodaenko noted we changed the policy and follow it. Fuel Core reviewers don't accept the patch if it's not submitted to upstream first. Fuel Library does all its best to have less divergence with community. * A patch is submitted in a Puppet module and quite often does not land because there is no activity, no tests or is abandonned later because fixed in Fuel Library. I've noticed the patch is fixed in Fuel Library though. John F. Kennedy said: Ask not what your country can do for you — ask what you can do for your country IMO, it's a communication issue and related more to Puppet OpenStack community that to Fuel Library folks. In Fuel Library when patch from external contributor has some problems we cherry-pick it, update a patchset to succeed our expectations. Meanwhile we contact the contributor over IRC or email and explain why we did that. That's very important as contributor may not know all architectural details. He may not know the details of CI tests or details how we test. That's the good attitude to help newcomers. So they will be naturally involved to community. Yes, it takes time for Fuel Library folks, but we continue doing that way as we think that communication with contributor is a key of success. I have looked over patches in progress. I may be wrong but I didn't find that Puppet OpenStack community updated patch to pass CI. It's not complex to cherry-pick and fix failed tests. It's also not complex to contact person over IRC or in email to explain what needs to be done. Trust me, usually it takes once. Smart creatives are clever enough not to make same mistakes twice. RAW copy/paste between upstream modules code and your forks. In term of Licensing, I'm even not sure you have the right to do that (I'm not a CLA expert though) but well... in term of authorship and statistics on code, I'm not sure it's fair. Using submodules with custom patches would have been great to respect the authors who created the original code and you could have personalize the manifests. This happens. People fork projects when they have some communication issues or different architectural views. However, I like Compiz and Beryl story. After some time both communities negotiated the issues and merged the projects. Merging projects and make some concession is more respectful in terms of attitude between smart creatives. We as a community don't do a great job watching bugs, so personally I'd prefer that fuel developers just push patches, filing a bug too if you want. (Note: we do need to improve our bug tracking!) Thank you Matt for bringing this up. That's definitely needs improvement. I asked people in Fuel Library in person if they know how to open a bug in Puppet OpenStack community. They also said We usually create a review and add someone's from community to review. Dmitry Borodaenko already pointed to How To contribute guide and policy that everyone in Fuel follows. It helps a lot for our external contributors. Also we do strictly require Closes-Bug or Implements: blueprint in every review. There only few bugs on https://bugs.launchpad.net/puppet-openstack/+bugs Though, there is no assignee or milestones. Some bugs can be invalidated. Some bugs require additional info but nobody asked to provide that info. Velocity of review is also a big problem fur Fuel. Sometimes it takes 3-6 months even when all tests are written and tested. That happened when we worked on Juno where Fuel library submitted many patches before official packages were released. This should be improved also. Going back to resolution and action items: We as all smart creatives have own vision. As Emilien mentioned Fuel developers don't participate in weekly meetings or mailing lists. That's true :( Though we participate in summit and follow the best practices in code styling and test coverages. I believe that communication improvement may resolve such issues. If we both start involving people to reviews and talk personally over IRC that will be a bit win for both projects. I believe we'll have synergy so there will be no divergence. Fuel as well as other projects like TripleO will use upstream manifests from master without any modifications. The developers will work on enhancement functionality speeding up the development process. From Fuel side I see that some engineers will be involved to review process. They will participate in weekly meetings. They also be
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On Fri, 2015-06-12 at 02:43 -0700, Dmitry Borodaenko wrote: On Thu, Jun 11, 2015 at 11:43:09PM -0400, Emilien Macchi wrote: What about code history and respect of commit ownership? I'm personally wondering if it's fair to copy/paste several thousands of lines of code from another Open-Source project without asking to the community or notifying the authors before. I know it's Open-Source and Apache 2.0 but well... :-) Being able to copy code without having to ask for permission is exactly what Free Software (and more recently, Open Source) is for. Copy and Paste forking of code into compatibly licenced code (without asking permission) is legally fine as long as you observe the licence, but it comes with a huge technical penalty: 1. Copy and Paste forks can't share patches: a patch for one has to be manually applied to the other. The amount of manual intervention required grows as the forks move out of sync. 2. Usually one fork gets more attention than the other, so the patch problem of point 1 eventually causes the less attended fork to become unmaintainable (or if not patched, eventually unusable). 3. In the odd chance both forks receive equal attention, you're still expending way over 2x the effort you would have on a single code base. There's no rule of thumb for this: we all paste snippets (pieces of code of up to around 10 lines or so). Sometimes these snippets contain errors and suddenly hundreds of places need fixing. The way around this problem is to share code, either by inclusion, modularity or linking. The reason we paste snippets is because sharing for them is enormous effort. However, as the size of the paste grows, so does the fork penalty and it becomes advantageous to avoid it and the effort of sharing the code looks a lot less problematic. Even in the case where the fork is simply patch the original for bug fixes and some new functionality, the fork penalty rules apply. The data that supports all of this came from Red Hat and SUSE. The end of the 2.4 kernel release cycle for them was a disaster with patch sets larger than the actual kernel itself. Sorting through the resulting rubble is where the upstream first policy actually came from. James __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On 06/12/2015 08:29 AM, Flavio Percoco wrote: On 12/06/15 03:04 -0700, Dmitry Borodaenko wrote: On Fri, Jun 12, 2015 at 09:31:45AM +0200, Flavio Percoco wrote: On 11/06/15 17:36 +0300, Matthew Mosesohn wrote: Secondly, I'd like to point out that Fuel is not so different from what other teams are doing. At the Summit, I heard from others who all maintain internal Gerrits and internal forks of the modules. The difference is that Fuel is being worked on in the open in StackForge. Anyone is free to contribute to Fuel as he or she wishes, take our patches, or review changesets. TBH, I really dislike the fact that there are internal forks but as long as they are kept internal, I don't really care. Internal may apply to other projects Matt is referring to, but it does not apply to Fuel. Fuel's forks of upstream puppet modules are not internal, they're embedded into the fuel-library repository, which, along with the rest of Fuel source code, is fully public. Yup, I was referring to other projects too. I should've been more explicit but thanks for clarifying. It's not correct to just copy/paste code, sure, but at least they are not making it publicly consumable with the wrong attributions. We are making Fuel publicly consumable, and, as I've pointed out in previous email, we're keeping all attributions in the source code intact. I do prefer (and I believe Emiliem does as well) to have Fuel in the open, And yet in your previous statements you say that publishing Fuel source code is somehow worse than keeping one's modifications of open source code unavailable to public. Which one is it? I was referring to other projects :) I like Fuel open, I like every project open but I'd very much want them to do it right. +1 Open-Source is not just about publishing your source code on github I'm sure we all know this statement, but in practice this is not always happening. Cheers, Flavio __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Emilien Macchi signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On 06/12/2015 05:43 AM, Dmitry Borodaenko wrote: On Thu, Jun 11, 2015 at 11:43:09PM -0400, Emilien Macchi wrote: What about code history and respect of commit ownership? I'm personally wondering if it's fair to copy/paste several thousands of lines of code from another Open-Source project without asking to the community or notifying the authors before. I know it's Open-Source and Apache 2.0 but well... :-) Being able to copy code without having to ask for permission is exactly what Free Software (and more recently, Open Source) is for. You can't rely on commit history and even changelog to track attribution and licensing, source tree itself should contain all appropriate copyright notices and licenses, and we keep all of those intact: https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/cinder/Modulefile#L3-L4 https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/cinder/LICENSE Besides, there's a historic precedent that stripping commit history is acceptable even with GPL: https://lwn.net/Articles/432012/ Oh I'm pretty sure you don't violate any law. I'm just not sure it was the right way to use the Puppet modules. I don't think there is value to discuss more about this 'copy/paste' thing, let's move on to interesting stuffs. Should not it be the way around? Puppet OpenStack modules provide the original code. If there is a bug, it has to be fixed in the modules. Puppet OpenStack developers don't have time/bandwidth and moreover don't want to periodically have a look at Fuel git history. I'm not sure this is the best solution for the community. (...) The reality (and again I won't blame any patch, you can find them on Gerrit) is that most of patches are not merged and in staled status. If I can suggest something, the policy should be more upstream first where you submit a patch upstream, you backport it downstream, and in the until it's merged you should make sure it land upstream after community review process. The last step is I think the problem I'm mentioning here and part of the root cause of this topic. Yes, this right here is the biggest point of contention in the whole discussion. The most problematic implication of what you're asking for is the additional effort that it would require from Fuel developers. When you say that Puppet OpenStack developers don't have time to look at Fuel git history for bugfixes, and then observe that actually Fuel developers do propose their patches to upstream, but those patches are stalled in the community review process, this indicates that you don't consider taking over and landing these patches a priority: We don't consider taking the patches? Why do you misinterpret my words this way here, and then few paragraphs later you demonstrate that you clearly understand the difference between taking patches and taking over patches? Please go on Gerrit, look at the patches and tell me if there is no review from Puppet OpenStack community. Most of the patchs are -1 or not passing unit testing which means the code can't be merged. Let me give you examples so you can see Puppet OpenStack folks is doing reviews on patchs from Fuel team: https://review.openstack.org/#/c/170485/ https://review.openstack.org/#/c/157004/ https://review.openstack.org/#/c/176924/ https://review.openstack.org/#/c/168848/ https://review.openstack.org/#/c/130937/ https://review.openstack.org/#/c/131710/ https://review.openstack.org/#/c/174811/ And this is only 'in progress' patches. A lot of fixed have been abandoned upstream. You can easily query them on Gerrit. Once again, this only disproves your concern that Puppet OpenStack developers would have to waste time digging through Fuel commit history to track down bugfixes. Fuel developers are taking care of this for you. Good to know so we have a first action: Fuel developers to track down every bug in Puppet modules that are fixed in Fuel, at least by submitting a patch upstream and making sure the review stays alive. If the developer can't spend more time on a pending patch, the developer can gently ask on the ML or IRC if someone from Puppet OpenStack group can takeover to finish the code, and use Co-Authored-By. The fact that you have started this thread means that you actually do care to get these bugfixes into Puppet OpenStack. If that's true, maybe you can consider a middleground approach: Fuel team agrees to propose all our changes to upstream (i.e. do a better job at something we've already committed to unilaterally), and you help us land the patches we propose, and take over those that get stalled when the submitter from Fuel team has moved on to other tasks? If I understand correctly, you're asking for Puppet OpenStack group to take over patches that are sent from Fuel team but have negative reviews (not passing unit tests, not compliant with Puppet best practices), just because they have to switch to
Re: [openstack-dev] [puppet] [fuel] more collaboration request
+1 for the thread, I would also like to hear from Mirantis on this. The Fork on fuel/puppet has been actively seen patching and consolidation.It seems like parallel effort why not merge it. regards /sanjay On Thu, Jun 11, 2015 at 9:12 AM, Emilien Macchi emil...@redhat.com wrote: Hi, Before reading this e-mail, please keep in mind: * I have a lot of admiration for Fuel and since I'm working on OpenStack Installers (at eNovance and now Red Hat), Fuel is something I always consider a good product. * This e-mail is about Fuel and Puppet, nothing about Mirantis. * I'm writing on behalf of my thoughts, and not on our group. * I'm using open mailing-list for open discussion. There is not bad spirit in this e-mail and I want to have a productive thread. I have some concerns I would like to share with you and hopefully find some solutions together. Since I've been working on Puppet OpenStack (2 years now), I see some situations that happen - according to me - too often: * A bug is reported in both Fuel Library and the Puppet module having trouble. A patch is provided in Fuel Library (your fork of Puppet OpenStack modules) but not in Puppet upstream module. That means you fix the bug for Fuel, and not for Puppet OpenStack community. It does not happen all the time but quite often. * A patch is submitted in a Puppet module and quite often does not land because there is no activity, no tests or is abandonned later because fixed in Fuel Library. I've noticed the patch is fixed in Fuel Library though. * RAW copy/paste between upstream modules code and your forks. In term of Licensing, I'm even not sure you have the right to do that (I'm not a CLA expert though) but well... in term of authorship and statistics on code, I'm not sure it's fair. Using submodules with custom patches would have been great to respect the authors who created the original code and you could have personalize the manifests. Note: you can see that I don't give any example because I'm not here to blame people or judge anyone. So the goal of my e-mail is to open the discussion and have a *real* collaboration between Fuel team which seems to have a lot of good Puppet engineers and Puppet OpenStack team. We had this kind of discussion at the Summit (in Vancouver and also Paris, and even before). Now I would like to officialy know if you are interested or not to be more involved. I'm also open at any feedback about Puppet OpenStack group and if something blocks you to contribute more. We have the same goals, having Puppet modules better. I think it can be win/win: you have less diff with upstream and we have more hands in our module maintenance. Thank you for reading so far, and I'm looking forward to reading from you. Best regards, -- Emilien Macchi __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sanjay Upadhyay http://saneax.blogspot.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
Hi Emilien, I can see why you might be unhappy with Fuel's actions with regards to the OpenStack Puppet modules. You could make this argument about many components in Fuel. The heart of the matter is that we bundle the upstream OpenStack Puppet modules with all the other modules, developed both upstream and by Fuel's developers in one single git repository. This decision, along with all the other decisions to put Fuel's components under its own repositories, was intended to add stability and granular control to the product. I'm not saying it's the most community-oriented approach, but Fuel would have never evolved and matured without it. The attribution in commits is lost because our directory namespace differs such that it can't just be merged cleanly. Revisiting submodules is an option, but it led to maintenance problems in the past. Secondly, I'd like to point out that Fuel is not so different from what other teams are doing. At the Summit, I heard from others who all maintain internal Gerrits and internal forks of the modules. The difference is that Fuel is being worked on in the open in StackForge. Anyone is free to contribute to Fuel as he or she wishes, take our patches, or review changesets. Starting in October 2014, the Fuel team has adopted a policy that we cannot merge any patches into the core Puppet OpenStack modules of Fuel without submitting a patch or at least a bug upstream. Our reviewers block patches consistently. The truth is that the upstream modules are quite excellent and we don't need to make changes so often. Our goal is to work with upstream modules or in the issue where upstream integration is impossible, we place that config in our own separate modules. The point you raised about fixing bugs in Fuel and not in Puppet OpenStack is definitely valid and something we need to collaborate on. The first and easiest option when a bug is applicable to both, we could use Launchpad to assign bugs to both Fuel project and puppet-$project so it gains visibility. If a bug is discovered in Puppet OpenStack after it's been reported and/or fixed in Fuel, then it would be best to ping someone in #fuel-dev on IRC and we can try to figure out how to get this applied upstream correctly. Our best results come from fixing upstream and I want to make sure that is clear. If you have specific bugs or commits you'd like to discuss, let's work them out. I believe I can get the Fuel Library team to agree to do a walk through all the open bugs in Puppet OpenStack and see if we have any related fixes or bug reports. Best Regards, Matthew Mosesohn On Thu, Jun 11, 2015 at 2:34 PM, Sanjay Upadhyay san...@gmail.com wrote: +1 for the thread, I would also like to hear from Mirantis on this. The Fork on fuel/puppet has been actively seen patching and consolidation.It seems like parallel effort why not merge it. regards /sanjay On Thu, Jun 11, 2015 at 9:12 AM, Emilien Macchi emil...@redhat.com wrote: Hi, Before reading this e-mail, please keep in mind: * I have a lot of admiration for Fuel and since I'm working on OpenStack Installers (at eNovance and now Red Hat), Fuel is something I always consider a good product. * This e-mail is about Fuel and Puppet, nothing about Mirantis. * I'm writing on behalf of my thoughts, and not on our group. * I'm using open mailing-list for open discussion. There is not bad spirit in this e-mail and I want to have a productive thread. I have some concerns I would like to share with you and hopefully find some solutions together. Since I've been working on Puppet OpenStack (2 years now), I see some situations that happen - according to me - too often: * A bug is reported in both Fuel Library and the Puppet module having trouble. A patch is provided in Fuel Library (your fork of Puppet OpenStack modules) but not in Puppet upstream module. That means you fix the bug for Fuel, and not for Puppet OpenStack community. It does not happen all the time but quite often. * A patch is submitted in a Puppet module and quite often does not land because there is no activity, no tests or is abandonned later because fixed in Fuel Library. I've noticed the patch is fixed in Fuel Library though. * RAW copy/paste between upstream modules code and your forks. In term of Licensing, I'm even not sure you have the right to do that (I'm not a CLA expert though) but well... in term of authorship and statistics on code, I'm not sure it's fair. Using submodules with custom patches would have been great to respect the authors who created the original code and you could have personalize the manifests. Note: you can see that I don't give any example because I'm not here to blame people or judge anyone. So the goal of my e-mail is to open the discussion and have a *real* collaboration between Fuel team which seems to have a lot of good Puppet engineers and Puppet OpenStack team. We had this kind of discussion at the Summit (in Vancouver and also Paris,
Re: [openstack-dev] [puppet] [fuel] more collaboration request
Hi Matthew, Thanks for your reply, please see inline: On 06/11/2015 10:36 AM, Matthew Mosesohn wrote: Hi Emilien, I can see why you might be unhappy with Fuel's actions with regards to the OpenStack Puppet modules. You could make this argument about many components in Fuel. The heart of the matter is that we bundle the upstream OpenStack Puppet modules with all the other modules, developed both upstream and by Fuel's developers in one single git repository. This decision, along with all the other decisions to put Fuel's components under its own repositories, was intended to add stability and granular control to the product. I'm not saying it's the most community-oriented approach, but Fuel would have never evolved and matured without it. The attribution in commits is lost because our directory namespace differs such that it can't just be merged cleanly. Revisiting submodules is an option, but it led to maintenance problems in the past. What kind of problems? You also could have used forks of modules, applied your patches and done rebase from time to time when you like. Using a Puppetfile in your installer and you're all set. The maintenance problems thing does not sound right to me here, I think it's more expensive to maintain files than git repositories. Secondly, I'd like to point out that Fuel is not so different from what other teams are doing. At the Summit, I heard from others who all maintain internal Gerrits and internal forks of the modules. True, and most of people are using forks which is totally fine in term of authorship respect. The difference is that Fuel is being worked on in the open in StackForge. Anyone is free to contribute to Fuel as he or she wishes, take our patches, or review changesets. Should not it be the way around? Puppet OpenStack modules provide the original code. If there is a bug, it has to be fixed in the modules. Puppet OpenStack developers don't have time/bandwidth and moreover don't want to periodically have a look at Fuel git history. I'm not sure this is the best solution for the community. Starting in October 2014, the Fuel team has adopted a policy that we cannot merge any patches into the core Puppet OpenStack modules of Fuel without submitting a patch or at least a bug upstream. Our reviewers block patches consistently. The truth is that the upstream modules are quite excellent and we don't need to make changes so often. Our goal is to work with upstream modules or in the issue where upstream integration is impossible, we place that config in our own separate modules. I'm sure you do respect this policy. The reality (and again I won't blame any patch, you can find them on Gerrit) is that most of patches are not merged and in staled status. If I can suggest something, the policy should be more upstream first where you submit a patch upstream, you backport it downstream, and in the until it's merged you should make sure it land upstream after community review process. The last step is I think the problem I'm mentioning here and part of the root cause of this topic. The point you raised about fixing bugs in Fuel and not in Puppet OpenStack is definitely valid and something we need to collaborate on. The first and easiest option when a bug is applicable to both, we could use Launchpad to assign bugs to both Fuel project and puppet-$project so it gains visibility. If a bug is discovered in Puppet OpenStack after it's been reported and/or fixed in Fuel, then it would be best to ping someone in #fuel-dev on IRC and we can try to figure out how to get this applied upstream correctly. Our best results come from fixing upstream and I want to make sure that is clear. Again, I'm not sure this is the right direction. The official channel for Puppet OpenStack modules is #puppet-openstack and this is the place to be when you're involved in the Puppet OpenStack community. I would suggest to rewrite this thing in if a bug is discovered in Puppet OpenStack after it's been reported or fixed in Fuel, then folks (from both groups) should collaborate on Puppet OpenStack official channel to fix it upstream. IMHO, Fuel IRC channel should relate to Fuel specific things. As a example, RDO has #rdo-puppet talking about Puppet OpenStack downstream (packstack, etc) things and use #puppet-openstack for upstream related bugs. I think this is the way to go if we want to improve our collaboration. If you have specific bugs or commits you'd like to discuss, let's work them out. I believe I can get the Fuel Library team to agree to do a walk through all the open bugs in Puppet OpenStack and see if we have any related fixes or bug reports. Like I already said, I won't share any patch/bug because I don't want to blame anyone here, this is not the goal. Going thru Launchpad and Gerrit, you'll easily see what I mean. At this stage, it's pretty clear we still need more discussion. Best Regards, Matthew Mosesohn On Thu, Jun 11, 2015 at
Re: [openstack-dev] [puppet] [fuel] more collaboration request
We as a community don't do a great job watching bugs, so personally I'd prefer that fuel developers just push patches, filing a bug too if you want. (Note: we do need to improve our bug tracking!) However, I don't think that asking puppet openstack devs to ask in the fuel channel if a given bug is fixed in fuel is a very sustainable model. I'm not sure of many projects where the upstream polls downstream to ask whether they've fixed bugs. Can we come up with a way to collaborate more that everyone likes? I think there's a lot of value in it for everyone, we all get a better product out of it. On Thu, Jun 11, 2015 at 8:36 AM, Matthew Mosesohn mmoses...@mirantis.com wrote: Hi Emilien, I can see why you might be unhappy with Fuel's actions with regards to the OpenStack Puppet modules. You could make this argument about many components in Fuel. The heart of the matter is that we bundle the upstream OpenStack Puppet modules with all the other modules, developed both upstream and by Fuel's developers in one single git repository. This decision, along with all the other decisions to put Fuel's components under its own repositories, was intended to add stability and granular control to the product. I'm not saying it's the most community-oriented approach, but Fuel would have never evolved and matured without it. The attribution in commits is lost because our directory namespace differs such that it can't just be merged cleanly. Revisiting submodules is an option, but it led to maintenance problems in the past. Secondly, I'd like to point out that Fuel is not so different from what other teams are doing. At the Summit, I heard from others who all maintain internal Gerrits and internal forks of the modules. The difference is that Fuel is being worked on in the open in StackForge. Anyone is free to contribute to Fuel as he or she wishes, take our patches, or review changesets. Starting in October 2014, the Fuel team has adopted a policy that we cannot merge any patches into the core Puppet OpenStack modules of Fuel without submitting a patch or at least a bug upstream. Our reviewers block patches consistently. The truth is that the upstream modules are quite excellent and we don't need to make changes so often. Our goal is to work with upstream modules or in the issue where upstream integration is impossible, we place that config in our own separate modules. The point you raised about fixing bugs in Fuel and not in Puppet OpenStack is definitely valid and something we need to collaborate on. The first and easiest option when a bug is applicable to both, we could use Launchpad to assign bugs to both Fuel project and puppet-$project so it gains visibility. If a bug is discovered in Puppet OpenStack after it's been reported and/or fixed in Fuel, then it would be best to ping someone in #fuel-dev on IRC and we can try to figure out how to get this applied upstream correctly. Our best results come from fixing upstream and I want to make sure that is clear. If you have specific bugs or commits you'd like to discuss, let's work them out. I believe I can get the Fuel Library team to agree to do a walk through all the open bugs in Puppet OpenStack and see if we have any related fixes or bug reports. Best Regards, Matthew Mosesohn On Thu, Jun 11, 2015 at 2:34 PM, Sanjay Upadhyay san...@gmail.com wrote: +1 for the thread, I would also like to hear from Mirantis on this. The Fork on fuel/puppet has been actively seen patching and consolidation.It seems like parallel effort why not merge it. regards /sanjay On Thu, Jun 11, 2015 at 9:12 AM, Emilien Macchi emil...@redhat.com wrote: Hi, Before reading this e-mail, please keep in mind: * I have a lot of admiration for Fuel and since I'm working on OpenStack Installers (at eNovance and now Red Hat), Fuel is something I always consider a good product. * This e-mail is about Fuel and Puppet, nothing about Mirantis. * I'm writing on behalf of my thoughts, and not on our group. * I'm using open mailing-list for open discussion. There is not bad spirit in this e-mail and I want to have a productive thread. I have some concerns I would like to share with you and hopefully find some solutions together. Since I've been working on Puppet OpenStack (2 years now), I see some situations that happen - according to me - too often: * A bug is reported in both Fuel Library and the Puppet module having trouble. A patch is provided in Fuel Library (your fork of Puppet OpenStack modules) but not in Puppet upstream module. That means you fix the bug for Fuel, and not for Puppet OpenStack community. It does not happen all the time but quite often. * A patch is submitted in a Puppet module and quite often does not land because there is no activity, no tests or is abandonned later because fixed in Fuel Library. I've noticed the patch is fixed in Fuel Library though. *
Re: [openstack-dev] [puppet] [fuel] more collaboration request
Hey Matt other OpenStack folks, I agree that pinging #fuel-dev would not be scalable for every request. But I think it was more of if someone notices something is fixed in fuel-library but not in an OpenStack puppet library, then by all means come and poke someone so we can try and get it in to upstream if we failed to do so. As for the inclusion of the OpenStack puppet modules within fuel-library, I also am not a fan of the way it is done today, but I'm new here so I can't speak for previous decisions. It would be nice if it followed a similar model to how the OpenStack python libraries are consumed. Leveraging something like Librarian where we can just increment version numbers or swap out repositories would be a nice option. That being said, there's a lot more tooling that is required in order to get to that and it's hard to prioritize things like that. Puppet modules aren't traditionally packaged via rpms/debs/etc so it's kinda of hard to do that dependency checking like we do for all the OpenStack services. I, personally and I'm sure there are others, would welcome any assistance or ideas on the best way to do something like that. I think this is where some community help would be much appreciated so that we can get to that point. I'd be happy to help with trying to push some of these ideas forward. I think there are probably others who detest having to manually pull in module updates every few months and battle test and merge issues, so anything to make everyones' lives easier would be welcomed. Thanks, -Alex Schultz On Thu, Jun 11, 2015 at 4:03 PM, Matt Fischer m...@mattfischer.com wrote: We as a community don't do a great job watching bugs, so personally I'd prefer that fuel developers just push patches, filing a bug too if you want. (Note: we do need to improve our bug tracking!) However, I don't think that asking puppet openstack devs to ask in the fuel channel if a given bug is fixed in fuel is a very sustainable model. I'm not sure of many projects where the upstream polls downstream to ask whether they've fixed bugs. Can we come up with a way to collaborate more that everyone likes? I think there's a lot of value in it for everyone, we all get a better product out of it. On Thu, Jun 11, 2015 at 8:36 AM, Matthew Mosesohn mmoses...@mirantis.com wrote: Hi Emilien, I can see why you might be unhappy with Fuel's actions with regards to the OpenStack Puppet modules. You could make this argument about many components in Fuel. The heart of the matter is that we bundle the upstream OpenStack Puppet modules with all the other modules, developed both upstream and by Fuel's developers in one single git repository. This decision, along with all the other decisions to put Fuel's components under its own repositories, was intended to add stability and granular control to the product. I'm not saying it's the most community-oriented approach, but Fuel would have never evolved and matured without it. The attribution in commits is lost because our directory namespace differs such that it can't just be merged cleanly. Revisiting submodules is an option, but it led to maintenance problems in the past. Secondly, I'd like to point out that Fuel is not so different from what other teams are doing. At the Summit, I heard from others who all maintain internal Gerrits and internal forks of the modules. The difference is that Fuel is being worked on in the open in StackForge. Anyone is free to contribute to Fuel as he or she wishes, take our patches, or review changesets. Starting in October 2014, the Fuel team has adopted a policy that we cannot merge any patches into the core Puppet OpenStack modules of Fuel without submitting a patch or at least a bug upstream. Our reviewers block patches consistently. The truth is that the upstream modules are quite excellent and we don't need to make changes so often. Our goal is to work with upstream modules or in the issue where upstream integration is impossible, we place that config in our own separate modules. The point you raised about fixing bugs in Fuel and not in Puppet OpenStack is definitely valid and something we need to collaborate on. The first and easiest option when a bug is applicable to both, we could use Launchpad to assign bugs to both Fuel project and puppet-$project so it gains visibility. If a bug is discovered in Puppet OpenStack after it's been reported and/or fixed in Fuel, then it would be best to ping someone in #fuel-dev on IRC and we can try to figure out how to get this applied upstream correctly. Our best results come from fixing upstream and I want to make sure that is clear. If you have specific bugs or commits you'd like to discuss, let's work them out. I believe I can get the Fuel Library team to agree to do a walk through all the open bugs in Puppet OpenStack and see if we have any related fixes or bug reports. Best Regards, Matthew Mosesohn On Thu, Jun 11, 2015 at
Re: [openstack-dev] [puppet] [fuel] more collaboration request
First of all, thank you Emilien for bringing this up, and thank you Matt for confirming our commitment to collaborate with puppet-openstack and other Puppet modules that Fuel developers consider upstream. I'd like to add some more concrete examples of what Fuel team has already done, is doing, and what more we can do to improve our collaboration with other Puppet developers. As Matt has pointed out, sharing bugfixes for forked Puppet modules is already a requirement for all Fuel developers: https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Fuel_library_for_puppet_manifests Here are the key bits that are specifically designed to simplify reconciling of changes between Fuel and its upstream: 1) If you are adding a module that is the work of another project and is already tracked in separate repo (...) review should also contain the commit hash from the upstream repo in the commit message. Using this reference to upstream commit, you can cleanly and automatically isolate Fuel specific changes to that module into a patch series with two useful properties: a) You can associate each deviant line with a commit in fuel-library repository that would refer to specific LP bug or blueprint and otherwise explain the change. b) The whole patch series can be cleanly applied on top of the specified commit in upstream git. This means you can automatically graft a branch made out of the patch series onto upstream git, and then use git rebase to make that branch mergeable with the current upstream git head. 2) submit patch to upstream module first, and once merged or +2 recieved from a core reviewer, the patch should be backported to Fuel library as well. Aside from the obvious benefit of immediate contribution to upstream, this rule helps to keep Fuel specific changes confined within Fuel-specific modules, and discourages arbitrary deviations of forked modules from upstream. It's important to understand that Fuel's How to contribute guide is not just a set of recommendations for external contributors: it is the primary definition of the engineering process that all Fuel developers, within and without Mirantis, are expected to follow. If you come across a violation of these rules in Fuel, it is a bug, you're more than welcome to report it Launchpad, and Fuel developers will address it. I understand that even with these rules in place the process isn't perfect, even if it were followed flawlessly. If you have ideas on what else we can tweak to make upstream's life easier, please share, we're always looking for ways to improve our processes. I hope you understand that there are some tradeoffs that are not worth considering. While we can (and do) set aside some time in every release to reconcile our local changes with upstream, we can't allow any external factors to affect our ability to meet our own deadlines. For example, we can and should push all our local changes to upstream, but we can't hold back a fix for a Critical bug in Fuel until it is found acceptable by upstream. We can further minimize the delta between our fork and an upstream module, even eliminate the delta altogether when possible, but we have to keep a copy of the module in a repository that we control. We can and should make a more efficient use of the time our engineers spend interacting with upstream, but only as long as that does not subtract from the effort we put into Fuel project's priorities. At the same time, it's important for Fuel developers to understand that even though Fuel source code is publicly available and tracking and minimizing deviation from upstream is encoded in our engineering process, it doesn't mean there's no room for improvement. Saying don't worry, be happy, we've got it under control, the ball is on your side is not good enough. P.S. There's been more emails on the thread while I was writing this, and some points raised are not covered in this email, I'll try to address them in another followup. -- Dmitry Borodaenko On Thu, Jun 11, 2015 at 05:36:57PM +0300, Matthew Mosesohn wrote: Hi Emilien, I can see why you might be unhappy with Fuel's actions with regards to the OpenStack Puppet modules. You could make this argument about many components in Fuel. The heart of the matter is that we bundle the upstream OpenStack Puppet modules with all the other modules, developed both upstream and by Fuel's developers in one single git repository. This decision, along with all the other decisions to put Fuel's components under its own repositories, was intended to add stability and granular control to the product. I'm not saying it's the most community-oriented approach, but Fuel would have never evolved and matured without it. The attribution in commits is lost because our directory namespace differs such that it can't just be merged cleanly. Revisiting submodules is an option, but it led to maintenance problems in the past. Secondly, I'd like to point out that Fuel is not
Re: [openstack-dev] [puppet] [fuel] more collaboration request
Hi, Before you read me, please remember I know almost nothing about puppet. :) On 06/11/2015 11:03 PM, Matt Fischer wrote: We as a community don't do a great job watching bugs, so personally I'd prefer that fuel developers just push patches, filing a bug too if you want. (Note: we do need to improve our bug tracking!) However, I don't think that asking puppet openstack devs to ask in the fuel channel if a given bug is fixed in fuel is a very sustainable model. I'm not sure of many projects where the upstream polls downstream to ask whether they've fixed bugs. Can we come up with a way to collaborate more that everyone likes? I think there's a lot of value in it for everyone, we all get a better product out of it. On Thu, Jun 11, 2015 at 8:36 AM, Matthew Mosesohn The point you raised about fixing bugs in Fuel and not in Puppet OpenStack is definitely valid and something we need to collaborate on. The first and easiest option when a bug is applicable to both, we could use Launchpad to assign bugs to both Fuel project and puppet-$project so it gains visibility. If a bug is discovered in Puppet OpenStack after it's been reported and/or fixed in Fuel, then it would be best to ping someone in #fuel-dev on IRC and we can try to figure out how to get this applied upstream correctly. Our best results come from fixing upstream and I want to make sure that is clear. Matt, I appreciate a lot who you are, and all the help you've given me so far, but what you are asking here is wrong. You shouldn't ask Emilien to track the work of the Fuel team, and ping them on IRC to contribute back. It should be up to them to directly fix upstream *first*, and *then* fix back in Fuel. If you have specific bugs or commits you'd like to discuss, let's work them out. I believe I can get the Fuel Library team to agree to do a walk through all the open bugs in Puppet OpenStack and see if we have any related fixes or bug reports. It shouldn't be the way either. The team working on fuel-library should be pro-active and doing the contributions, Emilien shouldn't have to discuss a specific bug of commits. I believe also that Emilien's reasoning goes beyond just one or 2 commits, it's a general thinking. On 06/11/2015 04:36 PM, Matthew Mosesohn wrote: I can see why you might be unhappy with Fuel's actions with regards to the OpenStack Puppet modules. You could make this argument about many components in Fuel. The heart of the matter is that we bundle the upstream OpenStack Puppet modules with all the other modules, developed both upstream and by Fuel's developers in one single git repository. This decision, along with all the other decisions to put Fuel's components under its own repositories, was intended to add stability and granular control to the product. I'm not saying it's the most community-oriented approach, but Fuel would have never evolved and matured without it. The attribution in commits is lost because our directory namespace differs such that it can't just be merged cleanly. Revisiting submodules is an option, but it led to maintenance problems in the past. This isn't the only place where we have a huge git repository doing everything. This IMO is a big mistake, which give us more work because we have to duplicate what's upstream and constantly rebase. This is yet another technical dept... This only works because we have a lot of Mirantis employee doing the work, so the inefficiency is counter balanced by the work force. But as you know, we're always pushing to the very limit of everyone to release a new version of MOS and Fuel, so maybe now is the time to rethink the way we work. To move forward, I really believe we (as in: Mirantis) should be: 1/ Rework fuel-library to use multiple git for puppet, and maybe work out a way to package these individually. 2/ Using unmodified version of upstream puppet as much as possible 3/ Work *only* on upstream puppet and not on a separate fork As a lot of the changes that I propose, this would be a one-off painful effort to kill this technical dept, but on the long run, we would really benefits from such reorganization. If we don't do the above, it's going to be business as usual, no mater how much efforts Mirantis engineer will put on: the pressure we have to deliver Fuel/MOS should shift from the fork to what's upstream. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On Thu, Jun 11, 2015 at 05:39:28PM -0400, Emilien Macchi wrote: On 06/11/2015 10:36 AM, Matthew Mosesohn wrote: I'm not saying it's the most community-oriented approach, but Fuel would have never evolved and matured without it. The attribution in commits is lost because our directory namespace differs such that it can't just be merged cleanly. Revisiting submodules is an option, but it led to maintenance problems in the past. What kind of problems? It was before I got involved with Fuel, but I can offer a guess. Managing submodules introduces operational overhead and with it more opportunities for failures and mishaps than a single flat git repo. Flat repo makes it more difficult to reconcile with upstream modules, but, when using the process I have described in my previous email to this thread, not as difficult as one could think. We also reconcile an average module with upstream much less frequently (no more than once per Fuel release) than we make commits to that module (many times per release), which also tilts the overhead balance if favor of using a flat repo. You also could have used forks of modules, applied your patches and done rebase from time to time when you like. Using a Puppetfile in your installer and you're all set. We do the apply patches and rebase from time to time part in a single repo, operating on a subdirectly within it doesn't actually add any overhead to this part of the workflow, as long as we keep track of sync commits properly. The maintenance problems thing does not sound right to me here, I think it's more expensive to maintain files than git repositories. To sum up, I don't think the difference is that great, or impact of doing it one way or the other that important. The current layout works well enough for Fuel team, and as you said yourself, developers of upstream modules are not likely to pro-actively harvest Fuel for bugfixes even if we change our repository structure. The difference is that Fuel is being worked on in the open in StackForge. Anyone is free to contribute to Fuel as he or she wishes, take our patches, or review changesets. Should not it be the way around? Puppet OpenStack modules provide the original code. If there is a bug, it has to be fixed in the modules. Puppet OpenStack developers don't have time/bandwidth and moreover don't want to periodically have a look at Fuel git history. I'm not sure this is the best solution for the community. (...) The reality (and again I won't blame any patch, you can find them on Gerrit) is that most of patches are not merged and in staled status. If I can suggest something, the policy should be more upstream first where you submit a patch upstream, you backport it downstream, and in the until it's merged you should make sure it land upstream after community review process. The last step is I think the problem I'm mentioning here and part of the root cause of this topic. Yes, this right here is the biggest point of contention in the whole discussion. The most problematic implication of what you're asking for is the additional effort that it would require from Fuel developers. When you say that Puppet OpenStack developers don't have time to look at Fuel git history for bugfixes, and then observe that actually Fuel developers do propose their patches to upstream, but those patches are stalled in the community review process, this indicates that you don't consider taking over and landing these patches a priority: http://lifehacker.com/5892948/instead-of-saying-i-dont-have-time-say-its-not-a-priority The fact that you have started this thread means that you actually do care to get these bugfixes into Puppet OpenStack. If that's true, maybe you can consider a middleground approach: Fuel team agrees to propose all our changes to upstream (i.e. do a better job at something we've already committed to unilaterally), and you help us land the patches we propose, and take over those that get stalled when the submitter from Fuel team has moved on to other tasks? A better alternative would be to make all upstream Puppet OpenStack directly usable in Fuel, but even if we figure out a way to make that work, it will take a long journey to get there. On the upstream side, Fuel core reviewers would have to also be upstream core reviewers, and Fuel CI would have to be allowed to vote on upstream commits. On Fuel side, we'd have to complete the reconciliation and modularization of all our forked modules, and move all Fuel CI to openstack-infra. The potential benefits for both sides are tremendous, but only after we essentially merge the two projects. Even if that's achievable, is that something whole Puppet OpenStack community is interested in? Again, I'm not sure this is the right direction. The official channel for Puppet OpenStack modules is #puppet-openstack and this is the place to be when you're involved in the Puppet OpenStack community. I would suggest to rewrite this thing in if a
Re: [openstack-dev] [puppet] [fuel] more collaboration request
Dmitry, thank you for taking your time. Please read inline: On 06/11/2015 07:12 PM, Dmitry Borodaenko wrote: First of all, thank you Emilien for bringing this up, and thank you Matt for confirming our commitment to collaborate with puppet-openstack and other Puppet modules that Fuel developers consider upstream. I'd like to add some more concrete examples of what Fuel team has already done, is doing, and what more we can do to improve our collaboration with other Puppet developers. As Matt has pointed out, sharing bugfixes for forked Puppet modules is already a requirement for all Fuel developers: https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Fuel_library_for_puppet_manifests The documentation is good, and well written. Here are the key bits that are specifically designed to simplify reconciling of changes between Fuel and its upstream: 1) If you are adding a module that is the work of another project and is already tracked in separate repo (...) review should also contain the commit hash from the upstream repo in the commit message. Using this reference to upstream commit, you can cleanly and automatically isolate Fuel specific changes to that module into a patch series with two useful properties: a) You can associate each deviant line with a commit in fuel-library repository that would refer to specific LP bug or blueprint and otherwise explain the change. b) The whole patch series can be cleanly applied on top of the specified commit in upstream git. This means you can automatically graft a branch made out of the patch series onto upstream git, and then use git rebase to make that branch mergeable with the current upstream git head. I spent some time to read Fuel Library, specially its custom manifests [1] and I don't see any example of these commits. I'm pretty sure I've missed them, can you give some example? [1] https://github.com/stackforge/fuel-library/commits/master/deployment/puppet/openstack/manifests/ 2) submit patch to upstream module first, and once merged or +2 recieved from a core reviewer, the patch should be backported to Fuel library as well. Aside from the obvious benefit of immediate contribution to upstream, this rule helps to keep Fuel specific changes confined within Fuel-specific modules, and discourages arbitrary deviations of forked modules from upstream. Please let me show you an example of a bug that has been fixed in Fuel, but not in upstream (puppet-nova here), *after* the new Fuel policy (mentioned by Matthew, October 2014). The bug: https://bugs.launchpad.net/fuel/+bug/1378962 Node Libvirt Unique UUID Not Generated On Deployment Patch in Fuel, merged released: https://review.openstack.org/#/c/128640/ Patch in puppet-nova, with negative feedback, without any +2 and staled for October 2014: https://review.openstack.org/#/c/131710 This is an example and again, I don't blame anyone here. I respect people who did the patches and I'm happy it's fixed in Fuel. It's important to understand that Fuel's How to contribute guide is not just a set of recommendations for external contributors: it is the primary definition of the engineering process that all Fuel developers, within and without Mirantis, are expected to follow. If you come across a violation of these rules in Fuel, it is a bug, you're more than welcome to report it Launchpad, and Fuel developers will address it. I understand that even with these rules in place the process isn't perfect, even if it were followed flawlessly. If you have ideas on what else we can tweak to make upstream's life easier, please share, we're always looking for ways to improve our processes. Like puppet-murano? Someone from Fuel team created first the module in Fuel, 6 months ago [1] and 3 months later someone from Fuel team created an empty repository in Stackforge [2]. By the way, Puppet OpenStack community does not have core permissions on this module and it's own by Murano team. [1] https://github.com/stackforge/fuel-library/tree/master/deployment/puppet/murano/ [2] https://review.openstack.org/#/c/155688/ Again another example, where I'm very happy to see the module in Fuel but nothing in Stackforge leveraged by Puppet OpenStack community. I hope you understand that there are some tradeoffs that are not worth considering. While we can (and do) set aside some time in every release to reconcile our local changes with upstream, we can't allow any external factors to affect our ability to meet our own deadlines. For example, we can and should push all our local changes to upstream, but we can't hold back a fix for a Critical bug in Fuel until it is found acceptable by upstream. We can further minimize the delta between our fork and an upstream module, even eliminate the delta altogether when possible, but we have to keep a copy of the module in a repository that we control. We can and should make a more efficient use of the time our
Re: [openstack-dev] [puppet] [fuel] more collaboration request
On 06/11/2015 10:31 PM, Dmitry Borodaenko wrote: On Thu, Jun 11, 2015 at 05:39:28PM -0400, Emilien Macchi wrote: On 06/11/2015 10:36 AM, Matthew Mosesohn wrote: I'm not saying it's the most community-oriented approach, but Fuel would have never evolved and matured without it. The attribution in commits is lost because our directory namespace differs such that it can't just be merged cleanly. Revisiting submodules is an option, but it led to maintenance problems in the past. What kind of problems? It was before I got involved with Fuel, but I can offer a guess. Managing submodules introduces operational overhead and with it more opportunities for failures and mishaps than a single flat git repo. Flat repo makes it more difficult to reconcile with upstream modules, but, when using the process I have described in my previous email to this thread, not as difficult as one could think. We also reconcile an average module with upstream much less frequently (no more than once per Fuel release) than we make commits to that module (many times per release), which also tilts the overhead balance if favor of using a flat repo. What about code history and respect of commit ownership? I'm personally wondering if it's fair to copy/paste several thousands of lines of code from another Open-Source project without asking to the community or notifying the authors before. I know it's Open-Source and Apache 2.0 but well... :-) You also could have used forks of modules, applied your patches and done rebase from time to time when you like. Using a Puppetfile in your installer and you're all set. We do the apply patches and rebase from time to time part in a single repo, operating on a subdirectly within it doesn't actually add any overhead to this part of the workflow, as long as we keep track of sync commits properly. The maintenance problems thing does not sound right to me here, I think it's more expensive to maintain files than git repositories. To sum up, I don't think the difference is that great, or impact of doing it one way or the other that important. The current layout works well enough for Fuel team, and as you said yourself, developers of upstream modules are not likely to pro-actively harvest Fuel for bugfixes even if we change our repository structure. The difference is that Fuel is being worked on in the open in StackForge. Anyone is free to contribute to Fuel as he or she wishes, take our patches, or review changesets. Should not it be the way around? Puppet OpenStack modules provide the original code. If there is a bug, it has to be fixed in the modules. Puppet OpenStack developers don't have time/bandwidth and moreover don't want to periodically have a look at Fuel git history. I'm not sure this is the best solution for the community. (...) The reality (and again I won't blame any patch, you can find them on Gerrit) is that most of patches are not merged and in staled status. If I can suggest something, the policy should be more upstream first where you submit a patch upstream, you backport it downstream, and in the until it's merged you should make sure it land upstream after community review process. The last step is I think the problem I'm mentioning here and part of the root cause of this topic. Yes, this right here is the biggest point of contention in the whole discussion. The most problematic implication of what you're asking for is the additional effort that it would require from Fuel developers. When you say that Puppet OpenStack developers don't have time to look at Fuel git history for bugfixes, and then observe that actually Fuel developers do propose their patches to upstream, but those patches are stalled in the community review process, this indicates that you don't consider taking over and landing these patches a priority: We don't consider taking the patches? Please go on Gerrit, look at the patches and tell me if there is no review from Puppet OpenStack community. Most of the patchs are -1 or not passing unit testing which means the code can't be merged. Let me give you examples so you can see Puppet OpenStack folks is doing reviews on patchs from Fuel team: https://review.openstack.org/#/c/170485/ https://review.openstack.org/#/c/157004/ https://review.openstack.org/#/c/176924/ https://review.openstack.org/#/c/168848/ https://review.openstack.org/#/c/130937/ https://review.openstack.org/#/c/131710/ https://review.openstack.org/#/c/174811/ And this is only 'in progress' patches. A lot of fixed have been abandoned upstream. You can easily query them on Gerrit. http://lifehacker.com/5892948/instead-of-saying-i-dont-have-time-say-its-not-a-priority The fact that you have started this thread means that you actually do care to get these bugfixes into Puppet OpenStack. If that's true, maybe you can consider a middleground approach: Fuel team agrees to propose all our changes to upstream (i.e. do a better job
[openstack-dev] [puppet] [fuel] more collaboration request
Hi, Before reading this e-mail, please keep in mind: * I have a lot of admiration for Fuel and since I'm working on OpenStack Installers (at eNovance and now Red Hat), Fuel is something I always consider a good product. * This e-mail is about Fuel and Puppet, nothing about Mirantis. * I'm writing on behalf of my thoughts, and not on our group. * I'm using open mailing-list for open discussion. There is not bad spirit in this e-mail and I want to have a productive thread. I have some concerns I would like to share with you and hopefully find some solutions together. Since I've been working on Puppet OpenStack (2 years now), I see some situations that happen - according to me - too often: * A bug is reported in both Fuel Library and the Puppet module having trouble. A patch is provided in Fuel Library (your fork of Puppet OpenStack modules) but not in Puppet upstream module. That means you fix the bug for Fuel, and not for Puppet OpenStack community. It does not happen all the time but quite often. * A patch is submitted in a Puppet module and quite often does not land because there is no activity, no tests or is abandonned later because fixed in Fuel Library. I've noticed the patch is fixed in Fuel Library though. * RAW copy/paste between upstream modules code and your forks. In term of Licensing, I'm even not sure you have the right to do that (I'm not a CLA expert though) but well... in term of authorship and statistics on code, I'm not sure it's fair. Using submodules with custom patches would have been great to respect the authors who created the original code and you could have personalize the manifests. Note: you can see that I don't give any example because I'm not here to blame people or judge anyone. So the goal of my e-mail is to open the discussion and have a *real* collaboration between Fuel team which seems to have a lot of good Puppet engineers and Puppet OpenStack team. We had this kind of discussion at the Summit (in Vancouver and also Paris, and even before). Now I would like to officialy know if you are interested or not to be more involved. I'm also open at any feedback about Puppet OpenStack group and if something blocks you to contribute more. We have the same goals, having Puppet modules better. I think it can be win/win: you have less diff with upstream and we have more hands in our module maintenance. Thank you for reading so far, and I'm looking forward to reading from you. Best regards, -- Emilien Macchi signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
