On 13.10.2013, at 01:26, "Joe Gordon" <joe.gord...@gmail.com<mailto:joe.gord...@gmail.com>> wrote:
On Sat, Oct 12, 2013 at 12:21 PM, Alessandro Pilotti <apilo...@cloudbasesolutions.com<mailto:apilo...@cloudbasesolutions.com>> wrote: On 12.10.2013, at 20:22, "Dan Smith" <d...@danplanet.com<mailto:d...@danplanet.com>> wrote: >> From the user perspective, splitting off the projects seems to be >> focussing on the ease of commit compared to the final user >> experience. > > I think what you describe is specifically the desire that originally > spawned the thread: making the merging of changes to the hyper-v driver > faster by having them not reviewed by the rest of the Nova team. It > seems to be what the hyper-v developers want, not necessarily what the > Nova team as a whole wants. > >> An 'extras' project without *strong* testing co-ordination with >> packagers such as SUSE and RedHat would end up with the consumers of >> the product facing the integration problems rather than resolving >> where they should be, within the OpenStack project itself. > > I don't think splitting out to -extras means that it loses strong > testing coordination (note that strong testing coordination does not > exist with the hyper-v driver at this point in time). Every patch to the > -extras tree could still be unit (and soon, integration) tested against > the current nova tree, using the proposed patch applied to the -extras > tree. It just means that a change against nova wouldn't trigger the > same, which is why the potential for "catch up" behavior would be required. > >> I am sympathetic to the 'extra' drivers problem such as Hyper-V and >> powervm, but I do not feel the right solution is to split. >> >> Assuming there is a summit session on how to address this, I can >> arrange a user representation in that session. > > Cool, I really think we're at the point where we know the advantages and > disadvantages of the various options and further face-to-face discussion > at the summit is what is going to move us to the next stage. > I agree. Looks like we are converging towards a common ground. I'm summing it up here, including a few additional details, for the benefit of who will not join us in HK (sorry, we'll party for you as well :-)): This sounds like a very myopic solution to the issue you originally raised, and I don't think it will solve the underlying issues. The solution I just proposed was based on the feedbacks received on this thread trying to make everybody happy, so if you find it "myopic" please be my guest and find a better one that suits all the different positions. :-) Taking a step back, you originally raised a concern about how we prioritize reviews with the havana-rc-potential tag. "In the past weeks we diligently marked bugs that are related to Havana features with the "havana-rc-potential" tag, which at least for what Nova is concerned, had absolutely no effect. Our code is sitting in the review queue as usual and, not being tagged for a release or prioritised, there's no guarantee that anybody will take a look at the patches in time for the release. Needless to say, this starts to feel like a Kafka novel. :-)" [1] If the issue is just better bug triage and prioritizing reviews, help us do that! [2] shows the current status of your hyper-v havana-rc-potential bugs. Currently there are only 7 bugs that have both tags. Of those 7, 3 have no pending patches to trunk, and one doesn't sound like it warrants a back port (https://bugs.launchpad.net/nova/+bug/1220256). Looking at the remaining 4, one is marked as a WIP by you (https://bugs.launchpad.net/nova/+bug/1231911 https://review.openstack.org/#/c/48645/) which leaves three patches for nova team to review. Three reviews open for a week doesn't sound like an issue that warrants a whole new repository. Sure, it's not the volume of reviews the subject here. This is just the icing on the cake on something that goes on since a while (see Havana feature freeze). You went on to clarify your position. "I'm not putting into discussion how much and well you guys are working (I actually firmly believe that you DO work very well), I'm just discussing about the way in which blueprints and bugs get prioritised. <snip> On the other side, to get our code reviewed and merged we are always dependent on the good will and best effort of core reviewers that don't necessarily know or care about specific driver, plugin or agent internals. This brings to even longer review cycles even considering that reviewers are clearly doing their best in understanding the patches and we couldn't be more thankful. "Best effort" has also a very specific meaning: in Nova all the Havana Hyper-V blueprints were marked as "low priority" (which can be translated in: "the only way to get them merged is to beg for reviews or maybe commit them on day 1 of the release cycle and pray") while most of the Hyper-V bugs had no priority at all (which can be translated in "make some noise on the ML and IRC or nobody will care"). :-) This reality unfortunately applies to most of the sub-projects (non only Hyper-V) and can be IMHO solved only by delegating more authonomy to the sub-project teams on their specific area of competence across OpenStack as a whole. Hopefully we'll manage to find a solution during the design summit as we are definitely not the only ones feeling this way, by judging on various threads in this ML." [3] Once again you raise the issue of bug triage and prioritization of reviews (and blueprints), so help us fix that! This isn't a virt driver only issue though. The issues you originally raise are only incidentally related to virt drivers and hyper-v. The same issues can be brought up as you point out by any sub-project (scheduling, APIs, DB, etc). So a fix for only virt drivers hardly sounds like an appropriate solution. I would like t propose an alternate two part solution to the issues you originally raised. * It sounds like nova needs to do a better job of bug triage, prioritizing reviews and blueprints. With so many bugs and reviews its very easy for us to mess this up. How to do a better job? I think we need to brainstorm on this a little. On this I agree with you: we definitely need to brainstorm. * Do a better job of tracking how knowledgeable a reviewer is about a subset of code, just because someone is in core doesn't mean they know the most about a specific component. I think its safe to say that on average the more someone commits patches and reviews a file the more he knows about it. With git and gerrit we have all this information we just need to use it. Here is a quick proof of concept of the idea, (as a proof of concept it just lists previous authors of the modified files): https://gist.github.com/jogo/6955514, along with some sample outputs http://paste.openstack.org/show/48340/. I'm case you didn't notice, none of the code reviewers has the slightest idea about what's going on in the Hyper-V driver beyond the surface (I cannot speak for other drivers, but I surely suspect that we are not alone here). As already stated in a previous mail, if you look at the reviews history, most of them are related to unit tests and formal stuff. None of the Nova reviewers ever bothered (to my knowledge) to setup a basic Hyper-V deployment, at least to get a basic idea of what they were reviewing. I definitely don't blame anybody for this. I just keep on saying that it does not make any sense for them to review them! Don't get me wrong, I'm absolutely thankful to all the people that helped with their +2 in all this time, but it just became annoying for us to beg for reviews and for reviewers to take responsibility in reviewing something completely alien. As a comparison with Neutron, where the collaboration went smooth from day one, here's what happened. The Neutron team had (has?) a policy in which every new plugin must have a core dev willing to take care of it. One core reviewer volunteered and guess what? He spent a few hours with us in setting up a fully working Hyper-V deployment. He gave his +2 on the initial plugin/agent commit after being sure that he understood every detail of the plugin and agent and that everything was working. In the process we learned a lot about Neutron details as well (Quantum at the time) and our collaboration goes on happily ever since. In Nova I only hear as an answer that Jenkins and the CI gates take care of the actual validation and that anybody can commit code blindly on whatever component. Well, although I strongly believe in the importance of test coverage, that does not mean that it gives the understanding to a reviewer to be authoritative on components that he/she does not understand. Do you guys really want to keep the drivers in Nova? Then let's stop talking about OUR will to get out of our "comfort zone" and let's organize a training day in which every Nova core dev gets his/her hands dirty on a full scale OpenStack Hyper-V deployment ("uhh Windows? I cannot, it's against my religion!"). And then let's reiterate for every other driver. We'll happily join in. ;-) If you don't like any of the options that this already long thread is providing, I'm absolutely open to discuss any constructive idea. But please, let's get out of this awful mess. OpenStack is still a young project. Let's make sure that we can hand it on to the next devs generations by getting rid of these management bottlenecks now! [1] http://lists.openstack.org/pipermail/openstack-dev/2013-October/016382.html <http://lists.openstack.org/pipermail/openstack-dev/2013-October/016382.html> [2] https://bugs.launchpad.net/nova/+bugs?field.searchtext=&orderby=-importance&search=Search&field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=havana-rc-potential+hyper-v+&field.tags_combinator=ALL&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on [3] http://lists.openstack.org/pipermail/openstack-dev/2013-October/016400.html 1) All the drivers will still be part of Nova. 2) One official project (nova-drivers-incubator?) or more than one will be created for the purpose of supporting a leaner and faster development pace of the drivers. 3) Current driver sub-project teams will informally elect their maintainer(s) which will have +2a rights on the new project or specific subtrees. 4) Periodically, code from the new project(s) must be merged into Nova. Only Nova core reviewers will have obviously +2a rights here. I propose to do it on scheduled days before every milestone, differentiated per driver to distribute the review effort (what about also having Nova core reviewers assigned to each driver? Dan was suggesting something similar some time ago). 5) All drivers will be treated equally and new features and bug fixes for master (except security ones) should land in the new project before moving to Nova. 6) CI gates for all drivers, once available, will be added to the new project as well. Only drivers code with a CI gate will be merged in Nova (starting with the Icehouse release as we already discussed). 7) Active communication should be maintained between the Nova core team and the drivers maintainers. This means something more than: "I wrote it on the ML didn't you see it?" :-) A couple if questions: will we keep version branches on the new project or just master? Bug fixes for older releases will be proposed to the incubator for the current release in development and to Nova for past versions branches? Please correct me if I missed something! Thanks, Alessandro > --Dan > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
