Re: [openstack-dev] [neutron] Proposing Assaf Muller for the Neutron Core Reviewer Team
+1 from me, long overdue! On May 28, 2015, at 9:42 AM, Kyle Mestery mest...@mestery.com wrote: Folks, I'd like to propose Assaf Muller to be a member of the Neutron core reviewer team. Assaf has been a long time contributor in Neutron, and he's also recently become my testing Lieutenant. His influence and knowledge in testing will be critical to the team in Liberty and beyond. In addition to that, he's done some fabulous work for Neutron around L3 HA and DVR. Assaf has become a trusted member of our community. His review stats place him in the pack with the rest of the Neutron core reviewers. I'd also like to take this time to remind everyone that reviewing code is a responsibility, in Neutron the same as other projects. And core reviewers are especially beholden to this responsibility. I'd also like to point out that +1/-1 reviews are very useful, and I encourage everyone to continue reviewing code even if you are not a core reviewer. Existing Neutron cores, please vote +1/-1 for the addition of Assaf to the core reviewer team. Thanks! Kyle [1] http://stackalytics.com/report/contribution/neutron-group/180 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] TC Candidacy
On Apr 24, 2015, at 6:17 AM, Luke Gorrie l...@tail-f.com wrote: Question regarding your candidacy: If I recall correctly you have spoken in favor of face to face discussions at mid-cycle meetings as a practical way to set priorities and move things forward. What are your current thoughts on the costs and benefits of this? I don't think our TC should be involved in a project's decision to hold a mid-cycle unless legitimate complaints about the openness of the mid-cycle remain unaddressed by the project's leadership. My experience is that face-to-face communication can aid in building the relationships and technical understanding essential to writing good software. The drawbacks are that proximity can be both expensive and exclusionary. I think it's up to each project community to decide whether a mid-cycle is justified, and if so, work hard to minimize the downsides. Foundation support is often available for those that can't afford to attend for financial reasons. For those that otherwise can't attend, there is the option of encouraging designation of a representative, ensuring that the mid-cycle is recorded or logged, and deferring decision-making on explored options to the wider community after the mid-cycle. Maru __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Question for the TC candidates
On Apr 23, 2015, at 9:14 AM, Chris Dent chd...@redhat.com wrote: What can and should the TC at large, and you specifically, do to ensure quality improves for the developers, end-users and operators of OpenStack as a full system, both as a project being developed and a product being used? I have strong opinions about how we can improve quality. I stated some of them in my candidate email. I think we have a more pressing concern, though. How do we ensure that our TC is capable of effecting the change we require? Our TC's primary mission used to be defining what OpenStack was and was not. The move to a 'big tent' model reduced the need to focus on that issue, and now our TC needs to direct its attention to improving our community and the software it delivers. This new mission changes the game, because top-down direction is not going to work. Without direct authority to make change happen, how can the TC hope to deliver? Indirect authority. 13 people of influence, with on-the-ground support developed and maintained at the project level, pulling in the same direction. That's the real power of the TC. And that's the power we're going to use to effect real change. This power is created by each TC member doing the hard work of engaging. It's writing, testing, debugging and documenting our software. It's conversations with developers, users, distros, and operators. It's developing an emotional connection - empathizing - with their constituents as they face the frustration caused by our community and software. It's making a difference so that their opinion is respected. This is the way things work in the individual projects, and our TC should differ only in the scale of the problems it tackles. This reinforces the idea, as has been suggested by other candidates, that the role of TC member needs to be taken more seriously. Change will not be decided because our TC knows best, but because its members collectively have first-hand experience with the problems at hand, have considered a diversity of potential solutions, and do the hard work of justifying their conclusions. Change will happen not because our TC commands it, but because project-level contributors have trust in our TC members and value their input. Only by being more engaged with the wider community can our TC hope to be an effective force for change. Respectfully, Maru __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] TC Candidacy
My name is Maru Newby, and I am announcing my candidacy for the Technical Committee (TC) election. tl;dr; I'm a relative unknown outside of Neutron, but I've been helping to drive high-level change in that project. I would welcome the opportunity to bring my energy and dedication to OpenStack-wide concerns as a member of our TC. If you're willing to read any part of this email, I hope it would be 'Why vote for me?'. If not, and you are as frustrated with the status quo as I am, I hope you will consider voting for candidates that support the goal of building a more engaged TC focused on ensuring that participation in OpenStack becomes more enjoyable and sustainable. Thank you for reading, and make sure to vote! * Why vote for me? If elected, I'm going to be a squeaky wheel advocating for anything and everything that empowers our development community to deliver useful software into the hands of users. If this puts me at odds with those that want to focus on issues like 'What is OpenStack' or 'How can we systematize project culture to make it easier to communicate?', good! I believe that you, as a technical contributor to OpenStack, need stronger representation of your viewpoints on our TC, and I'm one of the strongest advocates you're likely to find in this community. I am currently employed full time upstream, and I am willing and able to devote the resources necessary to do justice to the role. I also intend to advocate for focusing the OpenStack community on ambitious, but achievable, goals. I believe this will require making hard choices about which use cases we can reasonably expect to support. To me, the idea that OpenStack can be all things to all people is dangerous. It's just as likely that in trying to do it all, we do little of it well, and risk irrelevance. I think our primary competition is and will continue to be proprietary cloud, and my biggest fear is that we become unable to compete on cost due to the operational complexity required to make everyone happy. We need to keep our collective eyes on the ball! To be clear, I want to be a part of a TC with as diverse a group of viewpoints as possible to ensure that we have to work hard to achieve consensus. If agreement is reached too easily, there is probably something wrong. My wanting to push a developer-centric agenda doesn't mean that I don't respect the need to address other issues. My voice would be one of many, and I recognize that consensus requires compromise. * Why not vote for me? There are candidates more deserving of your vote if you think our TC shouldn't have a role in providing leadership to the OpenStack community. If you believe that our TC is already sufficiently developer-focused, then I also don't deserve your vote. This is in no way to suggest that I want to see our TC become a top-down decision-making body. This is still an open source project, after all, and I know first-hand the complexities of the culture involved. I do think it appropriate, though, for an elected body with as much visibility as our TC to participate more actively in addressing the challenges we face. * Key concerns There are any number of issues facing OpenStack today, but the items on the shortlist that follows are the concerns that I would like to see the TC champion in the near-term: ** Scaling our development effort The stated goal of our TC of late has been to 'get out of the way'. I wholeheartedly support this intention, and would like to see it extended to how our TC approaches governance beyond project evaluation. I think our TC should be responsible for proposing what we want to achieve rather than in how we should achieve it. Outcomes, rather than process. I think project-level contributors are best equipped to solve the problems of implementation. Our TC can take responsibility for setting OpenStack-wide expectations and in facilitating - rather than managing - cross-project advancement towards these goals. More than ever, we need to be pulling in the same direction, and I think our TC is an obvious candidate to rally ourselves around. I hope we all recognize that we can't scale OpenStack if decisions are constantly gated on a small group of 'tastemakers'. Delegation of trust is required, whether at the level of our TC or the individual projects. I think we need to take our TC's recent momentum to the project level and find ways to increase delegation of responsibility. Nova and Neutron, for example, have had to face ever-increasing demands with the same small pool of core reviewers, and many of you know all-too-well the pain that has resulted. Core reviewers - like the pre-big tent TC - have inadvertently become micromanagers as their responsibilities have evolved beyond their capacity to meet them. The model of direct trust - in which each core needs to trust every other core - doesn't scale due to fundamental human limits. This is at odds with OpenStack's need to grow, and I want our TC
Re: [openstack-dev] [Openstack-operators] [Neutron] The specs process, effective operators feedback and product management
On Apr 10, 2015, at 11:04 AM, Boris Pavlovic bo...@pavlovic.me wrote: Hi, I believe that specs are too detailed and too dev oriented for managers, operators and devops. They actually don't want/have time to write/read all the stuff in specs and that's why the communication between dev operators community is a broken. I would recommend to think about simpler approaches like making mechanism for proposing features/changes in projects. Like we have in Rally: https://rally.readthedocs.org/en/latest/feature_requests.html This is similar to specs but more concentrate on WHAT rather than HOW. +1 I think the line between HOW and WHAT are too often blurred in Neutron. Unless we’re able to improve our ability to communicate at an appropriate level of abstraction with non-dev stakeholders, meeting their needs will continue to be a struggle. Maru __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The Evolution of core developer to maintainer?
On Apr 2, 2015, at 3:26 AM, Thierry Carrez thie...@openstack.org wrote: Maru Newby wrote: [...] Many of us in the Neutron community find this taxonomy restrictive and not representative of all the work that makes the project possible. We seem to be after the same end goal. I just disagree that renaming core reviewers to maintainers is a positive step toward that goal. Worse, 'cores' are put on a pedastal, and not just in the project. Every summit a 'core reviewer dinner' is held that underscores the glorification of this designation. I deeply regret that, and communicated to the sponsor holding it the problem with this +2 dinner the very first time it was held. FWIW it's been renamed to VIP dinner and no longer limited to core reviewers, but I'd agree with you that the damage was already done. By proposing to rename 'core reviewer' to 'maintainer' the goal was to lay the groundwork for broadening the base of people whose valuable contribution could be recognized. The goal was to recognize not just review-related contributors, but also roles like doc/bug/test czar and cross-project liaison. The statue of the people filling these roles today is less if they are not also ‘core’, and that makes the work less attractive to many. That's where we disagree. You see renaming core reviewer to maintainer has a way to recognize a broader type of contributions. I see it as precisely resulting in the opposite. Simply renaming core reviewers to maintainers just keeps us using a single term (or class) to describe project leadership. And that class includes +2 reviewing duties. So you can't be a maintainer if you don't do core reviewing. That is exclusive, not inclusive. The important part of my statement above was ‘lay the groundwork for’. We were intended to change the name as a _precursor_ to changing the role itself to encompass more than just those with +2 rights. Nobody in their right mind would assume that changing the name by itself could fix the situation, but we thought it would be a good signal as to our intent to broaden the scope of recognized contribution. What we need to do instead is reviving the drivers concept (we can rename it maintainers if you really like that term), separate from the core reviewers concept. One can be a project driver and a core reviewer. And one can be a project driver *without* being a core reviewer. Now *that* allows to recognize all valuable contributions, and to be representative of all the work that makes the project possible. As Joe and I have said, Nova and Neutron already have drivers teams and they fill a different role from what you are suggesting. Can you think of a more appropriate name that isn’t already in use for what you are proposing? Maru __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The Evolution of core developer to maintainer?
On Apr 2, 2015, at 9:02 AM, Thierry Carrez thie...@openstack.org wrote: Maru Newby wrote: On Apr 2, 2015, at 3:26 AM, Thierry Carrez thie...@openstack.org wrote: What we need to do instead is reviving the drivers concept (we can rename it maintainers if you really like that term), separate from the core reviewers concept. One can be a project driver and a core reviewer. And one can be a project driver *without* being a core reviewer. Now *that* allows to recognize all valuable contributions, and to be representative of all the work that makes the project possible. As Joe and I have said, Nova and Neutron already have drivers teams and they fill a different role from what you are suggesting. The only team I heard you and Joe mention is the one approving specs, and that one is called nova-specs-core, not nova-drivers: https://review.openstack.org/#/admin/groups/302,members The one I'm speaking about always existed in the capacity I'm describing (project leadership, people actually making sure the project goes somewhere and all the boring tasks get done), and is defined at: https://launchpad.net/~nova-drivers Regardless of what you are seeing for Nova, the Neutron community calls this team ‘drivers’ and documents it as such in the wiki: https://wiki.openstack.org/wiki/Neutron-drivers Maru Can you think of a more appropriate name that isn’t already in use for what you are proposing? I'm not proposing anything new. I'm saying we should just use (or revive the use of) existing teams (nova-core, nova-drivers) to achieve our goals of being representative of all the work that makes the project possible. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The Evolution of core developer to maintainer?
On Apr 1, 2015, at 6:09 AM, Jeremy Stanley fu...@yuggoth.org wrote: And here is the crux of the situation, which I think bears highlighting. These empowered groups are (or at least started out as) nothing more than an attempt to map responsibilities onto the ACLs available to our projects in the tools we use to do the work. Coming up with some new pie-in-the-sky model of leadership hierarchy is an interesting thought exercise, but many people in this discussion are losing sight of the fact that the model we have is determined to a great extent by the tools we use. Change the tools and you may change the model, but changing the model doesn't automatically change the tools to support it (and those proposing a new model need to pony up the resources to implement it in _reality_, not just in _thought_). Responsibilities not tied to specific controls in our tools do exist in abundance, but they tend to be more fluid and ad-hoc because in most cases there's been no need to wrap authorization/enforcement around them. What I worry is happening is that as a community we're enshrining the arbitrary constructs which we invented to be able to configure our tools sanely. I see this discussion as an attempt to recognize those other responsibilities as well, but worry that creation of additional unnecessary authorization/enforcement process will emerge as a solution and drive us further into pointless bureaucracy. Given how important trust and relationships are to the functioning of individual projects, I think we’re past the point where we should allow our tooling to be the limiting factor in how we structure ourselves. Do we need finer-grained permissions in gerrit to enable something like subtree maintainers? I don't believe we do. In large projects like Neutron, there is no such thing as someone who knows everything anymore, so we all need to be aware of our limitations and know not to merge things we don't understand without oversight from those of our peers that do. Responsibility in this case could be subject to social rather than tool-based oversight. Maru __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The Evolution of core developer to maintainer?
On Apr 1, 2015, at 1:47 PM, Jeremy Stanley fu...@yuggoth.org wrote: On 2015-04-01 12:00:53 -0700 (-0700), Maru Newby wrote: Given how important trust and relationships are to the functioning of individual projects, I think we’re past the point where we should allow our tooling to be the limiting factor in how we structure ourselves. I'm definitely not suggesting that either, merely pointing out that if you have an ACL which, for example, defines the set of people able to push a particular button then it's helpful to have a term for that set of people. As soon as you start to conflate that specific permission with other roles and responsibilities then the term for it gets overloaded. To me a core reviewer is just that: people with accounts in the .*-core Gerrit groups granted the ability to push a review button indicating that a proposed change is suitable to merge. Whether or not those same people are also afforded permissions outside that system is orthogonal. I find your perspective on the term ‘core reviewer’ to be interesting indeed, and for me it underscores the need to consider whether using the term outside of gerrit is justified. Maru __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The Evolution of core developer to maintainer?
On Apr 1, 2015, at 2:52 AM, Thierry Carrez thie...@openstack.org wrote: - Some people are core reviewers and maintainers (or drivers, to reuse the openstack terminology we already have for that) - Some people are core reviewers only (because they can't commit 90% of their work time to work on project priorities) - Some people are maintainers/drivers only (because their project duties don't give them enough time to also do reviewing) - Some people are casual developers / reviewers (because they can't spend more than 30% of their day on project stuff) All those people are valuable. Simply renaming core reviewers to maintainers (creating a single super-developer class) just excludes valuable people. I hear that you believe that the proposal to rename 'core reviewer' to 'maintainer' in Neutron was intended to entrench privilege. Nothing could be further from the truth - it was actually intended to break it down. As per Joe’s recent reply, ‘drivers’ in Nova have to be core reviewers. This is true in Neutron as well. I think a more accurate taxonomy, at least in Neutron, is the following: - Everyone that participates in the project is a 'contributor' - Some contributors are 'core reviewers' - members of the team with merge rights on a primary repo and a responsibility to actively review for that repo. - Some core reviewers are 'drivers' - members of a team with merge rights on the spec repo and a responsibility to actively review for that repo. This is obviously a gross simplification, but it should serve for what I'm trying to communicate. Many of us in the Neutron community find this taxonomy restrictive and not representative of all the work that makes the project possible. Worse, 'cores' are put on a pedastal, and not just in the project. Every summit a 'core reviewer dinner' is held that underscores the glorification of this designation. By proposing to rename 'core reviewer' to 'maintainer' the goal was to lay the groundwork for broadening the base of people whose valuable contribution could be recognized. The goal was to recognize not just review-related contributors, but also roles like doc/bug/test czar and cross-project liaison. The statue of the people filling these roles today is less if they are not also ‘core’, and that makes the work less attractive to many. Given the TC's apparent mandate to define the organizational taxonomy that a project like Neutron is allowed to use, I would ask you and your fellow committee members to consider addressing the role that the current taxonomy plays in valuing reviewing ahead of other forms of contribution. It provides disincentive against other forms of contribution, since they aren’t recognized on an equal footing, and I think this needs to change if we want to ensure the long-term viability of projects like Neutron (if not OpenStack as a whole). Maru __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] What do people think of YAPF (like gofmt, for python)?
On Mar 25, 2015, at 4:22 PM, Monty Taylor mord...@inaugust.com wrote: On 03/25/2015 05:50 PM, Maru Newby wrote: I am excited by the release of YAPF [1], a gofmt-like too for python. I think it has the potential to simplify style enforcement, and as much as I appreciate our current hacking checks, I’d be much happier not requiring developers to manually conform to them. Maybe we can consider automation in a manner similar to that employed by the go codereview tool [2]? I played with it for a few minutes and although it's configurable, it's still pretty limited in terms of expressiveness. That said - although I do appreciate the theory of auto-formatting (seriously, one less thing, right?) I think it would be problematic for us. You can't ship git hooks in a git repo, so we can't help our users know to run it before pushing. In a world where getting set up in openstack is already non-trivial, I think requiring 2500 developers to add a new git hook to every repo that they do something with would be a bit of a disaster. When you combine that with the people who won't know, will make a patch, send it up, and have it rejected --- oy. Chaos. git review is used by a ton of people who write in non-python. I think adding openstack-specific style enforcement to it would make it way less generally useful. In general, if you find it interesting, there's certainly nothing wrong with tracking it and poking at it from time to time. But I honestly think that the logistical issue is pretty large, and one that the payoff from solving would not be worth the effort. I agree with points raised by you and jeblair regarding the potential for problems in applying auto-formatting without developer intervention. I apologize if my use of the word 'automation' muddied the waters, as the go codereview tool's support of the 'gofmt' command is limited to explicit invocation rather than being triggered in a hook. It doesn't appear acceptable for git-review to be granted similar capability, but really that is a nicety. Individual projects can as easily provide their own style definitions, include YAPF as a test dependency, and document how to invoke it. I think there is value in extending our current solution of automated style enforcement to provide a component that can be manually invoked to auto-format to the enforced style. I don't see virtue in applying manual effort to maintaining code style. YAPF may not be ready for our use just yet, but I think it's worth pursuing. Maru __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [infra] What do people think of YAPF (like gofmt, for python)?
I am excited by the release of YAPF [1], a gofmt-like too for python. I think it has the potential to simplify style enforcement, and as much as I appreciate our current hacking checks, I’d be much happier not requiring developers to manually conform to them. Maybe we can consider automation in a manner similar to that employed by the go codereview tool [2]? Cheers, Maru 1: https://github.com/google/yapf 2: https://godoc.org/golang.org/x/review/git-codereview#hdr-Gofmt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa][neutron] Moving network api test development to Neutron repo
tl;dr; As per a discussion in Paris [1], development of Neutron's API tests is moving from the Tempest repo to the Neutron repo. If you are developing API tests for Neutron in Tempest, please be advised that, effective immediately, your efforts should be directed towards the Neutron repo. The current set of Neutron API tests in Tempest has been copied (along with supporting infrastructure) to neutron/tests/tempest [1]. Tests in this path are run as part of the neutron-dsvm-api job, which will shortly be gating [2]. Test changes that previously targeted the tempest/api/network path in the Tempest repo should target neutron/tests/tempest/network/api in the Neutron repo until further notice. Automatic conversion from a Tempest change to a Neutron change is possible: - cd [path to neutron repo] - ./tools/copy_api_tests_from_tempest.sh [path to tempest working directory] As per the Tempest guidelines for test removal [3], the tests currently in Tempest will remain in Tempest and continue to run as part of the existing jobs until we can target tests in the Neutron repo against stable branches and enable the use of the in-repo tests by defcore/refstack. Finally, guidelines for API test development in the Neutron repo are in the works and will be proposed shortly. The guidelines will define policy intended to protect against backwards compatible changes to our API. Thanks, Maru 1: https://etherpad.openstack.org/p/kilo-crossproject-move-func-tests-to-projects 2: https://github.com/openstack/neutron/tree/master/neutron/tests/tempest 3: https://review.openstack.org/#/c/164886 4: https://wiki.openstack.org/wiki/QA/Tempest-test-removal __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Proposal to add Ihar Hrachyshka as a Neutron Core Reviewer
+1 from me, Ihar has been doing great work and it will be great to have him finally able to merge! On Mar 4, 2015, at 11:42 AM, Kyle Mestery mest...@mestery.com wrote: I'd like to propose that we add Ihar Hrachyshka to the Neutron core reviewer team. Ihar has been doing a great job reviewing in Neutron as evidence by his stats [1]. Ihar is the Oslo liaison for Neutron, he's been doing a great job keeping Neutron current there. He's already a critical reviewer for all the Neutron repositories. In addition, he's a stable maintainer. Ihar makes himself available in IRC, and has done a great job working with the entire Neutron team. His reviews are thoughtful and he really takes time to work with code submitters to ensure his feedback is addressed. I'd also like to again remind everyone that reviewing code is a responsibility, in Neutron the same as other projects. And core reviewers are especially beholden to this responsibility. I'd also like to point out and reinforce that +1/-1 reviews are super useful, and I encourage everyone to continue reviewing code across Neutron as well as the other OpenStack projects, regardless of your status as a core reviewer on these projects. Existing Neutron cores, please vote +1/-1 on this proposal to add Ihar to the core reviewer team. Thanks! Kyle [1] http://stackalytics.com/report/contribution/neutron-group/90 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron] The future of Xen + OVS support
There have been a couple of patches [1] [2] proposed to neutron recently that could impact our support for Xen+OVS, but I don’t see an easy way for us to validate such changes. I’m not aware of a 3rd party CI job that runs against Xen, and I don’t know of any active contributors able to manually validate changes. Unless this changes, I think we should consider deprecating support for Xen in kilo. We can’t afford to block changes for fear of breaking something that nobody seems to care about. I’m hoping this post serves as a wake-up call to anyone that wants to see neutron maintain its support for Xen, and that they will be willing to devote resources to ensure that support for it continues. I’ve also added this issue to next week’s irc meeting [3]. Maru 1: https://review.openstack.org/#/c/148969/ 2: https://review.openstack.org/#/c/158805 3: https://wiki.openstack.org/wiki/Network/Meetings#On_Demand_Agenda __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] The future of Xen + OVS support
On Mar 3, 2015, at 1:06 PM, Kevin Benton blak...@gmail.com wrote: It might make sense to send this to the users list as well. There may be a large deployment out there with resources willing to at least test changes even if they don't have any upstream development resources. Good call, I’ve cross-posted to the user list as you suggest. On Tue, Mar 3, 2015 at 12:50 PM, Maru Newby ma...@redhat.com wrote: There have been a couple of patches [1] [2] proposed to neutron recently that could impact our support for Xen+OVS, but I don’t see an easy way for us to validate such changes. I’m not aware of a 3rd party CI job that runs against Xen, and I don’t know of any active contributors able to manually validate changes. Unless this changes, I think we should consider deprecating support for Xen in kilo. We can’t afford to block changes for fear of breaking something that nobody seems to care about. I’m hoping this post serves as a wake-up call to anyone that wants to see neutron maintain its support for Xen, and that they will be willing to devote resources to ensure that support for it continues. I’ve also added this issue to next week’s irc meeting [3]. Maru 1: https://review.openstack.org/#/c/148969/ 2: https://review.openstack.org/#/c/158805 3: https://wiki.openstack.org/wiki/Network/Meetings#On_Demand_Agenda __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] The future of Xen + OVS support
On Mar 3, 2015, at 1:44 PM, Bob Ball bob.b...@citrix.com wrote: We certainly care about this but converting the XenServer CI to Neutron+OVS has been hitting a number of issues (notably a few concurrency ones – although there are a few missing features) that we’ve been trying to sort through. I am certainly hoping that we’ll have everything stable enough to set up a CI specifically for this combination before the Kilo release and would be very keen not to deprecate XenAPI support in Kilo. Certainly even if we don’t make the Kilo release for the CI we will still be pushing for it in L. Bob I guess that answers the question of whether anyone cares enough to work on enabling 3rd party CI, and that it may be premature to consider deprecation. Until CI is live, though, and in the absence of resources committed to fixing breakage when it is detected, I’m not sure it’s reasonable that core reviewers have to consider Xen support in determining whether a patch is ready to merge. Given that, it’s entirely possible that we’ll end up shipping broken Xen support for kilo, especially given the changes coming to rootwrap and ovs interaction. Should we as a community consider that acceptable with the hope that fixes will be proposed and backported in a timely-enough fashion? Or maybe we should consider moving Xen support (which is centered on the ovs agent) out of the tree so that it can be maintained independent of the OpenStack release cycle? Maru From: Kevin Benton [mailto:blak...@gmail.com] Sent: 03 March 2015 21:06 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] The future of Xen + OVS support It might make sense to send this to the users list as well. There may be a large deployment out there with resources willing to at least test changes even if they don't have any upstream development resources. On Tue, Mar 3, 2015 at 12:50 PM, Maru Newby ma...@redhat.com wrote: There have been a couple of patches [1] [2] proposed to neutron recently that could impact our support for Xen+OVS, but I don’t see an easy way for us to validate such changes. I’m not aware of a 3rd party CI job that runs against Xen, and I don’t know of any active contributors able to manually validate changes. Unless this changes, I think we should consider deprecating support for Xen in kilo. We can’t afford to block changes for fear of breaking something that nobody seems to care about. I’m hoping this post serves as a wake-up call to anyone that wants to see neutron maintain its support for Xen, and that they will be willing to devote resources to ensure that support for it continues. I’ve also added this issue to next week’s irc meeting [3]. Maru 1: https://review.openstack.org/#/c/148969/ 2: https://review.openstack.org/#/c/158805 3: https://wiki.openstack.org/wiki/Network/Meetings#On_Demand_Agenda __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Kevin Benton __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Changes to the core team
+1 to Doug’s addition. He’s been a workhorse in moving lbaas and the services split forward and his being able to finally merge code in the main repo should be a boon to our merge velocity. Many thanks to Sumit for his long-serving dedication as a core reviewer. I hope to see him return to the fold soon! Maru On Jan 15, 2015, at 2:31 PM, Kyle Mestery mest...@mestery.com wrote: The last time we looked at core reviewer stats was in December [1]. In looking at the current stats, I'm going to propose some changes to the core team. Reviews are the most important part of being a core reviewer, so we need to ensure cores are doing reviews. The stats for the 90 day period [2] indicate some changes are needed for core reviewers who are no longer reviewing on pace with the other core reviewers. First of all, I'm removing Sumit Naiksatam from neutron-core. Sumit has been a core reviewer for a long time, and his past contributions are very much thanked by the entire OpenStack Neutron team. If Sumit jumps back in with thoughtful reviews in the future, we can look at getting him back as a Neutron core reviewer. But for now, his stats indicate he's not reviewing at a level consistent with the rest of the Neutron core reviewers. As part of the change, I'd like to propose Doug Wiegley as a new Neutron core reviewer. Doug has been actively reviewing code across not only all the Neutron projects, but also other projects such as infra. His help and work in the services split in December were the reason we were so successful in making that happen. Doug has also been instrumental in the Neutron LBaaS V2 rollout, as well as helping to merge code in the other neutron service repositories. I'd also like to take this time to remind everyone that reviewing code is a responsibility, in Neutron the same as other projects. And core reviewers are especially beholden to this responsibility. I'd also like to point out that +1/-1 reviews are very useful, and I encourage everyone to continue reviewing code even if you are not a core reviewer. Existing neutron cores, please vote +1/-1 for the addition of Doug to the core team. Thanks! Kyle [1] http://lists.openstack.org/pipermail/openstack-dev/2014-December/051986.html [2] http://russellbryant.net/openstack-stats/neutron-reviewers-90.txt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The state of nova-network to neutron migration
As per a recent exchange on #openstack-neutron, I’ve been asked to present my views on this effort. What follows is in no way intended to detract from the hard work and dedication of those undertaking it, but I think that their energy could be better spent. At nova’s juno mid-cycle in July, there was a discussion about deprecating nova-network. Most of the work-items on the TC’s gap analysis [1] had been covered off, with one notable exception: Gap 6, the requirement to provide a migration plan between nova-network and neutron, had stalled over questions of implementation strategy. In my recollection of the conversation that followed, broad consensus was reached that the costs of automating a reliable and fault-tolerant migration strategy would be considerable. The technical complexity of targeting a fixed deployment scenario would be challenging enough, and targeting heterogenous scenarios would magnify that complexity many-fold. Given the cost and high risks associated with implementing an automated solution, everyone seemed to agree that it was not worth pursuing. Our understanding was that not pursuing an automated solution could still be in keeping with the TC’s requirements for deprecation, which required that a migration plan be formulated but not that it be automated. Documentation was deemed sufficient, and that was to be the path forward in covering Gap 6. The documentation would allow deployers and operators to devise migration strategies to suit their individual requirements. Then, when the Kilo summit schedule was announced, there was a session scheduled in the nova track for discussing how to implement an automated migration. I only managed to catch the tail end of the session, but the etherpad [2] makes no mention of the rationale for requiring an automated migration in the first place. It was like the discussion at the mid-cycle, and all the talk of the risks outweighing the potential benefits of such an effort, had simply not occurred. So, in the interests of a full and open discussion on this matter, can someone please explain to me why the risks discussed at the mid-cycle were suddenly deemed justifiable, seemingly against all technical rationale? Criticism has been leveled at the neutron project for our supposed inaction in implementing an automated solution, and I don’t think I’m the only one who is concerned that this is an unreasonable requirement imposed without due consideration to the risks involved. Yes, most of us want to see nova-network deprecated, but why is the lack of migration automation blocking that? An automated migration was not a requirement in the TC’s original assessment of the preconditions for deprecation. I think that if neutron is deemed to be of sufficiently high quality that it can serve as an effective replacement for nova-network, and we can document a migration plan between them, then deprecation should proceed. Maru 1: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee/Neutron_Gap_Coverage 2: https://etherpad.openstack.org/p/kilo-nova-nova-network-to-neutron On Dec 19, 2014, at 8:59 AM, Anita Kuno ante...@anteaya.info wrote: Rather than waste your time making excuses let me state where we are and where I would like to get to, also sharing my thoughts about how you can get involved if you want to see this happen as badly as I have been told you do. Where we are: * a great deal of foundation work has been accomplished to achieve parity with nova-network and neutron to the extent that those involved are ready for migration plans to be formulated and be put in place * a summit session happened with notes and intentions[0] * people took responsibility and promptly got swamped with other responsibilities * spec deadlines arose and in neutron's case have passed * currently a neutron spec [1] is a work in progress (and it needs significant work still) and a nova spec is required and doesn't have a first draft or a champion Where I would like to get to: * I need people in addition to Oleg Bondarev to be available to help come up with ideas and words to describe them to create the specs in a very short amount of time (Oleg is doing great work and is a fabulous person, yay Oleg, he just can't do this alone) * specifically I need a contact on the nova side of this complex problem, similar to Oleg on the neutron side * we need to have a way for people involved with this effort to find each other, talk to each other and track progress * we need to have representation at both nova and neutron weekly meetings to communicate status and needs We are at K-2 and our current status is insufficient to expect this work will be accomplished by the end of K-3. I will be championing this work, in whatever state, so at least it doesn't fall off the map. If you would like to help this effort please get in contact. I will be thinking of ways to
Re: [openstack-dev] The state of nova-network to neutron migration
On Jan 8, 2015, at 3:54 PM, Sean Dague s...@dague.net wrote: On 01/08/2015 06:41 PM, Maru Newby wrote: As per a recent exchange on #openstack-neutron, I’ve been asked to present my views on this effort. What follows is in no way intended to detract from the hard work and dedication of those undertaking it, but I think that their energy could be better spent. At nova’s juno mid-cycle in July, there was a discussion about deprecating nova-network. Most of the work-items on the TC’s gap analysis [1] had been covered off, with one notable exception: Gap 6, the requirement to provide a migration plan between nova-network and neutron, had stalled over questions of implementation strategy. In my recollection of the conversation that followed, broad consensus was reached that the costs of automating a reliable and fault-tolerant migration strategy would be considerable. The technical complexity of targeting a fixed deployment scenario would be challenging enough, and targeting heterogenous scenarios would magnify that complexity many-fold. Given the cost and high risks associated with implementing an automated solution, everyone seemed to agree that it was not worth pursuing. Our understanding was that not pursuing an automated solution could still be in keeping with the TC’s requirements for deprecation, which required that a migration plan be formulated but not that it be automated. Documentation was deemed sufficient, and that was to be the path forward in covering Gap 6. The documentation would allow deployers and operators to devise migration strategies to suit their individual requirements. Then, when the Kilo summit schedule was announced, there was a session scheduled in the nova track for discussing how to implement an automated migration. I only managed to catch the tail end of the session, but the etherpad [2] makes no mention of the rationale for requiring an automated migration in the first place. It was like the discussion at the mid-cycle, and all the talk of the risks outweighing the potential benefits of such an effort, had simply not occurred. So, in the interests of a full and open discussion on this matter, can someone please explain to me why the risks discussed at the mid-cycle were suddenly deemed justifiable, seemingly against all technical rationale? Criticism has been leveled at the neutron project for our supposed inaction in implementing an automated solution, and I don’t think I’m the only one who is concerned that this is an unreasonable requirement imposed without due consideration to the risks involved. Yes, most of us want to see nova-network deprecated, but why is the lack of migration automation blocking that? An automated migration was not a requirement in the TC’s original assessment of the preconditions for deprecation. I think that if neutron is deemed to be of sufficiently high quality that it can serve as an effective replacement for nova-network, and we can document a migration plan between them, then deprecation should proceed. Maru The crux of it comes from the fact that the operator voice (especially those folks with large nova-network deploys) wasn't represented there. Once we got back from the mid-cycle and brought it to the list, there was some very understandable push back on deprecating without a migration plan. I think it’s clear that a migration plan is required. An automated migration, not so much. I believe we landed at the need for the common case, flat multi host networking, to be migrated to something equivalent in neutron land (dvr?). And it needs to be something that Metacloud and CERN can get behind, as they represent 2 very large nova-network deploys (and have reasonably well defined down time allowances for this). This doesn't have to be automation for all cases, but we need to support a happy path from one to the other that's repeatable, reasonably automatic (as much as possible), and provides minimum down time for guests running on the environment. The fact that operators running nova-network would like the upstream community to pay for implementing an automated migration solution for them is hardly surprising. It is less clear to me that implementing such a solution, with all the attendant cost and risks, should take priority over efforts that benefit a broader swath of the community. Are the operators in question so strapped for resources that they are not able to automate their migrations themselves, provided a sufficiently detailed plan to do so? Maru ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] UniqueConstraint for name and tenant_id in security group
On Dec 12, 2014, at 1:40 PM, Sean Dague s...@dague.net wrote: On 12/12/2014 01:05 PM, Maru Newby wrote: On Dec 11, 2014, at 2:27 PM, Sean Dague s...@dague.net wrote: On 12/11/2014 04:16 PM, Jay Pipes wrote: On 12/11/2014 04:07 PM, Vishvananda Ishaya wrote: On Dec 11, 2014, at 1:04 PM, Jay Pipes jaypi...@gmail.com wrote: On 12/11/2014 04:01 PM, Vishvananda Ishaya wrote: On Dec 11, 2014, at 8:00 AM, Henry Gessau ges...@cisco.com wrote: On Thu, Dec 11, 2014, Mark McClain m...@mcclain.xyz wrote: On Dec 11, 2014, at 8:43 AM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: I'm generally in favor of making name attributes opaque, utf-8 strings that are entirely user-defined and have no constraints on them. I consider the name to be just a tag that the user places on some resource. It is the resource's ID that is unique. I do realize that Nova takes a different approach to *some* resources, including the security group name. End of the day, it's probably just a personal preference whether names should be unique to a tenant/user or not. Maru had asked me my opinion on whether names should be unique and I answered my personal opinion that no, they should not be, and if Neutron needed to ensure that there was one and only one default security group for a tenant, that a way to accomplish such a thing in a race-free way, without use of SELECT FOR UPDATE, was to use the approach I put into the pastebin on the review above. I agree with Jay. We should not care about how a user names the resource. There other ways to prevent this race and Jay’s suggestion is a good one. However we should open a bug against Horizon because the user experience there is terrible with duplicate security group names. The reason security group names are unique is that the ec2 api supports source rule specifications by tenant_id (user_id in amazon) and name, so not enforcing uniqueness means that invocation in the ec2 api will either fail or be non-deterministic in some way. So we should couple our API evolution to EC2 API then? -jay No I was just pointing out the historical reason for uniqueness, and hopefully encouraging someone to find the best behavior for the ec2 api if we are going to keep the incompatibility there. Also I personally feel the ux is better with unique names, but it is only a slight preference. Sorry for snapping, you made a fair point. Yeh, honestly, I agree with Vish. I do feel that the UX of that constraint is useful. Otherwise you get into having to show people UUIDs in a lot more places. While those are good for consistency, they are kind of terrible to show to people. While there is a good case for the UX of unique names - it also makes orchestration via tools like puppet a heck of a lot simpler - the fact is that most OpenStack resources do not require unique names. That being the case, why would we want security groups to deviate from this convention? Maybe the other ones are the broken ones? Honestly, any sanely usable system makes names unique inside a container. Like files in a directory. In this case the tenant is the container, which makes sense. It is one of many places that OpenStack is not consistent. But I'd rather make things consistent and more usable than consistent and less. You might prefer less consistency for the sake of usability, but for me, consistency is a large enough factor in usability that allowing seemingly arbitrary deviation doesn’t seem like a huge win. Regardless, I’d like to see us came to decisions on API usability on an OpenStack-wide basis, so the API working group is probably where this discussion should continue. Maru ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] UniqueConstraint for name and tenant_id in security group
On Dec 15, 2014, at 9:13 AM, Assaf Muller amul...@redhat.com wrote: - Original Message - -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 I was (rightfully) asked to share my comments on the matter that I left in gerrit here. See below. On 12/12/14 22:40, Sean Dague wrote: On 12/12/2014 01:05 PM, Maru Newby wrote: On Dec 11, 2014, at 2:27 PM, Sean Dague s...@dague.net wrote: On 12/11/2014 04:16 PM, Jay Pipes wrote: On 12/11/2014 04:07 PM, Vishvananda Ishaya wrote: On Dec 11, 2014, at 1:04 PM, Jay Pipes jaypi...@gmail.com wrote: On 12/11/2014 04:01 PM, Vishvananda Ishaya wrote: On Dec 11, 2014, at 8:00 AM, Henry Gessau ges...@cisco.com wrote: On Thu, Dec 11, 2014, Mark McClain m...@mcclain.xyz wrote: On Dec 11, 2014, at 8:43 AM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: I'm generally in favor of making name attributes opaque, utf-8 strings that are entirely user-defined and have no constraints on them. I consider the name to be just a tag that the user places on some resource. It is the resource's ID that is unique. I do realize that Nova takes a different approach to *some* resources, including the security group name. End of the day, it's probably just a personal preference whether names should be unique to a tenant/user or not. Maru had asked me my opinion on whether names should be unique and I answered my personal opinion that no, they should not be, and if Neutron needed to ensure that there was one and only one default security group for a tenant, that a way to accomplish such a thing in a race-free way, without use of SELECT FOR UPDATE, was to use the approach I put into the pastebin on the review above. I agree with Jay. We should not care about how a user names the resource. There other ways to prevent this race and Jay’s suggestion is a good one. However we should open a bug against Horizon because the user experience there is terrible with duplicate security group names. The reason security group names are unique is that the ec2 api supports source rule specifications by tenant_id (user_id in amazon) and name, so not enforcing uniqueness means that invocation in the ec2 api will either fail or be non-deterministic in some way. So we should couple our API evolution to EC2 API then? -jay No I was just pointing out the historical reason for uniqueness, and hopefully encouraging someone to find the best behavior for the ec2 api if we are going to keep the incompatibility there. Also I personally feel the ux is better with unique names, but it is only a slight preference. Sorry for snapping, you made a fair point. Yeh, honestly, I agree with Vish. I do feel that the UX of that constraint is useful. Otherwise you get into having to show people UUIDs in a lot more places. While those are good for consistency, they are kind of terrible to show to people. While there is a good case for the UX of unique names - it also makes orchestration via tools like puppet a heck of a lot simpler - the fact is that most OpenStack resources do not require unique names. That being the case, why would we want security groups to deviate from this convention? Maybe the other ones are the broken ones? Honestly, any sanely usable system makes names unique inside a container. Like files in a directory. Correct. Or take git: it does not use hashes to identify objects, right? In this case the tenant is the container, which makes sense. It is one of many places that OpenStack is not consistent. But I'd rather make things consistent and more usable than consistent and less. Are we only proposing to make security group name unique? I assume that, since that's what we currently have in review. The change would make API *more* inconsistent, not less, since other objects still use uuid for identification. You may say that we should move *all* neutron objects to the new identification system by name. But what's the real benefit? If there are problems in UX (client, horizon, ...), we should fix the view and not data models used. If we decide we want users to avoid using objects with the same names, fine, let's add warnings in UI (probably with an option to disable it so that we don't push the validation into their throats). Finally, I have concern about us changing user visible object attributes like names during db migrations, as it's proposed in the patch discussed here. I think such behaviour can be quite unexpected for some users, if not breaking their workflow and/or scripts. My belief is that responsible upstream does not apply ad-hoc changes to API to fix a race condition that is easily solvable in other ways (see Assaf's proposal to introduce a new DefaultSecurityGroups table in patchset 12 comments). As usual you explain yourself better than I can... I think my main original objection to the patch is that it feels like
Re: [openstack-dev] [Neutron] UniqueConstraint for name and tenant_id in security group
On Dec 11, 2014, at 2:27 PM, Sean Dague s...@dague.net wrote: On 12/11/2014 04:16 PM, Jay Pipes wrote: On 12/11/2014 04:07 PM, Vishvananda Ishaya wrote: On Dec 11, 2014, at 1:04 PM, Jay Pipes jaypi...@gmail.com wrote: On 12/11/2014 04:01 PM, Vishvananda Ishaya wrote: On Dec 11, 2014, at 8:00 AM, Henry Gessau ges...@cisco.com wrote: On Thu, Dec 11, 2014, Mark McClain m...@mcclain.xyz wrote: On Dec 11, 2014, at 8:43 AM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: I'm generally in favor of making name attributes opaque, utf-8 strings that are entirely user-defined and have no constraints on them. I consider the name to be just a tag that the user places on some resource. It is the resource's ID that is unique. I do realize that Nova takes a different approach to *some* resources, including the security group name. End of the day, it's probably just a personal preference whether names should be unique to a tenant/user or not. Maru had asked me my opinion on whether names should be unique and I answered my personal opinion that no, they should not be, and if Neutron needed to ensure that there was one and only one default security group for a tenant, that a way to accomplish such a thing in a race-free way, without use of SELECT FOR UPDATE, was to use the approach I put into the pastebin on the review above. I agree with Jay. We should not care about how a user names the resource. There other ways to prevent this race and Jay’s suggestion is a good one. However we should open a bug against Horizon because the user experience there is terrible with duplicate security group names. The reason security group names are unique is that the ec2 api supports source rule specifications by tenant_id (user_id in amazon) and name, so not enforcing uniqueness means that invocation in the ec2 api will either fail or be non-deterministic in some way. So we should couple our API evolution to EC2 API then? -jay No I was just pointing out the historical reason for uniqueness, and hopefully encouraging someone to find the best behavior for the ec2 api if we are going to keep the incompatibility there. Also I personally feel the ux is better with unique names, but it is only a slight preference. Sorry for snapping, you made a fair point. Yeh, honestly, I agree with Vish. I do feel that the UX of that constraint is useful. Otherwise you get into having to show people UUIDs in a lot more places. While those are good for consistency, they are kind of terrible to show to people. While there is a good case for the UX of unique names - it also makes orchestration via tools like puppet a heck of a lot simpler - the fact is that most OpenStack resources do not require unique names. That being the case, why would we want security groups to deviate from this convention? Maru ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Neutron] out-of-tree plugin for Mech driver/L2 and vif_driver
On Dec 9, 2014, at 7:04 AM, Daniel P. Berrange berra...@redhat.com wrote: On Tue, Dec 09, 2014 at 10:53:19AM +0100, Maxime Leroy wrote: I have also proposed a blueprint to have a new plugin mechanism in nova to load external vif driver. (nova-specs: https://review.openstack.org/#/c/136827/ and nova (rfc patch): https://review.openstack.org/#/c/136857/) From my point-of-view of a developer having a plugin framework for internal/external vif driver seems to be a good thing. It makes the code more modular and introduce a clear api for vif driver classes. So far, it raises legitimate questions concerning API stability and public API that request a wider discussion on the ML (as asking by John Garbut). I think having a plugin mechanism and a clear api for vif driver is not going against this policy: http://docs.openstack.org/developer/nova/devref/policies.html#out-of-tree-support. There is no needs to have a stable API. It is up to the owner of the external VIF driver to ensure that its driver is supported by the latest API. And not the nova community to manage a stable API for this external VIF driver. Does it make senses ? Experiance has shown that even if it is documented as unsupported, once the extension point exists, vendors users will ignore the small print about support status. There will be complaints raised every time it gets broken until we end up being forced to maintain it as stable API whether we want to or not. That's not a route we want to go down. Is the support contract for a given API have to be binary - ‘supported’ vs ‘unsupported’? The stability requirements for REST API’s that end users and all kinds of tooling consume are arguably different from those of an internal API, and recognizing this difference could be useful. Considering the network V2 API, L2/ML2 mechanism driver and VIF driver need to exchange information such as: binding:vif_type and binding:vif_details. From my understanding, 'binding:vif_type' and 'binding:vif_details' as a field part of the public network api. There is no validation constraints for these fields (see http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html), it means that any value is accepted by the API. So, the values set in 'binding:vif_type' and 'binding:vif_details' are not part of the public API. Is my understanding correct ? The VIF parameters are mapped into the nova.network.model.VIF class which is doing some crude validation. I would anticipate that this validation will be increasing over time, because any functional data flowing over the API and so needs to be carefully managed for upgrade reasons. Even if the Neutron impl is out of tree, I would still expect both Nova and Neutron core to sign off on any new VIF type name and its associated details (if any). What other reasons am I missing to not have VIF driver classes as a public extension point ? Having to find install VIF driver classes from countless different vendors, each hiding their code away in their own obsecure website, will lead to awful end user experiance when deploying Nova. Users are better served by having it all provided when they deploy Nova IMHO Shipping drivers in-tree makes sense for a purely open source solution for the reasons you mention. The logic doesn’t necessarily extend to deployment of a proprietary solution, though. If a given OpenStack deployment is intended to integrate with such a solution, it is likely that the distro/operator/deployer will have a direct relationship with the solution provider and the required software (including VIF driver(s), if necessary) is likely to have a well-defined distribution channel. If every vendor goes off works in their own isolated world we also loose the scope to align the implementations, so that common concepts work the same way in all cases and allow us to minimize the number of new VIF types required. The proposed vhostuser VIF type is a good example of this - it allows a single Nova VIF driver to be capable of potentially supporting multiple different impls on the Neutron side. If every vendor worked in their own world, we would have ended up with multiple VIF drivers doing the same thing in Nova, each with their own set of bugs quirks. I’m not sure the suggestion is that every vendor go off and do their own thing. Rather, the option for out-of-tree drivers could be made available to those that are pursuing initiatives that aren’t found to be in-keeping with Nova’s current priorities. I believe that allowing out-of-tree extensions is essential to ensuring the long-term viability of OpenStack. There is only so much experimental work that is going to be acceptable in core OpenStack projects, if only to ensure stability. Yes, there is the potential for duplicative effort with results of varying quality, but that’s the price of competitive innovation whether in the field of ideas or
Re: [openstack-dev] [Neutron] Core/Vendor code decomposition
On Dec 7, 2014, at 10:51 AM, Gary Kotton gkot...@vmware.com wrote: Hi Kyle, I am not missing the point. I understand the proposal. I just think that it has some shortcomings (unless I misunderstand, which will certainly not be the first time and most definitely not the last). The thinning out is to have a shim in place. I understand this and this will be the entry point for the plugin. I do not have a concern for this. My concern is that we are not doing this with the ML2 off the bat. That should lead by example as it is our reference architecture. Lets not kid anyone, but we are going to hit some problems with the decomposition. I would prefer that it be done with the default implementation. Why? The proposal is to move vendor-specific logic out of the tree to increase vendor control over such code while decreasing load on reviewers. ML2 doesn’t contain vendor-specific logic - that’s the province of ML2 drivers - so it is not a good target for the proposed decomposition by itself. • Cause we will fix them quicker as it is something that prevent Neutron from moving forwards • We will just need to fix in one place first and not in N (where N is the vendor plugins) • This is a community effort – so we will have a lot more eyes on it • It will provide a reference architecture for all new plugins that want to be added to the tree • It will provide a working example for plugin that are already in tree and are to be replaced by the shim If we really want to do this, we can say freeze all development (which is just approvals for patches) for a few days so that we will can just focus on this. I stated what I think should be the process on the review. For those who do not feel like finding the link: • Create a stack forge project for ML2 • Create the shim in Neutron • Update devstack for the to use the two repos and the shim When #3 is up and running we switch for that to be the gate. Then we start a stopwatch on all other plugins. As was pointed out on the spec (see Miguel’s comment on r15), the ML2 plugin and the OVS mechanism driver need to remain in the main Neutron repo for now. Neutron gates on ML2+OVS and landing a breaking change in the Neutron repo along with its corresponding fix to a separate ML2 repo would be all but impossible under the current integrated gating scheme. Plugins/drivers that do not gate Neutron have no such constraint. Maru Sure, I’ll catch you on IRC tomorrow. I guess that you guys will bash out the details at the meetup. Sadly I will not be able to attend – so you will have to delay on the tar and feathers. Thanks Gary From: mest...@mestery.com mest...@mestery.com Reply-To: OpenStack List openstack-dev@lists.openstack.org Date: Sunday, December 7, 2014 at 7:19 PM To: OpenStack List openstack-dev@lists.openstack.org Cc: openst...@lists.openstack.org openst...@lists.openstack.org Subject: Re: [openstack-dev] [Neutron] Core/Vendor code decomposition Gary, you are still miss the point of this proposal. Please see my comments in review. We are not forcing things out of tree, we are thinning them. The text you quoted in the review makes that clear. We will look at further decomposing ML2 post Kilo, but we have to be realistic with what we can accomplish during Kilo. Find me on IRC Monday morning and we can discuss further if you still have questions and concerns. Thanks! Kyle On Sun, Dec 7, 2014 at 2:08 AM, Gary Kotton gkot...@vmware.com wrote: Hi, I have raised my concerns on the proposal. I think that all plugins should be treated on an equal footing. My main concern is having the ML2 plugin in tree whilst the others will be moved out of tree will be problematic. I think that the model will be complete if the ML2 was also out of tree. This will help crystalize the idea and make sure that the model works correctly. Thanks Gary From: Armando M. arma...@gmail.com Reply-To: OpenStack List openstack-dev@lists.openstack.org Date: Saturday, December 6, 2014 at 1:04 AM To: OpenStack List openstack-dev@lists.openstack.org, openst...@lists.openstack.org openst...@lists.openstack.org Subject: [openstack-dev] [Neutron] Core/Vendor code decomposition Hi folks, For a few weeks now the Neutron team has worked tirelessly on [1]. This initiative stems from the fact that as the project matures, evolution of processes and contribution guidelines need to evolve with it. This is to ensure that the project can keep on thriving in order to meet the needs of an ever growing community. The effort of documenting intentions, and fleshing out the various details of the proposal is about to reach an end, and we'll soon kick the tires to put the proposal into practice. Since the spec has grown pretty big, I'll try to capture the tl;dr below. If you have any comment please do not hesitate to raise them here and/or
Re: [openstack-dev] [NFV] Patches tagging with NFVImpact
Am I the only one wondering whether introducing arbitrary tagging into our commit messages sets a bad precedent? Or was there a discussion on this topic that I missed? Maru On Jun 17, 2014, at 2:27 PM, Sylvain Bauza sba...@redhat.com wrote: Hi, There is an action for creating a Gerrit dashboard to show all NFV-related specs and patches. As it requires to have a specific label for discriminating them, I'm adding NFVImpact in all the patches proposed in [1]. The side effect is that it creates another patchset and so runs another Jenkins check so don't worry about it. I'll do a team status by tomorrow but meanwhile, please make sure that if you upload a new patch to Gerrit, it will be marked as NFVImpact in the commit message. Many thanks, -Sylvain [1] https://wiki.openstack.org/wiki/Teams/NFV#Active_Blueprints ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron]issue about dvr flows and port timestamp
On Oct 29, 2014, at 8:12 AM, Yangxurong yangxur...@huawei.com wrote: Hi, I’m not sure whether following issue is problematic, and both, our team do some effort, so I submit two blueprints: [1.] optimize dvr flows: Currently, accurate ovs flows in terms of full mac are used to communicate among distributed router, but here comes problem : (1)the more distributed router node, the more flows; (2)different distributed router across DC cannot communicate through tunnel and additional operation under same mac prefix configuration. So it would be useful to shift the complete-matching of mac to fuzzy Matching, like prefix matching, reducing the number of flows and leading to communicate among different DC though configuring same mac prefix through tunnel. Link: https://blueprints.launchpad.net/neutron/+spec/optimize-dvr-flows I think we need to focus on paying down technical debt (both in the code and on the testing side) related to dvr before we seriously consider the kind of optimization that you are proposing. I’m also unclear as to why we would want to pursue a solution to a problem whose severity doesn’t appear to be clear (I’m not sure whether the following issue is problematic…). Maru [2.]add port timestamp: It would be worth adding timestamp fields including create_at, update_at and delete_at in the table of port in neutron, so users can monitor port change conveniently, for example portal or management center wants to query the ports that have changed or refreshed during the latest 5sec in a large scale application. If not, it's time-consuming and low effectiveness. Link: https://blueprints.launchpad.net/neutron/+spec/add-port-timestamp Any response I will appreciate. Thanks, Xurong Yang ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [NFV] Patches tagging with NFVImpact
On Oct 29, 2014, at 12:25 PM, Steve Gordon sgor...@redhat.com wrote: - Original Message - From: Maru Newby ma...@redhat.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Am I the only one wondering whether introducing arbitrary tagging into our commit messages sets a bad precedent? Or was there a discussion on this topic that I missed? Maru Any particular reason for digging this up from June? We already changed the process so that we did not need to do this. Thanks, Steve I had a mail filter applied when I looked at the list this morning and I failed to notice that this was an old thread until I had already sent my reply. That said, we have Neutron specs up for review that still include this tag in the commit message and now I know at least that I can ask for their removal. Maru On Jun 17, 2014, at 2:27 PM, Sylvain Bauza sba...@redhat.com wrote: Hi, There is an action for creating a Gerrit dashboard to show all NFV-related specs and patches. As it requires to have a specific label for discriminating them, I'm adding NFVImpact in all the patches proposed in [1]. The side effect is that it creates another patchset and so runs another Jenkins check so don't worry about it. I'll do a team status by tomorrow but meanwhile, please make sure that if you upload a new patch to Gerrit, it will be marked as NFVImpact in the commit message. Many thanks, -Sylvain [1] https://wiki.openstack.org/wiki/Teams/NFV#Active_Blueprints ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Steve Gordon, RHCE Sr. Technical Product Manager, Red Hat Enterprise Linux OpenStack Platform ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron]issue about dvr flows and port timestamp
On Oct 29, 2014, at 1:36 PM, Hly henry4...@gmail.com wrote: Sent from my iPad On 2014-10-29, at 下午6:33, Maru Newby ma...@redhat.com wrote: On Oct 29, 2014, at 8:12 AM, Yangxurong yangxur...@huawei.com wrote: Hi, I’m not sure whether following issue is problematic, and both, our team do some effort, so I submit two blueprints: [1.] optimize dvr flows: Currently, accurate ovs flows in terms of full mac are used to communicate among distributed router, but here comes problem : (1)the more distributed router node, the more flows; (2)different distributed router across DC cannot communicate through tunnel and additional operation under same mac prefix configuration. So it would be useful to shift the complete-matching of mac to fuzzy Matching, like prefix matching, reducing the number of flows and leading to communicate among different DC though configuring same mac prefix through tunnel. Link: https://blueprints.launchpad.net/neutron/+spec/optimize-dvr-flows I think we need to focus on paying down technical debt (both in the code and on the testing side) related to dvr before we seriously consider the kind of optimization that you are proposing. I’m also unclear as to why we would want to pursue a solution to a problem whose severity doesn’t appear to be clear (I’m not sure whether the following issue is problematic…). DVR stability is the first class for sure, but if the code and logic could be less and simpler, there is more chance of stability. By my understanding, since DVR mac range has been configured as a prefix, so prefix based judgement instead of one by one flow setup triggered by mesh-like message notifying would simplify the code logic, thus indirectly contribute to overall stability. Also, it would remove hundreds of flows in the ovs in a middle scale cluster, very helpful for trouble shooting. If we’re going to refactor/optimize/whatever any part of Neutron, the first requirement is that we can maintain the expectations of the code (i.e. ensure good test coverage) across changes. DVR is no different from any other feature in this regard. I would welcome any effort you want to devote to improving DVR’s testing story such that the kind of optimizations you are proposing would have less potential to cause regressions. In addition to baseline test coverage, I would also expect to see test additions validating flow generation such that reviewers would be able to easily identify the benefit of your proposed optimizations by comparing the results before and after. Maru Wu Maru [2.]add port timestamp: It would be worth adding timestamp fields including create_at, update_at and delete_at in the table of port in neutron, so users can monitor port change conveniently, for example portal or management center wants to query the ports that have changed or refreshed during the latest 5sec in a large scale application. If not, it's time-consuming and low effectiveness. Link: https://blueprints.launchpad.net/neutron/+spec/add-port-timestamp Any response I will appreciate. Thanks, Xurong Yang ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][stable] metadata agent performance
On Oct 22, 2014, at 12:53 AM, Jakub Libosvar libos...@redhat.com wrote: On 10/22/2014 02:26 AM, Maru Newby wrote: We merged caching support for the metadata agent in juno, and backported to icehouse. It was enabled by default in juno, but disabled by default in icehouse to satisfy the stable maint requirement of not changing functional behavior. While performance of the agent was improved with caching enabled, it regressed a reported 8x when caching was disabled [1]. This means that by default, the caching backport severely impacts icehouse Neutron's performance. So, what is the way forward? We definitely need to document the problem for both icehouse and juno. Is documentation enough? Or can we enable caching by default in icehouse? Or remove the backport entirely. There is also a proposal to replace the metadata agent’s use of the neutron client in favor of rpc [2]. There were comments on an old bug suggesting we didn’t want to do this [3], but assuming that we want this change in Kilo, is backporting even a possibility given that it implies a behavioral change to be useful? Thanks, Maru 1: https://bugs.launchpad.net/cloud-archive/+bug/1361357 2: https://review.openstack.org/#/c/121782 3: https://bugs.launchpad.net/neutron/+bug/1092043 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I thought the performance regression was caused by wrong keystone token caching leading to authentication per neutron client instance. Fix was backported to Icehouse [1]. Does it mean this patch hasn't solved the problem and regression is somewhere else? As you say (and as per Ihar and Akihiro’s responses), the problem was fixed. I was confused by the bug that Oleg’s RPC proposal hard targeted as a related bug and thought the problem wasn’t yet resolved, but looking at it again it appears the bug is a Ubuntu distro issue. Accordingly, I’ve removed Neutron as a target for that bug. Apologies for the confusion. Maru Kuba [1] https://review.openstack.org/#/c/120418/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron][stable] metadata agent performance
We merged caching support for the metadata agent in juno, and backported to icehouse. It was enabled by default in juno, but disabled by default in icehouse to satisfy the stable maint requirement of not changing functional behavior. While performance of the agent was improved with caching enabled, it regressed a reported 8x when caching was disabled [1]. This means that by default, the caching backport severely impacts icehouse Neutron's performance. So, what is the way forward? We definitely need to document the problem for both icehouse and juno. Is documentation enough? Or can we enable caching by default in icehouse? Or remove the backport entirely. There is also a proposal to replace the metadata agent’s use of the neutron client in favor of rpc [2]. There were comments on an old bug suggesting we didn’t want to do this [3], but assuming that we want this change in Kilo, is backporting even a possibility given that it implies a behavioral change to be useful? Thanks, Maru 1: https://bugs.launchpad.net/cloud-archive/+bug/1361357 2: https://review.openstack.org/#/c/121782 3: https://bugs.launchpad.net/neutron/+bug/1092043 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron][db] Ensuring db isolation between tests
For legacy reasons, the Neutron test suite creates and destroys a db for each test. There is a patch proposed to create the tables once and then ensure the tables are wiped at the end of each test [1], providing a performance improvement of ~10%. I was wondering if this is the best way to provide isolation, since I’ve heard that isolation via per-test transactions should also work. The patch author reported problems with this approach - apparently nested commits were not being rolled back. Is there some trick to isolating with transactions that wouldn’t be immediately obvious? Thanks, Maru 1: https://review.openstack.org/#/c/122028/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Incubator concerns from packaging perspective
On Aug 26, 2014, at 5:06 PM, Pradeep Kilambi (pkilambi) pkila...@cisco.com wrote: On 8/26/14, 4:49 AM, Maru Newby ma...@redhat.com wrote: On Aug 25, 2014, at 4:39 PM, Pradeep Kilambi (pkilambi) pkila...@cisco.com wrote: On 8/23/14, 5:36 PM, Maru Newby ma...@redhat.com wrote: On Aug 23, 2014, at 4:06 AM, Sumit Naiksatam sumitnaiksa...@gmail.com wrote: On Thu, Aug 21, 2014 at 7:28 AM, Kyle Mestery mest...@mestery.com wrote: On Thu, Aug 21, 2014 at 5:12 AM, Ihar Hrachyshka ihrac...@redhat.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 20/08/14 18:28, Salvatore Orlando wrote: Some comments inline. Salvatore On 20 August 2014 17:38, Ihar Hrachyshka ihrac...@redhat.com mailto:ihrac...@redhat.com wrote: Hi all, I've read the proposal for incubator as described at [1], and I have several comments/concerns/suggestions to this. Overall, the idea of giving some space for experimentation that does not alienate parts of community from Neutron is good. In that way, we may relax review rules and quicken turnaround for preview features without loosing control on those features too much. Though the way it's to be implemented leaves several concerns, as follows: 1. From packaging perspective, having a separate repository and tarballs seems not optimal. As a packager, I would better deal with a single tarball instead of two. Meaning, it would be better to keep the code in the same tree. I know that we're afraid of shipping the code for which some users may expect the usual level of support and stability and compatibility. This can be solved by making it explicit that the incubated code is unsupported and used on your user's risk. 1) The experimental code wouldn't probably be installed unless explicitly requested, and 2) it would be put in a separate namespace (like 'preview', 'experimental', or 'staging', as the call it in Linux kernel world [2]). This would facilitate keeping commit history instead of loosing it during graduation. Yes, I know that people don't like to be called experimental or preview or incubator... And maybe neutron-labs repo sounds more appealing than an 'experimental' subtree in the core project. Well, there are lots of EXPERIMENTAL features in Linux kernel that we actively use (for example, btrfs is still considered experimental by Linux kernel devs, while being exposed as a supported option to RHEL7 users), so I don't see how that naming concern is significant. I think this is the whole point of the discussion around the incubator and the reason for which, to the best of my knowledge, no proposal has been accepted yet. I wonder where discussion around the proposal is running. Is it public? The discussion started out privately as the incubation proposal was put together, but it's now on the mailing list, in person, and in IRC meetings. Lets keep the discussion going on list now. In the spirit of keeping the discussion going, I think we probably need to iterate in practice on this idea a little bit before we can crystallize on the policy and process for this new repo. Here are few ideas on how we can start this iteration: * Namespace for the new repo: Should this be in the neutron namespace, or a completely different namespace like neutron labs? Perhaps creating a separate namespace will help the packagers to avoid issues of conflicting package owners of the namespace. I don¹t think there is a technical requirement to choose a new namespace. Python supports sharing a namespace, and packaging can support this feature (see: oslo.*). From what I understand there can be overlapping code between neutron and incubator to override/modify existing python/config files. In which case, packaging(for Eg: rpm) will raise a path conflict. So we probably will need to worry about namespaces? Doug's suggestion to use a separate namespace to indicate that the incubator codebase isn’t fully supported is a good idea and what I had in mind as a non-technical reason for a new namespace. I still assert that the potential for path conflicts can be avoided easily enough, and is not a good reason on its own to use a different namespace. * Dependency on Neutron (core) repository: We would need to sort this out so that we can get UTs to run and pass in the new repo. Can we set the dependency on Neutron milestone releases? We already publish tar balls for the milestone releases, but I am not sure we publish these as packages to pypi. If not could we start doing that? With this in place, the incubator would always lag the Neutron core by at the most one milestone release. Given that it is possible to specify a dependency as a branch/hash/tag in a git repo [1], I¹m not sure it¹s worth figuring out how to target tarballs. Master branch of the incubation repo could then target the master branch of the Neutron repo and always be assured of being
Re: [openstack-dev] [infra][qa][neutron] Neutron full job, advanced services, and the integrated gate
On Aug 27, 2014, at 1:47 AM, Salvatore Orlando sorla...@nicira.com wrote: TL; DR A few folks are proposing to stop running tests for neutron advanced services [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only on the neutron gate. Reason: projects like nova are 100% orthogonal to neutron advanced services. Also, there have been episodes in the past of unreliability of tests for these services, and it would be good to limit affected projects considering that more api tests and scenarios are being added. Given how many rechecks I’ve had to do to merge what are effectively no-op patches to infra/config, most often due to the full neutron job exhibiting sporadic failures, I fully support this change. I think we need time to stabilize the tests for advanced services against just neutron before we consider slowing down merges for other projects. - So far the neutron full job runs tests (api and scenarios) for neutron core functionality as well as neutron advanced services, which run as neutron service plugin. It's highly unlikely, if not impossible, that changes in projects such as nova, glance or ceilometer can have an impact on the stability of these services. On the other hand, instability in these services can trigger gate failures in unrelated projects as long as tests for these services are run in the neutron full job in the integrated gate. There have already been several gate-breaking bugs in lbaas scenario tests are firewall api tests. Admittedly, advanced services do not have the same level of coverage as core neutron functionality. Therefore as more tests are being added, there is an increased possibility of unearthing dormant bugs. For this reason we are proposing to not run anymore tests for neutron advanced services in the integrated gate, but keep them running on the neutron gate. This means we will have two neutron jobs: 1) check-tempest-dsvm-neutron-full which will run only core neutron functionality 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron full job is today. The former will be part of the integrated gate, the latter will be part of the neutron gate. Considering that other integrating services should not have an impact on neutron advanced services, this should not make gate testing asymmetric. However, there might be exceptions for: - orchestration project like heat which in the future might leverage capabilities like load balancing - oslo-* libraries, as changes in them might have an impact on neutron advanced services, since they consume those libraries Another good question is whether extended tests should be performed as part of functional or tempest checks. My take on this is that scenario tests should always be part of tempest. On the other hand I reckon API tests should exclusively be part of functional tests, but as so far tempest is running a gazillion of API tests, this is probably a discussion for the medium/long term. As you say, tempest should retain responsibility for ‘golden-path’ integration tests involving other OpenStack services (’scenario tests’). Everything else should eventually be in-tree, though the transition period to achieve this is likely to be multi-cycle. m. In order to add this new job there are a few patches under review: [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it. [3] Are the patches implementing a blueprint which will enable us to specify for which extensions test should be executed. Finally, one more note about smoketests. Although we're planning to get rid of them soon, we still have failures in the pg job because of [4]. For this reasons smoketests are still running for postgres in the integrated gate. As load balancing and firewall API tests are part of it, they should be removed from the smoke test executed on the integrated gate ([5], [6]). This is a temporary measure until the postgres issue is fixed. Regards, Salvatore [1] https://review.openstack.org/#/c/114933/ [2] https://review.openstack.org/#/c/114932/ [3] https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z [4] https://bugs.launchpad.net/nova/+bug/1305892 [5] https://review.openstack.org/#/c/115022/ [6] https://review.openstack.org/#/c/115023/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Incubator concerns from packaging perspective
On Aug 25, 2014, at 4:39 PM, Pradeep Kilambi (pkilambi) pkila...@cisco.com wrote: On 8/23/14, 5:36 PM, Maru Newby ma...@redhat.com wrote: On Aug 23, 2014, at 4:06 AM, Sumit Naiksatam sumitnaiksa...@gmail.com wrote: On Thu, Aug 21, 2014 at 7:28 AM, Kyle Mestery mest...@mestery.com wrote: On Thu, Aug 21, 2014 at 5:12 AM, Ihar Hrachyshka ihrac...@redhat.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 20/08/14 18:28, Salvatore Orlando wrote: Some comments inline. Salvatore On 20 August 2014 17:38, Ihar Hrachyshka ihrac...@redhat.com mailto:ihrac...@redhat.com wrote: Hi all, I've read the proposal for incubator as described at [1], and I have several comments/concerns/suggestions to this. Overall, the idea of giving some space for experimentation that does not alienate parts of community from Neutron is good. In that way, we may relax review rules and quicken turnaround for preview features without loosing control on those features too much. Though the way it's to be implemented leaves several concerns, as follows: 1. From packaging perspective, having a separate repository and tarballs seems not optimal. As a packager, I would better deal with a single tarball instead of two. Meaning, it would be better to keep the code in the same tree. I know that we're afraid of shipping the code for which some users may expect the usual level of support and stability and compatibility. This can be solved by making it explicit that the incubated code is unsupported and used on your user's risk. 1) The experimental code wouldn't probably be installed unless explicitly requested, and 2) it would be put in a separate namespace (like 'preview', 'experimental', or 'staging', as the call it in Linux kernel world [2]). This would facilitate keeping commit history instead of loosing it during graduation. Yes, I know that people don't like to be called experimental or preview or incubator... And maybe neutron-labs repo sounds more appealing than an 'experimental' subtree in the core project. Well, there are lots of EXPERIMENTAL features in Linux kernel that we actively use (for example, btrfs is still considered experimental by Linux kernel devs, while being exposed as a supported option to RHEL7 users), so I don't see how that naming concern is significant. I think this is the whole point of the discussion around the incubator and the reason for which, to the best of my knowledge, no proposal has been accepted yet. I wonder where discussion around the proposal is running. Is it public? The discussion started out privately as the incubation proposal was put together, but it's now on the mailing list, in person, and in IRC meetings. Lets keep the discussion going on list now. In the spirit of keeping the discussion going, I think we probably need to iterate in practice on this idea a little bit before we can crystallize on the policy and process for this new repo. Here are few ideas on how we can start this iteration: * Namespace for the new repo: Should this be in the neutron namespace, or a completely different namespace like neutron labs? Perhaps creating a separate namespace will help the packagers to avoid issues of conflicting package owners of the namespace. I don¹t think there is a technical requirement to choose a new namespace. Python supports sharing a namespace, and packaging can support this feature (see: oslo.*). From what I understand there can be overlapping code between neutron and incubator to override/modify existing python/config files. In which case, packaging(for Eg: rpm) will raise a path conflict. So we probably will need to worry about namespaces? Doug's suggestion to use a separate namespace to indicate that the incubator codebase isn’t fully supported is a good idea and what I had in mind as a non-technical reason for a new namespace. I still assert that the potential for path conflicts can be avoided easily enough, and is not a good reason on its own to use a different namespace. * Dependency on Neutron (core) repository: We would need to sort this out so that we can get UTs to run and pass in the new repo. Can we set the dependency on Neutron milestone releases? We already publish tar balls for the milestone releases, but I am not sure we publish these as packages to pypi. If not could we start doing that? With this in place, the incubator would always lag the Neutron core by at the most one milestone release. Given that it is possible to specify a dependency as a branch/hash/tag in a git repo [1], I¹m not sure it¹s worth figuring out how to target tarballs. Master branch of the incubation repo could then target the master branch of the Neutron repo and always be assured of being current, and then released versions could target milestone tags or released versions. 1: http://pip.readthedocs.org/en/latest/reference/pip_install.html#git
Re: [openstack-dev] [neutron][cisco] Cisco Nexus requires patched ncclient
On Aug 25, 2014, at 11:38 AM, Robert Collins robe...@robertcollins.net wrote: Fwiw I would like to see such dependencies listed so that we can properly install trunk via automation. Do you mean that you want the requirements for all plugins listed somewhere in the tree? Or specifically in Neutron’s requirements.txt? On 24 Aug 2014 10:43, Maru Newby ma...@redhat.com wrote: On Aug 14, 2014, at 1:55 PM, Ihar Hrachyshka ihrac...@redhat.com wrote: Signed PGP part FYI: I've uploaded a review for openstack/requirements to add the upstream module into the list of potential dependencies [1]. Once it's merged, I'm going to introduce this new requirement for Neutron. The only reason someone would install ncclient would be because they had a brocade or cisco solution that they wanted to integrate with and I don't think that justifies making Neutron depend on the library. Maru [1]: https://review.openstack.org/114213 /Ihar On 12/08/14 16:27, Ihar Hrachyshka wrote: Hey all, as per [1], Cisco Nexus ML2 plugin requires a patched version of ncclient from github. I wonder: - whether this information is still current; - why don't we depend on ncclient thru our requirements.txt file. [1]: https://wiki.openstack.org/wiki/Neutron/ML2/MechCiscoNexus Cheers, /Ihar ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][cisco] Cisco Nexus requires patched ncclient
On Aug 24, 2014, at 5:14 PM, Henry Gessau ges...@cisco.com wrote: Ihar Hrachyshka ihrac...@redhat.com wrote: Now, maybe putting the module into requirements.txt is an overkill (though I doubt it). In that case, we could be interested in getting the info in some other centralized way. Maru is of the opinion that it is overkill. I feel the same way, but I am not involved much with deployment issues so my feelings should not sway anyone. Note that ncclient is not the only package used by vendor solutions that is not listed in requirements.txt. The ones I am aware of are: ncclient (cisco and brocade) heleosapi (embrane) a10_neutron_lbaas (a10networks) Maybe we should start exploring some other centralized way to list these type of dependencies? I think each plugin should be able to have its own requirements.txt file to aid in manual deployment and in packaging of a given plugin. This suggests the need to maintain a global plugin requirements file (in the tree) to ensure use of only a single version of any dependencies used across more than one plugin. Given that 3rd party CI is likely having to install these dependencies anyway, I think it would be good to make this deployment reproducible while avoiding the need to add new dependencies to core Neutron. Maru ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron] Runtime checks vs Sanity checks
Kevin Benton has proposed adding a runtime check for netns permission problems: https://review.openstack.org/#/c/109736/ There seems to be consensus on the patch that this is something that we want to do at runtime, but that would seem to run counter to the precedent that host-specific issues such as this one be considered a deployment-time responsibility. The addition of the sanity check framework would seem to support the latter case: https://github.com/openstack/neutron/blob/master/neutron/cmd/sanity_check.py Thoughts? Maru ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Incubator concerns from packaging perspective
On Aug 23, 2014, at 4:06 AM, Sumit Naiksatam sumitnaiksa...@gmail.com wrote: On Thu, Aug 21, 2014 at 7:28 AM, Kyle Mestery mest...@mestery.com wrote: On Thu, Aug 21, 2014 at 5:12 AM, Ihar Hrachyshka ihrac...@redhat.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 20/08/14 18:28, Salvatore Orlando wrote: Some comments inline. Salvatore On 20 August 2014 17:38, Ihar Hrachyshka ihrac...@redhat.com mailto:ihrac...@redhat.com wrote: Hi all, I've read the proposal for incubator as described at [1], and I have several comments/concerns/suggestions to this. Overall, the idea of giving some space for experimentation that does not alienate parts of community from Neutron is good. In that way, we may relax review rules and quicken turnaround for preview features without loosing control on those features too much. Though the way it's to be implemented leaves several concerns, as follows: 1. From packaging perspective, having a separate repository and tarballs seems not optimal. As a packager, I would better deal with a single tarball instead of two. Meaning, it would be better to keep the code in the same tree. I know that we're afraid of shipping the code for which some users may expect the usual level of support and stability and compatibility. This can be solved by making it explicit that the incubated code is unsupported and used on your user's risk. 1) The experimental code wouldn't probably be installed unless explicitly requested, and 2) it would be put in a separate namespace (like 'preview', 'experimental', or 'staging', as the call it in Linux kernel world [2]). This would facilitate keeping commit history instead of loosing it during graduation. Yes, I know that people don't like to be called experimental or preview or incubator... And maybe neutron-labs repo sounds more appealing than an 'experimental' subtree in the core project. Well, there are lots of EXPERIMENTAL features in Linux kernel that we actively use (for example, btrfs is still considered experimental by Linux kernel devs, while being exposed as a supported option to RHEL7 users), so I don't see how that naming concern is significant. I think this is the whole point of the discussion around the incubator and the reason for which, to the best of my knowledge, no proposal has been accepted yet. I wonder where discussion around the proposal is running. Is it public? The discussion started out privately as the incubation proposal was put together, but it's now on the mailing list, in person, and in IRC meetings. Lets keep the discussion going on list now. In the spirit of keeping the discussion going, I think we probably need to iterate in practice on this idea a little bit before we can crystallize on the policy and process for this new repo. Here are few ideas on how we can start this iteration: * Namespace for the new repo: Should this be in the neutron namespace, or a completely different namespace like neutron labs? Perhaps creating a separate namespace will help the packagers to avoid issues of conflicting package owners of the namespace. I don’t think there is a technical requirement to choose a new namespace. Python supports sharing a namespace, and packaging can support this feature (see: oslo.*). * Dependency on Neutron (core) repository: We would need to sort this out so that we can get UTs to run and pass in the new repo. Can we set the dependency on Neutron milestone releases? We already publish tar balls for the milestone releases, but I am not sure we publish these as packages to pypi. If not could we start doing that? With this in place, the incubator would always lag the Neutron core by at the most one milestone release. Given that it is possible to specify a dependency as a branch/hash/tag in a git repo [1], I’m not sure it’s worth figuring out how to target tarballs. Master branch of the incubation repo could then target the master branch of the Neutron repo and always be assured of being current, and then released versions could target milestone tags or released versions. 1: http://pip.readthedocs.org/en/latest/reference/pip_install.html#git * Modules overlapping with the Neutron (core) repository: We could initially start with the features that required very little or no changes to the Neutron core, to avoid getting into the issue of blocking on changes to the Neutron (core) repository before progress can be made in the incubator. +1 I agree that it would be in an incubated effort’s best interest to put off doing invasive changes to the Neutron tree as long as possible to ensure sufficient time to hash out the best approach. * Packaging of ancillary code (CLI, Horizon, Heat): We start by adding these as subdirectories inside each feature. The packaging folks are going to find it difficult to package this. However, can we start with this approach, and have a parallel discussion
Re: [openstack-dev] [neutron] Incubator concerns from packaging perspective
On Aug 20, 2014, at 6:28 PM, Salvatore Orlando sorla...@nicira.com wrote: Some comments inline. Salvatore On 20 August 2014 17:38, Ihar Hrachyshka ihrac...@redhat.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi all, I've read the proposal for incubator as described at [1], and I have several comments/concerns/suggestions to this. Overall, the idea of giving some space for experimentation that does not alienate parts of community from Neutron is good. In that way, we may relax review rules and quicken turnaround for preview features without loosing control on those features too much. Though the way it's to be implemented leaves several concerns, as follows: 1. From packaging perspective, having a separate repository and tarballs seems not optimal. As a packager, I would better deal with a single tarball instead of two. Meaning, it would be better to keep the code in the same tree. I know that we're afraid of shipping the code for which some users may expect the usual level of support and stability and compatibility. This can be solved by making it explicit that the incubated code is unsupported and used on your user's risk. 1) The experimental code wouldn't probably be installed unless explicitly requested, and 2) it would be put in a separate namespace (like 'preview', 'experimental', or 'staging', as the call it in Linux kernel world [2]). This would facilitate keeping commit history instead of loosing it during graduation. Yes, I know that people don't like to be called experimental or preview or incubator... And maybe neutron-labs repo sounds more appealing than an 'experimental' subtree in the core project. Well, there are lots of EXPERIMENTAL features in Linux kernel that we actively use (for example, btrfs is still considered experimental by Linux kernel devs, while being exposed as a supported option to RHEL7 users), so I don't see how that naming concern is significant. I think this is the whole point of the discussion around the incubator and the reason for which, to the best of my knowledge, no proposal has been accepted yet. 2. If those 'extras' are really moved into a separate repository and tarballs, this will raise questions on whether packagers even want to cope with it before graduation. When it comes to supporting another build manifest for a piece of code of unknown quality, this is not the same as just cutting part of the code into a separate experimental/labs package. So unless I'm explicitly asked to package the incubator, I wouldn't probably touch it myself. This is just too much effort (btw the same applies to moving plugins out of the tree - once it's done, distros will probably need to reconsider which plugins they really want to package; at the moment, those plugins do not require lots of time to ship them, but having ~20 separate build manifests for each of them is just too hard to handle without clear incentive). One reason instead for moving plugins out of the main tree is allowing their maintainers to have full control over them. If there was a way with gerrit or similars to give somebody rights to merge code only on a subtree I probably would not even consider the option of moving plugin and drivers away. From my perspective it's not that I don't want them in the main tree, it's that I don't think it's fair for core team reviewers to take responsibility of approving code that they can't fully tests (3rd partt CI helps, but is still far from having a decent level of coverage). 3. The fact that neutron-incubator is not going to maintain any stable branches for security fixes and major failures concerns me too. In downstream, we don't generally ship the latest and greatest from PyPI. Meaning, we'll need to maintain our own downstream stable branches for major fixes. [BTW we already do that for python clients.] This is a valid point. We need to find an appropriate trade off. My thinking was that incubated projects could be treated just like client libraries from a branch perspective. 4. Another unclear part of the proposal is that notion of keeping Horizon and client changes required for incubator features in neutron-incubator. AFAIK the repo will be governed by Neutron Core team, and I doubt the team is ready to review Horizon changes (?). I think I don't understand how we're going to handle that. Can we just postpone Horizon work till graduation? I too do not think it's a great idea, mostly because there will be horizon bits not shipped with horizon, and not verified by horizon core team. I think it would be ok to have horizon support for neutron incubator. It won't be the first time that support for experimental features is added in horizon. 5. The wiki page says that graduation will require full test coverage. Does it mean 100% coverage in 'coverage' report? I don't think our existing code is even near that point, so maybe
Re: [openstack-dev] [neutron][cisco] Cisco Nexus requires patched ncclient
On Aug 14, 2014, at 1:55 PM, Ihar Hrachyshka ihrac...@redhat.com wrote: Signed PGP part FYI: I've uploaded a review for openstack/requirements to add the upstream module into the list of potential dependencies [1]. Once it's merged, I'm going to introduce this new requirement for Neutron. The only reason someone would install ncclient would be because they had a brocade or cisco solution that they wanted to integrate with and I don't think that justifies making Neutron depend on the library. Maru [1]: https://review.openstack.org/114213 /Ihar On 12/08/14 16:27, Ihar Hrachyshka wrote: Hey all, as per [1], Cisco Nexus ML2 plugin requires a patched version of ncclient from github. I wonder: - whether this information is still current; - why don't we depend on ncclient thru our requirements.txt file. [1]: https://wiki.openstack.org/wiki/Neutron/ML2/MechCiscoNexus Cheers, /Ihar ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][core] Expectations of core reviewers
On Aug 14, 2014, at 8:52 AM, Russell Bryant rbry...@redhat.com wrote: On 08/14/2014 11:40 AM, David Kranz wrote: On 08/14/2014 10:54 AM, Matt Riedemann wrote: On 8/14/2014 3:47 AM, Daniel P. Berrange wrote: On Thu, Aug 14, 2014 at 09:24:36AM +1000, Michael Still wrote: On Thu, Aug 14, 2014 at 3:09 AM, Dan Smith d...@danplanet.com wrote: I'm not questioning the value of f2f - I'm questioning the idea of doing f2f meetings sooo many times a year. OpenStack is very much the outlier here among open source projects - the vast majority of projects get along very well with much less f2f time and a far smaller % of their contributors attend those f2f meetings that do happen. So I really do question what is missing from OpenStack's community interaction that makes us believe that having 4 f2f meetings a year is critical to our success. How many is too many? So far, I have found the midcycles to be extremely productive -- productive in a way that we don't see at the summits, and I think other attendees agree. Obviously if budgets start limiting them, then we'll have to deal with it, but I don't want to stop meeting preemptively. I agree they're very productive. Let's pick on the nova v3 API case as an example... We had failed as a community to reach a consensus using our existing discussion mechanisms (hundreds of emails, at least three specs, phone calls between the various parties, et cetera), yet at the summit and then a midcycle meetup we managed to nail down an agreement on a very contentious and complicated topic. We thought we had agreement on v3 API after Atlanta f2f summit and after Hong Kong f2f too. So I wouldn't neccessarily say that we needed another f2f meeting to resolve that, but rather than this is a very complex topic that takes a long time to resolve no matter how we discuss it and the discussions had just happened to reach a natural conclusion this time around. But lets see if this agreement actually sticks this time I can see the argument that travel cost is an issue, but I think its also not a very strong argument. We have companies spending millions of dollars on OpenStack -- surely spending a relatively small amount on travel to keep the development team as efficient as possible isn't a big deal? I wouldn't be at all surprised if the financial costs of the v3 API debate (staff time mainly) were much higher than the travel costs of those involved in the summit and midcycle discussions which sorted it out. I think the travel cost really is a big issue. Due to the number of people who had to travel to the many mid-cycle meetups, a good number of people I work with no longer have the ability to go to the Paris design summit. This is going to make it harder for them to feel a proper engaged part of our community. I can only see this situation get worse over time if greater emphasis is placed on attending the mid-cycle meetups. Travelling to places to talk to people isn't a great solution, but it is the most effective one we've found so far. We should continue to experiment with other options, but until we find something that works as well as meetups, I think we need to keep having them. IMHO, the reasons to cut back would be: - People leaving with a well, that was useless... feeling - Not enough people able to travel to make it worthwhile So far, neither of those have been outcomes of the midcycles we've had, so I think we're doing okay. The design summits are structured differently, where we see a lot more diverse attendance because of the colocation with the user summit. It doesn't lend itself well to long and in-depth discussions about specific things, but it's very useful for what it gives us in the way of exposure. We could try to have less of that at the summit and more midcycle-ish time, but I think it's unlikely to achieve the same level of usefulness in that environment. Specifically, the lack of colocation with too many other projects has been a benefit. This time, Mark and Maru where there from Neutron. Last time, Mark from Neutron and the other Mark from Glance were there. If they were having meetups in other rooms (like at summit) they wouldn't have been there exposed to discussions that didn't seem like they'd have a component for their participation, but did after all (re: nova and glance and who should own flavors). I agree. The ability to focus on the issues that were blocking nova was very important. That's hard to do at a design summit when there is so much happening at the same time. Maybe we should change the way we structure the design summit to improve that. If there are critical issues blocking nova, it feels like it is better to be able to discuss and resolve as much as possible at the start of the dev cycle rather than in the middle of the dev cycle because I feel that means we are causing ourselves pain during milestone 1/2. Just speaking from
Re: [openstack-dev] [nova][core] Expectations of core reviewers
On Aug 13, 2014, at 10:32 PM, Michael Still mi...@stillhq.com wrote: On Thu, Aug 14, 2014 at 2:48 PM, Joe Gordon joe.gord...@gmail.com wrote: On Wed, Aug 13, 2014 at 8:31 PM, Michael Still mi...@stillhq.com wrote: On Thu, Aug 14, 2014 at 1:24 PM, Jay Pipes jaypi...@gmail.com wrote: Just wanted to quickly weigh in with my thoughts on this important topic. I very much valued the face-to-face interaction that came from the mid-cycle meetup in Beaverton (it was the only one I've ever been to). That said, I do not believe it should be a requirement that cores make it to the face-to-face meetings in-person. A number of folks have brought up very valid concerns about personal/family time, travel costs and burnout. I'm not proposing they be a requirement. I am proposing that they be strongly encouraged. I believe that the issue raised about furthering the divide between core and non-core folks is actually the biggest reason I don't support a mandate to have cores at the face-to-face meetings, and I think we should make our best efforts to support quality virtual meetings that can be done on a more frequent basis than the face-to-face meetings that would be optional. I am all for online meetings, but we don't have a practical way to do them at the moment apart from IRC. Until someone has a concrete proposal that's been shown to work, I feel its a straw man argument. What about making it easier for remote people to participate at the mid-cycle meetups? Set up some microphones and a Google hangout? At least that way attending the mid-cycle is not all or nothing. We did something like this last cycle (IIRC we didn't have enough mics) and it worked pretty well. As I said, I'm open to experimenting, but I need someone other than me to own that. I'm simply too busy to get to it. However, I don't think we should throw away the thing that works for best for us now, until we have a working replacement. I'm very much in favour of work being done on a replacement though. +1 I agree that that mid-cycles may not be sustainable over the long-term due to issues of travel cost (financial and otherwise) and a lack of inclusiveness, but I don't think they should stop happening until a suitably productive alternative has been found. Maru ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Which changes need accompanying bugs?
On Aug 13, 2014, at 11:11 AM, Kevin Benton blak...@gmail.com wrote: Is the pylint static analysis that caught that error prone to false positives? If not, I agree that it would be really nice if that were made part of the tox check so these don't have to be fixed after the fact. To me that particular patch seems like one that should be accompanied with a unit test because it's a failure case with cleanup code that isn't being touched by the unit tests. +1 As a general rule I would like to see test addition included with any fix targeting a bug that was merged due to a lack of coverage. m. On Aug 13, 2014 8:34 AM, Armando M. arma...@gmail.com wrote: I am gonna add more color to this story by posting my replies on review [1]: Hi Angus, You touched on a number of points. Let me try to give you an answer to all of them. (I'll create a bug report too. I still haven't worked out which class of changes need an accompanying bug report and which don't.) The long story can be read below: https://wiki.openstack.org/wiki/BugFilingRecommendations https://wiki.openstack.org/wiki/GitCommitMessages IMO, there's a grey area for some of the issues you found, but when I am faced with a bug, I tend to answer myself? Would a bug report be useful to someone else? The author of the code? A consumer of the code? Not everyone follow the core review system all the time, whereas Launchpad is pretty much the tool everyone uses to stay abreast with the OpenStack release cycle. Obviously if you're fixing a grammar nit, or filing a cosmetic change that has no functional impact then I warrant the lack of a test, but in this case you're fixing a genuine error: let's say we want to backport this to icehouse, how else would we make the release manager of that? He/she is looking at Launchpad. I can add a unittest for this particular code path, but it would only check this particular short segment of code, would need to be maintained as the code changes, and wouldn't catch another occurrence somewhere else. This seems an unsatisfying return on the additional work :( You're looking at this from the wrong perspective. This is not about ensuring that other code paths are valid, but that this code path stays valid over time, ensuring that the code path is exercised and that no other regression of any kind creep in. The reason why this error was introduced in the first place is because the code wasn't tested when it should have. If you don't get that this mechanical effort of fixing errors by static analysis is kind of ineffective, which leads me to the last point I actually found this via static analysis using pylint - and my question is: should I create some sort of pylint unittest that tries to catch this class of problem across the entire codebase? [...] I value what you're doing, however I would see even more value if we prevented these types of errors from occurring in the first place via automation. You run pylint today, but what about tomorrow, or a week from now? Are you gonna be filing pylint fixes for ever? We might be better off automating the check and catch these types of errors before they land in the tree. This means that the work you are doing it two-pronged: a) automate the detection of some failures by hooking this into tox.ini via HACKING/pep8 or equivalent mechanism and b) file all the fixes that require these validation tests to pass; c) everyone is happy, or at least they should be. I'd welcome to explore a better strategy to ensure a better quality of the code base, without some degree of automation, nothing will stop these conversation from happening again. Cheers, Armando [1] https://review.openstack.org/#/c/113777/ On 13 August 2014 03:02, Ihar Hrachyshka ihrac...@redhat.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 13/08/14 09:28, Angus Lees wrote: I'm doing various small cleanup changes as I explore the neutron codebase. Some of these cleanups are to fix actual bugs discovered in the code. Almost all of them are tiny and obviously correct. A recurring reviewer comment is that the change should have had an accompanying bug report and that they would rather that change was not submitted without one (or at least, they've -1'ed my change). I often didn't discover these issues by encountering an actual production issue so I'm unsure what to include in the bug report other than basically a copy of the change description. I also haven't worked out the pattern yet of which changes should have a bug and which don't need one. There's a section describing blueprints in NeutronDevelopment but nothing on bugs. It would be great if someone who understands the nuances here could add some words on when to file bugs: Which type of changes should have accompanying bug reports? What is the purpose of that bug, and what should it contain? It was discussed before at:
[openstack-dev] Fwd: [nova][core] Expectations of core reviewers
On Aug 13, 2014, at 2:57 AM, Daniel P. Berrange berra...@redhat.com wrote:
On Wed, Aug 13, 2014 at 08:57:40AM +1000, Michael Still wrote:
Hi.
One of the action items from the nova midcycle was that I was asked to
make nova's expectations of core reviews more clear. This email is an
attempt at that.
Nova expects a minimum level of sustained code reviews from cores. In
the past this has been generally held to be in the order of two code
reviews a day, which is a pretty low bar compared to the review
workload of many cores. I feel that existing cores understand this
requirement well, and I am mostly stating it here for completeness.
Additionally, there is increasing levels of concern that cores need to
be on the same page about the criteria we hold code to, as well as the
overall direction of nova. While the weekly meetings help here, it was
agreed that summit attendance is really important to cores. Its the
way we decide where we're going for the next cycle, as well as a
chance to make sure that people are all pulling in the same direction
and trust each other.
There is also a strong preference for midcycle meetup attendance,
although I understand that can sometimes be hard to arrange. My stance
is that I'd like core's to try to attend, but understand that
sometimes people will miss one. In response to the increasing
importance of midcycles over time, I commit to trying to get the dates
for these events announced further in advance.
Personally I'm going to find it really hard to justify long distance
travel 4 times a year for OpenStack for personal / family reasons,
let alone company cost. I couldn't attend Icehouse mid-cycle because
I just had too much travel in a short time to be able to do another
week long trip away from family. I couldn't attend Juno mid-cycle
because it clashed we personal holiday. There are other opensource
related conferences that I also have to attend (LinuxCon, FOSDEM,
KVM Forum, etc), etc so doubling the expected number of openstack
conferences from 2 to 4 is really very undesirable from my POV.
I might be able to attend the occassional mid-cycle meetup if the
location was convenient, but in general I don't see myself being
able to attend them regularly.
I tend to view the fact that we're emphasising the need of in-person
meetups to be somewhat of an indication of failure of our community
operation. The majority of open source projects work very effectively
with far less face-to-face time. OpenStack is fortunate that companies
are currently willing to spend 6/7-figure sums flying 1000's of
developers around the world many times a year, but I don't see that
lasting forever so I'm concerned about baking the idea of f2f midcycle
meetups into our way of life even more strongly.
I was fortunate to attend both the Nova and Neutron mid-cycles last month, and
I can attest to how productive these gatherings were. Discussion moved quickly
and misunderstandings were rapidly resolved. Informal ('water-cooler')
conversation led to many interactions that might not otherwise have occurred.
Given your attendance of summit and other open source conferences, though, I'm
assuming the value of f2f is not in question.
Nothing good is ever free. The financial cost and exclusionary nature of an
in-person meetup should definitely be weighed against the opportunity for
focused and high-bandwidth communication. It's clear to myself and other
attendees just how valuable the recent mid-cycles were in terms of making
technical decisions and building the relationships to support their
implementation. Maybe it isn't sustainable over the long-term to meet so
often, but I don't think that should preclude us from deriving benefit in the
short-term. I also don't think we should ignore the opportunity for more
effective decision-making on the grounds that not everyone can directly
participate. Not everyone is able to attend summit, but it is nonetheless a
critical part of our community's decision-making process. The topic lists for
a mid-cycle are published beforehand, just like summit, to allow non-attendees
the chance to present their views in advance and/or designate one or more
attendees to advocate on
their behalf. It's not perfect, but the alternative - not holding mid-cycles
- would seem to be a case of throwing out the baby with the bathwater.
Maru
Given that we consider these physical events so important, I'd like
people to let me know if they have travel funding issues. I can then
approach the Foundation about funding travel if that is required.
Travel funding is certainly an issue, but I'm not sure that Foundation
funding would be a solution, because the impact probably isn't directly
on the core devs. Speaking with my Red Hat on, if the midcycle meetup
is important enough, the core devs will likely get the funding to attend.
The fallout of this though is that every attendee at a mid-cycle summit
Re: [openstack-dev] [nova][core] Expectations of core reviewers
On Aug 13, 2014, at 2:57 AM, Daniel P. Berrange berra...@redhat.com wrote:
On Wed, Aug 13, 2014 at 08:57:40AM +1000, Michael Still wrote:
Hi.
One of the action items from the nova midcycle was that I was asked to
make nova's expectations of core reviews more clear. This email is an
attempt at that.
Nova expects a minimum level of sustained code reviews from cores. In
the past this has been generally held to be in the order of two code
reviews a day, which is a pretty low bar compared to the review
workload of many cores. I feel that existing cores understand this
requirement well, and I am mostly stating it here for completeness.
Additionally, there is increasing levels of concern that cores need to
be on the same page about the criteria we hold code to, as well as the
overall direction of nova. While the weekly meetings help here, it was
agreed that summit attendance is really important to cores. Its the
way we decide where we're going for the next cycle, as well as a
chance to make sure that people are all pulling in the same direction
and trust each other.
There is also a strong preference for midcycle meetup attendance,
although I understand that can sometimes be hard to arrange. My stance
is that I'd like core's to try to attend, but understand that
sometimes people will miss one. In response to the increasing
importance of midcycles over time, I commit to trying to get the dates
for these events announced further in advance.
Personally I'm going to find it really hard to justify long distance
travel 4 times a year for OpenStack for personal / family reasons,
let alone company cost. I couldn't attend Icehouse mid-cycle because
I just had too much travel in a short time to be able to do another
week long trip away from family. I couldn't attend Juno mid-cycle
because it clashed we personal holiday. There are other opensource
related conferences that I also have to attend (LinuxCon, FOSDEM,
KVM Forum, etc), etc so doubling the expected number of openstack
conferences from 2 to 4 is really very undesirable from my POV.
I might be able to attend the occassional mid-cycle meetup if the
location was convenient, but in general I don't see myself being
able to attend them regularly.
I tend to view the fact that we're emphasising the need of in-person
meetups to be somewhat of an indication of failure of our community
operation. The majority of open source projects work very effectively
with far less face-to-face time. OpenStack is fortunate that companies
are currently willing to spend 6/7-figure sums flying 1000's of
developers around the world many times a year, but I don't see that
lasting forever so I'm concerned about baking the idea of f2f midcycle
meetups into our way of life even more strongly.
I was fortunate to attend both the Nova and Neutron mid-cycles last month, and
I can attest to how productive these gatherings were. Discussion moved quickly
and misunderstandings were rapidly resolved. Informal ('water-cooler')
conversation led to many interactions that might not otherwise have occurred.
Given your attendance of summit and other open source conferences, though, I'm
assuming the value of f2f is not in question.
Nothing good is ever free. The financial cost and exclusionary nature of an
in-person meetup should definitely be weighed against the opportunity for
focused and high-bandwidth communication. It's clear to myself and other
attendees just how valuable the recent mid-cycles were in terms of making
technical decisions and building the relationships to support their
implementation. Maybe it isn't sustainable over the long-term to meet so
often, but I don't think that should preclude us from deriving benefit in the
short-term. I also don't think we should ignore the opportunity for more
effective decision-making on the grounds that not everyone can directly
participate. Not everyone is able to attend summit, but it is nonetheless a
critical part of our community's decision-making process. The topic lists for a
mid-cycle are published beforehand, just like summit, to allow non-attendees
the chance to present their views in advance and/or designate one or more
attendees to advocate on
their behalf. It's not perfect, but the alternative - not holding mid-cycles
- would seem to be a case of throwing out the baby with the bathwater.
Maru
Given that we consider these physical events so important, I'd like
people to let me know if they have travel funding issues. I can then
approach the Foundation about funding travel if that is required.
Travel funding is certainly an issue, but I'm not sure that Foundation
funding would be a solution, because the impact probably isn't directly
on the core devs. Speaking with my Red Hat on, if the midcycle meetup
is important enough, the core devs will likely get the funding to attend.
The fallout of this though is that every attendee at a mid-cycle summit
[openstack-dev] Fwd: [nova][core] Expectations of core reviewers
My apologies, I managed to break the thread here. Please respond to the thread
with subject 'Re: [openstack-dev] [nova][core] Expectations of core reviewers'
in preference to this one.
Maru
On Aug 13, 2014, at 9:01 AM, Maru Newby ma...@redhat.com wrote:
On Aug 13, 2014, at 2:57 AM, Daniel P. Berrange berra...@redhat.com wrote:
On Wed, Aug 13, 2014 at 08:57:40AM +1000, Michael Still wrote:
Hi.
One of the action items from the nova midcycle was that I was asked to
make nova's expectations of core reviews more clear. This email is an
attempt at that.
Nova expects a minimum level of sustained code reviews from cores. In
the past this has been generally held to be in the order of two code
reviews a day, which is a pretty low bar compared to the review
workload of many cores. I feel that existing cores understand this
requirement well, and I am mostly stating it here for completeness.
Additionally, there is increasing levels of concern that cores need to
be on the same page about the criteria we hold code to, as well as the
overall direction of nova. While the weekly meetings help here, it was
agreed that summit attendance is really important to cores. Its the
way we decide where we're going for the next cycle, as well as a
chance to make sure that people are all pulling in the same direction
and trust each other.
There is also a strong preference for midcycle meetup attendance,
although I understand that can sometimes be hard to arrange. My stance
is that I'd like core's to try to attend, but understand that
sometimes people will miss one. In response to the increasing
importance of midcycles over time, I commit to trying to get the dates
for these events announced further in advance.
Personally I'm going to find it really hard to justify long distance
travel 4 times a year for OpenStack for personal / family reasons,
let alone company cost. I couldn't attend Icehouse mid-cycle because
I just had too much travel in a short time to be able to do another
week long trip away from family. I couldn't attend Juno mid-cycle
because it clashed we personal holiday. There are other opensource
related conferences that I also have to attend (LinuxCon, FOSDEM,
KVM Forum, etc), etc so doubling the expected number of openstack
conferences from 2 to 4 is really very undesirable from my POV.
I might be able to attend the occassional mid-cycle meetup if the
location was convenient, but in general I don't see myself being
able to attend them regularly.
I tend to view the fact that we're emphasising the need of in-person
meetups to be somewhat of an indication of failure of our community
operation. The majority of open source projects work very effectively
with far less face-to-face time. OpenStack is fortunate that companies
are currently willing to spend 6/7-figure sums flying 1000's of
developers around the world many times a year, but I don't see that
lasting forever so I'm concerned about baking the idea of f2f midcycle
meetups into our way of life even more strongly.
I was fortunate to attend both the Nova and Neutron mid-cycles last month,
and I can attest to how productive these gatherings were. Discussion moved
quickly and misunderstandings were rapidly resolved. Informal
('water-cooler') conversation led to many interactions that might not
otherwise have occurred. Given your attendance of summit and other open
source conferences, though, I'm assuming the value of f2f is not in question.
Nothing good is ever free. The financial cost and exclusionary nature of an
in-person meetup should definitely be weighed against the opportunity for
focused and high-bandwidth communication. It's clear to myself and other
attendees just how valuable the recent mid-cycles were in terms of making
technical decisions and building the relationships to support their
implementation. Maybe it isn't sustainable over the long-term to meet so
often, but I don't think that should preclude us from deriving benefit in the
short-term. I also don't think we should ignore the opportunity for more
effective decision-making on the grounds that not everyone can directly
participate. Not everyone is able to attend summit, but it is nonetheless a
critical part of our community's decision-making process. The topic lists
for a mid-cycle are published beforehand, just like summit, to allow
non-attendees the chance to present their views in advance and/or designate
one or more attendees to advocate
on their behalf. It's not perfect, but the alternative - not holding
mid-cycles - would seem to be a case of throwing out the baby with the
bathwater.
Maru
Given that we consider these physical events so important, I'd like
people to let me know if they have travel funding issues. I can then
approach the Foundation about funding travel if that is required.
Travel funding is certainly an issue, but I'm not sure that Foundation
Re: [openstack-dev] [neutron] Rotating the weekly Neutron meeting
+1 Maru On Aug 13, 2014, at 7:05 AM, Kyle Mestery mest...@mestery.com wrote: Per this week's Neutron meeting [1], it was decided that offering a rotating meeting slot for the weekly Neutron meeting would be a good thing. This will allow for a much easier time for people in Asia/Pacific timezones, as well as for people in Europe. So, I'd like to propose we rotate the weekly as follows: Monday 2100UTC Tuesday 1400UTC If people are ok with these time slots, I'll set this up and we'll likely start with this new schedule in September, after the FPF. Thanks! Kyle [1] http://eavesdrop.openstack.org/meetings/networking/2014/networking.2014-08-11-21.00.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Group Based Policy and the way forward
On Aug 8, 2014, at 10:56 AM, Kevin Benton blak...@gmail.com wrote: There is an enforcement component to the group policy that allows you to use the current APIs and it's the reason that group policy is integrated into the neutron project. If someone uses the current APIs, the group policy plugin will make sure they don't violate any policy constraints before passing the request into the regular core/service plugins. The enforcement requirement might be easier to implement through code-based integration, but a separate service could provide the same guarantee against constraint violation by proxying v2 API calls for an endpoint to which access was restricted. Apologies if I've missed discussion of this (it's a big thread), but won't enforcement by group policy on the v2 API have the potential to violate stability requirements? If new errors related to enforcement can result from API calls, that would seem to be a change in behavior. Do we have a precedent for allowing extensions or new services to modify core behavior in this way? m. On Fri, Aug 8, 2014 at 11:02 AM, Salvatore Orlando sorla...@nicira.com wrote: It might be because of the wording used, but it seems to me that you're making it sound like the group policy effort could have been completely orthogonal to neutron as we know it now. What I understood is that the declarative abstraction offered by group policy could do without any existing neutron entity leveraging native drivers, but can actually be used also with existing neutron plugins through the mapping driver - which will provide a sort of backward compatibility. And still in that case I'm not sure one would be able to use traditional neutron API (or legacy as it has been called), since I don't know if the mapping driver is bidirectional. I know this probably stems from my ignorance on the subject - I had unfortunately very little time to catch-up with this effort in the past months. Salvatore On 8 August 2014 18:49, Ivar Lazzaro ivarlazz...@gmail.com wrote: Hi Jay, You can choose. The whole purpose of this is about flexibility, if you want to use GBP API 'only' with a specific driver you just can. Additionally, given the 'ML2 like' architecture, the reference mapping driver can ideally run alongside by filling the core Neutron constructs without ever 'disturbing' your own driver (I'm not entirely sure about this but it seems feasible). I hope this answers your question, Ivar. On Fri, Aug 8, 2014 at 6:28 PM, Jay Pipes jaypi...@gmail.com wrote: On 08/08/2014 08:55 AM, Kevin Benton wrote: The existing constructs will not change. A followup question on the above... If GPB API is merged into Neutron, the next logical steps (from what I can tell) will be to add drivers that handle policy-based payloads/requests. Some of these drivers, AFAICT, will *not* be deconstructing these policy requests into the low-level port, network, and subnet creation/attachment/detachment commands, but instead will be calling out as-is to hardware that speaks the higher-level abstraction API [1], not the lower-level port/subnet/network APIs. The low-level APIs would essentially be consumed entirely within the policy-based driver, which would effectively mean that the only way a system would be able to orchestrate networking in systems using these drivers would be via the high-level policy API. Is that correct? Very sorry if I haven't explained clearly my question... this is a tough question to frame eloquently :( Thanks, -jay [1] http://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-infrastructure/index.html On Aug 8, 2014 9:49 AM, CARVER, PAUL pc2...@att.com mailto:pc2...@att.com wrote: Wuhongning [mailto:wuhongn...@huawei.com mailto:wuhongn...@huawei.com] wrote: Does it make sense to move all advanced extension out of ML2, like security group, qos...? Then we can just talk about advanced service itself, without bothering basic neutron object (network/subnet/port) A modular layer 3 (ML3) analogous to ML2 sounds like a good idea. I still think it's too late in the game to be shooting down all the work that the GBP team has put in unless there's a really clean and effective way of running AND iterating on GBP in conjunction with Neutron without being part of the Juno release. As far as I can tell they've worked really hard to follow the process and accommodate input. They shouldn't have to wait multiple more releases on a hypothetical refactoring of how L3+ vs L2 is structured. But, just so I'm not making a horrible mistake, can someone reassure me that GBP isn't removing the constructs of network/subnet/port from Neutron? I'm under the impression that GBP is adding a higher level abstraction but that it's not ripping basic
Re: [openstack-dev] [neutron] [nova] Weekly nova-network / neutron parity meeting
On Jul 17, 2014, at 8:12 AM, Kyle Mestery mest...@mestery.com wrote: On Thu, Jul 17, 2014 at 6:42 AM, Thierry Carrez thie...@openstack.org wrote: Kyle Mestery wrote: As we're getting down to the wire in Juno, I'd like to propose we have a weekly meeting on the nova-network and neutron parity effort. I'd like to start this meeting next week, and I'd like to propose Wednesday at 1500 UTC on #openstack-meeting-3 as the time and location. This conflicts on the agenda with the PHP SDK Team Meeting. That said, I'm not sure they still run this one regularly. We could do Wednesday or Friday at 1500 UTC if that works better for folks. I'd prefer Wednesday, though. We could also do 1400UTC Wednesday. Perhaps I'll split the difference and do 1430 UTC Wednesday. Does that sound ok to everyone? Thanks, Kyle +1 from me. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron][group-based-policy] Should we revisit the priority of group-based policy?
At the summit session last week for group-based policy, there were many concerns voiced about the approach being undertaken. I think those concerns deserve a wider audience, and I'm going to highlight some of them here. The primary concern seemed to be related to the complexity of the approach implemented for the POC. A number of session participants voiced concern that the simpler approach documented in the original proposal [1] (described in the section titled 'Policies applied between groups') had not been implemented in addition to or instead of what appeared in the POC (described in the section titled 'Policies applied as a group API'). The simpler approach was considered by those participants as having the advantage of clarity and immediate usefulness, whereas the complex approach was deemed hard to understand and without immediate utility. A secondary but no less important concern is related to the impact on Neutron of the approach implemented in the POC. The POC was developed monolithically, without oversight through gerrit, and the resulting patches were excessive in size (~4700 [2] and ~1500 [3] lines). Such large patches are effectively impossible to review. Even broken down into reviewable chunks, though, it does not seem realistic to target juno-1 for merging this kind of complexity. The impact on stability could be considerable, and it is questionable whether the necessary review effort should be devoted to fast-tracking group-based policy at all, let alone an approach that is considered by many to be unnecessarily complicated. The blueprint for group policy [4] is currently listed as a 'High' priority. With the above concerns in mind, does it make sense to continue prioritizing an effort that at present would seem to require considerably more resources than the benefit it appears to promise? Maru 1: https://etherpad.openstack.org/p/group-based-policy 2: https://review.openstack.org/93853 3: https://review.openstack.org/93935 4: https://blueprints.launchpad.net/neutron/+spec/group-based-policy-abstraction ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][group-based-policy] Should we revisit the priority of group-based policy?
On May 22, 2014, at 11:03 AM, Maru Newby ma...@redhat.com wrote: At the summit session last week for group-based policy, there were many concerns voiced about the approach being undertaken. I think those concerns deserve a wider audience, and I'm going to highlight some of them here. The primary concern seemed to be related to the complexity of the approach implemented for the POC. A number of session participants voiced concern that the simpler approach documented in the original proposal [1] (described in the section titled 'Policies applied between groups') had not been implemented in addition to or instead of what appeared in the POC (described in the section titled 'Policies applied as a group API'). The simpler approach was considered by those participants as having the advantage of clarity and immediate usefulness, whereas the complex approach was deemed hard to understand and without immediate utility. A secondary but no less important concern is related to the impact on Neutron of the approach implemented in the POC. The POC was developed monolithically, without oversight through gerrit, and the resulting patches were excessive in size (~4700 [2] and ~1500 [3] lines). Such large patches are effectively impossible to review. Even broken down into reviewable chunks, though, it does not seem realistic to target juno-1 for merging this kind of complexity. The impact on stability could be considerable, and it is questionable whether the necessary review effort should be devoted to fast-tracking group-based policy at all, let alone an approach that is considered by many to be unnecessarily complicated. The blueprint for group policy [4] is currently listed as a 'High' priority. With the above concerns in mind, does it make sense to continue prioritizing an effort that at present would seem to require considerably more resources than the benefit it appears to promise? Maru 1: https://etherpad.openstack.org/p/group-based-policy Apologies, this link is to the summit session etherpad. The link to the original proposal is: https://docs.google.com/document/d/1ZbOFxAoibZbJmDWx1oOrOsDcov6Cuom5aaBIrupCD9E/edit 2: https://review.openstack.org/93853 3: https://review.openstack.org/93935 4: https://blueprints.launchpad.net/neutron/+spec/group-based-policy-abstraction ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][group-based-policy] Should we revisit the priority of group-based policy?
that we have adopted. If you don't like them, you are free to work to effect change, but it won't happen by simply wishing it so. And as per the summit session, there appears to be significant user demand for a subset of the POC that represents the 'link-based' approach. If we are to talk about taking user interests into account, I think the starting point would be implementing what they are asking for first and and evolving more complex solutions from there. m. 3. Re: Could dev/review cycles be better spent on refactoring I think that most people agree that policy control is an important feature that fundamentally improves neutron (by solving the automation and scale issues). In a large project, multiple sub-projects can, and for a healthy project should, work in parallel. I understand that the neutron core team is stretched. But we still need to be able to balance the needs of today (paying off the technical debt/existing-issues by doing refactoring) with needs of tomorrow (new features like GP and LBaaS). GP effort was started in Havana, and now we are trying to get this in Juno. I think that is reasonable and a long enough cycle for a high priority project to be able to get some core attention. Again I refer to LBaaS experience, as they struggled with very similar issues. 4. Re: If refactored neutron was available, would a simpler option become more viable We would love to be able to answer that question. We have been trying to understand the refactoring work to understand this (see another ML thread) and we are open to understanding your position on that. We will call the ad-hoc meeting that you suggested and we would like to understand the refactoring work that might be reused for simpler policy implementation. At the same time, we would like to build on what is available today, and when the required refactored neutron becomes available (say Juno or K-release), we are more than happy to adapt to it at that time. Serializing all development around an effort that is still in inception phase is not a good solution. We are looking forward to participating in the core refactoring work, and based on the final spec that come up with, we would love to be able to eventually make the policy implementation simpler. Regards, Mandeep On Thu, May 22, 2014 at 11:44 AM, Armando M. arma...@gmail.com wrote: I would second Maru's concerns, and I would also like to add the following: We need to acknowledge the fact that there are certain architectural aspects of Neutron as a project that need to be addressed; at the summit we talked about the core refactoring, a task oriented API, etc. To me these items have been neglected far too much over the past and would need a higher priority and a lot more attention during the Juno cycle. Being stretched as we are I wonder if dev/review cycles wouldn't be better spent devoting more time to these efforts rather than GP. That said, I appreciate that GP is important and needs to move forward, but at the same time I am thinking that there must be a better way for addressing it and yet relieve some of the pressure that GP complexity imposes to the Neutron team. One aspect it was discussed at the summit was that the type of approach shown in [2] and [3] below, was chosen because of lack of proper integration hooks...so I am advocating: let's talk about those first before ruling them out in favor of a monolithic approach that seems to violate some engineering principles, like modularity and loose decoupling of system components. I think we didn't have enough time during the summit to iron out some of the concerns voiced here, and it seems like the IRC meeting for Group Policy would not be the right venue to try and establish a common ground among the people driving this effort and the rest of the core team. Shall we try and have an ad-hoc meeting and an ad-hoc agenda to find a consensus? Many thanks, Armando On 22 May 2014 11:38, Maru Newby ma...@redhat.com wrote: On May 22, 2014, at 11:03 AM, Maru Newby ma...@redhat.com wrote: At the summit session last week for group-based policy, there were many concerns voiced about the approach being undertaken. I think those concerns deserve a wider audience, and I'm going to highlight some of them here. The primary concern seemed to be related to the complexity of the approach implemented for the POC. A number of session participants voiced concern that the simpler approach documented in the original proposal [1] (described in the section titled 'Policies applied between groups') had not been implemented in addition to or instead of what appeared in the POC (described in the section titled 'Policies applied as a group API'). The simpler approach was considered by those participants as having the advantage of clarity and immediate usefulness, whereas the complex approach was deemed hard to understand
Re: [openstack-dev] [neutron][group-based-policy] Should we revisit the priority of group-based policy?
On May 22, 2014, at 4:35 PM, Mandeep Dhami dh...@noironetworks.com wrote: each patch needs to receive core reviewer attention and that subsequent patches incorporate their feedback. At least two core neutron members were involved in creating the PoC, and at least two more cores were involved in reviews at various times. In addition to them, senior developers from at least seven networking companies were involved in developing this code. I concede that this code was on github for a few weeks, as that made the prototyping faster and allowed us to fail faster, but it was open and reviewed with the team above (and with the cores in that team). Based on our learning from that prototype activity, and feedback of those cores, we are upstreaming the improved production code to gerrit. All that involvement from the neutron core reviewers was critical in keeping the larger PoC team above focused on neutron norms and expectations from design and code. The feedback from reviewers needs to be provided on openstack infrastructure rather than outside it so that it is both visible to all reviewers (not just those directly involved) and that an enduring history of the process is retained. These requirements were not met in working in github on the POC, regardless of your protestations of how 'open' that work was and of who was involved. This isn't to suggest that out-of-tree prototyping isn't useful - of course it is. But I think it important to recognize that out-of-tree development is unlikely to be an effective way to develop code that can be easily merged to Neutron, and that the project can ill-afford the additional review cost it is likely to impose. As such, and as was agreed to in the irc meeting this morning, the way forward is to recognize that the POC is best considered a prototype useful in informing efforts to iterate in the open. m. On Thu, May 22, 2014 at 4:03 PM, Maru Newby ma...@redhat.com wrote: On May 22, 2014, at 1:59 PM, Mandeep Dhami dh...@noironetworks.com wrote: Maru's concerns are that: 1. It is large 2. It is complex As per the discussion in the irc meeting today, I hope it is clear now that eventual size and complexity are not real issue. Rather, I am concerned at how we get there. I keep talking about 'iterating in the open', and want to make it clear what I mean by this. It involves proposing a reviewable patch to openstack gerrit, working with reviewers to get the patch merged, and then incorporating their feedback into the overall design to drive the implementation of future patches. 'Iterating in the open' does not imply working outside of gerrit to create a monolithic codebase that needs to be manually decomposed into reviewable chunks at the end. I understand that this may be an effective way to create a POC, but it is not an effective way to produce code that can be merged into Neutron. Core reviewers have a mandate to ensure the quality of every patch, and their feedback is likely to have an impact on subsequent implementation. And Armando's related concerns are: 3. Could dev/review cycles be better spent on refactoring 4. If refactored neutron was available, would a simpler option become more viable Let me address them in that order. 1. Re: It is large Group policy has an ambitious goal - provide devop teams with policy based controls that are usable at scale and with automation (say a higher governance layer like Congress). The fact that meeting a large challenge requires more code is natural. We understand that challenge, and that is why we did a prototype (as PoC that was demonstrated on the summit). And based on that learning we are incrementally creating patches for building the group based policy. Just because a task is large, we as neutron can not shy away from building it. That will only drive people who need it out side neutron (as we are seeing with the frustration that the LBaaS team had because they have a requirement that is large as well). Again, the amount of code is not the problem. How code is introduced into the tree, and how the design is socialized (both with developers and users), _is_ of critical importance. Neutron is not alone in requiring an 'iterate in the open' approach - it is a characteristic common to many open source projects. 2. Re: It is complex Complexity depends on the context. Our goal was to make the end-user's life simpler (and more automated). To achieve some of that simplicity, we required a little more complexity in the implementation. We decide to make that arbitrage - a little higher complexity in implementation to allow for simpler usage. But we were careful and did not want to impose that complexity on every use case - hence a lot of that is optional (and exercised only if the use case needs it). Unfortunately the model, has to model all
Re: [openstack-dev] [neutron] Proposed changes to core team
On May 21, 2014, at 1:59 PM, Kyle Mestery mest...@noironetworks.com wrote: Neutron cores, please vote +1/-1 for the proposed addition of Carl Baldwin to Neutron core. +1 from me ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Request to consider merging OVS ARP responder
The patch in question is doing an invasive runtime check [1] to determine whether arp header modification is supported by ovs. I seem to remember a lack of consensus on whether invasive runtime checks were acceptable around vxlan detection, and I would appreciate confirmation from other cores that this approach is considered acceptable for this patch. m. 1: https://review.openstack.org/#/c/49227/41/neutron/agent/linux/ovs_lib.py On May 19, 2014, at 1:07 PM, Assaf Muller amul...@redhat.com wrote: Hello Mark, Bob and Kyle, You've all been involved with the OVS ARP responder patch: https://review.openstack.org/#/c/49227/ I kindly request that you revisit this patch - It's ready from my point of view, if that means anything. The last few patchsets have been nitpicks. Looking at the history, the patch has already had a large amount of +1's and +2's through it's various stages. I think it'd be technically superior if this patch was merged before further progress is made in the DVR patch series. Thanks for your time, Assaf Muller, Cloud Networking Engineer Red Hat ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Neutron][QA] Partial fix for unittest memory consumption
Memory usage due to plugin+mock leakage is addressed by the following patch: https://review.openstack.org/#/c/92793/ I'm seeing residual (post-test) memory usage decrease from ~4.5gb to ~1.3gb with 12 concurrent test runners. Of the 1.3gb, sqlalchemy is taking the lion share at ~500mb, so that will be my next target. Maru ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
On Mar 26, 2014, at 12:59 PM, Joe Gordon joe.gord...@gmail.com wrote: On Tue, Mar 25, 2014 at 1:49 AM, Maru Newby ma...@redhat.com wrote: On Mar 21, 2014, at 9:01 AM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project itself) does not count and they have to be in Tempest'. This has made us as a team wonder if we should migrate all our functional tests to Tempest. But from Matt's response, I think it is reasonable to continue in our current path have the functional tests in Marconi coexist along with the tests in Tempest. I think that what is being asked, really is that the functional tests could be a single set of tests that would become a part of the tempest repository and that these tests would have an ENV variable as part of the configuration that would allow either no Keystone or Keystone or some such, if that is the only configuration issue that separates running the tests isolated vs. integrated. The functional tests need to be as much as possible a single set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with each other/development. If they only run in the integrated environment, that's ok, but if you want to run them isolated to make debugging easier, then it should be a configuration option and a separate test job. So, if my assumptions are correct, QA only requires functional tests for integrated runs, but if the project QAs/Devs want to run isolated for dev and devtest purposes, more power to them. Just keep it a single set of functional tests and put them in the Tempest repository so that if a failure happens, anyone can find the test and do the debug work without digging into a separate project repository. Hopefully, the tests as designed could easily take a new configuration directive and a short bit of work with OS QA will get the integrated FTs working as well as the isolated ones. --Rocky This issue has been much debated. There are some active members of our community who believe that all the functional tests should live outside of tempest in the projects, albeit with the same idea that such tests could be run either as part of today's real tempest runs or mocked in various ways to allow component isolation or better performance. Maru Newby posted a patch with an example of one way to do this but I think it expired and I don't have a pointer. I think the best place for functional api tests to be maintained is in the projects themselves. The domain expertise required to write api tests is likely to be greater among project resources, and they should be tasked with writing api tests pre-merge. The current 'merge-first, test-later' procedure of maintaining api tests in the Tempest repo makes that impossible. Worse, the cost of developing functional api tests is higher in the integration environment that is the Tempest default. If an API is made and documented properly what domain expertise would be needed to use it? The opposite is true for tempest and the tests themselves. The tempest team focuses on just tests so they know how to write good tests and are able to leverage common underlying framework code. Given that documentation is typically finalized only late in the cycle, are you suggesting that we forego api testing until possibly well after the code has been written? Plus, it is a rare api that doesn't have to evolve in response to real-world experience with early implementations. The sooner we can write functional api tests, the sooner we can identify shortcomings that need to be addressed
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
On Mar 26, 2014, at 1:52 PM, Sean Dague s...@dague.net wrote: On 03/25/2014 04:49 AM, Maru Newby wrote: On Mar 21, 2014, at 9:01 AM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project itself) does not count and they have to be in Tempest'. This has made us as a team wonder if we should migrate all our functional tests to Tempest. But from Matt's response, I think it is reasonable to continue in our current path have the functional tests in Marconi coexist along with the tests in Tempest. I think that what is being asked, really is that the functional tests could be a single set of tests that would become a part of the tempest repository and that these tests would have an ENV variable as part of the configuration that would allow either no Keystone or Keystone or some such, if that is the only configuration issue that separates running the tests isolated vs. integrated. The functional tests need to be as much as possible a single set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with each other/development. If they only run in the integrated environment, that's ok, but if you want to run them isolated to make debugging easier, then it should be a configuration option and a separate test job. So, if my assumptions are correct, QA only requires functional tests for integrated runs, but if the project QAs/Devs want to run isolated for dev and devtest purposes, more power to them. Just keep it a single set of functional tests and put them in the Tempest repository so that if a failure happens, anyone can find the test and do the debug work without digging into a separate project repository. Hopefully, the tests as designed could easily take a new configuration directive and a short bit of work with OS QA will get the integrated FTs working as well as the isolated ones. --Rocky This issue has been much debated. There are some active members of our community who believe that all the functional tests should live outside of tempest in the projects, albeit with the same idea that such tests could be run either as part of today's real tempest runs or mocked in various ways to allow component isolation or better performance. Maru Newby posted a patch with an example of one way to do this but I think it expired and I don't have a pointer. I think the best place for functional api tests to be maintained is in the projects themselves. The domain expertise required to write api tests is likely to be greater among project resources, and they should be tasked with writing api tests pre-merge. The current 'merge-first, test-later' procedure of maintaining api tests in the Tempest repo makes that impossible. Worse, the cost of developing functional api tests is higher in the integration environment that is the Tempest default. I disagree. I think all that ends up doing is creating greater variance in quality between components in OpenStack. And it means now no one feels responsible when a bad test in a project some where causes a gate block. In my experience, projects have greater incentive to maintain tests that are in the same tree as the code they are testing. The tests can run pre-merge, and the same eyes that review the code can review the tests for that code. Any lag between when code is written and tests are written, at least below the integration level, can be detrimental due to the fact that the mental models for the code naturally decay over time. Does your experience on this differ? As you say, though, maybe its worth taking the hit of not putting as much testing in-tree as possible. Maybe
Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
On Mar 21, 2014, at 9:01 AM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response - just like an end user would, but without keystone. If one of these tests fail, it is because there is a bug in the Marconi code , and not because its interaction with Keystone caused it to fail. That being said there are certain cases where having a project specific functional test makes sense. For example swift has a functional test job that starts swift in devstack. But, those things are normally handled on a per case basis. In general if the project is meant to be part of the larger OpenStack ecosystem then Tempest is the place to put functional testing. That way you know it works with all of the other components. The thing is in openstack what seems like a project isolated functional test almost always involves another project in real use cases. (for example keystone auth with api requests) One of the concerns we heard in the review was 'having the functional tests elsewhere (I.e within the project itself) does not count and they have to be in Tempest'. This has made us as a team wonder if we should migrate all our functional tests to Tempest. But from Matt's response, I think it is reasonable to continue in our current path have the functional tests in Marconi coexist along with the tests in Tempest. I think that what is being asked, really is that the functional tests could be a single set of tests that would become a part of the tempest repository and that these tests would have an ENV variable as part of the configuration that would allow either no Keystone or Keystone or some such, if that is the only configuration issue that separates running the tests isolated vs. integrated. The functional tests need to be as much as possible a single set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with each other/development. If they only run in the integrated environment, that's ok, but if you want to run them isolated to make debugging easier, then it should be a configuration option and a separate test job. So, if my assumptions are correct, QA only requires functional tests for integrated runs, but if the project QAs/Devs want to run isolated for dev and devtest purposes, more power to them. Just keep it a single set of functional tests and put them in the Tempest repository so that if a failure happens, anyone can find the test and do the debug work without digging into a separate project repository. Hopefully, the tests as designed could easily take a new configuration directive and a short bit of work with OS QA will get the integrated FTs working as well as the isolated ones. --Rocky This issue has been much debated. There are some active members of our community who believe that all the functional tests should live outside of tempest in the projects, albeit with the same idea that such tests could be run either as part of today's real tempest runs or mocked in various ways to allow component isolation or better performance. Maru Newby posted a patch with an example of one way to do this but I think it expired and I don't have a pointer. I think the best place for functional api tests to be maintained is in the projects themselves. The domain expertise required to write api tests is likely to be greater among project resources, and they should be tasked with writing api tests pre-merge. The current 'merge-first, test-later' procedure of maintaining api tests in the Tempest repo makes that impossible. Worse, the cost of developing functional api tests is higher in the integration environment that is the Tempest default. The patch in question [1] proposes allowing pre-merge functional api test maintenance and test reuse in an integration environment. m. 1: https://review.openstack.org/#/c/72585/ IMO there are valid arguments on both sides, but I hope every one could agree that functional tests should not be arbitrarily split between projects and tempest as they are now. The Tempest README states a desire for complete coverage of the OpenStack API but Tempest is not close to that. We have been discussing and then ignoring this issue for some time but I think the recent action to say that Tempest will be used to determine if something can use the OpenStack trademark will force more completeness on tempest (more tests, that is). I think we need to resolve this issue but it won't be easy and modifying existing api tests to be more flexible will be a lot of work. But at least new projects could get on the right
Re: [openstack-dev] [Neutron][FYI] Bookmarklet for neutron gerrit review
+1
I think there should be naming standard for all reviews (e.g. [test] Jenkins,
[test-external] VMware) so that gerrit css can colorize automatic review
comments no matter the project.
m.
On Mar 7, 2014, at 2:08 AM, Chmouel Boudjnah chmo...@enovance.com wrote:
if peoples like this why don't we have it directly on the reviews?
Chmouel.
On Tue, Mar 4, 2014 at 10:00 PM, Carl Baldwin c...@ecbaldwin.net wrote:
Nachi,
Great! I'd been meaning to do something like this. I took yours and
tweaked it a bit to highlight failed Jenkins builds in red and grey
other Jenkins messages. Human reviews are left in blue.
javascript:(function(){
list = document.querySelectorAll('td.GJEA35ODGC');
for(i in list) {
title = list[i];
if(! title.innerHTML) { continue; }
text = title.nextSibling;
if (text.innerHTML.search('Build failed') 0) {
title.style.color='red'
} else if(title.innerHTML.search('Jenkins|CI|Ryu|Testing|Mine') = 0)
{
title.style.color='#66'
} else {
title.style.color='blue'
}
}
})()
Carl
On Wed, Feb 26, 2014 at 12:31 PM, Nachi Ueno na...@ntti3.com wrote:
Hi folks
I wrote an bookmarklet for neutron gerrit review.
This bookmarklet make the comment title for 3rd party ci as gray.
javascript:(function(){list =
document.querySelectorAll('td.GJEA35ODGC'); for(i in
list){if(!list[i].innerHTML){continue;};if(list[i].innerHTML
list[i].innerHTML.search('CI|Ryu|Testing|Mine')
0){list[i].style.color='#66'}else{list[i].style.color='red'}};})()
enjoy :)
Nachi
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa][neutron] API tests in the Neutron tree
At the last 2 summits, I've suggested that API tests could be maintained in the Neutron tree and reused by Tempest. I've finally submitted some patches that demonstrate this concept: https://review.openstack.org/#/c/72585/ (implements a unit test for the lifecycle of the network resource) https://review.openstack.org/#/c/72588/ (runs the test with tempest rest clients) My hope is to make API test maintenance a responsibility of the Neutron team. The API compatibility of each Neutron plugin has to be validated by Neutron tests anyway, and if the tests are structured as I am proposing, Tempest can reuse those efforts rather than duplicating them. I've added this topic to this week's agenda, and I would really appreciate it interested parties would take a look at the patches in question to prepare themselves to participate in the discussion. m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][neutron] API tests in the Neutron tree
On Feb 12, 2014, at 12:36 PM, Sean Dague s...@dague.net wrote: On 02/12/2014 01:48 PM, Maru Newby wrote: At the last 2 summits, I've suggested that API tests could be maintained in the Neutron tree and reused by Tempest. I've finally submitted some patches that demonstrate this concept: https://review.openstack.org/#/c/72585/ (implements a unit test for the lifecycle of the network resource) https://review.openstack.org/#/c/72588/ (runs the test with tempest rest clients) My hope is to make API test maintenance a responsibility of the Neutron team. The API compatibility of each Neutron plugin has to be validated by Neutron tests anyway, and if the tests are structured as I am proposing, Tempest can reuse those efforts rather than duplicating them. I've added this topic to this week's agenda, and I would really appreciate it interested parties would take a look at the patches in question to prepare themselves to participate in the discussion. m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Realistically, having API tests duplicated in the Tempest tree is a feature, not a bug. tempest/api is there for double book keep accounting, and it has been really effective at preventing accidental breakage of our APIs (which used to happen all the time), so I don't think putting API testing in neutron obviates that. Given how limited our testing resources are, might it be worth considering whether 'double-entry accounting' is actually the best way to preventing accidental breakage going forward? Might reasonable alternatives exist, such as clearly separating api tests from other tests in the neutron tree and giving review oversight only to qualified individuals? Today most projects (excepting swift... which I'll get to in a second) think about testing in 2 ways. Unit tests driven by tox in a venv, and tempest tests in a devstack environment. Because of this dualism, people have put increasingly more awkward live environments in the tox unit tests jobs. For instance, IIRC, neutron actually starts a full wsgi stack to tests every single in tree plugin, instead of just testing the driver call down path. Yeah, and this full-stack approach means that neutron tests are incredibly hard to maintain and debug, to say nothing of the time penalty (approaching the duration of a tempest smoke run). Part of the benefit of the proposal in question is to allow targeting the plugins directly with the same tests that will run against the full stack. m. Swift did something a little different. They have 3 classes of things. Unit tests, Tempest Tests, and Swift Functional tests. The Swift functional tests run in a devstack, but not with Tempest. Instead they run their own suite. This test suite only runs on the swift project, not on other projects, and it's something they can use to test functional scenarios that wouldn't fit in tempest, and extend to their heart's content, not having to worry about the interaction with other components, because it's only pushing on swift. Going down this third path with neutron testing is I think very valuable. Honestly, I'd like to encourage more projects to do this as well. It would give them a greater component assuredness before entering the integrated gate. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa][neutron] API tests in the Neutron tree
On Feb 12, 2014, at 1:59 PM, Sean Dague s...@dague.net wrote: On 02/12/2014 04:25 PM, Maru Newby wrote: On Feb 12, 2014, at 12:36 PM, Sean Dague s...@dague.net wrote: On 02/12/2014 01:48 PM, Maru Newby wrote: At the last 2 summits, I've suggested that API tests could be maintained in the Neutron tree and reused by Tempest. I've finally submitted some patches that demonstrate this concept: https://review.openstack.org/#/c/72585/ (implements a unit test for the lifecycle of the network resource) https://review.openstack.org/#/c/72588/ (runs the test with tempest rest clients) My hope is to make API test maintenance a responsibility of the Neutron team. The API compatibility of each Neutron plugin has to be validated by Neutron tests anyway, and if the tests are structured as I am proposing, Tempest can reuse those efforts rather than duplicating them. I've added this topic to this week's agenda, and I would really appreciate it interested parties would take a look at the patches in question to prepare themselves to participate in the discussion. m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Realistically, having API tests duplicated in the Tempest tree is a feature, not a bug. tempest/api is there for double book keep accounting, and it has been really effective at preventing accidental breakage of our APIs (which used to happen all the time), so I don't think putting API testing in neutron obviates that. Given how limited our testing resources are, might it be worth considering whether 'double-entry accounting' is actually the best way to preventing accidental breakage going forward? Might reasonable alternatives exist, such as clearly separating api tests from other tests in the neutron tree and giving review oversight only to qualified individuals? Our direct experience is that if we don't do this, within 2 weeks some project will have landed API breaking changes. This approach actually takes a lot of review load off the core reviewers, so reverting to a model which puts more work back on the review team (given the current review load), isn't something I think we want. Just so I'm clear, is there anything I could say that would change your mind? m. I get that there is a cost with this. But there is a cost of all of this. And because API tests should be write once for the API (and specifically not evolving), the upfront cost vs. the continually paid cost of review time in tree has been the better trade off. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Tempest][Production] Tempest / the gate / real world load
I'm afraid I missed this topic the first time around, and I think it bears revisiting. tl;dr: I think we should consider ensuring gate stability in the face of resource-starved services by some combination of more intelligent test design and better handling of resource starvation (for example, rate-limiting). Stress-testing would be more effective if it were explicitly focused on real-world usage scenarios and run separately from the gate. I think stress-testing is about the 'when' of failure, whereas the gate is about 'if'. I don't think it can be argued that OpenStack services (especially Neutron) can do better to ensure reliability under load. Running things in parallel in the gate shone a bright light on many problem areas and that was inarguably a good thing. Now that we have a better sense of the problem, though, it may be time to think about evolving our approach. From the perspective of gating commits, I think it makes sense to (a) minimize gate execution time and (b) provide some guarantees of reliability under reasonable load. I don't think either of these requires continuing to evaluate unrealistic usage scenarios against services running in a severely resource-starved environment. Every service eventually falls over when too much is asked of it. These kinds of failure are not likely to be particularly deterministic, so wouldn't it make sense to avoid triggering them in the gate as much as possible? In the specific case of Neutron, the current approach to testing isolation involves creating and tearing down networks at a tremendous rate. I'm not sure anyone can argue that this constitutes a usage scenario that is likely to appear in production, but because it causes problems in the gate, we've had to prioritize working on it over initiatives that might prove more useful to operators. While this may have been a necessary stop on the road to Neutron stability, I think it may be worth considering whether we want the gate to continue having an outsized role in defining optimization priorities. Thoughts? m. On Dec 12, 2013, at 11:23 AM, Robert Collins robe...@robertcollins.net wrote: A few times now we've run into patches for devstack-gate / devstack that change default configuration to handle 'tempest load'. For instance - https://review.openstack.org/61137 (Sorry Salvatore I'm not picking on you really!) So there appears to be a meme that the gate is particularly stressful - a bad environment - and that real world situations have less load. This could happen a few ways: (a) deployers might separate out components more; (b) they might have faster machines; (c) they might have less concurrent activity. (a) - unlikely! Deployers will cram stuff together as much as they can to save overheads. Big clouds will have components split out - yes, but they will also have correspondingly more load to drive that split out. (b) Perhaps, but not orders of magnitude faster, the clouds we run on are running on fairly recent hardware, and by using big instances we don't get crammed it with that many other tenants. (c) Almost certainly not. Tempest currently does a maximum of four concurrent requests. A small business cloud could easily have 5 or 6 people making concurrent requests from time to time, and bigger but not huge clouds will certainly have that. Their /average/ rate of API requests may be much lower, but when they point service orchestration tools at it -- particularly tools that walk their dependencies in parallel - load is going to be much much higher than what we generate with Tempest. tl;dr : if we need to change a config file setting in devstack-gate or devstack *other than* setting up the specific scenario, think thrice - should it be a production default and set in the relevant projects default config setting. Cheers, Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Neutron][qa] The 'spec' parameter of mock.patch()
I recently saw a case [1] where a misspelled assertion method (asoptt_called_once_with vs assert_called_once_with) did not result in a test failure because the object it was called on was created by mock.patch() without any of the spec/spec_set/autospec parameters being set. Might it make sense to require that calls to mock.patch() set autospec=True [2]? m. 1: https://review.openstack.org/#/c/61105/7/neutron/tests/unit/openvswitch/test_ovs_lib.py (line 162) 2: http://www.voidspace.org.uk/python/mock/patch.html#mock.patch ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] DHCP Agent Reliability
On Dec 13, 2013, at 8:06 PM, Isaku Yamahata isaku.yamah...@gmail.com wrote: On Fri, Dec 06, 2013 at 04:30:17PM +0900, Maru Newby ma...@redhat.com wrote: On Dec 5, 2013, at 5:21 PM, Isaku Yamahata isaku.yamah...@gmail.com wrote: On Wed, Dec 04, 2013 at 12:37:19PM +0900, Maru Newby ma...@redhat.com wrote: In the current architecture, the Neutron service handles RPC and WSGI with a single process and is prone to being overloaded such that agent heartbeats can be delayed beyond the limit for the agent being declared 'down'. Even if we increased the agent timeout as Yongsheg suggests, there is no guarantee that we can accurately detect whether an agent is 'live' with the current architecture. Given that amqp can ensure eventual delivery - it is a queue - is sending a notification blind such a bad idea? In the best case the agent isn't really down and can process the notification. In the worst case, the agent really is down but will be brought up eventually by a deployment's monitoring solution and process the notification when it returns. What am I missing? Do you mean overload of neutron server? Not neutron agent. So event agent sends periodic 'live' report, the reports are piled up unprocessed by server. When server sends notification, it considers agent dead wrongly. Not because agent didn't send live reports due to overload of agent. Is this understanding correct? Your interpretation is likely correct. The demands on the service are going to be much higher by virtue of having to field RPC requests from all the agents to interact with the database on their behalf. Is this strongly indicating thread-starvation. i.e. too much unfair thread scheduling. Given that eventlet is cooperative threading, should sleep(0) to hogging thread? I'm afraid that's a question for a profiler: https://github.com/colinhowe/eventlet_profiler m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Neutron] Cores - Prioritize merging migration fixes after tox change merges
As per Anita's email, we're not to approve anything until the following tox fix merges: https://review.openstack.org/#/c/60825 Please keep an eye on the change, and once it merges, make sure that the following patches merge before regular approval rules resume: https://review.openstack.org/#/c/61677 https://review.openstack.org/#/c/61663 Without these migration patches, devstack is broken for neutron. Thanks! Maru ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] DHCP Agent Reliability
On Dec 10, 2013, at 4:47 PM, Isaku Yamahata isaku.yamah...@gmail.com wrote:
On Tue, Dec 10, 2013 at 07:28:10PM +1300,
Robert Collins robe...@robertcollins.net wrote:
On 10 December 2013 19:16, Isaku Yamahata isaku.yamah...@gmail.com wrote:
Answering myself. If connection is closed, it will reconnects automatically
at rpc layer. See neutron.openstack.common.rpc.impl_{kombu, qpid}.py.
So notifications during reconnects can be lost if AMQP service is set
to discard notifications during no subscriber.
Which is fine: the agent repulls the full set it's running on that
machine, and life goes on.
On what event?
Polling in agent seems effectively disabled by self.needs_resync with
the current code.
If the agent is not connected, it is either down (needs_resync will be set to
True on launch) or experiencing a loss of connectivity to the amqp service
(needs_resync will have been set to True on error). The loss of notifications
is not a problem in either case.
m.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] DHCP Agent Reliability
On Dec 5, 2013, at 4:43 PM, Édouard Thuleau thul...@gmail.com wrote: There also another bug you can link/duplicate with #1192381 is https://bugs.launchpad.net/neutron/+bug/1185916. I proposed a fix but it's not the good way. I abandoned it. Édouard. Thank you for pointing this out! m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] DHCP Agent Reliability
On Dec 10, 2013, at 6:36 PM, Isaku Yamahata isaku.yamah...@gmail.com wrote: On Mon, Dec 09, 2013 at 08:43:59AM +1300, Robert Collins robe...@robertcollins.net wrote: listening: when an agent connects after an outage, it first starts listening, then does a poll for updates it missed. Are you suggesting that processing of notifications and full state synchronization are able to cooperate safely? Or hoping that it will be so in the future? I'm saying that you can avoid race conditions by a combination of 'subscribe to changes' + 'give me the full state'. Like this? https://review.openstack.org/#/c/61057/ This patch is just to confirm the idea. I'm afraid the proposed patch is no more reliable than the current approach of using file-based locking. I am working on an alternate patch that puts the rpc event loop in the dhcp agent so that better coordination between full synchronization and notification handling is possible. This approach has already been taken with the L3 agent and work on the L2 agent is in process. m. -- Isaku Yamahata isaku.yamah...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] DHCP Agent Reliability
On Dec 11, 2013, at 8:39 AM, Isaku Yamahata isaku.yamah...@gmail.com wrote: On Wed, Dec 11, 2013 at 01:23:36AM +0900, Maru Newby ma...@redhat.com wrote: On Dec 10, 2013, at 6:36 PM, Isaku Yamahata isaku.yamah...@gmail.com wrote: On Mon, Dec 09, 2013 at 08:43:59AM +1300, Robert Collins robe...@robertcollins.net wrote: listening: when an agent connects after an outage, it first starts listening, then does a poll for updates it missed. Are you suggesting that processing of notifications and full state synchronization are able to cooperate safely? Or hoping that it will be so in the future? I'm saying that you can avoid race conditions by a combination of 'subscribe to changes' + 'give me the full state'. Like this? https://review.openstack.org/#/c/61057/ This patch is just to confirm the idea. I'm afraid the proposed patch is no more reliable than the current approach of using file-based locking. I am working on an alternate patch that puts the rpc event loop in the dhcp agent so that better coordination between full synchronization and notification handling is possible. This approach has already been taken with the L3 agent and work on the L2 agent is in process. You objected against agent polling in the discussion. But you're now proposing polling now. Did you change your mind? Uh, no. I'm proposing better coordination between notification processing and full state synchronization beyond simple exclusionary primitives (utils.synchronize etc). I apologize if my language was unclear. m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo][neutron] Post-mortem debugging for tests?
Are any other projects interested in adding back the post-mortem debugging support we lost in the move away from nose? I have a patch in review for neutron and salv-orlando asked whether oslo might be the better place for it. https://review.openstack.org/#/c/43776/ m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] DHCP Agent Reliability
On Dec 7, 2013, at 6:21 PM, Robert Collins robe...@robertcollins.net wrote: On 7 December 2013 21:53, Isaku Yamahata isaku.yamah...@gmail.com wrote: Case 3: Hardware failure. So an agent on the node is gone. Another agent will run on another node. If AMQP service is set up not to lose notification, notifications will be piled up and stress AMQP service. I would say single node failure isn't catastrophic. So we should have AMQP set to discard notifications if there is noone What are the semantics of AMQP discarding notifications when a consumer is no longer present? Can this be relied upon to ensure that potentially stale notifications do not remain in the queue when an agent restarts? listening: when an agent connects after an outage, it first starts listening, then does a poll for updates it missed. Are you suggesting that processing of notifications and full state synchronization are able to cooperate safely? Or hoping that it will be so in the future? m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] request-id in API response
On Dec 3, 2013, at 12:18 AM, Joe Gordon joe.gord...@gmail.com wrote: On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson m...@not.mn wrote: Just to add to the story, Swift uses X-Trans-Id and generates it in the outer-most catch_errors middleware. Swift's catch errors middleware is responsible for ensuring that the transaction id exists on each request, and that all errors previously uncaught, anywhere in the pipeline, are caught and logged. If there is not a common way to do this, yet, I submit it as a great template for solving this problem. It's simple, scalable, and well-tested (ie tests and running in prod for years). https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py Leaving aside error handling and only focusing on the transaction id (or request id) generation, since OpenStack services are exposed to untrusted clients, how would you propose communicating the appropriate transaction id to a different service? I can see great benefit to having a glance transaction ID carry through to Swift requests (and so on), but how should the transaction id be communicated? It's not sensitive info, but I can imagine a pretty big problem when trying to track down errors if a client application decides to set eg the X-Set-Transaction-Id header on every request to the same thing. -1 to cross service request IDs, for the reasons John mentions above. Thanks for bringing this up, and I'd welcome a patch in Swift that would use a common library to generate the transaction id, if it were installed. I can see that there would be huge advantage to operators to trace requests through multiple systems. Another option would be for each system that calls an another OpenStack system to expect and log the transaction ID for the request that was given. This would be looser coupling and be more forgiving for a heterogeneous cluster. Eg when Glance makes a call to Swift, Glance cloud log the transaction id that Swift used (from the Swift response). Likewise, when Swift makes a call to Keystone, Swift could log the Keystone transaction id. This wouldn't result in a single transaction id across all systems, but it would provide markers so an admin could trace the request. There was a session on this at the summit, and although the notes are a little scarce this was the conclusion we came up with. Every time a cross service call is made, we will log and send a notification for ceilometer to consume, with the request-ids of both request ids. One of the benefits of this approach is that we can easily generate a tree of all the API calls that are made (and clearly show when multiple calls are made to the same service), something that just a cross service request id would have trouble with. Is wise to trust anything a client provides to ensure traceability? If a user receives a request id back from Nova, then submits that request id in an unrelated request to Neutron, the traceability would be effectively corrupted. If the consensus is that we don't want to securely deliver request ids for inter-service calls, how about requiring a service to log its request id along with the request id returned from a call to another service to achieve the a similar result? The catch is that every call point (or client instantiation?) would have to be modified to pass the request id instead of just logging at one place in each service. Is that a cost worth paying? m. https://etherpad.openstack.org/p/icehouse-summit-qa-gate-debugability With that in mind I think having a standard x-openstack-request-id makes things a little more uniform, and means that adding new services doesn't require new logic to handle new request ids. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] DHCP Agent Reliability
On Dec 5, 2013, at 6:43 AM, Carl Baldwin c...@ecbaldwin.net wrote: I have offered up https://review.openstack.org/#/c/60082/ as a backport to Havana. Interest was expressed in the blueprint for doing this even before this thread. If there is consensus for this as the stop-gap then it is there for the merging. However, I do not want to discourage discussion of other stop-gap solutions like what Maru proposed in the original post. Carl Awesome. No worries, I'm still planning on submitting a patch to improve notification reliability. We seem to be cpu bound now in processing RPC messages. Do you think it would be reasonable to run multiple processes for RPC? m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] DHCP Agent Reliability
On Dec 5, 2013, at 5:21 PM, Isaku Yamahata isaku.yamah...@gmail.com wrote: On Wed, Dec 04, 2013 at 12:37:19PM +0900, Maru Newby ma...@redhat.com wrote: In the current architecture, the Neutron service handles RPC and WSGI with a single process and is prone to being overloaded such that agent heartbeats can be delayed beyond the limit for the agent being declared 'down'. Even if we increased the agent timeout as Yongsheg suggests, there is no guarantee that we can accurately detect whether an agent is 'live' with the current architecture. Given that amqp can ensure eventual delivery - it is a queue - is sending a notification blind such a bad idea? In the best case the agent isn't really down and can process the notification. In the worst case, the agent really is down but will be brought up eventually by a deployment's monitoring solution and process the notification when it returns. What am I missing? Do you mean overload of neutron server? Not neutron agent. So event agent sends periodic 'live' report, the reports are piled up unprocessed by server. When server sends notification, it considers agent dead wrongly. Not because agent didn't send live reports due to overload of agent. Is this understanding correct? Your interpretation is likely correct. The demands on the service are going to be much higher by virtue of having to field RPC requests from all the agents to interact with the database on their behalf. Please consider that while a good solution will track notification delivery and success, we may need 2 solutions: 1. A 'good-enough', minimally-invasive stop-gap that can be back-ported to grizzly and havana. How about twisting DhcpAgent._periodic_resync_helper? If no notification is received form server from last sleep, it calls self.sync_state() even if self.needs_resync = False. Thus the inconsistency between agent and server due to losing notification will be fixed. Unless I'm missing something, wouldn't forcing more and potentially unnecessary resyncs increase the load on the Neutron service and negatively impact reliability? 2. A 'best-effort' refactor that maximizes the reliability of the DHCP agent. I'm hoping that coming up with a solution to #1 will allow us the breathing room to work on #2 in this cycle. Loss of notifications is somewhat inevitable, I think. (Or logging tasks to stable storage shared between server and agent) And Unconditionally sending notifications would cause problem. Regarding sending notifications unconditionally, what specifically are you worried about? I can imagine 2 scenarios: Case 1: Send notification to an agent that is incorrectly reported as down. Result: Agent receives notification and acts on it. Case 2: Send notification to an agent that is actually down. Result: Agent comes up eventually (in a production environment this should be a given) and calls sync_state(). We definitely need to make sync_state more reliable, though (I discuss the specifics later in this message). Notifications could of course be dropped if AMQP queues are not persistent and are lost, but I don't think there needs to be a code-based remedy for this. An operator is likely to deploy the AMQP service in HA to prevent the queues from being lost, and know to restart everything in the event of catastrophic failure. That's not to say we don't have work to do, though. An agent is responsible for communicating resource state changes to the service, but the service neither detects nor reacts when the state of a resource is scheduled to change and fails to do so in a reasonable timeframe. Thus, as in the bug that prompted this discussion, it is up to the user to detect the failure (a VM without connectivity). Ideally, Neutron should be tracking resource state changes with sufficient detail and reviewing them periodically to allow timely failure detection and remediation. However, such a change is unlikely to be a candidate for backport so it will have to wait. You mentioned agent crash. Server crash should also be taken care of for reliability. Admin also sometimes wants to restart neutron server/agents for some reasons. Agent can crash after receiving notifications before start processing actual tasks. Server can crash after commiting changes to DB before sending notifications. In such cases, notification will be lost. Polling to resync would be necessary somewhere. Agreed, we need to consider the cases of both agent and service failure. In the case of service failure, thanks to recently merged patches, the dhcp agent will at least force a resync in the event of an error in communicating with the server. However, there is no guarantee that the agent will communicate with the server during the downtime. While polling is one possible solution, might it be preferable for the service to simply notify the agents when it starts? The dhcp agent can already receive
Re: [openstack-dev] request-id in API response
On Dec 6, 2013, at 1:09 AM, John Dickinson m...@not.mn wrote: On Dec 5, 2013, at 1:36 AM, Maru Newby ma...@redhat.com wrote: On Dec 3, 2013, at 12:18 AM, Joe Gordon joe.gord...@gmail.com wrote: On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson m...@not.mn wrote: Just to add to the story, Swift uses X-Trans-Id and generates it in the outer-most catch_errors middleware. Swift's catch errors middleware is responsible for ensuring that the transaction id exists on each request, and that all errors previously uncaught, anywhere in the pipeline, are caught and logged. If there is not a common way to do this, yet, I submit it as a great template for solving this problem. It's simple, scalable, and well-tested (ie tests and running in prod for years). https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py Leaving aside error handling and only focusing on the transaction id (or request id) generation, since OpenStack services are exposed to untrusted clients, how would you propose communicating the appropriate transaction id to a different service? I can see great benefit to having a glance transaction ID carry through to Swift requests (and so on), but how should the transaction id be communicated? It's not sensitive info, but I can imagine a pretty big problem when trying to track down errors if a client application decides to set eg the X-Set-Transaction-Id header on every request to the same thing. -1 to cross service request IDs, for the reasons John mentions above. Thanks for bringing this up, and I'd welcome a patch in Swift that would use a common library to generate the transaction id, if it were installed. I can see that there would be huge advantage to operators to trace requests through multiple systems. Another option would be for each system that calls an another OpenStack system to expect and log the transaction ID for the request that was given. This would be looser coupling and be more forgiving for a heterogeneous cluster. Eg when Glance makes a call to Swift, Glance cloud log the transaction id that Swift used (from the Swift response). Likewise, when Swift makes a call to Keystone, Swift could log the Keystone transaction id. This wouldn't result in a single transaction id across all systems, but it would provide markers so an admin could trace the request. There was a session on this at the summit, and although the notes are a little scarce this was the conclusion we came up with. Every time a cross service call is made, we will log and send a notification for ceilometer to consume, with the request-ids of both request ids. One of the benefits of this approach is that we can easily generate a tree of all the API calls that are made (and clearly show when multiple calls are made to the same service), something that just a cross service request id would have trouble with. Is wise to trust anything a client provides to ensure traceability? If a user receives a request id back from Nova, then submits that request id in an unrelated request to Neutron, the traceability would be effectively corrupted. If the consensus is that we don't want to securely deliver request ids for inter-service calls, how about requiring a service to log its request id along with the request id returned from a call to another service to achieve the a similar result? Yes, this is what I was proposing. I think this is the best path forward. Ok, great. And as per your suggestion, a middleware-based error handler will soon be proposed for oslo that will secondarily ensure that a request id is added to the request. The catch is that every call point (or client instantiation?) would have to be modified to pass the request id instead of just logging at one place in each service. Is that a cost worth paying? Perhaps this is my ignorance of how other projects work today, but does this not already happen? Is it possible to get a response from an API call to an OpenStack project that doesn't include a request id? We'll have it in Neutron real-soon-now, and then I think the answer will be 'yes'. On reflection, it should be easy enough for a given service to ensure that calls to other services are automatically instrumented to log request id pairs. Again, probably something for oslo. m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] DHCP Agent Reliability
On Dec 4, 2013, at 8:55 AM, Carl Baldwin c...@ecbaldwin.net wrote: Stephen, all, I agree that there may be some opportunity to split things out a bit. However, I'm not sure what the best way will be. I recall that Mark mentioned breaking out the processes that handle API requests and RPC from each other at the summit. Anyway, it is something that has been discussed. I actually wanted to point out that the neutron server now has the ability to run a configurable number of sub-processes to handle a heavier load. Introduced with this commit: https://review.openstack.org/#/c/37131/ Set api_workers to something 1 and restart the server. The server can also be run on more than one physical host in combination with multiple child processes. I completely misunderstood the import of the commit in question. Being able to run the wsgi server(s) out of process is a nice improvement, thank you for making it happen. Has there been any discussion around making the default for api_workers 0 (at least 1) to ensure that the default configuration separates wsgi and rpc load? This also seems like a great candidate for backporting to havana and maybe even grizzly, although api_workers should probably be defaulted to 0 in those cases. FYI, I re-ran the test that attempted to boot 75 micro VM's simultaneously with api_workers = 2, with mixed results. The increased wsgi throughput resulted in almost half of the boot requests failing with 500 errors due to QueuePool errors (https://bugs.launchpad.net/neutron/+bug/1160442) in Neutron. It also appears that maximizing the number of wsgi requests has the side-effect of increasing the RPC load on the main process, and this means that the problem of dhcp notifications being dropped is little improved. I intend to submit a fix that ensures that notifications are sent regardless of agent status, in any case. m. Carl On Tue, Dec 3, 2013 at 9:47 AM, Stephen Gran stephen.g...@theguardian.com wrote: On 03/12/13 16:08, Maru Newby wrote: I've been investigating a bug that is preventing VM's from receiving IP addresses when a Neutron service is under high load: https://bugs.launchpad.net/neutron/+bug/1192381 High load causes the DHCP agent's status updates to be delayed, causing the Neutron service to assume that the agent is down. This results in the Neutron service not sending notifications of port addition to the DHCP agent. At present, the notifications are simply dropped. A simple fix is to send notifications regardless of agent status. Does anybody have any objections to this stop-gap approach? I'm not clear on the implications of sending notifications to agents that are down, but I'm hoping for a simple fix that can be backported to both havana and grizzly (yes, this bug has been with us that long). Fixing this problem for real, though, will likely be more involved. The proposal to replace the current wsgi framework with Pecan may increase the Neutron service's scalability, but should we continue to use a 'fire and forget' approach to notification? Being able to track the success or failure of a given action outside of the logs would seem pretty important, and allow for more effective coordination with Nova than is currently possible. It strikes me that we ask an awful lot of a single neutron-server instance - it has to take state updates from all the agents, it has to do scheduling, it has to respond to API requests, and it has to communicate about actual changes with the agents. Maybe breaking some of these out the way nova has a scheduler and a conductor and so on might be a good model (I know there are things people are unhappy about with nova-scheduler, but imagine how much worse it would be if it was built into the API). Doing all of those tasks, and doing it largely single threaded, is just asking for overload. Cheers, -- Stephen Gran Senior Systems Integrator - theguardian.com Please consider the environment before printing this email. -- Visit theguardian.com On your mobile, download the Guardian iPhone app theguardian.com/iphone and our iPad edition theguardian.com/iPad Save up to 33% by subscribing to the Guardian and Observer - choose the papers you want and get full digital access. Visit subscribe.theguardian.com This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News Media Limited A member of Guardian Media Group plc
Re: [openstack-dev] [Openstack][qa][Tempest][Network][Neutron] tenant_networks_reachable
On Dec 4, 2013, at 6:34 PM, Yair Fried yfr...@redhat.com wrote: Hi In tempest.conf, Is the parameter tenant_networks_reachable affected by anything other than neutron using ip name-spaces (which is the default setting) or not, and if not using it - if the tempest host is the same as the neutron server host? It is entirely possible for a tenant network to be configured to be reachable/routable. The common case, though, is for a tenant network to be non-routable and isolated by network namespaces. Even if tempest were to run on the network controller, networks isolated by network namespaces would not be reachable. m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Neutron] DHCP Agent Reliability
I've been investigating a bug that is preventing VM's from receiving IP addresses when a Neutron service is under high load: https://bugs.launchpad.net/neutron/+bug/1192381 High load causes the DHCP agent's status updates to be delayed, causing the Neutron service to assume that the agent is down. This results in the Neutron service not sending notifications of port addition to the DHCP agent. At present, the notifications are simply dropped. A simple fix is to send notifications regardless of agent status. Does anybody have any objections to this stop-gap approach? I'm not clear on the implications of sending notifications to agents that are down, but I'm hoping for a simple fix that can be backported to both havana and grizzly (yes, this bug has been with us that long). Fixing this problem for real, though, will likely be more involved. The proposal to replace the current wsgi framework with Pecan may increase the Neutron service's scalability, but should we continue to use a 'fire and forget' approach to notification? Being able to track the success or failure of a given action outside of the logs would seem pretty important, and allow for more effective coordination with Nova than is currently possible. Maru ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Neutron] Tuning QueuePool parameters?
I recently ran into this bug while trying to concurrently boot a large number (75) of VMs: https://bugs.launchpad.net/neutron/+bug/1160442 I see that the fix for the bug added configuration of SQLAlchemy QueuePool parameters that should prevent the boot failures I was seeing. However, I don't see a good explanation on the bug as to what values to set the configuration to or why the defaults weren't updated to something sane. If that information is available somewhere, please share! I'm not sure why this bug is considered fixed if it's still possible to trigger it with no clear path to resolution. Maru ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] DHCP Agent Reliability
On Dec 4, 2013, at 1:47 AM, Stephen Gran stephen.g...@theguardian.com wrote: On 03/12/13 16:08, Maru Newby wrote: I've been investigating a bug that is preventing VM's from receiving IP addresses when a Neutron service is under high load: https://bugs.launchpad.net/neutron/+bug/1192381 High load causes the DHCP agent's status updates to be delayed, causing the Neutron service to assume that the agent is down. This results in the Neutron service not sending notifications of port addition to the DHCP agent. At present, the notifications are simply dropped. A simple fix is to send notifications regardless of agent status. Does anybody have any objections to this stop-gap approach? I'm not clear on the implications of sending notifications to agents that are down, but I'm hoping for a simple fix that can be backported to both havana and grizzly (yes, this bug has been with us that long). Fixing this problem for real, though, will likely be more involved. The proposal to replace the current wsgi framework with Pecan may increase the Neutron service's scalability, but should we continue to use a 'fire and forget' approach to notification? Being able to track the success or failure of a given action outside of the logs would seem pretty important, and allow for more effective coordination with Nova than is currently possible. It strikes me that we ask an awful lot of a single neutron-server instance - it has to take state updates from all the agents, it has to do scheduling, it has to respond to API requests, and it has to communicate about actual changes with the agents. Maybe breaking some of these out the way nova has a scheduler and a conductor and so on might be a good model (I know there are things people are unhappy about with nova-scheduler, but imagine how much worse it would be if it was built into the API). Doing all of those tasks, and doing it largely single threaded, is just asking for overload. I'm sorry if it wasn't clear in my original message, but my primary concern lies with the reliability rather than the scalability of the Neutron service. Carl's addition of multiple workers is a good stop-gap to minimize the impact of blocking IO calls in the current architecture, and we already have consensus on the need to separate RPC and WSGI functions as part of the Pecan rewrite. I am worried, though, that we are not being sufficiently diligent in how we manage state transitions through notifications. Managing transitions and their associate error states is needlessly complicated by the current ad-hoc approach, and I'd appreciate input on the part of distributed systems experts as to how we could do better. m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] DHCP Agent Reliability
On Dec 4, 2013, at 11:02 AM, Yongsheng Gong gong...@unitedstack.com wrote: another way is to have a large agent_down_time, by default it is 9 secs. I don't believe that increasing the timeout by itself is a good solution. Relying on the agent state to know whether to send a notification has simply proven unreliable with the current architecture of a poorly-performing single process server handling both RPC and WSGI. m. On Wed, Dec 4, 2013 at 7:55 AM, Carl Baldwin c...@ecbaldwin.net wrote: Stephen, all, I agree that there may be some opportunity to split things out a bit. However, I'm not sure what the best way will be. I recall that Mark mentioned breaking out the processes that handle API requests and RPC from each other at the summit. Anyway, it is something that has been discussed. I actually wanted to point out that the neutron server now has the ability to run a configurable number of sub-processes to handle a heavier load. Introduced with this commit: https://review.openstack.org/#/c/37131/ Set api_workers to something 1 and restart the server. The server can also be run on more than one physical host in combination with multiple child processes. Carl On Tue, Dec 3, 2013 at 9:47 AM, Stephen Gran stephen.g...@theguardian.com wrote: On 03/12/13 16:08, Maru Newby wrote: I've been investigating a bug that is preventing VM's from receiving IP addresses when a Neutron service is under high load: https://bugs.launchpad.net/neutron/+bug/1192381 High load causes the DHCP agent's status updates to be delayed, causing the Neutron service to assume that the agent is down. This results in the Neutron service not sending notifications of port addition to the DHCP agent. At present, the notifications are simply dropped. A simple fix is to send notifications regardless of agent status. Does anybody have any objections to this stop-gap approach? I'm not clear on the implications of sending notifications to agents that are down, but I'm hoping for a simple fix that can be backported to both havana and grizzly (yes, this bug has been with us that long). Fixing this problem for real, though, will likely be more involved. The proposal to replace the current wsgi framework with Pecan may increase the Neutron service's scalability, but should we continue to use a 'fire and forget' approach to notification? Being able to track the success or failure of a given action outside of the logs would seem pretty important, and allow for more effective coordination with Nova than is currently possible. It strikes me that we ask an awful lot of a single neutron-server instance - it has to take state updates from all the agents, it has to do scheduling, it has to respond to API requests, and it has to communicate about actual changes with the agents. Maybe breaking some of these out the way nova has a scheduler and a conductor and so on might be a good model (I know there are things people are unhappy about with nova-scheduler, but imagine how much worse it would be if it was built into the API). Doing all of those tasks, and doing it largely single threaded, is just asking for overload. Cheers, -- Stephen Gran Senior Systems Integrator - theguardian.com Please consider the environment before printing this email. -- Visit theguardian.com On your mobile, download the Guardian iPhone app theguardian.com/iphone and our iPad edition theguardian.com/iPad Save up to 33% by subscribing to the Guardian and Observer - choose the papers you want and get full digital access. Visit subscribe.theguardian.com This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News Media Limited A member of Guardian Media Group plc Registered Office PO Box 68164 Kings Place 90 York Way London N1P 2AP Registered in England Number 908396 -- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list
Re: [openstack-dev] [Neutron] DHCP Agent Reliability
On Dec 4, 2013, at 11:57 AM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Maru Newby's message of 2013-12-03 08:08:09 -0800: I've been investigating a bug that is preventing VM's from receiving IP addresses when a Neutron service is under high load: https://bugs.launchpad.net/neutron/+bug/1192381 High load causes the DHCP agent's status updates to be delayed, causing the Neutron service to assume that the agent is down. This results in the Neutron service not sending notifications of port addition to the DHCP agent. At present, the notifications are simply dropped. A simple fix is to send notifications regardless of agent status. Does anybody have any objections to this stop-gap approach? I'm not clear on the implications of sending notifications to agents that are down, but I'm hoping for a simple fix that can be backported to both havana and grizzly (yes, this bug has been with us that long). Fixing this problem for real, though, will likely be more involved. The proposal to replace the current wsgi framework with Pecan may increase the Neutron service's scalability, but should we continue to use a 'fire and forget' approach to notification? Being able to track the success or failure of a given action outside of the logs would seem pretty important, and allow for more effective coordination with Nova than is currently possible. Dropping requests without triggering a user-visible error is a pretty serious problem. You didn't mention if you have filed a bug about that. If not, please do or let us know here so we can investigate and file a bug. There is a bug linked to in the original message that I am already working on. The fact that that bug title is 'dhcp agent doesn't configure ports' rather than 'dhcp notifications are silently dropped' is incidental. It seems to me that they should be put into a queue to be retried. Sending the notifications blindly is almost as bad as dropping them, as you have no idea if the agent is alive or not. This is more the kind of discussion I was looking for. In the current architecture, the Neutron service handles RPC and WSGI with a single process and is prone to being overloaded such that agent heartbeats can be delayed beyond the limit for the agent being declared 'down'. Even if we increased the agent timeout as Yongsheg suggests, there is no guarantee that we can accurately detect whether an agent is 'live' with the current architecture. Given that amqp can ensure eventual delivery - it is a queue - is sending a notification blind such a bad idea? In the best case the agent isn't really down and can process the notification. In the worst case, the agent really is down but will be brought up eventually by a deployment's monitoring solution and process the notification when it returns. What am I missing? Please consider that while a good solution will track notification delivery and success, we may need 2 solutions: 1. A 'good-enough', minimally-invasive stop-gap that can be back-ported to grizzly and havana. 2. A 'best-effort' refactor that maximizes the reliability of the DHCP agent. I'm hoping that coming up with a solution to #1 will allow us the breathing room to work on #2 in this cycle. m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Stop logging non-exceptional conditions as ERROR
On Dec 2, 2013, at 10:19 PM, Joe Gordon joe.gord...@gmail.com wrote: On Dec 2, 2013 3:39 AM, Maru Newby ma...@redhat.com wrote: On Dec 2, 2013, at 2:07 AM, Anita Kuno ante...@anteaya.info wrote: Great initiative putting this plan together, Maru. Thanks for doing this. Thanks for volunteering to help, Salvatore (I'm thinking of asking for you to be cloned - once that becomes available.) if you add your patch urls (as you create them) to the blueprint Maru started [0] that would help to track the work. Armando, thanks for doing this work as well. Could you add the urls of the patches you reference to the exceptional-conditions blueprint? For icehouse-1 to be a realistic goal for this assessment and clean-up, patches for this would need to be up by Tuesday Dec. 3 at the latest (does 13:00 UTC sound like a reasonable target?) so that they can make it through review and check testing, gate testing and merging prior to the Thursday Dec. 5 deadline for icehouse-1. I would really like to see this, I just want the timeline to be conscious. My mistake, getting this done by Tuesday does not seem realistic. icehouse-2, then. With icehouse-2 being the nova-network feature freeze reevaluation point (possibly lifting it) I think gating on new stacktraces by icehouse-2 is too late. Even a huge whitelist of errors is better then letting new errors in. No question that it needs to happen asap. If we're talking about milestones, though, and icehouse-1 patches need to be in by Tuesday, I don't think icehouse-1 is realistic. It will have to be early in icehouse-2. m. m. I would like to say talk to me tomorrow in -neutron to ensure you are getting the support you need to achieve this but I will be flying (wifi uncertain). I do hope that some additional individuals come forward to help with this. Thanks Maru, Salvatore and Armando, Anita. [0] https://blueprints.launchpad.net/neutron/+spec/log-only-exceptional-conditions-as-error On 11/30/2013 08:24 PM, Maru Newby wrote: On Nov 28, 2013, at 1:08 AM, Salvatore Orlando sorla...@nicira.com wrote: Thanks Maru, This is something my team had on the backlog for a while. I will push some patches to contribute towards this effort in the next few days. Let me know if you're already thinking of targeting the completion of this job for a specific deadline. I'm thinking this could be a task for those not involved in fixing race conditions, and be done in parallel. I guess that would be for icehouse-1 then? My hope would be that the early signs of race conditions would then be caught earlier. m. Salvatore On 27 November 2013 17:50, Maru Newby ma...@redhat.com wrote: Just a heads up, the console output for neutron gate jobs is about to get a lot noisier. Any log output that contains 'ERROR' is going to be dumped into the console output so that we can identify and eliminate unnecessary error logging. Once we've cleaned things up, the presence of unexpected (non-whitelisted) error output can be used to fail jobs, as per the following Tempest blueprint: https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors I've filed a related Neutron blueprint for eliminating the unnecessary error logging: https://blueprints.launchpad.net/neutron/+spec/log-only-exceptional-conditions-as-error I'm looking for volunteers to help with this effort, please reply in this thread if you're willing to assist. Thanks, Maru ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Stop logging non-exceptional conditions as ERROR
On Dec 2, 2013, at 2:07 AM, Anita Kuno ante...@anteaya.info wrote: Great initiative putting this plan together, Maru. Thanks for doing this. Thanks for volunteering to help, Salvatore (I'm thinking of asking for you to be cloned - once that becomes available.) if you add your patch urls (as you create them) to the blueprint Maru started [0] that would help to track the work. Armando, thanks for doing this work as well. Could you add the urls of the patches you reference to the exceptional-conditions blueprint? For icehouse-1 to be a realistic goal for this assessment and clean-up, patches for this would need to be up by Tuesday Dec. 3 at the latest (does 13:00 UTC sound like a reasonable target?) so that they can make it through review and check testing, gate testing and merging prior to the Thursday Dec. 5 deadline for icehouse-1. I would really like to see this, I just want the timeline to be conscious. My mistake, getting this done by Tuesday does not seem realistic. icehouse-2, then. m. I would like to say talk to me tomorrow in -neutron to ensure you are getting the support you need to achieve this but I will be flying (wifi uncertain). I do hope that some additional individuals come forward to help with this. Thanks Maru, Salvatore and Armando, Anita. [0] https://blueprints.launchpad.net/neutron/+spec/log-only-exceptional-conditions-as-error On 11/30/2013 08:24 PM, Maru Newby wrote: On Nov 28, 2013, at 1:08 AM, Salvatore Orlando sorla...@nicira.com wrote: Thanks Maru, This is something my team had on the backlog for a while. I will push some patches to contribute towards this effort in the next few days. Let me know if you're already thinking of targeting the completion of this job for a specific deadline. I'm thinking this could be a task for those not involved in fixing race conditions, and be done in parallel. I guess that would be for icehouse-1 then? My hope would be that the early signs of race conditions would then be caught earlier. m. Salvatore On 27 November 2013 17:50, Maru Newby ma...@redhat.com wrote: Just a heads up, the console output for neutron gate jobs is about to get a lot noisier. Any log output that contains 'ERROR' is going to be dumped into the console output so that we can identify and eliminate unnecessary error logging. Once we've cleaned things up, the presence of unexpected (non-whitelisted) error output can be used to fail jobs, as per the following Tempest blueprint: https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors I've filed a related Neutron blueprint for eliminating the unnecessary error logging: https://blueprints.launchpad.net/neutron/+spec/log-only-exceptional-conditions-as-error I'm looking for volunteers to help with this effort, please reply in this thread if you're willing to assist. Thanks, Maru ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] request-id in API response
On Nov 30, 2013, at 1:00 AM, Sean Dague s...@dague.net wrote: On 11/29/2013 10:33 AM, Jay Pipes wrote: On 11/28/2013 07:45 AM, Akihiro Motoki wrote: Hi, I am working on adding request-id to API response in Neutron. After I checked what header is used in other projects header name varies project by project. It seems there is no consensus what header is recommended and it is better to have some consensus. nova: x-compute-request-id cinder: x-compute-request-id glance: x-openstack-request-id neutron: x-network-request-id (under review) request-id is assigned and used inside of each project now, so x-service-request-id looks good. On the other hand, if we have a plan to enhance request-id across projects, x-openstack-request-id looks better. My vote is for: x-openstack-request-id With an implementation of create a request UUID if none exists yet in some standardized WSGI middleware... Agreed. I don't think I see any value in having these have different service names, having just x-openstack-request-id across all the services seems a far better idea, and come back through and fix nova and cinder to be that as well. +1 An openstack request id should be service agnostic to allow tracking of a request across many services (e.g. a call to nova to boot a VM should generate a request id that is provided to other services in requests to provision said VM). All services would ideally share a facility for generating new request ids and for securely accepting request ids from other services. m. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Stop logging non-exceptional conditions as ERROR
On Nov 28, 2013, at 1:08 AM, Salvatore Orlando sorla...@nicira.com wrote: Thanks Maru, This is something my team had on the backlog for a while. I will push some patches to contribute towards this effort in the next few days. Let me know if you're already thinking of targeting the completion of this job for a specific deadline. I'm thinking this could be a task for those not involved in fixing race conditions, and be done in parallel. I guess that would be for icehouse-1 then? My hope would be that the early signs of race conditions would then be caught earlier. m. Salvatore On 27 November 2013 17:50, Maru Newby ma...@redhat.com wrote: Just a heads up, the console output for neutron gate jobs is about to get a lot noisier. Any log output that contains 'ERROR' is going to be dumped into the console output so that we can identify and eliminate unnecessary error logging. Once we've cleaned things up, the presence of unexpected (non-whitelisted) error output can be used to fail jobs, as per the following Tempest blueprint: https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors I've filed a related Neutron blueprint for eliminating the unnecessary error logging: https://blueprints.launchpad.net/neutron/+spec/log-only-exceptional-conditions-as-error I'm looking for volunteers to help with this effort, please reply in this thread if you're willing to assist. Thanks, Maru ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Neutron] Rename 'tenant' to 'project' as per Keystone?
Keystone is almost finished replacing the term 'tenant' with 'project' (see recent thread https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg09709.html) and we might want to think about how and when Neutron makes a similar transition. It's an unlikely priority in the near term given the focus on stability, but I wanted to broach the topic for discussion in case people think it might be worth attempting later in the cycle. I've filed a preliminary blueprint in any case: https://blueprints.launchpad.net/neutron/+spec/rename-tenant-to-project Maru ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Neutron] Tenant isolation gate failures?
The tenant isolation gates that have been failing so frequently seem to be passing all of a sudden. I didn't see any merges that claimed to fix the issue, so maybe this is just a lull due to a lower volume of gate jobs. If it was intentional, though, I would appreciate knowing which patch or patches resolved the problem. Thanks in advance, Maru ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] About multihost patch review
On Aug 26, 2013, at 4:06 PM, Edgar Magana emag...@plumgrid.com wrote: Hi Developers, Let me explain my point of view on this topic and please share your thoughts in order to merge this new feature ASAP. My understanding is that multi-host is nova-network HA and we are implementing this bp https://blueprints.launchpad.net/neutron/+spec/quantum-multihost for the same reason. So, If in neutron configuration admin enables multi-host: etc/dhcp_agent.ini # Support multi host networks # enable_multihost = False Why do tenants needs to be aware of this? They should just create networks in the way they normally do and not by adding the multihost extension. I was pretty confused until I looked at the nova-network HA doc [1]. The proposed design would seem to emulate nova-network's multi-host HA option, where it was necessary to both run nova-network on every compute node and create a network explicitly as multi-host. I'm not sure why nova-network was implemented in this way, since it would appear that multi-host is basically all-or-nothing. Once nova-network services are running on every compute node, what does it mean to create a network that is not multi-host? So, to Edgar's question - is there a reason other than 'be like nova-network' for requiring neutron multi-host to be configured per-network? m. 1: http://docs.openstack.org/trunk/openstack-compute/admin/content/existing-ha-networking-options.html I could be totally wrong and crazy, so please provide some feedback. Thanks, Edgar From: Yongsheng Gong gong...@unitedstack.com Date: Monday, August 26, 2013 2:58 PM To: Kyle Mestery (kmestery) kmest...@cisco.com, Aaron Rosen aro...@nicira.com, Armando Migliaccio amigliac...@vmware.com, Akihiro MOTOKI amot...@gmail.com, Edgar Magana emag...@plumgrid.com, Maru Newby ma...@redhat.com, Nachi Ueno na...@nttmcl.com, Salvatore Orlando sorla...@nicira.com, Sumit Naiksatam sumit.naiksa...@bigswitch.com, Mark McClain mark.mccl...@dreamhost.com, Gary Kotton gkot...@vmware.com, Robert Kukura rkuk...@redhat.com Cc: OpenStack List openstack-dev@lists.openstack.org Subject: Re: About multihost patch review Hi, Edgar Magana has commented to say: 'This is the part that for me is confusing and I will need some clarification from the community. Do we expect to have the multi-host feature as an extension or something that will natural work as long as the deployment include more than one Network Node. In my opinion, Neutron deployments with more than one Network Node by default should call DHCP agents in all those nodes without the need to use an extension. If the community has decided to do this by extensions, then I am fine' at https://review.openstack.org/#/c/37919/11/neutron/extensions/multihostnetwork.py I have commented back, what is your opinion about it? Regards, Yong Sheng Gong On Fri, Aug 16, 2013 at 9:28 PM, Kyle Mestery (kmestery) kmest...@cisco.com wrote: Hi Yong: I'll review this and try it out today. Thanks, Kyle On Aug 15, 2013, at 10:01 PM, Yongsheng Gong gong...@unitedstack.com wrote: The multihost patch is there for a long long time, can someone help to review? https://review.openstack.org/#/c/37919/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] About multihost patch review
On Aug 26, 2013, at 9:39 PM, Yongsheng Gong gong...@unitedstack.com wrote: First 'be like nova-network' is a merit for some deployments. I'm afraid 'merit' is a bit vague for me. Would you please elaborate? second, To allow admin to decide which network will be multihosted at runtime will enable the neutron to continue using the current network node (dhcp agent) mode at the same time. If multi-host and non- multi-host networks are permitted to co-exist (because configuration is per-network), won't compute nodes have to be allowed to be heterogenous (some multi-host capable, some not)? And won't Nova then need to schedule VMs configured with multi-host networks on compatible nodes? I don't recall mention of this issue in the blueprint or design doc, and would appreciate pointers to where this decision was documented. If we force the network multihosted when the configuration enable_multihost is true, and then administrator wants to transfer to normal neutron way, he/she must modify the configuration item and then restart. I'm afraid I don't follow - are you suggesting that configuring multi-host globally will be harder on admins than the change under review? Switching to non multi-host under the current proposal involves reconfiguring and restarting of an awful lot of agents, to say nothing of the db changes. m. On Tue, Aug 27, 2013 at 9:14 AM, Maru Newby ma...@redhat.com wrote: On Aug 26, 2013, at 4:06 PM, Edgar Magana emag...@plumgrid.com wrote: Hi Developers, Let me explain my point of view on this topic and please share your thoughts in order to merge this new feature ASAP. My understanding is that multi-host is nova-network HA and we are implementing this bp https://blueprints.launchpad.net/neutron/+spec/quantum-multihost for the same reason. So, If in neutron configuration admin enables multi-host: etc/dhcp_agent.ini # Support multi host networks # enable_multihost = False Why do tenants needs to be aware of this? They should just create networks in the way they normally do and not by adding the multihost extension. I was pretty confused until I looked at the nova-network HA doc [1]. The proposed design would seem to emulate nova-network's multi-host HA option, where it was necessary to both run nova-network on every compute node and create a network explicitly as multi-host. I'm not sure why nova-network was implemented in this way, since it would appear that multi-host is basically all-or-nothing. Once nova-network services are running on every compute node, what does it mean to create a network that is not multi-host? So, to Edgar's question - is there a reason other than 'be like nova-network' for requiring neutron multi-host to be configured per-network? m. 1: http://docs.openstack.org/trunk/openstack-compute/admin/content/existing-ha-networking-options.html I could be totally wrong and crazy, so please provide some feedback. Thanks, Edgar From: Yongsheng Gong gong...@unitedstack.com Date: Monday, August 26, 2013 2:58 PM To: Kyle Mestery (kmestery) kmest...@cisco.com, Aaron Rosen aro...@nicira.com, Armando Migliaccio amigliac...@vmware.com, Akihiro MOTOKI amot...@gmail.com, Edgar Magana emag...@plumgrid.com, Maru Newby ma...@redhat.com, Nachi Ueno na...@nttmcl.com, Salvatore Orlando sorla...@nicira.com, Sumit Naiksatam sumit.naiksa...@bigswitch.com, Mark McClain mark.mccl...@dreamhost.com, Gary Kotton gkot...@vmware.com, Robert Kukura rkuk...@redhat.com Cc: OpenStack List openstack-dev@lists.openstack.org Subject: Re: About multihost patch review Hi, Edgar Magana has commented to say: 'This is the part that for me is confusing and I will need some clarification from the community. Do we expect to have the multi-host feature as an extension or something that will natural work as long as the deployment include more than one Network Node. In my opinion, Neutron deployments with more than one Network Node by default should call DHCP agents in all those nodes without the need to use an extension. If the community has decided to do this by extensions, then I am fine' at https://review.openstack.org/#/c/37919/11/neutron/extensions/multihostnetwork.py I have commented back, what is your opinion about it? Regards, Yong Sheng Gong On Fri, Aug 16, 2013 at 9:28 PM, Kyle Mestery (kmestery) kmest...@cisco.com wrote: Hi Yong: I'll review this and try it out today. Thanks, Kyle On Aug 15, 2013, at 10:01 PM, Yongsheng Gong gong...@unitedstack.com wrote: The multihost patch is there for a long long time, can someone help to review? https://review.openstack.org/#/c/37919/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Neutron + Grenade (or other upgrade testing)
On Aug 26, 2013, at 10:23 AM, Dean Troyer dtro...@gmail.com wrote: On Mon, Aug 26, 2013 at 10:50 AM, Maru Newby ma...@redhat.com wrote: Is anyone working on/planning on adding support for neutron to grenade? Or is there any other automated upgrade testing going on for neutron? We deliberately avoided migrations in Grenade (like Nova Volume - Cinder) as we wanted to focus on upgrades within projects. Migrations will necessarily be much more complicated, especially Nova Network - Neutron. At some point Neutron should be added to Grenade, but only as a release upgrade step for some basic configuration. That said, I'm sure there would be great appreciation for a recipe to duplicate an existing Nova Network config in Neutron. We can debate if that belongs in Grenade should it ever exist… I was referring to upgrades within projects - in this case Quantum to Neutron. I'm assuming that belongs in grenade? m. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] About multihost patch review
On Aug 27, 2013, at 3:27 PM, Tom Fifield t...@openstack.org wrote: On 27/08/13 15:23, Maru Newby wrote: On Aug 26, 2013, at 9:39 PM, Yongsheng Gong gong...@unitedstack.com wrote: First 'be like nova-network' is a merit for some deployments. I'm afraid 'merit' is a bit vague for me. Would you please elaborate? One area of 'merit' in this area is for migration from nova-network to neutron. If there's something exactly analogous to something that already exists, its easier to move across. I apologize for being unclear, but I don't think there is any question that neutron needs a multi-host HA capability. The question is not one of function, but of implementation. I don't believe that the design of a feature being proposed for neutron should be acceptable simply because it reuses an implementation strategy used by nova-network. Neutron's architecture may allow different decisions to be made, and we may have learned from nova-network's example. In any case, reviewers need to understand the 'why' behind design decisions, and it doesn't appear to me that there is sufficient documentation justifying the current proposal's approach. Only once we have more information will we be able to make an educated decision as to the quality of the proposal. m. second, To allow admin to decide which network will be multihosted at runtime will enable the neutron to continue using the current network node (dhcp agent) mode at the same time. If multi-host and non- multi-host networks are permitted to co-exist (because configuration is per-network), won't compute nodes have to be allowed to be heterogenous (some multi-host capable, some not)? And won't Nova then need to schedule VMs configured with multi-host networks on compatible nodes? I don't recall mention of this issue in the blueprint or design doc, and would appreciate pointers to where this decision was documented. If we force the network multihosted when the configuration enable_multihost is true, and then administrator wants to transfer to normal neutron way, he/she must modify the configuration item and then restart. I'm afraid I don't follow - are you suggesting that configuring multi-host globally will be harder on admins than the change under review? Switching to non multi-host under the current proposal involves reconfiguring and restarting of an awful lot of agents, to say nothing of the db changes. m. On Tue, Aug 27, 2013 at 9:14 AM, Maru Newby ma...@redhat.com wrote: On Aug 26, 2013, at 4:06 PM, Edgar Magana emag...@plumgrid.com wrote: Hi Developers, Let me explain my point of view on this topic and please share your thoughts in order to merge this new feature ASAP. My understanding is that multi-host is nova-network HA and we are implementing this bp https://blueprints.launchpad.net/neutron/+spec/quantum-multihost for the same reason. So, If in neutron configuration admin enables multi-host: etc/dhcp_agent.ini # Support multi host networks # enable_multihost = False Why do tenants needs to be aware of this? They should just create networks in the way they normally do and not by adding the multihost extension. I was pretty confused until I looked at the nova-network HA doc [1]. The proposed design would seem to emulate nova-network's multi-host HA option, where it was necessary to both run nova-network on every compute node and create a network explicitly as multi-host. I'm not sure why nova-network was implemented in this way, since it would appear that multi-host is basically all-or-nothing. Once nova-network services are running on every compute node, what does it mean to create a network that is not multi-host? So, to Edgar's question - is there a reason other than 'be like nova-network' for requiring neutron multi-host to be configured per-network? m. 1: http://docs.openstack.org/trunk/openstack-compute/admin/content/existing-ha-networking-options.html I could be totally wrong and crazy, so please provide some feedback. Thanks, Edgar From: Yongsheng Gong gong...@unitedstack.com Date: Monday, August 26, 2013 2:58 PM To: Kyle Mestery (kmestery) kmest...@cisco.com, Aaron Rosen aro...@nicira.com, Armando Migliaccio amigliac...@vmware.com, Akihiro MOTOKI amot...@gmail.com, Edgar Magana emag...@plumgrid.com, Maru Newby ma...@redhat.com, Nachi Ueno na...@nttmcl.com, Salvatore Orlando sorla...@nicira.com, Sumit Naiksatam sumit.naiksa...@bigswitch.com, Mark McClain mark.mccl...@dreamhost.com, Gary Kotton gkot...@vmware.com, Robert Kukura rkuk...@redhat.com Cc: OpenStack List openstack-dev@lists.openstack.org Subject: Re: About multihost patch review Hi, Edgar Magana has commented to say: 'This is the part that for me is confusing and I will need some clarification from the community. Do we expect to have the multi-host feature as an extension or something
[openstack-dev] Neutron + Grenade (or other upgrade testing)
Is anyone working on/planning on adding support for neutron to grenade? Or is there any other automated upgrade testing going on for neutron? Thanks, m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Code review study
On Aug 15, 2013, at 12:50 PM, Joe Gordon joe.gord...@gmail.com wrote: • On Thu, Aug 15, 2013 at 12:22 PM, Sam Harwell sam.harw...@rackspace.com wrote: I like to take a different approach. If my commit message is going to take more than a couple lines for people to understand the decisions I made, I go and make an issue in the issue tracker before committing locally and then reference that issue in the commit message. This helps in a few ways: 1. If I find a technical or grammatical error in the commit message, it can be corrected. 2. Developers can provide feedback on the subject matter independently of the implementation, as well as feedback on the implementation itself. 3. I like the ability to include formatting and hyperlinks in my documentation of the commit. This pattern has one slight issue, which is: • Do not assume the reviewer has access to external web services/site. In 6 months time when someone is on a train/plane/coach/beach/pub troubleshooting a problem browsing GIT history, there is no guarantee they will have access to the online bug tracker, or online blueprint documents. The great step forward with distributed SCM is that you no longer need to be online to have access to all information about the code repository. The commit message should be totally self-contained, to maintain that benefit. I'm not sure I agree with this. It can't be true in all cases, so it can hardly be considered a rule. A guideline, maybe - something to strive for. But not all artifacts of the development process are amenable to being stuffed into code or the commits associated with them. A dvcs is great and all, but unless one is working in a silo, online resources are all but mandatory. m. https://wiki.openstack.org/wiki/GitCommitMessages#Information_in_commit_messages Sam From: Christopher Yeoh [mailto:cbky...@gmail.com] Sent: Thursday, August 15, 2013 7:12 AM To: OpenStack Development Mailing List Subject: Re: [openstack-dev] Code review study On Thu, Aug 15, 2013 at 11:42 AM, Robert Collins robe...@robertcollins.net wrote: This may interest data-driven types here. https://www.ibm.com/developerworks/rational/library/11-proven-practices-for-peer-review/ Note specifically the citation of 200-400 lines as the knee of the review effectiveness curve: that's lower than I thought - I thought 200 was clearly fine - but no. Very interesting article. One other point which I think is pretty relevant is point 4 about getting authors to annotate the code better (and for those who haven't read it, they don't mean comments in the code but separately) because it results in the authors picking up more bugs before they even submit the code. So I wonder if its worth asking people to write more detailed commit logs which include some reasoning about why some of the more complex changes were done in a certain way and not just what is implemented or fixed. As it is many of the commit messages are often very succinct so I think it would help on the review efficiency side too. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Code review study
On Aug 16, 2013, at 2:12 AM, Robert Collins robe...@robertcollins.net wrote: On 16 August 2013 20:15, Maru Newby ma...@redhat.com wrote: This pattern has one slight issue, which is: • Do not assume the reviewer has access to external web services/site. In 6 months time when someone is on a train/plane/coach/beach/pub troubleshooting a problem browsing GIT history, there is no guarantee they will have access to the online bug tracker, or online blueprint documents. The great step forward with distributed SCM is that you no longer need to be online to have access to all information about the code repository. The commit message should be totally self-contained, to maintain that benefit. I'm not sure I agree with this. It can't be true in all cases, so it can hardly be considered a rule. A guideline, maybe - something to strive for. But not all artifacts of the development process are amenable to being stuffed into code or the commits associated with them. A dvcs is great and all, but unless one is working in a silo, online resources are all but mandatory. In a very strict sense you're right, but consider that for anyone doing fast iterative development the need to go hit a website is a huge slowdown : at least in most of the world :). You're suggesting that it's possible to do _fast_ iterative development on a distributed system of immense and largely undocumented complexity (like openstack)? I'd like to be working on the code you're working on! ;) m. So - while I agree that it's something to strive for, I think we should invert it and say 'not having everything in the repo is something we should permit occasional exceptions to'. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Gate breakage process - Let's fix! (related but not specific to neutron)
Neutron has been in and out of the gate for the better part of the past month, and it didn't slow the pace of development one bit. Most Neutron developers kept on working as if nothing was wrong, blithely merging changes with no guarantees that they weren't introducing new breakage. New bugs were indeed merged, greatly increasing the time and effort required to get Neutron back in the gate. I don't think this is sustainable, and I'd like to make a suggestion for how to minimize the impact of gate breakage. For the record, I don't think consistent gate breakage in one project should be allowed to hold up the development of other projects. The current approach of skipping tests or otherwise making a given job non-voting for innocent projects should continue. It is arguably worth taking the risk of relaxing gating for those innocent projects rather than halting development unnecessarily. However, I don't think it is a good idea to relax a broken gate for the offending project. So if a broken job/test is clearly Neutron related, it should continue to gate Neutron, effectively preventing merges until the problem is fixed. This would both raise the visibility of breakage beyond the person responsible for fixing it, and prevent additional breakage from slipping past were the gating to be relaxed. Thoughts? m. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
