Re: [openstack-dev] [Neutron] [Nova] [Cinder] [tc] Should Openstack project maintained by core team keep only API/DB in the future?
On 28 April 2015 at 10:14, Duncan Thomas duncan.tho...@gmail.com wrote: If we allow third party CI to fail and wait for vendors to fix their stuff, experience has shown that they won't, and there'll be broken or barely functional drivers out there, and no easy way for the community to exert pressure to fix them up. Can't the user community exert pressure on the driver developers directly by talking to them, or indirectly by not using their drivers? How come OpenStack upstream wants to tell the developers what is needed before the users get a chance to take a look? I would love to see OpenStack upstream acting more like a resource to support users and developers (e.g. providing 3rd party CI hooks upon requst) and less like gatekeepers with big sticks to wave at people who don't drop their own priorities and Follow The Process. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] TC Candidacy
On 23 April 2015 at 01:17, Maru Newby ma...@redhat.com wrote: My name is Maru Newby, and I am announcing my candidacy for the Technical Committee (TC) election. Cool! ** Growing our contributors Question regarding your candidacy: If I recall correctly you have spoken in favor of face to face discussions at mid-cycle meetings as a practical way to set priorities and move things forward. What are your current thoughts on the costs and benefits of this? Cheers, -Luke __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][NFV] VIF_VHOSTUSER
On 1 September 2014 09:10, loy wolfe loywo...@gmail.com wrote: If the neutron side MD is just for snabbswitch, then I thinks there is no change to be merged into the tree. Maybe we can learn from sriov nic, although backend is vendor specific, but the MD is generic, can support snabb, dpdkovs, ans other userspace vswitch, etc. That is an interesting idea. The Snabb mech driver simply asks Neutron/Nova to bind the port with VIF_VHOSTUSER. If this is the requirement for other drivers too then it would seem that we have good potential for sharing code. Perhaps we could rename mech_snabb to mech_vhostuser, like we have already renamed the VIF code. Going forward we would like the Snabb driver to become more mainstream in the way it manages its agent on the compute host. Currently we use a simple brute force approach [1] that is intended to protect us from synchronisation bugs (race conditions) and be compatible with multiple versions of OpenStack (e.g. the one we deploy with + the one we upstream towards). If we did not have to support a production version based on a stable OpenStack release then we might have been less conservative here. Cheers, -Luke [1] Snabb NFV architecture: https://github.com/SnabbCo/snabbswitch/wiki/Snabb-NFV-Architecture ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova][NFV] VIF_VHOSTUSER
Howdy! I am writing to ask whether it will be possible to merge VIF_VHOSTUSER [1] in Juno? VIF_VHOSTUSER adds support for a QEMU 2.1 has a feature called vhost-user [2] that allows a guest to do Virtio-net I/O via a userspace vswitch. This makes it convenient to deploy new vswitches that are optimized for NFV workloads, of which there are now several both open source and proprietary. The complication is that we have no CI coverage for this feature in Juno. Originally we had anticipated merging a Neutron driver that would exercise vhost-user but the Neutron core team requested that we develop that outside of the Neutron tree for the time being instead [3]. We are hoping that the Nova team will be willing to merge the feature even so. Within the NFV subgroup it would help us to share more code with each other and also be good for our morale :) particularly as the QEMU work was done especially for use with OpenStack. Cheers, -Luke [1] https://review.openstack.org/#/c/96140/ [2] http://www.virtualopensystems.com/en/solutions/guides/snabbswitch-qemu/ [3] https://review.openstack.org/#/c/116476/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Incubator concerns from packaging perspective
On 21 August 2014 12:12, Ihar Hrachyshka ihrac...@redhat.com wrote: Let the ones that are primarily interested in good quality of that code (vendors) to drive development. And if some plugins become garbage, it's bad news for specific vendors; if neutron screws because of lack of concentration on core features and open source plugins, everyone is doomed. Completely agree with this sentiment. Is there a crisp distinction between a vendor plugin and an open source plugin though? The Snabb NFV (http://snabb.co/nfv.html) driver superficially looks like a vendor plugin but is actually completely open source. The development is driven by end-user organisations who want to make the standard upstream Neutron support their NFV use cases. We are looking for a good way to engage with the upstream community. In this cycle we have found kindred spirits in the NFV subteam., but we did not find a good way to engage with Neutron upstream (see https://review.openstack.org/#/c/116476/). It would be wonderful if there is a suitable process available for us to use in Kilo e.g. incubation. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][QoS] Request to be considered for neutron-incubator
On 19 August 2014 23:15, Alan Kavanagh alan.kavan...@ericsson.com wrote: +1, I am hoping this is just a short term holding point and this will eventually be merged into main branch as this is a feature a lot of companies, us included would definitely benefit from having supported and many thanks to Sean for sticking with this and continue to push this. Agreed. Thank you Sean for the great work on QoS. We are looking forward to seeing this merged to master one day and meanwhile maintaining it on our NFV branch for our own use. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] fair standards for all hypervisor drivers
On 8 August 2014 15:27, Russell Bryant rbry...@redhat.com wrote: It sounds like what you're working on is a separate thing. Roger. Just wanted to check if our work could have some broader utility, but as you say we do have a specific use case in mind. Cheers! -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] fair standards for all hypervisor drivers
On 8 August 2014 02:06, Michael Still mi...@stillhq.com wrote: 1: I think that ultimately should live in infra as part of check, but I'd be ok with it starting as a third party if that delivers us something faster. I'd be happy enough to donate resources to get that going if we decide to go with this plan. Can we cooperate somehow? We are already working on bringing up a third party CI covering QEMU 2.1 and Libvirt 1.2.7. The intention of this CI is to test the software configuration that we are recommending for NFV deployments (including vhost-user feature which appeared in those releases), and to provide CI cover for the code we are offering for Neutron. Michele Paolino is working on this and the relevant nova/devstack changes. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron][ml2] Mech driver as out-of-tree add-on
Howdy! Rumor has it that it's easy to distribute ML2 mech drivers as out-of-tree add-on modules. Is this true? Has it been done? Where would one find an example? Cheers! -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron][third-party] Protocol for bringing up CI for a new driver?
Howdy, Could somebody please clarify the protocol for bringing up a CI for a new Neutron driver? Specifically, how does one resolve the chicken-and-egg situation of: 1. CI should be enabled before the driver is merged. 2. CI should test the refspecs given by Gerrit, which will not include the code for the new driver itself until after it has been merged. So what is the common practice for using the new driver in tests during the time window prior to its merge? Cheers! -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Protocol for bringing up CI for a new driver?
Hi Salvatore, On 5 August 2014 10:34, Salvatore Orlando sorla...@nicira.com wrote: Once in place, the CI system should be able to pick up the patches from the new plugin or driver on gerrit. In my opinion, successful CI runs against those patches should constitute a sufficient proof of the validity of the CI system. That sounds good to me. Is there already an easy way to pick up the patches with devstack? (would one create a local repo, do some git-fu to merge the driver, then point devstack's NEUTRON_REPO there?) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Fwd: [NFV][CI][third-party] The plan to bring up Snabb NFV CI for Juno-3
Hi Steve, Thanks for the continuing help! On 29 July 2014 20:25, Steve Gordon sgor...@redhat.com wrote: It appears to me the expectation/desire from Mark and Maru here is to see a lot more justification of the use cases for this driver and the direction of the current implementation I am attempting to satisfy this. I've posted more information on the code review and also on the ml: http://lists.openstack.org/pipermail/openstack-dev/2014-July/041662.html I hope this moves us a step towards satisfying the objections. I am optimistic because the use case is an important one and to me it seems like the design is in line with existing practice in Neutron. Typically third party CI is only provided/required for Nova when adding/maintaining a new hypervisor driver - at least that seems to be the case so far. We are not adding a new hypervisor driver. but the VIF_VHOSTUSER feature does depend on a very recent QEMU (= 2.1) and Libvirt (= 1.2.7). I would like to understand what (if any) CI implications this version requirement has on the Nova side. I know in your earlier email you mentioned also wanting to use this third party CI to also test a number of other scenarios, particularly: * Test with NFV-oriented features that are upstream in OpenStack. * Test with NFV-oriented changes that are not yet upstream e.g. Neutron QoS API. I am not sure how this would work - perhaps I misunderstand what you are proposing? As it stands the third-party CI jobs ideally run on each change *submitted* to gerrit so features that are not yet *merged* still receive this CI testing today both from the CI managed by infra and the existing third-party CI jobs? Or are you simply highlighting that you wish to test same with snabbswitch? Just not quite understanding why these were called out as separate cases. Sorry, I didn't explain myself very clearly. On reflection, the question is quite generic, and I will reraise it under the [3rd-party][neutron] label. There seems to be a chicken-and-egg situation where the CI is supposed to run tests with a new Neutron driver enabled, but that new Neutron itself driver is not merged and so it will not available with the Git refspec that Gerrit supplies to the CI. Likely there is already a standard practice for this that we can find out about from the 3rd party gurus e.g. cherry-pick the driver into all tests. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Neutron][ml2] Snabb NFV mech driver - background of the specification
Howdy! Our feature Snabb NFV mech driver has become controversial on Gerrit. Mark and Maru suspect that it is fundamentally ill-conceived. This mail is to explain the background and advertise that we are available for discussion. (No pressure, I just want to volunteer relevant information.) Blueprint: https://blueprints.launchpad.net/neutron/+spec/snabb-nfv-mech-driver This feature represents the work that myself and other independent open source developers are doing to support Deutsche Telekom's TeraStream NFV project. TeraStream is a new design for national ISPs and they have revisited every layer of the network all the way down to the optics. TeraStream hosts all network services inside an OpenStack cluster. Our job is to make Neutron support TeraStream's next round of requirements. We have been talking about this in great detail with everybody we can find at the summits in Hong Kong and in Atlanta, and we will be in Paris too. Still, the Neutron community is big, so probably most people have no idea what we are trying to accomplish. The main requirement is performance. On each compute node we want to do around 25 million packets per second of processing, and we want to do that within virtual machines using Virtio-net and with OpenStack's L2 networking features available. This will allow us to host applications like Address Family Translation (~NAT) inside VMs even when the need to process all the traffic for a national ISP. We have solved the performance problem by developing the vhost-user feature in QEMU 2.1 and exploiting it in Snabb Switch. That work is upstream now and we are eager to add support in OpenStack too. We think this is important work and we want to share it with the community. Our other requirement is robustness. We need a really simple control layer that has predictable failure modes, no performance surprises, and is easy to troubleshoot. People have advised us against using the LinuxBridge/OVSPlugin management layer at this time. So we have created an expedient short-term solution, which we don't propose to bring in tree and I needn't bore you with the details of, to use until we find a good in-tree solution. Our goal in submitting this code to Juno is to make it easily available to other NFV enthusiasts who may want to adopt it too. The reason we didn't promote this work early in the cycle is that we have only recently had the breakthrough of getting upstream into QEMU + Libvirt and that had been a gating requirement for the Nova (VIF_VHOSTUSER) to be taken seriously. Thanks for reading such a long message! More on TeraStream: Short: http://blog.ipspace.net/2013/11/deutsche-telekom-terastream-designed.html Long: https://ripe67.ripe.net/archives/video/3/ More on the relationship to NFV use cases in the NFV subgroup wiki page: https://wiki.openstack.org/wiki/Teams/NFV Cheers, -Luke (lukego) P.S. Happy for a video chat on Skype or Hangout to explain more if that helps ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [NFV][CI] The plan to bring up Snabb NFV CI for Juno-3
Greetings fellow NFV'stas! I would like to explain and solicit feedback on our plan to support a new open source NFV system in Juno. This work is approved as low-priority/best-effort for Juno-3. (Yes, we do understand that we are fighting the odds in terms of the Juno schedule.) We are developing a practical open source NFV implementation for OpenStack. This is for people who want to run tens of millions of packets per second through Virtio-net on each compute node. The work involves contributing code upstream to a dependent chain of projects: snabbswitch - QEMU - Libvirt - Nova - Neutron Recently we had a breakthrough: QEMU upstream merged the vhost-user feature that we developed and this convinced the kind maintainers of Libvirt, Nova, and Neutron to let us target code to them in parallel. Now Libvirt has accepted our code upstream too and the last pieces are Nova and Neutron. (Then we can start work on Version 2.) Previously our upstreaming effort has been obstructed: people understandably wanted to see our QEMU code accepted before they would take us seriously. So it is an exciting time for us and our upstreaming work. Just now we have ramped up our OpenStack development effort in response to getting approved for Juno-3. Michele Paolino has joined in: he is experienced with Libvirt and is the one who upstreamed our code there. Nikolay Nikolaev is joining in too: he did the bulk of the development on vhost-user and the upstreaming of it into QEMU. Here is what the three of us are working on for Juno-3: * VIF_VHOSTUSER support in Nova. https://blueprints.launchpad.net/nova/+spec/vif-vhostuser * Snabb NFV mech driver for Neutron. https://blueprints.launchpad.net/neutron/+spec/snabb-nfv-mech-driver * NFV CI: OpenStack 3rd party CI that covers our entire software ecosystem (snabbswitch + QEMU + Libvirt + Nova + Neutron). We are already getting great support from the community. Thank you everybody for that, and meta-thankyou to the people who setup the NFV subgroup which has been a fantastic enabler. For the code changes, the ball is in our court now to get them into shape in time. For the CI, I think it's worth having a discussion to make sure we are on the same page and have the same expectations. Here is how I visualize our ideal NFV CI for Juno: * Run Tempest tests for Nova and Neutron. * Test with the relevant versions of Libvirt, QEMU, and snabbswitch. * Test with NFV-oriented features that are upstream in OpenStack. * Test with NFV-oriented changes that are not yet upstream e.g. Neutron QoS API. * Operate reliably with a strong track record. * Be easy for other people to replicate if they want to run their own NFV CI. This CI should then provide assurance for us that our whole ecosystem is running compatibly, for OpenStack that the code going upstream is continuously tested, and for end users that the software they plan to deploy works (either based on our tests, if they are deploying the same software that we use, or based on their own tests if they want to operate a customised CI). How does this CI idea sound to the community and to others who are interested in related NFV-oriented features? That was quite a brain-dump... we have been working on this for quite some time but mostly on the parts outside of the OpenStack tree until now. For more information about our open source NFV project you can read the humble home page: http://snabb.co/nfv.html and if you want to talk nuts and bolts you can find us on Github: https://github.com/SnabbCo/snabbswitch and Google Groups: https://groups.google.com/forum/#!forum/snabb-devel We are independent open source developers and we are working to support Deutsche Telekom's TeraStream NFV project. Cheers! -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [not-only-neutron] How to Contribute upstream in OpenStack Neutron
On 28 July 2014 11:37, Salvatore Orlando sorla...@nicira.com wrote: Therefore the likeness of your patch merging depends on the specific nature of the -1 you received. This is really a key point. Here is a pattern that's worth recognising: If your code is in reasonable shape but there is no urgent need to complete the merge then the reviewer might praise with faint damnation. That is, keep giving you -1 reviews for minor reasons until closer to the end of the merge window. If you are an overzealous newbie you might think you need to respond to every such comment with an immediate revision, and that you might then be rewarded with a +1, but then you would just be waving a dead chicken [0] and better advised to slow down a little. (Says me who went through 19 patch sets on his first small contribution :-)). I would hope that new contributors won't feel too much pressure to look busy. This can be a tough call when your job is to take all reasonable steps to have code accepted. It's one thing to be overzealous about answering nitpick reviews but it would be really unfortunate if you felt that you always needed to (extreme example) have an agenda item in all relevant weekly meetings e.g. NFV + Nova + Neutron + ML2 + Third party. In any case the whole process probably goes much more smoothly once you have a chance to attend a Summit and make some friends. That might not be bad as a first step for new contributors if they have that possibility. (But then the Summit is very expensive until after your first commit is merged and you are recognised as a contributor.) [0]: http://zvon.org/comp/r/ref-Jargon_file.html#Terms~wave_a_dead_chicken ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [NFV][CI][third-party] The plan to bring up Snabb NFV CI for Juno-3
Hi Steve, On 29 July 2014 17:21, Steve Gordon sgor...@redhat.com wrote: I've added the [third-party] tag as well to ensure this catches the broadest segment of relevant people. Thanks! are any modifications to upstream Open vSwitch required to support Snabb? Good question. No, this uses a separate vswitch called Snabb Switch. Snabb Switch is a small user-space program that you assign some network interfaces to. It runs independent of any other networking you are doing on other ports (OVS, DPDK-OVS, SR-IOV, etc). Have you already attempted to solicit some core reviewers in Nova and Neutron How does one normally do that? We are getting help but I am not exactly sure how people have found us beyond chat in #openstack-nfv :-). Two Neutron core reviewers are making the requirements there very clear to us, both on the code and the CI. One Nova core reviewer is helping us too. I would like to better understand CI requirements on the Nova side (e.g. does the Neutron tempest testing regime provide adequate coverage for Nova or do we need to do more?). This is our first contribution to Nova so there is a risk that we overlook something important. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [NFV][CI] The plan to bring up Snabb NFV CI for Juno-3
On 29 July 2014 10:48, Luke Gorrie l...@snabb.co wrote: We are developing a practical open source NFV implementation for OpenStack. This is for people who want to run tens of millions of packets per second through Virtio-net on each compute node. Incidentally, we do currently achieve ~ line rate with our target workload of 6x10G with 256-byte packets and all traffic being looped through VMs over Virtio-net. Here is a benchmark output from our testbed right now: On :07:00.0 got 4.462 On :07:00.1 got 4.462 On :24:00.0 got 4.454 On :24:00.1 got 4.452 On :27:00.0 got 4.455 On :27:00.1 got 4.455 Rate(Mpps): 26.74 That is with each packet received off the wire by Snabb Switch, looped through a QEMU guest (running Ubuntu w/ DPDK) over vhost-user, then transmitted by Snabb Switch back onto the wire. That is one packet received and transmitted on each port every 225 nanoseconds. Surprisingly, the whole traffic plane is written in Lua and is only a small amount of code. We are really proud of the work we are doing and hope it will become a part of the open source networking landscape for many years to come. People who like this sort of thing are advised to get in touch with us and join in the fun :). Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [not-only-neutron] How to Contribute upstream in OpenStack Neutron
On 25 July 2014 20:05, Stefano Maffulli stef...@openstack.org wrote: Indeed, communication is key. I'm not sure how you envision to implement this though. We do send a message to first time contributors[1] to explain them how the review process works and give them very basic suggestions on how to react to comments (including what to do if things seem stuck). The main issue here though is that few people read emails, it's a basic fact of life. That welcome message does seem to do a really good job of setting expectations. Can you explain more what you have in mind? Here are some other topics that seem to take some time to develop a mental model of: How quickly and how often should you revise your patchset after a -1? (Is it better to give the community a week or so to collectively comment? Or should you revise ASAP after every negative review?) How do you know if your change is likely to merge? (If you have had 15 rounds of -1 votes and the last milestone deadline is a few days away, should you relax because your code is so thoroughly reviewed or should you despair because it should have been merged by now?) In the final days before a merge deadline, would it be rude to poke the person responsible for merging, or would it be negligent not to? How do you decide which IRC meetings to attend? (For meetings that occur at difficult times outside of working hours in your timezone, when are you expected to attend them? Is it okay to focus on email/informal communication if that suits you better and gets the job done?) If you're new to the project and you don't know anybody, who can you ask about this stuff? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Infra][Neutron] Request voting for Tail-f CI account
Thanks everybody! Onward :-) On 24 July 2014 19:41, Anita Kuno ante...@anteaya.info wrote: On 07/24/2014 01:18 PM, Kyle Mestery wrote: On Thu, Jul 24, 2014 at 12:03 PM, Collins, Sean sean_colli...@cable.comcast.com wrote: On Wed, Jul 23, 2014 at 11:19:13AM EDT, Luke Gorrie wrote: Tail-f NCS: I want to keep this feature well maintained and compliant with all the rules. I am the person who wrote this driver originally, I have been the responsible person for 90% of its lifetime, I am the person who setup the current CI, and I am the one responsible for smooth operation of that CI. I am reviewing its results with my morning coffee and have been doing so for the past 6 weeks. I would like to have it start voting and I believe that it and I are ready for that. I am responsive to email, I am usually on IRC (lukego), and in case of emergency you can SMS/call my mobile on +41 79 244 32 17. So... Let's be friends again? (and do ever cooler stuff in Kilo?) Luke was kind enough to reach out to me, and we had a discussion in order to bury the hatchet. Posting his contact details and being available to discuss things has put my mind at ease, I am ready to move forward. +1 He also reached out to me, so I'm also happy to add this back and move forward with burying the hatchet. I'm all for second chances in general, and Luke's gone out of his way to work with people upstream in a much more efficient and effective manner. Thanks, Kyle Well done, Luke. It takes a lot of work to dig oneself out of a hole and create good relationships where there need to be some. It is a tough job and not everyone chooses to do it. You chose to and you succeeded. I commend your work. I'm glad we have a good resolution in this space. Thanks to all involved for their persistence and hard work. Well done, Anita. -- Sean M. Collins ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [not-only-neutron] How to Contribute upstream in OpenStack Neutron
On 24 July 2014 17:09, Kyle Mestery mest...@mestery.com wrote: I've received a lot of emails lately, mostly private, from people who feel they are being left out of the Neutron process. I'm unsure if other projects have people who feel this way, thus the uniquely worded subject above. I wanted to broadly address these concerns with this email. I have one idea for low-hanging fruit to put new contributors more at ease: to explain a little about both when and why the Merge button is finally pressed on a change. I mean so that new contributors won't have doubts like is it bad that my change isn't merged yet?, am I being too meek?, am I being too pushy?, have I missed a step somewhere?, how often should I skip dinner with my family to attend more/different IRC meetings?, and so on. I have had a good experience with this but that is many thanks to friendly people giving me informal feedback and reassurance outside of the Gerrit workflow and official docs. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Infra][Neutron] Request voting for Tail-f CI account
On 22 July 2014 11:06, Luke Gorrie l...@tail-f.com wrote: End of Part One. Let's skip Part Two. That is just more frustration. Let's talk about Part Three in which we all do awesome CI hacking in Juno together :-). Here is what I want to achieve in Juno: NFV CI: Myself and my colleagues are developing the open source Neutron networking for Deutsche Telekom's TeraStream project (and we want to bring up a CI that tests this configuration. That will exercise new and exciting-for-NFV features of QEMU, Libvirt, Nova, and Neutron. This should serve several purposes: making TeraStream a success story for OpenStack and Neutron, making the whole design easy to replicate for other users (it's already open source), and providing test coverage for more OpenStack features. (Good stuff for everybody, I hope! More info: http://blog.ipspace.net/2013/11/deutsche-telekom-terastream-designed.html) People: I want to onboard great new open source hackers into the OpenStack community and get them contributing to CI. I am right now bringing new people up to speed on OpenStack development and working with them on bringing up our NFV CI this month. shellci: I want to make shellci a practical alternative for CI operators whose style is more screen+bash+awk than jenkins+zuul+nodepool. The development is already done, and it works great in my own tests, so now we plan to battle test it on the NFV CI. (link: https://github.com/SnabbCo/shellci) Tail-f NCS: I want to keep this feature well maintained and compliant with all the rules. I am the person who wrote this driver originally, I have been the responsible person for 90% of its lifetime, I am the person who setup the current CI, and I am the one responsible for smooth operation of that CI. I am reviewing its results with my morning coffee and have been doing so for the past 6 weeks. I would like to have it start voting and I believe that it and I are ready for that. I am responsive to email, I am usually on IRC (lukego), and in case of emergency you can SMS/call my mobile on +41 79 244 32 17. So... Let's be friends again? (and do ever cooler stuff in Kilo?) Cheers! -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Infra][Neutron] Request voting for Tail-f CI account
Hi Sean, On 21 July 2014 22:53, Collins, Sean sean_colli...@cable.comcast.com wrote: The fact that I tried to reach out to the person who was listed as the contact back in November to try and resolve the –1 that this CI system gave, and never received a response until the public mailing list thread about revoking voting rights for Tail-F, makes me believe that the Tail-F CI system is still not ready to have that kind of privilege. Especially if the account was idle from around February, until June – that is a huge gap, if I understand correctly? I understand your frustration. It seems like the experience of bringing up our CI has been miserable for all concerned. I am sad about that. It does not seem that it should have worked out this way, since everybody concerned is a competent person and acting in good faith. I hope we can finally clear this up and then continue with contributing to OpenStack on good terms with everybody. Back in November we were feeling eager to be good citizens and we wanted to be amongst the first to setup a 3rd party CI for Neutron. We were trying to be proactive: our driver was already in Havana and the deadlines for us to setup the CI were far in the future. My colleague Tobbe was also planning to take the lead on development of our OpenStack code from me and we thought the perfect first step would be to setup our CI system, since that would get him familiar with the code and since neither of us had prior experience operating an OpenStack CI. We read through the 3rd Party CI setup instructions and created a CI. Our initial setup ran Jenkins and would use a custom script to create a one-shot VM and inside that it would run the Neutron unit tests together with a patch that made our driver talk to our real external system. This got quite good test coverage because the unit tests really exercise the ML2 interface quite well. (Likely we should have used Tempest instead, as everybody does nowadays include us, but we didn't know that back then.) This seemed to work well and so we let it run. Honestly, we did not really know what would happen with our results after they were posted, and we did not have a definite goal for what service level we should uphold. That was surely naive, but I think understandable. We were relatively new and minor contributors to OpenStack and we were amongst the first wave of Neutron people to setup a CI. We hadn't yet had the opportunity to learn from the mistakes of others or see how reviews are used by the upstream people and systems. We were also perhaps a little too relaxed because our total contribution was around 150 lines of code that only run when explicitly enabled, and we had our own test procedure in place separately from OpenStack CI that we had been using since Havana, so it did not feel like we had much potential to impact other OpenStack users and developers with our code. Anyway. The test runs started to fail unexpectedly, for a boring kind of reason like that OpenStack needed a newer version of a library and our CI script lacked a pip upgrade command that would pick it up, so all tests would fail until manual intervention. So what happens when the CI falls down and needs help to come back up? First of all, it creates a big problem for upstream developers and slows down work on OpenStack (ouch). Second, you poor guys who are having problems try to contact the person responsible, but all you have is one work email address and IRC nick. In that case, you guys did not get a response. I think that was for the very pedestrian reason that my colleague who was responsible was on vacation and didn't appreciate that an operational issue with our CI would create an urgent problem for other people and must be attended to at all times. This must have been bad for you guys since you were stuck waiting on us and couldn't fix the problem on your side. I was also contacted by email, as the previous contact person for that driver, but the message simply asked me to confirm my colleague's email address and did not tell me that there was a problem that we had to resolve. So eventually the problem boiled over and when we started getting publicly flamed on the mailing list then I finally saw that there was an issue and called up my colleague directly who *then* jumped into account to sort it out (logging into gerrit and reversing old negative votes, and so on). So what do we take away from this first experience? To me it just looks like processes to fix: people operating 3rd party CIs need to better understand the required service level, there should be multiple contact points to deal with mundane stuff like vacations and illness, and that people should operate their CI successfully for a while before voting is enabled. It sucks that work was interrupted and people got mad, but at the end of the day this happened with everybody acting in good faith, and it shows us what kind of problems to prevent in the future. This is where it became a bit
Re: [openstack-dev] [Infra][Neutron] Request voting for Tail-f CI account
On 22 July 2014 11:06, Luke Gorrie l...@tail-f.com wrote: This must have been bad for you guys since you were stuck waiting on us and couldn't fix the problem on your side. I was also contacted by email, as the previous contact person for that driver, but the message simply asked me to confirm my colleague's email address and did not tell me that there was a problem that we had to resolve. (I checked and that is not true: actually it did tell me that there was a problem, and I just didn't get that it was urgent. This narrative is a little clouded with emotion at this point I must admit :-)). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Infra][Neutron] Request voting for Tail-f CI account
Howdy! I am writing to request voting rights for the Tail-f CI account. This runs tests for the Tail-f NCS ML2 mechanism driver in Neutron. This account has been non-votingly testing ML2 changes and posting results since June 10th. It has made around 500 test runs in that time. I am monitoring its operation daily. The recent changes that it has posted results for are here: https://review.openstack.org/#/dashboard/9695 The top level logs directory is here: http://openstack-ci.tail-f.com:81/html/ci-logs/ We reviewed its output in the 3rd party meeting last week. Two issues were raised: - Using an IP address instead of a DNS name. That's now corrected. - Running only a small set of Tempest tests. I'm working on expanding that (in a separate staging environment.) The account has a rich history :-). Initially we brought it online back around Nov 2013 early in the Icehouse cycle. That didn't work out so well: we had a bunch of operational issues and as OpenStack newbies we were oblivious to the impact they had on other people's workflows -- we were mortified to learn that we had created a disruption. Since then we have been more conservative which is why the account was mostly idle until June. I reckon we have a pretty good understanding of the expectations on CI operators now and I would like to enable the voting permission. I have recently developed a new CI daemon that I hope to migrate this account over to in the future: https://github.com/SnabbCo/shellci. Cheers, -Luke NB: Last week we had a half-dozen or so errors due to the infamous ansible versioning issue. I didn't retrigger all of those since this was a widespread issue and our CI wasn't voting. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] Update on third party CI in Neutron
On 11 July 2014 17:56, Kyle Mestery mest...@noironetworks.com wrote: 1. Tail-F 1. Inconsistent past runs, need updates on status. I've updated the Etherpad for our Tail-f CI and will be at the meeting. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
On 9 July 2014 20:24, Sullivan, Jon Paul jonpaul.sulli...@hp.com wrote: Incidentally, is there already way to review what votes my CI (or indeed anybody's) is casting via an openstack.org web interface? You can look at the individual account dashboards in Gerrit, like: https://review.openstack.org/#/dashboard/12019 This search seems to omit many entries that I am interested in. For example, the Tail-f CI has tested reviewed hundreds of Neutron changes recently but the search comes up empty. Is this an issue with accounts that don't have the voting bit set? For the use case of bringing new CIs online I would find it very useful to be able to review all the reviews that the CI makes even before it has voting enabled. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
Howdy! I've been operating a shellci for a while now and overall it is very smooth. The main new feature now is to automatically retrigger events that neither definitely succeed (exit status 100) nor definitely fail (exit status 101). In this case the CI will vote 0 with the logs and then automatically schedule a new test, up to a max of 5 tests, until it can conclude +1 or -1. This seems to work well. On my setup I often see tests fail due to network problems (e.g. timeout during a Git checkout) and automatic retries filer away this class of error very neatly, ultimately casting the +1/-1 votes based purely on the relatively reliable contents of tempest.log. (Thanks Jim Gray for pointing out that detectable intermittent bugs are really easy to deal with http://www.hpl.hp.com/techreports/tandem/TR-85.7.pdf). [The network problems themselves are a little suspicious but for now I see them as a blessing that helps me make shellci robust and I don't worry too much about the root cause.] I'm running 5 parallel builds at the moment. I've tried ramping that up to 10 or 20 but then I start to see Vagrant/VirtualBox startup errors. Again, I'm using this as a test case for the automatic retries, and not digging too deep yet. I'd like to be able to track the results via RSS feeds. Can I do this via openstack.org or should I support that directly in shellci? disk space will become an issue soon. Currently I'm running around 500 builds per day and each one generates around 8MB of logs. Compression and redundancy-avoidance may go a long way towards reducing his problem, however it seems a pity to have a scarcity mentality when it comes to logging (I'd prefer to do more rather than less). Any bright ideas for conveniently archiving some terabytes of logs? Overall I'd say that shellci is quite reliable now. The main reason I don't recommend it to others yet is that I'm a bit too busy to provide support with setting it up this week. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
On 10 July 2014 10:06, Luke Gorrie l...@snabb.co wrote: The main new feature now is to automatically retrigger events that neither definitely succeed (exit status 100) nor definitely fail (exit status 101). In this case the CI will vote 0 with the logs and then automatically schedule a new test, up to a max of 5 tests, until it can conclude +1 or -1. ... and while I have your attention, I may as well show you the code for that feature, to help describe the genera style of shellci: https://github.com/SnabbCo/shellci/commit/d9716d83c85e532ad05d04fcd72af3995f66899d ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
Hi again Jon Paul, My mistake! This seems to be exactly what I was looking for, thank you. (I goofed the query which is why I thought it was lacking.) Cheers :) -Luke On 10 July 2014 09:17, Luke Gorrie l...@snabb.co wrote: On 9 July 2014 20:24, Sullivan, Jon Paul jonpaul.sulli...@hp.com wrote: Incidentally, is there already way to review what votes my CI (or indeed anybody's) is casting via an openstack.org web interface? You can look at the individual account dashboards in Gerrit, like: https://review.openstack.org/#/dashboard/12019 This search seems to omit many entries that I am interested in. For example, the Tail-f CI has tested reviewed hundreds of Neutron changes recently but the search comes up empty. Is this an issue with accounts that don't have the voting bit set? For the use case of bringing new CIs online I would find it very useful to be able to review all the reviews that the CI makes even before it has voting enabled. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
On 3 July 2014 19:05, Luke Gorrie l...@snabb.co wrote: Time to make it start running real tempest tests. Howdy! shellci now supports running n parallel build processes and by default runs each test with devstack+tempest in a one-shot Vagrant VM. The README is updated on Github: https://github.com/SnabbCo/shellci I'm running an additional non-voting instance that runs five parallel builds and triggers on all OpenStack projects. For the curious, this instance's logs are at http://horgen.snabb.co/shellci/log/ and the build directories are under http://horgen.snabb.co/shellci/tests/. This week I should discover how much maintenance is needed to keep it humming along and then we'll see if I can recommend it to anybody else or not. (I don't recommend it yet but I did try to make the README detailed enough in case there is anybody who wants to play now.) Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [third party] - minimum response time for 3rd party CI responses
On 3 July 2014 19:02, Jay Pipes jaypi...@gmail.com wrote: devstack-gate works very well for what it is supposed to do: Yeah, I would actually love to use devstack-gate. I tried that first. There are two problems for me as a user: First I didn't manage to get it up and running reliably in a reasonable time frame (one week). In that time I was only starting to develop a mental model of how to troubleshoot problems. Getting support on IRC is awkward and especially so from my timezone (and especially-especially so while being a dad to a newborn baby). Second it exposes me to criticism for being lazy and/or incompetent because people think it's very easy to setup. This easily escalates into threats to delete all of my code from OpenStack for being a bad CI citizen, despite the fact that from my perspective I am starting early and working hard. (Havana was fun, and I'm writing working code, so how do I end up being cast as a bad guy?) Hacking up a custom CI in shell is pure desperation on my part because I need something that I am able to operate and maintain. Let's compare notes over a coffee in Paris :-). Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [third party] - minimum response time for 3rd party CI responses
On 3 July 2014 02:44, Michael Still mi...@stillhq.com wrote: The main purpose is to let change reviewers know that a change might be problematic for a piece of code not well tested by the gate Just a thought: A sampling approach could be a reasonable way to stay responsive under heavy load and still give a strong signal to reviewers about whether a change is likely to be problematic. I mean: Kevin mentions that his CI gets an hours-long queue during peak review season. One way to deal with that could be skipping some events e.g. toss a coin to decide whether to test the next revision of a change that he has already +1'd previously. That would keep responsiveness under control even when throughput is a problem. (A bit like how a router manages a congested input queue or how a sampling profiler keeps overhead low.) Could be worth keeping the rules flexible enough to permit this kind of thing, at least? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
On 1 July 2014 19:12, Luke Gorrie l...@tail-f.com wrote: It does not yet run devstack/tempest and I hope to reuse that part from somebody else's efforts. shellci is happily voting on the sandbox with the Snabb NFV CI account so far: http://egg.snabb.co:81/shellci/shellci.log Time to make it start running real tempest tests. I whipped up a simple Vagrantfile that runs devstack and tempest in a disposable VM. The idea is that out-of-the-box you get a setup that runs tempest and votes on the results. Then you customize local.conf, tempest.conf, and optionally the whole script to do the appropriate testing for your driver. (Or, if you like, skip this part and supply your own testing script to do whatever you like.) Vagrant scripts only in a Gist for now: https://gist.github.com/lukego/bdefc792b8255d141e4c I'll see how the performance looks. Vagrant probably slows down serial performance but should make independent parallel runs easy. I ordered a hetzner.de server with 128GB RAM and if that comes through tomorrow we'll see how that plays out. The plan for parallelism is sharding. Each gerrit-stream event will be hashed into one of N buckets and then you can run N copies of the testing script (on whatever machine(s)) and each copy chooses a different hash bucket to trigger on. Let's see how promising (or not) that looks tomorrow :-). If it works out for me then hopefully somebody else will want to kick the tires next week. (We'll need a separate 100 line budget for the Vagrant/devstack/tempest stuff by the look of it! Lies, damned lies, budgets...) Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova} NFV patches
On 2 July 2014 10:39, Gary Kotton gkot...@vmware.com wrote: There are some patches that are relevant to the NFV support. There are as follows: Additionally, we who are building Deutsche Telekom's open source NFV implementation will be able to make that available to the whole community if the VIF_VHOSTUSER spec is approved for Juno. Then we could help others to deploy this too which would be great. Blueprint: https://blueprints.launchpad.net/nova/+spec/vif-vhostuser Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [third party] - minimum response time for 3rd party CI responses
On 30 June 2014 21:04, Kevin Benton blak...@gmail.com wrote: As a maintainer of a small CI system that tends to get backed up during milestone rush hours, it would be nice if we were allowed up to 12 hours. However, as a developer this seems like too long to have to wait for the results of a patch. Interesting question! Taking one hundred steps back :-) what is the main purpose of the 3rd party reviews, and what are the practical consequences when they are not promptly available? Is the main purpose to allow 3rd parties to automatically object to changes that will cause them problems, and the practical consequence of a slow review being that OpenStack may merge code that will cause a problem for that third party? How do genuine negative reviews by 3rd party CIs play out in practice? (Do the change author and the 3rd party get together offline to work out the problem? Or does the change-author treat Gerrit as an edit-compile-run loop to fix the problem themselves?) I'd love to see links to such reviews, if anybody has some? (I've only seen positive reviews and false-negative reviews from 3rd party CIs so far in my limited experience.) Generally it seems like 12 hours is the blink of an eye in terms of the whole lifecycle of a change, or alternatively an eternity in terms of somebody sitting around waiting to take action on the result. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [third party] - minimum response time for 3rd party CI responses
On 2 July 2014 20:33, Luke Gorrie l...@snabb.co wrote: I'd love to see links to such reviews, if anybody has some? (I've only seen positive reviews and false-negative reviews from 3rd party CIs so far in my limited experience.) I didn't say what I meant: reviews where a 3rd party CI has genuinely objected to a change that was acceptable to all the other CIs i.e. a change that created a problem specifically for that third party. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
Howdy! I wrote a new version of shellci today and have it up and running and voting on the sandbox. It's described on the Github page: https://github.com/SnabbCo/shellci Currently this is simple shell scripts to receive review.openstack.org gerrit events, run tests and determine results, then post reviews. It does not yet run devstack/tempest and I hope to reuse that part from somebody else's efforts. (I know ODL have done reusable work here and I plan to look into that. Anybody else?) Ideas and encouragement welcome as always. Let's find out if this point in the design space is practical or not. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron][third-party] Simple and robust CI script?
Howdy! Paging other 3rd party CI operators... I would like to run a simple and robust 3rd party CI. Simple as in a small number of moving parts, robust as in unlikely to make mistakes due to unexpected problems. I'm imagining: - 100 lines of shell for the implementation. - Minimum of daemons. (Just Jenkins? Ideally not even that...) - Robust detection of operational problems (CI bugs) vs system-under-test problems (changes that should be voted against). - Effective notifications on all errors or negative votes. Does anybody already have an implementation like this that they would like to share? (Is anybody else wanting something in this direction?) I've been iterating on 3rd party CI mechanisms (currently onto my 3rd from-scratch setup) and I have not been completely satisfied with any of them. I would love to hear from someone who has a minimalist implementation that they are happy with :). Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
On 30 June 2014 17:34, Kyle Mestery mest...@noironetworks.com wrote: It would be great to get you to join the 3rd Party meeting [1] in #openstack-meeting at 1800UTC to discuss this. Can you make it today Luke? Yes, I'll be there. Currently I'm looking into the simplest 3rd party CI that could possibly work which would be less sophisticated than devstack-gate but easier for the operator to understand and be responsible for. (Or: CI in 100 lines of shell with no Jenkins or other large pieces of software to install.) I'm on #openstack-neutron if anybody wants to kick around ideas or share scripts that I can borrow ideas from (thanks ODL gang for doing this already). Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
On 30 June 2014 19:37, Asselin, Ramy ramy.asse...@hp.com wrote: Not sure if these are “minimalist” but at least they setup automagically, so you don’t need to do it from scratch: I'm aiming to do exactly the opposite of this i.e. no automagic. My experience has been that the really heavy-duty CI setups are too difficult to understand and troubleshoot for end-users like me. I spent a week working on installing Jay's tools, with very generous help from Jay himself, and for me it was a dead end [*]. I now consider these to be tools for openstack-infra insiders rather than guys like me who are maintaining tiny drivers. I don't know if other people have had a different experience from me. It does seem like 3rd party CI is broadly a lot more problematic than advertised. I have seen only relatively few frank experience reports from people operating CIs and I suspect there is a rich and interesting untold story there :-). (Maybe it would be interesting to do a survey some day.) [*] http://openstack-dev-openstack.git.net/2014-March/030022.html other posts in that thread. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
I have a really early sketch of this project on Github now. shellci - OpenStack 3rd party CI in 100 lines of shell https://github.com/SnabbCo/shellci This is not finished yet but I'll try to use it for the new Neutron mech driver that I want to contribute to Juno. Ideas and encouragement welcome :-). Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] Neutron 3rd Party CI status dashboard
On 30 June 2014 21:08, Anita Kuno ante...@anteaya.info wrote: I am disappointed to realize that Ilya (or stackalytics, I don't know where this is coming from) is unwilling to cease making up definitions of success for third party ci systems to allow the openstack community to arrive at its own definition. There is indeed a risk that the new dashboards won't give a meaningful view of whether a 3rd party CI is voting correctly or not. However, there is an elephant in the room and a much more important problem: To measure how accurately a CI is voting says much more about a driver author's Gerrit-fu ability to operate a CI system than it does about whether the code they have contributed to OpenStack actually works, and the latter is what is actually important. To my mind the whole 3rd party testing discussion should refocus on helping developers maintain good working code and less on waving you will be kicked out of OpenStack if you don't keep your swarm of nebulous daemons running 24/7. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][neutron][NFV] Mid cycle sprints
On 18 June 2014 12:00, Carlos Gonçalves m...@cgoncalves.pt wrote: I’ve added Joao Soares (Portugal Telecom) and myself (Instituto de Telecomunicacoes) to https://wiki.openstack.org/wiki/Sprints/ParisJuno2014 for a Neutron and NFV meetup. Please add yourselves as well so that we can have a better idea of who’s showing interest in participating. Looks like we have the numbers :-) with 5 people the NFV part would be one of the biggest of the sprint. That's good enough for me. I'm making travel arrangements to be in Paris next week and I have updated the Wiki to confirm my interest. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][Swift][third-party] Most Third Party CI's failing
On 17 June 2014 09:55, Luke Gorrie l...@tail-f.com wrote: I have a problem that appeared at the same time and may be related? testr list-tests in the tempest directory is failing with an obscure error message. Seems to be exactly the situation described here: https://bugs.launchpad.net/subunit/+bug/1278539 Any tips? How should I go about getting help with this? Mailing list + IRC is not getting anybody's attention and I want to get my CI back online. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][Swift][third-party] Most Third Party CI's failing
On 18 June 2014 15:48, Salvatore Orlando sorla...@nicira.com wrote: Hi Luke, That kind of message usually shows up in unit tests job when there is some syntax error or circular import. But I think that it's not your case. Usually you see an import error message towards the end of the garbage. If you can point me to a failing log of your CI I can have a look at it and see if I can help you. Thanks, Salvatore! I have a log here: http://88.198.8.227:81/html/ci-logs/problem-1.log and on this machine I can reproduce the problem using the steps in the bug that I referenced above: ci@egg:/tmp$ mkdir bug ci@egg:/tmp$ cd !$ cd bug ci@egg:/tmp/bug$ git clone https://github.com/openstack/tempest.git Cloning into 'tempest'... remote: Reusing existing pack: 35264, done. remote: Counting objects: 229, done. remote: Compressing objects: 100% (221/221), done. remote: Total 35493 (delta 105), reused 25 (delta 8) Receiving objects: 100% (35493/35493), 8.51 MiB | 1.61 MiB/s, done. Resolving deltas: 100% (25835/25835), done. Checking connectivity... done. ci@egg:/tmp/bug$ cd tempest ci@egg:/tmp/bug/tempest$ sudo pip install -r requirements.txt Requirement already satisfied (use --upgrade to upgrade): pbr=0.6,!=0.7,1.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): anyjson=0.3.3 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): httplib2=0.7.5 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): jsonschema=2.0.0,3.0.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): testtools=0.9.34 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 5)) Requirement already satisfied (use --upgrade to upgrade): lxml=2.3 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 6)) Requirement already satisfied (use --upgrade to upgrade): boto=2.12.0,!=2.13.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 7)) Requirement already satisfied (use --upgrade to upgrade): paramiko=1.13.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 8)) Requirement already satisfied (use --upgrade to upgrade): netaddr=0.7.6 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 9)) Requirement already satisfied (use --upgrade to upgrade): python-glanceclient=0.9.0 in /opt/stack/python-glanceclient (from -r requirements.txt (line 10)) Requirement already satisfied (use --upgrade to upgrade): python-keystoneclient=0.8.0 in /opt/stack/python-keystoneclient (from -r requirements.txt (line 11)) Requirement already satisfied (use --upgrade to upgrade): python-novaclient=2.17.0 in /opt/stack/python-novaclient (from -r requirements.txt (line 12)) Requirement already satisfied (use --upgrade to upgrade): python-neutronclient=2.3.4,3 in /opt/stack/python-neutronclient (from -r requirements.txt (line 13)) Requirement already satisfied (use --upgrade to upgrade): python-cinderclient=1.0.6 in /opt/stack/python-cinderclient (from -r requirements.txt (line 14)) Requirement already satisfied (use --upgrade to upgrade): python-heatclient=0.2.9 in /opt/stack/python-heatclient (from -r requirements.txt (line 15)) Requirement already satisfied (use --upgrade to upgrade): python-ironicclient in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 16)) Requirement already satisfied (use --upgrade to upgrade): python-saharaclient=0.6.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 17)) Requirement already satisfied (use --upgrade to upgrade): python-swiftclient=2.0.2 in /opt/stack/python-swiftclient (from -r requirements.txt (line 18)) Requirement already satisfied (use --upgrade to upgrade): testresources=0.2.4 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 19)) Requirement already satisfied (use --upgrade to upgrade): testrepository=0.0.18 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 20)) Requirement already satisfied (use --upgrade to upgrade): oslo.config=1.2.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 21)) Requirement already satisfied (use --upgrade to upgrade): six=1.6.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 22)) Requirement already satisfied (use --upgrade to upgrade): iso8601=0.1.9 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 23)) Requirement already satisfied (use --upgrade to upgrade): fixtures=0.3.14 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 24)) Requirement already satisfied (use --upgrade to upgrade): testscenarios=0.4 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 25))
Re: [openstack-dev] [Neutron][Swift][third-party] Most Third Party CI's failing
On 18 June 2014 18:24, Salvatore Orlando sorla...@nicira.com wrote: it seems something is not quite right with your tempest environment - you have import errors at startup [1] This might be happening because of missing dependencies, or, if you have applied some custom patches to tempest trunk, possibly those are causing some problems. Interesting. I'm using a fresh checkout of tempest from master on github and running pip install -r requirements.txt in tempest/. Any ideas on what I should check next? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][Swift][third-party] Most Third Party CI's failing
On 15 June 2014 02:45, Sukhdev Kapur sukhdevka...@gmail.com wrote: I thought I send out this message in case other CI maintainers are investigating this issue. I have a problem that appeared at the same time and may be related? testr list-tests in the tempest directory is failing with an obscure error message. Seems to be exactly the situation described here: https://bugs.launchpad.net/subunit/+bug/1278539 Any tips? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][neutron][NFV] Mid cycle sprints
On 13 June 2014 11:45, Carlos Gonçalves m...@cgoncalves.pt wrote: Neutron and NFV team members, who’s interested in meeting in Paris, or if not available on the date set by eNovance in other time and place? I'd be very interested in an NFV meet up in Paris in July. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][ml2] Too much shim rest proxy mechanism drivers in ML2
On 10 June 2014 18:18, Irena Berezovsky ire...@mellanox.com wrote: Hi Luke, Very impressive solution! Thanks for the kind words! I do not think there is a problem to keep agent out of the tree in a short term, but would highly recommend to put it upstream in a longer term. You will benefit from quite valuable community review. And most important it will allow to keep your code as much as possible aligned with neutron code base. Once there are some general changes done by other people, your code will be taken into account and won’t be broken accidentally. Yes, I agree. We do want to be upstream going forward. For the Juno cycle our modest goal is that users will be able to deploy our software with an unmodified OpenStack Juno. On the OpenStack side this only requires a simple VIF type (Nova) and ML2 mech driver (Neturon). Behind the scenes we are doing a lot of work together with the upstream QEMU community to make this possible. Hopefully we will be able to expand our upstream OpenStack ambitions in the K-cycle. Looking forward to setting the next goals in Paris :-). Currently our project is relatively new and unknown, but I hope there will be a lot of involvement within the OpenStack community going forward. Our goal is to create a practical NFV solution that is 100% open source and we hope this will be useful to a lot of people. (http://snabb.co/nfv.html) P.S. Thanks for the reminder about modular ML2 agent! Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][ml2] Too much shim rest proxy mechanism drivers in ML2
Hi Irena, Thanks for the very interesting perspective! On 10 June 2014 10:57, Irena Berezovsky ire...@mellanox.com wrote: *[IrenaB] The DB access approach was previously used by OVS and LinuxBridge Agents and at some point (~Grizzly Release) was changed to use RPC communication.* That is very interesting. I've been involved in OpenStack since the Havana cycle and was not familiar with the old design. I'm optimistic about the scalability of our implementation. We have sanity-tested with 300 compute nodes and a 300ms sync interval. I am sure we will find some parts that we need to spend optimization energy on, however. The other scalability aspect we are being careful of is the cost of individual update operations. (In LinuxBridge that would be the iptables, ebtables, etc commands.) In our implementation the compute nodes preprocess the Neutron config into a small config file for the local traffic plane and then load that in one atomic operation (SIGHUP style). Again, I am sure we will find cases that we need to spend optimization effort on, but the design seems scalable to me thanks to the atomicity. For concreteness, here is the agent we are running on the DB node to make the Neutron config available: https://github.com/SnabbCo/snabbswitch/blob/master/src/designs/neutron/neutron-sync-master and here is the agent that pulls it onto the compute node: https://github.com/SnabbCo/snabbswitch/blob/master/src/designs/neutron/neutron-sync-agent TL;DR we snapshot the config with mysqldump and distribute it with git. Here's the sanity test I referred to: https://groups.google.com/d/msg/snabb-devel/blmDuCgoknc/PP_oMgopiB4J I will be glad to report on our experience and what we change based on our deployment experience during the Juno cycle. *[IrenaB] I think that for “Non SDN Controller” Mechanism Drivers there will be need for some sort of agent to handle port update events even though it might not be required in order to bind the port.* True. Indeed, we do have an agent running on the compute host, and it we are synchronizing it with port updates based on the mechanism described above. Really what I mean is: Can we keep our agent out-of-tree and apart from ML2 and decide for ourselves how to keep it synchronized (instead of using the MQ)? Is there a precedent for doing things this way in an ML2 mech driver (e.g. one of the SDNs)? Cheers! -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [driverlog] Tail-f CI and it's lack of running and it's DriverLog status
Howdy!
Here is a successful Sandbox test from right now:
https://review.openstack.org/#/c/99061/. I don't immediately see how to
list all historical sandbox tests. (The previous ones are from before the
Summit anyway.)
I enabled the CI for the openstack/neutron Gerrit feed now. Here is a
change that it tested right now: https://review.openstack.org/#/c/95526/.
(Voting is disabled on the account and the config conservatively is set to
vote 0 on failure.)
Yes, I believe you can just submit a patch to DriverLog to reflect the
current status.
DriverLog is sourced from default_data.json (
https://github.com/stackforge/driverlog/blob/master/etc/default_data.json#L1007)?
If so then it does reflect the current status:
ci: {
id: tailfncs,
success_pattern: Successful,
failure_pattern: Failed
}
i.e. it specifies which CI account is associated with this driver, and that
corresponds to a CI that is now up and running.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [driverlog] Tail-f CI and it's lack of running and it's DriverLog status
Hi Sean, On 10 June 2014 18:09, Collins, Sean sean_colli...@cable.comcast.com wrote: One of the links that is posted in that review comment for the Tail-f NCS Jenkins timed out for me. http://egg.snabb.co:8080/job/jenkins-ncs/19/ I notice that there is another link included in that review that does work and has the logs: http://88.198.8.227:81/html/ci-logs/2014-06-10_15-52-09 Can you comment on what the egg.snabb.co URL is for? I saw that too and removed the inaccessible link. The more recent reviews show only the link to the logs. The egg.snabb.co URL was inserted by the BUILD_STATS string in Jenkins's template for Gerrit messages. It's a link to the Jenkins web interface that I use for administration, and the timeout is because it's protected behind a firewall. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][ml2] Too much shim rest proxy mechanism drivers in ML2
On 6 June 2014 10:17, henry hly henry4...@gmail.com wrote: ML2 mechanism drivers are becoming another kind of plugins. Although they can be loaded together, but can not work with each other. [...] Could we remove all device related adaption(rest/ssh/netconf/of... proxy) from these mechanism driver to the agent side, leaving only necessary code in the plugin? In the Snabb NFV mech driver [*] we are trying a design that you might find interesting. We stripped the mech driver down to bare bones and declared that the agent has to access the Neutron configuration independently. In practice this means that our out-of-tree agent is connecting to Neutron's MySQL database directly instead of being fed config changes by custom sync code in ML2. This means there are very little work for the mech driver to do (in our case check configuration and perform special port binding). We are also trying to avoid running an OVS/LinuxBridge-style agent on the compute hosts in order to keep the code footprint small. I hope we will succeed -- I'd love to hear if somebody else is running agent-less? Currently we depend on a really ugly workaround to make VIF binding succeed and we are looking for a clean alternative: https://github.com/lukego/neutron/commit/31d6d0657aeae9fd97a63e4d53da34fb86be92f7 [*] Snabb NFV mech driver code: https://review.openstack.org/95711 Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [driverlog] Tail-f CI and it's lack of running and it's DriverLog status
Howdy Kyle, On 9 June 2014 22:37, Kyle Mestery mest...@noironetworks.com wrote: After talking with various infra folks, we've noticed the Tail-f CI system is not voting anymore. According to some informal research, the last run for this CI setup was in April [1]. Can you verify this system is still running? We will need this to be working by the middle of Juno-2, with a history of voting or we may remove the Tail-f driver from the tree. The history is that I have debugged the CI setup using the Sandbox repo hooks. Then I shut that down. The next step is to bring it up and connect it to the Neutron project Gerrit hook. I'll get on to that -- thanks for the prod. I am being very conservative about making changes to the way I interact with the core CI infrastructure because frankly I am scared of accidentally creating unintended wide-reaching consequences :). Also, along these lines, I'm curious why DriverLog reports this driver Green and as tested [2]. What is the criteria for this? I'd like to propose a patch changing this driver from Green to something else since it's not running for the past few months. Fair question. I am happy to make the DriverLog reflect reality. Is DriverLog doing this based on the presence of a 'ci' section in default_data.json? (Is the needed patch to temporarily remove that section?) I'll focus on getting my CI hooked up to the Neutron project hook in order to moot this issue anyway. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][NFV] NFV BoF at design summit
Can't wait :-). On 14 May 2014 19:06, Chris Wright chr...@redhat.com wrote: Thursday at 1:30 PM in the Neutron Pod we'll do an NFV BoF. If you are at design summit and interested in Neutron + NFV please come join us. thanks, -chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [3rd party testing] How to setup CI? Take #2
Howdy! Here's some follow-up on setting up devstack-vm-gate as a 3rd party. On 13 March 2014 15:30, Luke Gorrie l...@tail-f.com wrote: 1. I need to enable an ML2 mech driver. How can I do this? I have been trying to create a localrc with a Q_ML2_PLUGIN_MECHANISM_DRIVERS=... line, but it appears that the KEEP_LOCALRC option in devstack-gate is broken (confirmed on #openstack-infra). 2. How do I streamline which tests are run? I tried added export DEVSTACK_GATE_TEMPEST_REGEX=network in the Jenkins job configuration but I don't see any effect. (word on #openstack-infra is this option is not used by them so status unknown.) Now we have diagnosed (on #openstack-qa) and submitted fixes to devstack-gate for both of these problems. Links: https://review.openstack.org/#/c/80359/ (for localrc) and https://review.openstack.org/#/c/80566/ (for regex). 3. How do I have Jenkins copy the log files into a directory on the Jenkins master node (that I can serve up with Apache)? This is left as an exercise to the reader in the blog tutorial but I would love a cheat, since I am getting plenty of exercise already :-). This is still open for me. I have some tips from IRC (thanks Jay) but I haven't been able to make them work yet. I also have the meta-question: How can I test changes/fixes to devstack-gate? We found a solution for this now. If you add this line to the Jenkins job: export SKIP_DEVSTACK_GATE_PROJECT=1 then I can edit /opt/stack/new/devstack-gate/devstack-vm-gate.sh without it being overwritten on each test run. That makes it possible to do work on the script. (Important: remember to also remove the lines from the Jenkins job that do a git reset --hard to HEAD.) I also have an issue that worries me. I once started seeing tempest tests failing due to a resource leak where the kernel ran out of loopback mounts and that broke tempest. This issue hasn't popped up again. Overall it's fun to be able to hang out on IRC and make improvements to the OpenStack infrastructure tools. On the other hand, I've now invested about a week of effort and I still don't have the basic devstack-vm-gate working reliably, let alone testing the driver that I am interested in. So I find it's a bit tough as a small vendor to comply with the new CI rules. Lack of familiarity with the overall toolchain + 30-minute turnaround time on testing each change really kills my productivity. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [3rd party testing] How to setup CI? Take #2
Howdy! I have some tech questions I'd love some pointers on from people who've succeeded in setting up CI for Neutron based on the upstream devstack-gate. Here are the parts where I'm blocked now: 1. I need to enable an ML2 mech driver. How can I do this? I have been trying to create a localrc with a Q_ML2_PLUGIN_MECHANISM_DRIVERS=... line, but it appears that the KEEP_LOCALRC option in devstack-gate is broken (confirmed on #openstack-infra). 2. How do I streamline which tests are run? I tried added export DEVSTACK_GATE_TEMPEST_REGEX=network in the Jenkins job configuration but I don't see any effect. (word on #openstack-infra is this option is not used by them so status unknown.) 3. How do I have Jenkins copy the log files into a directory on the Jenkins master node (that I can serve up with Apache)? This is left as an exercise to the reader in the blog tutorial but I would love a cheat, since I am getting plenty of exercise already :-). I also have the meta-question: How can I test changes/fixes to devstack-gate? I've attempted many times to modify how scripts work, but I don't have a global understanding of the whole openstack-infra setup, and somehow my changes always end up being clobbered by a fresh checkout from the upstream repo on Github. That's crazy frustrating when it takes 10+ minutes to fire up a test via Jenkins even when I'm only e.g. trying to add an echo to a shell script somewhere to see what's in an environment variable at a certain point in a script. I'd love a faster edit-compile-run loop, especially one that doesn't involve needing to get changed merged upstream into the official openstack-infra repo. I also have an issue that worries me. I once started seeing tempest tests failing due to a resource leak where the kernel ran out of loopback mounts and that broke tempest. Here is what I saw: root@egg-slave:~# losetup -a /dev/loop0: [fc00]:5248399 (/opt/stack/data/swift/drives/images/swift.img) /dev/loop1: [fc00]:5248409 (/opt/stack/data/stack-volumes-backing-file) /dev/loop2: [fc00]:5248467 (/opt/stack/data/swift/drives/images/swift.img) /dev/loop3: [fc00]:5248496 (/opt/stack/data/swift/drives/images/swift.img) /dev/loop4: [fc00]:5248702 (/opt/stack/data/swift/drives/images/swift.img) /dev/loop5: [fc00]:5248735 (/opt/stack/data/swift/drives/images/swift.img) /dev/loop6: [fc00]:5248814 (/opt/stack/data/swift/drives/images/swift.img) /dev/loop7: [fc00]:5248825 (/opt/stack/data/swift/drives/images/swift.img) and trying to remove this with 'losetup -d ...' had no effect. I rebooted. (I'm on Ubuntu 13.10.) This kind of spurious error has the potential to cause my CI to start casting negative votes (again) and upsetting everybody's workflows, not because my tests have actually found a problem but just because it's a non-trivial problem for me to keep a devstack-gate continuously operational. I hope that doesn't happen, but with this level of infrastructure complexity it does feel a little like playing russian roulette that the next glitch in devstack/devstack-gate/Jenkins/Gerrit/Zuul/Gearman/... will manifest itself in the copy that's running on my server. /vent Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [3rd party testing] How to setup CI? Take #2
oh and in my haste I forgot to say: thank you extremely much to everybody who's been giving me pointers on IRC and especially to Jay for the blog walkthrough! On 13 March 2014 15:30, Luke Gorrie l...@tail-f.com wrote: Howdy! I have some tech questions I'd love some pointers on from people who've succeeded in setting up CI for Neutron based on the upstream devstack-gate. Here are the parts where I'm blocked now: 1. I need to enable an ML2 mech driver. How can I do this? I have been trying to create a localrc with a Q_ML2_PLUGIN_MECHANISM_DRIVERS=... line, but it appears that the KEEP_LOCALRC option in devstack-gate is broken (confirmed on #openstack-infra). 2. How do I streamline which tests are run? I tried added export DEVSTACK_GATE_TEMPEST_REGEX=network in the Jenkins job configuration but I don't see any effect. (word on #openstack-infra is this option is not used by them so status unknown.) 3. How do I have Jenkins copy the log files into a directory on the Jenkins master node (that I can serve up with Apache)? This is left as an exercise to the reader in the blog tutorial but I would love a cheat, since I am getting plenty of exercise already :-). I also have the meta-question: How can I test changes/fixes to devstack-gate? I've attempted many times to modify how scripts work, but I don't have a global understanding of the whole openstack-infra setup, and somehow my changes always end up being clobbered by a fresh checkout from the upstream repo on Github. That's crazy frustrating when it takes 10+ minutes to fire up a test via Jenkins even when I'm only e.g. trying to add an echo to a shell script somewhere to see what's in an environment variable at a certain point in a script. I'd love a faster edit-compile-run loop, especially one that doesn't involve needing to get changed merged upstream into the official openstack-infra repo. I also have an issue that worries me. I once started seeing tempest tests failing due to a resource leak where the kernel ran out of loopback mounts and that broke tempest. Here is what I saw: root@egg-slave:~# losetup -a /dev/loop0: [fc00]:5248399 (/opt/stack/data/swift/drives/images/swift.img) /dev/loop1: [fc00]:5248409 (/opt/stack/data/stack-volumes-backing-file) /dev/loop2: [fc00]:5248467 (/opt/stack/data/swift/drives/images/swift.img) /dev/loop3: [fc00]:5248496 (/opt/stack/data/swift/drives/images/swift.img) /dev/loop4: [fc00]:5248702 (/opt/stack/data/swift/drives/images/swift.img) /dev/loop5: [fc00]:5248735 (/opt/stack/data/swift/drives/images/swift.img) /dev/loop6: [fc00]:5248814 (/opt/stack/data/swift/drives/images/swift.img) /dev/loop7: [fc00]:5248825 (/opt/stack/data/swift/drives/images/swift.img) and trying to remove this with 'losetup -d ...' had no effect. I rebooted. (I'm on Ubuntu 13.10.) This kind of spurious error has the potential to cause my CI to start casting negative votes (again) and upsetting everybody's workflows, not because my tests have actually found a problem but just because it's a non-trivial problem for me to keep a devstack-gate continuously operational. I hope that doesn't happen, but with this level of infrastructure complexity it does feel a little like playing russian roulette that the next glitch in devstack/devstack-gate/Jenkins/Gerrit/Zuul/Gearman/... will manifest itself in the copy that's running on my server. /vent Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [3rd party testing] How to setup CI? Take #2
On 4 March 2014 17:07, Jay Pipes jaypi...@gmail.com wrote: I would advise dropping the custom CI setup and going with a method that specifically uses the upstream openstack-dev/devstack and openstack-infra/devstack-gate projects. This sounds great to me. Thank you for all the work you are doing on simplifying the baseline CI setup. The ideal situation from my perspective would be to use a standard upstream script to create a working CI that can make real tempest runs and vote with my account based on the results (to the sandbox initially). Then I'd branch this script to do the setup that's specific for my driver and to selectively disable tests that are not relevant (if needed). Then once it's looking good the votes could move from the sandbox to the mainline. In future cycles when the CI requirements change I would pull the new upstream scripts and rebase my branch onto them. This could perhaps run in parallel into the sandbox before taking over the mainline work. This seems to be the direction that you are taking things and that sounds wonderful to me. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack-Dev] [Cinder] Open Source and community working together
On 3 March 2014 18:30, Thierry Carrez thie...@openstack.org wrote: My advice was therefore that you should not wait for that to happen to engage in cooperative behavior, because you don't want to be the first company to get singled out. Cooperative behavior is vague. Case in point: I have not successfully setup 3rd party CI for the ML2 driver that I've developed on behalf of a vendor. Does this make me one of your uncooperative vendors? Do I need to worry about being fired because somebody at OpenStack decides to name and shame the company I'm doing the work for and make an example? (Is that what the deprecated neutron drivers list will be used for?) If one project official says driver contributors have to comply with X, Y, Z by Icehouse-2 and then another project official says that uncooperative contributors are going to be nailed to the wall then, well, sucks to be contributors. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack-Dev] [Cinder] Open Source and community working together
On 4 March 2014 11:40, Thierry Carrez thie...@openstack.org wrote: This is a technical requirement, and failing to match those requirements is clearly not the same as engaging in deception or otherwise failing the OpenStack community code of conduct. Thank you for clearing that up! ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [3rd party testing] How to setup CI? Take #2
Hi Jay, (Switching Subject to third party testing) On 4 March 2014 15:13, Jay Pipes jaypi...@gmail.com wrote: Let me know how we can help you! Thanks for the invitation! I will take you up on it :-). My goal is to make sure the Tail-f NCS mechanism driver is fully supported in Icehouse. Question: How should I setup CI? Option 1: Should I debug the CI implementation we developed and tested back in December, which was not based on Tempest but rather on our private integration test method that we used in the Havana cycle? The debugging that is needed is more defensive programming in our automated attempt to setup OpenStack for test -- so that if the installation fails for some unexpected reason we don't vote based on that. Option 2: Should I start over with a standard Tempest test insead? If so, what's the best method to set it up (yours? Arista's? another?), and how do I know when that method is sufficiently debugged that it's time to start? I was on the 3rd party testing meeting last night (as 'lukego') and your recommendation for me was to hold off for a week or so and then try your method after your next update. That sounds totally fine to me in principle. However, this will mean that I don't have a mature test procedure in place by March 14th, and I'm concerned that there may be bad consequences on this. This date was mentioned as a deadline in the Neutron meeting last night, but I don't actually understand what the consequence of non-compliance is for established drivers such as this one. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack-Dev] [Cinder] Open Source and community working together
On 3 March 2014 11:27, Thierry Carrez thie...@openstack.org wrote: It will certainly hurt the first one we nail on the wall. So here is one reputational pressure: you don't want to be that company. [1] http://fnords.wordpress.com/2014/02/24/the-dilemma-of-open-innovation/ -1. That's a really harsh threat being made against a really vaguely defined group. I don't want to have to read between the lines on threats posted to openstack-dev to see if my reputation will be in tatters in the morning. This spoiled my day today, and I have nothing to do with whatever incident you guys with privileged information are talking about. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron][ml2] Maintaining support for the Tail-f NCS mech driver in Icehouse
Howdy! My name is Luke and I'm helping my friends at Tail-f Systems to support Neutron with their NCS [1] product. This went really smoothly for us on the Havana cycle, but lately we're having a harder time with Icehouse. In particular, our attempt to fulfill the 3rd party testing requirements has caused a lot of frustration for the #openstack-infra team around New Year. So I'm writing now to look for a good solution. Our goal for Icehouse is to keep our mechanism driver officially supported. The code already works, and has unit tests to keep it working. The risk we want to avoid is going on the dreaded deprecated list for some other reason, which would confuse our customers. For background, our mechanism driver is about 150 lines of code. It translates each network/port/subnet API call into a REST/JSON notification to an external system. That external system returns HTTP 200 OK. That's about all. It's a pretty trivial program. In December I sat down with Tobbe Tornqvist and we tried to setup Jenkins 3rd party testing. We created a Vagrantfile that spins up an Ubuntu VM, installs Neutron and NCS, and performs integration tests with them connected together. We hooked this into Jenkins with a service account. This went fine to start with, but after Christmas our tests started failing due to random setup issues with 'tox', and the script started making -1 votes. Those -1's created major headaches for the infrastructure team and others upstream, I am sorry to say, and ended up requiring a messy process of manual cleanup, and a public flogging for us on IRC. Bad feeling all around ... And that's where we are today. Now, reviewing the situation, I would love to find a solution that doesn't make us deprecated _and_ doesn't require us to be so deeply hooked into the OpenStack Jenkins infrastructure. Could we, for example, write an accurate emulator for the external system so that the MD code can be tested on OpenStack's own servers? That would suit us very well. It seems like a reasonable request to make given the simplicity of our driver code. Hoping for a simple solution... Cheers, -Luke friends at Tail-f [1] http://blog.ipspace.net/2013/05/tail-f-network-control-system-first.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [snabb-devel] RE: [Neutron] Building a new open source NFV system for Neutron
On 23 January 2014 17:42, Calum Loudon calum.lou...@metaswitch.com wrote: That sounds fantastic. As an NFV application developer I'm very pleased to see this contribution which looks to eliminate the key bottleneck hitting the performance of very high packet throughput apps on OpenStack. Thanks for the kind words! A couple of questions on features and implementation: 1. If I create a VM with say neutron and Open vSwitch then I get a TAP device + Linux bridge + veth device between the VM and the vSwitch, with the Linux bridge needed for implementing anti-spoofing iptables rules/ security group support. What will the stack look like with your NFV driver in place? Will your stack implement equivalent security functions, or will those functions not be available? Snabb NFV will implement equivalent security functions, and these will be configured via the standard Neutron APIs for Ports and Security Groups. Our goal is to offload most of these functions to the NIC using hardware features like Intel's Flow Director. Standard NICs actually have hardware sitting idle that can do most of the work that iptables/ebtables/ovs/bridge is doing for OpenStack -- we hope to put this hardware to work and free up the CPU for running VMs. The Snabb Switch traffic plane is internally structured as a network of apps that each implement one networking function and are connected by virtual Ethernet links (shared memory rings). This is a fairly accurate illustration of the internal components of Snabb NFV: https://github.com/SnabbCo/snabbswitch/blob/snabbnfv-readme/src/designs/nfv/README.md#snabb-switch-operation 2. Are you planning to support live migration? Good question! This is a priority if-and-only-if KVM's basic live migration mechanism is adequate for NFV applications. Do you know? (are some operators evaluating KVM for live migration and concluding that it is practical?) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Building a new open source NFV system for Neutron
Howdy Ian! Thanks for the background on the Passthrough work. I reckon the best choice for us now is to use the traditional Neutron APIs instead of Passthrough. I think they cover all of our use cases as it stands now (many thanks to you for your earlier help with working this out :)). The idea is to put the SR-IOV hardware to work behind-the-scenes of a normal software switch. We will definitely check out the Passthrough when it's ready and see if we should also support that somehow. On 11 January 2014 01:04, Ian Wells ijw.ubu...@cack.org.uk wrote: Hey Luke, If you look at the passthrough proposals, the overview is that part of the passthrough work is to ensure there's an PCI function available to allocate to the VM, and part is to pass that function on to the Neutron plugin via conventional means. There's nothing that actually mandates that you connect the SRIOV port using the passthrough mechanism, and we've been working on the assumption that we would be supporting the 'macvtap' method of attachment that Mellanox came up with some time ago. I think what we'll probably have is a set of standard attachments (including passthrough) added to the Nova drivers - you'll see in the virtualisation drivers that Neutron already gets to tell Nova how to attach the port and can pass auxiliary information - and we will pass the PCI path and, optionally, other parameters to Neutron in the port-update that precedes VIF plugging. That would leave you with the option of passing the path back and requesting an actual passthrough or coming up with some other mechanism of your own choosing (which may not involve changing Nova at all, if you're using your standard virtual plugging mechanism). -- Ian. On 10 January 2014 19:26, Luke Gorrie l...@snabb.co wrote: Hi Mike, On 10 January 2014 17:35, Michael Bright mjbrigh...@gmail.com wrote: Very pleased to see this initiative in the OpenStack/NFV space. Glad to hear it! A dumb question - how do you see this related to the ongoing [openstack-dev] [nova] [neutron] PCI pass-through network support discussion on this list? Do you see that work as one component within your proposed architecture for example or an alternative implementation? Good question. I'd like to answer separately about the underlying technology on the one hand and the OpenStack API on the other. The underlying technology of SR-IOV and IOMMU hardware capabilities are the same in PCI pass-through and Snabb NFV. The difference is that we introduce a very thin layer of software over the top that preserves the basic zero-copy operation while adding a Virtio-net abstraction towards the VM, packet filtering, tunneling, and policing (to start off with). The design goal is to add quite a bit of functionality with only a modest processing cost. The OpenStack API question is more open. How should we best map our functionality onto Neutron APIs? This is something we need to thrash out together with the community. Our current best guess - which surely needs much revision, and is not based on the PCI pass-through blueprint - is here: https://github.com/SnabbCo/snabbswitch/tree/snabbnfv-readme/src/designs/nfv#neutron-configuration Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Neutron] Building a new open source NFV system for Neutron
Howdy Stackers! We are developing a new open source Network Functions Virtualization driver for Neutron. I am writing to you now to ask for early advice that could help us to smoothly bring this work upstream into OpenStack Juno. The background is that we are open source developers working to satisfy the NFV requirements of large service provider networks including Deutsche Telekom's TeraStream project [1] [2]. We are developing a complete NFV stack for this purpose: from the DPDK-like traffic plane all the way up to the Neutron ML2 driver. We are developing against Havana, we attended the Icehouse summit and had a lot of great discussions in Hong Kong, and our ambition is to start bringing running code upstream into Juno. Our work is 100% open source and we want to work in the open with the wider OpenStack community. Currently we are in heads-down hacking mode on the core functionality, but it would be wonderful to connect with the upstream communities who we hope to be working with more in the future (that's you guys). More details on Github: https://github.com/SnabbCo/snabbswitch/tree/snabbnfv-readme/src/designs/nfv Thanks for reading! Cheers, -Luke [1] Ivan Pepelnjak on TeraStream: http://blog.ipspace.net/2013/11/deutsche-telekom-terastream-designed.html [2] Peter Löthberg's presentation on TeraStream at RIPE 67: https://ripe67.ripe.net/archives/video/3/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Building a new open source NFV system for Neutron
Hi Mike, On 10 January 2014 17:35, Michael Bright mjbrigh...@gmail.com wrote: Very pleased to see this initiative in the OpenStack/NFV space. Glad to hear it! A dumb question - how do you see this related to the ongoing [openstack-dev] [nova] [neutron] PCI pass-through network support discussion on this list? Do you see that work as one component within your proposed architecture for example or an alternative implementation? Good question. I'd like to answer separately about the underlying technology on the one hand and the OpenStack API on the other. The underlying technology of SR-IOV and IOMMU hardware capabilities are the same in PCI pass-through and Snabb NFV. The difference is that we introduce a very thin layer of software over the top that preserves the basic zero-copy operation while adding a Virtio-net abstraction towards the VM, packet filtering, tunneling, and policing (to start off with). The design goal is to add quite a bit of functionality with only a modest processing cost. The OpenStack API question is more open. How should we best map our functionality onto Neutron APIs? This is something we need to thrash out together with the community. Our current best guess - which surely needs much revision, and is not based on the PCI pass-through blueprint - is here: https://github.com/SnabbCo/snabbswitch/tree/snabbnfv-readme/src/designs/nfv#neutron-configuration Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Availability of external testing logs
Hi guys, On 6 January 2014 14:44, Anita Kuno ante...@anteaya.info wrote: If the account holder of this account is reading this email, responding to it would certainly be a good idea. Apologies for the disturbance! Please do go ahead and disable the voting rights while we work out what's going wrong and get the voting to work reliably. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Availability of external testing logs
On 6 January 2014 18:12, Collins, Sean sean_colli...@cable.comcast.com wrote: How should we handle existing -1's that have been posted? I suggest removing/ignoring those votes until we see if they are spurious. The Tail-f NCS plugin is very simple code and I'd say it's unlikely that any recent changes will have broken it in a non-trivial way. Probably we will find there is no serious issue and it's only teething problems on the test integration. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] IPv6 DHCP options for dnsmasq
On 21 October 2013 19:51, Sean M. Collins s...@coreitpro.com wrote: The motivation is to help Neutron work with IPv6 - which is a must-have for Comcast. Deutsche Telekom too. We are working on making Neutron interoperate well with a service provider network that's based on IPv6. I look forward to talking about this with people in Hong Kong :) Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Neutron] devstack with ml2 setup problem
Howdy! I'm trying to get ml2 up and running with devstack. I'm falling at the first hurdle - getting devstack working with Neutron. I would love a hint! Here is my localrc: disable_service n-net enable_service q-svc enable_service q-agt enable_service q-dhcp enable_service q-l3 enable_service q-meta enable_service neutron # Optional, to enable tempest configuration as part of devstack #enable_service tempest #Q_PLUGIN=ml2 #ENABLE_TENANT_VLANS=True #ML2_VLAN_RANGES=mynetwork:100:200 #Q_ML2_PLUGIN_MECHANISM_DRIVERS=log,ncs DATABASE_PASSWORD=admin RABBIT_PASSWORD=admin SERVICE_TOKEN=admin SERVICE_PASSWORD=admin ADMIN_PASSWORD=admin but after firing up stack and logging into the GUI the system seems not entirely healthy and I see messages like: *Error: *Unauthorized: Unable to retrieve usage information. *Error: *Unauthorized: Unable to retrieve limit information. *Error: *Unauthorized: Unable to retrieve project list. It had looked okay before I tried enabled Neutron. This is with Ubuntu raring in vagrant/virtualbox with 2GB RAM. Any tips appreciated! -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Running unit tests (was: REST-based ML2 MechanismDriver)
Hi guys, Can someone tell me the best way currently to run a subset of Neutron unit tests (e.g. ml2 ones)? The command I got the last time I asked has recently stopped working: On 6 June 2013 17:16, Luke Gorrie l...@snabb.co wrote: The .venv/bin/python run_tests.py ... trick works for me after . I am in business! I can run the full test suite with tox but haven't figured out how to run only a subset of test cases. Hints would be really welcome! Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [ml2] ML2 Sub-Team Meeting tomorrow
I also won't make the meeting today. I have now started writing code for the Tail-f NCS mechanism driver based on Andre's great work. Thanks Andre ! On 10 July 2013 06:53, Andre Pech ap...@aristanetworks.com wrote: Thanks Kyle, I'm unfortunately going to be on a plane during tomorrow's meeting, so wanted to send a quick update on the ML2 mechanism driver infrastructure ( https://blueprints.launchpad.net/neutron/+spec/ml2-mechanism-drivers). Thanks to everyone for the latest rounds of review, I believe that I've incorporated all of the feedback so far and appreciate people taking another look. I'm available most of tomorrow so will try to be responsive to comments so that we can hopefully get this merged and into H2. Thanks Andre On Tue, Jul 9, 2013 at 7:20 PM, Kyle Mestery (kmestery) kmest...@cisco.com wrote: Just a reminder, we have our weekly ML2 sub team meeting Wednesday at 1400UTC. The agenda is on the meeting page here: https://wiki.openstack.org/wiki/Meetings/ML2 We have a number of ML2 related blueprints in review with a hope of getting them in for H2 tomorrow, we'll focus on those in the meeting tomorrow for the most part. Thanks! Kyle ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [networking][ml2] ML2 Mechanism Driver API proposal
12 jun 2013 kl. 09:25 skrev Andre Pech: As promised at the ml2 kickoff meeting last week, attached is our basic proposal for the ml2 mechanism driver API. Great, that is fast work! After getting more familiar with the ml2 plugin code and looking at some of the other blueprints that are looking to make use of the MechanismDriver, we've instead gone with a more simple passthrough model using the existing plugin language of create_network / update_network / delete_network / create_port / update_port / delete_port. Is it intentional to exclude create_subnet / update_subnet / delete_subnet, which are also a part of the QuantumPluginBaseV2 API? We have currently included Subnets in the data model that we want to synchronize via our mechanism driver. So I would like to either include these subnet-related callbacks in the MechanisnDriver API, or find a good reason to eliminate Subnets from our data model so that they don't need to be synchronized. Kyle, do you guys have the same issue with the OpenDaylight plugin? Thanks very much Andre Arista gang for hacking this up so quickly. I am excited to get started on our mechanism driver. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
