Re: [openstack-dev] [infra] [cinder] CI via infra for the DRBD Cinder driver
this is a reflection of the discussion I just had on #openstack-infra; it's about (re-)using the central CI infrastructure for our Open-Source DRBD driver too. As this is done now, I'd like to discuss a few points. First of all -- thanks to everybody involved; if you happen to find someone (Europe is a bad timezone for Openstack, I guess), people on IRC are *very* helpful. Secondly, «Yup, so the two things I will start with is that multinode testing is still really rudimentary, we only just got tempest sort of working with it. ... » I didn't follow all the conversations on the ML - is there an update for multi-node testing? What's the policy? Good idea, but not needed or A must-have? Especially for distributed/replicated storage drivers (like DRBD ;) I guess it would make sense to have some. Third point, mostly of interest for other driver maintainers: we're following the various CI runs, and sporadically check failures to see whether we can improve our driver or the devstack installation script. For that we use Kibana on http://logstash.openstack.org, with a filter string of project:openstack/cinder AND build_status:FAILURE AND build_name:check-tempest-dsvm-full-drbd-devstack-nv AND Detailed logs This shows the initial line of the failed devstack runs (the one including the line with the log URL), which we can then paste into a script that fetches the (for us) relevant files for further analysis. The newest failures are already at the top. Another nice feature is using the existing Graphite installation to get a visual display of the success/failure ratio over time[1]; here we can see the impact of individual changes, eg. on June 19th we diagnosed (and fixed) an udev/blkid race with the kernel-attach, since then the number of failures has clearly gone down. I just pushed another patch that should us bring even more into the green area ;) One more idea is to watch http://status.openstack.org/zuul/? with a filter string of Cinder, and to open reviews that will finish soon, so that current behaviour can be easily checked. I hope that this helps other people a bit. Regards, Phil Ad 1: http://graphite.openstack.org/render/?width=600height=344_salt=1434709688.361from=-7daystitle=DRBD%20Cinder%2FDevstack%20statscolorList=red%2Cgreen%2Cbluetarget=stats_counts.zuul.pipeline.check.job.check-tempest-dsvm-full-drbd-devstack-nv.FAILUREtarget=stats_counts.zuul.pipeline.check.job.check-tempest-dsvm-full-drbd-devstack-nv.SUCCESS -- : Ing. Philipp Marek : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com : DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [cinder] CI via infra for the DRBD Cinder driver
Anita Kuno ante...@anteaya.info writes: I'd like to make sure that if Infra is saying that CI jobs on drivers can gate that this is a substantial difference from what I have been saying for a year, specifically that they can't and won't. For the purposes of this conversation, the distinction between drivers and any other type of project is not important. The only thing that can not gate is a CI system other than the one we run. That's not a change. Again, most driver CI systems are third-party systems because they must be for technical reasons (and therefore, they can not gate for that reason). If they can be run upstream, there's no reason they can't gate. And there are some instances where we would be strongly advised to -- the default open-source driver for a system, for instance. Most of those happen to be in-tree at the moment. -Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [cinder] CI via infra for the DRBD Cinder driver
On 02/24/2015 04:40 AM, Duncan Thomas wrote: For cinder, we don't want to gate on drivers, and will be unhappy if that gets turned on, so the story hasn't changed, Anita. I think Jim was pointing out the technical possibility, which is a fine and valid thing to point out. Cinder core are likely to continue not to want it, for the foreseeable future. Fair enough. My concern is that by making it technically possible and permissible from infra's side we are just opening up the projects to have to engage in yet another game of whack-a-mole since driver testing ops can and will try to turn it on. Thanks, Anita. On 24 February 2015 at 10:30, Anita Kuno ante...@anteaya.info wrote: On 02/23/2015 07:00 PM, James E. Blair wrote: Clark Boylan cboy...@sapwetik.org writes: The other thing is that we don't have the zuul code to vote with a different account deployed/merged yet. So initially you could run your job but it wouldn't vote against, say, cinder.» The stack that adds the necessary zuul stuff ends with https://review.openstack.org/#/c/121528/ I don't believe that we have any current plans to use that code in infra. I don't believe that it makes sense for us to create multiple accounts in Gerrit for our single system. It's quite a bit of overhead, and I don't believe it is necessary. To be clear, I think any policy that says that drivers must have third-party CI is an oversight. I believe that it's fine to require them to have CI, but if the CI can be provided by infra, it should be. I believe this misunderstanding comes from the fact that most out-of-tree drivers require specialized hardware, or non-free software, that can not be run in our system. As mentioned elsewhere, we're generally quite happy to have any open source component tested in the upstream infrastructure. Since this qualifies, I think the quickest and simplest way to proceed is to create a job that runs on the driver repo, and then create a non-voting version to run on the cinder repo. Additionally, if it proves stable, the Cinder developers could certainly choose to gate on this job as well. I'd like to make sure that if Infra is saying that CI jobs on drivers can gate that this is a substantial difference from what I have been saying for a year, specifically that they can't and won't. Every CI system/job author/operator that I have interacted with for the past year has been trying to figure out how to be in the gate. Since that implies a relationship in which development on the service can't proceed without a driver's say so (by passing the test in the gate) this is exactly the relationship I have been trying to ensure doesn't happen. By stating that driver jobs can gate on the service, this opens the flood gates of every CI operator increasing pressure to be in the gate as well (the pressure never stopped, it just took different forms). I disagree with the relationship where a service's development can only proceed at the speed of a driver (or driver's because where there is one, there are plenty more). Thanks, Anita. That's entirely up to them, but there's no policy or technical reason it can not. -Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [cinder] CI via infra for the DRBD Cinder driver
On Fri, Feb 20, 2015, at 11:24 AM, Philipp Marek wrote: Hi all, this is a reflection of the discussion I just had on #openstack-infra; it's about (re-)using the central CI infrastructure for our Open-Source DRBD driver too. The current status is: * The DRBD driver is already in Cinder, so DRBD-replicated Cinder storage using iSCSI to the hypervisors does work out-of-the-box. * The Nova-parts didn't make it in for Kilo; we'll try to get them into L. * I've got a lib/backends/drbd for devstack, that together with a matching local.conf can set up a node - at least for a limited set of distributions (as DRBD needs a kernel module, Ubuntu/Debian via DKMS are the easy way). [Please note that package installation is not yet done in this script yet - I'm not sure whether I can/may/should simply add an apt-repository.] Now, clarkb told me about two caveats: «Yup, so the two things I will start with is that multinode testing is still really rudimentary, we only just got tempest sort of working with it. So I might suggest running on a single node first to get the general thing working. You can follow the current state of multinode work at https://review.openstack.org/#/c/141530/ The other thing is that we don't have the zuul code to vote with a different account deployed/merged yet. So initially you could run your job but it wouldn't vote against, say, cinder.» The stack that adds the necessary zuul stuff ends with https://review.openstack.org/#/c/121528/ Cinder has a deadline for CI: March 19th; upon relaying that fact (resp. nearly correct date) clarkb said «thats about 3 weeks... probably at least for the zuul thing.» So, actually it's nearly 4 weeks, let's hope that it all works out. Actually, the multi-node testing will only be needed when we get the Nova parts in, because then it would make sense to test (Nova) via both iSCSI and the DRBD transport; for Cinder CI a single-node setup is sufficient. My remaining questions are: * Is it possible to have our driver tested via the common infrastructure? I think it should be, initially just reporting against your devstack plugin while things get sorted out then once zuul has the appropriate bits reporting like a third party test account but running upstream. * Is it okay to setup another apt-repository during the devstack run, to install the needed packages? I'm not sure whether our servers would simply be accessible, some firewall or filtering proxy could break such things easily. It is not ok in a gating job to do that but that isn't your intention so should be fine. That said, isn't drbd available in existing apt/yum repos? What special sauce do you need to pull in? * Apart from the cinder-backend script in devstack (which I'll have to finish first, see eg. package installation), is any other information needed from us? Other than following this thread and writing that devstack plugin probably not. Hope this helps, Clark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [infra] [cinder] CI via infra for the DRBD Cinder driver
Hi all, this is a reflection of the discussion I just had on #openstack-infra; it's about (re-)using the central CI infrastructure for our Open-Source DRBD driver too. The current status is: * The DRBD driver is already in Cinder, so DRBD-replicated Cinder storage using iSCSI to the hypervisors does work out-of-the-box. * The Nova-parts didn't make it in for Kilo; we'll try to get them into L. * I've got a lib/backends/drbd for devstack, that together with a matching local.conf can set up a node - at least for a limited set of distributions (as DRBD needs a kernel module, Ubuntu/Debian via DKMS are the easy way). [Please note that package installation is not yet done in this script yet - I'm not sure whether I can/may/should simply add an apt-repository.] Now, clarkb told me about two caveats: «Yup, so the two things I will start with is that multinode testing is still really rudimentary, we only just got tempest sort of working with it. So I might suggest running on a single node first to get the general thing working. The other thing is that we don't have the zuul code to vote with a different account deployed/merged yet. So initially you could run your job but it wouldn't vote against, say, cinder.» Cinder has a deadline for CI: March 19th; upon relaying that fact (resp. nearly correct date) clarkb said «thats about 3 weeks... probably at least for the zuul thing.» So, actually it's nearly 4 weeks, let's hope that it all works out. Actually, the multi-node testing will only be needed when we get the Nova parts in, because then it would make sense to test (Nova) via both iSCSI and the DRBD transport; for Cinder CI a single-node setup is sufficient. My remaining questions are: * Is it possible to have our driver tested via the common infrastructure? * Is it okay to setup another apt-repository during the devstack run, to install the needed packages? I'm not sure whether our servers would simply be accessible, some firewall or filtering proxy could break such things easily. * Apart from the cinder-backend script in devstack (which I'll have to finish first, see eg. package installation), is any other information needed from us? Thank you for your feedback and any help you can offer! Regards, Phil -- : Ing. Philipp Marek : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com : DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
