Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
On 9 July 2014 20:24, Sullivan, Jon Paul jonpaul.sulli...@hp.com wrote: Incidentally, is there already way to review what votes my CI (or indeed anybody's) is casting via an openstack.org web interface? You can look at the individual account dashboards in Gerrit, like: https://review.openstack.org/#/dashboard/12019 This search seems to omit many entries that I am interested in. For example, the Tail-f CI has tested reviewed hundreds of Neutron changes recently but the search comes up empty. Is this an issue with accounts that don't have the voting bit set? For the use case of bringing new CIs online I would find it very useful to be able to review all the reviews that the CI makes even before it has voting enabled. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
Howdy! I've been operating a shellci for a while now and overall it is very smooth. The main new feature now is to automatically retrigger events that neither definitely succeed (exit status 100) nor definitely fail (exit status 101). In this case the CI will vote 0 with the logs and then automatically schedule a new test, up to a max of 5 tests, until it can conclude +1 or -1. This seems to work well. On my setup I often see tests fail due to network problems (e.g. timeout during a Git checkout) and automatic retries filer away this class of error very neatly, ultimately casting the +1/-1 votes based purely on the relatively reliable contents of tempest.log. (Thanks Jim Gray for pointing out that detectable intermittent bugs are really easy to deal with http://www.hpl.hp.com/techreports/tandem/TR-85.7.pdf). [The network problems themselves are a little suspicious but for now I see them as a blessing that helps me make shellci robust and I don't worry too much about the root cause.] I'm running 5 parallel builds at the moment. I've tried ramping that up to 10 or 20 but then I start to see Vagrant/VirtualBox startup errors. Again, I'm using this as a test case for the automatic retries, and not digging too deep yet. I'd like to be able to track the results via RSS feeds. Can I do this via openstack.org or should I support that directly in shellci? disk space will become an issue soon. Currently I'm running around 500 builds per day and each one generates around 8MB of logs. Compression and redundancy-avoidance may go a long way towards reducing his problem, however it seems a pity to have a scarcity mentality when it comes to logging (I'd prefer to do more rather than less). Any bright ideas for conveniently archiving some terabytes of logs? Overall I'd say that shellci is quite reliable now. The main reason I don't recommend it to others yet is that I'm a bit too busy to provide support with setting it up this week. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
On 10 July 2014 10:06, Luke Gorrie l...@snabb.co wrote: The main new feature now is to automatically retrigger events that neither definitely succeed (exit status 100) nor definitely fail (exit status 101). In this case the CI will vote 0 with the logs and then automatically schedule a new test, up to a max of 5 tests, until it can conclude +1 or -1. ... and while I have your attention, I may as well show you the code for that feature, to help describe the genera style of shellci: https://github.com/SnabbCo/shellci/commit/d9716d83c85e532ad05d04fcd72af3995f66899d ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
If I understand correctly, you are having it comment when there is a retrigger due to an internal CI failure? If so, please don't do this because it makes the Gerrit reviews very noisy and it provides nothing relevant to the contributor submitting the patch. Nobody wants a CI to report that it has nothing useful to say, especially up to five times on a single patch. :-) Howdy! I've been operating a shellci for a while now and overall it is very smooth. The main new feature now is to automatically retrigger events that neither definitely succeed (exit status 100) nor definitely fail (exit status 101). In this case the CI will vote 0 with the logs and then automatically schedule a new test, up to a max of 5 tests, until it can conclude +1 or -1. This seems to work well. On my setup I often see tests fail due to network problems (e.g. timeout during a Git checkout) and automatic retries filer away this class of error very neatly, ultimately casting the +1/-1 votes based purely on the relatively reliable contents of tempest.log. (Thanks Jim Gray for pointing out that detectable intermittent bugs are really easy to deal with http://www.hpl.hp.com/techreports/tandem/TR-85.7.pdf). [The network problems themselves are a little suspicious but for now I see them as a blessing that helps me make shellci robust and I don't worry too much about the root cause.] I'm running 5 parallel builds at the moment. I've tried ramping that up to 10 or 20 but then I start to see Vagrant/VirtualBox startup errors. Again, I'm using this as a test case for the automatic retries, and not digging too deep yet. I'd like to be able to track the results via RSS feeds. Can I do this via openstack.org or should I support that directly in shellci? disk space will become an issue soon. Currently I'm running around 500 builds per day and each one generates around 8MB of logs. Compression and redundancy-avoidance may go a long way towards reducing his problem, however it seems a pity to have a scarcity mentality when it comes to logging (I'd prefer to do more rather than less). Any bright ideas for conveniently archiving some terabytes of logs? Overall I'd say that shellci is quite reliable now. The main reason I don't recommend it to others yet is that I'm a bit too busy to provide support with setting it up this week. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
For log storage I would definitely start with compression since these are just plain text. Make sure you enable gzip decompression in your web server software so people can still view the log files in their browser. Before spending tons of disk space on log storage, I would also have it purge logs from successful runs that are more than a few weeks old. The ones people are interested in are the ones that break. On Jul 10, 2014 1:08 AM, Luke Gorrie l...@snabb.co wrote: Howdy! I've been operating a shellci for a while now and overall it is very smooth. The main new feature now is to automatically retrigger events that neither definitely succeed (exit status 100) nor definitely fail (exit status 101). In this case the CI will vote 0 with the logs and then automatically schedule a new test, up to a max of 5 tests, until it can conclude +1 or -1. This seems to work well. On my setup I often see tests fail due to network problems (e.g. timeout during a Git checkout) and automatic retries filer away this class of error very neatly, ultimately casting the +1/-1 votes based purely on the relatively reliable contents of tempest.log. (Thanks Jim Gray for pointing out that detectable intermittent bugs are really easy to deal with http://www.hpl.hp.com/techreports/tandem/TR-85.7.pdf). [The network problems themselves are a little suspicious but for now I see them as a blessing that helps me make shellci robust and I don't worry too much about the root cause.] I'm running 5 parallel builds at the moment. I've tried ramping that up to 10 or 20 but then I start to see Vagrant/VirtualBox startup errors. Again, I'm using this as a test case for the automatic retries, and not digging too deep yet. I'd like to be able to track the results via RSS feeds. Can I do this via openstack.org or should I support that directly in shellci? disk space will become an issue soon. Currently I'm running around 500 builds per day and each one generates around 8MB of logs. Compression and redundancy-avoidance may go a long way towards reducing his problem, however it seems a pity to have a scarcity mentality when it comes to logging (I'd prefer to do more rather than less). Any bright ideas for conveniently archiving some terabytes of logs? Overall I'd say that shellci is quite reliable now. The main reason I don't recommend it to others yet is that I'm a bit too busy to provide support with setting it up this week. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
Hi again Jon Paul, My mistake! This seems to be exactly what I was looking for, thank you. (I goofed the query which is why I thought it was lacking.) Cheers :) -Luke On 10 July 2014 09:17, Luke Gorrie l...@snabb.co wrote: On 9 July 2014 20:24, Sullivan, Jon Paul jonpaul.sulli...@hp.com wrote: Incidentally, is there already way to review what votes my CI (or indeed anybody's) is casting via an openstack.org web interface? You can look at the individual account dashboards in Gerrit, like: https://review.openstack.org/#/dashboard/12019 This search seems to omit many entries that I am interested in. For example, the Tail-f CI has tested reviewed hundreds of Neutron changes recently but the search comes up empty. Is this an issue with accounts that don't have the voting bit set? For the use case of bringing new CIs online I would find it very useful to be able to review all the reviews that the CI makes even before it has voting enabled. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
From: Luke Gorrie [mailto:l...@snabb.co] Sent: 07 July 2014 10:54 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron][third-party] Simple and robust CI script? On 7 July 2014 11:41, Luke Gorrie l...@snabb.comailto:l...@snabb.co wrote: I'm running an additional non-voting instance that runs five parallel builds and triggers on all OpenStack projects. To clarify: by non-voting I mean not posting any results to review.openstack.orghttp://review.openstack.org at all, to avoid noise. (Posting comments is only enabled for the instance that tracks openstack-dev/sandbox.) Incidentally, is there already way to review what votes my CI (or indeed anybody's) is casting via an openstack.orghttp://openstack.org web interface? You can look at the individual account dashboards in Gerrit, like: https://review.openstack.org/#/dashboard/12019 You can get the user id through the REST API, using a simple wget: “wget https://review.openstack.org/accounts/openstack...@snabb.co” Thanks, Jon-Paul Sullivan ☺ Cloud Services - @hpcloud Postal Address: Hewlett-Packard Galway Limited, Ballybrit Business Park, Galway. Registered Office: Hewlett-Packard Galway Limited, 63-74 Sir John Rogerson's Quay, Dublin 2. Registered Number: 361933 The contents of this message and any attachments to it are confidential and may be legally privileged. If you have received this message in error you should delete it from your system immediately and advise the sender. To any recipient of this message within HP, unless otherwise stated, you should consider this message and attachments as HP CONFIDENTIAL. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
On 3 July 2014 19:05, Luke Gorrie l...@snabb.co wrote: Time to make it start running real tempest tests. Howdy! shellci now supports running n parallel build processes and by default runs each test with devstack+tempest in a one-shot Vagrant VM. The README is updated on Github: https://github.com/SnabbCo/shellci I'm running an additional non-voting instance that runs five parallel builds and triggers on all OpenStack projects. For the curious, this instance's logs are at http://horgen.snabb.co/shellci/log/ and the build directories are under http://horgen.snabb.co/shellci/tests/. This week I should discover how much maintenance is needed to keep it humming along and then we'll see if I can recommend it to anybody else or not. (I don't recommend it yet but I did try to make the README detailed enough in case there is anybody who wants to play now.) Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
On Mon, Jul 7, 2014 at 4:41 AM, Luke Gorrie l...@snabb.co wrote: On 3 July 2014 19:05, Luke Gorrie l...@snabb.co wrote: Time to make it start running real tempest tests. Howdy! shellci now supports running n parallel build processes and by default runs each test with devstack+tempest in a one-shot Vagrant VM. The README is updated on Github: https://github.com/SnabbCo/shellci I'm running an additional non-voting instance that runs five parallel builds and triggers on all OpenStack projects. For the curious, this instance's logs are at http://horgen.snabb.co/shellci/log/ and the build directories are under http://horgen.snabb.co/shellci/tests/. This week I should discover how much maintenance is needed to keep it humming along and then we'll see if I can recommend it to anybody else or not. (I don't recommend it yet but I did try to make the README detailed enough in case there is anybody who wants to play now.) This sounds promising Luke. I for one will be kicking the tires on this and having a look once you bless it. Thanks, Kyle Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
On 1 July 2014 19:12, Luke Gorrie l...@tail-f.com wrote: It does not yet run devstack/tempest and I hope to reuse that part from somebody else's efforts. shellci is happily voting on the sandbox with the Snabb NFV CI account so far: http://egg.snabb.co:81/shellci/shellci.log Time to make it start running real tempest tests. I whipped up a simple Vagrantfile that runs devstack and tempest in a disposable VM. The idea is that out-of-the-box you get a setup that runs tempest and votes on the results. Then you customize local.conf, tempest.conf, and optionally the whole script to do the appropriate testing for your driver. (Or, if you like, skip this part and supply your own testing script to do whatever you like.) Vagrant scripts only in a Gist for now: https://gist.github.com/lukego/bdefc792b8255d141e4c I'll see how the performance looks. Vagrant probably slows down serial performance but should make independent parallel runs easy. I ordered a hetzner.de server with 128GB RAM and if that comes through tomorrow we'll see how that plays out. The plan for parallelism is sharding. Each gerrit-stream event will be hashed into one of N buckets and then you can run N copies of the testing script (on whatever machine(s)) and each copy chooses a different hash bucket to trigger on. Let's see how promising (or not) that looks tomorrow :-). If it works out for me then hopefully somebody else will want to kick the tires next week. (We'll need a separate 100 line budget for the Vagrant/devstack/tempest stuff by the look of it! Lies, damned lies, budgets...) Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
Howdy! I wrote a new version of shellci today and have it up and running and voting on the sandbox. It's described on the Github page: https://github.com/SnabbCo/shellci Currently this is simple shell scripts to receive review.openstack.org gerrit events, run tests and determine results, then post reviews. It does not yet run devstack/tempest and I hope to reuse that part from somebody else's efforts. (I know ODL have done reusable work here and I plan to look into that. Anybody else?) Ideas and encouragement welcome as always. Let's find out if this point in the design space is practical or not. Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron][third-party] Simple and robust CI script?
Howdy! Paging other 3rd party CI operators... I would like to run a simple and robust 3rd party CI. Simple as in a small number of moving parts, robust as in unlikely to make mistakes due to unexpected problems. I'm imagining: - 100 lines of shell for the implementation. - Minimum of daemons. (Just Jenkins? Ideally not even that...) - Robust detection of operational problems (CI bugs) vs system-under-test problems (changes that should be voted against). - Effective notifications on all errors or negative votes. Does anybody already have an implementation like this that they would like to share? (Is anybody else wanting something in this direction?) I've been iterating on 3rd party CI mechanisms (currently onto my 3rd from-scratch setup) and I have not been completely satisfied with any of them. I would love to hear from someone who has a minimalist implementation that they are happy with :). Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
On 30 June 2014 17:34, Kyle Mestery mest...@noironetworks.com wrote: It would be great to get you to join the 3rd Party meeting [1] in #openstack-meeting at 1800UTC to discuss this. Can you make it today Luke? Yes, I'll be there. Currently I'm looking into the simplest 3rd party CI that could possibly work which would be less sophisticated than devstack-gate but easier for the operator to understand and be responsible for. (Or: CI in 100 lines of shell with no Jenkins or other large pieces of software to install.) I'm on #openstack-neutron if anybody wants to kick around ideas or share scripts that I can borrow ideas from (thanks ODL gang for doing this already). Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
Not sure if these are “minimalist” but at least they setup automagically, so you don’t need to do it from scratch: You can check out these repos which automate the 3rd party ci setup: Jay Pipe’s solution: https://github.com/jaypipes/os-ext-testing https://github.com/jaypipes/os-ext-testing-data My WIP fork of Jay Pipe’s that also includes Nodepool: https://github.com/rasselin/os-ext-testing https://github.com/rasselin/os-ext-testing-data Ramy From: Luke Gorrie [mailto:l...@tail-f.com] Sent: Monday, June 30, 2014 9:26 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron][third-party] Simple and robust CI script? On 30 June 2014 17:34, Kyle Mestery mest...@noironetworks.commailto:mest...@noironetworks.com wrote: It would be great to get you to join the 3rd Party meeting [1] in #openstack-meeting at 1800UTC to discuss this. Can you make it today Luke? Yes, I'll be there. Currently I'm looking into the simplest 3rd party CI that could possibly work which would be less sophisticated than devstack-gate but easier for the operator to understand and be responsible for. (Or: CI in 100 lines of shell with no Jenkins or other large pieces of software to install.) I'm on #openstack-neutron if anybody wants to kick around ideas or share scripts that I can borrow ideas from (thanks ODL gang for doing this already). Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
On 30 June 2014 19:37, Asselin, Ramy ramy.asse...@hp.com wrote: Not sure if these are “minimalist” but at least they setup automagically, so you don’t need to do it from scratch: I'm aiming to do exactly the opposite of this i.e. no automagic. My experience has been that the really heavy-duty CI setups are too difficult to understand and troubleshoot for end-users like me. I spent a week working on installing Jay's tools, with very generous help from Jay himself, and for me it was a dead end [*]. I now consider these to be tools for openstack-infra insiders rather than guys like me who are maintaining tiny drivers. I don't know if other people have had a different experience from me. It does seem like 3rd party CI is broadly a lot more problematic than advertised. I have seen only relatively few frank experience reports from people operating CIs and I suspect there is a rich and interesting untold story there :-). (Maybe it would be interesting to do a survey some day.) [*] http://openstack-dev-openstack.git.net/2014-March/030022.html other posts in that thread. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][third-party] Simple and robust CI script?
I have a really early sketch of this project on Github now. shellci - OpenStack 3rd party CI in 100 lines of shell https://github.com/SnabbCo/shellci This is not finished yet but I'll try to use it for the new Neutron mech driver that I want to contribute to Juno. Ideas and encouragement welcome :-). Cheers, -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
