Re: [openstack-dev] DB Datasets CI broken
On Sat, Oct 18, 2014 at 11:02 AM, Jeremy Stanley wrote: > On 2014-10-18 10:45:23 +1100 (+1100), Michael Still wrote: > [...] >> Is it possible to add a verification step to nodepool so that it >> doesn't mark a new image as ready unless it passes some basic sanity >> checks? > > Back in the beforetime, when devstack-gate had scripts which managed > the worker pool as scheduled Jenkins jobs, it would run DevStack > exercises on a test boot of the new image before using it to boot > real images. Of course you can imagine the number of perfectly good > images which were thrown away because of nondeterministic bugs > causing false negative results there, so we probably wouldn't want > to duplicate that exactly, but perhaps something more lightweight > would be a reasonable compromise. > > Anyway, I consider it a good feature request (others may disagree), > just nobody's reimplemented it in nodepool to date. Yeah, I'm starting to think along the lines of adding a simple sanity check to the shell worker in turbo hipster before the real tests run. Things like checking if the git directory exists, and contains a git repo with the branches we need. We could run that pre-flight script (or a variant) of it on images before marking them as real. For reference, what we think happened here is that the cache of SQL databases baked into the image was rsynced from our master while jhesketh was in the process of updating the SQL databases to a more recent version of OpenStack. Michael -- Rackspace Australia ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] DB Datasets CI broken
On 2014-10-18 10:45:23 +1100 (+1100), Michael Still wrote: [...] > Is it possible to add a verification step to nodepool so that it > doesn't mark a new image as ready unless it passes some basic sanity > checks? Back in the beforetime, when devstack-gate had scripts which managed the worker pool as scheduled Jenkins jobs, it would run DevStack exercises on a test boot of the new image before using it to boot real images. Of course you can imagine the number of perfectly good images which were thrown away because of nondeterministic bugs causing false negative results there, so we probably wouldn't want to duplicate that exactly, but perhaps something more lightweight would be a reasonable compromise. Anyway, I consider it a good feature request (others may disagree), just nobody's reimplemented it in nodepool to date. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] DB Datasets CI broken
This was a bad image in nodepool. I've rebuilt the image and killed our pool of workers running the old image and things seem to be ok now. I'm in the process of enqueueing rechecks for every failed turbo-hipster run now, but they'll take some time to all get executed. Thanks for your patience everyone. Is it possible to add a verification step to nodepool so that it doesn't mark a new image as ready unless it passes some basic sanity checks? Thanks, Michael On Sat, Oct 18, 2014 at 8:44 AM, Michael Still wrote: > Hi, > > I've just noticed that the DB Datasets CI (the artist formerly known > as turbo hipster) is failing for many patches. I'm looking into it > now. > > Michael > > -- > Rackspace Australia -- Rackspace Australia ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] DB Datasets CI broken
Hi, I've just noticed that the DB Datasets CI (the artist formerly known as turbo hipster) is failing for many patches. I'm looking into it now. Michael -- Rackspace Australia ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
