Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

The proposed undercloud installation jobs dependency [0] worked, see the jobs start time [1], [2] confirms that. The resulting delay for the full pipeline is an ~80 minutes, as it was expected. So PTAL folks, I propose to try it out in real gating and see how the tripleo zuul queue gets relieve

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On 5/28/18 11:43 AM, Bogdan Dobrelya wrote: On 5/25/18 6:40 PM, Tristan Cacqueray wrote: Hello Bogdan, Perhaps this has something to do with jobs evaluation order, it may be worth trying to add the dependencies list in the project-templates, like it is done here for example: http://git.openstac

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On 5/25/18 6:40 PM, Tristan Cacqueray wrote: Hello Bogdan, Perhaps this has something to do with jobs evaluation order, it may be worth trying to add the dependencies list in the project-templates, like it is done here for example: http://git.openstack.org/cgit/openstack-infra/project-config/tre

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

Hello Bogdan, Perhaps this has something to do with jobs evaluation order, it may be worth trying to add the dependencies list in the project-templates, like it is done here for example: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml#n9799 It also easier t

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

Job dependencies seem ignored by zuul, see jobs [0],[1],[2] started simultaneously. While I expected them run one by one. According to the patch 568536 [3], [1] is a dependency for [2] and [3]. The same can be observed for the remaining patches in the topic [4]. Is that a bug or I misunderstood

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On 2018-05-16 15:17:52 +0200 (+0200), Bogdan Dobrelya wrote: [...] > My understanding, I may be totally wrong, is that unlike to > packages and repos (do not count OSTree [0]), containers use > layers and can be exported into tarballs with built-in > de-duplication. This makes idea of tossing those

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On 5/16/18 2:17 PM, Jeremy Stanley wrote: On 2018-05-16 11:31:30 +0200 (+0200), Bogdan Dobrelya wrote: [...] I'm pretty sure though with broader containers adoption, openstack infra will catch up eventually, so we all could benefit our upstream CI jobs with affinity based and co-located data ava

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On 2018-05-16 11:31:30 +0200 (+0200), Bogdan Dobrelya wrote: [...] > I'm pretty sure though with broader containers adoption, openstack > infra will catch up eventually, so we all could benefit our > upstream CI jobs with affinity based and co-located data available > around for consequent build st

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On 5/15/18 10:31 PM, Wesley Hayutin wrote: On Tue, May 15, 2018 at 1:29 PM James E. Blair > wrote: Jeremy Stanley mailto:fu...@yuggoth.org>> writes: > On 2018-05-15 09:40:28 -0700 (-0700), James E. Blair wrote: > [...] >> We're also talking abou

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On 2018-05-15 14:52:26 -0600 (-0600), Wesley Hayutin wrote: [...] > The content would then sync to a swift file server on a central > point for ALL the openstack providers or it would be sync'd to > each cloud? [...] We haven't previously requested that all the Infra provider donors support Swift,

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On Tue, May 15, 2018 at 11:42 AM Jeremy Stanley wrote: > On 2018-05-15 17:31:07 +0200 (+0200), Bogdan Dobrelya wrote: > [...] > > * upload into a swift container, with an automatic expiration set, the > > de-duplicated and compressed tarball created with something like: > > # docker save $(dock

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On Mon, May 14, 2018 at 3:16 PM Sagi Shnaidman wrote: > Hi, Bogdan > > I like the idea with undercloud job. Actually if undercloud fails, I'd > stop all other jobs, because it doens't make sense to run them. Seeing the > same failure in 10 jobs doesn't add too much. So maybe adding undercloud > j

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On Tue, May 15, 2018 at 1:29 PM James E. Blair wrote: > Jeremy Stanley writes: > > > On 2018-05-15 09:40:28 -0700 (-0700), James E. Blair wrote: > > [...] > >> We're also talking about making a new kind of job which can continue to > >> run after it's "finished" so that you could use it to do so

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

Jeremy Stanley writes: > On 2018-05-15 09:40:28 -0700 (-0700), James E. Blair wrote: > [...] >> We're also talking about making a new kind of job which can continue to >> run after it's "finished" so that you could use it to do something like >> host a container registry that's used by other jobs

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On 2018-05-15 09:40:28 -0700 (-0700), James E. Blair wrote: [...] > We're also talking about making a new kind of job which can continue to > run after it's "finished" so that you could use it to do something like > host a container registry that's used by other jobs running on the > change. We do

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

Bogdan Dobrelya writes: > * check out testing depends-on things, (Zuul should have done this for you, but yes.) > * build repos and all tripleo docker images from these repos, > * upload into a swift container, with an automatic expiration set, the > de-duplicated and compressed tarball created

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On 5/15/18 5:08 PM, Sagi Shnaidman wrote: Bogdan, I think before final decisions we need to know exactly - what a price we need to pay? Without exact numbers it will be difficult to discuss about. I we need to wait 80 mins of undercloud-containers job to finish for starting all other jobs, it

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On 2018-05-15 17:31:07 +0200 (+0200), Bogdan Dobrelya wrote: [...] > * upload into a swift container, with an automatic expiration set, the > de-duplicated and compressed tarball created with something like: > # docker save $(docker images -q) | gzip -1 > all.tar.xz > (I expect it will be somethi

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On 5/15/18 4:30 PM, James E. Blair wrote: Bogdan Dobrelya writes: Added a few more patches [0], [1] by the discussion results. PTAL folks. Wrt remaining in the topic, I'd propose to give it a try and revert it, if it proved to be worse than better. Thank you for feedback! The next step could

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

Bogdan, I think before final decisions we need to know exactly - what a price we need to pay? Without exact numbers it will be difficult to discuss about. I we need to wait 80 mins of undercloud-containers job to finish for starting all other jobs, it will be about 4.5 hours to wait for result (+

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

Bogdan Dobrelya writes: > Added a few more patches [0], [1] by the discussion results. PTAL folks. > Wrt remaining in the topic, I'd propose to give it a try and revert > it, if it proved to be worse than better. > Thank you for feedback! > > The next step could be reusing artifacts, like DLRN re

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On 2018-05-15 15:22:14 +0200 (+0200), Bogdan Dobrelya wrote: [...] > I mean pipelines as jobs executed in batches, ordered via defined > dependencies, like gitlab pipelines [0]. And those batches can > also be thought of steps, or whatever we call that. [...] Got it. So Zuul refers to that relatio

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On 5/15/18 2:30 PM, Jeremy Stanley wrote: On 2018-05-15 14:07:56 +0200 (+0200), Bogdan Dobrelya wrote: [...] How pipelines may help solve it? Pipelines only alleviate, not solve the problem of waiting. We only want to build pipelines for the main zuul check process, omitting gating and RDO CI (f

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On 2018-05-15 14:07:56 +0200 (+0200), Bogdan Dobrelya wrote: [...] > How pipelines may help solve it? > Pipelines only alleviate, not solve the problem of waiting. We only want to > build pipelines for the main zuul check process, omitting gating and RDO CI > (for now). > > Where are two cases to

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

Let me clarify the problem I want to solve with pipelines. It is getting *hard* to develop things and move patches to the Happy End (merged): - Patches wait too long for CI jobs to start. It should be minutes and not hours of waiting. - If a patch fails a job w/o a good reason, the consequent r

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

Added a few more patches [0], [1] by the discussion results. PTAL folks. Wrt remaining in the topic, I'd propose to give it a try and revert it, if it proved to be worse than better. Thank you for feedback! The next step could be reusing artifacts, like DLRN repos and containers built for patc

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On 5/14/18 10:06 PM, Alex Schultz wrote: On Mon, May 14, 2018 at 10:15 AM, Bogdan Dobrelya wrote: An update for your review please folks Bogdan Dobrelya writes: Hello. As Zuul documentation [0] explains, the names "check", "gate", and "post" may be altered for more advanced pipelines. Is

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On 5/14/18 9:15 PM, Sagi Shnaidman wrote: Hi, Bogdan I like the idea with undercloud job. Actually if undercloud fails, I'd stop all other jobs, because it doens't make sense to run them. Seeing the same failure in 10 jobs doesn't add too much. So maybe adding undercloud job as dependency for

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

On Mon, May 14, 2018 at 10:15 AM, Bogdan Dobrelya wrote: > An update for your review please folks > >> Bogdan Dobrelya writes: >> >>> Hello. >>> As Zuul documentation [0] explains, the names "check", "gate", and >>> "post" may be altered for more advanced pipelines. Is it doable to >>> introduce

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

Hi, Bogdan I like the idea with undercloud job. Actually if undercloud fails, I'd stop all other jobs, because it doens't make sense to run them. Seeing the same failure in 10 jobs doesn't add too much. So maybe adding undercloud job as dependency for all multinode jobs would be great idea. I thin

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

An update for your review please folks Bogdan Dobrelya writes: Hello. As Zuul documentation [0] explains, the names "check", "gate", and "post" may be altered for more advanced pipelines. Is it doable to introduce, for particular openstack projects, multiple check stages/steps as check-1, ch

Re: [openstack-dev] [ci][infra][tripleo] Multi-staged check pipelines for Zuul v3 proposal

Bogdan Dobrelya writes: > Hello. > As Zuul documentation [0] explains, the names "check", "gate", and > "post" may be altered for more advanced pipelines. Is it doable to > introduce, for particular openstack projects, multiple check > stages/steps as check-1, check-2 and so on? And is it possib