Re: [openstack-dev] [tripleo][ci] Having more that one queue for gate pipeline at tripleo

2018-10-11 Thread Clark Boylan 
On Thu, Oct 11, 2018, at 7:17 AM, Ben Nemec wrote:
> 
> 
> On 10/11/18 8:53 AM, Felix Enrique Llorente Pastora wrote:
> > So for example, I don't see why changes at tripleo-quickstart can be 
> > reset if tripleo-ui fails, this is the kind of thing that maybe can be 
> > optimize.
> 
> Because if two incompatible changes are proposed to tripleo-quickstart 
> and tripleo-ui and both end up in parallel gate queues at the same time, 
> it's possible both queues could get wedged. Quickstart and the UI are 
> not completely independent projects. Quickstart has roles for deploying 
> the UI, which means there is a connection there.
> 
> I think the only way you could have independent gate queues is if you 
> had two disjoint sets of projects that could be gated without any use of 
> projects from the other set. I don't think it's possible to divide 
> TripleO in that way, but if I'm wrong then maybe you could do multiple 
> queues.

To follow up on this the Gate pipeline queue that your projects belong to are 
how you indicate to Zuul that there is coupling between these projects. Having 
things set up in this way allows you to ensure (through the Gate and Zuul's 
speculative future states) that a change to one project in the queue can't 
break another because they are tested together.

If your concern is "time to merge" splitting queues won't help all that much 
unless you put all of the unreliable broken code with broken tests in one queue 
and have the reliable code in another queue. Zuul tests everything in parallel 
within a queue. This means that if your code base and its tests are reliable 
you can merge 20 changes all at once and the time to merge for all 20 changes 
is the same as a single change. Problems arise when tests fail and these future 
states have to be updated and retested. This will affect one or many queues.

The fix here is to work on making reliable test jobs so that you can merge all 
20 changes in the span of time it takes to merge a single change.  This isn't 
necessarily easy, but helps you merge more code and be confident it works too.

Clark

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][ci] Having more that one queue for gate pipeline at tripleo

2018-10-11 Thread Ben Nemec 



On 10/11/18 8:53 AM, Felix Enrique Llorente Pastora wrote:
So for example, I don't see why changes at tripleo-quickstart can be 
reset if tripleo-ui fails, this is the kind of thing that maybe can be 
optimize.


Because if two incompatible changes are proposed to tripleo-quickstart 
and tripleo-ui and both end up in parallel gate queues at the same time, 
it's possible both queues could get wedged. Quickstart and the UI are 
not completely independent projects. Quickstart has roles for deploying 
the UI, which means there is a connection there.


I think the only way you could have independent gate queues is if you 
had two disjoint sets of projects that could be gated without any use of 
projects from the other set. I don't think it's possible to divide 
TripleO in that way, but if I'm wrong then maybe you could do multiple 
queues.




On Thu, Oct 11, 2018 at 1:17 PM Emilien Macchi > wrote:




On Thu, Oct 11, 2018 at 10:01 AM Felix Enrique Llorente Pastora
mailto:ellor...@redhat.com>> wrote:

Hello there,

    After suffering a lot from zuul's tripleo gate piepeline
queue reseting after failures on patches I have ask myself what
would happend if we have more than one queue for gating tripleo.

    After a quick read here
https://zuul-ci.org/docs/zuul/user/gating.html, I have found the
following:

"If changes with cross-project dependencies do not share a
change queue then Zuul is unable to enqueue them together, and
the first will be required to merge before the second is enqueued."

    So it make sense to share zuul queue, but maybe only one
queue for all tripleo projects is too  much, for example sharing
queue between tripleo-ui and tripleo-quickstart, maybe we need
for example to queues for product stuff and one for CI, so
product does not get resetted if CI fails in a patch.

    What do you think ?

Probably a wrong example, as TripleO UI gate is using CI jobs
running tripleo-quickstart scenarios.
We could create more queues for projects which are really
independent from each other but we need to be very careful about it.
-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Quique Llorente

Openstack TripleO CI

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][ci] Having more that one queue for gate pipeline at tripleo

2018-10-11 Thread Felix Enrique Llorente Pastora 
So for example, I don't see why changes at tripleo-quickstart can be reset
if tripleo-ui fails, this is the kind of thing that maybe can be optimize.

On Thu, Oct 11, 2018 at 1:17 PM Emilien Macchi  wrote:

>
>
> On Thu, Oct 11, 2018 at 10:01 AM Felix Enrique Llorente Pastora <
> ellor...@redhat.com> wrote:
>
>> Hello there,
>>
>>After suffering a lot from zuul's tripleo gate piepeline queue
>> reseting after failures on patches I have ask myself what would happend if
>> we have more than one queue for gating tripleo.
>>
>>After a quick read here https://zuul-ci.org/docs/zuul/user/gating.html,
>> I have found the following:
>>
>> "If changes with cross-project dependencies do not share a change queue
>> then Zuul is unable to enqueue them together, and the first will be
>> required to merge before the second is enqueued."
>>
>>So it make sense to share zuul queue, but maybe only one queue for all
>> tripleo projects is too  much, for example sharing queue between tripleo-ui
>> and tripleo-quickstart, maybe we need for example to queues for product
>> stuff and one for CI, so product does not get resetted if CI fails in a
>> patch.
>>
>>What do you think ?
>>
>
> Probably a wrong example, as TripleO UI gate is using CI jobs running
> tripleo-quickstart scenarios.
> We could create more queues for projects which are really independent from
> each other but we need to be very careful about it.
> --
> Emilien Macchi
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


-- 
Quique Llorente

Openstack TripleO CI
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][ci] Having more that one queue for gate pipeline at tripleo

2018-10-11 Thread Emilien Macchi 
On Thu, Oct 11, 2018 at 10:01 AM Felix Enrique Llorente Pastora <
ellor...@redhat.com> wrote:

> Hello there,
>
>After suffering a lot from zuul's tripleo gate piepeline queue reseting
> after failures on patches I have ask myself what would happend if we have
> more than one queue for gating tripleo.
>
>After a quick read here https://zuul-ci.org/docs/zuul/user/gating.html,
> I have found the following:
>
> "If changes with cross-project dependencies do not share a change queue
> then Zuul is unable to enqueue them together, and the first will be
> required to merge before the second is enqueued."
>
>So it make sense to share zuul queue, but maybe only one queue for all
> tripleo projects is too  much, for example sharing queue between tripleo-ui
> and tripleo-quickstart, maybe we need for example to queues for product
> stuff and one for CI, so product does not get resetted if CI fails in a
> patch.
>
>What do you think ?
>

Probably a wrong example, as TripleO UI gate is using CI jobs running
tripleo-quickstart scenarios.
We could create more queues for projects which are really independent from
each other but we need to be very careful about it.
-- 
Emilien Macchi
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo][ci] Having more that one queue for gate pipeline at tripleo

2018-10-11 Thread Felix Enrique Llorente Pastora 
Hello there,

   After suffering a lot from zuul's tripleo gate piepeline queue reseting
after failures on patches I have ask myself what would happend if we have
more than one queue for gating tripleo.

   After a quick read here https://zuul-ci.org/docs/zuul/user/gating.html,
I have found the following:

"If changes with cross-project dependencies do not share a change queue
then Zuul is unable to enqueue them together, and the first will be
required to merge before the second is enqueued."

   So it make sense to share zuul queue, but maybe only one queue for all
tripleo projects is too  much, for example sharing queue between tripleo-ui
and tripleo-quickstart, maybe we need for example to queues for product
stuff and one for CI, so product does not get resetted if CI fails in a
patch.

   What do you think ?

-- 
Quique Llorente

Openstack TripleO CI
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev