Re: [openstack-dev] [ironic][qa][grenade] Release blocked on grenade job not testing from newton

2017-02-10 Thread Vasyl Saienko 
The root cause why ironic grenade job was broken is described in
https://bugs.launchpad.net/ironic/+bug/1663371
In two words, during Ocata we removed DEFAULT_IMAGE_NAME setting logic from
devstack [0]. As soon stable/ocata
was cut for devstack variable DEFAULT_IMAGE_NAME is not visible in grenade,
and default value in nova resources.sh
script was picked [1].

It was fixed by [2] by sourcing ironic vars (now we set DEFAULT_IMAGE_NAME
there) in grenade, to made it available for all
grenade scripts.

The problem described earlier (we do testing upgrade from master to master)
still exist.
It affects not only ironic, but all projects that do not have latest stable
branch (ie stable/ocata now). As soon they cut it all becomes
to normal. But we have a short period of time when all projects testing
upgrades from master to master.
Related bug [3].



[0]
https://github.com/openstack-dev/devstack/commit/d89b175321ac293454ad15caaee13c0ae46b0bd6
[1]
https://github.com/openstack-dev/grenade/blob/master/projects/60_nova/resources.sh#L31
[2] https://review.openstack.org/#/c/431369/
[3] https://bugs.launchpad.net/grenade/+bug/1663505

On Thu, Feb 9, 2017 at 4:02 PM, Jim Rollenhagen 
wrote:

> On Thu, Feb 9, 2017 at 7:00 AM, Jim Rollenhagen 
> wrote:
>
>> Hey folks,
>>
>> Ironic plans to release Ocata this week, once we have a couple small
>> patches
>> and a release note cleanup landed.
>>
>> However, our grenade job is now testing master->master, best I can tell.
>> This
>> is pretty clearly due to this d-s-g commit:
>> https://github.com/openstack-infra/devstack-gate/commit/9c75
>> 2b02fbd57c7021a7c9295bf4d68a0d1973a8
>>
>> Evidence:
>>
>> * it appears to be checking out a change on master into the old side:
>>   http://logs.openstack.org/44/354744/10/check/gate-grenade-ds
>> vm-ironic-ubuntu-xenial/4b395ff/logs/grenade.sh.txt.gz#_
>> 2017-02-09_07_15_32_979
>>
>> * and somewhat coincidentally, our grenade job seems to be broken when
>> master
>>   (ocata) is on the old side, because we now select instance images in our
>>   devstack plugin:
>>   http://logs.openstack.org/44/354744/10/check/gate-grenade-ds
>> vm-ironic-ubuntu-xenial/4b395ff/logs/grenade.sh.txt.gz#_
>> 2017-02-09_08_07_10_946
>>
>> So, we're currently blocking the ironic release on this, as obviously we
>> don't
>> want to release if we don't know upgrades work. As I see it, we have two
>> options:
>>
>> 1) Somehow fix devstack-gate and configure our jobs in project-config
>> such that
>> this job upgrades newton->master. I might need some help on navigating
>> this
>> one.
>>
>> 2) Make our grenade job non-voting for now, release 7.0.0 anyway, and
>> immediately make sure that the stable/ocata branch runs grenade as
>> expected and
>> passes. If it isn't passing, fix what we need to and cut 7.0.1 ASAP.
>>
>
> After talking to Doug and Sean on IRC, I think this is the best
> option. We don't necessarily need to make it non-voting if we
> can fix it quickly (Vasyl is working on this already).
>
> We still have a week to release from the Ocata branch if we need
> to get more things in. They'll just need to go through the backport
> process.
>
> // jim
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][qa][grenade] Release blocked on grenade job not testing from newton

2017-02-09 Thread Jim Rollenhagen 
On Thu, Feb 9, 2017 at 7:00 AM, Jim Rollenhagen 
wrote:

> Hey folks,
>
> Ironic plans to release Ocata this week, once we have a couple small
> patches
> and a release note cleanup landed.
>
> However, our grenade job is now testing master->master, best I can tell.
> This
> is pretty clearly due to this d-s-g commit:
> https://github.com/openstack-infra/devstack-gate/commit/
> 9c752b02fbd57c7021a7c9295bf4d68a0d1973a8
>
> Evidence:
>
> * it appears to be checking out a change on master into the old side:
>   http://logs.openstack.org/44/354744/10/check/gate-grenade-
> dsvm-ironic-ubuntu-xenial/4b395ff/logs/grenade.sh.txt.
> gz#_2017-02-09_07_15_32_979
>
> * and somewhat coincidentally, our grenade job seems to be broken when
> master
>   (ocata) is on the old side, because we now select instance images in our
>   devstack plugin:
>   http://logs.openstack.org/44/354744/10/check/gate-grenade-
> dsvm-ironic-ubuntu-xenial/4b395ff/logs/grenade.sh.txt.
> gz#_2017-02-09_08_07_10_946
>
> So, we're currently blocking the ironic release on this, as obviously we
> don't
> want to release if we don't know upgrades work. As I see it, we have two
> options:
>
> 1) Somehow fix devstack-gate and configure our jobs in project-config such
> that
> this job upgrades newton->master. I might need some help on navigating this
> one.
>
> 2) Make our grenade job non-voting for now, release 7.0.0 anyway, and
> immediately make sure that the stable/ocata branch runs grenade as
> expected and
> passes. If it isn't passing, fix what we need to and cut 7.0.1 ASAP.
>

After talking to Doug and Sean on IRC, I think this is the best
option. We don't necessarily need to make it non-voting if we
can fix it quickly (Vasyl is working on this already).

We still have a week to release from the Ocata branch if we need
to get more things in. They'll just need to go through the backport
process.

// jim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][qa][grenade] Release blocked on grenade job not testing from newton

2017-02-09 Thread Sean Dague 
On 02/09/2017 08:22 AM, Jim Rollenhagen wrote:
> On Thu, Feb 9, 2017 at 7:51 AM, Sean Dague  > wrote:
> 
> On 02/09/2017 07:00 AM, Jim Rollenhagen wrote:
> > Hey folks,
> >
> > Ironic plans to release Ocata this week, once we have a couple
> small patches
> > and a release note cleanup landed.
> >
> > However, our grenade job is now testing master->master, best I can
> tell.
> > This
> > is pretty clearly due to this d-s-g commit:
> >
> 
> https://github.com/openstack-infra/devstack-gate/commit/9c752b02fbd57c7021a7c9295bf4d68a0d1973a8
> 
> 
> >
> > Evidence:
> >
> > * it appears to be checking out a change on master into the old side:
> >
> >
> 
> http://logs.openstack.org/44/354744/10/check/gate-grenade-dsvm-ironic-ubuntu-xenial/4b395ff/logs/grenade.sh.txt.gz#_2017-02-09_07_15_32_979
> 
> 
> >
> > * and somewhat coincidentally, our grenade job seems to be broken when
> > master
> >   (ocata) is on the old side, because we now select instance
> images in our
> >   devstack plugin:
> >
> >
> 
> http://logs.openstack.org/44/354744/10/check/gate-grenade-dsvm-ironic-ubuntu-xenial/4b395ff/logs/grenade.sh.txt.gz#_2017-02-09_08_07_10_946
> 
> 
> >
> > So, we're currently blocking the ironic release on this, as
> obviously we
> > don't
> > want to release if we don't know upgrades work. As I see it, we
> have two
> > options:
> >
> > 1) Somehow fix devstack-gate and configure our jobs in project-config
> > such that
> > this job upgrades newton->master. I might need some help on
> navigating this
> > one.
> >
> > 2) Make our grenade job non-voting for now, release 7.0.0 anyway, and
> > immediately make sure that the stable/ocata branch runs grenade as
> > expected and
> > passes. If it isn't passing, fix what we need to and cut 7.0.1 ASAP.
> >
> > Thoughts?
> 
> By now ironic should have a stable/ocata branch, right? That's the thing
> that is missing -
> 
> https://github.com/openstack-infra/devstack-gate/commit/9c752b02fbd57c7021a7c9295bf4d68a0d1973a8
> 
> 
> 
> A stable/ocata branch doesn't require a release. It's an expectation
> that all the projects have one of those at this point.
> 
> I'd be -1 putting project specific branch selection logic inside
> devstack-gate. Things are already complicated enough there.
> 
> 
> Ironic's release model is cycle-with-intermediary, meaning we don't do
> release
> candidates, and don't necessarily branch at the same time. The deadline
> is next
> week; we were hoping to get ahead of the game this week.
> 
> And a branch does require a release as a branch point (though in most
> cases that's
> an RC1 release), if the project is released at all, AIUI.

Ok, sorry, my failure to understand that's all linked.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][qa][grenade] Release blocked on grenade job not testing from newton

2017-02-09 Thread Jim Rollenhagen 
On Thu, Feb 9, 2017 at 7:51 AM, Sean Dague  wrote:

> On 02/09/2017 07:00 AM, Jim Rollenhagen wrote:
> > Hey folks,
> >
> > Ironic plans to release Ocata this week, once we have a couple small
> patches
> > and a release note cleanup landed.
> >
> > However, our grenade job is now testing master->master, best I can tell.
> > This
> > is pretty clearly due to this d-s-g commit:
> > https://github.com/openstack-infra/devstack-gate/commit/
> 9c752b02fbd57c7021a7c9295bf4d68a0d1973a8
> >
> > Evidence:
> >
> > * it appears to be checking out a change on master into the old side:
> >
> > http://logs.openstack.org/44/354744/10/check/gate-grenade-
> dsvm-ironic-ubuntu-xenial/4b395ff/logs/grenade.sh.txt.
> gz#_2017-02-09_07_15_32_979
> >
> > * and somewhat coincidentally, our grenade job seems to be broken when
> > master
> >   (ocata) is on the old side, because we now select instance images in
> our
> >   devstack plugin:
> >
> > http://logs.openstack.org/44/354744/10/check/gate-grenade-
> dsvm-ironic-ubuntu-xenial/4b395ff/logs/grenade.sh.txt.
> gz#_2017-02-09_08_07_10_946
> >
> > So, we're currently blocking the ironic release on this, as obviously we
> > don't
> > want to release if we don't know upgrades work. As I see it, we have two
> > options:
> >
> > 1) Somehow fix devstack-gate and configure our jobs in project-config
> > such that
> > this job upgrades newton->master. I might need some help on navigating
> this
> > one.
> >
> > 2) Make our grenade job non-voting for now, release 7.0.0 anyway, and
> > immediately make sure that the stable/ocata branch runs grenade as
> > expected and
> > passes. If it isn't passing, fix what we need to and cut 7.0.1 ASAP.
> >
> > Thoughts?
>
> By now ironic should have a stable/ocata branch, right? That's the thing
> that is missing -
> https://github.com/openstack-infra/devstack-gate/commit/
> 9c752b02fbd57c7021a7c9295bf4d68a0d1973a8
>
> A stable/ocata branch doesn't require a release. It's an expectation
> that all the projects have one of those at this point.
>
> I'd be -1 putting project specific branch selection logic inside
> devstack-gate. Things are already complicated enough there.
>

Ironic's release model is cycle-with-intermediary, meaning we don't do
release
candidates, and don't necessarily branch at the same time. The deadline is
next
week; we were hoping to get ahead of the game this week.

And a branch does require a release as a branch point (though in most cases
that's
an RC1 release), if the project is released at all, AIUI.

// jim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][qa][grenade] Release blocked on grenade job not testing from newton

2017-02-09 Thread Doug Hellmann 
Excerpts from Sean Dague's message of 2017-02-09 07:51:41 -0500:
> On 02/09/2017 07:00 AM, Jim Rollenhagen wrote:
> > Hey folks,
> > 
> > Ironic plans to release Ocata this week, once we have a couple small patches
> > and a release note cleanup landed.
> > 
> > However, our grenade job is now testing master->master, best I can tell.
> > This
> > is pretty clearly due to this d-s-g commit:
> > https://github.com/openstack-infra/devstack-gate/commit/9c752b02fbd57c7021a7c9295bf4d68a0d1973a8
> > 
> > Evidence:
> > 
> > * it appears to be checking out a change on master into the old side:
> >  
> > http://logs.openstack.org/44/354744/10/check/gate-grenade-dsvm-ironic-ubuntu-xenial/4b395ff/logs/grenade.sh.txt.gz#_2017-02-09_07_15_32_979
> > 
> > * and somewhat coincidentally, our grenade job seems to be broken when
> > master
> >   (ocata) is on the old side, because we now select instance images in our
> >   devstack plugin:
> >  
> > http://logs.openstack.org/44/354744/10/check/gate-grenade-dsvm-ironic-ubuntu-xenial/4b395ff/logs/grenade.sh.txt.gz#_2017-02-09_08_07_10_946
> > 
> > So, we're currently blocking the ironic release on this, as obviously we
> > don't
> > want to release if we don't know upgrades work. As I see it, we have two
> > options:
> > 
> > 1) Somehow fix devstack-gate and configure our jobs in project-config
> > such that
> > this job upgrades newton->master. I might need some help on navigating this
> > one.
> > 
> > 2) Make our grenade job non-voting for now, release 7.0.0 anyway, and
> > immediately make sure that the stable/ocata branch runs grenade as
> > expected and
> > passes. If it isn't passing, fix what we need to and cut 7.0.1 ASAP.
> > 
> > Thoughts?
> 
> By now ironic should have a stable/ocata branch, right? That's the thing
> that is missing -
> https://github.com/openstack-infra/devstack-gate/commit/9c752b02fbd57c7021a7c9295bf4d68a0d1973a8
> 
> A stable/ocata branch doesn't require a release. It's an expectation
> that all the projects have one of those at this point.

We only create stable branches from releases, so it does require a
release of some kind.

Doug

> 
> I'd be -1 putting project specific branch selection logic inside
> devstack-gate. Things are already complicated enough there.
> 
> -Sean
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][qa][grenade] Release blocked on grenade job not testing from newton

2017-02-09 Thread Sean Dague 
On 02/09/2017 07:00 AM, Jim Rollenhagen wrote:
> Hey folks,
> 
> Ironic plans to release Ocata this week, once we have a couple small patches
> and a release note cleanup landed.
> 
> However, our grenade job is now testing master->master, best I can tell.
> This
> is pretty clearly due to this d-s-g commit:
> https://github.com/openstack-infra/devstack-gate/commit/9c752b02fbd57c7021a7c9295bf4d68a0d1973a8
> 
> Evidence:
> 
> * it appears to be checking out a change on master into the old side:
>  
> http://logs.openstack.org/44/354744/10/check/gate-grenade-dsvm-ironic-ubuntu-xenial/4b395ff/logs/grenade.sh.txt.gz#_2017-02-09_07_15_32_979
> 
> * and somewhat coincidentally, our grenade job seems to be broken when
> master
>   (ocata) is on the old side, because we now select instance images in our
>   devstack plugin:
>  
> http://logs.openstack.org/44/354744/10/check/gate-grenade-dsvm-ironic-ubuntu-xenial/4b395ff/logs/grenade.sh.txt.gz#_2017-02-09_08_07_10_946
> 
> So, we're currently blocking the ironic release on this, as obviously we
> don't
> want to release if we don't know upgrades work. As I see it, we have two
> options:
> 
> 1) Somehow fix devstack-gate and configure our jobs in project-config
> such that
> this job upgrades newton->master. I might need some help on navigating this
> one.
> 
> 2) Make our grenade job non-voting for now, release 7.0.0 anyway, and
> immediately make sure that the stable/ocata branch runs grenade as
> expected and
> passes. If it isn't passing, fix what we need to and cut 7.0.1 ASAP.
> 
> Thoughts?

By now ironic should have a stable/ocata branch, right? That's the thing
that is missing -
https://github.com/openstack-infra/devstack-gate/commit/9c752b02fbd57c7021a7c9295bf4d68a0d1973a8

A stable/ocata branch doesn't require a release. It's an expectation
that all the projects have one of those at this point.

I'd be -1 putting project specific branch selection logic inside
devstack-gate. Things are already complicated enough there.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ironic][qa][grenade] Release blocked on grenade job not testing from newton

2017-02-09 Thread Jim Rollenhagen 
Hey folks,

Ironic plans to release Ocata this week, once we have a couple small patches
and a release note cleanup landed.

However, our grenade job is now testing master->master, best I can tell.
This
is pretty clearly due to this d-s-g commit:
https://github.com/openstack-infra/devstack-gate/commit/9c752b02fbd57c7021a7c9295bf4d68a0d1973a8

Evidence:

* it appears to be checking out a change on master into the old side:

http://logs.openstack.org/44/354744/10/check/gate-grenade-dsvm-ironic-ubuntu-xenial/4b395ff/logs/grenade.sh.txt.gz#_2017-02-09_07_15_32_979

* and somewhat coincidentally, our grenade job seems to be broken when
master
  (ocata) is on the old side, because we now select instance images in our
  devstack plugin:

http://logs.openstack.org/44/354744/10/check/gate-grenade-dsvm-ironic-ubuntu-xenial/4b395ff/logs/grenade.sh.txt.gz#_2017-02-09_08_07_10_946

So, we're currently blocking the ironic release on this, as obviously we
don't
want to release if we don't know upgrades work. As I see it, we have two
options:

1) Somehow fix devstack-gate and configure our jobs in project-config such
that
this job upgrades newton->master. I might need some help on navigating this
one.

2) Make our grenade job non-voting for now, release 7.0.0 anyway, and
immediately make sure that the stable/ocata branch runs grenade as expected
and
passes. If it isn't passing, fix what we need to and cut 7.0.1 ASAP.

Thoughts?

// jim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev