Re: [openstack-dev] [ironic] state of the stable/mitaka branches

2017-03-03 Thread Jeremy Stanley
On 2017-03-02 09:56:28 -0800 (-0800), Ihar Hrachyshka wrote:
> On Thu, Mar 2, 2017 at 8:13 AM, Pavlo Shchelokovskyy
>  wrote:
> > I'm also kind of wondering what the grenade job in stable/newton will test
> > after mitaka EOL? upgrade from mitaka-eol tag to stable/newton branch? Then
> > even that might be affected if devstack-gate + project config will not be
> > able to set *_ssh in enabled drivers while grenade will try to use them.
> 
> When a branch is EOLed, grenade jobs using it for old side of the
> cloud are deprovisioned.

By way of explanation the reason for this is that once a branch is
no longer supported and gets closed for new changes, we're unable to
update it to keep it runnable such that we can avoid having it break
the "old" side of those upgrade jobs. From a downstream perspective,
it means that we recommend upgrading _before_ the version you're
running reaches EOL rather than after (because we no longer continue
testing the upgrade process from that point on).
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] state of the stable/mitaka branches

2017-03-02 Thread Ihar Hrachyshka
On Thu, Mar 2, 2017 at 8:13 AM, Pavlo Shchelokovskyy
 wrote:
> I'm also kind of wondering what the grenade job in stable/newton will test
> after mitaka EOL? upgrade from mitaka-eol tag to stable/newton branch? Then
> even that might be affected if devstack-gate + project config will not be
> able to set *_ssh in enabled drivers while grenade will try to use them.

When a branch is EOLed, grenade jobs using it for old side of the
cloud are deprovisioned.

Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] state of the stable/mitaka branches

2017-03-02 Thread Pavlo Shchelokovskyy
Hi Dmitry,

> I'm not sure why removing of *_ssh drivers from master should
> necessary break stable/mitaka, where these drivers are present. Could
> you elaborate?

My main concern is the following: both project-config and devstack-gate are
branch-less. When removing the *_ssh drivers from tree, we have to remove
them from 'enabled_drivers' list in ironic.conf too (so that conductor
would not fail to start). Most our jobs are not setting enabled_drivers
themselves, but use whatever devstack-gate is setting. Currently it enables
either pxe_ssh+pxe_ipmitool, or agent_ssh+agent_ipmitool (depending on
deploy driver starting with 'agent' or not). If we drop *_ssh from
enabled_drivers in devstack-gate, then the jobs that actually use *_ssh
will fail instead.

AFAIU this affects only mitaka branch, so a possible way of moving forward
is actually let it EOL and then continue with *_ssh drivers removal.

I'm also kind of wondering what the grenade job in stable/newton will test
after mitaka EOL? upgrade from mitaka-eol tag to stable/newton branch? Then
even that might be affected if devstack-gate + project config will not be
able to set *_ssh in enabled drivers while grenade will try to use them.

On Thu, Mar 2, 2017 at 12:39 PM, Dmitry Tantsur  wrote:

> On 03/01/2017 08:19 PM, Jay Faulkner wrote:
>
>>
>> On Mar 1, 2017, at 11:15 AM, Pavlo Shchelokovskyy <
>>> pshchelokovs...@mirantis.com> wrote:
>>>
>>> Greetings ironicers,
>>>
>>> I'd like to discuss the state of the gates in ironic and other related
>>> projects for stable/mitaka branch.
>>>
>>> Today while making some test patches to old branches I discovered the
>>> following problems:
>>>
>>> python-ironicclient/stable/mitaka
>>> All unit-test-like jobs are broken due to not handling upper
>>> constraints. Because of it a newer than actually supported
>>> python-openstackclient is installed, which already lacks some modules
>>> python-ironicclient tries to import (these were moved to osc-lib).
>>> I've proposed a patch that copies current way of dealing with upper
>>> constraints in tox envs [0], gates are passing.
>>>
>>> ironic/stable/mitaka
>>> While not actually being gated on, using virtualbmc+ipmitool drivers is
>>> broken. The reason is again related to upper constraints as what happens is
>>> old enough version of pyghmi (from mitaka upper constraints) is installed
>>> with most recent virtualbmc (not in upper constraints), and those versions
>>> are incompatible.
>>> This highlights a question whether we should propose virtualbmc to upper
>>> constraints too to avoid such problems in the future.
>>> Meanwhile a quick fix would be to hard-code the supported virtualbmc
>>> version in the ironic's devstack plugin for mitaka release.
>>> Although not strictly supported for Mitaka release, I'd like that
>>> functionality to be working on stable/mitaka gates to test for upcoming
>>> removal of *_ssh drivers.
>>>
>>> I did not test other projects yet.
>>>
>>>
>> I can attest jobs are broken for stable/mitaka on ironic-lib as well —
>> our jobs build docs unconditionally, and ironic-lib had no docs in Mitaka.
>>
>
> Oh, fun.
>
> Well, the docs job is easy to exclude. But we seem to have
> virtualbmc-based jobs there. As I already wrote in another message, I'm not
> even sure they're supposed to work..


Yes, the ironic-lib:mitaka is kind of completely broken currently :( There
are several pieces that need to be fixed to get it back to healthy state
(even if only to leave it healthy for EOL). Luckily (or not?), most of the
fixes go thru other projects, so it is doable without one giant squashed
commit:

- docs job - need to be disabled for ironic-lib/mitaka in project-config

- gate-tempest-dsvm-ironic-lib-* - two of them are gating, so they must be
fixed. they are affected by that very virtualbmc+pyghmi incompatibility.
The fix would be to
1) introduce virtualbmc to requirements:master and cherry-pick that (with
version changes) all the way down to mitaka. patch to master is already
here [0]
2) make a patch to ironic's devstack plugin master to install virtualbmc
minding u-c (3 chars fix :) ) and cherry-pick that all the way down to
mitaka as well (test patch to mitaka is here [1], will have to be replaced
with proper cherry-pick from master)
This is already kind of tested with this layered depends-on patch [2] -
those ipmitool jobs are passing

- ironic-lib-coverage-* - in ironic-lib:mitaka tox jobs also do not use
u-c, but it seems it is not the reason of failures here, as the current
"tox -recover" does not pass neither with u-c from mitaka, nor without
them. I have a fix, which is again single character :) (wondering how it
was working before :) ), and could be probably introduced together with a
cherry-pick for u-c handling. But for it to merge, all the other jobs must
be fixed beforehand.

[0] https://review.openstack.org/#/c/440348/
[1] https://review.openstack.org/#/c/440559/
[2] https://review.openstack.org/#/c/440562/


Re: [openstack-dev] [ironic] state of the stable/mitaka branches

2017-03-02 Thread Dmitry Tantsur

On 03/01/2017 08:19 PM, Jay Faulkner wrote:



On Mar 1, 2017, at 11:15 AM, Pavlo Shchelokovskyy 
 wrote:

Greetings ironicers,

I'd like to discuss the state of the gates in ironic and other related projects 
for stable/mitaka branch.

Today while making some test patches to old branches I discovered the following 
problems:

python-ironicclient/stable/mitaka
All unit-test-like jobs are broken due to not handling upper constraints. 
Because of it a newer than actually supported python-openstackclient is 
installed, which already lacks some modules python-ironicclient tries to import 
(these were moved to osc-lib).
I've proposed a patch that copies current way of dealing with upper constraints 
in tox envs [0], gates are passing.

ironic/stable/mitaka
While not actually being gated on, using virtualbmc+ipmitool drivers is broken. 
The reason is again related to upper constraints as what happens is old enough 
version of pyghmi (from mitaka upper constraints) is installed with most recent 
virtualbmc (not in upper constraints), and those versions are incompatible.
This highlights a question whether we should propose virtualbmc to upper 
constraints too to avoid such problems in the future.
Meanwhile a quick fix would be to hard-code the supported virtualbmc version in 
the ironic's devstack plugin for mitaka release.
Although not strictly supported for Mitaka release, I'd like that functionality 
to be working on stable/mitaka gates to test for upcoming removal of *_ssh 
drivers.

I did not test other projects yet.



I can attest jobs are broken for stable/mitaka on ironic-lib as well — our jobs 
build docs unconditionally, and ironic-lib had no docs in Mitaka.


Oh, fun.

Well, the docs job is easy to exclude. But we seem to have virtualbmc-based jobs 
there. As I already wrote in another message, I'm not even sure they're supposed 
to work..




-
Jay Faulkner
OSIC


With all the above, the question is should we really fix the gates for the 
mitaka branch now? According to OpenStack release page [1] the Mitaka release 
will reach end-of-life on April 10, 2017.

[0] https://review.openstack.org/#/c/439742/
[1] https://releases.openstack.org/#release-series

Cheers,
Dr. Pavlo Shchelokovskyy
Senior Software Engineer
Mirantis Inc
www.mirantis.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] state of the stable/mitaka branches

2017-03-02 Thread Dmitry Tantsur

On 03/01/2017 08:15 PM, Pavlo Shchelokovskyy wrote:

Greetings ironicers,

I'd like to discuss the state of the gates in ironic and other related projects
for stable/mitaka branch.


Hi!

Thanks for raising this. I need to apologize, I haven't been doing great job as 
a stable liaison recently. I'll try to fix stuff in the coming days.




Today while making some test patches to old branches I discovered the following
problems:

python-ironicclient/stable/mitaka
All unit-test-like jobs are broken due to not handling upper constraints.
Because of it a newer than actually supported python-openstackclient is
installed, which already lacks some modules python-ironicclient tries to import
(these were moved to osc-lib).
I've proposed a patch that copies current way of dealing with upper constraints
in tox envs [0], gates are passing.


Thanks, I've just approved this patch.



ironic/stable/mitaka
While not actually being gated on, using virtualbmc+ipmitool drivers is broken.


This was not a supported combination back then, so I'm not sure it's good time 
to start right now.



The reason is again related to upper constraints as what happens is old enough
version of pyghmi (from mitaka upper constraints) is installed with most recent
virtualbmc (not in upper constraints), and those versions are incompatible.
This highlights a question whether we should propose virtualbmc to upper
constraints too to avoid such problems in the future.
Meanwhile a quick fix would be to hard-code the supported virtualbmc version in
the ironic's devstack plugin for mitaka release.
Although not strictly supported for Mitaka release, I'd like that functionality
to be working on stable/mitaka gates to test for upcoming removal of *_ssh 
drivers.


There were important changes of pyghmi that made virtualbmc possible at all. I 
don't think any versions can work with old pyghmi.


I'm not sure why removing of *_ssh drivers from master should necessary break 
stable/mitaka, where these drivers are present. Could you elaborate?




I did not test other projects yet.

With all the above, the question is should we really fix the gates for the
mitaka branch now? According to OpenStack release page [1] the Mitaka release
will reach end-of-life on April 10, 2017.


I'd prefer we fix them, I'll look into the problems raised in this thread.



[0] https://review.openstack.org/#/c/439742/
[1] https://releases.openstack.org/#release-series

Cheers,
Dr. Pavlo Shchelokovskyy
Senior Software Engineer
Mirantis Inc
www.mirantis.com 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] state of the stable/mitaka branches

2017-03-01 Thread Tony Breeds
On Wed, Mar 01, 2017 at 09:29:14PM +, Jeremy Stanley wrote:
> On 2017-03-01 13:24:09 -0800 (-0800), Ihar Hrachyshka wrote:
> [...]
> > Other projects spent some time upfront and adopted constraints
> > quite a while ago. I am surprised that there are still stable
> > branches that don't do that.
> [...]
> 
> Yep, I had to backport it for some oslo.middleware stable branches
> recently so we could get a security fix through. There are likely
> some still lurking out there we just haven't spotted because they
> receive new changes on those branches infrequently (or never).

So I did a quick grep and this is what I get:

BRANCH   ALLMERGED  OPEN  HELP  SKIP
origin/master252   23115 2 4
origin/stable/ocata  202   184 014 4
origin/stable/newton 193   104 085 4
origin/stable/mitaka 16871 093 4

This is based on repos that have opted into being managed by the thr
requirements team.

This shows that of the 252 repos that have an origin/master branch 231 have
merged constraints support, 15 have open reviews to do so 2 need some form of
help becuase they shoudl support constratints but they're difficult and 4
shoudln't support constraints.

The 4 projects in SKIP are:
openstack/requirements  
SKIP  !
openstack/tacker-horizon
SKIP  !
openstack/tempest   
SKIP  !
openstack/tempest-lib   
SKIP  !

and there's certainly scope to re-evaluate that.  I suspect that
openstack/tacker-horizon should really be moved to the help section but this
was a quick "see how we're traveling" script.

You can see that the number of projects that need "HELP" gets higher as we get
to older releases.  For most of them it shoudl be a simple matter of
cherry-picking the patch on the nearest branch and then just using the correct
branch to thet the file form the requirements repo.

So clearly there's scope for projects teams and the requirements team to do
work here, but right now it isn't on the plan for this cycle.

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] state of the stable/mitaka branches

2017-03-01 Thread Tony Breeds
On Wed, Mar 01, 2017 at 01:24:09PM -0800, Ihar Hrachyshka wrote:
> I am surprised that there are still
> stable branches that don't do that. It's so much easier to maintain
> them with constraints in place!

Taking a tanget.  At the begining of the Ocata cycle we only had 55%
constraints usage on *master*  That's up much closer to 90% these days.

So master and ocata are looking better but there is still plenty of backport
potential to newton (and possibly mitaka).

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] state of the stable/mitaka branches

2017-03-01 Thread Jeremy Stanley
On 2017-03-01 13:24:09 -0800 (-0800), Ihar Hrachyshka wrote:
[...]
> Other projects spent some time upfront and adopted constraints
> quite a while ago. I am surprised that there are still stable
> branches that don't do that.
[...]

Yep, I had to backport it for some oslo.middleware stable branches
recently so we could get a security fix through. There are likely
some still lurking out there we just haven't spotted because they
receive new changes on those branches infrequently (or never).
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] state of the stable/mitaka branches

2017-03-01 Thread Ihar Hrachyshka
Agreed. All I am saying is that as long as there was no change in the
policy, projects are expected to keep up.

I see that upper-constraints.txt mentioned in the email several times.
I believe it's the least that the project could do to fix the branch,
and lack of the fix doesn't seem like a good enough reason to drop the
ball in the middle (for the project as a whole, not for any specific
contributor). Other projects spent some time upfront and adopted
constraints quite a while ago. I am surprised that there are still
stable branches that don't do that. It's so much easier to maintain
them with constraints in place!

Ihar

On Wed, Mar 1, 2017 at 12:34 PM, Jeremy Stanley  wrote:
> On 2017-03-01 11:36:47 -0800 (-0800), Ihar Hrachyshka wrote:
>> On Wed, Mar 1, 2017 at 11:15 AM, Pavlo Shchelokovskyy
>>  wrote:
>> > With all the above, the question is should we really fix the gates for the
>> > mitaka branch now? According to OpenStack release page [1] the Mitaka
>> > release will reach end-of-life on April 10, 2017.
>>
>> Yes we should. It's part of the contract with consumers that rely on
>> follows:stable-policy tag owned by Ironic and other projects.
>
> It's a two-way street though. As a community we agreed to extend
> stable support timelines on the promise that the people consuming
> those would step up to keep them testable. If key projects are
> having trouble testing stable/mitaka now and the people relying on
> that aren't helping fix the situation, then it's time to again
> reevaluate our earlier choices for support duration.
> --
> Jeremy Stanley
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] state of the stable/mitaka branches

2017-03-01 Thread Jeremy Stanley
On 2017-03-01 11:36:47 -0800 (-0800), Ihar Hrachyshka wrote:
> On Wed, Mar 1, 2017 at 11:15 AM, Pavlo Shchelokovskyy
>  wrote:
> > With all the above, the question is should we really fix the gates for the
> > mitaka branch now? According to OpenStack release page [1] the Mitaka
> > release will reach end-of-life on April 10, 2017.
> 
> Yes we should. It's part of the contract with consumers that rely on
> follows:stable-policy tag owned by Ironic and other projects.

It's a two-way street though. As a community we agreed to extend
stable support timelines on the promise that the people consuming
those would step up to keep them testable. If key projects are
having trouble testing stable/mitaka now and the people relying on
that aren't helping fix the situation, then it's time to again
reevaluate our earlier choices for support duration.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] state of the stable/mitaka branches

2017-03-01 Thread Ihar Hrachyshka
On Wed, Mar 1, 2017 at 11:15 AM, Pavlo Shchelokovskyy
 wrote:
> With all the above, the question is should we really fix the gates for the
> mitaka branch now? According to OpenStack release page [1] the Mitaka
> release will reach end-of-life on April 10, 2017.

Yes we should. It's part of the contract with consumers that rely on
follows:stable-policy tag owned by Ironic and other projects.

Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] state of the stable/mitaka branches

2017-03-01 Thread Jay Faulkner

> On Mar 1, 2017, at 11:15 AM, Pavlo Shchelokovskyy 
>  wrote:
> 
> Greetings ironicers,
> 
> I'd like to discuss the state of the gates in ironic and other related 
> projects for stable/mitaka branch.
> 
> Today while making some test patches to old branches I discovered the 
> following problems:
> 
> python-ironicclient/stable/mitaka
> All unit-test-like jobs are broken due to not handling upper constraints. 
> Because of it a newer than actually supported python-openstackclient is 
> installed, which already lacks some modules python-ironicclient tries to 
> import (these were moved to osc-lib).
> I've proposed a patch that copies current way of dealing with upper 
> constraints in tox envs [0], gates are passing.
> 
> ironic/stable/mitaka
> While not actually being gated on, using virtualbmc+ipmitool drivers is 
> broken. The reason is again related to upper constraints as what happens is 
> old enough version of pyghmi (from mitaka upper constraints) is installed 
> with most recent virtualbmc (not in upper constraints), and those versions 
> are incompatible.
> This highlights a question whether we should propose virtualbmc to upper 
> constraints too to avoid such problems in the future.
> Meanwhile a quick fix would be to hard-code the supported virtualbmc version 
> in the ironic's devstack plugin for mitaka release.
> Although not strictly supported for Mitaka release, I'd like that 
> functionality to be working on stable/mitaka gates to test for upcoming 
> removal of *_ssh drivers.
> 
> I did not test other projects yet.
> 

I can attest jobs are broken for stable/mitaka on ironic-lib as well — our jobs 
build docs unconditionally, and ironic-lib had no docs in Mitaka.

-
Jay Faulkner
OSIC

> With all the above, the question is should we really fix the gates for the 
> mitaka branch now? According to OpenStack release page [1] the Mitaka release 
> will reach end-of-life on April 10, 2017.
> 
> [0] https://review.openstack.org/#/c/439742/
> [1] https://releases.openstack.org/#release-series
> 
> Cheers,
> Dr. Pavlo Shchelokovskyy
> Senior Software Engineer
> Mirantis Inc
> www.mirantis.com
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev