Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-10 Thread Matthew Treinish 
On Tue, Feb 10, 2015 at 11:50:28AM -0500, David Kranz wrote:
> On 02/10/2015 10:35 AM, Matthew Treinish wrote:
> >On Tue, Feb 10, 2015 at 11:19:20AM +0100, Thierry Carrez wrote:
> >>Joe, Matt & Matthew:
> >>
> >>I hear your frustration with broken stable branches. With my
> >>vulnerability management team member hat, responsible for landing
> >>patches there with a strict deadline, I can certainly relate with the
> >>frustration of having to dive in to unbork the branch in the first
> >>place, rather than concentrate on the work you initially planned on doing.
> >>
> >>That said, wearing my stable team member hat, I think it's a bit unfair
> >>to say that things are worse than they were and call for dramatic
> >>action. The stable branch team put a structure in place to try to
> >>continuously fix the stable branches rather than reactively fix it when
> >>we need it to work. Those champions have been quite active[1] unbreaking
> >>it in the past months. I'd argue that the branch is broken much less
> >>often than it used to. That doesn't mean it's never broken, though, or
> >>that those people are magicians.
> >I don't at all for 2 reasons. The first being in every discussion we had at 2
> >summits I raised the increased maint. burden for a longer support window and
> >was told that people were going to stand up so it wouldn't be an issue. I 
> >have
> >yet to see that happen. I have not seen anything to date that would convince
> >me that we are at all ready to be maintaining 3 stable branches at once.
> >
> >The second is while I've seen that etherpad, I still view their still being a
> >huge disconnect here about what actually maintaining the branches requires. 
> >The
> >issue which I'm raising is about issues related to the gating infrastructure 
> >and
> >how to ensure that things stay working. There is a non-linear overhead 
> >involved
> >with making sure any gating job stays working. (on stable or master) People 
> >need
> >to take ownership of jobs to make sure they keep working.
> >
> >>One issue in the current situation is that the two groups (you and the
> >>stable maintainers) seem to work in parallel rather than collaborate.
> >>It's quite telling that the two groups maintained separate etherpads to
> >>keep track of the fixes that needed landing.
> >I don't actually view it as that. Just looking at the etherpad it has a very
> >small subset of the actual types of issues we're raising here.
> >
> >For example, there was a week in late Nov. when 2 consecutive oslo project
> >releases broke the stable gates. After we unwound all of this and landed the
> >fixes in the branches the next step was to changes to make sure we didn't 
> >allow
> >breakages in the same way:
> >
> >http://lists.openstack.org/pipermail/openstack-dev/2014-November/051206.html
> >
> >This was also happened at the same time as a new testtools stack release 
> >which
> >broke every branch (including master). Another example is all of the 
> >setuptools
> >stack churn from the famed Christmas releases. That was another critical
> >infrastructure piece that fell apart and was mostly handled by the infra 
> >team.
> >All of these things are getting fixed because they have to be, to make sure
> >development on master can continue not because those with a vested interest 
> >in
> >the stable branches working for 15 months are working on them.
> >
> >The other aspect here are development efforts to make things more stable in 
> >this
> >space. Things like the effort to pin the requirements on stable branches 
> >which
> >Joe is spearheading. These are critical to the long term success of the 
> >stable
> >branches yet no one has stepped up to help with it.
> >
> >I view this as a disconnect between what people think maintaining a stable
> >branch means and what it actually entails. Sure, the backporting of fixes to
> >intermittent failures is part of it. But, the most effort is spent on making
> >sure the gating machinery stays well oiled and doesn't breakdown.
> >
> >>[1] https://etherpad.openstack.org/p/stable-tracker
> >>
> >>Matthew Treinish wrote:
> >>>So I think it's time we called the icehouse branch and marked it EOL. We
> >>>originally conditioned the longer support window on extra people stepping
> >>>forward to keep things working. I believe this latest issue is just the 
> >>>latest
> >>>indication that this hasn't happened. Issue 1 listed above is being caused 
> >>>by
> >>>the icehouse branch during upgrades. The fact that a stable release was 
> >>>pushed
> >>>at the same time things were wedged on the juno branch is just the latest
> >>>evidence to me that things aren't being maintained as they should be. 
> >>>Looking at
> >>>the #openstack-qa irc log from today or the etherpad about trying to sort 
> >>>this
> >>>issue should be an indication that no one has stepped up to help with the
> >>>maintenance and it shows given the poor state of the branch.
> >>I disagree with the assessment. People have stepped up. I

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-10 Thread David Kranz 

On 02/10/2015 12:20 PM, Jeremy Stanley wrote:

On 2015-02-10 11:50:28 -0500 (-0500), David Kranz wrote:
[...]

I would rather give up branchless tempest than the ability for
real distributors/deployers/operators to collaborate on stable
branches.

[...]

Keep in mind that branchless tempest came about in part due to
downstream use cases as well, not merely as a means to simplify our
testing implementation. Specifically, the interoperability (defcore,
refstack) push was for a testing framework and testset which could
work against multiple deployed environments regardless of what
release(s) they're running and without having to decide among
multiple versions of a tool to do so (especially since they might be
mixing components from multiple OpenStack integrated releases at any
given point in time).
Yes, but that goes out the window in the real world because tempest is 
not really branchless when we periodically
throw out older releases, as we must.  And the earlier we toss out 
things like icehouse, the less branchless it is from the 
interoperability perspective.
Also, tempest is really based on api versions of services, not 
integrated releases, so I'm not sure where mixing components comes into 
play.


In any event, this is a tradeoff and since refstack or whomever has to 
deal with releases that are no longer supported upstream anyway,
they could just do whatever the solution is from the get-go. That said, 
I feel like the current situation is caused by a perfect storm of 
branchless tempest, unpinned versions, and running multiple releases on 
the same machine so there could be other ways to untangle things. I just 
think it is a bad idea to throw the concept of stable branches overboard 
just because the folks who care about it can't deal with the current 
complexity. Once we simplify it, some way or another, I am sure more 
folks will step up or those who have already can get more done.


 -David


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-10 Thread Thierry Carrez 
James E. Blair wrote:
> Thierry Carrez  writes:
> 
>> I also disagree with the proposed solution. We announced a support
>> timeframe for Icehouse, our downstream users made plans around it, so we
>> should stick to it as much as we can.
> 
> To be fair, if we did that, we did not communicate accurately the
> sentiment of the room at the Kilo summit.  From:
> 
>   https://etherpad.openstack.org/p/kilo-summit-ops-stable-branch
> 
>   Stable branches starting with stable/icehouse are intended to be
>   supported for 15 months (previously it was 9 months), but it depends
>   on the community dedicating resources to maintain stable/*
> 
> There was definitely skepticism in that room.  I would characterize it
> as something like "some people wanted 15 months" and other people said
> "that is unlikely to happen based on our track record".  I think the
> consensus was akin to "okay, we'll try it and see what happens but no
> promises".

I think you are confusing two etherpads there. That was the Ops summit
session etherpad (I was in that room, and tried to encourage people to
step up there by adding that note to the etherpad).

I suspect the room you want to express the "sentiment" or "skepticism"
from was the Design Summit one, which had *this* etherpad:

https://etherpad.openstack.org/p/kilo-relmgt-stable-branches

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-10 Thread Jeremy Stanley 
On 2015-02-10 11:50:28 -0500 (-0500), David Kranz wrote:
[...]
> I would rather give up branchless tempest than the ability for
> real distributors/deployers/operators to collaborate on stable
> branches.
[...]

Keep in mind that branchless tempest came about in part due to
downstream use cases as well, not merely as a means to simplify our
testing implementation. Specifically, the interoperability (defcore,
refstack) push was for a testing framework and testset which could
work against multiple deployed environments regardless of what
release(s) they're running and without having to decide among
multiple versions of a tool to do so (especially since they might be
mixing components from multiple OpenStack integrated releases at any
given point in time).
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-10 Thread David Kranz 

On 02/10/2015 10:35 AM, Matthew Treinish wrote:

On Tue, Feb 10, 2015 at 11:19:20AM +0100, Thierry Carrez wrote:

Joe, Matt & Matthew:

I hear your frustration with broken stable branches. With my
vulnerability management team member hat, responsible for landing
patches there with a strict deadline, I can certainly relate with the
frustration of having to dive in to unbork the branch in the first
place, rather than concentrate on the work you initially planned on doing.

That said, wearing my stable team member hat, I think it's a bit unfair
to say that things are worse than they were and call for dramatic
action. The stable branch team put a structure in place to try to
continuously fix the stable branches rather than reactively fix it when
we need it to work. Those champions have been quite active[1] unbreaking
it in the past months. I'd argue that the branch is broken much less
often than it used to. That doesn't mean it's never broken, though, or
that those people are magicians.

I don't at all for 2 reasons. The first being in every discussion we had at 2
summits I raised the increased maint. burden for a longer support window and
was told that people were going to stand up so it wouldn't be an issue. I have
yet to see that happen. I have not seen anything to date that would convince
me that we are at all ready to be maintaining 3 stable branches at once.

The second is while I've seen that etherpad, I still view their still being a
huge disconnect here about what actually maintaining the branches requires. The
issue which I'm raising is about issues related to the gating infrastructure and
how to ensure that things stay working. There is a non-linear overhead involved
with making sure any gating job stays working. (on stable or master) People need
to take ownership of jobs to make sure they keep working.


One issue in the current situation is that the two groups (you and the
stable maintainers) seem to work in parallel rather than collaborate.
It's quite telling that the two groups maintained separate etherpads to
keep track of the fixes that needed landing.

I don't actually view it as that. Just looking at the etherpad it has a very
small subset of the actual types of issues we're raising here.

For example, there was a week in late Nov. when 2 consecutive oslo project
releases broke the stable gates. After we unwound all of this and landed the
fixes in the branches the next step was to changes to make sure we didn't allow
breakages in the same way:

http://lists.openstack.org/pipermail/openstack-dev/2014-November/051206.html

This was also happened at the same time as a new testtools stack release which
broke every branch (including master). Another example is all of the setuptools
stack churn from the famed Christmas releases. That was another critical
infrastructure piece that fell apart and was mostly handled by the infra team.
All of these things are getting fixed because they have to be, to make sure
development on master can continue not because those with a vested interest in
the stable branches working for 15 months are working on them.

The other aspect here are development efforts to make things more stable in this
space. Things like the effort to pin the requirements on stable branches which
Joe is spearheading. These are critical to the long term success of the stable
branches yet no one has stepped up to help with it.

I view this as a disconnect between what people think maintaining a stable
branch means and what it actually entails. Sure, the backporting of fixes to
intermittent failures is part of it. But, the most effort is spent on making
sure the gating machinery stays well oiled and doesn't breakdown.


[1] https://etherpad.openstack.org/p/stable-tracker

Matthew Treinish wrote:

So I think it's time we called the icehouse branch and marked it EOL. We
originally conditioned the longer support window on extra people stepping
forward to keep things working. I believe this latest issue is just the latest
indication that this hasn't happened. Issue 1 listed above is being caused by
the icehouse branch during upgrades. The fact that a stable release was pushed
at the same time things were wedged on the juno branch is just the latest
evidence to me that things aren't being maintained as they should be. Looking at
the #openstack-qa irc log from today or the etherpad about trying to sort this
issue should be an indication that no one has stepped up to help with the
maintenance and it shows given the poor state of the branch.

I disagree with the assessment. People have stepped up. I think the
stable branches are less often broken than they were, and stable branch
champions (as their tracking etherpad shows) have made a difference.
There just has been more issues as usual recently and they probably
couldn't keep track. It's not a fun job to babysit stable branches,
belittling the stable branch champions results is not the best way to
encourage them to continue in this position. I agre

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-10 Thread James E. Blair 
Thierry Carrez  writes:

> I also disagree with the proposed solution. We announced a support
> timeframe for Icehouse, our downstream users made plans around it, so we
> should stick to it as much as we can.

To be fair, if we did that, we did not communicate accurately the
sentiment of the room at the Kilo summit.  From:

  https://etherpad.openstack.org/p/kilo-summit-ops-stable-branch

  Stable branches starting with stable/icehouse are intended to be
  supported for 15 months (previously it was 9 months), but it depends
  on the community dedicating resources to maintain stable/*

There was definitely skepticism in that room.  I would characterize it
as something like "some people wanted 15 months" and other people said
"that is unlikely to happen based on our track record".  I think the
consensus was akin to "okay, we'll try it and see what happens but no
promises".

-Jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-10 Thread Matthew Treinish 
On Tue, Feb 10, 2015 at 11:19:20AM +0100, Thierry Carrez wrote:
> Joe, Matt & Matthew:
> 
> I hear your frustration with broken stable branches. With my
> vulnerability management team member hat, responsible for landing
> patches there with a strict deadline, I can certainly relate with the
> frustration of having to dive in to unbork the branch in the first
> place, rather than concentrate on the work you initially planned on doing.
> 
> That said, wearing my stable team member hat, I think it's a bit unfair
> to say that things are worse than they were and call for dramatic
> action. The stable branch team put a structure in place to try to
> continuously fix the stable branches rather than reactively fix it when
> we need it to work. Those champions have been quite active[1] unbreaking
> it in the past months. I'd argue that the branch is broken much less
> often than it used to. That doesn't mean it's never broken, though, or
> that those people are magicians.

I don't at all for 2 reasons. The first being in every discussion we had at 2
summits I raised the increased maint. burden for a longer support window and
was told that people were going to stand up so it wouldn't be an issue. I have
yet to see that happen. I have not seen anything to date that would convince
me that we are at all ready to be maintaining 3 stable branches at once.

The second is while I've seen that etherpad, I still view their still being a
huge disconnect here about what actually maintaining the branches requires. The
issue which I'm raising is about issues related to the gating infrastructure and
how to ensure that things stay working. There is a non-linear overhead involved
with making sure any gating job stays working. (on stable or master) People need
to take ownership of jobs to make sure they keep working.

> 
> One issue in the current situation is that the two groups (you and the
> stable maintainers) seem to work in parallel rather than collaborate.
> It's quite telling that the two groups maintained separate etherpads to
> keep track of the fixes that needed landing.

I don't actually view it as that. Just looking at the etherpad it has a very
small subset of the actual types of issues we're raising here. 

For example, there was a week in late Nov. when 2 consecutive oslo project
releases broke the stable gates. After we unwound all of this and landed the
fixes in the branches the next step was to changes to make sure we didn't allow
breakages in the same way:

http://lists.openstack.org/pipermail/openstack-dev/2014-November/051206.html

This was also happened at the same time as a new testtools stack release which
broke every branch (including master). Another example is all of the setuptools
stack churn from the famed Christmas releases. That was another critical
infrastructure piece that fell apart and was mostly handled by the infra team.
All of these things are getting fixed because they have to be, to make sure
development on master can continue not because those with a vested interest in
the stable branches working for 15 months are working on them.

The other aspect here are development efforts to make things more stable in this
space. Things like the effort to pin the requirements on stable branches which
Joe is spearheading. These are critical to the long term success of the stable
branches yet no one has stepped up to help with it.

I view this as a disconnect between what people think maintaining a stable
branch means and what it actually entails. Sure, the backporting of fixes to
intermittent failures is part of it. But, the most effort is spent on making
sure the gating machinery stays well oiled and doesn't breakdown.

> 
> [1] https://etherpad.openstack.org/p/stable-tracker
> 
> Matthew Treinish wrote:
> > So I think it's time we called the icehouse branch and marked it EOL. We
> > originally conditioned the longer support window on extra people stepping
> > forward to keep things working. I believe this latest issue is just the 
> > latest
> > indication that this hasn't happened. Issue 1 listed above is being caused 
> > by
> > the icehouse branch during upgrades. The fact that a stable release was 
> > pushed
> > at the same time things were wedged on the juno branch is just the latest
> > evidence to me that things aren't being maintained as they should be. 
> > Looking at
> > the #openstack-qa irc log from today or the etherpad about trying to sort 
> > this
> > issue should be an indication that no one has stepped up to help with the
> > maintenance and it shows given the poor state of the branch.
> 
> I disagree with the assessment. People have stepped up. I think the
> stable branches are less often broken than they were, and stable branch
> champions (as their tracking etherpad shows) have made a difference.
> There just has been more issues as usual recently and they probably
> couldn't keep track. It's not a fun job to babysit stable branches,
> belittling the stable branch champions resu

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-10 Thread Doug Hellmann 


On Tue, Feb 10, 2015, at 05:19 AM, Thierry Carrez wrote:
> Joe, Matt & Matthew:
> 
> I hear your frustration with broken stable branches. With my
> vulnerability management team member hat, responsible for landing
> patches there with a strict deadline, I can certainly relate with the
> frustration of having to dive in to unbork the branch in the first
> place, rather than concentrate on the work you initially planned on
> doing.
> 
> That said, wearing my stable team member hat, I think it's a bit unfair
> to say that things are worse than they were and call for dramatic
> action. The stable branch team put a structure in place to try to
> continuously fix the stable branches rather than reactively fix it when
> we need it to work. Those champions have been quite active[1] unbreaking
> it in the past months. I'd argue that the branch is broken much less
> often than it used to. That doesn't mean it's never broken, though, or
> that those people are magicians.
> 
> One issue in the current situation is that the two groups (you and the
> stable maintainers) seem to work in parallel rather than collaborate.
> It's quite telling that the two groups maintained separate etherpads to
> keep track of the fixes that needed landing.

I agree we should figure out a way to communicate better between the
various teams involved in fixing the issues. Consolidating some of the
communication channels we use should help. For example, if we settle on
a single channel to use on IRC then we can log that channel so anyone
can review the history of a given issue, either to catch up on what
happened overnight or to understand how an issue was resolved long after
the fact. At one point we created #openstack-gate for this, but I don't
see many people there now and it's not being logged. Maybe we should
just use #openstack-dev, since that isn't as active for other purposes
now?

> 
> [1] https://etherpad.openstack.org/p/stable-tracker
> 
> Matthew Treinish wrote:
> > So I think it's time we called the icehouse branch and marked it EOL. We
> > originally conditioned the longer support window on extra people stepping
> > forward to keep things working. I believe this latest issue is just the 
> > latest
> > indication that this hasn't happened. Issue 1 listed above is being caused 
> > by
> > the icehouse branch during upgrades. The fact that a stable release was 
> > pushed
> > at the same time things were wedged on the juno branch is just the latest
> > evidence to me that things aren't being maintained as they should be. 
> > Looking at
> > the #openstack-qa irc log from today or the etherpad about trying to sort 
> > this
> > issue should be an indication that no one has stepped up to help with the
> > maintenance and it shows given the poor state of the branch.
> 
> I disagree with the assessment. People have stepped up. I think the
> stable branches are less often broken than they were, and stable branch
> champions (as their tracking etherpad shows) have made a difference.

They seem to be pretty consistently broken, at least lately, but the
causes have also been consistent. This cycle we changed the way we
manage libraries (dropping alpha release versions) and test tools
(branchless tempest) without fully understanding the effects those
changes would have throughout the complex testing systems we have in
place. As the changes have rippled through the system, we've found more
and more unintended consequences, some of which have required making
other big changes to the way test jobs are defined and to how we manage
requirements. I'm not convinced we would have identified all of those
issues even if we had spent more time reasoning through the changes
before making them, given the complexity. So, although it has been a
frustrating period, in retrospect I think there were probably a few
things we could have done differently early on but it was largely
necessary to do it this way to uncover the issues we couldn't predict.

I think we're approaching a good state, too. The series of patches Joe,
Matt, and Sean came up with yesterday should unwedge icehouse. Then we
can resume the work to cap the requirements lists, and that will avoid
breaking backwards compatibility test jobs with new releases of
libraries. We've also stopped testing master branches of libraries
against the stable branches where we won't be running the code, so
development on the master branches of those libraries should already be
unblocked (although we're not releasing anything until this is sorted
out, which is another sort of block).

> There just has been more issues as usual recently and they probably
> couldn't keep track. It's not a fun job to babysit stable branches,
> belittling the stable branch champions results is not the best way to
> encourage them to continue in this position. I agree that they could
> work more with the QA team when they get overwhelmed, and raise more red
> flags when they just can't keep up.
> 
> I also disagree with the propos

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-10 Thread Ihar Hrachyshka 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

my name was called out here, so I think I need to present my thoughts
on the matter. :) And sorry for writing too many words below.

===

As a disclaimer, I was not the one to support long support term for
stable branches. On the previous summit, I spoke up on sessions about
stable branches to avoid raising the bar even more, and even for
lowering it. One of my arguments against long stable branch support
term is that with 6 month release cycle in mind, we end up with 3, 4,
5, ... stable branches to support in parallel, and it's just too much
to take for anyone. Unfortunately, the session decided to keep 15
months support cycle for Juno.

I really believe we should support one stable release for the most of
our time, leaving ~1 month overlap right after the latest major
release (probably till the first minor .1 update).

That being said, I don't support the way it's proposed here to just
drop a stable branch due to some intermittent issues with it, instead
of fixing it, and the process around it. We made a public statement on
support term, and it would be irresponsible to break it.

===

I also disagree that stable-maint team does not step in to fixing the
issues. There are people who are actively tracking the branches and
doing appropriate fixes here and there. I'm sorry to hear that some of
you still need to step in from time to time to fix stable branches,
but I want to assure you you're not alone in suffering through it.

There were changes in how the team works recently:
- - we actually care about periodic jobs that report issues in project
trees on daily basis;
- - we have a common etherpad to track current stable branch state;
- - we have champions assigned to each of stable branches;
- - we have a generally better fix cycle for issues that arise in stable
branches.

We still have issues to solve, for example the way we handle stable
branch freezes seems to be not optional. We should also be more
proactive to cap library versions right after new release (I'll try to
follow up on that after the current issues are solved). I think we
should also reduce support cycle, especially since there seems to be
little interest from d/s consumers to work collaboratively on those
branches.

===

Part of the problem with stable branches is that we still run them
against bleeding edge. It's true for both external libraries (recent
eventlet, boto issues) and our own projects (clients, oslo libraries,
tempest). This should not be the case, and I'm glad there are people
who are working on pinning pypi libraries on stable branches. Oslo
project now also understand that they should maintain stable branches
for all their libraries (it was not generally the case half a year ago).

I know that sometimes we try to attempt branchlesness, for it's easier
to live without backports, and with due care, it usually works. But it
still fails sometimes, because in reality our backwards compatibility
claims are more of 'best effort' kind, and people do mistakes.

So I hope that once the current issues are solved, we move forward
with pinning all the version for pypi libraries.

I'm also open to help with the effort, though I need to admit that I'm
not involved in current 'qa' program, and so I have no deep
understanding on how it all actually works.

===

I see several problems shown by the gate issues. First, we obviously
lack on communication between 'qa' and 'stable-maint' people. F.e. QA
people were not aware of stable-tracker etherpad maintained by
stable-maint team; while I personally was not aware of who is working
on unwedging the gate, or even whether the issue was critical or even
existed (the first report I got from periodic-checks that showed the
issues comes from today at the night, my local time).

I've updated stable-branch wiki page with contacts of stable branch
champions, so that next time you know whom to reach.

I'd also like to say that you can always reach me for any stable
branch related issues no matter whether it's Icehouse or Juno. The
quicker we get a ping from affected parties, the quicker we start
looking into it.

I'll add #openstack-qa channel into my auto-connect list, I hope it
will help to raise awareness about gate issues on my side, and about
stable process on yours. You're also invited to #openstack-stable were
people show up (sometimes).

===

I would also like to note that stable-maint team lacks a lot of ACLs
to actually resolve any of those issues. Specifically,

- - we don't have +2 for devstack, so we can't merge several patches
that are ready to be pushed to unwedge the branch, specifically:
- - https://review.openstack.org/#/c/154252/
- - https://review.openstack.org/#/c/154217/;

- - we don't have +2 for tempest since it's branch-less, and we are not
cores there;

- - we don't have +2 at requirements/master that sometimes requires
ninja merges in case a gate failure is introduced by a new external
library release.

What we currently have

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-10 Thread Thierry Carrez 
Joe, Matt & Matthew:

I hear your frustration with broken stable branches. With my
vulnerability management team member hat, responsible for landing
patches there with a strict deadline, I can certainly relate with the
frustration of having to dive in to unbork the branch in the first
place, rather than concentrate on the work you initially planned on doing.

That said, wearing my stable team member hat, I think it's a bit unfair
to say that things are worse than they were and call for dramatic
action. The stable branch team put a structure in place to try to
continuously fix the stable branches rather than reactively fix it when
we need it to work. Those champions have been quite active[1] unbreaking
it in the past months. I'd argue that the branch is broken much less
often than it used to. That doesn't mean it's never broken, though, or
that those people are magicians.

One issue in the current situation is that the two groups (you and the
stable maintainers) seem to work in parallel rather than collaborate.
It's quite telling that the two groups maintained separate etherpads to
keep track of the fixes that needed landing.

[1] https://etherpad.openstack.org/p/stable-tracker

Matthew Treinish wrote:
> So I think it's time we called the icehouse branch and marked it EOL. We
> originally conditioned the longer support window on extra people stepping
> forward to keep things working. I believe this latest issue is just the latest
> indication that this hasn't happened. Issue 1 listed above is being caused by
> the icehouse branch during upgrades. The fact that a stable release was pushed
> at the same time things were wedged on the juno branch is just the latest
> evidence to me that things aren't being maintained as they should be. Looking 
> at
> the #openstack-qa irc log from today or the etherpad about trying to sort this
> issue should be an indication that no one has stepped up to help with the
> maintenance and it shows given the poor state of the branch.

I disagree with the assessment. People have stepped up. I think the
stable branches are less often broken than they were, and stable branch
champions (as their tracking etherpad shows) have made a difference.
There just has been more issues as usual recently and they probably
couldn't keep track. It's not a fun job to babysit stable branches,
belittling the stable branch champions results is not the best way to
encourage them to continue in this position. I agree that they could
work more with the QA team when they get overwhelmed, and raise more red
flags when they just can't keep up.

I also disagree with the proposed solution. We announced a support
timeframe for Icehouse, our downstream users made plans around it, so we
should stick to it as much as we can. If we dropped stable branch
support every time a patch can't be landed there, there would just not
be any stable branch.

Joe Gordon wrote:
> Where is it documented that Adam is the Juno branch champion and Ihar is
> Icehouse's? I didn't see it anywhere in the wiki.

It was announced here:
http://lists.openstack.org/pipermail/openstack-dev/2014-November/050390.html

I agree it should be documented on the wiki.

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-09 Thread Joe Gordon 
On Mon, Feb 9, 2015 at 3:10 PM, Alan Pevec  wrote:

> >> >> Tracking etherpad:
> >> >> https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015
>
> BTW there is a tracking etherpad updated by
> https://wiki.openstack.org/wiki/StableBranch#Stable_branch_champions
> https://etherpad.openstack.org/p/stable-tracker
> linked in https://wiki.openstack.org/wiki/StableBranch#Gate_Status and
> announced on this list
> http://lists.openstack.org/pipermail/openstack-dev/2015-January/05.html
>
> From crossed items in "Recently closed" section you can see that
> branch champions have been busy.
>
>
There are two main audiences for stable branches:

* Downstream consumers.
* Upstream developers working on master who need a working stable branch.

I cannot comment on how well the first group is being supported. But as a
member of the second group, I am constantly frustrated by how frequently
broken stable branches ruin my day.


> > You are missing the fact that a bunch of us (Matt Treinish, myself and
> > others) are frustrated by the fact that we end up fixing stable branches
> > whenever they break because we touch tempest, grenade and other projects
> > that require working stable branches. But we do not want to be working on
> > stable branches ourselves.  I begrudgingly stepped up to work on pinning
> all
> > requirements on stable branches, to reduce the number of times stable
> > branches break and ruin my day. But my plan to cap dependencies has been
> > delayed several times by stable branches breaking again and again, along
> > with unwinding undesired behaviors in our testing harnesses.
> >
> > Most recently, stable/juno grenade broke on February 4th (due to the
> release
> > of tempest-lib 0.2.0). This caused bug
>
> So that's a change in tooling, not stable branch itself. Idea when 15
> months for Icehouse was discussed was that branchless Tempest would
> make it easier, but now it turns out that's making both tooling and
> stable branch unhappy.
>

I don't think its reasonable to assume maintaining the stable/branch
excludes actively supporting and improving are testing harness and related
tooling. Our tooling is constantly changing and supporting stable branches
means working on our tooling to make sure its functioning as expected for
stable branches.

Also cutting a new release of a library is not a 'change in tooling'.


> > What I expect to happen when issues like this arise is interested parties
> > work together to fix things and be proactive and make stable testing more
> > robust. Instead we currently have people who have no desire to work on
> > stable branches maintaining them.
>
> At least parts of stable team have been pro-active (see above
> etherpad) but I guess we have a communication issue here: has
> anyonetried to contact stable branch champions (juno=Adam,
> icehouse=Ihar) and what exactly do you expect stable team to do?
> AFAICT these are all changes in tooling where stable-maint is not core
> (devstack, tempest)...
>


Where is it documented that Adam is the Juno branch champion and Ihar is
Icehouse's? I didn't see it anywhere in the wiki.

If something breaks in stable/juno and grenade on master seizes up, what
should we do? When issues are blocking development we should not have to
wait for any one person to respond -- single points of failure are bad. So
I don't think 'has anyone tried to contact us' is the right question to
ask. A better question to ask is 'have stable branches recently prevented
development'

So who should I contact to help me freeze all stable/* dependencies? Or
better yet, someone to drive the effort instead.


> BTW Icehouse 2014.1.4 was planned[*] for Feb 19 with freeze starting
> on Feb 12, I'll delay it for now until we sort the current situation
> out.
>
>
> Cheers,
> Alan
>
>
> [*]
> https://wiki.openstack.org/wiki/StableBranchRelease#Planned_stable.2Ficehouse_releases
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-09 Thread Matthew Treinish 
On Mon, Feb 09, 2015 at 03:10:28PM -0600, Matt Riedemann wrote:
> 
> 
> On 2/9/2015 2:56 PM, Matthew Treinish wrote:
> >On Mon, Feb 09, 2015 at 01:24:34PM -0600, Matt Riedemann wrote:
> >>
> >>
> >>On 2/9/2015 12:23 PM, Joe Gordon wrote:
> >>>
> >>>On Feb 9, 2015 10:04 AM, "Matt Riedemann"  >>>> wrote:
> 
> There are at least two blocking bugs:
> 
> 1. https://bugs.launchpad.net/grenade/+bug/1419913
> 
> Sounds like jogo is working a javelin fix for this. I'm not aware of
> >>>a patch to review though.
> >>>
> >>>We need to stop trying to install tempest in the same env as stable/* code.
> >>>
> >>>I should be able to revise/respond to comments shortly.
> >>>
> >>>https://review.openstack.org/#/c/153080/
> >>>
> >>>https://review.openstack.org/#/c/153702/
> >>>
> >>>This is also blocking my effort to pin stable dependencies (Dean's
> >>>devstack changes are needed before we can pin stable dependencies as well).
> >>>
> 
> 2. https://bugs.launchpad.net/ceilometer/+bug/1419919
> 
> I'm not sure yet what's going on with this one.
> 
> >>
> >>Tracking etherpad:
> >>
> >>https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015
> >
> >
> >So I think it's time we called the icehouse branch and marked it EOL. We
> >originally conditioned the longer support window on extra people stepping
> >forward to keep things working. I believe this latest issue is just the 
> >latest
> >indication that this hasn't happened. Issue 1 listed above is being caused by
> >the icehouse branch during upgrades. The fact that a stable release was 
> >pushed
> >at the same time things were wedged on the juno branch is just the latest
> >evidence to me that things aren't being maintained as they should be. 
> >Looking at
> >the #openstack-qa irc log from today or the etherpad about trying to sort 
> >this
> >issue should be an indication that no one has stepped up to help with the
> >maintenance and it shows given the poor state of the branch.
> >
> >If I'm not mistaken with our original support window lengths Icehouse would 
> >be
> >EOL'd around now. So it's time we stopped pretending we'll be maintaining 
> >this
> >branch for several more months and just go through the normal EOL procedure.
> >
> >-Matt Treinish
> >
> >
> >

> 
> Until we've figured out what's going on and how to unwind it (maybe it's
> just jogo's changes that are blocked right now due to wedged
> stable/icehouse), I'd -1 this.  I think branchless tempest is definitely
> running into some issues here with requirements being uncapped and that's
> throwing a huge wrench into things and EOL'ing stable/icehouse at this point
> kind of gives an easy out w/o first have the full picture of what's busted
> and why we can't work out of it.
> 

I disagree, I feel we've got a pretty good handle on what the cause of the
latest issues here are, the etherpad outlines things pretty well. Additionally,
part of the set of patches from jogo are attempting to address the tempest
install issue by isolating the tempest tox venv from the rest of the python deps
so the divergent reqs aren't an issue in the future. However, we can't land any
of that without unwinding this whole mess.

The other aspect here is this is not just the latest issue. The issues around
the maintenance of icehouse branch has been a continual issue. I know I've had
to personally go and sort out tons of issues on the icehouse branch (mostly
related to new library releases) the past few months because it ends up blocking
tempest development.

-Matt Treinish


pgpEOxVdTVFoq.pgp
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-09 Thread Alan Pevec 
>> >> Tracking etherpad:
>> >> https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015

BTW there is a tracking etherpad updated by
https://wiki.openstack.org/wiki/StableBranch#Stable_branch_champions
https://etherpad.openstack.org/p/stable-tracker
linked in https://wiki.openstack.org/wiki/StableBranch#Gate_Status and
announced on this list
http://lists.openstack.org/pipermail/openstack-dev/2015-January/05.html

>From crossed items in "Recently closed" section you can see that
branch champions have been busy.

> You are missing the fact that a bunch of us (Matt Treinish, myself and
> others) are frustrated by the fact that we end up fixing stable branches
> whenever they break because we touch tempest, grenade and other projects
> that require working stable branches. But we do not want to be working on
> stable branches ourselves.  I begrudgingly stepped up to work on pinning all
> requirements on stable branches, to reduce the number of times stable
> branches break and ruin my day. But my plan to cap dependencies has been
> delayed several times by stable branches breaking again and again, along
> with unwinding undesired behaviors in our testing harnesses.
>
> Most recently, stable/juno grenade broke on February 4th (due to the release
> of tempest-lib 0.2.0). This caused bug

So that's a change in tooling, not stable branch itself. Idea when 15
months for Icehouse was discussed was that branchless Tempest would
make it easier, but now it turns out that's making both tooling and
stable branch unhappy.

> What I expect to happen when issues like this arise is interested parties
> work together to fix things and be proactive and make stable testing more
> robust. Instead we currently have people who have no desire to work on
> stable branches maintaining them.

At least parts of stable team have been pro-active (see above
etherpad) but I guess we have a communication issue here: has
anyonetried to contact stable branch champions (juno=Adam,
icehouse=Ihar) and what exactly do you expect stable team to do?
AFAICT these are all changes in tooling where stable-maint is not core
(devstack, tempest)...

BTW Icehouse 2014.1.4 was planned[*] for Feb 19 with freeze starting
on Feb 12, I'll delay it for now until we sort the current situation
out.


Cheers,
Alan


[*] 
https://wiki.openstack.org/wiki/StableBranchRelease#Planned_stable.2Ficehouse_releases

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-09 Thread Joe Gordon 
On Mon, Feb 9, 2015 at 2:20 PM, Joe Gordon  wrote:

>
>
> On Mon, Feb 9, 2015 at 1:02 PM, John Griffith 
> wrote:
>
>> On Mon, Feb 9, 2015 at 1:56 PM, Matthew Treinish 
>> wrote:
>> > On Mon, Feb 09, 2015 at 01:24:34PM -0600, Matt Riedemann wrote:
>> >>
>> >>
>> >> On 2/9/2015 12:23 PM, Joe Gordon wrote:
>> >> >
>> >> >On Feb 9, 2015 10:04 AM, "Matt Riedemann" > >> >> wrote:
>> >> > >
>> >> > > There are at least two blocking bugs:
>> >> > >
>> >> > > 1. https://bugs.launchpad.net/grenade/+bug/1419913
>> >> > >
>> >> > > Sounds like jogo is working a javelin fix for this. I'm not aware
>> of
>> >> >a patch to review though.
>> >> >
>> >> >We need to stop trying to install tempest in the same env as stable/*
>> code.
>> >> >
>> >> >I should be able to revise/respond to comments shortly.
>> >> >
>> >> >https://review.openstack.org/#/c/153080/
>> >> >
>> >> >https://review.openstack.org/#/c/153702/
>> >> >
>> >> >This is also blocking my effort to pin stable dependencies (Dean's
>> >> >devstack changes are needed before we can pin stable dependencies as
>> well).
>> >> >
>> >> > >
>> >> > > 2. https://bugs.launchpad.net/ceilometer/+bug/1419919
>> >> > >
>> >> > > I'm not sure yet what's going on with this one.
>> >> > >
>> >>
>> >> Tracking etherpad:
>> >>
>> >> https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015
>> >
>> >
>> > So I think it's time we called the icehouse branch and marked it EOL. We
>> > originally conditioned the longer support window on extra people
>> stepping
>> > forward to keep things working. I believe this latest issue is just the
>> latest
>> > indication that this hasn't happened. Issue 1 listed above is being
>> caused by
>> > the icehouse branch during upgrades. The fact that a stable release was
>> pushed
>> > at the same time things were wedged on the juno branch is just the
>> latest
>> > evidence to me that things aren't being maintained as they should be.
>> Looking at
>> > the #openstack-qa irc log from today or the etherpad about trying to
>> sort this
>> > issue should be an indication that no one has stepped up to help with
>> the
>> > maintenance and it shows given the poor state of the branch.
>> >
>> > If I'm not mistaken with our original support window lengths Icehouse
>> would be
>> > EOL'd around now. So it's time we stopped pretending we'll be
>> maintaining this
>> > branch for several more months and just go through the normal EOL
>> procedure.
>> >
>>
>
>> Was this serious?  I mean, we just say; 'sorry, yes we said support
>> until X; but now it's hard so we're going to drop it'.
>>
>> Tell me I'm missing something here?
>>
>
> You are missing the fact that a bunch of us (Matt Treinish, myself and
> others) are frustrated by the fact that we end up fixing stable branches
> whenever they break because we touch tempest, grenade and other projects
> that require working stable branches. But we do not want to be working on
> stable branches ourselves.  I begrudgingly stepped up to work on pinning
> all requirements on stable branches, to reduce the number of times stable
> branches break and ruin my day. But my plan to cap dependencies has been
> delayed several times by stable branches breaking again and again, along
> with unwinding undesired behaviors in our testing harnesses.
>

Note: At least 3 of us just spent most of the day working on this instead
of working on developing on other things.


>
> Most recently, stable/juno grenade broke on February 4th (due to the
> release of tempest-lib 0.2.0). This caused bug
> https://bugs.launchpad.net/grenade/+bug/1419913
> 
>  (" pkg_resources.ContextualVersionConflict: (oslo.config 1.4.0
> (/usr/local/lib/python2.7/dist-packages),
> Requirement.parse('oslo.config>=1.6.0'), set(['tempest-lib']))". This
> specific bug is caused because we install master tempest (due to branchless
> tempest) on stable/icehouse and sync in stable/icehouse global requirements
> which not surprisingly has a conflict with tempest's requirements.  So the
> solution here is stop installing tempest and requiring it  to work with
> stable/icehouse, stable/juno and master's version of global-requirements.
> But that doesn't work because master tempest has an uncapped version of
> boto but nova stable/icehouse only works with the capped version of
> Icehouse. So we get this https://review.openstack.org/#/c/154217/1/. So
> now we are exploring dropping the EC2 tests on stable/icehouse. If that
> works, we still need to land roughly 4 more patches to unwedge this
> stable/juno grenade and prevent this type of issue from happening in the
> future.
>
> Lets say we EOL Icehouse, we stop running grenade on stable/juno patches.
> Meaning this bug goes away all together and stable/juno is unwedged and I
> can move forward with pinning all stable/juno requirements.
>
> What I expect to happen when issues like this arise is interested parties
> work together to fix

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-09 Thread Joe Gordon 
On Mon, Feb 9, 2015 at 1:02 PM, John Griffith 
wrote:

> On Mon, Feb 9, 2015 at 1:56 PM, Matthew Treinish 
> wrote:
> > On Mon, Feb 09, 2015 at 01:24:34PM -0600, Matt Riedemann wrote:
> >>
> >>
> >> On 2/9/2015 12:23 PM, Joe Gordon wrote:
> >> >
> >> >On Feb 9, 2015 10:04 AM, "Matt Riedemann"  >> >> wrote:
> >> > >
> >> > > There are at least two blocking bugs:
> >> > >
> >> > > 1. https://bugs.launchpad.net/grenade/+bug/1419913
> >> > >
> >> > > Sounds like jogo is working a javelin fix for this. I'm not aware of
> >> >a patch to review though.
> >> >
> >> >We need to stop trying to install tempest in the same env as stable/*
> code.
> >> >
> >> >I should be able to revise/respond to comments shortly.
> >> >
> >> >https://review.openstack.org/#/c/153080/
> >> >
> >> >https://review.openstack.org/#/c/153702/
> >> >
> >> >This is also blocking my effort to pin stable dependencies (Dean's
> >> >devstack changes are needed before we can pin stable dependencies as
> well).
> >> >
> >> > >
> >> > > 2. https://bugs.launchpad.net/ceilometer/+bug/1419919
> >> > >
> >> > > I'm not sure yet what's going on with this one.
> >> > >
> >>
> >> Tracking etherpad:
> >>
> >> https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015
> >
> >
> > So I think it's time we called the icehouse branch and marked it EOL. We
> > originally conditioned the longer support window on extra people stepping
> > forward to keep things working. I believe this latest issue is just the
> latest
> > indication that this hasn't happened. Issue 1 listed above is being
> caused by
> > the icehouse branch during upgrades. The fact that a stable release was
> pushed
> > at the same time things were wedged on the juno branch is just the latest
> > evidence to me that things aren't being maintained as they should be.
> Looking at
> > the #openstack-qa irc log from today or the etherpad about trying to
> sort this
> > issue should be an indication that no one has stepped up to help with the
> > maintenance and it shows given the poor state of the branch.
> >
> > If I'm not mistaken with our original support window lengths Icehouse
> would be
> > EOL'd around now. So it's time we stopped pretending we'll be
> maintaining this
> > branch for several more months and just go through the normal EOL
> procedure.
> >
>

> Was this serious?  I mean, we just say; 'sorry, yes we said support
> until X; but now it's hard so we're going to drop it'.
>
> Tell me I'm missing something here?
>

You are missing the fact that a bunch of us (Matt Treinish, myself and
others) are frustrated by the fact that we end up fixing stable branches
whenever they break because we touch tempest, grenade and other projects
that require working stable branches. But we do not want to be working on
stable branches ourselves.  I begrudgingly stepped up to work on pinning
all requirements on stable branches, to reduce the number of times stable
branches break and ruin my day. But my plan to cap dependencies has been
delayed several times by stable branches breaking again and again, along
with unwinding undesired behaviors in our testing harnesses.

Most recently, stable/juno grenade broke on February 4th (due to the
release of tempest-lib 0.2.0). This caused bug
https://bugs.launchpad.net/grenade/+bug/1419913

 (" pkg_resources.ContextualVersionConflict: (oslo.config 1.4.0
(/usr/local/lib/python2.7/dist-packages),
Requirement.parse('oslo.config>=1.6.0'), set(['tempest-lib']))". This
specific bug is caused because we install master tempest (due to branchless
tempest) on stable/icehouse and sync in stable/icehouse global requirements
which not surprisingly has a conflict with tempest's requirements.  So the
solution here is stop installing tempest and requiring it  to work with
stable/icehouse, stable/juno and master's version of global-requirements.
But that doesn't work because master tempest has an uncapped version of
boto but nova stable/icehouse only works with the capped version of
Icehouse. So we get this https://review.openstack.org/#/c/154217/1/. So now
we are exploring dropping the EC2 tests on stable/icehouse. If that works,
we still need to land roughly 4 more patches to unwedge this stable/juno
grenade and prevent this type of issue from happening in the future.

Lets say we EOL Icehouse, we stop running grenade on stable/juno patches.
Meaning this bug goes away all together and stable/juno is unwedged and I
can move forward with pinning all stable/juno requirements.

What I expect to happen when issues like this arise is interested parties
work together to fix things and be proactive and make stable testing more
robust. Instead we currently have people who have no desire to work on
stable branches maintaining them. Pinning all direct stable/* requirements
isn't enough to make sure stable/* doesn't break. There are transitive
dependencies that can change (I have a plan on how to pin those t

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-09 Thread Matt Riedemann 



On 2/9/2015 2:56 PM, Matthew Treinish wrote:

On Mon, Feb 09, 2015 at 01:24:34PM -0600, Matt Riedemann wrote:



On 2/9/2015 12:23 PM, Joe Gordon wrote:


On Feb 9, 2015 10:04 AM, "Matt Riedemann" mailto:mrie...@linux.vnet.ibm.com>> wrote:


There are at least two blocking bugs:

1. https://bugs.launchpad.net/grenade/+bug/1419913

Sounds like jogo is working a javelin fix for this. I'm not aware of

a patch to review though.

We need to stop trying to install tempest in the same env as stable/* code.

I should be able to revise/respond to comments shortly.

https://review.openstack.org/#/c/153080/

https://review.openstack.org/#/c/153702/

This is also blocking my effort to pin stable dependencies (Dean's
devstack changes are needed before we can pin stable dependencies as well).



2. https://bugs.launchpad.net/ceilometer/+bug/1419919

I'm not sure yet what's going on with this one.



Tracking etherpad:

https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015



So I think it's time we called the icehouse branch and marked it EOL. We
originally conditioned the longer support window on extra people stepping
forward to keep things working. I believe this latest issue is just the latest
indication that this hasn't happened. Issue 1 listed above is being caused by
the icehouse branch during upgrades. The fact that a stable release was pushed
at the same time things were wedged on the juno branch is just the latest
evidence to me that things aren't being maintained as they should be. Looking at
the #openstack-qa irc log from today or the etherpad about trying to sort this
issue should be an indication that no one has stepped up to help with the
maintenance and it shows given the poor state of the branch.

If I'm not mistaken with our original support window lengths Icehouse would be
EOL'd around now. So it's time we stopped pretending we'll be maintaining this
branch for several more months and just go through the normal EOL procedure.

-Matt Treinish



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



Until we've figured out what's going on and how to unwind it (maybe it's 
just jogo's changes that are blocked right now due to wedged 
stable/icehouse), I'd -1 this.  I think branchless tempest is definitely 
running into some issues here with requirements being uncapped and 
that's throwing a huge wrench into things and EOL'ing stable/icehouse at 
this point kind of gives an easy out w/o first have the full picture of 
what's busted and why we can't work out of it.


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-09 Thread John Griffith 
On Mon, Feb 9, 2015 at 1:56 PM, Matthew Treinish  wrote:
> On Mon, Feb 09, 2015 at 01:24:34PM -0600, Matt Riedemann wrote:
>>
>>
>> On 2/9/2015 12:23 PM, Joe Gordon wrote:
>> >
>> >On Feb 9, 2015 10:04 AM, "Matt Riedemann" > >> wrote:
>> > >
>> > > There are at least two blocking bugs:
>> > >
>> > > 1. https://bugs.launchpad.net/grenade/+bug/1419913
>> > >
>> > > Sounds like jogo is working a javelin fix for this. I'm not aware of
>> >a patch to review though.
>> >
>> >We need to stop trying to install tempest in the same env as stable/* code.
>> >
>> >I should be able to revise/respond to comments shortly.
>> >
>> >https://review.openstack.org/#/c/153080/
>> >
>> >https://review.openstack.org/#/c/153702/
>> >
>> >This is also blocking my effort to pin stable dependencies (Dean's
>> >devstack changes are needed before we can pin stable dependencies as well).
>> >
>> > >
>> > > 2. https://bugs.launchpad.net/ceilometer/+bug/1419919
>> > >
>> > > I'm not sure yet what's going on with this one.
>> > >
>>
>> Tracking etherpad:
>>
>> https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015
>
>
> So I think it's time we called the icehouse branch and marked it EOL. We
> originally conditioned the longer support window on extra people stepping
> forward to keep things working. I believe this latest issue is just the latest
> indication that this hasn't happened. Issue 1 listed above is being caused by
> the icehouse branch during upgrades. The fact that a stable release was pushed
> at the same time things were wedged on the juno branch is just the latest
> evidence to me that things aren't being maintained as they should be. Looking 
> at
> the #openstack-qa irc log from today or the etherpad about trying to sort this
> issue should be an indication that no one has stepped up to help with the
> maintenance and it shows given the poor state of the branch.
>
> If I'm not mistaken with our original support window lengths Icehouse would be
> EOL'd around now. So it's time we stopped pretending we'll be maintaining this
> branch for several more months and just go through the normal EOL procedure.
>

Was this serious?  I mean, we just say; 'sorry, yes we said support
until X; but now it's hard so we're going to drop it'.

Tell me I'm missing something here?

> -Matt Treinish
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-09 Thread Matthew Treinish 
On Mon, Feb 09, 2015 at 01:24:34PM -0600, Matt Riedemann wrote:
> 
> 
> On 2/9/2015 12:23 PM, Joe Gordon wrote:
> >
> >On Feb 9, 2015 10:04 AM, "Matt Riedemann"  >> wrote:
> > >
> > > There are at least two blocking bugs:
> > >
> > > 1. https://bugs.launchpad.net/grenade/+bug/1419913
> > >
> > > Sounds like jogo is working a javelin fix for this. I'm not aware of
> >a patch to review though.
> >
> >We need to stop trying to install tempest in the same env as stable/* code.
> >
> >I should be able to revise/respond to comments shortly.
> >
> >https://review.openstack.org/#/c/153080/
> >
> >https://review.openstack.org/#/c/153702/
> >
> >This is also blocking my effort to pin stable dependencies (Dean's
> >devstack changes are needed before we can pin stable dependencies as well).
> >
> > >
> > > 2. https://bugs.launchpad.net/ceilometer/+bug/1419919
> > >
> > > I'm not sure yet what's going on with this one.
> > >
> 
> Tracking etherpad:
> 
> https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015


So I think it's time we called the icehouse branch and marked it EOL. We
originally conditioned the longer support window on extra people stepping
forward to keep things working. I believe this latest issue is just the latest
indication that this hasn't happened. Issue 1 listed above is being caused by
the icehouse branch during upgrades. The fact that a stable release was pushed
at the same time things were wedged on the juno branch is just the latest
evidence to me that things aren't being maintained as they should be. Looking at
the #openstack-qa irc log from today or the etherpad about trying to sort this
issue should be an indication that no one has stepped up to help with the
maintenance and it shows given the poor state of the branch.

If I'm not mistaken with our original support window lengths Icehouse would be
EOL'd around now. So it's time we stopped pretending we'll be maintaining this
branch for several more months and just go through the normal EOL procedure.

-Matt Treinish


pgpwWu0TsiRvn.pgp
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-09 Thread Matt Riedemann 



On 2/9/2015 12:23 PM, Joe Gordon wrote:


On Feb 9, 2015 10:04 AM, "Matt Riedemann" mailto:mrie...@linux.vnet.ibm.com>> wrote:
 >
 > There are at least two blocking bugs:
 >
 > 1. https://bugs.launchpad.net/grenade/+bug/1419913
 >
 > Sounds like jogo is working a javelin fix for this. I'm not aware of
a patch to review though.

We need to stop trying to install tempest in the same env as stable/* code.

I should be able to revise/respond to comments shortly.

https://review.openstack.org/#/c/153080/

https://review.openstack.org/#/c/153702/

This is also blocking my effort to pin stable dependencies (Dean's
devstack changes are needed before we can pin stable dependencies as well).

 >
 > 2. https://bugs.launchpad.net/ceilometer/+bug/1419919
 >
 > I'm not sure yet what's going on with this one.
 >
 > --
 >
 > Thanks,
 >
 > Matt Riedemann
 >
 >
 >
__
 > OpenStack Development Mailing List (not for usage questions)
 > Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

 > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



Tracking etherpad:

https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015

--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-09 Thread Matt Riedemann 



On 2/9/2015 12:03 PM, Matt Riedemann wrote:

There are at least two blocking bugs:

1. https://bugs.launchpad.net/grenade/+bug/1419913

Sounds like jogo is working a javelin fix for this. I'm not aware of a
patch to review though.

2. https://bugs.launchpad.net/ceilometer/+bug/1419919

I'm not sure yet what's going on with this one.



Looks like the versions haven't been bumped for the integrated release 
projects on stable/juno yet so I did that here:


https://review.openstack.org/#/q/Ib8a29258d99de75b49a9b19aef36bb99bc5fcac0,n,z

--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable] juno is fubar in the gate

2015-02-09 Thread Joe Gordon 
On Feb 9, 2015 10:04 AM, "Matt Riedemann" 
wrote:
>
> There are at least two blocking bugs:
>
> 1. https://bugs.launchpad.net/grenade/+bug/1419913
>
> Sounds like jogo is working a javelin fix for this. I'm not aware of a
patch to review though.

We need to stop trying to install tempest in the same env as stable/* code.

I should be able to revise/respond to comments shortly.

https://review.openstack.org/#/c/153080/

https://review.openstack.org/#/c/153702/

This is also blocking my effort to pin stable dependencies (Dean's devstack
changes are needed before we can pin stable dependencies as well).

>
> 2. https://bugs.launchpad.net/ceilometer/+bug/1419919
>
> I'm not sure yet what's going on with this one.
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [stable] juno is fubar in the gate

2015-02-09 Thread Matt Riedemann 

There are at least two blocking bugs:

1. https://bugs.launchpad.net/grenade/+bug/1419913

Sounds like jogo is working a javelin fix for this. I'm not aware of a 
patch to review though.


2. https://bugs.launchpad.net/ceilometer/+bug/1419919

I'm not sure yet what's going on with this one.

--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev