Re: [openstack-dev] [stable] juno is fubar in the gate
On Tue, Feb 10, 2015, at 05:19 AM, Thierry Carrez wrote: Joe, Matt Matthew: I hear your frustration with broken stable branches. With my vulnerability management team member hat, responsible for landing patches there with a strict deadline, I can certainly relate with the frustration of having to dive in to unbork the branch in the first place, rather than concentrate on the work you initially planned on doing. That said, wearing my stable team member hat, I think it's a bit unfair to say that things are worse than they were and call for dramatic action. The stable branch team put a structure in place to try to continuously fix the stable branches rather than reactively fix it when we need it to work. Those champions have been quite active[1] unbreaking it in the past months. I'd argue that the branch is broken much less often than it used to. That doesn't mean it's never broken, though, or that those people are magicians. One issue in the current situation is that the two groups (you and the stable maintainers) seem to work in parallel rather than collaborate. It's quite telling that the two groups maintained separate etherpads to keep track of the fixes that needed landing. I agree we should figure out a way to communicate better between the various teams involved in fixing the issues. Consolidating some of the communication channels we use should help. For example, if we settle on a single channel to use on IRC then we can log that channel so anyone can review the history of a given issue, either to catch up on what happened overnight or to understand how an issue was resolved long after the fact. At one point we created #openstack-gate for this, but I don't see many people there now and it's not being logged. Maybe we should just use #openstack-dev, since that isn't as active for other purposes now? [1] https://etherpad.openstack.org/p/stable-tracker Matthew Treinish wrote: So I think it's time we called the icehouse branch and marked it EOL. We originally conditioned the longer support window on extra people stepping forward to keep things working. I believe this latest issue is just the latest indication that this hasn't happened. Issue 1 listed above is being caused by the icehouse branch during upgrades. The fact that a stable release was pushed at the same time things were wedged on the juno branch is just the latest evidence to me that things aren't being maintained as they should be. Looking at the #openstack-qa irc log from today or the etherpad about trying to sort this issue should be an indication that no one has stepped up to help with the maintenance and it shows given the poor state of the branch. I disagree with the assessment. People have stepped up. I think the stable branches are less often broken than they were, and stable branch champions (as their tracking etherpad shows) have made a difference. They seem to be pretty consistently broken, at least lately, but the causes have also been consistent. This cycle we changed the way we manage libraries (dropping alpha release versions) and test tools (branchless tempest) without fully understanding the effects those changes would have throughout the complex testing systems we have in place. As the changes have rippled through the system, we've found more and more unintended consequences, some of which have required making other big changes to the way test jobs are defined and to how we manage requirements. I'm not convinced we would have identified all of those issues even if we had spent more time reasoning through the changes before making them, given the complexity. So, although it has been a frustrating period, in retrospect I think there were probably a few things we could have done differently early on but it was largely necessary to do it this way to uncover the issues we couldn't predict. I think we're approaching a good state, too. The series of patches Joe, Matt, and Sean came up with yesterday should unwedge icehouse. Then we can resume the work to cap the requirements lists, and that will avoid breaking backwards compatibility test jobs with new releases of libraries. We've also stopped testing master branches of libraries against the stable branches where we won't be running the code, so development on the master branches of those libraries should already be unblocked (although we're not releasing anything until this is sorted out, which is another sort of block). There just has been more issues as usual recently and they probably couldn't keep track. It's not a fun job to babysit stable branches, belittling the stable branch champions results is not the best way to encourage them to continue in this position. I agree that they could work more with the QA team when they get overwhelmed, and raise more red flags when they just can't keep up. I also disagree with the proposed solution. We announced a support timeframe for Icehouse, our
Re: [openstack-dev] [stable] juno is fubar in the gate
On Tue, Feb 10, 2015 at 11:19:20AM +0100, Thierry Carrez wrote: Joe, Matt Matthew: I hear your frustration with broken stable branches. With my vulnerability management team member hat, responsible for landing patches there with a strict deadline, I can certainly relate with the frustration of having to dive in to unbork the branch in the first place, rather than concentrate on the work you initially planned on doing. That said, wearing my stable team member hat, I think it's a bit unfair to say that things are worse than they were and call for dramatic action. The stable branch team put a structure in place to try to continuously fix the stable branches rather than reactively fix it when we need it to work. Those champions have been quite active[1] unbreaking it in the past months. I'd argue that the branch is broken much less often than it used to. That doesn't mean it's never broken, though, or that those people are magicians. I don't at all for 2 reasons. The first being in every discussion we had at 2 summits I raised the increased maint. burden for a longer support window and was told that people were going to stand up so it wouldn't be an issue. I have yet to see that happen. I have not seen anything to date that would convince me that we are at all ready to be maintaining 3 stable branches at once. The second is while I've seen that etherpad, I still view their still being a huge disconnect here about what actually maintaining the branches requires. The issue which I'm raising is about issues related to the gating infrastructure and how to ensure that things stay working. There is a non-linear overhead involved with making sure any gating job stays working. (on stable or master) People need to take ownership of jobs to make sure they keep working. One issue in the current situation is that the two groups (you and the stable maintainers) seem to work in parallel rather than collaborate. It's quite telling that the two groups maintained separate etherpads to keep track of the fixes that needed landing. I don't actually view it as that. Just looking at the etherpad it has a very small subset of the actual types of issues we're raising here. For example, there was a week in late Nov. when 2 consecutive oslo project releases broke the stable gates. After we unwound all of this and landed the fixes in the branches the next step was to changes to make sure we didn't allow breakages in the same way: http://lists.openstack.org/pipermail/openstack-dev/2014-November/051206.html This was also happened at the same time as a new testtools stack release which broke every branch (including master). Another example is all of the setuptools stack churn from the famed Christmas releases. That was another critical infrastructure piece that fell apart and was mostly handled by the infra team. All of these things are getting fixed because they have to be, to make sure development on master can continue not because those with a vested interest in the stable branches working for 15 months are working on them. The other aspect here are development efforts to make things more stable in this space. Things like the effort to pin the requirements on stable branches which Joe is spearheading. These are critical to the long term success of the stable branches yet no one has stepped up to help with it. I view this as a disconnect between what people think maintaining a stable branch means and what it actually entails. Sure, the backporting of fixes to intermittent failures is part of it. But, the most effort is spent on making sure the gating machinery stays well oiled and doesn't breakdown. [1] https://etherpad.openstack.org/p/stable-tracker Matthew Treinish wrote: So I think it's time we called the icehouse branch and marked it EOL. We originally conditioned the longer support window on extra people stepping forward to keep things working. I believe this latest issue is just the latest indication that this hasn't happened. Issue 1 listed above is being caused by the icehouse branch during upgrades. The fact that a stable release was pushed at the same time things were wedged on the juno branch is just the latest evidence to me that things aren't being maintained as they should be. Looking at the #openstack-qa irc log from today or the etherpad about trying to sort this issue should be an indication that no one has stepped up to help with the maintenance and it shows given the poor state of the branch. I disagree with the assessment. People have stepped up. I think the stable branches are less often broken than they were, and stable branch champions (as their tracking etherpad shows) have made a difference. There just has been more issues as usual recently and they probably couldn't keep track. It's not a fun job to babysit stable branches, belittling the stable branch champions results is not the best way to encourage them to continue in this
Re: [openstack-dev] [stable] juno is fubar in the gate
Thierry Carrez thie...@openstack.org writes: I also disagree with the proposed solution. We announced a support timeframe for Icehouse, our downstream users made plans around it, so we should stick to it as much as we can. To be fair, if we did that, we did not communicate accurately the sentiment of the room at the Kilo summit. From: https://etherpad.openstack.org/p/kilo-summit-ops-stable-branch Stable branches starting with stable/icehouse are intended to be supported for 15 months (previously it was 9 months), but it depends on the community dedicating resources to maintain stable/* There was definitely skepticism in that room. I would characterize it as something like some people wanted 15 months and other people said that is unlikely to happen based on our track record. I think the consensus was akin to okay, we'll try it and see what happens but no promises. -Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable] juno is fubar in the gate
On 02/10/2015 10:35 AM, Matthew Treinish wrote: On Tue, Feb 10, 2015 at 11:19:20AM +0100, Thierry Carrez wrote: Joe, Matt Matthew: I hear your frustration with broken stable branches. With my vulnerability management team member hat, responsible for landing patches there with a strict deadline, I can certainly relate with the frustration of having to dive in to unbork the branch in the first place, rather than concentrate on the work you initially planned on doing. That said, wearing my stable team member hat, I think it's a bit unfair to say that things are worse than they were and call for dramatic action. The stable branch team put a structure in place to try to continuously fix the stable branches rather than reactively fix it when we need it to work. Those champions have been quite active[1] unbreaking it in the past months. I'd argue that the branch is broken much less often than it used to. That doesn't mean it's never broken, though, or that those people are magicians. I don't at all for 2 reasons. The first being in every discussion we had at 2 summits I raised the increased maint. burden for a longer support window and was told that people were going to stand up so it wouldn't be an issue. I have yet to see that happen. I have not seen anything to date that would convince me that we are at all ready to be maintaining 3 stable branches at once. The second is while I've seen that etherpad, I still view their still being a huge disconnect here about what actually maintaining the branches requires. The issue which I'm raising is about issues related to the gating infrastructure and how to ensure that things stay working. There is a non-linear overhead involved with making sure any gating job stays working. (on stable or master) People need to take ownership of jobs to make sure they keep working. One issue in the current situation is that the two groups (you and the stable maintainers) seem to work in parallel rather than collaborate. It's quite telling that the two groups maintained separate etherpads to keep track of the fixes that needed landing. I don't actually view it as that. Just looking at the etherpad it has a very small subset of the actual types of issues we're raising here. For example, there was a week in late Nov. when 2 consecutive oslo project releases broke the stable gates. After we unwound all of this and landed the fixes in the branches the next step was to changes to make sure we didn't allow breakages in the same way: http://lists.openstack.org/pipermail/openstack-dev/2014-November/051206.html This was also happened at the same time as a new testtools stack release which broke every branch (including master). Another example is all of the setuptools stack churn from the famed Christmas releases. That was another critical infrastructure piece that fell apart and was mostly handled by the infra team. All of these things are getting fixed because they have to be, to make sure development on master can continue not because those with a vested interest in the stable branches working for 15 months are working on them. The other aspect here are development efforts to make things more stable in this space. Things like the effort to pin the requirements on stable branches which Joe is spearheading. These are critical to the long term success of the stable branches yet no one has stepped up to help with it. I view this as a disconnect between what people think maintaining a stable branch means and what it actually entails. Sure, the backporting of fixes to intermittent failures is part of it. But, the most effort is spent on making sure the gating machinery stays well oiled and doesn't breakdown. [1] https://etherpad.openstack.org/p/stable-tracker Matthew Treinish wrote: So I think it's time we called the icehouse branch and marked it EOL. We originally conditioned the longer support window on extra people stepping forward to keep things working. I believe this latest issue is just the latest indication that this hasn't happened. Issue 1 listed above is being caused by the icehouse branch during upgrades. The fact that a stable release was pushed at the same time things were wedged on the juno branch is just the latest evidence to me that things aren't being maintained as they should be. Looking at the #openstack-qa irc log from today or the etherpad about trying to sort this issue should be an indication that no one has stepped up to help with the maintenance and it shows given the poor state of the branch. I disagree with the assessment. People have stepped up. I think the stable branches are less often broken than they were, and stable branch champions (as their tracking etherpad shows) have made a difference. There just has been more issues as usual recently and they probably couldn't keep track. It's not a fun job to babysit stable branches, belittling the stable branch champions results is not the best way to encourage them to continue in this position. I
Re: [openstack-dev] [stable] juno is fubar in the gate
On Tue, Feb 10, 2015 at 11:50:28AM -0500, David Kranz wrote: On 02/10/2015 10:35 AM, Matthew Treinish wrote: On Tue, Feb 10, 2015 at 11:19:20AM +0100, Thierry Carrez wrote: Joe, Matt Matthew: I hear your frustration with broken stable branches. With my vulnerability management team member hat, responsible for landing patches there with a strict deadline, I can certainly relate with the frustration of having to dive in to unbork the branch in the first place, rather than concentrate on the work you initially planned on doing. That said, wearing my stable team member hat, I think it's a bit unfair to say that things are worse than they were and call for dramatic action. The stable branch team put a structure in place to try to continuously fix the stable branches rather than reactively fix it when we need it to work. Those champions have been quite active[1] unbreaking it in the past months. I'd argue that the branch is broken much less often than it used to. That doesn't mean it's never broken, though, or that those people are magicians. I don't at all for 2 reasons. The first being in every discussion we had at 2 summits I raised the increased maint. burden for a longer support window and was told that people were going to stand up so it wouldn't be an issue. I have yet to see that happen. I have not seen anything to date that would convince me that we are at all ready to be maintaining 3 stable branches at once. The second is while I've seen that etherpad, I still view their still being a huge disconnect here about what actually maintaining the branches requires. The issue which I'm raising is about issues related to the gating infrastructure and how to ensure that things stay working. There is a non-linear overhead involved with making sure any gating job stays working. (on stable or master) People need to take ownership of jobs to make sure they keep working. One issue in the current situation is that the two groups (you and the stable maintainers) seem to work in parallel rather than collaborate. It's quite telling that the two groups maintained separate etherpads to keep track of the fixes that needed landing. I don't actually view it as that. Just looking at the etherpad it has a very small subset of the actual types of issues we're raising here. For example, there was a week in late Nov. when 2 consecutive oslo project releases broke the stable gates. After we unwound all of this and landed the fixes in the branches the next step was to changes to make sure we didn't allow breakages in the same way: http://lists.openstack.org/pipermail/openstack-dev/2014-November/051206.html This was also happened at the same time as a new testtools stack release which broke every branch (including master). Another example is all of the setuptools stack churn from the famed Christmas releases. That was another critical infrastructure piece that fell apart and was mostly handled by the infra team. All of these things are getting fixed because they have to be, to make sure development on master can continue not because those with a vested interest in the stable branches working for 15 months are working on them. The other aspect here are development efforts to make things more stable in this space. Things like the effort to pin the requirements on stable branches which Joe is spearheading. These are critical to the long term success of the stable branches yet no one has stepped up to help with it. I view this as a disconnect between what people think maintaining a stable branch means and what it actually entails. Sure, the backporting of fixes to intermittent failures is part of it. But, the most effort is spent on making sure the gating machinery stays well oiled and doesn't breakdown. [1] https://etherpad.openstack.org/p/stable-tracker Matthew Treinish wrote: So I think it's time we called the icehouse branch and marked it EOL. We originally conditioned the longer support window on extra people stepping forward to keep things working. I believe this latest issue is just the latest indication that this hasn't happened. Issue 1 listed above is being caused by the icehouse branch during upgrades. The fact that a stable release was pushed at the same time things were wedged on the juno branch is just the latest evidence to me that things aren't being maintained as they should be. Looking at the #openstack-qa irc log from today or the etherpad about trying to sort this issue should be an indication that no one has stepped up to help with the maintenance and it shows given the poor state of the branch. I disagree with the assessment. People have stepped up. I think the stable branches are less often broken than they were, and stable branch champions (as their tracking etherpad shows) have made a difference. There just has been more issues as usual recently and they probably couldn't keep track.
Re: [openstack-dev] [stable] juno is fubar in the gate
On 2015-02-10 11:50:28 -0500 (-0500), David Kranz wrote: [...] I would rather give up branchless tempest than the ability for real distributors/deployers/operators to collaborate on stable branches. [...] Keep in mind that branchless tempest came about in part due to downstream use cases as well, not merely as a means to simplify our testing implementation. Specifically, the interoperability (defcore, refstack) push was for a testing framework and testset which could work against multiple deployed environments regardless of what release(s) they're running and without having to decide among multiple versions of a tool to do so (especially since they might be mixing components from multiple OpenStack integrated releases at any given point in time). -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable] juno is fubar in the gate
On 02/10/2015 12:20 PM, Jeremy Stanley wrote: On 2015-02-10 11:50:28 -0500 (-0500), David Kranz wrote: [...] I would rather give up branchless tempest than the ability for real distributors/deployers/operators to collaborate on stable branches. [...] Keep in mind that branchless tempest came about in part due to downstream use cases as well, not merely as a means to simplify our testing implementation. Specifically, the interoperability (defcore, refstack) push was for a testing framework and testset which could work against multiple deployed environments regardless of what release(s) they're running and without having to decide among multiple versions of a tool to do so (especially since they might be mixing components from multiple OpenStack integrated releases at any given point in time). Yes, but that goes out the window in the real world because tempest is not really branchless when we periodically throw out older releases, as we must. And the earlier we toss out things like icehouse, the less branchless it is from the interoperability perspective. Also, tempest is really based on api versions of services, not integrated releases, so I'm not sure where mixing components comes into play. In any event, this is a tradeoff and since refstack or whomever has to deal with releases that are no longer supported upstream anyway, they could just do whatever the solution is from the get-go. That said, I feel like the current situation is caused by a perfect storm of branchless tempest, unpinned versions, and running multiple releases on the same machine so there could be other ways to untangle things. I just think it is a bad idea to throw the concept of stable branches overboard just because the folks who care about it can't deal with the current complexity. Once we simplify it, some way or another, I am sure more folks will step up or those who have already can get more done. -David __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable] juno is fubar in the gate
James E. Blair wrote: Thierry Carrez thie...@openstack.org writes: I also disagree with the proposed solution. We announced a support timeframe for Icehouse, our downstream users made plans around it, so we should stick to it as much as we can. To be fair, if we did that, we did not communicate accurately the sentiment of the room at the Kilo summit. From: https://etherpad.openstack.org/p/kilo-summit-ops-stable-branch Stable branches starting with stable/icehouse are intended to be supported for 15 months (previously it was 9 months), but it depends on the community dedicating resources to maintain stable/* There was definitely skepticism in that room. I would characterize it as something like some people wanted 15 months and other people said that is unlikely to happen based on our track record. I think the consensus was akin to okay, we'll try it and see what happens but no promises. I think you are confusing two etherpads there. That was the Ops summit session etherpad (I was in that room, and tried to encourage people to step up there by adding that note to the etherpad). I suspect the room you want to express the sentiment or skepticism from was the Design Summit one, which had *this* etherpad: https://etherpad.openstack.org/p/kilo-relmgt-stable-branches -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable] juno is fubar in the gate
Joe, Matt Matthew: I hear your frustration with broken stable branches. With my vulnerability management team member hat, responsible for landing patches there with a strict deadline, I can certainly relate with the frustration of having to dive in to unbork the branch in the first place, rather than concentrate on the work you initially planned on doing. That said, wearing my stable team member hat, I think it's a bit unfair to say that things are worse than they were and call for dramatic action. The stable branch team put a structure in place to try to continuously fix the stable branches rather than reactively fix it when we need it to work. Those champions have been quite active[1] unbreaking it in the past months. I'd argue that the branch is broken much less often than it used to. That doesn't mean it's never broken, though, or that those people are magicians. One issue in the current situation is that the two groups (you and the stable maintainers) seem to work in parallel rather than collaborate. It's quite telling that the two groups maintained separate etherpads to keep track of the fixes that needed landing. [1] https://etherpad.openstack.org/p/stable-tracker Matthew Treinish wrote: So I think it's time we called the icehouse branch and marked it EOL. We originally conditioned the longer support window on extra people stepping forward to keep things working. I believe this latest issue is just the latest indication that this hasn't happened. Issue 1 listed above is being caused by the icehouse branch during upgrades. The fact that a stable release was pushed at the same time things were wedged on the juno branch is just the latest evidence to me that things aren't being maintained as they should be. Looking at the #openstack-qa irc log from today or the etherpad about trying to sort this issue should be an indication that no one has stepped up to help with the maintenance and it shows given the poor state of the branch. I disagree with the assessment. People have stepped up. I think the stable branches are less often broken than they were, and stable branch champions (as their tracking etherpad shows) have made a difference. There just has been more issues as usual recently and they probably couldn't keep track. It's not a fun job to babysit stable branches, belittling the stable branch champions results is not the best way to encourage them to continue in this position. I agree that they could work more with the QA team when they get overwhelmed, and raise more red flags when they just can't keep up. I also disagree with the proposed solution. We announced a support timeframe for Icehouse, our downstream users made plans around it, so we should stick to it as much as we can. If we dropped stable branch support every time a patch can't be landed there, there would just not be any stable branch. Joe Gordon wrote: Where is it documented that Adam is the Juno branch champion and Ihar is Icehouse's? I didn't see it anywhere in the wiki. It was announced here: http://lists.openstack.org/pipermail/openstack-dev/2014-November/050390.html I agree it should be documented on the wiki. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable] juno is fubar in the gate
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, my name was called out here, so I think I need to present my thoughts on the matter. :) And sorry for writing too many words below. === As a disclaimer, I was not the one to support long support term for stable branches. On the previous summit, I spoke up on sessions about stable branches to avoid raising the bar even more, and even for lowering it. One of my arguments against long stable branch support term is that with 6 month release cycle in mind, we end up with 3, 4, 5, ... stable branches to support in parallel, and it's just too much to take for anyone. Unfortunately, the session decided to keep 15 months support cycle for Juno. I really believe we should support one stable release for the most of our time, leaving ~1 month overlap right after the latest major release (probably till the first minor .1 update). That being said, I don't support the way it's proposed here to just drop a stable branch due to some intermittent issues with it, instead of fixing it, and the process around it. We made a public statement on support term, and it would be irresponsible to break it. === I also disagree that stable-maint team does not step in to fixing the issues. There are people who are actively tracking the branches and doing appropriate fixes here and there. I'm sorry to hear that some of you still need to step in from time to time to fix stable branches, but I want to assure you you're not alone in suffering through it. There were changes in how the team works recently: - - we actually care about periodic jobs that report issues in project trees on daily basis; - - we have a common etherpad to track current stable branch state; - - we have champions assigned to each of stable branches; - - we have a generally better fix cycle for issues that arise in stable branches. We still have issues to solve, for example the way we handle stable branch freezes seems to be not optional. We should also be more proactive to cap library versions right after new release (I'll try to follow up on that after the current issues are solved). I think we should also reduce support cycle, especially since there seems to be little interest from d/s consumers to work collaboratively on those branches. === Part of the problem with stable branches is that we still run them against bleeding edge. It's true for both external libraries (recent eventlet, boto issues) and our own projects (clients, oslo libraries, tempest). This should not be the case, and I'm glad there are people who are working on pinning pypi libraries on stable branches. Oslo project now also understand that they should maintain stable branches for all their libraries (it was not generally the case half a year ago). I know that sometimes we try to attempt branchlesness, for it's easier to live without backports, and with due care, it usually works. But it still fails sometimes, because in reality our backwards compatibility claims are more of 'best effort' kind, and people do mistakes. So I hope that once the current issues are solved, we move forward with pinning all the version for pypi libraries. I'm also open to help with the effort, though I need to admit that I'm not involved in current 'qa' program, and so I have no deep understanding on how it all actually works. === I see several problems shown by the gate issues. First, we obviously lack on communication between 'qa' and 'stable-maint' people. F.e. QA people were not aware of stable-tracker etherpad maintained by stable-maint team; while I personally was not aware of who is working on unwedging the gate, or even whether the issue was critical or even existed (the first report I got from periodic-checks that showed the issues comes from today at the night, my local time). I've updated stable-branch wiki page with contacts of stable branch champions, so that next time you know whom to reach. I'd also like to say that you can always reach me for any stable branch related issues no matter whether it's Icehouse or Juno. The quicker we get a ping from affected parties, the quicker we start looking into it. I'll add #openstack-qa channel into my auto-connect list, I hope it will help to raise awareness about gate issues on my side, and about stable process on yours. You're also invited to #openstack-stable were people show up (sometimes). === I would also like to note that stable-maint team lacks a lot of ACLs to actually resolve any of those issues. Specifically, - - we don't have +2 for devstack, so we can't merge several patches that are ready to be pushed to unwedge the branch, specifically: - - https://review.openstack.org/#/c/154252/ - - https://review.openstack.org/#/c/154217/; - - we don't have +2 for tempest since it's branch-less, and we are not cores there; - - we don't have +2 at requirements/master that sometimes requires ninja merges in case a gate failure is introduced by a new external library release. What we currently
Re: [openstack-dev] [stable] juno is fubar in the gate
On 2/9/2015 12:23 PM, Joe Gordon wrote: On Feb 9, 2015 10:04 AM, Matt Riedemann mrie...@linux.vnet.ibm.com mailto:mrie...@linux.vnet.ibm.com wrote: There are at least two blocking bugs: 1. https://bugs.launchpad.net/grenade/+bug/1419913 Sounds like jogo is working a javelin fix for this. I'm not aware of a patch to review though. We need to stop trying to install tempest in the same env as stable/* code. I should be able to revise/respond to comments shortly. https://review.openstack.org/#/c/153080/ https://review.openstack.org/#/c/153702/ This is also blocking my effort to pin stable dependencies (Dean's devstack changes are needed before we can pin stable dependencies as well). 2. https://bugs.launchpad.net/ceilometer/+bug/1419919 I'm not sure yet what's going on with this one. -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Tracking etherpad: https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015 -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [stable] juno is fubar in the gate
There are at least two blocking bugs: 1. https://bugs.launchpad.net/grenade/+bug/1419913 Sounds like jogo is working a javelin fix for this. I'm not aware of a patch to review though. 2. https://bugs.launchpad.net/ceilometer/+bug/1419919 I'm not sure yet what's going on with this one. -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable] juno is fubar in the gate
On Feb 9, 2015 10:04 AM, Matt Riedemann mrie...@linux.vnet.ibm.com wrote: There are at least two blocking bugs: 1. https://bugs.launchpad.net/grenade/+bug/1419913 Sounds like jogo is working a javelin fix for this. I'm not aware of a patch to review though. We need to stop trying to install tempest in the same env as stable/* code. I should be able to revise/respond to comments shortly. https://review.openstack.org/#/c/153080/ https://review.openstack.org/#/c/153702/ This is also blocking my effort to pin stable dependencies (Dean's devstack changes are needed before we can pin stable dependencies as well). 2. https://bugs.launchpad.net/ceilometer/+bug/1419919 I'm not sure yet what's going on with this one. -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable] juno is fubar in the gate
On 2/9/2015 12:03 PM, Matt Riedemann wrote: There are at least two blocking bugs: 1. https://bugs.launchpad.net/grenade/+bug/1419913 Sounds like jogo is working a javelin fix for this. I'm not aware of a patch to review though. 2. https://bugs.launchpad.net/ceilometer/+bug/1419919 I'm not sure yet what's going on with this one. Looks like the versions haven't been bumped for the integrated release projects on stable/juno yet so I did that here: https://review.openstack.org/#/q/Ib8a29258d99de75b49a9b19aef36bb99bc5fcac0,n,z -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable] juno is fubar in the gate
On Mon, Feb 9, 2015 at 3:10 PM, Alan Pevec ape...@gmail.com wrote: Tracking etherpad: https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015 BTW there is a tracking etherpad updated by https://wiki.openstack.org/wiki/StableBranch#Stable_branch_champions https://etherpad.openstack.org/p/stable-tracker linked in https://wiki.openstack.org/wiki/StableBranch#Gate_Status and announced on this list http://lists.openstack.org/pipermail/openstack-dev/2015-January/05.html From crossed items in Recently closed section you can see that branch champions have been busy. There are two main audiences for stable branches: * Downstream consumers. * Upstream developers working on master who need a working stable branch. I cannot comment on how well the first group is being supported. But as a member of the second group, I am constantly frustrated by how frequently broken stable branches ruin my day. You are missing the fact that a bunch of us (Matt Treinish, myself and others) are frustrated by the fact that we end up fixing stable branches whenever they break because we touch tempest, grenade and other projects that require working stable branches. But we do not want to be working on stable branches ourselves. I begrudgingly stepped up to work on pinning all requirements on stable branches, to reduce the number of times stable branches break and ruin my day. But my plan to cap dependencies has been delayed several times by stable branches breaking again and again, along with unwinding undesired behaviors in our testing harnesses. Most recently, stable/juno grenade broke on February 4th (due to the release of tempest-lib 0.2.0). This caused bug So that's a change in tooling, not stable branch itself. Idea when 15 months for Icehouse was discussed was that branchless Tempest would make it easier, but now it turns out that's making both tooling and stable branch unhappy. I don't think its reasonable to assume maintaining the stable/branch excludes actively supporting and improving are testing harness and related tooling. Our tooling is constantly changing and supporting stable branches means working on our tooling to make sure its functioning as expected for stable branches. Also cutting a new release of a library is not a 'change in tooling'. What I expect to happen when issues like this arise is interested parties work together to fix things and be proactive and make stable testing more robust. Instead we currently have people who have no desire to work on stable branches maintaining them. At least parts of stable team have been pro-active (see above etherpad) but I guess we have a communication issue here: has anyonetried to contact stable branch champions (juno=Adam, icehouse=Ihar) and what exactly do you expect stable team to do? AFAICT these are all changes in tooling where stable-maint is not core (devstack, tempest)... Where is it documented that Adam is the Juno branch champion and Ihar is Icehouse's? I didn't see it anywhere in the wiki. If something breaks in stable/juno and grenade on master seizes up, what should we do? When issues are blocking development we should not have to wait for any one person to respond -- single points of failure are bad. So I don't think 'has anyone tried to contact us' is the right question to ask. A better question to ask is 'have stable branches recently prevented development' So who should I contact to help me freeze all stable/* dependencies? Or better yet, someone to drive the effort instead. BTW Icehouse 2014.1.4 was planned[*] for Feb 19 with freeze starting on Feb 12, I'll delay it for now until we sort the current situation out. Cheers, Alan [*] https://wiki.openstack.org/wiki/StableBranchRelease#Planned_stable.2Ficehouse_releases __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable] juno is fubar in the gate
On Mon, Feb 9, 2015 at 1:56 PM, Matthew Treinish mtrein...@kortar.org wrote: On Mon, Feb 09, 2015 at 01:24:34PM -0600, Matt Riedemann wrote: On 2/9/2015 12:23 PM, Joe Gordon wrote: On Feb 9, 2015 10:04 AM, Matt Riedemann mrie...@linux.vnet.ibm.com mailto:mrie...@linux.vnet.ibm.com wrote: There are at least two blocking bugs: 1. https://bugs.launchpad.net/grenade/+bug/1419913 Sounds like jogo is working a javelin fix for this. I'm not aware of a patch to review though. We need to stop trying to install tempest in the same env as stable/* code. I should be able to revise/respond to comments shortly. https://review.openstack.org/#/c/153080/ https://review.openstack.org/#/c/153702/ This is also blocking my effort to pin stable dependencies (Dean's devstack changes are needed before we can pin stable dependencies as well). 2. https://bugs.launchpad.net/ceilometer/+bug/1419919 I'm not sure yet what's going on with this one. Tracking etherpad: https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015 So I think it's time we called the icehouse branch and marked it EOL. We originally conditioned the longer support window on extra people stepping forward to keep things working. I believe this latest issue is just the latest indication that this hasn't happened. Issue 1 listed above is being caused by the icehouse branch during upgrades. The fact that a stable release was pushed at the same time things were wedged on the juno branch is just the latest evidence to me that things aren't being maintained as they should be. Looking at the #openstack-qa irc log from today or the etherpad about trying to sort this issue should be an indication that no one has stepped up to help with the maintenance and it shows given the poor state of the branch. If I'm not mistaken with our original support window lengths Icehouse would be EOL'd around now. So it's time we stopped pretending we'll be maintaining this branch for several more months and just go through the normal EOL procedure. Was this serious? I mean, we just say; 'sorry, yes we said support until X; but now it's hard so we're going to drop it'. Tell me I'm missing something here? -Matt Treinish __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable] juno is fubar in the gate
On Mon, Feb 09, 2015 at 01:24:34PM -0600, Matt Riedemann wrote: On 2/9/2015 12:23 PM, Joe Gordon wrote: On Feb 9, 2015 10:04 AM, Matt Riedemann mrie...@linux.vnet.ibm.com mailto:mrie...@linux.vnet.ibm.com wrote: There are at least two blocking bugs: 1. https://bugs.launchpad.net/grenade/+bug/1419913 Sounds like jogo is working a javelin fix for this. I'm not aware of a patch to review though. We need to stop trying to install tempest in the same env as stable/* code. I should be able to revise/respond to comments shortly. https://review.openstack.org/#/c/153080/ https://review.openstack.org/#/c/153702/ This is also blocking my effort to pin stable dependencies (Dean's devstack changes are needed before we can pin stable dependencies as well). 2. https://bugs.launchpad.net/ceilometer/+bug/1419919 I'm not sure yet what's going on with this one. Tracking etherpad: https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015 So I think it's time we called the icehouse branch and marked it EOL. We originally conditioned the longer support window on extra people stepping forward to keep things working. I believe this latest issue is just the latest indication that this hasn't happened. Issue 1 listed above is being caused by the icehouse branch during upgrades. The fact that a stable release was pushed at the same time things were wedged on the juno branch is just the latest evidence to me that things aren't being maintained as they should be. Looking at the #openstack-qa irc log from today or the etherpad about trying to sort this issue should be an indication that no one has stepped up to help with the maintenance and it shows given the poor state of the branch. If I'm not mistaken with our original support window lengths Icehouse would be EOL'd around now. So it's time we stopped pretending we'll be maintaining this branch for several more months and just go through the normal EOL procedure. -Matt Treinish pgpwWu0TsiRvn.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable] juno is fubar in the gate
On 2/9/2015 2:56 PM, Matthew Treinish wrote: On Mon, Feb 09, 2015 at 01:24:34PM -0600, Matt Riedemann wrote: On 2/9/2015 12:23 PM, Joe Gordon wrote: On Feb 9, 2015 10:04 AM, Matt Riedemann mrie...@linux.vnet.ibm.com mailto:mrie...@linux.vnet.ibm.com wrote: There are at least two blocking bugs: 1. https://bugs.launchpad.net/grenade/+bug/1419913 Sounds like jogo is working a javelin fix for this. I'm not aware of a patch to review though. We need to stop trying to install tempest in the same env as stable/* code. I should be able to revise/respond to comments shortly. https://review.openstack.org/#/c/153080/ https://review.openstack.org/#/c/153702/ This is also blocking my effort to pin stable dependencies (Dean's devstack changes are needed before we can pin stable dependencies as well). 2. https://bugs.launchpad.net/ceilometer/+bug/1419919 I'm not sure yet what's going on with this one. Tracking etherpad: https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015 So I think it's time we called the icehouse branch and marked it EOL. We originally conditioned the longer support window on extra people stepping forward to keep things working. I believe this latest issue is just the latest indication that this hasn't happened. Issue 1 listed above is being caused by the icehouse branch during upgrades. The fact that a stable release was pushed at the same time things were wedged on the juno branch is just the latest evidence to me that things aren't being maintained as they should be. Looking at the #openstack-qa irc log from today or the etherpad about trying to sort this issue should be an indication that no one has stepped up to help with the maintenance and it shows given the poor state of the branch. If I'm not mistaken with our original support window lengths Icehouse would be EOL'd around now. So it's time we stopped pretending we'll be maintaining this branch for several more months and just go through the normal EOL procedure. -Matt Treinish __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Until we've figured out what's going on and how to unwind it (maybe it's just jogo's changes that are blocked right now due to wedged stable/icehouse), I'd -1 this. I think branchless tempest is definitely running into some issues here with requirements being uncapped and that's throwing a huge wrench into things and EOL'ing stable/icehouse at this point kind of gives an easy out w/o first have the full picture of what's busted and why we can't work out of it. -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable] juno is fubar in the gate
On Mon, Feb 9, 2015 at 2:20 PM, Joe Gordon joe.gord...@gmail.com wrote:
On Mon, Feb 9, 2015 at 1:02 PM, John Griffith john.griffi...@gmail.com
wrote:
On Mon, Feb 9, 2015 at 1:56 PM, Matthew Treinish mtrein...@kortar.org
wrote:
On Mon, Feb 09, 2015 at 01:24:34PM -0600, Matt Riedemann wrote:
On 2/9/2015 12:23 PM, Joe Gordon wrote:
On Feb 9, 2015 10:04 AM, Matt Riedemann mrie...@linux.vnet.ibm.com
mailto:mrie...@linux.vnet.ibm.com wrote:
There are at least two blocking bugs:
1. https://bugs.launchpad.net/grenade/+bug/1419913
Sounds like jogo is working a javelin fix for this. I'm not aware
of
a patch to review though.
We need to stop trying to install tempest in the same env as stable/*
code.
I should be able to revise/respond to comments shortly.
https://review.openstack.org/#/c/153080/
https://review.openstack.org/#/c/153702/
This is also blocking my effort to pin stable dependencies (Dean's
devstack changes are needed before we can pin stable dependencies as
well).
2. https://bugs.launchpad.net/ceilometer/+bug/1419919
I'm not sure yet what's going on with this one.
Tracking etherpad:
https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015
So I think it's time we called the icehouse branch and marked it EOL. We
originally conditioned the longer support window on extra people
stepping
forward to keep things working. I believe this latest issue is just the
latest
indication that this hasn't happened. Issue 1 listed above is being
caused by
the icehouse branch during upgrades. The fact that a stable release was
pushed
at the same time things were wedged on the juno branch is just the
latest
evidence to me that things aren't being maintained as they should be.
Looking at
the #openstack-qa irc log from today or the etherpad about trying to
sort this
issue should be an indication that no one has stepped up to help with
the
maintenance and it shows given the poor state of the branch.
If I'm not mistaken with our original support window lengths Icehouse
would be
EOL'd around now. So it's time we stopped pretending we'll be
maintaining this
branch for several more months and just go through the normal EOL
procedure.
Was this serious? I mean, we just say; 'sorry, yes we said support
until X; but now it's hard so we're going to drop it'.
Tell me I'm missing something here?
You are missing the fact that a bunch of us (Matt Treinish, myself and
others) are frustrated by the fact that we end up fixing stable branches
whenever they break because we touch tempest, grenade and other projects
that require working stable branches. But we do not want to be working on
stable branches ourselves. I begrudgingly stepped up to work on pinning
all requirements on stable branches, to reduce the number of times stable
branches break and ruin my day. But my plan to cap dependencies has been
delayed several times by stable branches breaking again and again, along
with unwinding undesired behaviors in our testing harnesses.
Note: At least 3 of us just spent most of the day working on this instead
of working on developing on other things.
Most recently, stable/juno grenade broke on February 4th (due to the
release of tempest-lib 0.2.0). This caused bug
https://bugs.launchpad.net/grenade/+bug/1419913
https://bugs.launchpad.net/grenade/+bug/1419913
( pkg_resources.ContextualVersionConflict: (oslo.config 1.4.0
(/usr/local/lib/python2.7/dist-packages),
Requirement.parse('oslo.config=1.6.0'), set(['tempest-lib'])). This
specific bug is caused because we install master tempest (due to branchless
tempest) on stable/icehouse and sync in stable/icehouse global requirements
which not surprisingly has a conflict with tempest's requirements. So the
solution here is stop installing tempest and requiring it to work with
stable/icehouse, stable/juno and master's version of global-requirements.
But that doesn't work because master tempest has an uncapped version of
boto but nova stable/icehouse only works with the capped version of
Icehouse. So we get this https://review.openstack.org/#/c/154217/1/. So
now we are exploring dropping the EC2 tests on stable/icehouse. If that
works, we still need to land roughly 4 more patches to unwedge this
stable/juno grenade and prevent this type of issue from happening in the
future.
Lets say we EOL Icehouse, we stop running grenade on stable/juno patches.
Meaning this bug goes away all together and stable/juno is unwedged and I
can move forward with pinning all stable/juno requirements.
What I expect to happen when issues like this arise is interested parties
work together to fix things and be proactive and make stable testing more
robust. Instead we currently have people who have no desire to work on
stable branches maintaining them. Pinning all direct stable/* requirements
isn't enough to make sure
Re: [openstack-dev] [stable] juno is fubar in the gate
On Mon, Feb 9, 2015 at 1:02 PM, John Griffith john.griffi...@gmail.com
wrote:
On Mon, Feb 9, 2015 at 1:56 PM, Matthew Treinish mtrein...@kortar.org
wrote:
On Mon, Feb 09, 2015 at 01:24:34PM -0600, Matt Riedemann wrote:
On 2/9/2015 12:23 PM, Joe Gordon wrote:
On Feb 9, 2015 10:04 AM, Matt Riedemann mrie...@linux.vnet.ibm.com
mailto:mrie...@linux.vnet.ibm.com wrote:
There are at least two blocking bugs:
1. https://bugs.launchpad.net/grenade/+bug/1419913
Sounds like jogo is working a javelin fix for this. I'm not aware of
a patch to review though.
We need to stop trying to install tempest in the same env as stable/*
code.
I should be able to revise/respond to comments shortly.
https://review.openstack.org/#/c/153080/
https://review.openstack.org/#/c/153702/
This is also blocking my effort to pin stable dependencies (Dean's
devstack changes are needed before we can pin stable dependencies as
well).
2. https://bugs.launchpad.net/ceilometer/+bug/1419919
I'm not sure yet what's going on with this one.
Tracking etherpad:
https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015
So I think it's time we called the icehouse branch and marked it EOL. We
originally conditioned the longer support window on extra people stepping
forward to keep things working. I believe this latest issue is just the
latest
indication that this hasn't happened. Issue 1 listed above is being
caused by
the icehouse branch during upgrades. The fact that a stable release was
pushed
at the same time things were wedged on the juno branch is just the latest
evidence to me that things aren't being maintained as they should be.
Looking at
the #openstack-qa irc log from today or the etherpad about trying to
sort this
issue should be an indication that no one has stepped up to help with the
maintenance and it shows given the poor state of the branch.
If I'm not mistaken with our original support window lengths Icehouse
would be
EOL'd around now. So it's time we stopped pretending we'll be
maintaining this
branch for several more months and just go through the normal EOL
procedure.
Was this serious? I mean, we just say; 'sorry, yes we said support
until X; but now it's hard so we're going to drop it'.
Tell me I'm missing something here?
You are missing the fact that a bunch of us (Matt Treinish, myself and
others) are frustrated by the fact that we end up fixing stable branches
whenever they break because we touch tempest, grenade and other projects
that require working stable branches. But we do not want to be working on
stable branches ourselves. I begrudgingly stepped up to work on pinning
all requirements on stable branches, to reduce the number of times stable
branches break and ruin my day. But my plan to cap dependencies has been
delayed several times by stable branches breaking again and again, along
with unwinding undesired behaviors in our testing harnesses.
Most recently, stable/juno grenade broke on February 4th (due to the
release of tempest-lib 0.2.0). This caused bug
https://bugs.launchpad.net/grenade/+bug/1419913
https://bugs.launchpad.net/grenade/+bug/1419913
( pkg_resources.ContextualVersionConflict: (oslo.config 1.4.0
(/usr/local/lib/python2.7/dist-packages),
Requirement.parse('oslo.config=1.6.0'), set(['tempest-lib'])). This
specific bug is caused because we install master tempest (due to branchless
tempest) on stable/icehouse and sync in stable/icehouse global requirements
which not surprisingly has a conflict with tempest's requirements. So the
solution here is stop installing tempest and requiring it to work with
stable/icehouse, stable/juno and master's version of global-requirements.
But that doesn't work because master tempest has an uncapped version of
boto but nova stable/icehouse only works with the capped version of
Icehouse. So we get this https://review.openstack.org/#/c/154217/1/. So now
we are exploring dropping the EC2 tests on stable/icehouse. If that works,
we still need to land roughly 4 more patches to unwedge this stable/juno
grenade and prevent this type of issue from happening in the future.
Lets say we EOL Icehouse, we stop running grenade on stable/juno patches.
Meaning this bug goes away all together and stable/juno is unwedged and I
can move forward with pinning all stable/juno requirements.
What I expect to happen when issues like this arise is interested parties
work together to fix things and be proactive and make stable testing more
robust. Instead we currently have people who have no desire to work on
stable branches maintaining them. Pinning all direct stable/* requirements
isn't enough to make sure stable/* doesn't break. There are transitive
dependencies that can change (I have a plan on how to pin those too, but it
will take time and I can use some help), and changing packages etc. can
break things as well. Having a reactive stable
Re: [openstack-dev] [stable] juno is fubar in the gate
Tracking etherpad: https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015 BTW there is a tracking etherpad updated by https://wiki.openstack.org/wiki/StableBranch#Stable_branch_champions https://etherpad.openstack.org/p/stable-tracker linked in https://wiki.openstack.org/wiki/StableBranch#Gate_Status and announced on this list http://lists.openstack.org/pipermail/openstack-dev/2015-January/05.html From crossed items in Recently closed section you can see that branch champions have been busy. You are missing the fact that a bunch of us (Matt Treinish, myself and others) are frustrated by the fact that we end up fixing stable branches whenever they break because we touch tempest, grenade and other projects that require working stable branches. But we do not want to be working on stable branches ourselves. I begrudgingly stepped up to work on pinning all requirements on stable branches, to reduce the number of times stable branches break and ruin my day. But my plan to cap dependencies has been delayed several times by stable branches breaking again and again, along with unwinding undesired behaviors in our testing harnesses. Most recently, stable/juno grenade broke on February 4th (due to the release of tempest-lib 0.2.0). This caused bug So that's a change in tooling, not stable branch itself. Idea when 15 months for Icehouse was discussed was that branchless Tempest would make it easier, but now it turns out that's making both tooling and stable branch unhappy. What I expect to happen when issues like this arise is interested parties work together to fix things and be proactive and make stable testing more robust. Instead we currently have people who have no desire to work on stable branches maintaining them. At least parts of stable team have been pro-active (see above etherpad) but I guess we have a communication issue here: has anyonetried to contact stable branch champions (juno=Adam, icehouse=Ihar) and what exactly do you expect stable team to do? AFAICT these are all changes in tooling where stable-maint is not core (devstack, tempest)... BTW Icehouse 2014.1.4 was planned[*] for Feb 19 with freeze starting on Feb 12, I'll delay it for now until we sort the current situation out. Cheers, Alan [*] https://wiki.openstack.org/wiki/StableBranchRelease#Planned_stable.2Ficehouse_releases __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable] juno is fubar in the gate
On Mon, Feb 09, 2015 at 03:10:28PM -0600, Matt Riedemann wrote: On 2/9/2015 2:56 PM, Matthew Treinish wrote: On Mon, Feb 09, 2015 at 01:24:34PM -0600, Matt Riedemann wrote: On 2/9/2015 12:23 PM, Joe Gordon wrote: On Feb 9, 2015 10:04 AM, Matt Riedemann mrie...@linux.vnet.ibm.com mailto:mrie...@linux.vnet.ibm.com wrote: There are at least two blocking bugs: 1. https://bugs.launchpad.net/grenade/+bug/1419913 Sounds like jogo is working a javelin fix for this. I'm not aware of a patch to review though. We need to stop trying to install tempest in the same env as stable/* code. I should be able to revise/respond to comments shortly. https://review.openstack.org/#/c/153080/ https://review.openstack.org/#/c/153702/ This is also blocking my effort to pin stable dependencies (Dean's devstack changes are needed before we can pin stable dependencies as well). 2. https://bugs.launchpad.net/ceilometer/+bug/1419919 I'm not sure yet what's going on with this one. Tracking etherpad: https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015 So I think it's time we called the icehouse branch and marked it EOL. We originally conditioned the longer support window on extra people stepping forward to keep things working. I believe this latest issue is just the latest indication that this hasn't happened. Issue 1 listed above is being caused by the icehouse branch during upgrades. The fact that a stable release was pushed at the same time things were wedged on the juno branch is just the latest evidence to me that things aren't being maintained as they should be. Looking at the #openstack-qa irc log from today or the etherpad about trying to sort this issue should be an indication that no one has stepped up to help with the maintenance and it shows given the poor state of the branch. If I'm not mistaken with our original support window lengths Icehouse would be EOL'd around now. So it's time we stopped pretending we'll be maintaining this branch for several more months and just go through the normal EOL procedure. -Matt Treinish Until we've figured out what's going on and how to unwind it (maybe it's just jogo's changes that are blocked right now due to wedged stable/icehouse), I'd -1 this. I think branchless tempest is definitely running into some issues here with requirements being uncapped and that's throwing a huge wrench into things and EOL'ing stable/icehouse at this point kind of gives an easy out w/o first have the full picture of what's busted and why we can't work out of it. I disagree, I feel we've got a pretty good handle on what the cause of the latest issues here are, the etherpad outlines things pretty well. Additionally, part of the set of patches from jogo are attempting to address the tempest install issue by isolating the tempest tox venv from the rest of the python deps so the divergent reqs aren't an issue in the future. However, we can't land any of that without unwinding this whole mess. The other aspect here is this is not just the latest issue. The issues around the maintenance of icehouse branch has been a continual issue. I know I've had to personally go and sort out tons of issues on the icehouse branch (mostly related to new library releases) the past few months because it ends up blocking tempest development. -Matt Treinish pgpEOxVdTVFoq.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
