Re: [openstack-dev] [heat]Policy on upgades required config changes
Hi Keith Clint, On Tue, Mar 11, 2014 at 05:05:21AM +, Keith Bray wrote: I want to echo Clint's responses... We do run close to Heat master here at Rackspace, and we'd be happy to set up a non-voting job to notify when a review would break Heat on our cloud if that would be beneficial. Some of the breaks we have seen have been things that simply weren't caught in code review (a human intensive effort), were specific to the way we configure Heat for large-scale cloud use, applicable to the entire Heat project, and not necessarily service provider specific. I appreciate the feedback and I've certainly learned something during this process and will endeavor to provide uniformly backwards compatible changes in future. I certainly agree we can do things better next time :) Hopefully you can appreciate that the auth related features I've been working on have been a large and difficult undertaking, and that once the transitional pain has passed will bring considerable benefits for both users and deployers. One frustration I have is lack of review feedback for most of the instance-users and v3 keystone work (except for a small and dedicated subset of the heat-core team, thanks!). So my feedback to you is if you're running close to master, we really really need your help during the review process, to avoid post-merge stress for everyone :) Re gate CI - it sounds like a great idea, voting and non-voting feedback is hugely valuable in addition to human reviewer feedback, so hopefully we can work towards getting such tests in place. Anyway, apologies again for any inconvenience, hopefully all is working OK now with the fallback patch I provided. Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat]Policy on upgades required config changes
On 03/04/2014 12:39 PM, Steven Hardy wrote: Hi all, As some of you know, I've been working on the instance-users blueprint[1]. This blueprint implementation requires three new items to be added to the heat.conf, or some resources (those which create keystone users) will not work: https://review.openstack.org/#/c/73978/ https://review.openstack.org/#/c/76035/ So on upgrade, the deployer must create a keystone domain and domain-admin user, add the details to heat.conf, as already been done in devstack[2]. The changes requried for this to work have already landed in devstack, but it was discussed to day and Clint suggested this may be unacceptable upgrade behavior - I'm not sure so looking for guidance/comments. My plan was/is: - Make devstack work - Talk to tripleo folks to assist in any transition (what prompted this discussion) - Document the upgrade requirements in the Icehouse release notes so the wider community can upgrade from Havana. - Try to give a heads-up to those maintaining downstream heat deployment tools (e.g stackforge/puppet-heat) that some tweaks will be required for Icehouse. However some have suggested there may be an openstack-wide policy which requires peoples old config files to continue working indefinitely on upgrade between versions - is this right? If so where is it documented? This is basically enforced in code in grenade, the language for this actually got lost in the project requirements discussion in the TC, I'll bring that back in the post graduation requirements discussion we're having again. The issue is - Heat still doesn't materially participate in grenade. Heat is substantially far behind the other integrated projects in it's integration with the upstream testing. Only monday did we finally start gating on a real unit of work for Heat (the heat-slow jobs). If I was letter grading projects right now on upstream testing I'd give Nova an A, Neutron a C (still no full run, no working grenade), and Heat a D. So in short. Heat did the wrong thing. You should be able to use your configs from the last release. This is what all the mature projects in OpenStack do. In the event that you *have* to make a change like that it requires an UpgradeImpact tag in the commit. And those should be limited really aggressively. This is the whole point of the deprecation cycle. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat]Policy on upgades required config changes
On Tue, Mar 11, 2014 at 07:04:32AM -0400, Sean Dague wrote: On 03/04/2014 12:39 PM, Steven Hardy wrote: Hi all, As some of you know, I've been working on the instance-users blueprint[1]. This blueprint implementation requires three new items to be added to the heat.conf, or some resources (those which create keystone users) will not work: https://review.openstack.org/#/c/73978/ https://review.openstack.org/#/c/76035/ So on upgrade, the deployer must create a keystone domain and domain-admin user, add the details to heat.conf, as already been done in devstack[2]. The changes requried for this to work have already landed in devstack, but it was discussed to day and Clint suggested this may be unacceptable upgrade behavior - I'm not sure so looking for guidance/comments. My plan was/is: - Make devstack work - Talk to tripleo folks to assist in any transition (what prompted this discussion) - Document the upgrade requirements in the Icehouse release notes so the wider community can upgrade from Havana. - Try to give a heads-up to those maintaining downstream heat deployment tools (e.g stackforge/puppet-heat) that some tweaks will be required for Icehouse. However some have suggested there may be an openstack-wide policy which requires peoples old config files to continue working indefinitely on upgrade between versions - is this right? If so where is it documented? This is basically enforced in code in grenade, the language for this actually got lost in the project requirements discussion in the TC, I'll bring that back in the post graduation requirements discussion we're having again. The issue is - Heat still doesn't materially participate in grenade. Heat is substantially far behind the other integrated projects in it's integration with the upstream testing. Only monday did we finally start gating on a real unit of work for Heat (the heat-slow jobs). If I was letter grading projects right now on upstream testing I'd give Nova an A, Neutron a C (still no full run, no working grenade), and Heat a D. Thanks for this, I know we have a lot more work to do in tempest, but evidently grenade integration is something we should priotitize as soon as possible. Any volunteers out there? :) So in short. Heat did the wrong thing. You should be able to use your configs from the last release. This is what all the mature projects in OpenStack do. In the event that you *have* to make a change like that it requires an UpgradeImpact tag in the commit. And those should be limited really aggressively. This is the whole point of the deprecation cycle. Ok, got that message loud and clear now, thanks ;) Do you have a link to docs which describe the deprecation cycle and openstack-wide policy for introducing backwards incompatible changes? The thing I'm still not that clear on, is if we want to eventually require a specific config option, and we can't just have an upgrade requirement to add it as I was expecting - is it enough to just output a warning for one release cycle then require it? Then I guess my question is how do we rationalize the requirements of trunk-chasing downstream users wrt the time based releases as part of the deprecation cycle policy? i.e if we branch stable/icehouse then I immediately post a patch removing the deprecated fallback path, it may still break downstream users who don't care about the stable-branch process and I have no way of knowing (other than, as in this case, finding out too late when they shout at me..). Thanks for contributing to the discussion, hopefully it's not only me who's somewhat confused by the process, and the requirement to satisfy two quite different sets of release constraints for downstream deployers. Perhaps we need a wiki page similar to the StableBranch page which spells out the requirements for projects wrt trunk-chasing deployers, unless one exists already?. Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat]Policy on upgades required config changes
On 03/11/2014 04:04 AM, Sean Dague wrote: On 03/04/2014 12:39 PM, Steven Hardy wrote: Hi all, As some of you know, I've been working on the instance-users blueprint[1]. This blueprint implementation requires three new items to be added to the heat.conf, or some resources (those which create keystone users) will not work: https://review.openstack.org/#/c/73978/ https://review.openstack.org/#/c/76035/ So on upgrade, the deployer must create a keystone domain and domain-admin user, add the details to heat.conf, as already been done in devstack[2]. The changes requried for this to work have already landed in devstack, but it was discussed to day and Clint suggested this may be unacceptable upgrade behavior - I'm not sure so looking for guidance/comments. My plan was/is: - Make devstack work - Talk to tripleo folks to assist in any transition (what prompted this discussion) - Document the upgrade requirements in the Icehouse release notes so the wider community can upgrade from Havana. - Try to give a heads-up to those maintaining downstream heat deployment tools (e.g stackforge/puppet-heat) that some tweaks will be required for Icehouse. However some have suggested there may be an openstack-wide policy which requires peoples old config files to continue working indefinitely on upgrade between versions - is this right? If so where is it documented? This is basically enforced in code in grenade, the language for this actually got lost in the project requirements discussion in the TC, I'll bring that back in the post graduation requirements discussion we're having again. The issue is - Heat still doesn't materially participate in grenade. Heat is substantially far behind the other integrated projects in it's integration with the upstream testing. Only monday did we finally start gating on a real unit of work for Heat (the heat-slow jobs). If I was letter grading projects right now on upstream testing I'd give Nova an A, Neutron a C (still no full run, no working grenade), and Heat a D. Sean, I agree the Heat community hasn't done a bang-up job of getting integrated with Tempest. We only have 50 functional tests implemented. The community clearly needs to do more and provide better functional coverage with Heat. It is inappropriate to say Only monday did we finally start gating because that was a huge move in the right direction. It took alot of effort and should not be so easily dismissed. Clearly the community, and especially the core developers, are making an effort. Keep in mind we have to balance upstream development work, answering user questions, staying on top of a 5 page review queue, keeping relationships and track of the various integrated projects which are consuming Heat as a building block, plus all of the demands of our day jobs. We just don't have enough bandwidth on the core team to tackle writing all of the tempest test cases ourselves. We have made an effort to distribute this work to the overall heat community via wishlist bugs in Heat which several new folks have picked up. I hope to see our coverage improve over time, especially with more advanced scenario tests through this effort. Regards -steve So in short. Heat did the wrong thing. You should be able to use your configs from the last release. This is what all the mature projects in OpenStack do. In the event that you *have* to make a change like that it requires an UpgradeImpact tag in the commit. And those should be limited really aggressively. This is the whole point of the deprecation cycle. -Sean ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat]Policy on upgades required config changes
On 03/11/2014 10:15 AM, Steven Dake wrote: On 03/11/2014 04:04 AM, Sean Dague wrote: On 03/04/2014 12:39 PM, Steven Hardy wrote: Hi all, As some of you know, I've been working on the instance-users blueprint[1]. This blueprint implementation requires three new items to be added to the heat.conf, or some resources (those which create keystone users) will not work: https://review.openstack.org/#/c/73978/ https://review.openstack.org/#/c/76035/ So on upgrade, the deployer must create a keystone domain and domain-admin user, add the details to heat.conf, as already been done in devstack[2]. The changes requried for this to work have already landed in devstack, but it was discussed to day and Clint suggested this may be unacceptable upgrade behavior - I'm not sure so looking for guidance/comments. My plan was/is: - Make devstack work - Talk to tripleo folks to assist in any transition (what prompted this discussion) - Document the upgrade requirements in the Icehouse release notes so the wider community can upgrade from Havana. - Try to give a heads-up to those maintaining downstream heat deployment tools (e.g stackforge/puppet-heat) that some tweaks will be required for Icehouse. However some have suggested there may be an openstack-wide policy which requires peoples old config files to continue working indefinitely on upgrade between versions - is this right? If so where is it documented? This is basically enforced in code in grenade, the language for this actually got lost in the project requirements discussion in the TC, I'll bring that back in the post graduation requirements discussion we're having again. The issue is - Heat still doesn't materially participate in grenade. Heat is substantially far behind the other integrated projects in it's integration with the upstream testing. Only monday did we finally start gating on a real unit of work for Heat (the heat-slow jobs). If I was letter grading projects right now on upstream testing I'd give Nova an A, Neutron a C (still no full run, no working grenade), and Heat a D. Sean, I agree the Heat community hasn't done a bang-up job of getting integrated with Tempest. We only have 50 functional tests implemented. The community clearly needs to do more and provide better functional coverage with Heat. It is inappropriate to say Only monday did we finally start gating because that was a huge move in the right direction. It took alot of effort and should not be so easily dismissed. Clearly the community, and especially the core developers, are making an effort. Keep in mind we have to balance upstream development work, answering user questions, staying on top of a 5 page review queue, keeping relationships and track of the various integrated projects which are consuming Heat as a building block, plus all of the demands of our day jobs. I agree it was a huge step in the right direction. It's not clear to me why expressing that this was very recent was inappropriate. Recent conversations have made me realize that a lot of the Heat core team doesn't realize that Heat's participation in upstream gating is below average, so I decided to be blunt about it. Because it was only after being blunt about that with the Neutron team in Hong Kong did we get any real motion on it (Neutron has seen huge gains this cycle). All the integrated projects have the same challenges. Upstream QA is really important. It not only protects heat from itself, it protects it from changes in other projects. We just don't have enough bandwidth on the core team to tackle writing all of the tempest test cases ourselves. We have made an effort to distribute this work to the overall heat community via wishlist bugs in Heat which several new folks have picked up. I hope to see our coverage improve over time, especially with more advanced scenario tests through this effort. Bandwidth is a problem for everyone. It's a matter of priorities. The fact that realistic upstream gating is considered wishlist priority in from a Heat perspective is something I find troubling. Putting the investment into realistic scenarios in Tempest / gate is going to be a huge timesaving for the Heat team. It will ensure Heat is functioning at every commit (not just releases), it will protect Heat from chasing breaking issues in Keystone or Nova, and it will mean that we'll expose more subtle issues that only come with being able to do data analysis on 10k runs. I get it's never fun to hear that a project is below average on a metric that's important to the OpenStack community. But if we aren't honest and open about these things they never change. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
Re: [openstack-dev] [heat]Policy on upgades required config changes
On 03/11/2014 07:35 AM, Sean Dague wrote: On 03/11/2014 10:15 AM, Steven Dake wrote: On 03/11/2014 04:04 AM, Sean Dague wrote: On 03/04/2014 12:39 PM, Steven Hardy wrote: Hi all, As some of you know, I've been working on the instance-users blueprint[1]. This blueprint implementation requires three new items to be added to the heat.conf, or some resources (those which create keystone users) will not work: https://review.openstack.org/#/c/73978/ https://review.openstack.org/#/c/76035/ So on upgrade, the deployer must create a keystone domain and domain-admin user, add the details to heat.conf, as already been done in devstack[2]. The changes requried for this to work have already landed in devstack, but it was discussed to day and Clint suggested this may be unacceptable upgrade behavior - I'm not sure so looking for guidance/comments. My plan was/is: - Make devstack work - Talk to tripleo folks to assist in any transition (what prompted this discussion) - Document the upgrade requirements in the Icehouse release notes so the wider community can upgrade from Havana. - Try to give a heads-up to those maintaining downstream heat deployment tools (e.g stackforge/puppet-heat) that some tweaks will be required for Icehouse. However some have suggested there may be an openstack-wide policy which requires peoples old config files to continue working indefinitely on upgrade between versions - is this right? If so where is it documented? This is basically enforced in code in grenade, the language for this actually got lost in the project requirements discussion in the TC, I'll bring that back in the post graduation requirements discussion we're having again. The issue is - Heat still doesn't materially participate in grenade. Heat is substantially far behind the other integrated projects in it's integration with the upstream testing. Only monday did we finally start gating on a real unit of work for Heat (the heat-slow jobs). If I was letter grading projects right now on upstream testing I'd give Nova an A, Neutron a C (still no full run, no working grenade), and Heat a D. Sean, I agree the Heat community hasn't done a bang-up job of getting integrated with Tempest. We only have 50 functional tests implemented. The community clearly needs to do more and provide better functional coverage with Heat. It is inappropriate to say Only monday did we finally start gating because that was a huge move in the right direction. It took alot of effort and should not be so easily dismissed. Clearly the community, and especially the core developers, are making an effort. Keep in mind we have to balance upstream development work, answering user questions, staying on top of a 5 page review queue, keeping relationships and track of the various integrated projects which are consuming Heat as a building block, plus all of the demands of our day jobs. I agree it was a huge step in the right direction. It's not clear to me why expressing that this was very recent was inappropriate. Recent conversations have made me realize that a lot of the Heat core team doesn't realize that Heat's participation in upstream gating is below average, so I decided to be blunt about it. Because it was only after being blunt about that with the Neutron team in Hong Kong did we get any real motion on it (Neutron has seen huge gains this cycle). All the integrated projects have the same challenges. Upstream QA is really important. It not only protects heat from itself, it protects it from changes in other projects. We just don't have enough bandwidth on the core team to tackle writing all of the tempest test cases ourselves. We have made an effort to distribute this work to the overall heat community via wishlist bugs in Heat which several new folks have picked up. I hope to see our coverage improve over time, especially with more advanced scenario tests through this effort. Bandwidth is a problem for everyone. It's a matter of priorities. The fact that realistic upstream gating is considered wishlist priority in from a Heat perspective is something I find troubling. Sean, Unfortunately the root of the problem is there is no way to track in one place the suggested test cases for projects. The Tempest community doesn't want test cases in the tempest launchpad tracker. At one point we were told to track the work using etherpads, which is absolutely ridiculous. So we must resort to using wishlist priority. In all cases, a user bug that has a negative impact on operation of Heat is higher priority then implementing functional testing. I get that if we had functional testing, maybe that bug wouldn't have been filed in the first case. However, we are in a situation where we already have the bugs, and they already need to be addressed. If the test cases were stored in tempest launchpad, they could be properly prioritized from a upstream-testing POV. The purpose of the Heat launchpad tracker is to
Re: [openstack-dev] [heat]Policy on upgades required config changes
On 11/03/14 01:05, Keith Bray wrote: We do run close to Heat master here at Rackspace, and we'd be happy to set up a non-voting job to notify when a review would break Heat on our cloud if that would be beneficial. Some of the breaks we have seen have been things that simply weren't caught in code review (a human intensive effort), were specific to the way we configure Heat for large-scale cloud use, applicable to the entire Heat project, and not necessarily service provider specific. +1, thanks Keith that sounds like a great idea. it's obviously not possible to test every configuration, but testing a typical large operator configuration would be a big plus for the project. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat]Policy on upgades required config changes
Excerpts from Steven Hardy's message of 2014-03-05 04:24:51 -0800: On Tue, Mar 04, 2014 at 02:06:16PM -0800, Clint Byrum wrote: Excerpts from Steven Hardy's message of 2014-03-04 09:39:21 -0800: Hi all, As some of you know, I've been working on the instance-users blueprint[1]. This blueprint implementation requires three new items to be added to the heat.conf, or some resources (those which create keystone users) will not work: https://review.openstack.org/#/c/73978/ https://review.openstack.org/#/c/76035/ So on upgrade, the deployer must create a keystone domain and domain-admin user, add the details to heat.conf, as already been done in devstack[2]. The changes requried for this to work have already landed in devstack, but it was discussed to day and Clint suggested this may be unacceptable upgrade behavior - I'm not sure so looking for guidance/comments. My plan was/is: - Make devstack work - Talk to tripleo folks to assist in any transition (what prompted this discussion) - Document the upgrade requirements in the Icehouse release notes so the wider community can upgrade from Havana. - Try to give a heads-up to those maintaining downstream heat deployment tools (e.g stackforge/puppet-heat) that some tweaks will be required for Icehouse. However some have suggested there may be an openstack-wide policy which requires peoples old config files to continue working indefinitely on upgrade between versions - is this right? If so where is it documented? I don't think I said indefinitely, and I certainly did not mean indefinitely. What is required though, is that we be able to upgrade to the next release without requiring a new config setting. So log a warning for one cycle, then it's OK to expect the config after that? Correct. I'm still unclear if there's an openstack-wide policy on this, as the whole time-based release with release-notes (which all of openstack is structured around and adheres to) seems to basically be an uncomfortable fit for folks like tripleo who are trunk chasing and doing CI. So we're continuous delivery focused, but we are not special. HP Cloud and Rackspace both do this, and really anyone running a large cloud will most likely do so with CD, as the value proposition is that you don't have big scary upgrades, you just keep incrementally upgrading and getting newer, better code. We can only do this if we have excellent testing, which upstream already does and which the public clouds all do privately as well of course. Changes like the one that was merged last week in Heat turn into stressful fire drills for those deployment teams. Also as we scramble to deal with these things in TripleO (as all of our users are now unable to spin up new images), it is clear that it is more than just a setting. One must create domain users carefully and roll out a new password. Such are the pitfalls of life at the bleeding edge ;) This is mildly annoying as a stance, as that's not how we've been operating with all of the other services of OpenStack. We're not crazy for wanting to deploy master and for wanting master to keep working. We are a _little_ crazy for wanting that without being in the gate. Seriously though, apologies for the inconvenience - I have been asking for feedback on these patches for at least a month, but clearly I should've asked harder. Mea culpa too, I did not realize what impact this would have until it was too late. As was discussed on IRC yesterday, I think some sort of (initially non-voting) feedback from tripleo CI to heat gerrit is pretty much essential given that you're so highly coupled to us or this will just keep happening. TripleO will be in the gate some day (hopefully soon!) and then this will be less of an issue as you'd see failures early on, and could open bugs and get us to fix our issue sooner. However you'd still need to provide the backward compatibility for a single cycle. Servers aren't upgraded instantly, and keystone may not be ready for this v3/domain change until after users have fully rolled out Icehouse. Whether one is CD or stable release focused, one still needs a simple solution to rolling out a massive update. What I'm suggesting is that we should instead _warn_ that the old behavior is being used and will be deprecated. At this point, out of urgency, we're landing fixes. But in the future, this should be considered carefully. Ok, well I raised this bug: https://bugs.launchpad.net/heat/+bug/1287980 So we can modify the stuff so that it falls back to the old behavior gracefully and will solve the issue for folks on the time-based releases. Hopefully we can work towards the tripleo gate feedback so next time this is less of a suprise for all of us ;) Yes, and hopefully it is also clear that we really do need to make things simpler for upgraders,
Re: [openstack-dev] [heat]Policy on upgades required config changes
I want to echo Clint's responses... We do run close to Heat master here at Rackspace, and we'd be happy to set up a non-voting job to notify when a review would break Heat on our cloud if that would be beneficial. Some of the breaks we have seen have been things that simply weren't caught in code review (a human intensive effort), were specific to the way we configure Heat for large-scale cloud use, applicable to the entire Heat project, and not necessarily service provider specific. -Keith On 3/10/14 5:19 PM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Steven Hardy's message of 2014-03-05 04:24:51 -0800: On Tue, Mar 04, 2014 at 02:06:16PM -0800, Clint Byrum wrote: Excerpts from Steven Hardy's message of 2014-03-04 09:39:21 -0800: Hi all, As some of you know, I've been working on the instance-users blueprint[1]. This blueprint implementation requires three new items to be added to the heat.conf, or some resources (those which create keystone users) will not work: https://review.openstack.org/#/c/73978/ https://review.openstack.org/#/c/76035/ So on upgrade, the deployer must create a keystone domain and domain-admin user, add the details to heat.conf, as already been done in devstack[2]. The changes requried for this to work have already landed in devstack, but it was discussed to day and Clint suggested this may be unacceptable upgrade behavior - I'm not sure so looking for guidance/comments. My plan was/is: - Make devstack work - Talk to tripleo folks to assist in any transition (what prompted this discussion) - Document the upgrade requirements in the Icehouse release notes so the wider community can upgrade from Havana. - Try to give a heads-up to those maintaining downstream heat deployment tools (e.g stackforge/puppet-heat) that some tweaks will be required for Icehouse. However some have suggested there may be an openstack-wide policy which requires peoples old config files to continue working indefinitely on upgrade between versions - is this right? If so where is it documented? I don't think I said indefinitely, and I certainly did not mean indefinitely. What is required though, is that we be able to upgrade to the next release without requiring a new config setting. So log a warning for one cycle, then it's OK to expect the config after that? Correct. I'm still unclear if there's an openstack-wide policy on this, as the whole time-based release with release-notes (which all of openstack is structured around and adheres to) seems to basically be an uncomfortable fit for folks like tripleo who are trunk chasing and doing CI. So we're continuous delivery focused, but we are not special. HP Cloud and Rackspace both do this, and really anyone running a large cloud will most likely do so with CD, as the value proposition is that you don't have big scary upgrades, you just keep incrementally upgrading and getting newer, better code. We can only do this if we have excellent testing, which upstream already does and which the public clouds all do privately as well of course. Changes like the one that was merged last week in Heat turn into stressful fire drills for those deployment teams. Also as we scramble to deal with these things in TripleO (as all of our users are now unable to spin up new images), it is clear that it is more than just a setting. One must create domain users carefully and roll out a new password. Such are the pitfalls of life at the bleeding edge ;) This is mildly annoying as a stance, as that's not how we've been operating with all of the other services of OpenStack. We're not crazy for wanting to deploy master and for wanting master to keep working. We are a _little_ crazy for wanting that without being in the gate. Seriously though, apologies for the inconvenience - I have been asking for feedback on these patches for at least a month, but clearly I should've asked harder. Mea culpa too, I did not realize what impact this would have until it was too late. As was discussed on IRC yesterday, I think some sort of (initially non-voting) feedback from tripleo CI to heat gerrit is pretty much essential given that you're so highly coupled to us or this will just keep happening. TripleO will be in the gate some day (hopefully soon!) and then this will be less of an issue as you'd see failures early on, and could open bugs and get us to fix our issue sooner. However you'd still need to provide the backward compatibility for a single cycle. Servers aren't upgraded instantly, and keystone may not be ready for this v3/domain change until after users have fully rolled out Icehouse. Whether one is CD or stable release focused, one still needs a simple solution to rolling out a massive update. What I'm suggesting is that we should instead _warn_ that the old behavior is being used and will be deprecated.
Re: [openstack-dev] [heat]Policy on upgades required config changes
On Tue, Mar 04, 2014 at 02:06:16PM -0800, Clint Byrum wrote: Excerpts from Steven Hardy's message of 2014-03-04 09:39:21 -0800: Hi all, As some of you know, I've been working on the instance-users blueprint[1]. This blueprint implementation requires three new items to be added to the heat.conf, or some resources (those which create keystone users) will not work: https://review.openstack.org/#/c/73978/ https://review.openstack.org/#/c/76035/ So on upgrade, the deployer must create a keystone domain and domain-admin user, add the details to heat.conf, as already been done in devstack[2]. The changes requried for this to work have already landed in devstack, but it was discussed to day and Clint suggested this may be unacceptable upgrade behavior - I'm not sure so looking for guidance/comments. My plan was/is: - Make devstack work - Talk to tripleo folks to assist in any transition (what prompted this discussion) - Document the upgrade requirements in the Icehouse release notes so the wider community can upgrade from Havana. - Try to give a heads-up to those maintaining downstream heat deployment tools (e.g stackforge/puppet-heat) that some tweaks will be required for Icehouse. However some have suggested there may be an openstack-wide policy which requires peoples old config files to continue working indefinitely on upgrade between versions - is this right? If so where is it documented? I don't think I said indefinitely, and I certainly did not mean indefinitely. What is required though, is that we be able to upgrade to the next release without requiring a new config setting. So log a warning for one cycle, then it's OK to expect the config after that? I'm still unclear if there's an openstack-wide policy on this, as the whole time-based release with release-notes (which all of openstack is structured around and adheres to) seems to basically be an uncomfortable fit for folks like tripleo who are trunk chasing and doing CI. Also as we scramble to deal with these things in TripleO (as all of our users are now unable to spin up new images), it is clear that it is more than just a setting. One must create domain users carefully and roll out a new password. Such are the pitfalls of life at the bleeding edge ;) Seriously though, apologies for the inconvenience - I have been asking for feedback on these patches for at least a month, but clearly I should've asked harder. As was discussed on IRC yesterday, I think some sort of (initially non-voting) feedback from tripleo CI to heat gerrit is pretty much essential given that you're so highly coupled to us or this will just keep happening. What I'm suggesting is that we should instead _warn_ that the old behavior is being used and will be deprecated. At this point, out of urgency, we're landing fixes. But in the future, this should be considered carefully. Ok, well I raised this bug: https://bugs.launchpad.net/heat/+bug/1287980 So we can modify the stuff so that it falls back to the old behavior gracefully and will solve the issue for folks on the time-based releases. Hopefully we can work towards the tripleo gate feedback so next time this is less of a suprise for all of us ;) Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [heat]Policy on upgades required config changes
Hi all, As some of you know, I've been working on the instance-users blueprint[1]. This blueprint implementation requires three new items to be added to the heat.conf, or some resources (those which create keystone users) will not work: https://review.openstack.org/#/c/73978/ https://review.openstack.org/#/c/76035/ So on upgrade, the deployer must create a keystone domain and domain-admin user, add the details to heat.conf, as already been done in devstack[2]. The changes requried for this to work have already landed in devstack, but it was discussed to day and Clint suggested this may be unacceptable upgrade behavior - I'm not sure so looking for guidance/comments. My plan was/is: - Make devstack work - Talk to tripleo folks to assist in any transition (what prompted this discussion) - Document the upgrade requirements in the Icehouse release notes so the wider community can upgrade from Havana. - Try to give a heads-up to those maintaining downstream heat deployment tools (e.g stackforge/puppet-heat) that some tweaks will be required for Icehouse. However some have suggested there may be an openstack-wide policy which requires peoples old config files to continue working indefinitely on upgrade between versions - is this right? If so where is it documented? The code itself will handle backwards compatibility where existing stacks were created with the old code, but I had assumed (as a concession to code simplicity) that some documented upgrade procedure would be acceptable rather than hacking in some way to support the previous (broken, ref bug #1089261) behavior when the config values are not found. If anyone can clarify the requirement/expectation around config files and upgrades that would be most helpful, thanks! Steve [1] https://blueprints.launchpad.net/heat/+spec/instance-users [2] https://review.openstack.org/#/c/73324/ https://review.openstack.org/#/c/75424/ https://review.openstack.org/#/c/76036/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat]Policy on upgades required config changes
Excerpts from Steven Hardy's message of 2014-03-04 09:39:21 -0800: Hi all, As some of you know, I've been working on the instance-users blueprint[1]. This blueprint implementation requires three new items to be added to the heat.conf, or some resources (those which create keystone users) will not work: https://review.openstack.org/#/c/73978/ https://review.openstack.org/#/c/76035/ So on upgrade, the deployer must create a keystone domain and domain-admin user, add the details to heat.conf, as already been done in devstack[2]. The changes requried for this to work have already landed in devstack, but it was discussed to day and Clint suggested this may be unacceptable upgrade behavior - I'm not sure so looking for guidance/comments. My plan was/is: - Make devstack work - Talk to tripleo folks to assist in any transition (what prompted this discussion) - Document the upgrade requirements in the Icehouse release notes so the wider community can upgrade from Havana. - Try to give a heads-up to those maintaining downstream heat deployment tools (e.g stackforge/puppet-heat) that some tweaks will be required for Icehouse. However some have suggested there may be an openstack-wide policy which requires peoples old config files to continue working indefinitely on upgrade between versions - is this right? If so where is it documented? I don't think I said indefinitely, and I certainly did not mean indefinitely. What is required though, is that we be able to upgrade to the next release without requiring a new config setting. Also as we scramble to deal with these things in TripleO (as all of our users are now unable to spin up new images), it is clear that it is more than just a setting. One must create domain users carefully and roll out a new password. What I'm suggesting is that we should instead _warn_ that the old behavior is being used and will be deprecated. At this point, out of urgency, we're landing fixes. But in the future, this should be considered carefully. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
