Re: [openstack-dev] [tripleo] tripleo-heat-templates, vendor plugins and the new hiera hook
On 27/01/17 16:23, Dan Prince wrote: > On Wed, 2017-01-25 at 14:59 +0200, Marios Andreou wrote: >> Hi, as part of the composable upgrades workflow shaping up for Newton >> to >> Ocata, we need to install the new hiera hook that was first added >> with >> [1] and disable the old hook and data as part of the upgrade >> initialization [2]. Most of the existing hieradata was ported to use >> the >> new hook in [3]. The deletion of the old hiera data is necessary for >> the >> Ocata upgrade, but it also means it will break any plugins still >> using >> the 'old' os-apply-config hiera hook. > > > Nice catch on the old vendor hieradata. I clearly missed those for the record... actually thanks to slagle - see discussion at https://review.openstack.org/#/c/424715 > interfaces for the in-tree extraconfig data when doing these conversion > (sorry about that). Would be nice to get some sort of coverage on these > interfaces I guess. > > The new hook uses Json and is much cleaner. We were accumulating a lot > of hacks in t-h-t to work around deficiencies with the old o-a-c YAML > element mechanism. What this means is a conversion tool is hard. Not > impossible, but might not get 100% of the cases I think due to the > differences in how YAML and Json can handle arrays and such with all > the conversions going on. > > Updating the rest of the in-tree interfaces (like you have done) should > get most of it. For any out of tree extraconfig code that relies on the > old heira element would it be reasonable to fail-fast instead? There > isn't a great place to do this unfortunately but a couple of options: > > 1) in the agent hook itself: https://review.openstack.org/#/c/426241/1 > > 2) in the old hiera hook: https://review.openstack.org/#/c/425955/ > > Option #1 handles signals more nicely but couples the old and new > implementations a bit with the extra check. Option #2 doesn't currently > handle signaling nicely (as shardy pointed out in the review). fwiw #1 makes more sense to me (I didn't see that until just now, after having reviewed #2 first thing today) just so the operator gets an early (earlier at least) exit with some error message. thanks > > Dan > >> >> In order to be able to upgrade to Ocata any templates that define >> hiera >> data need to be using the new hiera hook and then the overcloud nodes >> need to have the new hook installed (installing is done in [2] as a >> matter of necessity, and that is what prompted this email in the >> first >> place). I've had a go at updating all the plugin templates that are >> still using the old hiera data with a review at [4] which I have -1 >> for now. >> >> I'll try and reach out to some individuals more directly as well but >> wanted to get the review at [4] and this email out as a first step, >> >> thanks, marios >> >> [1] https://review.openstack.org/#/c/379733/ >> [2] >> https://review.openstack.org/#/c/424715/2/extraconfig/tasks/newton_oc >> ata_upgrade_init_common.sh >> [3] https://review.openstack.org/#/c/384757/ >> [4] https://review.openstack.org/#/c/425154/ >> >> _ >> _ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubs >> cribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] tripleo-heat-templates, vendor plugins and the new hiera hook
On Wed, 2017-01-25 at 14:59 +0200, Marios Andreou wrote: > Hi, as part of the composable upgrades workflow shaping up for Newton > to > Ocata, we need to install the new hiera hook that was first added > with > [1] and disable the old hook and data as part of the upgrade > initialization [2]. Most of the existing hieradata was ported to use > the > new hook in [3]. The deletion of the old hiera data is necessary for > the > Ocata upgrade, but it also means it will break any plugins still > using > the 'old' os-apply-config hiera hook. Nice catch on the old vendor hieradata. I clearly missed those interfaces for the in-tree extraconfig data when doing these conversion (sorry about that). Would be nice to get some sort of coverage on these interfaces I guess. The new hook uses Json and is much cleaner. We were accumulating a lot of hacks in t-h-t to work around deficiencies with the old o-a-c YAML element mechanism. What this means is a conversion tool is hard. Not impossible, but might not get 100% of the cases I think due to the differences in how YAML and Json can handle arrays and such with all the conversions going on. Updating the rest of the in-tree interfaces (like you have done) should get most of it. For any out of tree extraconfig code that relies on the old heira element would it be reasonable to fail-fast instead? There isn't a great place to do this unfortunately but a couple of options: 1) in the agent hook itself: https://review.openstack.org/#/c/426241/1 2) in the old hiera hook: https://review.openstack.org/#/c/425955/ Option #1 handles signals more nicely but couples the old and new implementations a bit with the extra check. Option #2 doesn't currently handle signaling nicely (as shardy pointed out in the review). Dan > > In order to be able to upgrade to Ocata any templates that define > hiera > data need to be using the new hiera hook and then the overcloud nodes > need to have the new hook installed (installing is done in [2] as a > matter of necessity, and that is what prompted this email in the > first > place). I've had a go at updating all the plugin templates that are > still using the old hiera data with a review at [4] which I have -1 > for now. > > I'll try and reach out to some individuals more directly as well but > wanted to get the review at [4] and this email out as a first step, > > thanks, marios > > [1] https://review.openstack.org/#/c/379733/ > [2] > https://review.openstack.org/#/c/424715/2/extraconfig/tasks/newton_oc > ata_upgrade_init_common.sh > [3] https://review.openstack.org/#/c/384757/ > [4] https://review.openstack.org/#/c/425154/ > > _ > _ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubs > cribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] tripleo-heat-templates, vendor plugins and the new hiera hook
Hi, Steven Hardy writes: > On Wed, Jan 25, 2017 at 02:59:42PM +0200, Marios Andreou wrote: >> Hi, as part of the composable upgrades workflow shaping up for Newton to >> Ocata, we need to install the new hiera hook that was first added with >> [1] and disable the old hook and data as part of the upgrade >> initialization [2]. Most of the existing hieradata was ported to use the >> new hook in [3]. The deletion of the old hiera data is necessary for the >> Ocata upgrade, but it also means it will break any plugins still using >> the 'old' os-apply-config hiera hook. >> >> In order to be able to upgrade to Ocata any templates that define hiera >> data need to be using the new hiera hook and then the overcloud nodes >> need to have the new hook installed (installing is done in [2] as a >> matter of necessity, and that is what prompted this email in the first >> place). I've had a go at updating all the plugin templates that are >> still using the old hiera data with a review at [4] which I have -1 for now. >> >> I'll try and reach out to some individuals more directly as well but >> wanted to get the review at [4] and this email out as a first step, > > Thanks for raising this marios, and yeah it's unfortunate as we've had to > do a switch from the old to new hiera hook this release with out a > transition where both work. > > I think we probably need to do the following: > > 1. Convert anything in t-h-t refering to the old hook to the new (seems you > have this in progress, we need to ensure it all lands before ocata) > > 2. Write a good release note for t-h-t explaining the change, referencing > docs which show how to convert to use the new hook > > 3. Figure out a way to make the 99-refresh-completed script signal failure > if anyone tries to deploy with the old hook (vs potentially silently > failing then hanging the deploy, which I think is what will happen atm). I've created a bug to make sure this isn't forgotten before the release: https://bugs.launchpad.net/tripleo/+bug/1659540 > > I think ensuring a good error path should mitigate this change, since it's > fairly simple for folks to switch to the new hook provided we can document > it and point to those docs in the error I think. > > Be good to get input from Dan on this too, as he might have ideas on how we > could maintain both hooks for one release. This would be ideal, and would remove the need for the previous bug. Thanks, > > Thanks! > > Steve > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sofer Athlan-Guyot __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] tripleo-heat-templates, vendor plugins and the new hiera hook
On 01/25/2017 04:32 PM, Steven Hardy wrote: > On Wed, Jan 25, 2017 at 02:59:42PM +0200, Marios Andreou wrote: >> Hi, as part of the composable upgrades workflow shaping up for Newton to >> Ocata, we need to install the new hiera hook that was first added with >> [1] and disable the old hook and data as part of the upgrade >> initialization [2]. Most of the existing hieradata was ported to use the >> new hook in [3]. The deletion of the old hiera data is necessary for the >> Ocata upgrade, but it also means it will break any plugins still using >> the 'old' os-apply-config hiera hook. >> >> In order to be able to upgrade to Ocata any templates that define hiera >> data need to be using the new hiera hook and then the overcloud nodes >> need to have the new hook installed (installing is done in [2] as a >> matter of necessity, and that is what prompted this email in the first >> place). I've had a go at updating all the plugin templates that are >> still using the old hiera data with a review at [4] which I have -1 for now. >> >> I'll try and reach out to some individuals more directly as well but >> wanted to get the review at [4] and this email out as a first step, > > Thanks for raising this marios, and yeah it's unfortunate as we've had to > do a switch from the old to new hiera hook this release with out a > transition where both work. > > I think we probably need to do the following: > > 1. Convert anything in t-h-t refering to the old hook to the new (seems you > have this in progress, we need to ensure it all lands before ocata) was working on this as well today for Ceph https://review.openstack.org/#/c/425288/ thanks marios -- Giulio Fidente GPG KEY: 08D733BA __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] tripleo-heat-templates, vendor plugins and the new hiera hook
On Wed, Jan 25, 2017 at 02:59:42PM +0200, Marios Andreou wrote: > Hi, as part of the composable upgrades workflow shaping up for Newton to > Ocata, we need to install the new hiera hook that was first added with > [1] and disable the old hook and data as part of the upgrade > initialization [2]. Most of the existing hieradata was ported to use the > new hook in [3]. The deletion of the old hiera data is necessary for the > Ocata upgrade, but it also means it will break any plugins still using > the 'old' os-apply-config hiera hook. > > In order to be able to upgrade to Ocata any templates that define hiera > data need to be using the new hiera hook and then the overcloud nodes > need to have the new hook installed (installing is done in [2] as a > matter of necessity, and that is what prompted this email in the first > place). I've had a go at updating all the plugin templates that are > still using the old hiera data with a review at [4] which I have -1 for now. > > I'll try and reach out to some individuals more directly as well but > wanted to get the review at [4] and this email out as a first step, Thanks for raising this marios, and yeah it's unfortunate as we've had to do a switch from the old to new hiera hook this release with out a transition where both work. I think we probably need to do the following: 1. Convert anything in t-h-t refering to the old hook to the new (seems you have this in progress, we need to ensure it all lands before ocata) 2. Write a good release note for t-h-t explaining the change, referencing docs which show how to convert to use the new hook 3. Figure out a way to make the 99-refresh-completed script signal failure if anyone tries to deploy with the old hook (vs potentially silently failing then hanging the deploy, which I think is what will happen atm). I think ensuring a good error path should mitigate this change, since it's fairly simple for folks to switch to the new hook provided we can document it and point to those docs in the error I think. Be good to get input from Dan on this too, as he might have ideas on how we could maintain both hooks for one release. Thanks! Steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [tripleo] tripleo-heat-templates, vendor plugins and the new hiera hook
Hi, as part of the composable upgrades workflow shaping up for Newton to Ocata, we need to install the new hiera hook that was first added with [1] and disable the old hook and data as part of the upgrade initialization [2]. Most of the existing hieradata was ported to use the new hook in [3]. The deletion of the old hiera data is necessary for the Ocata upgrade, but it also means it will break any plugins still using the 'old' os-apply-config hiera hook. In order to be able to upgrade to Ocata any templates that define hiera data need to be using the new hiera hook and then the overcloud nodes need to have the new hook installed (installing is done in [2] as a matter of necessity, and that is what prompted this email in the first place). I've had a go at updating all the plugin templates that are still using the old hiera data with a review at [4] which I have -1 for now. I'll try and reach out to some individuals more directly as well but wanted to get the review at [4] and this email out as a first step, thanks, marios [1] https://review.openstack.org/#/c/379733/ [2] https://review.openstack.org/#/c/424715/2/extraconfig/tasks/newton_ocata_upgrade_init_common.sh [3] https://review.openstack.org/#/c/384757/ [4] https://review.openstack.org/#/c/425154/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev