Re: [openstack-dev] [tripleo] tripleo-heat-templates, vendor plugins and the new hiera hook

[Marios Andreou]

On 27/01/17 16:23, Dan Prince wrote:
> On Wed, 2017-01-25 at 14:59 +0200, Marios Andreou wrote:
>> Hi, as part of the composable upgrades workflow shaping up for Newton
>> to
>> Ocata, we need to install the new hiera hook that was first added
>> with
>> [1] and disable the old hook and data as part of the upgrade
>> initialization [2]. Most of the existing hieradata was ported to use
>> the
>> new hook in [3]. The deletion of the old hiera data is necessary for
>> the
>> Ocata upgrade, but it also means it will break any plugins still
>> using
>> the 'old' os-apply-config hiera hook.
> 
> 
> Nice catch on the old vendor hieradata. I clearly missed those

for the record... actually thanks to slagle - see discussion at
https://review.openstack.org/#/c/424715

> interfaces for the in-tree extraconfig data when doing these conversion
> (sorry about that). Would be nice to get some sort of coverage on these
> interfaces I guess.
> 
> The new hook uses Json and is much cleaner. We were accumulating a lot
> of hacks in t-h-t to work around deficiencies with the old o-a-c YAML
> element mechanism. What this means is a conversion tool is hard. Not
> impossible, but might not get 100% of the cases I think due to the
> differences in how YAML and Json can handle arrays and such with all
> the conversions going on.
> 
> Updating the rest of the in-tree interfaces (like you have done) should
> get most of it. For any out of tree extraconfig code that relies on the
> old heira element would it be reasonable to fail-fast instead? There
> isn't a great place to do this unfortunately but a couple of options:
> 
> 1) in the agent hook itself: https://review.openstack.org/#/c/426241/1
> 
> 2) in the old hiera hook: https://review.openstack.org/#/c/425955/
> 
> Option #1 handles signals more nicely but couples the old and new
> implementations a bit with the extra check. Option #2 doesn't currently
> handle signaling nicely (as shardy pointed out in the review).

fwiw #1 makes more sense to me (I didn't see that until just now, after
having reviewed #2 first thing today) just so the operator gets an early
(earlier at least) exit with some error message.

thanks

> 
> Dan
> 
>>
>> In order to be able to upgrade to Ocata any templates that define
>> hiera
>> data need to be using the new hiera hook and then the overcloud nodes
>> need to have the new hook installed (installing is done in [2] as a
>> matter of necessity, and that is what prompted this email in the
>> first
>> place). I've had a go at updating all the plugin templates that are
>> still using the old hiera data with a review at [4] which I have -1
>> for now.
>>
>> I'll try and reach out to some individuals more directly as well but
>> wanted to get the review at [4] and this email out as a first step,
>>
>> thanks, marios
>>
>> [1] https://review.openstack.org/#/c/379733/
>> [2]
>> https://review.openstack.org/#/c/424715/2/extraconfig/tasks/newton_oc
>> ata_upgrade_init_common.sh
>> [3] https://review.openstack.org/#/c/384757/
>> [4] https://review.openstack.org/#/c/425154/
>>
>> _
>> _
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubs
>> cribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] tripleo-heat-templates, vendor plugins and the new hiera hook

[Dan Prince]

On Wed, 2017-01-25 at 14:59 +0200, Marios Andreou wrote:
> Hi, as part of the composable upgrades workflow shaping up for Newton
> to
> Ocata, we need to install the new hiera hook that was first added
> with
> [1] and disable the old hook and data as part of the upgrade
> initialization [2]. Most of the existing hieradata was ported to use
> the
> new hook in [3]. The deletion of the old hiera data is necessary for
> the
> Ocata upgrade, but it also means it will break any plugins still
> using
> the 'old' os-apply-config hiera hook.


Nice catch on the old vendor hieradata. I clearly missed those
interfaces for the in-tree extraconfig data when doing these conversion
(sorry about that). Would be nice to get some sort of coverage on these
interfaces I guess.

The new hook uses Json and is much cleaner. We were accumulating a lot
of hacks in t-h-t to work around deficiencies with the old o-a-c YAML
element mechanism. What this means is a conversion tool is hard. Not
impossible, but might not get 100% of the cases I think due to the
differences in how YAML and Json can handle arrays and such with all
the conversions going on.

Updating the rest of the in-tree interfaces (like you have done) should
get most of it. For any out of tree extraconfig code that relies on the
old heira element would it be reasonable to fail-fast instead? There
isn't a great place to do this unfortunately but a couple of options:

1) in the agent hook itself: https://review.openstack.org/#/c/426241/1

2) in the old hiera hook: https://review.openstack.org/#/c/425955/

Option #1 handles signals more nicely but couples the old and new
implementations a bit with the extra check. Option #2 doesn't currently
handle signaling nicely (as shardy pointed out in the review).

Dan

> 
> In order to be able to upgrade to Ocata any templates that define
> hiera
> data need to be using the new hiera hook and then the overcloud nodes
> need to have the new hook installed (installing is done in [2] as a
> matter of necessity, and that is what prompted this email in the
> first
> place). I've had a go at updating all the plugin templates that are
> still using the old hiera data with a review at [4] which I have -1
> for now.
> 
> I'll try and reach out to some individuals more directly as well but
> wanted to get the review at [4] and this email out as a first step,
> 
> thanks, marios
> 
> [1] https://review.openstack.org/#/c/379733/
> [2]
> https://review.openstack.org/#/c/424715/2/extraconfig/tasks/newton_oc
> ata_upgrade_init_common.sh
> [3] https://review.openstack.org/#/c/384757/
> [4] https://review.openstack.org/#/c/425154/
> 
> _
> _
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubs
> cribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] tripleo-heat-templates, vendor plugins and the new hiera hook

[Sofer Athlan-Guyot]

Hi,

Steven Hardy  writes:

> On Wed, Jan 25, 2017 at 02:59:42PM +0200, Marios Andreou wrote:
>> Hi, as part of the composable upgrades workflow shaping up for Newton to
>> Ocata, we need to install the new hiera hook that was first added with
>> [1] and disable the old hook and data as part of the upgrade
>> initialization [2]. Most of the existing hieradata was ported to use the
>> new hook in [3]. The deletion of the old hiera data is necessary for the
>> Ocata upgrade, but it also means it will break any plugins still using
>> the 'old' os-apply-config hiera hook.
>> 
>> In order to be able to upgrade to Ocata any templates that define hiera
>> data need to be using the new hiera hook and then the overcloud nodes
>> need to have the new hook installed (installing is done in [2] as a
>> matter of necessity, and that is what prompted this email in the first
>> place). I've had a go at updating all the plugin templates that are
>> still using the old hiera data with a review at [4] which I have -1 for now.
>> 
>> I'll try and reach out to some individuals more directly as well but
>> wanted to get the review at [4] and this email out as a first step,
>
> Thanks for raising this marios, and yeah it's unfortunate as we've had to
> do a switch from the old to new hiera hook this release with out a
> transition where both work.
>
> I think we probably need to do the following:
>
> 1. Convert anything in t-h-t refering to the old hook to the new (seems you
> have this in progress, we need to ensure it all lands before ocata)
>
> 2. Write a good release note for t-h-t explaining the change, referencing
> docs which show how to convert to use the new hook
>
> 3. Figure out a way to make the 99-refresh-completed script signal failure
> if anyone tries to deploy with the old hook (vs potentially silently
> failing then hanging the deploy, which I think is what will happen atm).

I've created a bug to make sure this isn't forgotten before the release:

  https://bugs.launchpad.net/tripleo/+bug/1659540

>
> I think ensuring a good error path should mitigate this change, since it's
> fairly simple for folks to switch to the new hook provided we can document
> it and point to those docs in the error I think.
>
> Be good to get input from Dan on this too, as he might have ideas on how we
> could maintain both hooks for one release.

This would be ideal, and would remove the need for the previous bug.

Thanks,

>
> Thanks!
>
> Steve
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
Sofer Athlan-Guyot

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] tripleo-heat-templates, vendor plugins and the new hiera hook

[Giulio Fidente]

On 01/25/2017 04:32 PM, Steven Hardy wrote:
> On Wed, Jan 25, 2017 at 02:59:42PM +0200, Marios Andreou wrote:
>> Hi, as part of the composable upgrades workflow shaping up for Newton to
>> Ocata, we need to install the new hiera hook that was first added with
>> [1] and disable the old hook and data as part of the upgrade
>> initialization [2]. Most of the existing hieradata was ported to use the
>> new hook in [3]. The deletion of the old hiera data is necessary for the
>> Ocata upgrade, but it also means it will break any plugins still using
>> the 'old' os-apply-config hiera hook.
>>
>> In order to be able to upgrade to Ocata any templates that define hiera
>> data need to be using the new hiera hook and then the overcloud nodes
>> need to have the new hook installed (installing is done in [2] as a
>> matter of necessity, and that is what prompted this email in the first
>> place). I've had a go at updating all the plugin templates that are
>> still using the old hiera data with a review at [4] which I have -1 for now.
>>
>> I'll try and reach out to some individuals more directly as well but
>> wanted to get the review at [4] and this email out as a first step,
> 
> Thanks for raising this marios, and yeah it's unfortunate as we've had to
> do a switch from the old to new hiera hook this release with out a
> transition where both work.
> 
> I think we probably need to do the following:
> 
> 1. Convert anything in t-h-t refering to the old hook to the new (seems you
> have this in progress, we need to ensure it all lands before ocata)

was working on this as well today for Ceph

https://review.openstack.org/#/c/425288/

thanks marios
-- 
Giulio Fidente
GPG KEY: 08D733BA

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] tripleo-heat-templates, vendor plugins and the new hiera hook

[Steven Hardy]

On Wed, Jan 25, 2017 at 02:59:42PM +0200, Marios Andreou wrote:
> Hi, as part of the composable upgrades workflow shaping up for Newton to
> Ocata, we need to install the new hiera hook that was first added with
> [1] and disable the old hook and data as part of the upgrade
> initialization [2]. Most of the existing hieradata was ported to use the
> new hook in [3]. The deletion of the old hiera data is necessary for the
> Ocata upgrade, but it also means it will break any plugins still using
> the 'old' os-apply-config hiera hook.
> 
> In order to be able to upgrade to Ocata any templates that define hiera
> data need to be using the new hiera hook and then the overcloud nodes
> need to have the new hook installed (installing is done in [2] as a
> matter of necessity, and that is what prompted this email in the first
> place). I've had a go at updating all the plugin templates that are
> still using the old hiera data with a review at [4] which I have -1 for now.
> 
> I'll try and reach out to some individuals more directly as well but
> wanted to get the review at [4] and this email out as a first step,

Thanks for raising this marios, and yeah it's unfortunate as we've had to
do a switch from the old to new hiera hook this release with out a
transition where both work.

I think we probably need to do the following:

1. Convert anything in t-h-t refering to the old hook to the new (seems you
have this in progress, we need to ensure it all lands before ocata)

2. Write a good release note for t-h-t explaining the change, referencing
docs which show how to convert to use the new hook

3. Figure out a way to make the 99-refresh-completed script signal failure
if anyone tries to deploy with the old hook (vs potentially silently
failing then hanging the deploy, which I think is what will happen atm).

I think ensuring a good error path should mitigate this change, since it's
fairly simple for folks to switch to the new hook provided we can document
it and point to those docs in the error I think.

Be good to get input from Dan on this too, as he might have ideas on how we
could maintain both hooks for one release.

Thanks!

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] tripleo-heat-templates, vendor plugins and the new hiera hook

[Marios Andreou]

Hi, as part of the composable upgrades workflow shaping up for Newton to
Ocata, we need to install the new hiera hook that was first added with
[1] and disable the old hook and data as part of the upgrade
initialization [2]. Most of the existing hieradata was ported to use the
new hook in [3]. The deletion of the old hiera data is necessary for the
Ocata upgrade, but it also means it will break any plugins still using
the 'old' os-apply-config hiera hook.

In order to be able to upgrade to Ocata any templates that define hiera
data need to be using the new hiera hook and then the overcloud nodes
need to have the new hook installed (installing is done in [2] as a
matter of necessity, and that is what prompted this email in the first
place). I've had a go at updating all the plugin templates that are
still using the old hiera data with a review at [4] which I have -1 for now.

I'll try and reach out to some individuals more directly as well but
wanted to get the review at [4] and this email out as a first step,

thanks, marios

[1] https://review.openstack.org/#/c/379733/
[2]
https://review.openstack.org/#/c/424715/2/extraconfig/tasks/newton_ocata_upgrade_init_common.sh
[3] https://review.openstack.org/#/c/384757/
[4] https://review.openstack.org/#/c/425154/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev