Re: [openstack-dev] [TripleO] [Heat] TripleO Heat Templates and merge.py
On 06/04/14 23:27, Steve Baker wrote: > On 05/04/14 04:47, Tomas Sedovic wrote: >> Hi All, >> >> > The maintenance burden of merge.py can be gradually reduced if features > in it can be deleted when they are no longer needed. At some point in > this process merge.py will need to accept HOT templates, and risk of > breakage during this changeover would be reduced the smaller merge.py is. > > How about this for the task order? > 1. remove OpenStack::ImageBuilder::Elements support from merge.py > 2. move to software-config based templates > 3. remove the following from merge.py >3.1. merging params and resources >3.2. FileInclude >3.3. OpenStack::Role > 4. port tripleo templates and merge.py to HOT > 5. use some HOT replacement for Merge::Map, delete Merge::Map from tripleo > 6. move to resource providers/scaling groups for scaling > 7. rm -f merge.py I like this. Clint's already working on #2. I can tackle #1 and help review & test the software config changes. We can deal with the rest afterwards. One note on 3.1: until we switch to provider resources or get_file, we can't drop the "merging params and resources" feature. We can drop FileInclude, OpenStack::Role and deep merge (e.g. joining `notCompute0Config` from `overcloud-source.yaml` and `swift-source.yaml` example from my email), but we'll have to keep the functionality of putting multiple templates together for a bit longer. That said, I don't think switching to provider resources is going to be a drastic change. > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] [Heat] TripleO Heat Templates and merge.py
On 04/06/2014 11:27 PM, Steve Baker wrote:
On 05/04/14 04:47, Tomas Sedovic wrote:
Hi All,
I was wondering if the time has come to document what exactly are we
doing with tripleo-heat-templates and merge.py[1], figure out what needs
to happen to move away and raise the necessary blueprints on Heat and
TripleO side.
(merge.py is a script we use to build the final TripleO Heat templates
from smaller chunks)
There probably isn't an immediate need for us to drop merge.py, but its
existence either indicates deficiencies within Heat or our unfamiliarity
with some of Heat's features (possibly both).
I worry that the longer we stay with merge.py the harder it will be to
move forward. We're still adding new features and fixing bugs in it (at
a slow pace but still).
Below is my understanding of the main marge.py functionality and a rough
plan of what I think might be a good direction to move to. It is almost
certainly incomplete -- please do poke holes in this. I'm hoping we'll
get to a point where everyone's clear on what exactly merge.py does and
why. We can then document that and raise the appropriate blueprints.
## merge.py features ##
1. Merging parameters and resources
Any uniquely-named parameters and resources from multiple templates are
put together into the final template.
If a resource of the same name is in multiple templates, an error is
raised. Unless it's of a whitelisted type (nova server, launch
configuration, etc.) in which case they're all merged into a single
resource.
For example: merge.py overcloud-source.yaml swift-source.yaml
The final template has all the parameters from both. Moreover, these two
resources will be joined together:
overcloud-source.yaml
notCompute0Config:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
ImageId: '0'
InstanceType: '0'
Metadata:
admin-password: {Ref: AdminPassword}
admin-token: {Ref: AdminToken}
bootstack:
public_interface_ip:
Ref: NeutronPublicInterfaceIP
swift-source.yaml
notCompute0Config:
Type: AWS::AutoScaling::LaunchConfiguration
Metadata:
swift:
devices:
...
hash: {Ref: SwiftHashSuffix}
service-password: {Ref: SwiftPassword}
The final template will contain:
notCompute0Config:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
ImageId: '0'
InstanceType: '0'
Metadata:
admin-password: {Ref: AdminPassword}
admin-token: {Ref: AdminToken}
bootstack:
public_interface_ip:
Ref: NeutronPublicInterfaceIP
swift:
devices:
...
hash: {Ref: SwiftHashSuffix}
service-password: {Ref: SwiftPassword}
We use this to keep the templates more manageable (instead of having one
huge file) and also to be able to pick the components we want: instead
of `undercloud-bm-source.yaml` we can pick `undercloud-vm-source` (which
uses the VirtualPowerManager driver) or `ironic-vm-source`.
2. FileInclude
If you have a pseudo resource with the type of `FileInclude`, we will
look at the specified Path and SubKey and put the resulting dictionary in:
overcloud-source.yaml
NovaCompute0Config:
Type: FileInclude
Path: nova-compute-instance.yaml
SubKey: Resources.NovaCompute0Config
Parameters:
NeutronNetworkType: "gre"
NeutronEnableTunnelling: "True"
nova-compute-instance.yaml
NovaCompute0Config:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
InstanceType: '0'
ImageId: '0'
Metadata:
keystone:
host: {Ref: KeystoneHost}
neutron:
host: {Ref: NeutronHost}
tenant_network_type: {Ref: NeutronNetworkType}
network_vlan_ranges: {Ref: NeutronNetworkVLANRanges}
bridge_mappings: {Ref: NeutronBridgeMappings}
enable_tunneling: {Ref: NeutronEnableTunnelling}
physical_bridge: {Ref: NeutronPhysicalBridge}
public_interface: {Ref: NeutronPublicInterface}
service-password:
Ref: NeutronPassword
admin-password: {Ref: AdminPassword}
The result:
NovaCompute0Config:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
InstanceType: '0'
ImageId: '0'
Metadata:
keystone:
host: {Ref: KeystoneHost}
neutron:
host: {Ref: NeutronHost}
tenant_network_type: "gre"
network_vlan_ranges: {Ref: NeutronNetworkVLANRanges}
bridge_mappings: {Ref: NeutronBridgeMappings}
enable_tunneling: "True"
physical_bridge: {Ref: NeutronPhysicalBridge}
public_interface: {Ref: NeutronPublicInterface}
service-password:
Ref: NeutronPassword
admin-password: {Ref: AdminPassword}
Note the `NeutronNetworkType` and `NeutronEnableTunneling` parameter
substitution.
This is usefu
Re: [openstack-dev] [TripleO] [Heat] TripleO Heat Templates and merge.py
On 07/04/14 00:27, Steve Baker wrote:
> On 05/04/14 04:47, Tomas Sedovic wrote:
>> Hi All,
>>
>> I was wondering if the time has come to document what exactly are we
>> doing with tripleo-heat-templates and merge.py[1], figure out what needs
>> to happen to move away and raise the necessary blueprints on Heat and
>> TripleO side.
>>
>> (merge.py is a script we use to build the final TripleO Heat templates
>> from smaller chunks)
>>
>> There probably isn't an immediate need for us to drop merge.py, but its
>> existence either indicates deficiencies within Heat or our unfamiliarity
>> with some of Heat's features (possibly both).
>>
>> I worry that the longer we stay with merge.py the harder it will be to
>> move forward. We're still adding new features and fixing bugs in it (at
>> a slow pace but still).
>>
>> Below is my understanding of the main marge.py functionality and a rough
>> plan of what I think might be a good direction to move to. It is almost
>> certainly incomplete -- please do poke holes in this. I'm hoping we'll
>> get to a point where everyone's clear on what exactly merge.py does and
>> why. We can then document that and raise the appropriate blueprints.
>>
>>
>> ## merge.py features ##
>>
>>
>> 1. Merging parameters and resources
>>
>> Any uniquely-named parameters and resources from multiple templates are
>> put together into the final template.
>>
>> If a resource of the same name is in multiple templates, an error is
>> raised. Unless it's of a whitelisted type (nova server, launch
>> configuration, etc.) in which case they're all merged into a single
>> resource.
>>
>> For example: merge.py overcloud-source.yaml swift-source.yaml
>>
>> The final template has all the parameters from both. Moreover, these two
>> resources will be joined together:
>>
>> overcloud-source.yaml
>>
>> notCompute0Config:
>> Type: AWS::AutoScaling::LaunchConfiguration
>> Properties:
>> ImageId: '0'
>> InstanceType: '0'
>> Metadata:
>> admin-password: {Ref: AdminPassword}
>> admin-token: {Ref: AdminToken}
>> bootstack:
>> public_interface_ip:
>> Ref: NeutronPublicInterfaceIP
>>
>>
>> swift-source.yaml
>>
>> notCompute0Config:
>> Type: AWS::AutoScaling::LaunchConfiguration
>> Metadata:
>> swift:
>> devices:
>> ...
>> hash: {Ref: SwiftHashSuffix}
>> service-password: {Ref: SwiftPassword}
>>
>>
>> The final template will contain:
>>
>> notCompute0Config:
>> Type: AWS::AutoScaling::LaunchConfiguration
>> Properties:
>> ImageId: '0'
>> InstanceType: '0'
>> Metadata:
>> admin-password: {Ref: AdminPassword}
>> admin-token: {Ref: AdminToken}
>> bootstack:
>> public_interface_ip:
>> Ref: NeutronPublicInterfaceIP
>> swift:
>> devices:
>> ...
>> hash: {Ref: SwiftHashSuffix}
>> service-password: {Ref: SwiftPassword}
>>
>>
>> We use this to keep the templates more manageable (instead of having one
>> huge file) and also to be able to pick the components we want: instead
>> of `undercloud-bm-source.yaml` we can pick `undercloud-vm-source` (which
>> uses the VirtualPowerManager driver) or `ironic-vm-source`.
>>
>>
>>
>> 2. FileInclude
>>
>> If you have a pseudo resource with the type of `FileInclude`, we will
>> look at the specified Path and SubKey and put the resulting dictionary in:
>>
>> overcloud-source.yaml
>>
>> NovaCompute0Config:
>> Type: FileInclude
>> Path: nova-compute-instance.yaml
>> SubKey: Resources.NovaCompute0Config
>> Parameters:
>> NeutronNetworkType: "gre"
>> NeutronEnableTunnelling: "True"
>>
>>
>> nova-compute-instance.yaml
>>
>> NovaCompute0Config:
>> Type: AWS::AutoScaling::LaunchConfiguration
>> Properties:
>> InstanceType: '0'
>> ImageId: '0'
>> Metadata:
>> keystone:
>> host: {Ref: KeystoneHost}
>> neutron:
>> host: {Ref: NeutronHost}
>> tenant_network_type: {Ref: NeutronNetworkType}
>> network_vlan_ranges: {Ref: NeutronNetworkVLANRanges}
>> bridge_mappings: {Ref: NeutronBridgeMappings}
>> enable_tunneling: {Ref: NeutronEnableTunnelling}
>> physical_bridge: {Ref: NeutronPhysicalBridge}
>> public_interface: {Ref: NeutronPublicInterface}
>> service-password:
>> Ref: NeutronPassword
>> admin-password: {Ref: AdminPassword}
>>
>> The result:
>>
>> NovaCompute0Config:
>> Type: AWS::AutoScaling::LaunchConfiguration
>> Properties:
>> InstanceType: '0'
>> ImageId: '0'
>> Metadata:
>> keystone:
>> host: {Ref: KeystoneHost}
>> neutron:
>> host: {Ref: NeutronHost}
>> tenant_network_type: "gre"
>> network_vlan_ranges: {Ref: NeutronNetworkVLANRanges}
>> bridge_mappings: {Ref: NeutronBridgeMappings}
>> enab
Re: [openstack-dev] [TripleO] [Heat] TripleO Heat Templates and merge.py
On 05/04/14 04:47, Tomas Sedovic wrote:
> Hi All,
>
> I was wondering if the time has come to document what exactly are we
> doing with tripleo-heat-templates and merge.py[1], figure out what needs
> to happen to move away and raise the necessary blueprints on Heat and
> TripleO side.
>
> (merge.py is a script we use to build the final TripleO Heat templates
> from smaller chunks)
>
> There probably isn't an immediate need for us to drop merge.py, but its
> existence either indicates deficiencies within Heat or our unfamiliarity
> with some of Heat's features (possibly both).
>
> I worry that the longer we stay with merge.py the harder it will be to
> move forward. We're still adding new features and fixing bugs in it (at
> a slow pace but still).
>
> Below is my understanding of the main marge.py functionality and a rough
> plan of what I think might be a good direction to move to. It is almost
> certainly incomplete -- please do poke holes in this. I'm hoping we'll
> get to a point where everyone's clear on what exactly merge.py does and
> why. We can then document that and raise the appropriate blueprints.
>
>
> ## merge.py features ##
>
>
> 1. Merging parameters and resources
>
> Any uniquely-named parameters and resources from multiple templates are
> put together into the final template.
>
> If a resource of the same name is in multiple templates, an error is
> raised. Unless it's of a whitelisted type (nova server, launch
> configuration, etc.) in which case they're all merged into a single
> resource.
>
> For example: merge.py overcloud-source.yaml swift-source.yaml
>
> The final template has all the parameters from both. Moreover, these two
> resources will be joined together:
>
> overcloud-source.yaml
>
> notCompute0Config:
> Type: AWS::AutoScaling::LaunchConfiguration
> Properties:
> ImageId: '0'
> InstanceType: '0'
> Metadata:
> admin-password: {Ref: AdminPassword}
> admin-token: {Ref: AdminToken}
> bootstack:
> public_interface_ip:
> Ref: NeutronPublicInterfaceIP
>
>
> swift-source.yaml
>
> notCompute0Config:
> Type: AWS::AutoScaling::LaunchConfiguration
> Metadata:
> swift:
> devices:
> ...
> hash: {Ref: SwiftHashSuffix}
> service-password: {Ref: SwiftPassword}
>
>
> The final template will contain:
>
> notCompute0Config:
> Type: AWS::AutoScaling::LaunchConfiguration
> Properties:
> ImageId: '0'
> InstanceType: '0'
> Metadata:
> admin-password: {Ref: AdminPassword}
> admin-token: {Ref: AdminToken}
> bootstack:
> public_interface_ip:
> Ref: NeutronPublicInterfaceIP
> swift:
> devices:
> ...
> hash: {Ref: SwiftHashSuffix}
> service-password: {Ref: SwiftPassword}
>
>
> We use this to keep the templates more manageable (instead of having one
> huge file) and also to be able to pick the components we want: instead
> of `undercloud-bm-source.yaml` we can pick `undercloud-vm-source` (which
> uses the VirtualPowerManager driver) or `ironic-vm-source`.
>
>
>
> 2. FileInclude
>
> If you have a pseudo resource with the type of `FileInclude`, we will
> look at the specified Path and SubKey and put the resulting dictionary in:
>
> overcloud-source.yaml
>
> NovaCompute0Config:
> Type: FileInclude
> Path: nova-compute-instance.yaml
> SubKey: Resources.NovaCompute0Config
> Parameters:
> NeutronNetworkType: "gre"
> NeutronEnableTunnelling: "True"
>
>
> nova-compute-instance.yaml
>
> NovaCompute0Config:
> Type: AWS::AutoScaling::LaunchConfiguration
> Properties:
> InstanceType: '0'
> ImageId: '0'
> Metadata:
> keystone:
> host: {Ref: KeystoneHost}
> neutron:
> host: {Ref: NeutronHost}
> tenant_network_type: {Ref: NeutronNetworkType}
> network_vlan_ranges: {Ref: NeutronNetworkVLANRanges}
> bridge_mappings: {Ref: NeutronBridgeMappings}
> enable_tunneling: {Ref: NeutronEnableTunnelling}
> physical_bridge: {Ref: NeutronPhysicalBridge}
> public_interface: {Ref: NeutronPublicInterface}
> service-password:
> Ref: NeutronPassword
> admin-password: {Ref: AdminPassword}
>
> The result:
>
> NovaCompute0Config:
> Type: AWS::AutoScaling::LaunchConfiguration
> Properties:
> InstanceType: '0'
> ImageId: '0'
> Metadata:
> keystone:
> host: {Ref: KeystoneHost}
> neutron:
> host: {Ref: NeutronHost}
> tenant_network_type: "gre"
> network_vlan_ranges: {Ref: NeutronNetworkVLANRanges}
> bridge_mappings: {Ref: NeutronBridgeMappings}
> enable_tunneling: "True"
> physical_bridge: {Ref: NeutronPhysicalBridge}
> public_interface: {Ref: NeutronPublicInterface}
> service-password:
> Ref: Neutron
Re: [openstack-dev] [TripleO] [Heat] TripleO Heat Templates and merge.py
On 04/04/14 13:58, Clint Byrum wrote: >We could keep roughly the same structure: a separate template for each >OpenStack service (compute, block storage, object storage, ironic, nova >baremetal). We would then use Heat environments to treat each of these >templates as a custom resource (e.g. OS::TripleO::Nova, >OS::TripleO::Swift, etc.). > I've never fully embraced providers for composition. Perhaps I've missed that as a key feature. An example of this would be helpful. I think if we deprecated all of merge.py except the "merge unique params and resources into one template" part, we could probably just start using nested stacks for that and drop merge.py. However, I'm not a huge fan of nested stacks as they are a bit clunky. Maybe providers would make that better? Anyway, I think I need to see how this would actually work before I can really grasp it. AIUI this use case is pretty much a canonical example of where you'd want to use providers. You have a server that you would like to treat as just a server, but can't because it comes with a WaitCondition and a random string generator (or whatever) into the bargain. So you group those resources together into a provider template behind a server-like facade, and just treat them like the single server you'd prefer them to be. This could actually be a big win where you're creating multiple ones with a similar configuration, because you can parametrise it and move it inside the template and then you only need to specify the custom parts rather than repeat the whole declaration when you add more resources in the same "group". From there moving into scaling groups when the time comes should be trivial. I'm actually pushing for the autoscaling code to literally use the providers mechanism to implement scaling of stacks, but the ResourceGroup does something basically equivalent too - splitting your scaling unit into a separate template is in all cases the first step. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] [Heat] TripleO Heat Templates and merge.py
Excerpts from Tomas Sedovic's message of 2014-04-04 08:47:46 -0700:
> Hi All,
>
> I was wondering if the time has come to document what exactly are we
> doing with tripleo-heat-templates and merge.py[1], figure out what needs
> to happen to move away and raise the necessary blueprints on Heat and
> TripleO side.
>
Yes indeed, it is time.
> (merge.py is a script we use to build the final TripleO Heat templates
> from smaller chunks)
>
> There probably isn't an immediate need for us to drop merge.py, but its
> existence either indicates deficiencies within Heat or our unfamiliarity
> with some of Heat's features (possibly both).
>
> I worry that the longer we stay with merge.py the harder it will be to
> move forward. We're still adding new features and fixing bugs in it (at
> a slow pace but still).
>
Merge.py is where we've amassed our debt. We'll pay it back by moving
features into Heat. A huge debt payment is coming in the form of
software config migration, which you mention at the bottom of this
message.
> Below is my understanding of the main marge.py functionality and a rough
> plan of what I think might be a good direction to move to. It is almost
> certainly incomplete -- please do poke holes in this. I'm hoping we'll
> get to a point where everyone's clear on what exactly merge.py does and
> why. We can then document that and raise the appropriate blueprints.
>
>
> ## merge.py features ##
>
>
> 1. Merging parameters and resources
>
> Any uniquely-named parameters and resources from multiple templates are
> put together into the final template.
>
> If a resource of the same name is in multiple templates, an error is
> raised. Unless it's of a whitelisted type (nova server, launch
> configuration, etc.) in which case they're all merged into a single
> resource.
>
> For example: merge.py overcloud-source.yaml swift-source.yaml
>
> The final template has all the parameters from both. Moreover, these two
> resources will be joined together:
>
> overcloud-source.yaml
>
> notCompute0Config:
> Type: AWS::AutoScaling::LaunchConfiguration
> Properties:
> ImageId: '0'
> InstanceType: '0'
> Metadata:
> admin-password: {Ref: AdminPassword}
> admin-token: {Ref: AdminToken}
> bootstack:
> public_interface_ip:
> Ref: NeutronPublicInterfaceIP
>
>
> swift-source.yaml
>
> notCompute0Config:
> Type: AWS::AutoScaling::LaunchConfiguration
> Metadata:
> swift:
> devices:
> ...
> hash: {Ref: SwiftHashSuffix}
> service-password: {Ref: SwiftPassword}
>
>
> The final template will contain:
>
> notCompute0Config:
> Type: AWS::AutoScaling::LaunchConfiguration
> Properties:
> ImageId: '0'
> InstanceType: '0'
> Metadata:
> admin-password: {Ref: AdminPassword}
> admin-token: {Ref: AdminToken}
> bootstack:
> public_interface_ip:
> Ref: NeutronPublicInterfaceIP
> swift:
> devices:
> ...
> hash: {Ref: SwiftHashSuffix}
> service-password: {Ref: SwiftPassword}
>
>
> We use this to keep the templates more manageable (instead of having one
> huge file) and also to be able to pick the components we want: instead
> of `undercloud-bm-source.yaml` we can pick `undercloud-vm-source` (which
> uses the VirtualPowerManager driver) or `ironic-vm-source`.
>
The merging of white-listed types is superseded entirely by
OS::Heat::StructuredConfig and OS::Heat::StructuredDeployment. I would
move that we replace all uses of it with those, and deprecate the
feature.
>
>
> 2. FileInclude
>
> If you have a pseudo resource with the type of `FileInclude`, we will
> look at the specified Path and SubKey and put the resulting dictionary in:
>
> overcloud-source.yaml
>
> NovaCompute0Config:
> Type: FileInclude
> Path: nova-compute-instance.yaml
> SubKey: Resources.NovaCompute0Config
> Parameters:
> NeutronNetworkType: "gre"
> NeutronEnableTunnelling: "True"
>
>
> nova-compute-instance.yaml
>
> NovaCompute0Config:
> Type: AWS::AutoScaling::LaunchConfiguration
> Properties:
> InstanceType: '0'
> ImageId: '0'
> Metadata:
> keystone:
> host: {Ref: KeystoneHost}
> neutron:
> host: {Ref: NeutronHost}
> tenant_network_type: {Ref: NeutronNetworkType}
> network_vlan_ranges: {Ref: NeutronNetworkVLANRanges}
> bridge_mappings: {Ref: NeutronBridgeMappings}
> enable_tunneling: {Ref: NeutronEnableTunnelling}
> physical_bridge: {Ref: NeutronPhysicalBridge}
> public_interface: {Ref: NeutronPublicInterface}
> service-password:
> Ref: NeutronPassword
> admin-password: {Ref: AdminPassword}
>
> The result:
>
> NovaCompute0Config:
> Type: AWS::AutoScaling::LaunchConfiguration
> Properties:
> InstanceType: '0'
>
