Re: [openstack-dev] [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

[Flavio Percoco]

On 15/06/17 13:06 -0400, Emilien Macchi wrote:

I missed [tripleo] tag.

On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi  wrote:

If you haven't followed the "Configuration management with etcd /
confd" thread [1], Doug found out that using confd to generate
configuration files wouldn't work for the Cinder case where we don't
know in advance of the deployment what settings to tell confd to look
at.
We are still looking for a generic way to generate *.conf files for
OpenStack, that would be usable by Deployment tools and operators.
Right now, Doug and I are investigating some tooling that would be
useful to achieve this goal.

Doug has prototyped an Ansible role that would generate configuration
files by consumming 2 things:

* Configuration schema, generated by Ben's work with Machine Readable
Sample Config.
  $ oslo-config-generator --namespace cinder --format yaml > cinder-schema.yaml

It also needs: https://review.openstack.org/#/c/474306/ to generate
some extra data not included in the original version.

* Parameters values provided in config_data directly in the playbook:
   config_data:
 DEFAULT:
   transport_url: rabbit://user:password@hostname
   verbose: true

There are 2 options disabled by default but which would be useful for
production environments:
* Set to true to always show all configuration values: config_show_defaults
* Set to true to show the help text: config_show_help: true

The Ansible module is available on github:
https://github.com/dhellmann/oslo-config-ansible

To try this out, just run:
  $ ansible-playbook ./playbook.yml

You can quickly see the output of cinder.conf:
https://clbin.com/HmS58


What are the next steps:

* Getting feedback from Deployment Tools and operators on the concept
of this module.
  Maybe this module could replace what is done by Kolla with
merge_configs and OpenStack Ansible with config_template.
* On the TripleO side, we would like to see if this module could
replace the Puppet OpenStack modules that are now mostly used for
generating configuration files for containers.
  A transition path would be having Heat to generate Ansible vars
files and give it to this module. We could integrate the playbook into
a new task in the composable services, something like
  "os_gen_config_tasks", a bit like we already have for upgrade tasks,
also driven by Ansible.
* Another similar option to what Doug did is to write a standalone
tool that would generate configuration, and for Ansible users we would
write a new module to use this tool.
  Example:
  Step 1. oslo-config-generator --namespace cinder --format yaml >
cinder-schema.yaml (note this tool already exists)
  Step 2. Create config_data.yaml in a specific format with
parameters values for what we want to configure (note this format
doesn't exist yet but look at what Doug did in the role, we could use
the same kind of schema).
  Step 3. oslo-gen-config -i config_data.yaml -s schema.yaml >
cinder.conf (note this tool doesn't exist yet)

  For Ansible users, we would write an Ansible module that would
take in entry 2 files: the schema and the data. The module would just
run the tool provided by oslo.config.
  Example:
  - name: Generate cinder.conf
oslo-gen-config: schema=cinder-schema.yaml
   data=config_data.yaml



I finally caught up with this thread and got the time to get back to y'all.
Sorry.

I like the roles version more because it's flexible and easier to distribute. We
can upload it to galaxy, package it, etc. Distributing ansible modules is a bit
painful right now and you end up adding them as roles in the playbook for the
modules to be loaded.

I'm about to work on a prototype and I'll use option #1 and perhaps we can
discuss further the option #2.

Flavio

--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

[Bogdan Dobrelya]

On 20.06.2017 17:27, Michał Jastrzębski wrote:
> On 19 June 2017 at 06:05, Doug Hellmann  wrote:
>> Excerpts from Michał Jastrzębski's message of 2017-06-16 15:50:54 -0700:
>>> So I'm trying to figure out how to actually use it.
>>>
>>> We (and any other container based deploy..) will run into some
>>> chicken/egg problem - you need to deploy container to generate big
>>> yaml with defaults, then you need to overload it with your
>>
>> The config schema file (the "big YAML with defaults") should be part of
>> the packaged software, so the deployment tool shouldn't need to generate
>> it unless you're handling drivers that are not included in tree.
> 
> Right that's what I was missing, I guess we can generate these during
> buildtime without big issues, then it will be embedded into container,
> shouldn't be too hard change and would work for both source and
> binary.
>>> configurations, validate if they're not deprecated, run container with
>>
>> It doesn't do it today, but the thing that converts the input data to
>> the INI file could automatically translate old option names to their new
>> names.
>>
>>> this ansible role (or module...really doesn't matter), spit out final
>>
>> Why does the config file need to be generated inside a container?
> 
> Outside of container you don't have oslo or nova (python libs), so to
> get access to these you need to do it inside container.

That could be another container I suppose. Like those containers used
for build deps, there could be as well a container for config management
deps. Docker multi-stage [0] could help to achieve that smooth, w/o
impacting the service containers.

> 
>>> confg, lay it down, deploy container again. And that will have to be
>>> done for every host class (as configs might differ host to host). Imho
>>> a bit too much for this to be appealing (but I might be wrong). I'd
>>> much rather have:
>>> 1. Yaml as input to oslo.config instead of broken ini
>>
>> I'm not opposed to switching to YAML, but it's a bit more involved than
>> just adding support in the parser. All of the work that has been done on
>> generating sample default files and documentation needs to be updated to
>> support YAML. We need a migration path to move everyone from INI to
>> YAML. And we need to update devstack and all of its plugins to edit the
>> new file format. There are probably more tasks involved in the
>> migration. I'm dealing with a couple of other projects right now, and
>> don't have time to plan all of that out myself. If someone else wants to
>> pick it up, I can help with reviews on the spec and code changes.
> 
> Switching is a big no, everyone would hate us with emotion pure as
> mountain spring water. It's to support both at same time, which makes
> it slightly more complex. We could make full switch after few releases
> of deprecation I guess. Anyway, agree, lots of work.
> 


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

[Michał Jastrzębski]

On 19 June 2017 at 06:05, Doug Hellmann  wrote:
> Excerpts from Michał Jastrzębski's message of 2017-06-16 15:50:54 -0700:
>> So I'm trying to figure out how to actually use it.
>>
>> We (and any other container based deploy..) will run into some
>> chicken/egg problem - you need to deploy container to generate big
>> yaml with defaults, then you need to overload it with your
>
> The config schema file (the "big YAML with defaults") should be part of
> the packaged software, so the deployment tool shouldn't need to generate
> it unless you're handling drivers that are not included in tree.

Right that's what I was missing, I guess we can generate these during
buildtime without big issues, then it will be embedded into container,
shouldn't be too hard change and would work for both source and
binary.
>> configurations, validate if they're not deprecated, run container with
>
> It doesn't do it today, but the thing that converts the input data to
> the INI file could automatically translate old option names to their new
> names.
>
>> this ansible role (or module...really doesn't matter), spit out final
>
> Why does the config file need to be generated inside a container?

Outside of container you don't have oslo or nova (python libs), so to
get access to these you need to do it inside container.

>> confg, lay it down, deploy container again. And that will have to be
>> done for every host class (as configs might differ host to host). Imho
>> a bit too much for this to be appealing (but I might be wrong). I'd
>> much rather have:
>> 1. Yaml as input to oslo.config instead of broken ini
>
> I'm not opposed to switching to YAML, but it's a bit more involved than
> just adding support in the parser. All of the work that has been done on
> generating sample default files and documentation needs to be updated to
> support YAML. We need a migration path to move everyone from INI to
> YAML. And we need to update devstack and all of its plugins to edit the
> new file format. There are probably more tasks involved in the
> migration. I'm dealing with a couple of other projects right now, and
> don't have time to plan all of that out myself. If someone else wants to
> pick it up, I can help with reviews on the spec and code changes.

Switching is a big no, everyone would hate us with emotion pure as
mountain spring water. It's to support both at same time, which makes
it slightly more complex. We could make full switch after few releases
of deprecation I guess. Anyway, agree, lots of work.

>
>> 2. Validator to throw an error if one of our regular,
>> template-rendered, configs is deprecated
>>
>> We can run this validator in gate to have quick feedback when
>> something gets deprecated.
>>
>> Thoughts?
>> Michal
>>
>> On 16 June 2017 at 13:24, Emilien Macchi  wrote:
>> > On Fri, Jun 16, 2017 at 11:09 AM, Jiří Stránský  wrote:
>> >> On 15.6.2017 19:06, Emilien Macchi wrote:
>> >>>
>> >>> I missed [tripleo] tag.
>> >>>
>> >>> On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi 
>> >>> wrote:
>> 
>>  If you haven't followed the "Configuration management with etcd /
>>  confd" thread [1], Doug found out that using confd to generate
>>  configuration files wouldn't work for the Cinder case where we don't
>>  know in advance of the deployment what settings to tell confd to look
>>  at.
>>  We are still looking for a generic way to generate *.conf files for
>>  OpenStack, that would be usable by Deployment tools and operators.
>>  Right now, Doug and I are investigating some tooling that would be
>>  useful to achieve this goal.
>> 
>>  Doug has prototyped an Ansible role that would generate configuration
>>  files by consumming 2 things:
>> 
>>  * Configuration schema, generated by Ben's work with Machine Readable
>>  Sample Config.
>> $ oslo-config-generator --namespace cinder --format yaml >
>>  cinder-schema.yaml
>> 
>>  It also needs: https://review.openstack.org/#/c/474306/ to generate
>>  some extra data not included in the original version.
>> 
>>  * Parameters values provided in config_data directly in the playbook:
>>  config_data:
>>    DEFAULT:
>>  transport_url: rabbit://user:password@hostname
>>  verbose: true
>> 
>>  There are 2 options disabled by default but which would be useful for
>>  production environments:
>>  * Set to true to always show all configuration values:
>>  config_show_defaults
>>  * Set to true to show the help text: config_show_help: true
>> 
>>  The Ansible module is available on github:
>>  https://github.com/dhellmann/oslo-config-ansible
>> 
>>  To try this out, just run:
>> $ ansible-playbook ./playbook.yml
>> 
>>  You can quickly see the output of cinder.conf:
>>   https://clbin.com/HmS58
>> 
>> 

Re: [openstack-dev] [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

[Doug Hellmann]

Excerpts from Michał Jastrzębski's message of 2017-06-16 15:50:54 -0700:
> So I'm trying to figure out how to actually use it.
> 
> We (and any other container based deploy..) will run into some
> chicken/egg problem - you need to deploy container to generate big
> yaml with defaults, then you need to overload it with your

The config schema file (the "big YAML with defaults") should be part of
the packaged software, so the deployment tool shouldn't need to generate
it unless you're handling drivers that are not included in tree.

> configurations, validate if they're not deprecated, run container with

It doesn't do it today, but the thing that converts the input data to
the INI file could automatically translate old option names to their new
names.

> this ansible role (or module...really doesn't matter), spit out final

Why does the config file need to be generated inside a container?

> confg, lay it down, deploy container again. And that will have to be
> done for every host class (as configs might differ host to host). Imho
> a bit too much for this to be appealing (but I might be wrong). I'd
> much rather have:
> 1. Yaml as input to oslo.config instead of broken ini

I'm not opposed to switching to YAML, but it's a bit more involved than
just adding support in the parser. All of the work that has been done on
generating sample default files and documentation needs to be updated to
support YAML. We need a migration path to move everyone from INI to
YAML. And we need to update devstack and all of its plugins to edit the
new file format. There are probably more tasks involved in the
migration. I'm dealing with a couple of other projects right now, and
don't have time to plan all of that out myself. If someone else wants to
pick it up, I can help with reviews on the spec and code changes.

> 2. Validator to throw an error if one of our regular,
> template-rendered, configs is deprecated
> 
> We can run this validator in gate to have quick feedback when
> something gets deprecated.
> 
> Thoughts?
> Michal
> 
> On 16 June 2017 at 13:24, Emilien Macchi  wrote:
> > On Fri, Jun 16, 2017 at 11:09 AM, Jiří Stránský  wrote:
> >> On 15.6.2017 19:06, Emilien Macchi wrote:
> >>>
> >>> I missed [tripleo] tag.
> >>>
> >>> On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi 
> >>> wrote:
> 
>  If you haven't followed the "Configuration management with etcd /
>  confd" thread [1], Doug found out that using confd to generate
>  configuration files wouldn't work for the Cinder case where we don't
>  know in advance of the deployment what settings to tell confd to look
>  at.
>  We are still looking for a generic way to generate *.conf files for
>  OpenStack, that would be usable by Deployment tools and operators.
>  Right now, Doug and I are investigating some tooling that would be
>  useful to achieve this goal.
> 
>  Doug has prototyped an Ansible role that would generate configuration
>  files by consumming 2 things:
> 
>  * Configuration schema, generated by Ben's work with Machine Readable
>  Sample Config.
> $ oslo-config-generator --namespace cinder --format yaml >
>  cinder-schema.yaml
> 
>  It also needs: https://review.openstack.org/#/c/474306/ to generate
>  some extra data not included in the original version.
> 
>  * Parameters values provided in config_data directly in the playbook:
>  config_data:
>    DEFAULT:
>  transport_url: rabbit://user:password@hostname
>  verbose: true
> 
>  There are 2 options disabled by default but which would be useful for
>  production environments:
>  * Set to true to always show all configuration values:
>  config_show_defaults
>  * Set to true to show the help text: config_show_help: true
> 
>  The Ansible module is available on github:
>  https://github.com/dhellmann/oslo-config-ansible
> 
>  To try this out, just run:
> $ ansible-playbook ./playbook.yml
> 
>  You can quickly see the output of cinder.conf:
>   https://clbin.com/HmS58
> 
> 
>  What are the next steps:
> 
>  * Getting feedback from Deployment Tools and operators on the concept
>  of this module.
> Maybe this module could replace what is done by Kolla with
>  merge_configs and OpenStack Ansible with config_template.
>  * On the TripleO side, we would like to see if this module could
>  replace the Puppet OpenStack modules that are now mostly used for
>  generating configuration files for containers.
> A transition path would be having Heat to generate Ansible vars
>  files and give it to this module. We could integrate the playbook into
>  a new task in the composable services, something like
> "os_gen_config_tasks", a bit like we already have for upgrade tasks,
>  also driven 

Re: [openstack-dev] [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

[Michał Jastrzębski]

So I'm trying to figure out how to actually use it.

We (and any other container based deploy..) will run into some
chicken/egg problem - you need to deploy container to generate big
yaml with defaults, then you need to overload it with your
configurations, validate if they're not deprecated, run container with
this ansible role (or module...really doesn't matter), spit out final
confg, lay it down, deploy container again. And that will have to be
done for every host class (as configs might differ host to host). Imho
a bit too much for this to be appealing (but I might be wrong). I'd
much rather have:
1. Yaml as input to oslo.config instead of broken ini
2. Validator to throw an error if one of our regular,
template-rendered, configs is deprecated

We can run this validator in gate to have quick feedback when
something gets deprecated.

Thoughts?
Michal

On 16 June 2017 at 13:24, Emilien Macchi  wrote:
> On Fri, Jun 16, 2017 at 11:09 AM, Jiří Stránský  wrote:
>> On 15.6.2017 19:06, Emilien Macchi wrote:
>>>
>>> I missed [tripleo] tag.
>>>
>>> On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi 
>>> wrote:

 If you haven't followed the "Configuration management with etcd /
 confd" thread [1], Doug found out that using confd to generate
 configuration files wouldn't work for the Cinder case where we don't
 know in advance of the deployment what settings to tell confd to look
 at.
 We are still looking for a generic way to generate *.conf files for
 OpenStack, that would be usable by Deployment tools and operators.
 Right now, Doug and I are investigating some tooling that would be
 useful to achieve this goal.

 Doug has prototyped an Ansible role that would generate configuration
 files by consumming 2 things:

 * Configuration schema, generated by Ben's work with Machine Readable
 Sample Config.
$ oslo-config-generator --namespace cinder --format yaml >
 cinder-schema.yaml

 It also needs: https://review.openstack.org/#/c/474306/ to generate
 some extra data not included in the original version.

 * Parameters values provided in config_data directly in the playbook:
 config_data:
   DEFAULT:
 transport_url: rabbit://user:password@hostname
 verbose: true

 There are 2 options disabled by default but which would be useful for
 production environments:
 * Set to true to always show all configuration values:
 config_show_defaults
 * Set to true to show the help text: config_show_help: true

 The Ansible module is available on github:
 https://github.com/dhellmann/oslo-config-ansible

 To try this out, just run:
$ ansible-playbook ./playbook.yml

 You can quickly see the output of cinder.conf:
  https://clbin.com/HmS58


 What are the next steps:

 * Getting feedback from Deployment Tools and operators on the concept
 of this module.
Maybe this module could replace what is done by Kolla with
 merge_configs and OpenStack Ansible with config_template.
 * On the TripleO side, we would like to see if this module could
 replace the Puppet OpenStack modules that are now mostly used for
 generating configuration files for containers.
A transition path would be having Heat to generate Ansible vars
 files and give it to this module. We could integrate the playbook into
 a new task in the composable services, something like
"os_gen_config_tasks", a bit like we already have for upgrade tasks,
 also driven by Ansible.
>>
>>
>> This sounds good to me, though one issue i can presently see is that Puppet
>> modules sometimes contain quite a bit of data processing logic ("smart"
>> variables which map 1-to-N rather than 1-to-1 to actual config values, and
>> often not just in openstack service configs, e.g. puppet-nova also
>> configures libvirt, etc.). Also we use some non-config aspects from the
>> Puppet modules (e.g. seeding Keystone tenants/services/endpoints/...). We'd
>> need to implement this functionality elsewhere when replacing the Puppet
>> modules. Not a blocker, but something to keep in mind.
>
> 2 interesting things:
>
> - For the logic that are done by puppet modules for some parameters:
> yes I agree, this problem isn't solved now. This thread talks about
> config management, with some data in entry, it's a very little step I
> know, but it's on purpose.
>   Once we figure how to do that, we can think about the data
> generation and where to put the logic (I think the logic is too
> opinionated to be in a common project, but I might be wrong).
>
> - Things like libvirt, mysql, etc will be managed by something else
> but Puppet I think; this is out of topic for now. For Keystone
> resources, same thing, we could use some native python clients or
> Ansible modules if we switch to 

Re: [openstack-dev] [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

[Emilien Macchi]

On Fri, Jun 16, 2017 at 11:09 AM, Jiří Stránský  wrote:
> On 15.6.2017 19:06, Emilien Macchi wrote:
>>
>> I missed [tripleo] tag.
>>
>> On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi 
>> wrote:
>>>
>>> If you haven't followed the "Configuration management with etcd /
>>> confd" thread [1], Doug found out that using confd to generate
>>> configuration files wouldn't work for the Cinder case where we don't
>>> know in advance of the deployment what settings to tell confd to look
>>> at.
>>> We are still looking for a generic way to generate *.conf files for
>>> OpenStack, that would be usable by Deployment tools and operators.
>>> Right now, Doug and I are investigating some tooling that would be
>>> useful to achieve this goal.
>>>
>>> Doug has prototyped an Ansible role that would generate configuration
>>> files by consumming 2 things:
>>>
>>> * Configuration schema, generated by Ben's work with Machine Readable
>>> Sample Config.
>>>$ oslo-config-generator --namespace cinder --format yaml >
>>> cinder-schema.yaml
>>>
>>> It also needs: https://review.openstack.org/#/c/474306/ to generate
>>> some extra data not included in the original version.
>>>
>>> * Parameters values provided in config_data directly in the playbook:
>>> config_data:
>>>   DEFAULT:
>>> transport_url: rabbit://user:password@hostname
>>> verbose: true
>>>
>>> There are 2 options disabled by default but which would be useful for
>>> production environments:
>>> * Set to true to always show all configuration values:
>>> config_show_defaults
>>> * Set to true to show the help text: config_show_help: true
>>>
>>> The Ansible module is available on github:
>>> https://github.com/dhellmann/oslo-config-ansible
>>>
>>> To try this out, just run:
>>>$ ansible-playbook ./playbook.yml
>>>
>>> You can quickly see the output of cinder.conf:
>>>  https://clbin.com/HmS58
>>>
>>>
>>> What are the next steps:
>>>
>>> * Getting feedback from Deployment Tools and operators on the concept
>>> of this module.
>>>Maybe this module could replace what is done by Kolla with
>>> merge_configs and OpenStack Ansible with config_template.
>>> * On the TripleO side, we would like to see if this module could
>>> replace the Puppet OpenStack modules that are now mostly used for
>>> generating configuration files for containers.
>>>A transition path would be having Heat to generate Ansible vars
>>> files and give it to this module. We could integrate the playbook into
>>> a new task in the composable services, something like
>>>"os_gen_config_tasks", a bit like we already have for upgrade tasks,
>>> also driven by Ansible.
>
>
> This sounds good to me, though one issue i can presently see is that Puppet
> modules sometimes contain quite a bit of data processing logic ("smart"
> variables which map 1-to-N rather than 1-to-1 to actual config values, and
> often not just in openstack service configs, e.g. puppet-nova also
> configures libvirt, etc.). Also we use some non-config aspects from the
> Puppet modules (e.g. seeding Keystone tenants/services/endpoints/...). We'd
> need to implement this functionality elsewhere when replacing the Puppet
> modules. Not a blocker, but something to keep in mind.

2 interesting things:

- For the logic that are done by puppet modules for some parameters:
yes I agree, this problem isn't solved now. This thread talks about
config management, with some data in entry, it's a very little step I
know, but it's on purpose.
  Once we figure how to do that, we can think about the data
generation and where to put the logic (I think the logic is too
opinionated to be in a common project, but I might be wrong).

- Things like libvirt, mysql, etc will be managed by something else
but Puppet I think; this is out of topic for now. For Keystone
resources, same thing, we could use some native python clients or
Ansible modules if we switch to Ansible, etc.

Again, topic is really on "give me an ini file".

>>> * Another similar option to what Doug did is to write a standalone
>>> tool that would generate configuration, and for Ansible users we would
>>> write a new module to use this tool.
>>>Example:
>>>Step 1. oslo-config-generator --namespace cinder --format yaml >
>>> cinder-schema.yaml (note this tool already exists)
>>>Step 2. Create config_data.yaml in a specific format with
>>> parameters values for what we want to configure (note this format
>>> doesn't exist yet but look at what Doug did in the role, we could use
>>> the same kind of schema).
>>>Step 3. oslo-gen-config -i config_data.yaml -s schema.yaml >
>>> cinder.conf (note this tool doesn't exist yet)
>
>
> +1 on standalone tool which can be used in different contexts (by different
> higher level tools), this sounds generally useful.

Ack, good feedback.

>>>
>>>For Ansible users, we would write an Ansible module that would
>>> take in entry 2 files: the schema and 

Re: [openstack-dev] [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

[Jiří Stránský]

On 15.6.2017 19:06, Emilien Macchi wrote:

I missed [tripleo] tag.

On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi  wrote:

If you haven't followed the "Configuration management with etcd /
confd" thread [1], Doug found out that using confd to generate
configuration files wouldn't work for the Cinder case where we don't
know in advance of the deployment what settings to tell confd to look
at.
We are still looking for a generic way to generate *.conf files for
OpenStack, that would be usable by Deployment tools and operators.
Right now, Doug and I are investigating some tooling that would be
useful to achieve this goal.

Doug has prototyped an Ansible role that would generate configuration
files by consumming 2 things:

* Configuration schema, generated by Ben's work with Machine Readable
Sample Config.
   $ oslo-config-generator --namespace cinder --format yaml > cinder-schema.yaml

It also needs: https://review.openstack.org/#/c/474306/ to generate
some extra data not included in the original version.

* Parameters values provided in config_data directly in the playbook:
config_data:
  DEFAULT:
transport_url: rabbit://user:password@hostname
verbose: true

There are 2 options disabled by default but which would be useful for
production environments:
* Set to true to always show all configuration values: config_show_defaults
* Set to true to show the help text: config_show_help: true

The Ansible module is available on github:
https://github.com/dhellmann/oslo-config-ansible

To try this out, just run:
   $ ansible-playbook ./playbook.yml

You can quickly see the output of cinder.conf:
 https://clbin.com/HmS58


What are the next steps:

* Getting feedback from Deployment Tools and operators on the concept
of this module.
   Maybe this module could replace what is done by Kolla with
merge_configs and OpenStack Ansible with config_template.
* On the TripleO side, we would like to see if this module could
replace the Puppet OpenStack modules that are now mostly used for
generating configuration files for containers.
   A transition path would be having Heat to generate Ansible vars
files and give it to this module. We could integrate the playbook into
a new task in the composable services, something like
   "os_gen_config_tasks", a bit like we already have for upgrade tasks,
also driven by Ansible.


This sounds good to me, though one issue i can presently see is that 
Puppet modules sometimes contain quite a bit of data processing logic 
("smart" variables which map 1-to-N rather than 1-to-1 to actual config 
values, and often not just in openstack service configs, e.g. 
puppet-nova also configures libvirt, etc.). Also we use some non-config 
aspects from the Puppet modules (e.g. seeding Keystone 
tenants/services/endpoints/...). We'd need to implement this 
functionality elsewhere when replacing the Puppet modules. Not a 
blocker, but something to keep in mind.



* Another similar option to what Doug did is to write a standalone
tool that would generate configuration, and for Ansible users we would
write a new module to use this tool.
   Example:
   Step 1. oslo-config-generator --namespace cinder --format yaml >
cinder-schema.yaml (note this tool already exists)
   Step 2. Create config_data.yaml in a specific format with
parameters values for what we want to configure (note this format
doesn't exist yet but look at what Doug did in the role, we could use
the same kind of schema).
   Step 3. oslo-gen-config -i config_data.yaml -s schema.yaml >
cinder.conf (note this tool doesn't exist yet)


+1 on standalone tool which can be used in different contexts (by 
different higher level tools), this sounds generally useful.




   For Ansible users, we would write an Ansible module that would
take in entry 2 files: the schema and the data. The module would just
run the tool provided by oslo.config.
   Example:
   - name: Generate cinder.conf
 oslo-gen-config: schema=cinder-schema.yaml
data=config_data.yaml


+1 for module rather than a role. "Take these inputs and produce that 
output" fits the module semantics better than role semantics IMO.


FWIW as i see it right now, this ^^ + ConfigMaps + immutable-config 
containers could result in a nicer/safer/more-debuggable containerized 
OpenStack setup than etcd + confd in daemon mode + mutable-config 
containers.





Please bring feedback and thoughts, it's really important to know what
folks from Installers think about this idea; again the ultimate goal
is to provide a reference tool to generate configuration in OpenStack,
in a way that scales and is friendly for our operators.

Thanks,

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-June/118176.html
--
Emilien Macchi






Have a good day,

Jirka

__
OpenStack Development Mailing List (not for usage 

[openstack-dev] [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

[Emilien Macchi]

I missed [tripleo] tag.

On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi  wrote:
> If you haven't followed the "Configuration management with etcd /
> confd" thread [1], Doug found out that using confd to generate
> configuration files wouldn't work for the Cinder case where we don't
> know in advance of the deployment what settings to tell confd to look
> at.
> We are still looking for a generic way to generate *.conf files for
> OpenStack, that would be usable by Deployment tools and operators.
> Right now, Doug and I are investigating some tooling that would be
> useful to achieve this goal.
>
> Doug has prototyped an Ansible role that would generate configuration
> files by consumming 2 things:
>
> * Configuration schema, generated by Ben's work with Machine Readable
> Sample Config.
>   $ oslo-config-generator --namespace cinder --format yaml > 
> cinder-schema.yaml
>
> It also needs: https://review.openstack.org/#/c/474306/ to generate
> some extra data not included in the original version.
>
> * Parameters values provided in config_data directly in the playbook:
>config_data:
>  DEFAULT:
>transport_url: rabbit://user:password@hostname
>verbose: true
>
> There are 2 options disabled by default but which would be useful for
> production environments:
> * Set to true to always show all configuration values: config_show_defaults
> * Set to true to show the help text: config_show_help: true
>
> The Ansible module is available on github:
> https://github.com/dhellmann/oslo-config-ansible
>
> To try this out, just run:
>   $ ansible-playbook ./playbook.yml
>
> You can quickly see the output of cinder.conf:
> https://clbin.com/HmS58
>
>
> What are the next steps:
>
> * Getting feedback from Deployment Tools and operators on the concept
> of this module.
>   Maybe this module could replace what is done by Kolla with
> merge_configs and OpenStack Ansible with config_template.
> * On the TripleO side, we would like to see if this module could
> replace the Puppet OpenStack modules that are now mostly used for
> generating configuration files for containers.
>   A transition path would be having Heat to generate Ansible vars
> files and give it to this module. We could integrate the playbook into
> a new task in the composable services, something like
>   "os_gen_config_tasks", a bit like we already have for upgrade tasks,
> also driven by Ansible.
> * Another similar option to what Doug did is to write a standalone
> tool that would generate configuration, and for Ansible users we would
> write a new module to use this tool.
>   Example:
>   Step 1. oslo-config-generator --namespace cinder --format yaml >
> cinder-schema.yaml (note this tool already exists)
>   Step 2. Create config_data.yaml in a specific format with
> parameters values for what we want to configure (note this format
> doesn't exist yet but look at what Doug did in the role, we could use
> the same kind of schema).
>   Step 3. oslo-gen-config -i config_data.yaml -s schema.yaml >
> cinder.conf (note this tool doesn't exist yet)
>
>   For Ansible users, we would write an Ansible module that would
> take in entry 2 files: the schema and the data. The module would just
> run the tool provided by oslo.config.
>   Example:
>   - name: Generate cinder.conf
> oslo-gen-config: schema=cinder-schema.yaml
>data=config_data.yaml
>
>
> Please bring feedback and thoughts, it's really important to know what
> folks from Installers think about this idea; again the ultimate goal
> is to provide a reference tool to generate configuration in OpenStack,
> in a way that scales and is friendly for our operators.
>
> Thanks,
>
> [1] http://lists.openstack.org/pipermail/openstack-dev/2017-June/118176.html
> --
> Emilien Macchi



-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev