Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Emilien Macchi]

On Sun, Apr 9, 2017 at 6:36 PM, Stephen Hindle  wrote:
> On Fri, Apr 7, 2017 at 3:00 PM, Doug Hellmann  wrote:
>>
>> Although I was a proponent of having the reload feature address
>> that issue, I'm not sure the complexity of the current implementation
>> is something we want to hang on to. In the new spec, I propose an
>> alternative treatment, which is to not cache mutable values in the
>> first place so the service never needs to receive a signal to reload.
>> The reload API is retained, and simply discards *everything* and
>> starts the configuration object over as though it was being freshly
>> created. This will be a big change, but the feature is new I think
>> the propose behavior better fits the spirit of the need anyway. Please
>> provide feedback if you think otherwise.
>>
>
> I'm still concerned about how this handles non-OpenStack services.
> Kolla currently provides containers for MySQL, RabbitMQ, Ceph, etc.  I
> agree with Paul Belanger, creating a 'special snowflake' for OpenStack
> config would be bad.  Personally, I agree something like confd could
> be used to keep configs 'out' of the containers, by generating them at
> run time.  This could work for both OS and Non-OS services, giving a
> consistent mechanism.

I still don't see anything wrong that would block us to make OpenStack
services talking to etcd directly, while some non-OpenStack services
wouldn't.
In fact, while oslo.db would get the config from etcd, confd (used by
some non-OpenStack services) would also use etcd backend.

The major thing with confd that I don't like is the fact that it uses
Templates files (like Kolla and some other projects are using). We
don't want to maintain templates files as OpenStack options tend to
change every time.
That's one of the major reasons why we think oslo.db could directly
talk to etcd to store the parameters / values.

>
>
> --
> Stephen Hindle - Senior Systems Engineer
> 480.807.8189 480.807.8189
> www.limelight.com Delivering Faster Better
>
> Join the conversation
>
> at Limelight Connect
>
> --
> The information in this message may be confidential.  It is intended solely
> for
> the addressee(s).  If you are not the intended recipient, any disclosure,
> copying or distribution of the message, or any action or omission taken by
> you
> in reliance on it, is prohibited and may be unlawful.  Please immediately
> contact the sender if you have received this message in error.
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Stephen Hindle]

On Fri, Apr 7, 2017 at 3:00 PM, Doug Hellmann  wrote:
>
> Although I was a proponent of having the reload feature address
> that issue, I'm not sure the complexity of the current implementation
> is something we want to hang on to. In the new spec, I propose an
> alternative treatment, which is to not cache mutable values in the
> first place so the service never needs to receive a signal to reload.
> The reload API is retained, and simply discards *everything* and
> starts the configuration object over as though it was being freshly
> created. This will be a big change, but the feature is new I think
> the propose behavior better fits the spirit of the need anyway. Please
> provide feedback if you think otherwise.
>

I'm still concerned about how this handles non-OpenStack services.
Kolla currently provides containers for MySQL, RabbitMQ, Ceph, etc.  I
agree with Paul Belanger, creating a 'special snowflake' for OpenStack
config would be bad.  Personally, I agree something like confd could
be used to keep configs 'out' of the containers, by generating them at
run time.  This could work for both OS and Non-OS services, giving a
consistent mechanism.



-- 
Stephen Hindle - Senior Systems Engineer
480.807.8189 480.807.8189
www.limelight.com Delivering Faster Better

Join the conversation

at Limelight Connect

-- 
The information in this message may be confidential.  It is intended solely 
for
the addressee(s).  If you are not the intended recipient, any disclosure,
copying or distribution of the message, or any action or omission taken by 
you
in reliance on it, is prohibited and may be unlawful.  Please immediately
contact the sender if you have received this message in error.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Doug Hellmann]

Excerpts from Emilien Macchi's message of 2017-04-07 11:16:34 -0400:

> I've resurrected an old spec: https://review.openstack.org/#/c/243114/
> - addressed some comments and put TODOs that Doug and I will work
> together.
> The target is set to Queens but we might provide some proof-of-concept
> during Pike to make progress.
> TripleO project is very interested by this feature and I'm pretty sure
> other deployment tools might be too. Any feedback on the work here is
> more than welcome!

After discussing this with Emilien for a while this afternoon, I've
completely rewritten the spec. I abandoned the old one in favor of
https://review.openstack.org/#/c/454897/, but include a reference
to the old one for attribution and archaeologists.

One of the hardest aspects to address was the "mutable" option
feature we recently added. The current implementation of that feature
includes a lot of protective logic to try to avoid letting an
application reload a config setting with a different value unless
the application has explicitly said it knows how to handle cases
where the value changes (think about connection strings for long-lived
database connections).

Although I was a proponent of having the reload feature address
that issue, I'm not sure the complexity of the current implementation
is something we want to hang on to. In the new spec, I propose an
alternative treatment, which is to not cache mutable values in the
first place so the service never needs to receive a signal to reload.
The reload API is retained, and simply discards *everything* and
starts the configuration object over as though it was being freshly
created. This will be a big change, but the feature is new I think
the propose behavior better fits the spirit of the need anyway. Please
provide feedback if you think otherwise.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Emilien Macchi]

On Thu, Apr 6, 2017 at 7:08 PM, Doug Hellmann  wrote:
> Excerpts from Emilien Macchi's message of 2017-04-06 18:17:59 -0400:
>> On Wed, Mar 22, 2017 at 11:23 AM, Flavio Percoco  wrote:
>> > On 15/03/17 15:40 -0400, Doug Hellmann wrote:
>> >>
>> >> Excerpts from Monty Taylor's message of 2017-03-15 04:36:24 +0100:
>> >>>
>> >>> On 03/14/2017 06:04 PM, Davanum Srinivas wrote:
>> >>> > Team,
>> >>> >
>> >>> > So one more thing popped up again on IRC:
>> >>> > https://etherpad.openstack.org/p/oslo.config_etcd_backend
>> >>> >
>> >>> > What do you think? interested in this work?
>> >>> >
>> >>> > Thanks,
>> >>> > Dims
>> >>> >
>> >>> > PS: Between this thread and the other one about Tooz/DLM and
>> >>> > os-lively, we can probably make a good case to add etcd as a base
>> >>> > always-on service.
>> >>>
>> >>> As I mentioned in the other thread, there was specific and strong
>> >>> anti-etcd sentiment in Tokyo which is why we decided to use an
>> >>> abstraction. I continue to be in favor of us having one known service in
>> >>> this space, but I do think that it's important to revisit that decision
>> >>> fully and in context of the concerns that were raised when we tried to
>> >>> pick one last time.
>> >>>
>> >>> It's worth noting that there is nothing particularly etcd-ish about
>> >>> storing config that couldn't also be done with zk and thus just be an
>> >>> additional api call or two added to Tooz with etcd and zk drivers for it.
>> >>>
>> >>
>> >> The fun* thing about working with these libraries is managing the
>> >> interdependencies. If we're going to have an abstraction library that
>> >> provides configuration options for seeing the backend, like we do in
>> >> oslo.db and olso.messaging, then the configuration library can't use it
>> >> or we have a circular dependency.
>> >>
>> >> Luckily, tooz does not currently use oslo.config. So, oslo.config could
>> >> use tooz and we could create an oslo.dlm library with a shallow
>> >> interface mapping config options to tooz calls to open connections or
>> >> whatever we need from tooz in an application. Then apps could use
>> >> oslo.dlm instead of calling into tooz directly and the configuration of
>> >> the backend would be hidden from the application developer.
>> >
>> >
>> > Replying here becasue I like the proposal, I like what Monty said and I 
>> > also
>> > like what Doug said. Most of the issues and concerns have been covered in
>> > this
>> > thread and I don't have much else to add other than +1.
>>
>> The one-million-dollar question now is: what are the next steps?
>> It sounds like an oslo spec would be nice to summarize the ideas here
>> and talk about design.
>>
>> I volunteer to help but I would need someone more familiar than I am with 
>> Oslo.
>> Please let me know if you're interested to work on it with me
>> otherwise I'll chase chase some of you :-)
>
> I can help from the Oslo side.

I've resurrected an old spec: https://review.openstack.org/#/c/243114/
- addressed some comments and put TODOs that Doug and I will work
together.
The target is set to Queens but we might provide some proof-of-concept
during Pike to make progress.
TripleO project is very interested by this feature and I'm pretty sure
other deployment tools might be too. Any feedback on the work here is
more than welcome!

Thanks,

> Doug
>
>>
>> Thanks for the nice discussions here, I think we've made good progress.
>>
>> >> Doug
>> >>
>> >> * your definition of "fun" may be different than mine
>> >
>> >
>> > Which is probably different than mine :)
>> >
>> > --
>> > @flaper87
>> > Flavio Percoco
>> >
>> > __
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Doug Hellmann]

Excerpts from Emilien Macchi's message of 2017-04-06 18:17:59 -0400:
> On Wed, Mar 22, 2017 at 11:23 AM, Flavio Percoco  wrote:
> > On 15/03/17 15:40 -0400, Doug Hellmann wrote:
> >>
> >> Excerpts from Monty Taylor's message of 2017-03-15 04:36:24 +0100:
> >>>
> >>> On 03/14/2017 06:04 PM, Davanum Srinivas wrote:
> >>> > Team,
> >>> >
> >>> > So one more thing popped up again on IRC:
> >>> > https://etherpad.openstack.org/p/oslo.config_etcd_backend
> >>> >
> >>> > What do you think? interested in this work?
> >>> >
> >>> > Thanks,
> >>> > Dims
> >>> >
> >>> > PS: Between this thread and the other one about Tooz/DLM and
> >>> > os-lively, we can probably make a good case to add etcd as a base
> >>> > always-on service.
> >>>
> >>> As I mentioned in the other thread, there was specific and strong
> >>> anti-etcd sentiment in Tokyo which is why we decided to use an
> >>> abstraction. I continue to be in favor of us having one known service in
> >>> this space, but I do think that it's important to revisit that decision
> >>> fully and in context of the concerns that were raised when we tried to
> >>> pick one last time.
> >>>
> >>> It's worth noting that there is nothing particularly etcd-ish about
> >>> storing config that couldn't also be done with zk and thus just be an
> >>> additional api call or two added to Tooz with etcd and zk drivers for it.
> >>>
> >>
> >> The fun* thing about working with these libraries is managing the
> >> interdependencies. If we're going to have an abstraction library that
> >> provides configuration options for seeing the backend, like we do in
> >> oslo.db and olso.messaging, then the configuration library can't use it
> >> or we have a circular dependency.
> >>
> >> Luckily, tooz does not currently use oslo.config. So, oslo.config could
> >> use tooz and we could create an oslo.dlm library with a shallow
> >> interface mapping config options to tooz calls to open connections or
> >> whatever we need from tooz in an application. Then apps could use
> >> oslo.dlm instead of calling into tooz directly and the configuration of
> >> the backend would be hidden from the application developer.
> >
> >
> > Replying here becasue I like the proposal, I like what Monty said and I also
> > like what Doug said. Most of the issues and concerns have been covered in
> > this
> > thread and I don't have much else to add other than +1.
> 
> The one-million-dollar question now is: what are the next steps?
> It sounds like an oslo spec would be nice to summarize the ideas here
> and talk about design.
> 
> I volunteer to help but I would need someone more familiar than I am with 
> Oslo.
> Please let me know if you're interested to work on it with me
> otherwise I'll chase chase some of you :-)

I can help from the Oslo side.

Doug

> 
> Thanks for the nice discussions here, I think we've made good progress.
> 
> >> Doug
> >>
> >> * your definition of "fun" may be different than mine
> >
> >
> > Which is probably different than mine :)
> >
> > --
> > @flaper87
> > Flavio Percoco
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Emilien Macchi]

On Wed, Mar 22, 2017 at 11:23 AM, Flavio Percoco  wrote:
> On 15/03/17 15:40 -0400, Doug Hellmann wrote:
>>
>> Excerpts from Monty Taylor's message of 2017-03-15 04:36:24 +0100:
>>>
>>> On 03/14/2017 06:04 PM, Davanum Srinivas wrote:
>>> > Team,
>>> >
>>> > So one more thing popped up again on IRC:
>>> > https://etherpad.openstack.org/p/oslo.config_etcd_backend
>>> >
>>> > What do you think? interested in this work?
>>> >
>>> > Thanks,
>>> > Dims
>>> >
>>> > PS: Between this thread and the other one about Tooz/DLM and
>>> > os-lively, we can probably make a good case to add etcd as a base
>>> > always-on service.
>>>
>>> As I mentioned in the other thread, there was specific and strong
>>> anti-etcd sentiment in Tokyo which is why we decided to use an
>>> abstraction. I continue to be in favor of us having one known service in
>>> this space, but I do think that it's important to revisit that decision
>>> fully and in context of the concerns that were raised when we tried to
>>> pick one last time.
>>>
>>> It's worth noting that there is nothing particularly etcd-ish about
>>> storing config that couldn't also be done with zk and thus just be an
>>> additional api call or two added to Tooz with etcd and zk drivers for it.
>>>
>>
>> The fun* thing about working with these libraries is managing the
>> interdependencies. If we're going to have an abstraction library that
>> provides configuration options for seeing the backend, like we do in
>> oslo.db and olso.messaging, then the configuration library can't use it
>> or we have a circular dependency.
>>
>> Luckily, tooz does not currently use oslo.config. So, oslo.config could
>> use tooz and we could create an oslo.dlm library with a shallow
>> interface mapping config options to tooz calls to open connections or
>> whatever we need from tooz in an application. Then apps could use
>> oslo.dlm instead of calling into tooz directly and the configuration of
>> the backend would be hidden from the application developer.
>
>
> Replying here becasue I like the proposal, I like what Monty said and I also
> like what Doug said. Most of the issues and concerns have been covered in
> this
> thread and I don't have much else to add other than +1.

The one-million-dollar question now is: what are the next steps?
It sounds like an oslo spec would be nice to summarize the ideas here
and talk about design.

I volunteer to help but I would need someone more familiar than I am with Oslo.
Please let me know if you're interested to work on it with me
otherwise I'll chase chase some of you :-)

Thanks for the nice discussions here, I think we've made good progress.

>> Doug
>>
>> * your definition of "fun" may be different than mine
>
>
> Which is probably different than mine :)
>
> --
> @flaper87
> Flavio Percoco
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[ChangBo Guo]

2017-03-31 2:55 GMT+08:00 Doug Hellmann :

> Excerpts from Paul Belanger's message of 2017-03-22 09:58:46 -0400:
> > On Tue, Mar 21, 2017 at 05:53:35PM -0600, Alex Schultz wrote:
> > > On Tue, Mar 21, 2017 at 5:35 PM, John Dickinson  wrote:
> > > >
> > > >
> > > > On 21 Mar 2017, at 15:34, Alex Schultz wrote:
> > > >
> > > >> On Tue, Mar 21, 2017 at 3:45 PM, John Dickinson  wrote:
> > > >>> I've been following this thread, but I must admit I seem to have
> missed something.
> > > >>>
> > > >>> What problem is being solved by storing per-server service
> configuration options in an external distributed CP system that is
> currently not possible with the existing pattern of using local text files?
> > > >>>
> > > >>
> > > >> This effort is partially to help the path to containerization where
> we
> > > >> are delivering the service code via container but don't want to
> > > >> necessarily deliver the configuration in the same fashion.  It's
> about
> > > >> ease of configuration where moving service -> config files (on many
> > > >> hosts/containers) to service -> config via etcd (single source
> > > >> cluster).  It's also about an alternative to configuration
> management
> > > >> where today we have many tools handling the files in various ways
> > > >> (templates, from repo, via code providers) and trying to come to a
> > > >> more unified way of representing the configuration such that the end
> > > >> result is the same for every deployment tool.  All tools load
> configs
> > > >> into $place and services can be configured to talk to $place.  It
> > > >> should be noted that configuration files won't go away because many
> of
> > > >> the companion services still rely on them (rabbit/mysql/apache/etc)
> so
> > > >> we're really talking about services that currently use oslo.
> > > >
> > > > Thanks for the explanation!
> > > >
> > > > So in the future, you expect a node in a clustered OpenStack service
> to be deployed and run as a container, and then that node queries a
> centralized etcd (or other) k/v store to load config options. And other
> services running in the (container? cluster?) will load config from local
> text files managed in some other way.
> > >
> > > No the goal is in the etcd mode, that it  may not be necessary to load
> > > the config files locally at all.  That being said there would still be
> > > support for having some configuration from a file and optionally
> > > provide a kv store as another config point.  'service --config-file
> > > /etc/service/service.conf --config-etcd proto://ip:port/slug'
> > >
> > Hmm, not sure I like this.  Having a service magically read from 2
> different
> > configuration source at run time, merge them, and reload, seems overly
> > complicated. And even harder to debug.
> >
> > > >
> > > > No wait. It's not the *services* that will load the config from a kv
> store--it's the config management system? So in the process of deploying a
> new container instance of a particular service, the deployment tool will
> pull the right values out of the kv system and inject those into the
> container, I'm guessing as a local text file that the service loads as
> normal?
> > > >
> > >
> > > No the thought is to have the services pull their configs from the kv
> > > store via oslo.config.  The point is hopefully to not require
> > > configuration files at all for containers.  The container would get
> > > where to pull it's configs from (ie. http://11.1.1.1:2730/magic/ or
> > > /etc/myconfigs/).  At that point it just becomes another place to load
> > > configurations from via oslo.config.  Configuration management comes
> > > in as a way to load the configs either as a file or into etcd.  Many
> > > operators (and deployment tools) are already using some form of
> > > configuration management so if we can integrate in a kv store output
> > > option, adoption becomes much easier than making everyone start from
> > > scratch.
> > >
> > > > This means you could have some (OpenStack?) service for inventory
> management (like Karbor) that is seeding the kv store, the cloud
> infrastructure software itself is "cloud aware" and queries the central
> distributed kv system for the correct-right-now config options, and the
> cloud service itself gets all the benefits of dynamic scaling of available
> hardware resources. That's pretty cool. Add hardware to the inventory, the
> cloud infra itself expands to make it available. Hardware fails, and the
> cloud infra resizes to adjust. Apps running on the infra keep doing their
> thing consuming the resources. It's clouds all the way down :-)
> > > >
> > > > Despite sounding pretty interesting, it also sounds like a lot of
> extra complexity. Maybe it's worth it. I don't know.
> > > >
> > >
> > > Yea there's extra complexity at least in the
> > > deployment/management/monitoring of the new service or maybe not.
> > > Keeping configuration files synced across 1000s of nodes (or

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Doug Hellmann]

Excerpts from Paul Belanger's message of 2017-03-22 09:58:46 -0400:
> On Tue, Mar 21, 2017 at 05:53:35PM -0600, Alex Schultz wrote:
> > On Tue, Mar 21, 2017 at 5:35 PM, John Dickinson  wrote:
> > >
> > >
> > > On 21 Mar 2017, at 15:34, Alex Schultz wrote:
> > >
> > >> On Tue, Mar 21, 2017 at 3:45 PM, John Dickinson  wrote:
> > >>> I've been following this thread, but I must admit I seem to have missed 
> > >>> something.
> > >>>
> > >>> What problem is being solved by storing per-server service 
> > >>> configuration options in an external distributed CP system that is 
> > >>> currently not possible with the existing pattern of using local text 
> > >>> files?
> > >>>
> > >>
> > >> This effort is partially to help the path to containerization where we
> > >> are delivering the service code via container but don't want to
> > >> necessarily deliver the configuration in the same fashion.  It's about
> > >> ease of configuration where moving service -> config files (on many
> > >> hosts/containers) to service -> config via etcd (single source
> > >> cluster).  It's also about an alternative to configuration management
> > >> where today we have many tools handling the files in various ways
> > >> (templates, from repo, via code providers) and trying to come to a
> > >> more unified way of representing the configuration such that the end
> > >> result is the same for every deployment tool.  All tools load configs
> > >> into $place and services can be configured to talk to $place.  It
> > >> should be noted that configuration files won't go away because many of
> > >> the companion services still rely on them (rabbit/mysql/apache/etc) so
> > >> we're really talking about services that currently use oslo.
> > >
> > > Thanks for the explanation!
> > >
> > > So in the future, you expect a node in a clustered OpenStack service to 
> > > be deployed and run as a container, and then that node queries a 
> > > centralized etcd (or other) k/v store to load config options. And other 
> > > services running in the (container? cluster?) will load config from local 
> > > text files managed in some other way.
> > 
> > No the goal is in the etcd mode, that it  may not be necessary to load
> > the config files locally at all.  That being said there would still be
> > support for having some configuration from a file and optionally
> > provide a kv store as another config point.  'service --config-file
> > /etc/service/service.conf --config-etcd proto://ip:port/slug'
> > 
> Hmm, not sure I like this.  Having a service magically read from 2 different
> configuration source at run time, merge them, and reload, seems overly
> complicated. And even harder to debug.
> 
> > >
> > > No wait. It's not the *services* that will load the config from a kv 
> > > store--it's the config management system? So in the process of deploying 
> > > a new container instance of a particular service, the deployment tool 
> > > will pull the right values out of the kv system and inject those into the 
> > > container, I'm guessing as a local text file that the service loads as 
> > > normal?
> > >
> > 
> > No the thought is to have the services pull their configs from the kv
> > store via oslo.config.  The point is hopefully to not require
> > configuration files at all for containers.  The container would get
> > where to pull it's configs from (ie. http://11.1.1.1:2730/magic/ or
> > /etc/myconfigs/).  At that point it just becomes another place to load
> > configurations from via oslo.config.  Configuration management comes
> > in as a way to load the configs either as a file or into etcd.  Many
> > operators (and deployment tools) are already using some form of
> > configuration management so if we can integrate in a kv store output
> > option, adoption becomes much easier than making everyone start from
> > scratch.
> > 
> > > This means you could have some (OpenStack?) service for inventory 
> > > management (like Karbor) that is seeding the kv store, the cloud 
> > > infrastructure software itself is "cloud aware" and queries the central 
> > > distributed kv system for the correct-right-now config options, and the 
> > > cloud service itself gets all the benefits of dynamic scaling of 
> > > available hardware resources. That's pretty cool. Add hardware to the 
> > > inventory, the cloud infra itself expands to make it available. Hardware 
> > > fails, and the cloud infra resizes to adjust. Apps running on the infra 
> > > keep doing their thing consuming the resources. It's clouds all the way 
> > > down :-)
> > >
> > > Despite sounding pretty interesting, it also sounds like a lot of extra 
> > > complexity. Maybe it's worth it. I don't know.
> > >
> > 
> > Yea there's extra complexity at least in the
> > deployment/management/monitoring of the new service or maybe not.
> > Keeping configuration files synced across 1000s of nodes (or
> > containers) can be just as hard however.
> > 
> Please correct me if I am 

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Flavio Percoco]

On 15/03/17 15:40 -0400, Doug Hellmann wrote:

Excerpts from Monty Taylor's message of 2017-03-15 04:36:24 +0100:

On 03/14/2017 06:04 PM, Davanum Srinivas wrote:
> Team,
>
> So one more thing popped up again on IRC:
> https://etherpad.openstack.org/p/oslo.config_etcd_backend
>
> What do you think? interested in this work?
>
> Thanks,
> Dims
>
> PS: Between this thread and the other one about Tooz/DLM and
> os-lively, we can probably make a good case to add etcd as a base
> always-on service.

As I mentioned in the other thread, there was specific and strong
anti-etcd sentiment in Tokyo which is why we decided to use an
abstraction. I continue to be in favor of us having one known service in
this space, but I do think that it's important to revisit that decision
fully and in context of the concerns that were raised when we tried to
pick one last time.

It's worth noting that there is nothing particularly etcd-ish about
storing config that couldn't also be done with zk and thus just be an
additional api call or two added to Tooz with etcd and zk drivers for it.



The fun* thing about working with these libraries is managing the
interdependencies. If we're going to have an abstraction library that
provides configuration options for seeing the backend, like we do in
oslo.db and olso.messaging, then the configuration library can't use it
or we have a circular dependency.

Luckily, tooz does not currently use oslo.config. So, oslo.config could
use tooz and we could create an oslo.dlm library with a shallow
interface mapping config options to tooz calls to open connections or
whatever we need from tooz in an application. Then apps could use
oslo.dlm instead of calling into tooz directly and the configuration of
the backend would be hidden from the application developer.


Replying here becasue I like the proposal, I like what Monty said and I also
like what Doug said. Most of the issues and concerns have been covered in this
thread and I don't have much else to add other than +1.


Doug

* your definition of "fun" may be different than mine


Which is probably different than mine :)

--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Thomas Herve]

On Wed, Mar 22, 2017 at 3:24 PM, Alex Schultz  wrote:
> On Wed, Mar 22, 2017 at 7:58 AM, Paul Belanger  wrote:
[snip]
>> Please correct me if I am wrong, because I still have my container training
>> wheels on. I understand the need for etcd, and operators to write their
>> configuration into it.  Why I am struggling with still, is why you need
>> oslo.config to support it.  There is nothing stopping an operator today from
>> using etcd / confd in a container, right?  I can only imagine countless other
>> services that run in containers using them.
>>
>
> We want oslo.config to support it as a source for configuration.
> Dealing with files in containers is complicated. If we can remove the
> requirement to munge configurations for containers,
> deployment/updating containers becomes easier.  The service container
> becomes a single artifact to be deployed with less moving parts which
> helps reduce complexity and errors.  The process for moving a single
> container artifact is a lot easier than moving container and updating
> configurations based on where it's landing.

I believe the point is that operators will need to have a solution for
non-oslo services, if we want to centralize configuration using etcd.
If that's the case, it's unclear what will be the benefit of having
direct support for etcd in oslo.

I have even a counter example: let's say you want to deploy heat api
using httpd (as recommended). You'll deploy it it in a container: you
then need confd to manage httpd config, but Heat would then talk
directly to etcd? I'm not sure the benefit would be gigantic.

To summarize: confd (or something equivalent) needs to be in the
equation. Should we "simply" standardize on it?

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Paul Belanger]

On Wed, Mar 22, 2017 at 08:24:52AM -0600, Alex Schultz wrote:
> On Wed, Mar 22, 2017 at 7:58 AM, Paul Belanger  wrote:
> > On Tue, Mar 21, 2017 at 05:53:35PM -0600, Alex Schultz wrote:
> >> On Tue, Mar 21, 2017 at 5:35 PM, John Dickinson  wrote:
> >> >
> >> >
> >> > On 21 Mar 2017, at 15:34, Alex Schultz wrote:
> >> >
> >> >> On Tue, Mar 21, 2017 at 3:45 PM, John Dickinson  wrote:
> >> >>> I've been following this thread, but I must admit I seem to have 
> >> >>> missed something.
> >> >>>
> >> >>> What problem is being solved by storing per-server service 
> >> >>> configuration options in an external distributed CP system that is 
> >> >>> currently not possible with the existing pattern of using local text 
> >> >>> files?
> >> >>>
> >> >>
> >> >> This effort is partially to help the path to containerization where we
> >> >> are delivering the service code via container but don't want to
> >> >> necessarily deliver the configuration in the same fashion.  It's about
> >> >> ease of configuration where moving service -> config files (on many
> >> >> hosts/containers) to service -> config via etcd (single source
> >> >> cluster).  It's also about an alternative to configuration management
> >> >> where today we have many tools handling the files in various ways
> >> >> (templates, from repo, via code providers) and trying to come to a
> >> >> more unified way of representing the configuration such that the end
> >> >> result is the same for every deployment tool.  All tools load configs
> >> >> into $place and services can be configured to talk to $place.  It
> >> >> should be noted that configuration files won't go away because many of
> >> >> the companion services still rely on them (rabbit/mysql/apache/etc) so
> >> >> we're really talking about services that currently use oslo.
> >> >
> >> > Thanks for the explanation!
> >> >
> >> > So in the future, you expect a node in a clustered OpenStack service to 
> >> > be deployed and run as a container, and then that node queries a 
> >> > centralized etcd (or other) k/v store to load config options. And other 
> >> > services running in the (container? cluster?) will load config from 
> >> > local text files managed in some other way.
> >>
> >> No the goal is in the etcd mode, that it  may not be necessary to load
> >> the config files locally at all.  That being said there would still be
> >> support for having some configuration from a file and optionally
> >> provide a kv store as another config point.  'service --config-file
> >> /etc/service/service.conf --config-etcd proto://ip:port/slug'
> >>
> > Hmm, not sure I like this.  Having a service magically read from 2 different
> > configuration source at run time, merge them, and reload, seems overly
> > complicated. And even harder to debug.
> >
> 
> That's something inherently supported by oslo.config today. We even do
> it for dist provided packaging (I also don't like it, but it's an
> established pattern).
> 
> >> >
> >> > No wait. It's not the *services* that will load the config from a kv 
> >> > store--it's the config management system? So in the process of deploying 
> >> > a new container instance of a particular service, the deployment tool 
> >> > will pull the right values out of the kv system and inject those into 
> >> > the container, I'm guessing as a local text file that the service loads 
> >> > as normal?
> >> >
> >>
> >> No the thought is to have the services pull their configs from the kv
> >> store via oslo.config.  The point is hopefully to not require
> >> configuration files at all for containers.  The container would get
> >> where to pull it's configs from (ie. http://11.1.1.1:2730/magic/ or
> >> /etc/myconfigs/).  At that point it just becomes another place to load
> >> configurations from via oslo.config.  Configuration management comes
> >> in as a way to load the configs either as a file or into etcd.  Many
> >> operators (and deployment tools) are already using some form of
> >> configuration management so if we can integrate in a kv store output
> >> option, adoption becomes much easier than making everyone start from
> >> scratch.
> >>
> >> > This means you could have some (OpenStack?) service for inventory 
> >> > management (like Karbor) that is seeding the kv store, the cloud 
> >> > infrastructure software itself is "cloud aware" and queries the central 
> >> > distributed kv system for the correct-right-now config options, and the 
> >> > cloud service itself gets all the benefits of dynamic scaling of 
> >> > available hardware resources. That's pretty cool. Add hardware to the 
> >> > inventory, the cloud infra itself expands to make it available. Hardware 
> >> > fails, and the cloud infra resizes to adjust. Apps running on the infra 
> >> > keep doing their thing consuming the resources. It's clouds all the way 
> >> > down :-)
> >> >
> >> > Despite sounding pretty interesting, it also sounds like a lot of extra 

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Alex Schultz]

On Wed, Mar 22, 2017 at 7:58 AM, Paul Belanger  wrote:
> On Tue, Mar 21, 2017 at 05:53:35PM -0600, Alex Schultz wrote:
>> On Tue, Mar 21, 2017 at 5:35 PM, John Dickinson  wrote:
>> >
>> >
>> > On 21 Mar 2017, at 15:34, Alex Schultz wrote:
>> >
>> >> On Tue, Mar 21, 2017 at 3:45 PM, John Dickinson  wrote:
>> >>> I've been following this thread, but I must admit I seem to have missed 
>> >>> something.
>> >>>
>> >>> What problem is being solved by storing per-server service configuration 
>> >>> options in an external distributed CP system that is currently not 
>> >>> possible with the existing pattern of using local text files?
>> >>>
>> >>
>> >> This effort is partially to help the path to containerization where we
>> >> are delivering the service code via container but don't want to
>> >> necessarily deliver the configuration in the same fashion.  It's about
>> >> ease of configuration where moving service -> config files (on many
>> >> hosts/containers) to service -> config via etcd (single source
>> >> cluster).  It's also about an alternative to configuration management
>> >> where today we have many tools handling the files in various ways
>> >> (templates, from repo, via code providers) and trying to come to a
>> >> more unified way of representing the configuration such that the end
>> >> result is the same for every deployment tool.  All tools load configs
>> >> into $place and services can be configured to talk to $place.  It
>> >> should be noted that configuration files won't go away because many of
>> >> the companion services still rely on them (rabbit/mysql/apache/etc) so
>> >> we're really talking about services that currently use oslo.
>> >
>> > Thanks for the explanation!
>> >
>> > So in the future, you expect a node in a clustered OpenStack service to be 
>> > deployed and run as a container, and then that node queries a centralized 
>> > etcd (or other) k/v store to load config options. And other services 
>> > running in the (container? cluster?) will load config from local text 
>> > files managed in some other way.
>>
>> No the goal is in the etcd mode, that it  may not be necessary to load
>> the config files locally at all.  That being said there would still be
>> support for having some configuration from a file and optionally
>> provide a kv store as another config point.  'service --config-file
>> /etc/service/service.conf --config-etcd proto://ip:port/slug'
>>
> Hmm, not sure I like this.  Having a service magically read from 2 different
> configuration source at run time, merge them, and reload, seems overly
> complicated. And even harder to debug.
>

That's something inherently supported by oslo.config today. We even do
it for dist provided packaging (I also don't like it, but it's an
established pattern).

>> >
>> > No wait. It's not the *services* that will load the config from a kv 
>> > store--it's the config management system? So in the process of deploying a 
>> > new container instance of a particular service, the deployment tool will 
>> > pull the right values out of the kv system and inject those into the 
>> > container, I'm guessing as a local text file that the service loads as 
>> > normal?
>> >
>>
>> No the thought is to have the services pull their configs from the kv
>> store via oslo.config.  The point is hopefully to not require
>> configuration files at all for containers.  The container would get
>> where to pull it's configs from (ie. http://11.1.1.1:2730/magic/ or
>> /etc/myconfigs/).  At that point it just becomes another place to load
>> configurations from via oslo.config.  Configuration management comes
>> in as a way to load the configs either as a file or into etcd.  Many
>> operators (and deployment tools) are already using some form of
>> configuration management so if we can integrate in a kv store output
>> option, adoption becomes much easier than making everyone start from
>> scratch.
>>
>> > This means you could have some (OpenStack?) service for inventory 
>> > management (like Karbor) that is seeding the kv store, the cloud 
>> > infrastructure software itself is "cloud aware" and queries the central 
>> > distributed kv system for the correct-right-now config options, and the 
>> > cloud service itself gets all the benefits of dynamic scaling of available 
>> > hardware resources. That's pretty cool. Add hardware to the inventory, the 
>> > cloud infra itself expands to make it available. Hardware fails, and the 
>> > cloud infra resizes to adjust. Apps running on the infra keep doing their 
>> > thing consuming the resources. It's clouds all the way down :-)
>> >
>> > Despite sounding pretty interesting, it also sounds like a lot of extra 
>> > complexity. Maybe it's worth it. I don't know.
>> >
>>
>> Yea there's extra complexity at least in the
>> deployment/management/monitoring of the new service or maybe not.
>> Keeping configuration files synced across 1000s of nodes (or
>> 

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Alex Schultz]

On Wed, Mar 22, 2017 at 12:23 AM, Tim Bell  wrote:
>
>> On 22 Mar 2017, at 00:53, Alex Schultz  wrote:
>>
>> On Tue, Mar 21, 2017 at 5:35 PM, John Dickinson  wrote:
>>>
>>>
>>> On 21 Mar 2017, at 15:34, Alex Schultz wrote:
>>>
 On Tue, Mar 21, 2017 at 3:45 PM, John Dickinson  wrote:
> I've been following this thread, but I must admit I seem to have missed 
> something.
>
> What problem is being solved by storing per-server service configuration 
> options in an external distributed CP system that is currently not 
> possible with the existing pattern of using local text files?
>

 This effort is partially to help the path to containerization where we
 are delivering the service code via container but don't want to
 necessarily deliver the configuration in the same fashion.  It's about
 ease of configuration where moving service -> config files (on many
 hosts/containers) to service -> config via etcd (single source
 cluster).  It's also about an alternative to configuration management
 where today we have many tools handling the files in various ways
 (templates, from repo, via code providers) and trying to come to a
 more unified way of representing the configuration such that the end
 result is the same for every deployment tool.  All tools load configs
 into $place and services can be configured to talk to $place.  It
 should be noted that configuration files won't go away because many of
 the companion services still rely on them (rabbit/mysql/apache/etc) so
 we're really talking about services that currently use oslo.
>>>
>>> Thanks for the explanation!
>>>
>>> So in the future, you expect a node in a clustered OpenStack service to be 
>>> deployed and run as a container, and then that node queries a centralized 
>>> etcd (or other) k/v store to load config options. And other services 
>>> running in the (container? cluster?) will load config from local text files 
>>> managed in some other way.
>>
>> No the goal is in the etcd mode, that it  may not be necessary to load
>> the config files locally at all.  That being said there would still be
>> support for having some configuration from a file and optionally
>> provide a kv store as another config point.  'service --config-file
>> /etc/service/service.conf --config-etcd proto://ip:port/slug'
>>
>>>
>>> No wait. It's not the *services* that will load the config from a kv 
>>> store--it's the config management system? So in the process of deploying a 
>>> new container instance of a particular service, the deployment tool will 
>>> pull the right values out of the kv system and inject those into the 
>>> container, I'm guessing as a local text file that the service loads as 
>>> normal?
>>>
>>
>> No the thought is to have the services pull their configs from the kv
>> store via oslo.config.  The point is hopefully to not require
>> configuration files at all for containers.  The container would get
>> where to pull it's configs from (ie. http://11.1.1.1:2730/magic/ or
>> /etc/myconfigs/).  At that point it just becomes another place to load
>> configurations from via oslo.config.  Configuration management comes
>> in as a way to load the configs either as a file or into etcd.  Many
>> operators (and deployment tools) are already using some form of
>> configuration management so if we can integrate in a kv store output
>> option, adoption becomes much easier than making everyone start from
>> scratch.
>>
>>> This means you could have some (OpenStack?) service for inventory 
>>> management (like Karbor) that is seeding the kv store, the cloud 
>>> infrastructure software itself is "cloud aware" and queries the central 
>>> distributed kv system for the correct-right-now config options, and the 
>>> cloud service itself gets all the benefits of dynamic scaling of available 
>>> hardware resources. That's pretty cool. Add hardware to the inventory, the 
>>> cloud infra itself expands to make it available. Hardware fails, and the 
>>> cloud infra resizes to adjust. Apps running on the infra keep doing their 
>>> thing consuming the resources. It's clouds all the way down :-)
>>>
>>> Despite sounding pretty interesting, it also sounds like a lot of extra 
>>> complexity. Maybe it's worth it. I don't know.
>>>
>>
>> Yea there's extra complexity at least in the
>> deployment/management/monitoring of the new service or maybe not.
>> Keeping configuration files synced across 1000s of nodes (or
>> containers) can be just as hard however.
>>
>
> Would there be a mechanism to stage configuration changes (such as a 
> QA/production environment) or have different configurations for different 
> hypervisors?
>

Yes my understanding is that the goal is not to have a single config
for all the deployed instances as that just doesn't make sense.  This
is primarily the problem with trying to distribute configs with 

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Paul Belanger]

On Tue, Mar 21, 2017 at 05:53:35PM -0600, Alex Schultz wrote:
> On Tue, Mar 21, 2017 at 5:35 PM, John Dickinson  wrote:
> >
> >
> > On 21 Mar 2017, at 15:34, Alex Schultz wrote:
> >
> >> On Tue, Mar 21, 2017 at 3:45 PM, John Dickinson  wrote:
> >>> I've been following this thread, but I must admit I seem to have missed 
> >>> something.
> >>>
> >>> What problem is being solved by storing per-server service configuration 
> >>> options in an external distributed CP system that is currently not 
> >>> possible with the existing pattern of using local text files?
> >>>
> >>
> >> This effort is partially to help the path to containerization where we
> >> are delivering the service code via container but don't want to
> >> necessarily deliver the configuration in the same fashion.  It's about
> >> ease of configuration where moving service -> config files (on many
> >> hosts/containers) to service -> config via etcd (single source
> >> cluster).  It's also about an alternative to configuration management
> >> where today we have many tools handling the files in various ways
> >> (templates, from repo, via code providers) and trying to come to a
> >> more unified way of representing the configuration such that the end
> >> result is the same for every deployment tool.  All tools load configs
> >> into $place and services can be configured to talk to $place.  It
> >> should be noted that configuration files won't go away because many of
> >> the companion services still rely on them (rabbit/mysql/apache/etc) so
> >> we're really talking about services that currently use oslo.
> >
> > Thanks for the explanation!
> >
> > So in the future, you expect a node in a clustered OpenStack service to be 
> > deployed and run as a container, and then that node queries a centralized 
> > etcd (or other) k/v store to load config options. And other services 
> > running in the (container? cluster?) will load config from local text files 
> > managed in some other way.
> 
> No the goal is in the etcd mode, that it  may not be necessary to load
> the config files locally at all.  That being said there would still be
> support for having some configuration from a file and optionally
> provide a kv store as another config point.  'service --config-file
> /etc/service/service.conf --config-etcd proto://ip:port/slug'
> 
Hmm, not sure I like this.  Having a service magically read from 2 different
configuration source at run time, merge them, and reload, seems overly
complicated. And even harder to debug.

> >
> > No wait. It's not the *services* that will load the config from a kv 
> > store--it's the config management system? So in the process of deploying a 
> > new container instance of a particular service, the deployment tool will 
> > pull the right values out of the kv system and inject those into the 
> > container, I'm guessing as a local text file that the service loads as 
> > normal?
> >
> 
> No the thought is to have the services pull their configs from the kv
> store via oslo.config.  The point is hopefully to not require
> configuration files at all for containers.  The container would get
> where to pull it's configs from (ie. http://11.1.1.1:2730/magic/ or
> /etc/myconfigs/).  At that point it just becomes another place to load
> configurations from via oslo.config.  Configuration management comes
> in as a way to load the configs either as a file or into etcd.  Many
> operators (and deployment tools) are already using some form of
> configuration management so if we can integrate in a kv store output
> option, adoption becomes much easier than making everyone start from
> scratch.
> 
> > This means you could have some (OpenStack?) service for inventory 
> > management (like Karbor) that is seeding the kv store, the cloud 
> > infrastructure software itself is "cloud aware" and queries the central 
> > distributed kv system for the correct-right-now config options, and the 
> > cloud service itself gets all the benefits of dynamic scaling of available 
> > hardware resources. That's pretty cool. Add hardware to the inventory, the 
> > cloud infra itself expands to make it available. Hardware fails, and the 
> > cloud infra resizes to adjust. Apps running on the infra keep doing their 
> > thing consuming the resources. It's clouds all the way down :-)
> >
> > Despite sounding pretty interesting, it also sounds like a lot of extra 
> > complexity. Maybe it's worth it. I don't know.
> >
> 
> Yea there's extra complexity at least in the
> deployment/management/monitoring of the new service or maybe not.
> Keeping configuration files synced across 1000s of nodes (or
> containers) can be just as hard however.
> 
Please correct me if I am wrong, because I still have my container training
wheels on. I understand the need for etcd, and operators to write their
configuration into it.  Why I am struggling with still, is why you need
oslo.config to support it.  There is nothing stopping an operator 

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Thomas Herve]

On Tue, Mar 21, 2017 at 10:26 PM, Davanum Srinivas  wrote:
> Jay,
>
> the /v3alpha HTTP API  (grpc-gateway) supports watch
> https://coreos.com/etcd/docs/latest/dev-guide/apispec/swagger/rpc.swagger.json

Ah, that's a great find, thanks. That means that we don't have to use
grpc, and can still talk to an HTTP endpoint that would integrate
better with our tools. I tested the API, and it works fine.

Regarding reloading, I don't think watch is mandatory for now. As a
first step, I would fetch the keys again upload reload, like we do for
files.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Tim Bell]

> On 22 Mar 2017, at 00:53, Alex Schultz  wrote:
> 
> On Tue, Mar 21, 2017 at 5:35 PM, John Dickinson  wrote:
>> 
>> 
>> On 21 Mar 2017, at 15:34, Alex Schultz wrote:
>> 
>>> On Tue, Mar 21, 2017 at 3:45 PM, John Dickinson  wrote:
 I've been following this thread, but I must admit I seem to have missed 
 something.
 
 What problem is being solved by storing per-server service configuration 
 options in an external distributed CP system that is currently not 
 possible with the existing pattern of using local text files?
 
>>> 
>>> This effort is partially to help the path to containerization where we
>>> are delivering the service code via container but don't want to
>>> necessarily deliver the configuration in the same fashion.  It's about
>>> ease of configuration where moving service -> config files (on many
>>> hosts/containers) to service -> config via etcd (single source
>>> cluster).  It's also about an alternative to configuration management
>>> where today we have many tools handling the files in various ways
>>> (templates, from repo, via code providers) and trying to come to a
>>> more unified way of representing the configuration such that the end
>>> result is the same for every deployment tool.  All tools load configs
>>> into $place and services can be configured to talk to $place.  It
>>> should be noted that configuration files won't go away because many of
>>> the companion services still rely on them (rabbit/mysql/apache/etc) so
>>> we're really talking about services that currently use oslo.
>> 
>> Thanks for the explanation!
>> 
>> So in the future, you expect a node in a clustered OpenStack service to be 
>> deployed and run as a container, and then that node queries a centralized 
>> etcd (or other) k/v store to load config options. And other services running 
>> in the (container? cluster?) will load config from local text files managed 
>> in some other way.
> 
> No the goal is in the etcd mode, that it  may not be necessary to load
> the config files locally at all.  That being said there would still be
> support for having some configuration from a file and optionally
> provide a kv store as another config point.  'service --config-file
> /etc/service/service.conf --config-etcd proto://ip:port/slug'
> 
>> 
>> No wait. It's not the *services* that will load the config from a kv 
>> store--it's the config management system? So in the process of deploying a 
>> new container instance of a particular service, the deployment tool will 
>> pull the right values out of the kv system and inject those into the 
>> container, I'm guessing as a local text file that the service loads as 
>> normal?
>> 
> 
> No the thought is to have the services pull their configs from the kv
> store via oslo.config.  The point is hopefully to not require
> configuration files at all for containers.  The container would get
> where to pull it's configs from (ie. http://11.1.1.1:2730/magic/ or
> /etc/myconfigs/).  At that point it just becomes another place to load
> configurations from via oslo.config.  Configuration management comes
> in as a way to load the configs either as a file or into etcd.  Many
> operators (and deployment tools) are already using some form of
> configuration management so if we can integrate in a kv store output
> option, adoption becomes much easier than making everyone start from
> scratch.
> 
>> This means you could have some (OpenStack?) service for inventory management 
>> (like Karbor) that is seeding the kv store, the cloud infrastructure 
>> software itself is "cloud aware" and queries the central distributed kv 
>> system for the correct-right-now config options, and the cloud service 
>> itself gets all the benefits of dynamic scaling of available hardware 
>> resources. That's pretty cool. Add hardware to the inventory, the cloud 
>> infra itself expands to make it available. Hardware fails, and the cloud 
>> infra resizes to adjust. Apps running on the infra keep doing their thing 
>> consuming the resources. It's clouds all the way down :-)
>> 
>> Despite sounding pretty interesting, it also sounds like a lot of extra 
>> complexity. Maybe it's worth it. I don't know.
>> 
> 
> Yea there's extra complexity at least in the
> deployment/management/monitoring of the new service or maybe not.
> Keeping configuration files synced across 1000s of nodes (or
> containers) can be just as hard however.
> 

Would there be a mechanism to stage configuration changes (such as a 
QA/production environment) or have different configurations for different 
hypervisors?

We have some of our hypervisors set for high performance which needs a slightly 
different nova.conf (such as CPU passthrough).

Tim

>> Thanks again for the explanation.
>> 
>> 
>> --John
>> 
>> 
>> 
>> 
>>> 
>>> Thanks,
>>> -Alex
>>> 
 
 --John
 
 
 
 
 On 21 Mar 2017, at 14:26, Davanum Srinivas wrote:
 
> 

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Alex Schultz]

On Tue, Mar 21, 2017 at 5:35 PM, John Dickinson  wrote:
>
>
> On 21 Mar 2017, at 15:34, Alex Schultz wrote:
>
>> On Tue, Mar 21, 2017 at 3:45 PM, John Dickinson  wrote:
>>> I've been following this thread, but I must admit I seem to have missed 
>>> something.
>>>
>>> What problem is being solved by storing per-server service configuration 
>>> options in an external distributed CP system that is currently not possible 
>>> with the existing pattern of using local text files?
>>>
>>
>> This effort is partially to help the path to containerization where we
>> are delivering the service code via container but don't want to
>> necessarily deliver the configuration in the same fashion.  It's about
>> ease of configuration where moving service -> config files (on many
>> hosts/containers) to service -> config via etcd (single source
>> cluster).  It's also about an alternative to configuration management
>> where today we have many tools handling the files in various ways
>> (templates, from repo, via code providers) and trying to come to a
>> more unified way of representing the configuration such that the end
>> result is the same for every deployment tool.  All tools load configs
>> into $place and services can be configured to talk to $place.  It
>> should be noted that configuration files won't go away because many of
>> the companion services still rely on them (rabbit/mysql/apache/etc) so
>> we're really talking about services that currently use oslo.
>
> Thanks for the explanation!
>
> So in the future, you expect a node in a clustered OpenStack service to be 
> deployed and run as a container, and then that node queries a centralized 
> etcd (or other) k/v store to load config options. And other services running 
> in the (container? cluster?) will load config from local text files managed 
> in some other way.

No the goal is in the etcd mode, that it  may not be necessary to load
the config files locally at all.  That being said there would still be
support for having some configuration from a file and optionally
provide a kv store as another config point.  'service --config-file
/etc/service/service.conf --config-etcd proto://ip:port/slug'

>
> No wait. It's not the *services* that will load the config from a kv 
> store--it's the config management system? So in the process of deploying a 
> new container instance of a particular service, the deployment tool will pull 
> the right values out of the kv system and inject those into the container, 
> I'm guessing as a local text file that the service loads as normal?
>

No the thought is to have the services pull their configs from the kv
store via oslo.config.  The point is hopefully to not require
configuration files at all for containers.  The container would get
where to pull it's configs from (ie. http://11.1.1.1:2730/magic/ or
/etc/myconfigs/).  At that point it just becomes another place to load
configurations from via oslo.config.  Configuration management comes
in as a way to load the configs either as a file or into etcd.  Many
operators (and deployment tools) are already using some form of
configuration management so if we can integrate in a kv store output
option, adoption becomes much easier than making everyone start from
scratch.

> This means you could have some (OpenStack?) service for inventory management 
> (like Karbor) that is seeding the kv store, the cloud infrastructure software 
> itself is "cloud aware" and queries the central distributed kv system for the 
> correct-right-now config options, and the cloud service itself gets all the 
> benefits of dynamic scaling of available hardware resources. That's pretty 
> cool. Add hardware to the inventory, the cloud infra itself expands to make 
> it available. Hardware fails, and the cloud infra resizes to adjust. Apps 
> running on the infra keep doing their thing consuming the resources. It's 
> clouds all the way down :-)
>
> Despite sounding pretty interesting, it also sounds like a lot of extra 
> complexity. Maybe it's worth it. I don't know.
>

Yea there's extra complexity at least in the
deployment/management/monitoring of the new service or maybe not.
Keeping configuration files synced across 1000s of nodes (or
containers) can be just as hard however.

> Thanks again for the explanation.
>
>
> --John
>
>
>
>
>>
>> Thanks,
>> -Alex
>>
>>>
>>> --John
>>>
>>>
>>>
>>>
>>> On 21 Mar 2017, at 14:26, Davanum Srinivas wrote:
>>>
 Jay,

 the /v3alpha HTTP API  (grpc-gateway) supports watch
 https://coreos.com/etcd/docs/latest/dev-guide/apispec/swagger/rpc.swagger.json

 -- Dims

 On Tue, Mar 21, 2017 at 5:22 PM, Jay Pipes  wrote:
> On 03/21/2017 04:29 PM, Clint Byrum wrote:
>>
>> Excerpts from Doug Hellmann's message of 2017-03-15 15:35:13 -0400:
>>>
>>> Excerpts from Thomas Herve's message of 2017-03-15 09:41:16 +0100:

 On Wed, Mar 15, 2017 at 12:05 AM, Joshua Harlow 

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[John Dickinson]

On 21 Mar 2017, at 15:34, Alex Schultz wrote:

> On Tue, Mar 21, 2017 at 3:45 PM, John Dickinson  wrote:
>> I've been following this thread, but I must admit I seem to have missed 
>> something.
>>
>> What problem is being solved by storing per-server service configuration 
>> options in an external distributed CP system that is currently not possible 
>> with the existing pattern of using local text files?
>>
>
> This effort is partially to help the path to containerization where we
> are delivering the service code via container but don't want to
> necessarily deliver the configuration in the same fashion.  It's about
> ease of configuration where moving service -> config files (on many
> hosts/containers) to service -> config via etcd (single source
> cluster).  It's also about an alternative to configuration management
> where today we have many tools handling the files in various ways
> (templates, from repo, via code providers) and trying to come to a
> more unified way of representing the configuration such that the end
> result is the same for every deployment tool.  All tools load configs
> into $place and services can be configured to talk to $place.  It
> should be noted that configuration files won't go away because many of
> the companion services still rely on them (rabbit/mysql/apache/etc) so
> we're really talking about services that currently use oslo.

Thanks for the explanation!

So in the future, you expect a node in a clustered OpenStack service to be 
deployed and run as a container, and then that node queries a centralized etcd 
(or other) k/v store to load config options. And other services running in the 
(container? cluster?) will load config from local text files managed in some 
other way.

No wait. It's not the *services* that will load the config from a kv 
store--it's the config management system? So in the process of deploying a new 
container instance of a particular service, the deployment tool will pull the 
right values out of the kv system and inject those into the container, I'm 
guessing as a local text file that the service loads as normal?

This means you could have some (OpenStack?) service for inventory management 
(like Karbor) that is seeding the kv store, the cloud infrastructure software 
itself is "cloud aware" and queries the central distributed kv system for the 
correct-right-now config options, and the cloud service itself gets all the 
benefits of dynamic scaling of available hardware resources. That's pretty 
cool. Add hardware to the inventory, the cloud infra itself expands to make it 
available. Hardware fails, and the cloud infra resizes to adjust. Apps running 
on the infra keep doing their thing consuming the resources. It's clouds all 
the way down :-)

Despite sounding pretty interesting, it also sounds like a lot of extra 
complexity. Maybe it's worth it. I don't know.

Thanks again for the explanation.


--John




>
> Thanks,
> -Alex
>
>>
>> --John
>>
>>
>>
>>
>> On 21 Mar 2017, at 14:26, Davanum Srinivas wrote:
>>
>>> Jay,
>>>
>>> the /v3alpha HTTP API  (grpc-gateway) supports watch
>>> https://coreos.com/etcd/docs/latest/dev-guide/apispec/swagger/rpc.swagger.json
>>>
>>> -- Dims
>>>
>>> On Tue, Mar 21, 2017 at 5:22 PM, Jay Pipes  wrote:
 On 03/21/2017 04:29 PM, Clint Byrum wrote:
>
> Excerpts from Doug Hellmann's message of 2017-03-15 15:35:13 -0400:
>>
>> Excerpts from Thomas Herve's message of 2017-03-15 09:41:16 +0100:
>>>
>>> On Wed, Mar 15, 2017 at 12:05 AM, Joshua Harlow 
>>> wrote:
>>>
 * How does reloading work (does it)?
>>>
>>>
>>> No. There is nothing that we can do in oslo that will make services
>>> magically reload configuration. It's also unclear to me if that's
>>> something to do. In a containerized environment, wouldn't it be
>>> simpler to deploy new services? Otherwise, supporting signal based
>>> reload as we do today should be trivial.
>>
>>
>> Reloading works today with files, that's why the question is important
>> to think through. There is a special flag to set on options that are
>> "mutable" and then there are functions within oslo.config to reload.
>> Those are usually triggered when a service gets a SIGHUP or something
>> similar.
>>
>> We need to decide what happens to a service's config when that API
>> is used and the backend is etcd. Maybe nothing, because every time
>> any config option is accessed the read goes all the way through to
>> etcd? Maybe a warning is logged because we don't support reloads?
>> Maybe an error is logged? Or maybe we flush the local cache and start
>> reading from etcd on future accesses?
>>
>
> etcd provides the ability to "watch" keys. So one would start a thread
> that just watches the keys you want to reload on, and when they change
> that thread will see a response and can 

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Alex Schultz]

On Tue, Mar 21, 2017 at 3:45 PM, John Dickinson  wrote:
> I've been following this thread, but I must admit I seem to have missed 
> something.
>
> What problem is being solved by storing per-server service configuration 
> options in an external distributed CP system that is currently not possible 
> with the existing pattern of using local text files?
>

This effort is partially to help the path to containerization where we
are delivering the service code via container but don't want to
necessarily deliver the configuration in the same fashion.  It's about
ease of configuration where moving service -> config files (on many
hosts/containers) to service -> config via etcd (single source
cluster).  It's also about an alternative to configuration management
where today we have many tools handling the files in various ways
(templates, from repo, via code providers) and trying to come to a
more unified way of representing the configuration such that the end
result is the same for every deployment tool.  All tools load configs
into $place and services can be configured to talk to $place.  It
should be noted that configuration files won't go away because many of
the companion services still rely on them (rabbit/mysql/apache/etc) so
we're really talking about services that currently use oslo.

Thanks,
-Alex

>
> --John
>
>
>
>
> On 21 Mar 2017, at 14:26, Davanum Srinivas wrote:
>
>> Jay,
>>
>> the /v3alpha HTTP API  (grpc-gateway) supports watch
>> https://coreos.com/etcd/docs/latest/dev-guide/apispec/swagger/rpc.swagger.json
>>
>> -- Dims
>>
>> On Tue, Mar 21, 2017 at 5:22 PM, Jay Pipes  wrote:
>>> On 03/21/2017 04:29 PM, Clint Byrum wrote:

 Excerpts from Doug Hellmann's message of 2017-03-15 15:35:13 -0400:
>
> Excerpts from Thomas Herve's message of 2017-03-15 09:41:16 +0100:
>>
>> On Wed, Mar 15, 2017 at 12:05 AM, Joshua Harlow 
>> wrote:
>>
>>> * How does reloading work (does it)?
>>
>>
>> No. There is nothing that we can do in oslo that will make services
>> magically reload configuration. It's also unclear to me if that's
>> something to do. In a containerized environment, wouldn't it be
>> simpler to deploy new services? Otherwise, supporting signal based
>> reload as we do today should be trivial.
>
>
> Reloading works today with files, that's why the question is important
> to think through. There is a special flag to set on options that are
> "mutable" and then there are functions within oslo.config to reload.
> Those are usually triggered when a service gets a SIGHUP or something
> similar.
>
> We need to decide what happens to a service's config when that API
> is used and the backend is etcd. Maybe nothing, because every time
> any config option is accessed the read goes all the way through to
> etcd? Maybe a warning is logged because we don't support reloads?
> Maybe an error is logged? Or maybe we flush the local cache and start
> reading from etcd on future accesses?
>

 etcd provides the ability to "watch" keys. So one would start a thread
 that just watches the keys you want to reload on, and when they change
 that thread will see a response and can reload appropriately.

 https://coreos.com/etcd/docs/latest/dev-guide/api_reference_v3.html
>>>
>>>
>>> Yep. Unfortunately, you won't be able to start an eventlet greenthread to
>>> watch an etcd3/gRPC key. The python grpc library is incompatible with
>>> eventlet/gevent's monkeypatching technique and causes a complete program
>>> hang if you try to communicate with the etcd3 server from a greenlet. Fun!
>>>
>>> So, either use etcd2 (the no-longer-being-worked-on HTTP API) or don't use
>>> eventlet in your client service.
>>>
>>> Best,
>>> -jay
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> --
>> Davanum Srinivas :: https://twitter.com/dims
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[John Dickinson]

I've been following this thread, but I must admit I seem to have missed 
something.

What problem is being solved by storing per-server service configuration 
options in an external distributed CP system that is currently not possible 
with the existing pattern of using local text files?


--John




On 21 Mar 2017, at 14:26, Davanum Srinivas wrote:

> Jay,
>
> the /v3alpha HTTP API  (grpc-gateway) supports watch
> https://coreos.com/etcd/docs/latest/dev-guide/apispec/swagger/rpc.swagger.json
>
> -- Dims
>
> On Tue, Mar 21, 2017 at 5:22 PM, Jay Pipes  wrote:
>> On 03/21/2017 04:29 PM, Clint Byrum wrote:
>>>
>>> Excerpts from Doug Hellmann's message of 2017-03-15 15:35:13 -0400:

 Excerpts from Thomas Herve's message of 2017-03-15 09:41:16 +0100:
>
> On Wed, Mar 15, 2017 at 12:05 AM, Joshua Harlow 
> wrote:
>
>> * How does reloading work (does it)?
>
>
> No. There is nothing that we can do in oslo that will make services
> magically reload configuration. It's also unclear to me if that's
> something to do. In a containerized environment, wouldn't it be
> simpler to deploy new services? Otherwise, supporting signal based
> reload as we do today should be trivial.


 Reloading works today with files, that's why the question is important
 to think through. There is a special flag to set on options that are
 "mutable" and then there are functions within oslo.config to reload.
 Those are usually triggered when a service gets a SIGHUP or something
 similar.

 We need to decide what happens to a service's config when that API
 is used and the backend is etcd. Maybe nothing, because every time
 any config option is accessed the read goes all the way through to
 etcd? Maybe a warning is logged because we don't support reloads?
 Maybe an error is logged? Or maybe we flush the local cache and start
 reading from etcd on future accesses?

>>>
>>> etcd provides the ability to "watch" keys. So one would start a thread
>>> that just watches the keys you want to reload on, and when they change
>>> that thread will see a response and can reload appropriately.
>>>
>>> https://coreos.com/etcd/docs/latest/dev-guide/api_reference_v3.html
>>
>>
>> Yep. Unfortunately, you won't be able to start an eventlet greenthread to
>> watch an etcd3/gRPC key. The python grpc library is incompatible with
>> eventlet/gevent's monkeypatching technique and causes a complete program
>> hang if you try to communicate with the etcd3 server from a greenlet. Fun!
>>
>> So, either use etcd2 (the no-longer-being-worked-on HTTP API) or don't use
>> eventlet in your client service.
>>
>> Best,
>> -jay
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> -- 
> Davanum Srinivas :: https://twitter.com/dims
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Davanum Srinivas]

Jay,

the /v3alpha HTTP API  (grpc-gateway) supports watch
https://coreos.com/etcd/docs/latest/dev-guide/apispec/swagger/rpc.swagger.json

-- Dims

On Tue, Mar 21, 2017 at 5:22 PM, Jay Pipes  wrote:
> On 03/21/2017 04:29 PM, Clint Byrum wrote:
>>
>> Excerpts from Doug Hellmann's message of 2017-03-15 15:35:13 -0400:
>>>
>>> Excerpts from Thomas Herve's message of 2017-03-15 09:41:16 +0100:

 On Wed, Mar 15, 2017 at 12:05 AM, Joshua Harlow 
 wrote:

> * How does reloading work (does it)?


 No. There is nothing that we can do in oslo that will make services
 magically reload configuration. It's also unclear to me if that's
 something to do. In a containerized environment, wouldn't it be
 simpler to deploy new services? Otherwise, supporting signal based
 reload as we do today should be trivial.
>>>
>>>
>>> Reloading works today with files, that's why the question is important
>>> to think through. There is a special flag to set on options that are
>>> "mutable" and then there are functions within oslo.config to reload.
>>> Those are usually triggered when a service gets a SIGHUP or something
>>> similar.
>>>
>>> We need to decide what happens to a service's config when that API
>>> is used and the backend is etcd. Maybe nothing, because every time
>>> any config option is accessed the read goes all the way through to
>>> etcd? Maybe a warning is logged because we don't support reloads?
>>> Maybe an error is logged? Or maybe we flush the local cache and start
>>> reading from etcd on future accesses?
>>>
>>
>> etcd provides the ability to "watch" keys. So one would start a thread
>> that just watches the keys you want to reload on, and when they change
>> that thread will see a response and can reload appropriately.
>>
>> https://coreos.com/etcd/docs/latest/dev-guide/api_reference_v3.html
>
>
> Yep. Unfortunately, you won't be able to start an eventlet greenthread to
> watch an etcd3/gRPC key. The python grpc library is incompatible with
> eventlet/gevent's monkeypatching technique and causes a complete program
> hang if you try to communicate with the etcd3 server from a greenlet. Fun!
>
> So, either use etcd2 (the no-longer-being-worked-on HTTP API) or don't use
> eventlet in your client service.
>
> Best,
> -jay
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Jay Pipes]

On 03/21/2017 04:29 PM, Clint Byrum wrote:

Excerpts from Doug Hellmann's message of 2017-03-15 15:35:13 -0400:

Excerpts from Thomas Herve's message of 2017-03-15 09:41:16 +0100:

On Wed, Mar 15, 2017 at 12:05 AM, Joshua Harlow  wrote:


* How does reloading work (does it)?


No. There is nothing that we can do in oslo that will make services
magically reload configuration. It's also unclear to me if that's
something to do. In a containerized environment, wouldn't it be
simpler to deploy new services? Otherwise, supporting signal based
reload as we do today should be trivial.


Reloading works today with files, that's why the question is important
to think through. There is a special flag to set on options that are
"mutable" and then there are functions within oslo.config to reload.
Those are usually triggered when a service gets a SIGHUP or something similar.

We need to decide what happens to a service's config when that API
is used and the backend is etcd. Maybe nothing, because every time
any config option is accessed the read goes all the way through to
etcd? Maybe a warning is logged because we don't support reloads?
Maybe an error is logged? Or maybe we flush the local cache and start
reading from etcd on future accesses?



etcd provides the ability to "watch" keys. So one would start a thread
that just watches the keys you want to reload on, and when they change
that thread will see a response and can reload appropriately.

https://coreos.com/etcd/docs/latest/dev-guide/api_reference_v3.html


Yep. Unfortunately, you won't be able to start an eventlet greenthread 
to watch an etcd3/gRPC key. The python grpc library is incompatible with 
eventlet/gevent's monkeypatching technique and causes a complete program 
hang if you try to communicate with the etcd3 server from a greenlet. Fun!


So, either use etcd2 (the no-longer-being-worked-on HTTP API) or don't 
use eventlet in your client service.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Clint Byrum]

Excerpts from Sean Dague's message of 2017-03-15 08:54:55 -0400:
> On 03/15/2017 02:16 AM, Clint Byrum wrote:
> > Excerpts from Monty Taylor's message of 2017-03-15 04:36:24 +0100:
> >> On 03/14/2017 06:04 PM, Davanum Srinivas wrote:
> >>> Team,
> >>>
> >>> So one more thing popped up again on IRC:
> >>> https://etherpad.openstack.org/p/oslo.config_etcd_backend
> >>>
> >>> What do you think? interested in this work?
> >>>
> >>> Thanks,
> >>> Dims
> >>>
> >>> PS: Between this thread and the other one about Tooz/DLM and
> >>> os-lively, we can probably make a good case to add etcd as a base
> >>> always-on service.
> >>
> >> As I mentioned in the other thread, there was specific and strong
> >> anti-etcd sentiment in Tokyo which is why we decided to use an
> >> abstraction. I continue to be in favor of us having one known service in
> >> this space, but I do think that it's important to revisit that decision
> >> fully and in context of the concerns that were raised when we tried to
> >> pick one last time.
> >>
> >> It's worth noting that there is nothing particularly etcd-ish about
> >> storing config that couldn't also be done with zk and thus just be an
> >> additional api call or two added to Tooz with etcd and zk drivers for it.
> >>
> > 
> > Combine that thought with the "please have an ingest/export" thought,
> > and I think you have a pretty operator-friendly transition path. Would
> > be pretty great to have a release of OpenStack that just lets you add
> > an '[etcd]', or '[config-service]' section maybe, to your config files,
> > and then once you've fully migrated everything, lets you delete all the
> > other sections. Then the admin nodes still have the full configs and
> > one can just edit configs in git and roll them out by ingesting.
> > 
> > (Then the magical rainbow fairy ponies teach our services to watch their
> > config service for changes and restart themselves).
> 
> Make sure to add:
> 
> ... (after fully quiescing, when they are not processing any inflight
> work, when they are part of a pool so that they can be rolling restarted
> without impacting other services trying to connect to them, with a
> rollback to past config should the new config cause a crash).
> 
> There are a ton of really interesting things about a network registry,
> that makes many things easier. However, from an operational point of
> view I would be concerned about the idea of services restarting
> themselves in a non orchestrated manner. Or that a single key set in the
> registry triggers a complete reboot of the cluster. It's definitely less
> clear to understand the linkage of the action that took down your cloud
> and why when the operator isn't explicit about "and restart this service
> now".
> 

It's big and powerful and scary. That's for sure. But it's not _that_
different than an Ansible playbook in its ability to massively complicate
or uncomplicate your life with a tiny amount of code. :)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Clint Byrum]

Excerpts from Doug Hellmann's message of 2017-03-15 15:35:13 -0400:
> Excerpts from Thomas Herve's message of 2017-03-15 09:41:16 +0100:
> > On Wed, Mar 15, 2017 at 12:05 AM, Joshua Harlow  
> > wrote:
> > 
> > > * How does reloading work (does it)?
> > 
> > No. There is nothing that we can do in oslo that will make services
> > magically reload configuration. It's also unclear to me if that's
> > something to do. In a containerized environment, wouldn't it be
> > simpler to deploy new services? Otherwise, supporting signal based
> > reload as we do today should be trivial.
> 
> Reloading works today with files, that's why the question is important
> to think through. There is a special flag to set on options that are
> "mutable" and then there are functions within oslo.config to reload.
> Those are usually triggered when a service gets a SIGHUP or something similar.
> 
> We need to decide what happens to a service's config when that API
> is used and the backend is etcd. Maybe nothing, because every time
> any config option is accessed the read goes all the way through to
> etcd? Maybe a warning is logged because we don't support reloads?
> Maybe an error is logged? Or maybe we flush the local cache and start
> reading from etcd on future accesses?
> 

etcd provides the ability to "watch" keys. So one would start a thread
that just watches the keys you want to reload on, and when they change
that thread will see a response and can reload appropriately.

https://coreos.com/etcd/docs/latest/dev-guide/api_reference_v3.html

see "service Watch"

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Emilien Macchi]

On Wed, Mar 15, 2017 at 3:53 PM, Paul Belanger  wrote:
> On Wed, Mar 15, 2017 at 09:41:16AM +0100, Thomas Herve wrote:
>> On Wed, Mar 15, 2017 at 12:05 AM, Joshua Harlow  
>> wrote:
>>
>> > * How does reloading work (does it)?
>>
>> No. There is nothing that we can do in oslo that will make services
>> magically reload configuration. It's also unclear to me if that's
>> something to do. In a containerized environment, wouldn't it be
>> simpler to deploy new services? Otherwise, supporting signal based
>> reload as we do today should be trivial.
>>
>> > * What's the operational experience (editing a ini file is about the lowest
>> > bar we can possible get to, for better and/or worse).
>> >
>> > * Does this need to be a new oslo.config backend or is it better suited by
>> > something like the following (external programs loop)::
>> >
>> >etcd_client = make_etcd_client(args)
>> >while True:
>> >has_changed = etcd_client.get_new_config("/blahblah") # or use a
>> > watch
>> >if has_changed:
>> >   fetch_and_write_ini_file(etcd_client)
>> >   trigger_reload()
>> >time.sleep(args.wait)
>>
>> That's confd: https://github.com/kelseyhightower/confd/ . Bonus
>> points; it supports a ton of other backends. One solution is to
>> provide templates and documentation to use confd with OpenStack.
>>
> ++
>
> Lets not get into the business of writing cfgmgmt tools in openstack, but 
> reuse
> what exists today.
>
> oslo.config should just write to etcd, and other tools would be used, confd, 
> to
> trigger things.

I agree with you and that's what we're investigating in TripleO:
http://lists.openstack.org/pipermail/openstack-dev/2017-March/114246.html

Both threads are related, I would love to hear some feedback on it or
directly in the etherpad:
https://etherpad.openstack.org/p/tripleo-etcd-transition

Thanks,

>
>> --
>> Thomas
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Paul Belanger]

On Wed, Mar 15, 2017 at 09:41:16AM +0100, Thomas Herve wrote:
> On Wed, Mar 15, 2017 at 12:05 AM, Joshua Harlow  wrote:
> 
> > * How does reloading work (does it)?
> 
> No. There is nothing that we can do in oslo that will make services
> magically reload configuration. It's also unclear to me if that's
> something to do. In a containerized environment, wouldn't it be
> simpler to deploy new services? Otherwise, supporting signal based
> reload as we do today should be trivial.
> 
> > * What's the operational experience (editing a ini file is about the lowest
> > bar we can possible get to, for better and/or worse).
> >
> > * Does this need to be a new oslo.config backend or is it better suited by
> > something like the following (external programs loop)::
> >
> >etcd_client = make_etcd_client(args)
> >while True:
> >has_changed = etcd_client.get_new_config("/blahblah") # or use a
> > watch
> >if has_changed:
> >   fetch_and_write_ini_file(etcd_client)
> >   trigger_reload()
> >time.sleep(args.wait)
> 
> That's confd: https://github.com/kelseyhightower/confd/ . Bonus
> points; it supports a ton of other backends. One solution is to
> provide templates and documentation to use confd with OpenStack.
> 
++

Lets not get into the business of writing cfgmgmt tools in openstack, but reuse
what exists today.

oslo.config should just write to etcd, and other tools would be used, confd, to
trigger things.

> -- 
> Thomas
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Doug Hellmann]

Excerpts from Monty Taylor's message of 2017-03-15 04:36:24 +0100:
> On 03/14/2017 06:04 PM, Davanum Srinivas wrote:
> > Team,
> > 
> > So one more thing popped up again on IRC:
> > https://etherpad.openstack.org/p/oslo.config_etcd_backend
> > 
> > What do you think? interested in this work?
> > 
> > Thanks,
> > Dims
> > 
> > PS: Between this thread and the other one about Tooz/DLM and
> > os-lively, we can probably make a good case to add etcd as a base
> > always-on service.
> 
> As I mentioned in the other thread, there was specific and strong
> anti-etcd sentiment in Tokyo which is why we decided to use an
> abstraction. I continue to be in favor of us having one known service in
> this space, but I do think that it's important to revisit that decision
> fully and in context of the concerns that were raised when we tried to
> pick one last time.
> 
> It's worth noting that there is nothing particularly etcd-ish about
> storing config that couldn't also be done with zk and thus just be an
> additional api call or two added to Tooz with etcd and zk drivers for it.
> 

The fun* thing about working with these libraries is managing the
interdependencies. If we're going to have an abstraction library that
provides configuration options for seeing the backend, like we do in
oslo.db and olso.messaging, then the configuration library can't use it
or we have a circular dependency.

Luckily, tooz does not currently use oslo.config. So, oslo.config could
use tooz and we could create an oslo.dlm library with a shallow
interface mapping config options to tooz calls to open connections or
whatever we need from tooz in an application. Then apps could use
oslo.dlm instead of calling into tooz directly and the configuration of
the backend would be hidden from the application developer.

Doug

* your definition of "fun" may be different than mine

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Doug Hellmann]

Excerpts from Thomas Herve's message of 2017-03-15 09:41:16 +0100:
> On Wed, Mar 15, 2017 at 12:05 AM, Joshua Harlow  wrote:
> 
> > * How does reloading work (does it)?
> 
> No. There is nothing that we can do in oslo that will make services
> magically reload configuration. It's also unclear to me if that's
> something to do. In a containerized environment, wouldn't it be
> simpler to deploy new services? Otherwise, supporting signal based
> reload as we do today should be trivial.

Reloading works today with files, that's why the question is important
to think through. There is a special flag to set on options that are
"mutable" and then there are functions within oslo.config to reload.
Those are usually triggered when a service gets a SIGHUP or something similar.

We need to decide what happens to a service's config when that API
is used and the backend is etcd. Maybe nothing, because every time
any config option is accessed the read goes all the way through to
etcd? Maybe a warning is logged because we don't support reloads?
Maybe an error is logged? Or maybe we flush the local cache and start
reading from etcd on future accesses?

> > * What's the operational experience (editing a ini file is about the lowest
> > bar we can possible get to, for better and/or worse).
> >
> > * Does this need to be a new oslo.config backend or is it better suited by
> > something like the following (external programs loop)::
> >
> >etcd_client = make_etcd_client(args)
> >while True:
> >has_changed = etcd_client.get_new_config("/blahblah") # or use a
> > watch
> >if has_changed:
> >   fetch_and_write_ini_file(etcd_client)
> >   trigger_reload()
> >time.sleep(args.wait)
> 
> That's confd: https://github.com/kelseyhightower/confd/ . Bonus
> points; it supports a ton of other backends. One solution is to
> provide templates and documentation to use confd with OpenStack.

That sounds like it might also be useful, but is probably a separate
issue.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Paul Belanger]

On Wed, Mar 15, 2017 at 04:23:09AM +0100, Monty Taylor wrote:
> On 03/15/2017 12:05 AM, Joshua Harlow wrote:
> > So just fyi, this has been talked about before (but prob in context of
> > zookeeper or various other pluggable config backends).
> > 
> > Some links:
> > 
> > - https://review.openstack.org/#/c/243114/
> > - https://review.openstack.org/#/c/243182/
> > - https://blueprints.launchpad.net/oslo.config/+spec/oslo-config-db
> > - https://review.openstack.org/#/c/130047/
> > 
> > I think the general questions that seem to reappear are around the
> > following:
> > 
> > * How does reloading work (does it)?
> > 
> > * What's the operational experience (editing a ini file is about the
> > lowest bar we can possible get to, for better and/or worse).
> 
> As a person who operates many softwares (but who does not necessarily
> operate OpenStack specifically) I will say that services that store
> their config in a service that do not have an injest/update facility
> from file are a GIANT PITA to deal with. Config management is great at
> laying down config files. It _can_ put things into services, but that's
> almost always more work.
> 
> Which is my way of saying - neat, but please please please whoever
> writes this make a simple facility that will let someone plop config
> into a file on disk and get that noticed and slurped into the config
> service. A one-liner command line tool that one runs on the config file
> to splat into the config service would be fine.
> 
So much this! As an operator, I am fine plopping a config files down on a remote
node and understand what that means to my workflow.

Opt out by default! :)

> > * Does this need to be a new oslo.config backend or is it better suited
> > by something like the following (external programs loop)::
> > 
> >etcd_client = make_etcd_client(args)
> >while True:
> >has_changed = etcd_client.get_new_config("/blahblah") # or use a
> > watch
> >if has_changed:
> >   fetch_and_write_ini_file(etcd_client)
> >   trigger_reload()
> >time.sleep(args.wait)
> > 
> > * Is an external loop better (maybe, maybe not?)
> > 
> > Pretty sure there are some etherpad discussions around this also somewhere.
> > 
> > Clint Byrum wrote:
> >> Excerpts from Davanum Srinivas's message of 2017-03-14 13:04:37 -0400:
> >>> Team,
> >>>
> >>> So one more thing popped up again on IRC:
> >>> https://etherpad.openstack.org/p/oslo.config_etcd_backend
> >>>
> >>> What do you think? interested in this work?
> >>>
> >>> Thanks,
> >>> Dims
> >>>
> >>> PS: Between this thread and the other one about Tooz/DLM and
> >>> os-lively, we can probably make a good case to add etcd as a base
> >>> always-on service.
> >>>
> >>
> >> This is a cool idea, and I think we should do it.
> >>
> >> A few loose ends I'd like to see in a spec:
> >>
> >> * Security Security Security. (Hoping if I say it 3 times a real
> >>security person will appear and ask the hard questions).
> >> * Explain clearly how operators would inspect, edit, and diff their
> >>configs.
> >>
> >> __
> >>
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Sean Dague]

On 03/15/2017 02:16 AM, Clint Byrum wrote:
> Excerpts from Monty Taylor's message of 2017-03-15 04:36:24 +0100:
>> On 03/14/2017 06:04 PM, Davanum Srinivas wrote:
>>> Team,
>>>
>>> So one more thing popped up again on IRC:
>>> https://etherpad.openstack.org/p/oslo.config_etcd_backend
>>>
>>> What do you think? interested in this work?
>>>
>>> Thanks,
>>> Dims
>>>
>>> PS: Between this thread and the other one about Tooz/DLM and
>>> os-lively, we can probably make a good case to add etcd as a base
>>> always-on service.
>>
>> As I mentioned in the other thread, there was specific and strong
>> anti-etcd sentiment in Tokyo which is why we decided to use an
>> abstraction. I continue to be in favor of us having one known service in
>> this space, but I do think that it's important to revisit that decision
>> fully and in context of the concerns that were raised when we tried to
>> pick one last time.
>>
>> It's worth noting that there is nothing particularly etcd-ish about
>> storing config that couldn't also be done with zk and thus just be an
>> additional api call or two added to Tooz with etcd and zk drivers for it.
>>
> 
> Combine that thought with the "please have an ingest/export" thought,
> and I think you have a pretty operator-friendly transition path. Would
> be pretty great to have a release of OpenStack that just lets you add
> an '[etcd]', or '[config-service]' section maybe, to your config files,
> and then once you've fully migrated everything, lets you delete all the
> other sections. Then the admin nodes still have the full configs and
> one can just edit configs in git and roll them out by ingesting.
> 
> (Then the magical rainbow fairy ponies teach our services to watch their
> config service for changes and restart themselves).

Make sure to add:

... (after fully quiescing, when they are not processing any inflight
work, when they are part of a pool so that they can be rolling restarted
without impacting other services trying to connect to them, with a
rollback to past config should the new config cause a crash).

There are a ton of really interesting things about a network registry,
that makes many things easier. However, from an operational point of
view I would be concerned about the idea of services restarting
themselves in a non orchestrated manner. Or that a single key set in the
registry triggers a complete reboot of the cluster. It's definitely less
clear to understand the linkage of the action that took down your cloud
and why when the operator isn't explicit about "and restart this service
now".

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[John Garbutt]

On 15 March 2017 at 03:23, Monty Taylor  wrote:
> On 03/15/2017 12:05 AM, Joshua Harlow wrote:
>> So just fyi, this has been talked about before (but prob in context of
>> zookeeper or various other pluggable config backends).
>>
>> Some links:
>>
>> - https://review.openstack.org/#/c/243114/
>> - https://review.openstack.org/#/c/243182/
>> - https://blueprints.launchpad.net/oslo.config/+spec/oslo-config-db
>> - https://review.openstack.org/#/c/130047/
>>
>> I think the general questions that seem to reappear are around the
>> following:
>>
>> * How does reloading work (does it)?
>>
>> * What's the operational experience (editing a ini file is about the
>> lowest bar we can possible get to, for better and/or worse).
>
> As a person who operates many softwares (but who does not necessarily
> operate OpenStack specifically) I will say that services that store
> their config in a service that do not have an injest/update facility
> from file are a GIANT PITA to deal with. Config management is great at
> laying down config files. It _can_ put things into services, but that's
> almost always more work.
>
> Which is my way of saying - neat, but please please please whoever
> writes this make a simple facility that will let someone plop config
> into a file on disk and get that noticed and slurped into the config
> service. A one-liner command line tool that one runs on the config file
> to splat into the config service would be fine.

+1 for keeping the simple use a config file working well.

(+1 for trying other things too, if they don't break the simple way)

Thanks,
John

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Thierry Carrez]

Joshua Harlow wrote:
> Monty Taylor wrote:
>> On 03/14/2017 06:04 PM, Davanum Srinivas wrote:
>>> PS: Between this thread and the other one about Tooz/DLM and
>>> os-lively, we can probably make a good case to add etcd as a base
>>> always-on service.
>>
>> As I mentioned in the other thread, there was specific and strong
>> anti-etcd sentiment in Tokyo which is why we decided to use an
>> abstraction. I continue to be in favor of us having one known service in
>> this space, but I do think that it's important to revisit that decision
>> fully and in context of the concerns that were raised when we tried to
>> pick one last time.
> 
> I'm in agreement with this.
> 
> I don't mind tooz either (it's good at what it is for) since I took a
> part in creating it... Given that I can't help but wonder how nice it
> would be to pick one (etcd, zookeeper, consul?) and just do nice things
> with it (perhaps u know even work with the etcd or zookeeper or consul
> developers [depending on which one we pick] on features and bug fixes
> and such).

If I remember correctly:

- the root of the anti-etcd sentiment was that it was less featureful
than ZooKeeper/Consul (I remember arguments that ZooKeeper was the only
one to do DLM correctly)

- the root of the anti-ZK sentiment was that it was a Java stack, which
comes with some extra complexity, some operational baggage and some
licensing concerns.

so it felt like enabling all three through an abstraction library would
give operators choice, and the features we needed actually led
themselves to abstraction well.

Now, a number of things happened since then:

- etcd3 is widely seen as doing DLM features as well as ZooKeeper or
Consul, and has closed (some say even surpassed) the performance gap

- as each day passes we (OpenStack) are growing a stronger
complementarity with the Kubernetes ecosystem, which relies on etcd as a
base service extensively

So unless someone can make an argument that Zookeeper has technical
benefits that etcd3 can't provide (and therefore should be kept as one
option under an abstraction layer), it's still time to directly pick
etcd3 as a base service. Skip the expand step if unnecessary, and go
directly to the contract phase.

PS: Nobody mentions Consul anymore -- In all cases I don't think it
provides any extra value to us over etcd3 that would justify the cost of
maintaining an abstraction driver for it.

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Thomas Herve]

On Wed, Mar 15, 2017 at 12:05 AM, Joshua Harlow  wrote:

> * How does reloading work (does it)?

No. There is nothing that we can do in oslo that will make services
magically reload configuration. It's also unclear to me if that's
something to do. In a containerized environment, wouldn't it be
simpler to deploy new services? Otherwise, supporting signal based
reload as we do today should be trivial.

> * What's the operational experience (editing a ini file is about the lowest
> bar we can possible get to, for better and/or worse).
>
> * Does this need to be a new oslo.config backend or is it better suited by
> something like the following (external programs loop)::
>
>etcd_client = make_etcd_client(args)
>while True:
>has_changed = etcd_client.get_new_config("/blahblah") # or use a
> watch
>if has_changed:
>   fetch_and_write_ini_file(etcd_client)
>   trigger_reload()
>time.sleep(args.wait)

That's confd: https://github.com/kelseyhightower/confd/ . Bonus
points; it supports a ton of other backends. One solution is to
provide templates and documentation to use confd with OpenStack.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Clint Byrum]

Excerpts from Monty Taylor's message of 2017-03-15 04:36:24 +0100:
> On 03/14/2017 06:04 PM, Davanum Srinivas wrote:
> > Team,
> > 
> > So one more thing popped up again on IRC:
> > https://etherpad.openstack.org/p/oslo.config_etcd_backend
> > 
> > What do you think? interested in this work?
> > 
> > Thanks,
> > Dims
> > 
> > PS: Between this thread and the other one about Tooz/DLM and
> > os-lively, we can probably make a good case to add etcd as a base
> > always-on service.
> 
> As I mentioned in the other thread, there was specific and strong
> anti-etcd sentiment in Tokyo which is why we decided to use an
> abstraction. I continue to be in favor of us having one known service in
> this space, but I do think that it's important to revisit that decision
> fully and in context of the concerns that were raised when we tried to
> pick one last time.
> 
> It's worth noting that there is nothing particularly etcd-ish about
> storing config that couldn't also be done with zk and thus just be an
> additional api call or two added to Tooz with etcd and zk drivers for it.
> 

Combine that thought with the "please have an ingest/export" thought,
and I think you have a pretty operator-friendly transition path. Would
be pretty great to have a release of OpenStack that just lets you add
an '[etcd]', or '[config-service]' section maybe, to your config files,
and then once you've fully migrated everything, lets you delete all the
other sections. Then the admin nodes still have the full configs and
one can just edit configs in git and roll them out by ingesting.

(Then the magical rainbow fairy ponies teach our services to watch their
config service for changes and restart themselves).

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Clint Byrum]

Excerpts from Fox, Kevin M's message of 2017-03-15 00:10:24 +:
> +1 for having the option. I think in general it is a great idea.
> 
> I think there are some devils in the implementation. How do you prevent
> a service from getting way more secrets then it strictly needs? Maybe
> this is the start of an activity thought to split out all secrets from
> config, which would be an awesome thing to do. Having secrets commingled
> with config has always rubbed me the wrong way.
> 

You'll always have at least one secret around that grants access to
the others.  Whether it is the root SSH key, or the TLS private key that
grants etcd.

This is where we might diverge into the key management discussion so we
can at least have audit trails. With config only in files, you could
use something like CloudKeep (defunct?) that would effectively make
config-not-secrets into templates that would be filled with secrets when
read from a fifo backed by a daemon that fetched secrets from Barbican.

But, we could, instead, have less moving parts by having oslo.config
provide the option to just use castellan (or oslo.keymanager if we do
rename it) to handle key management of a TLS private key for etcd auth,
and maybe an additional private key for decryption of secrets like db
passwords and such. If you don't have Barbican, you can either a) write
a castellan driver for your existing key manager, b) help finish the
Hashicorp Vault castellan driver that Robert Clark was hacking on in
ATL, or c) use the insecure on-disk one that basically makes it work
just like your httpd /etc/httpd/ssl.key dir.

Whatever is done, there's a lot of operator-unfriendliness if it's not
accompanied by nice tools to read the effective configuration. Something
like `/opt/openstack/bin/keystone-manage show-config`.

> The other issue I can think of is overridability. the current
> separate file per instantiated service has some flexibility that a
> simple implementation of just looking in etcd for the keys may not work
> for. Some, look here first, then look for overrides over here thing
> would work though.
> 

That was addressed in the original I think. This would work:

/opt/openstack/bin/nova-conductor --config=/etc/nova/nova.conf 
--etcd-url=http://foo:94895/common --etcd-url=https://foo:94895/my-hostname

As with current oslo.config behavior, later urls would override earlier
ones. I don't think anybody's suggesting doing away with configs. Just
that having them in some central micro service means having less mutable
things to worry about in your images, which is why Kolla's asking so
nicely. :)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Joshua Harlow]

Monty Taylor wrote:

On 03/14/2017 06:04 PM, Davanum Srinivas wrote:

Team,

So one more thing popped up again on IRC:
https://etherpad.openstack.org/p/oslo.config_etcd_backend

What do you think? interested in this work?

Thanks,
Dims

PS: Between this thread and the other one about Tooz/DLM and
os-lively, we can probably make a good case to add etcd as a base
always-on service.


As I mentioned in the other thread, there was specific and strong
anti-etcd sentiment in Tokyo which is why we decided to use an
abstraction. I continue to be in favor of us having one known service in
this space, but I do think that it's important to revisit that decision
fully and in context of the concerns that were raised when we tried to
pick one last time.


I'm in agreement with this.

I don't mind tooz either (it's good at what it is for) since I took a 
part in creating it... Given that I can't help but wonder how nice it 
would be to pick one (etcd, zookeeper, consul?) and just do nice things 
with it (perhaps u know even work with the etcd or zookeeper or consul 
developers [depending on which one we pick] on features and bug fixes 
and such).




It's worth noting that there is nothing particularly etcd-ish about
storing config that couldn't also be done with zk and thus just be an
additional api call or two added to Tooz with etcd and zk drivers for it.


Ya, to me zookeeper and etcd look pretty much the same now-a-days.

Which I guess is why https://github.com/coreos/zetcd ('A ZooKeeper 
"personality" for etcd) (although I'm not sure I'd want to run that, ha) 
exists as a thing.




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Monty Taylor]

On 03/14/2017 06:04 PM, Davanum Srinivas wrote:
> Team,
> 
> So one more thing popped up again on IRC:
> https://etherpad.openstack.org/p/oslo.config_etcd_backend
> 
> What do you think? interested in this work?
> 
> Thanks,
> Dims
> 
> PS: Between this thread and the other one about Tooz/DLM and
> os-lively, we can probably make a good case to add etcd as a base
> always-on service.

As I mentioned in the other thread, there was specific and strong
anti-etcd sentiment in Tokyo which is why we decided to use an
abstraction. I continue to be in favor of us having one known service in
this space, but I do think that it's important to revisit that decision
fully and in context of the concerns that were raised when we tried to
pick one last time.

It's worth noting that there is nothing particularly etcd-ish about
storing config that couldn't also be done with zk and thus just be an
additional api call or two added to Tooz with etcd and zk drivers for it.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Monty Taylor]

On 03/15/2017 12:05 AM, Joshua Harlow wrote:
> So just fyi, this has been talked about before (but prob in context of
> zookeeper or various other pluggable config backends).
> 
> Some links:
> 
> - https://review.openstack.org/#/c/243114/
> - https://review.openstack.org/#/c/243182/
> - https://blueprints.launchpad.net/oslo.config/+spec/oslo-config-db
> - https://review.openstack.org/#/c/130047/
> 
> I think the general questions that seem to reappear are around the
> following:
> 
> * How does reloading work (does it)?
> 
> * What's the operational experience (editing a ini file is about the
> lowest bar we can possible get to, for better and/or worse).

As a person who operates many softwares (but who does not necessarily
operate OpenStack specifically) I will say that services that store
their config in a service that do not have an injest/update facility
from file are a GIANT PITA to deal with. Config management is great at
laying down config files. It _can_ put things into services, but that's
almost always more work.

Which is my way of saying - neat, but please please please whoever
writes this make a simple facility that will let someone plop config
into a file on disk and get that noticed and slurped into the config
service. A one-liner command line tool that one runs on the config file
to splat into the config service would be fine.

> * Does this need to be a new oslo.config backend or is it better suited
> by something like the following (external programs loop)::
> 
>etcd_client = make_etcd_client(args)
>while True:
>has_changed = etcd_client.get_new_config("/blahblah") # or use a
> watch
>if has_changed:
>   fetch_and_write_ini_file(etcd_client)
>   trigger_reload()
>time.sleep(args.wait)
> 
> * Is an external loop better (maybe, maybe not?)
> 
> Pretty sure there are some etherpad discussions around this also somewhere.
> 
> Clint Byrum wrote:
>> Excerpts from Davanum Srinivas's message of 2017-03-14 13:04:37 -0400:
>>> Team,
>>>
>>> So one more thing popped up again on IRC:
>>> https://etherpad.openstack.org/p/oslo.config_etcd_backend
>>>
>>> What do you think? interested in this work?
>>>
>>> Thanks,
>>> Dims
>>>
>>> PS: Between this thread and the other one about Tooz/DLM and
>>> os-lively, we can probably make a good case to add etcd as a base
>>> always-on service.
>>>
>>
>> This is a cool idea, and I think we should do it.
>>
>> A few loose ends I'd like to see in a spec:
>>
>> * Security Security Security. (Hoping if I say it 3 times a real
>>security person will appear and ask the hard questions).
>> * Explain clearly how operators would inspect, edit, and diff their
>>configs.
>>
>> __
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Fox, Kevin M]

+1 for having the option. I think in general it is a great idea.

I think there are some devils in the implementation. How do you prevent a 
service from getting way more secrets then it strictly needs? Maybe this is the 
start of an activity thought to split out all secrets from config, which would 
be an awesome thing to do. Having secrets commingled with config has always 
rubbed me the wrong way.

The other issue I can think of is overridability. the current separate file per 
instantiated service has some flexibility that a simple implementation of just 
looking in etcd for the keys may not work for. Some, look here first, then look 
for overrides over here thing would work though.

Thanks,
Kevin

From: Michał Jastrzębski [inc...@gmail.com]
Sent: Tuesday, March 14, 2017 1:19 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] 
Storing configuration options in etcd(?)

So from kolla perspective that would be totally awesome. For
kolla-ansible it will make things easier, for kolla-k8s...well, it
would solve a lot (**a lot**) of issues we're dealing with.

Pretty pretty please? <:)

On 14 March 2017 at 10:48, Emilien Macchi <emil...@redhat.com> wrote:
> On Tue, Mar 14, 2017 at 1:04 PM, Davanum Srinivas <dava...@gmail.com> wrote:
>> Team,
>>
>> So one more thing popped up again on IRC:
>> https://etherpad.openstack.org/p/oslo.config_etcd_backend
>>
>> What do you think? interested in this work?
>
> Wow, it seems like our minds are crossing.
> We had this discussion at the PTG and I've seen this topic quite often
> mentioned during different sessions.
>
> It's also somehow mentioned here:
> https://etherpad.openstack.org/p/deployment-pike
> See "Allow services to make use of KV stores instead of just INI files
> for config".
>
> What Thomas is doing in the PoC is actually what we thought it could
> be the next steps forward configuration management in OpenStack in a
> way that could be shared across all tools.
> Partially related, Ben Nemec is working on a spec that would extract
> all OpenStack parameters and generate YAML files:
> https://review.openstack.org/#/c/440835/
> And we thought that we could re-use this file to inject the
> configuration into etcd.
>
> I see a connection here where :
>
> 1. With Ben's work, we would generate a list of parameters available
> in OpenStack and expose it to the User Interface of the deployment
> tool.
> 2. The deployment tool would grab inputs from users and write the
> values into etcd. The installers would also configure some parameters
> that users don't want to provide (with all the logic around).
> 3. OpenStack services would read the config directly from etcd, thanks
> to Thomas's work.
>
> That way, 1. and 3. belong to oslo.config and 2. is done by OpenStack
> deployment tools.
> Does it make sense?
>
> I see a lot of collaboration and consolidation here, in how we do
> configuration management in OpenStack. I hope we can move forward and
> find some consensus here; and why not proposing a first architecture
> for Pike.
>
> Thanks,
>
>> Thanks,
>> Dims
>>
>> PS: Between this thread and the other one about Tooz/DLM and
>> os-lively, we can probably make a good case to add etcd as a base
>> always-on service.
>>
>> --
>> Davanum Srinivas :: https://twitter.com/dims
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Emilien Macchi
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Emilien Macchi]

On Tue, Mar 14, 2017 at 6:17 PM, Clint Byrum  wrote:
> Excerpts from Davanum Srinivas's message of 2017-03-14 13:04:37 -0400:
>> Team,
>>
>> So one more thing popped up again on IRC:
>> https://etherpad.openstack.org/p/oslo.config_etcd_backend
>>
>> What do you think? interested in this work?
>>
>> Thanks,
>> Dims
>>
>> PS: Between this thread and the other one about Tooz/DLM and
>> os-lively, we can probably make a good case to add etcd as a base
>> always-on service.
>>
>
> This is a cool idea, and I think we should do it.
>
> A few loose ends I'd like to see in a spec:
>
> * Security Security Security. (Hoping if I say it 3 times a real
>   security person will appear and ask the hard questions).

I don't consider myself as a Security expert but in little knowledge:

- etcd v2 API allows authentification:
https://coreos.com/etcd/docs/latest/v2/authentication.html
- etcd supports SSL/TLS as well as authentication through client
certificates, both for clients to server as well as peer (server to
server / cluster) communication

Which sounds pretty secure at this stage, comparing to what we have
now: config files with passwords and secrets everywhere.

> * Explain clearly how operators would inspect, edit, and diff their
>   configs.

That's a good question and we clearly need a tool to query etcd and
grab all parameters + values from a project in particular.
One other aspect that we could see is, thanks to
https://review.openstack.org/#/c/440835/ - we would have a single
interface that expose all parameters in a human readable format
and let operators manage these parameters (through an UI or just by
reading in the file).

-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Joshua Harlow]

So just fyi, this has been talked about before (but prob in context of 
zookeeper or various other pluggable config backends).


Some links:

- https://review.openstack.org/#/c/243114/
- https://review.openstack.org/#/c/243182/
- https://blueprints.launchpad.net/oslo.config/+spec/oslo-config-db
- https://review.openstack.org/#/c/130047/

I think the general questions that seem to reappear are around the 
following:


* How does reloading work (does it)?

* What's the operational experience (editing a ini file is about the 
lowest bar we can possible get to, for better and/or worse).


* Does this need to be a new oslo.config backend or is it better suited 
by something like the following (external programs loop)::


   etcd_client = make_etcd_client(args)
   while True:
   has_changed = etcd_client.get_new_config("/blahblah") # or use a 
watch

   if has_changed:
  fetch_and_write_ini_file(etcd_client)
  trigger_reload()
   time.sleep(args.wait)

* Is an external loop better (maybe, maybe not?)

Pretty sure there are some etherpad discussions around this also somewhere.

Clint Byrum wrote:

Excerpts from Davanum Srinivas's message of 2017-03-14 13:04:37 -0400:

Team,

So one more thing popped up again on IRC:
https://etherpad.openstack.org/p/oslo.config_etcd_backend

What do you think? interested in this work?

Thanks,
Dims

PS: Between this thread and the other one about Tooz/DLM and
os-lively, we can probably make a good case to add etcd as a base
always-on service.



This is a cool idea, and I think we should do it.

A few loose ends I'd like to see in a spec:

* Security Security Security. (Hoping if I say it 3 times a real
   security person will appear and ask the hard questions).
* Explain clearly how operators would inspect, edit, and diff their
   configs.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Clint Byrum]

Excerpts from Davanum Srinivas's message of 2017-03-14 13:04:37 -0400:
> Team,
> 
> So one more thing popped up again on IRC:
> https://etherpad.openstack.org/p/oslo.config_etcd_backend
> 
> What do you think? interested in this work?
> 
> Thanks,
> Dims
> 
> PS: Between this thread and the other one about Tooz/DLM and
> os-lively, we can probably make a good case to add etcd as a base
> always-on service.
> 

This is a cool idea, and I think we should do it.

A few loose ends I'd like to see in a spec:

* Security Security Security. (Hoping if I say it 3 times a real
  security person will appear and ask the hard questions).
* Explain clearly how operators would inspect, edit, and diff their
  configs.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Michał Jastrzębski]

So from kolla perspective that would be totally awesome. For
kolla-ansible it will make things easier, for kolla-k8s...well, it
would solve a lot (**a lot**) of issues we're dealing with.

Pretty pretty please? <:)

On 14 March 2017 at 10:48, Emilien Macchi  wrote:
> On Tue, Mar 14, 2017 at 1:04 PM, Davanum Srinivas  wrote:
>> Team,
>>
>> So one more thing popped up again on IRC:
>> https://etherpad.openstack.org/p/oslo.config_etcd_backend
>>
>> What do you think? interested in this work?
>
> Wow, it seems like our minds are crossing.
> We had this discussion at the PTG and I've seen this topic quite often
> mentioned during different sessions.
>
> It's also somehow mentioned here:
> https://etherpad.openstack.org/p/deployment-pike
> See "Allow services to make use of KV stores instead of just INI files
> for config".
>
> What Thomas is doing in the PoC is actually what we thought it could
> be the next steps forward configuration management in OpenStack in a
> way that could be shared across all tools.
> Partially related, Ben Nemec is working on a spec that would extract
> all OpenStack parameters and generate YAML files:
> https://review.openstack.org/#/c/440835/
> And we thought that we could re-use this file to inject the
> configuration into etcd.
>
> I see a connection here where :
>
> 1. With Ben's work, we would generate a list of parameters available
> in OpenStack and expose it to the User Interface of the deployment
> tool.
> 2. The deployment tool would grab inputs from users and write the
> values into etcd. The installers would also configure some parameters
> that users don't want to provide (with all the logic around).
> 3. OpenStack services would read the config directly from etcd, thanks
> to Thomas's work.
>
> That way, 1. and 3. belong to oslo.config and 2. is done by OpenStack
> deployment tools.
> Does it make sense?
>
> I see a lot of collaboration and consolidation here, in how we do
> configuration management in OpenStack. I hope we can move forward and
> find some consensus here; and why not proposing a first architecture
> for Pike.
>
> Thanks,
>
>> Thanks,
>> Dims
>>
>> PS: Between this thread and the other one about Tooz/DLM and
>> os-lively, we can probably make a good case to add etcd as a base
>> always-on service.
>>
>> --
>> Davanum Srinivas :: https://twitter.com/dims
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Emilien Macchi
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Emilien Macchi]

On Tue, Mar 14, 2017 at 1:04 PM, Davanum Srinivas  wrote:
> Team,
>
> So one more thing popped up again on IRC:
> https://etherpad.openstack.org/p/oslo.config_etcd_backend
>
> What do you think? interested in this work?

Wow, it seems like our minds are crossing.
We had this discussion at the PTG and I've seen this topic quite often
mentioned during different sessions.

It's also somehow mentioned here:
https://etherpad.openstack.org/p/deployment-pike
See "Allow services to make use of KV stores instead of just INI files
for config".

What Thomas is doing in the PoC is actually what we thought it could
be the next steps forward configuration management in OpenStack in a
way that could be shared across all tools.
Partially related, Ben Nemec is working on a spec that would extract
all OpenStack parameters and generate YAML files:
https://review.openstack.org/#/c/440835/
And we thought that we could re-use this file to inject the
configuration into etcd.

I see a connection here where :

1. With Ben's work, we would generate a list of parameters available
in OpenStack and expose it to the User Interface of the deployment
tool.
2. The deployment tool would grab inputs from users and write the
values into etcd. The installers would also configure some parameters
that users don't want to provide (with all the logic around).
3. OpenStack services would read the config directly from etcd, thanks
to Thomas's work.

That way, 1. and 3. belong to oslo.config and 2. is done by OpenStack
deployment tools.
Does it make sense?

I see a lot of collaboration and consolidation here, in how we do
configuration management in OpenStack. I hope we can move forward and
find some consensus here; and why not proposing a first architecture
for Pike.

Thanks,

> Thanks,
> Dims
>
> PS: Between this thread and the other one about Tooz/DLM and
> os-lively, we can probably make a good case to add etcd as a base
> always-on service.
>
> --
> Davanum Srinivas :: https://twitter.com/dims
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Davanum Srinivas]

Team,

So one more thing popped up again on IRC:
https://etherpad.openstack.org/p/oslo.config_etcd_backend

What do you think? interested in this work?

Thanks,
Dims

PS: Between this thread and the other one about Tooz/DLM and
os-lively, we can probably make a good case to add etcd as a base
always-on service.

-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev