Re: [openstack-dev] [DevStack] Generalize config file settings
The biggest concern I have about the current implementation is that it handles conf files inconsistently - but as Dean pointed out on the review it opens a wider question. I will very often run devstack, find something not quite working unstack, change a value or two in localrc and then re-run devstack - this is against the policy that you must never change localrc. I understand that there are some very good reasons that we can't make all modifications to localrc work on an unstack/stack -- but a very large number of the options are only read at run time and do not have any persistent state. This change enforces a stricter do not modify localrc after the first run of stack.sh policy by making sure localrc is not updated if it exists - even if the local.conf sections had changed. Conversely, any changes in local.conf outside of the localrc section _will_ be honoured and pushed through to the respective config files. I think that we need to attempt to use all changes in local.conf, whether in localrc or not. Admittedly this might give situations where a second run of devstack is incompatible with the system state - but that's no worse than the current situation. Of course, the current implementation doesn't stop someone modifying localrc directly and re-stacking as they can now - but then we'd have a mismatch of what was in local.conf and what is in localrc which I'm sure would lead to significant confusion. In my mind the alternative would be to add a more restrictive check to make it very obvious what the expectations are - e.g. store all values from localrc in localrc.lastrun and if any of the values have changed then fail to stack. Thoughts? Bob From: Dean Troyer [mailto:dtro...@gmail.com] Sent: 16 September 2013 22:41 To: OpenStack Development Mailing List Subject: Re: [openstack-dev] [DevStack] Generalize config file settings On Fri, Sep 13, 2013 at 1:20 PM, Everett Toews everett.to...@rackspace.commailto:everett.to...@rackspace.com wrote: On Sep 13, 2013, at 6:10 AM, Sean Dague wrote: Because inevitably people ask for copies of other folks configs to duplicate things, and a single file is easier to pass around than a tree. But that would mean a unique parser to handle the top level stanza. +1 I share localrc files all the time. Well, I wrote a parser for a modified INI file format anyway and managed to put everything into a single local.conf file. There is a bit of clumsiness around localrc in order to maintain backward-compatibility and a deprecation cycle to go through for some existing config variables. The current incarnation is in https://review.openstack.org/#/c/46768/, try it out, let's see if it is any good. I did write up some more detailed docs at http://hackstack.org/x/blog/2013/09/07/devstack-local-config/ that will morph into the doc page in devstack.orghttp://devstack.org when we're done. dt -- Dean Troyer dtro...@gmail.commailto:dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Generalize config file settings
On Wed, Sep 18, 2013 at 4:46 AM, Bob Ball bob.b...@citrix.com wrote: I understand that there are some very good reasons that we can’t make all modifications to localrc work on an unstack/stack –– but a very large number of the options are only read at run time and do not have any persistent state. My apologies Bob, misunderstanding alert...it wasn't clear in the review message where I said we can't change localrc I meant stack.sh, not us as users. The only time stack.sh will change localrc is to add passwords entered at the prompts. There can be problems when localrc changes between stack and unstack but we're a dev/test tool and some hands just are not going to be held. The question I have is when we introduce local.conf, we can't source it directly to get the localrc values so currently it writes those to localrc if ther isn't already one present and continues as before. The question is what is the 'least surprise' user experience next time stack.sh runs, overwrite localrc again to pick up any changes that might be made to local.conf? Or stick with the current behaviour of never changing it and requiring the user to remove it before changes in local.conf get written back to it again. I prefer the former, overwriting localrc every time but that is a significant change in behaviour, even if it is consistent with what we do to the other OpenStack config files (rewriting them every time). dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Generalize config file settings
Ah! Sorry for the mis-understanding. In which case, would a simple fix of adding a localrc.generated be sufficient? Produce a localrc which simply imports localrc.generated and replace localrc.generated in all cases - still allowing users to override it if needed? Bob From: Dean Troyer [mailto:dtro...@gmail.com] Sent: 18 September 2013 13:37 To: OpenStack Development Mailing List Subject: Re: [openstack-dev] [DevStack] Generalize config file settings On Wed, Sep 18, 2013 at 4:46 AM, Bob Ball bob.b...@citrix.commailto:bob.b...@citrix.com wrote: I understand that there are some very good reasons that we can't make all modifications to localrc work on an unstack/stack -- but a very large number of the options are only read at run time and do not have any persistent state. My apologies Bob, misunderstanding alert...it wasn't clear in the review message where I said we can't change localrc I meant stack.sh, not us as users. The only time stack.sh will change localrc is to add passwords entered at the prompts. There can be problems when localrc changes between stack and unstack but we're a dev/test tool and some hands just are not going to be held. The question I have is when we introduce local.conf, we can't source it directly to get the localrc values so currently it writes those to localrc if ther isn't already one present and continues as before. The question is what is the 'least surprise' user experience next time stack.sh runs, overwrite localrc again to pick up any changes that might be made to local.conf? Or stick with the current behaviour of never changing it and requiring the user to remove it before changes in local.conf get written back to it again. I prefer the former, overwriting localrc every time but that is a significant change in behaviour, even if it is consistent with what we do to the other OpenStack config files (rewriting them every time). dt -- Dean Troyer dtro...@gmail.commailto:dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Generalize config file settings
On Wed, Sep 18, 2013 at 7:49 AM, Bob Ball bob.b...@citrix.com wrote: In which case, would a simple fix of adding a “localrc.generated” be sufficient? ** ** Produce a localrc which simply imports localrc.generated and replace localrc.generated in all cases – still allowing users to override it if needed? OK, how about this: - always extract localrc section from local.conf into localrc.generated (if it exists) - append localrc (if it exists) to localrc.generated This works for stack.sh, but localrc is used in a _lot_ of places, do we re-generate each time? We have to to keep the current semantics of localrc edits being effective immediately. I want to keep this as simple as possible... dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Generalize config file settings
I think it'd make more sense the other way round. If localrc does not exist, and we've created a localrc.generated create localrc with just source localrc.generated. That way anyone using localrc directly will get the right information. A quick grep suggests that all uses in devstack source the localrc, therefore that should work? It'd be easy to add a comment in the localrc or in the docs for local.conf that say only stack.sh will update localrc.generated so if you want live changes in other scripts you have to modify localrc? Bob From: Dean Troyer [mailto:dtro...@gmail.com] Sent: 18 September 2013 15:04 To: OpenStack Development Mailing List Subject: Re: [openstack-dev] [DevStack] Generalize config file settings On Wed, Sep 18, 2013 at 7:49 AM, Bob Ball bob.b...@citrix.commailto:bob.b...@citrix.com wrote: In which case, would a simple fix of adding a localrc.generated be sufficient? Produce a localrc which simply imports localrc.generated and replace localrc.generated in all cases - still allowing users to override it if needed? OK, how about this: - always extract localrc section from local.conf into localrc.generated (if it exists) - append localrc (if it exists) to localrc.generated This works for stack.sh, but localrc is used in a _lot_ of places, do we re-generate each time? We have to to keep the current semantics of localrc edits being effective immediately. I want to keep this as simple as possible... dt -- Dean Troyer dtro...@gmail.commailto:dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Generalize config file settings
On Wed, Sep 18, 2013 at 9:28 AM, Bob Ball bob.b...@citrix.com wrote: I think it’d make more sense the other way round. ** ** If localrc does not exist, and we’ve created a localrc.generated create localrc with just “source localrc.generated”. That way anyone using localrc directly will get the right information. A quick grep suggests that all uses in devstack source the localrc, therefore that should work? ** ** It’d be easy to add a comment in the localrc or in the docs for local.conf that say only stack.sh will update localrc.generated so if you want live changes in other scripts you have to modify localrc? The real goal is for the user to stop modifying localrc once they start using local.conf but keep the same semantics. If I could figure out how to stuff the config bits into localrc and still have it be sourcable I'd...ok, just tried that, it can work but is too fragile. How about this: - if localrc exists, use it. the rest of local.conf works as proposed. - if local.conf has a [[local:localrc]] section and localrc exists print a warning that it (local.conf localrc section) is being ignored - if local.conf has a [[local:localrc]] section and localrc does not exist, extract it to localrc.generated and use that - interactively-entered passwords will continue to be written to localrc if it exists but will not be written to local.conf; this way we can say local.conf is _never_ modified by DevStack. The value in creating localrc on first run is long gone, running DevStack with no config is not very useful anymore. This does the following: - localrc semantics do not change so there is no compatibility surprise - local.conf config function is seamlessly added - to gain the all-config-in-one-file aspect the user needs to 'convert' localrc to a local.conf [[local:localrc]] section; otherwise no change is necessary - local.generated will include the 'DO NOT EDIT' warning as it will be re-created as-required when stackrc sources it Am I missing anything? dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Generalize config file settings
On Fri, Sep 13, 2013 at 1:20 PM, Everett Toews everett.to...@rackspace.comwrote: On Sep 13, 2013, at 6:10 AM, Sean Dague wrote: Because inevitably people ask for copies of other folks configs to duplicate things, and a single file is easier to pass around than a tree. But that would mean a unique parser to handle the top level stanza. +1 I share localrc files all the time. Well, I wrote a parser for a modified INI file format anyway and managed to put everything into a single local.conf file. There is a bit of clumsiness around localrc in order to maintain backward-compatibility and a deprecation cycle to go through for some existing config variables. The current incarnation is in https://review.openstack.org/#/c/46768/, try it out, let's see if it is any good. I did write up some more detailed docs at http://hackstack.org/x/blog/2013/09/07/devstack-local-config/ that will morph into the doc page in devstack.org when we're done. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Generalize config file settings
On Fri, Sep 13, 2013 at 6:10 AM, Sean Dague s...@dague.net wrote: I like option a, though I'm not sure we need the full system path in the conf.d (that's pretty minor though). That was to avoid making assumptions about target files or encoding paths in filenames. It really needs to recognize XXX_CONF_DIR settings too. Because inevitably people ask for copies of other folks configs to duplicate things, and a single file is easier to pass around than a tree. But that would mean a unique parser to handle the top level stanza. That is a great point and one I did not have in mind. One possibility would be to include localrc in this mega-file and the _very_ first step would be to extract it if localrc doesn't already exist and run from there. I want to support the conf.d-style also because that is useful for outside projects to drop in what they require for changing included project configs; these would not necessarily be user-modifiable. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Generalize config file settings
On Sep 13, 2013, at 6:10 AM, Sean Dague wrote: Because inevitably people ask for copies of other folks configs to duplicate things, and a single file is easier to pass around than a tree. But that would mean a unique parser to handle the top level stanza. +1 I share localrc files all the time. Regards, Everett ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Generalize config file settings
On Thu, Sep 12, 2013 at 9:36 AM, Dean Troyer dtro...@gmail.com wrote: DevStack has long had a config setting in localrc called EXTRA_OPTS that allowed arbitrary settings to be added to /etc/nova/nova.conf [DEFAULT] section. Additional files and sections have recently been implemented with a similar scheme. I don't think this scales well as at a minimum every config file and section needs to be individually handled. I'd like to get some feedback on the following two proposals, or hear other ideas on how to generalize solving the problem of setting arbitrary configuration values. a) Create conf.d/*.conf files as needed and process each file present into a corresponding config file. These files would not be supplied by DevStack but created and maintained locally. Example: conf.d/etc/nova/nova.conf: [DEFAULT] use_syslog = True [osapi_v3] enabled = False b) Create a single service.local.conf file for each project (Nova, Cinder, etc) that contains a list of settings to be applied to the config files for that service. Example: nova.local.conf: # conf file names are parsed out of the section name below between '[' and the first ':' [/etc/nova/nova.conf:DEFAULT] use_syslog = True [/etc/nova/nova.conf:osapi_v3] enabled = False Both cases need to be able to specify the destination config file and section in addition to the attribute name and value. Thoughts? dt [Prompted by review https://review.openstack.org/44266] -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev +1 for not dumping in lib/xxx Option 'a' seems like a bit easier to manage in terms of number of files etc but I wouldn't have a strong preference between the two options presented. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Generalize config file settings
On Thu, Sep 12, 2013 at 9:44 PM, Monty Taylor mord...@inaugust.com wrote: os-apply-config Doesn't that just convert a json syntax to a file with the syntax Dean was describing? Maybe it's changed, but that's what I *thought* it did. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
