Re: [openstack-dev] [oslo] usage patterns for oslo.config
Mark - > I don't think I've seen code (except for obscure cases) which uses the > CONF global directly (as opposed to being passed CONF as a parameter) > but doesn't register the options at import time. > > Mark. > > Keystone uses the CONF global directly and doesn't register the options at import time. They're registered early when keystone is started, just before the call to CONF(). Here's an example use of the CONF global: http://git.openstack.org/cgit/openstack/keystone/tree/keystone/identity/controllers.py#n48 Here's the function that registers the CONF options: http://git.openstack.org/cgit/openstack/keystone/tree/keystone/common/config.py#n820 Here's the call to the function that registers the CONF options: http://git.openstack.org/cgit/openstack/keystone/tree/bin/keystone-all#n115 This was done so that reading the value of a config option during import will fail. - Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] usage patterns for oslo.config
On Tue, 2014-08-26 at 10:00 -0400, Doug Hellmann wrote: > On Aug 26, 2014, at 6:30 AM, Mark McLoughlin wrote: > > > On Mon, 2014-08-11 at 15:06 -0400, Doug Hellmann wrote: > >> On Aug 8, 2014, at 7:22 PM, Devananda van der Veen > >> wrote: > >> > >>> On Fri, Aug 8, 2014 at 12:41 PM, Doug Hellmann > >>> wrote: > > That’s right. The preferred approach is to put the register_opt() in > *runtime* code somewhere before the option will be used. That might be in > the constructor for a class that uses an option, for example, as > described > in > http://docs.openstack.org/developer/oslo.config/cfg.html#registering-options > > Doug > >>> > >>> Interesting. > >>> > >>> I've been following the prevailing example in Nova, which is to > >>> register opts at the top of a module, immediately after defining them. > >>> Is there a situation in which one approach is better than the other? > >> > >> The approach used in Nova is the “old” way of doing it. It works, but > >> assumes that all of the application code is modifying a global > >> configuration object. The runtime approach allows you to pass a > >> configuration object to a library, which makes it easier to mock the > >> configuration for testing and avoids having the configuration options > >> bleed into the public API of the library. We’ve started using the > >> runtime approach in new Oslo libraries that have configuration > >> options, but changing the implementation in existing application code > >> isn’t strictly necessary. > > > > I've been meaning to dig up some of the old threads and reviews to > > document how we got here. > > > > But briefly: > > > > * this global CONF variable originates from the gflags FLAGS variable > >in Nova before oslo.config > > > > * I was initially determined to get rid of any global variable use > >and did a lot of work to allow glance use oslo.config without a > >global variable > > > > * one example detail of this work - when you use paste.deploy to > >load an app, you have no ability to pass a config object > >through paste.deploy to the app. I wrote a little helper that > >used a thread-local variable to mimic this pass-through. > > > > * with glance done, I moved on to making keystone use oslo.config and > >initially didn't use the global variable. Then I ran into a veto > >from termie who felt very strongly that a global variable should be > >used. > > > > * in the end, I bought the argument that the use of a global variable > >was pretty deeply ingrained (especially in Nova) and that we should > >aim for consistent coding patterns across projects (i.e. Oslo > >shouldn't be just about shared code, but also shared patterns). The > >only realistic "standard pattern" we could hope for was the use of > >the global variable. > > > > * with that agreed, we reverted glance back to using a global > >variable and all projects followed suit > > > > * the case of libraries is different IMO - we'd be foolish to design > >APIs which lock us into using the global object > > > > So ... I wouldn't quite agree that this is "the new way" vs "the old > > way", but I think it would be reasonable to re-open the discussion about > > using the global object in our applications. Perhaps, at least, we could > > reduce our dependence on it. > > The aspect I was calling “old” was the “register options at import > time” pattern, not the use of a global. Whether we use a global or > not, registering options at runtime in a code path that will be using > them is better than relying on import ordering to ensure options are > registered before they are used. I don't think I've seen code (except for obscure cases) which uses the CONF global directly (as opposed to being passed CONF as a parameter) but doesn't register the options at import time. Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] usage patterns for oslo.config
On Aug 26, 2014, at 6:30 AM, Mark McLoughlin wrote: > On Mon, 2014-08-11 at 15:06 -0400, Doug Hellmann wrote: >> On Aug 8, 2014, at 7:22 PM, Devananda van der Veen >> wrote: >> >>> On Fri, Aug 8, 2014 at 12:41 PM, Doug Hellmann >>> wrote: That’s right. The preferred approach is to put the register_opt() in *runtime* code somewhere before the option will be used. That might be in the constructor for a class that uses an option, for example, as described in http://docs.openstack.org/developer/oslo.config/cfg.html#registering-options Doug >>> >>> Interesting. >>> >>> I've been following the prevailing example in Nova, which is to >>> register opts at the top of a module, immediately after defining them. >>> Is there a situation in which one approach is better than the other? >> >> The approach used in Nova is the “old” way of doing it. It works, but >> assumes that all of the application code is modifying a global >> configuration object. The runtime approach allows you to pass a >> configuration object to a library, which makes it easier to mock the >> configuration for testing and avoids having the configuration options >> bleed into the public API of the library. We’ve started using the >> runtime approach in new Oslo libraries that have configuration >> options, but changing the implementation in existing application code >> isn’t strictly necessary. > > I've been meaning to dig up some of the old threads and reviews to > document how we got here. > > But briefly: > > * this global CONF variable originates from the gflags FLAGS variable >in Nova before oslo.config > > * I was initially determined to get rid of any global variable use >and did a lot of work to allow glance use oslo.config without a >global variable > > * one example detail of this work - when you use paste.deploy to >load an app, you have no ability to pass a config object >through paste.deploy to the app. I wrote a little helper that >used a thread-local variable to mimic this pass-through. > > * with glance done, I moved on to making keystone use oslo.config and >initially didn't use the global variable. Then I ran into a veto >from termie who felt very strongly that a global variable should be >used. > > * in the end, I bought the argument that the use of a global variable >was pretty deeply ingrained (especially in Nova) and that we should >aim for consistent coding patterns across projects (i.e. Oslo >shouldn't be just about shared code, but also shared patterns). The >only realistic "standard pattern" we could hope for was the use of >the global variable. > > * with that agreed, we reverted glance back to using a global >variable and all projects followed suit > > * the case of libraries is different IMO - we'd be foolish to design >APIs which lock us into using the global object > > So ... I wouldn't quite agree that this is "the new way" vs "the old > way", but I think it would be reasonable to re-open the discussion about > using the global object in our applications. Perhaps, at least, we could > reduce our dependence on it. The aspect I was calling “old” was the “register options at import time” pattern, not the use of a global. Whether we use a global or not, registering options at runtime in a code path that will be using them is better than relying on import ordering to ensure options are registered before they are used. Doug > > Oh look, we have a FAQ on this: > > https://wiki.openstack.org/wiki/Oslo#Why_does_oslo.config_have_a_CONF_object.3F_Global_object_SUCK.21 > > Mark. > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] usage patterns for oslo.config
On Mon, 2014-08-11 at 15:06 -0400, Doug Hellmann wrote: > On Aug 8, 2014, at 7:22 PM, Devananda van der Veen > wrote: > > > On Fri, Aug 8, 2014 at 12:41 PM, Doug Hellmann > > wrote: > >> > >> That’s right. The preferred approach is to put the register_opt() in > >> *runtime* code somewhere before the option will be used. That might be in > >> the constructor for a class that uses an option, for example, as described > >> in > >> http://docs.openstack.org/developer/oslo.config/cfg.html#registering-options > >> > >> Doug > > > > Interesting. > > > > I've been following the prevailing example in Nova, which is to > > register opts at the top of a module, immediately after defining them. > > Is there a situation in which one approach is better than the other? > > The approach used in Nova is the “old” way of doing it. It works, but > assumes that all of the application code is modifying a global > configuration object. The runtime approach allows you to pass a > configuration object to a library, which makes it easier to mock the > configuration for testing and avoids having the configuration options > bleed into the public API of the library. We’ve started using the > runtime approach in new Oslo libraries that have configuration > options, but changing the implementation in existing application code > isn’t strictly necessary. I've been meaning to dig up some of the old threads and reviews to document how we got here. But briefly: * this global CONF variable originates from the gflags FLAGS variable in Nova before oslo.config * I was initially determined to get rid of any global variable use and did a lot of work to allow glance use oslo.config without a global variable * one example detail of this work - when you use paste.deploy to load an app, you have no ability to pass a config object through paste.deploy to the app. I wrote a little helper that used a thread-local variable to mimic this pass-through. * with glance done, I moved on to making keystone use oslo.config and initially didn't use the global variable. Then I ran into a veto from termie who felt very strongly that a global variable should be used. * in the end, I bought the argument that the use of a global variable was pretty deeply ingrained (especially in Nova) and that we should aim for consistent coding patterns across projects (i.e. Oslo shouldn't be just about shared code, but also shared patterns). The only realistic "standard pattern" we could hope for was the use of the global variable. * with that agreed, we reverted glance back to using a global variable and all projects followed suit * the case of libraries is different IMO - we'd be foolish to design APIs which lock us into using the global object So ... I wouldn't quite agree that this is "the new way" vs "the old way", but I think it would be reasonable to re-open the discussion about using the global object in our applications. Perhaps, at least, we could reduce our dependence on it. Oh look, we have a FAQ on this: https://wiki.openstack.org/wiki/Oslo#Why_does_oslo.config_have_a_CONF_object.3F_Global_object_SUCK.21 Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] usage patterns for oslo.config
On Aug 8, 2014, at 7:22 PM, Devananda van der Veen wrote: > On Fri, Aug 8, 2014 at 12:41 PM, Doug Hellmann wrote: >> >> That’s right. The preferred approach is to put the register_opt() in >> *runtime* code somewhere before the option will be used. That might be in >> the constructor for a class that uses an option, for example, as described >> in >> http://docs.openstack.org/developer/oslo.config/cfg.html#registering-options >> >> Doug > > Interesting. > > I've been following the prevailing example in Nova, which is to > register opts at the top of a module, immediately after defining them. > Is there a situation in which one approach is better than the other? The approach used in Nova is the “old” way of doing it. It works, but assumes that all of the application code is modifying a global configuration object. The runtime approach allows you to pass a configuration object to a library, which makes it easier to mock the configuration for testing and avoids having the configuration options bleed into the public API of the library. We’ve started using the runtime approach in new Oslo libraries that have configuration options, but changing the implementation in existing application code isn’t strictly necessary. Doug > > Thanks, > Devananda > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] usage patterns for oslo.config
On Fri, Aug 8, 2014 at 12:41 PM, Doug Hellmann wrote: > > That’s right. The preferred approach is to put the register_opt() in > *runtime* code somewhere before the option will be used. That might be in > the constructor for a class that uses an option, for example, as described > in > http://docs.openstack.org/developer/oslo.config/cfg.html#registering-options > > Doug Interesting. I've been following the prevailing example in Nova, which is to register opts at the top of a module, immediately after defining them. Is there a situation in which one approach is better than the other? Thanks, Devananda ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] usage patterns for oslo.config
On Aug 8, 2014, at 1:30 PM, Vishvananda Ishaya wrote:
> Hi Alistair,
>
> Modules can register their own options and there is no need to call
> reload_config_files. The config files are parsed and values stored in case
> the option is later declared. The only time you need to reload files is if
> you add new config files in the new module. See the example code:
>
> from oslo.config import cfg
> with open("foo", "w") as f:
> f.write("[DEFAULT]\nfoo=bar")
>
> cfg.CONF(["--config-file", "foo"])
> try:
> print cfg.CONF.foo
> except cfg.NoSuchOptError:
> print "NO OPT"
> # OUT: 'NO OPT'
>
> cfg.CONF.register_opt(cfg.StrOpt("foo"))
> print cfg.CONF.foo
> cfg.CONF.foo
> # OUT: ‘bar'
>
> One thing to keep in mind is you don’t want to use config values at import
> time, since this tends to be before the config files have been loaded.
>
> Vish
That’s right. The preferred approach is to put the register_opt() in *runtime*
code somewhere before the option will be used. That might be in the constructor
for a class that uses an option, for example, as described in
http://docs.openstack.org/developer/oslo.config/cfg.html#registering-options
Doug
>
> On Aug 8, 2014, at 6:40 AM, Coles, Alistair wrote:
>
>> I’ve been looking at the implications of applying oslo.config in Swift, and
>> I have a question about the best pattern for registering options.
>>
>> Looking at how keystone uses oslo.config, the pattern seems to be to have
>> all options declared and registered 'up-front' in a single place
>> (keystone/common/config.py) before loading wsgi pipeline/starting the
>> service. Is there another usage pattern where each middleware registers its
>> options independently ‘on-demand’ rather than maintaining them all in a
>> single place?
>>
>> I read about a pattern [1] whereby modules register opts during import, but
>> does that require there to be some point in the lifecycle where all required
>> modules are imported *before* parsing config files? Seems like that would
>> mean parsing the wsgi pipeline to ‘discover’ the middleware modules being
>> used, importing all those modules, then parsing config files, then loading
>> the wsgi pipeline?
>>
>> OR - is it acceptable for each middleware module to register its own options
>> if/when it is imported during wsgi pipeline loading
>> (CONF.register_options()) and then call CONF.reload_config_files() ?
>>
>> Thanks,
>> Alistair
>>
>> [1]
>> http://docs.openstack.org/developer/oslo.config/cfg.html#global-configopts
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] usage patterns for oslo.config
Hi Alistair,
Modules can register their own options and there is no need to call
reload_config_files. The config files are parsed and values stored in case the
option is later declared. The only time you need to reload files is if you add
new config files in the new module. See the example code:
from oslo.config import cfg
with open("foo", "w") as f:
f.write("[DEFAULT]\nfoo=bar")
cfg.CONF(["--config-file", "foo"])
try:
print cfg.CONF.foo
except cfg.NoSuchOptError:
print "NO OPT"
# OUT: 'NO OPT'
cfg.CONF.register_opt(cfg.StrOpt("foo"))
print cfg.CONF.foo
cfg.CONF.foo
# OUT: ‘bar'
One thing to keep in mind is you don’t want to use config values at import
time, since this tends to be before the config files have been loaded.
Vish
On Aug 8, 2014, at 6:40 AM, Coles, Alistair wrote:
> I’ve been looking at the implications of applying oslo.config in Swift, and I
> have a question about the best pattern for registering options.
>
> Looking at how keystone uses oslo.config, the pattern seems to be to have all
> options declared and registered 'up-front' in a single place
> (keystone/common/config.py) before loading wsgi pipeline/starting the
> service. Is there another usage pattern where each middleware registers its
> options independently ‘on-demand’ rather than maintaining them all in a
> single place?
>
> I read about a pattern [1] whereby modules register opts during import, but
> does that require there to be some point in the lifecycle where all required
> modules are imported *before* parsing config files? Seems like that would
> mean parsing the wsgi pipeline to ‘discover’ the middleware modules being
> used, importing all those modules, then parsing config files, then loading
> the wsgi pipeline?
>
> OR - is it acceptable for each middleware module to register its own options
> if/when it is imported during wsgi pipeline loading (CONF.register_options())
> and then call CONF.reload_config_files() ?
>
> Thanks,
> Alistair
>
> [1] http://docs.openstack.org/developer/oslo.config/cfg.html#global-configopts
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo] usage patterns for oslo.config
I've been looking at the implications of applying oslo.config in Swift, and I have a question about the best pattern for registering options. Looking at how keystone uses oslo.config, the pattern seems to be to have all options declared and registered 'up-front' in a single place (keystone/common/config.py) before loading wsgi pipeline/starting the service. Is there another usage pattern where each middleware registers its options independently 'on-demand' rather than maintaining them all in a single place? I read about a pattern [1] whereby modules register opts during import, but does that require there to be some point in the lifecycle where all required modules are imported *before* parsing config files? Seems like that would mean parsing the wsgi pipeline to 'discover' the middleware modules being used, importing all those modules, then parsing config files, then loading the wsgi pipeline? OR - is it acceptable for each middleware module to register its own options if/when it is imported during wsgi pipeline loading (CONF.register_options()) and then call CONF.reload_config_files() ? Thanks, Alistair [1] http://docs.openstack.org/developer/oslo.config/cfg.html#global-configopts ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
