Excerpts from Joshua Harlow's message of 2015-11-04 14:13:33 -0800: > Agreed, it'd be nice to have an 'audit' of the various projects configs > and try to categorize which ones should be reloadable (and the priority > to make it reloadable) and which ones are service discovery configs (and > probably shouldn't be in config in the first place) and which ones are > nice to haves to be configurable... (like rpc_response_timeout). > > The side-effects of making a few things configurable will likely cause a > whole bunch of other issues anyway (like how an application/library... > gets notified that a config has changed and it may need to re-adjust > itself to those new values which may include for example unloading a > driver, stopping a thread, starting a new driver...), so I'm thinking we > should start small and grow as needed (based on above prioritization). > > Log levels are high on my known list.
Right, anything that is going to take significant application rework to support should wait. The session identified a few relatively simple options that help with debugging when they are changed, including logging, and we should start with those. Doug > > Fox, Kevin M wrote: > > Ah, I hadn't considered the rabbit_hosts (discovery) case. yeah, that would > > be a useful thing to be able to tweak live. I haven't needed that feature > > yet, but could see how that flexibility could come in handy. > > > > Thanks, > > Kevin > > ________________________________________ > > From: Joshua Harlow [harlo...@fastmail.com] > > Sent: Wednesday, November 04, 2015 11:34 AM > > To: OpenStack Development Mailing List (not for usage questions) > > Subject: Re: [openstack-dev] [oslo][oslo.config][oslo.service] Dynamic > > Reconfiguration of OpenStack Services > > > > Along this line, thinks like the following are likely more changeable > > (and my guess is operators would want to change them when things start > > going badly), for example from a nova.conf that I have laying around... > > > > [DEFAULT] > > > > rabbit_hosts=... > > rpc_response_timeout=... > > default_notification_level=... > > default_log_levels=... > > > > [glance] > > > > api_servers=... > > > > (and more) > > > > Some of those I think should have higher priority as being > > reconfigurable, but I think operators should be asked what they think > > would be useful and prioritize those. > > > > Some of those really are service discovery 'types' (rabbit_hosts, > > glance/api_servers, keystone/api_servers) but fixing this is likely a > > longer term goal (see conversations in keystone). > > > > Joshua Harlow wrote: > >> gord chung wrote: > >>> we actually had a solution implemented in Ceilometer to handle this[1]. > >>> > >>> that said, based on the results of our survey[2], we found that most > >>> operators *never* update configuration files after the initial setup and > >>> if they did it was very rarely (monthly updates). the question related > >>> to Ceilometer and its pipeline configuration file so the results might > >>> be specific to Ceilometer. I think you should definitely query operators > >>> before undertaking any work. the last thing you want to do is implement > >>> a feature no one really needs/wants. > >>> > >>> [1] > >>> http://specs.openstack.org/openstack/ceilometer-specs/specs/liberty/reload-file-based-pipeline-configuration.html > >>> > >>> [2] > >>> http://lists.openstack.org/pipermail/openstack-dev/2015-September/075628.html > >>> > >> So my general though on the above is yes, definitely consult operators > >> to see if they would use this, although if a feature doesn't exist and > >> has never existed (say outside of ceilometer) then it's sort of hard to > >> get an accurate survey result from a group of people that have never had > >> the feature in the first place... Either way it should be done, just to > >> get more knowledge... > >> > >> I know operators (at yahoo!) want to be able to dynamically change the > >> logging level, and that's not a monthly task, but more of an 'as-needed' > >> one that would be very helpful when things start going badly... So > >> perhaps the set of reloadable configuration should start out small and > >> not encompass all the things... > >> > >>> On 04/11/2015 10:00 AM, Marian Horban wrote: > >>>> Hi guys, > >>>> > >>>> Unfortunately I haven't been on Tokio summit but I know that there was > >>>> discussion about dynamic reloading of configuration. > >>>> Etherpad refs: > >>>> https://etherpad.openstack.org/p/mitaka-cross-project-dynamic-config-services, > >>>> > >>>> > >>>> https://etherpad.openstack.org/p/mitaka-oslo-security-logging > >>>> > >>>> In this thread I want to discuss agreements reached on the summit and > >>>> discuss > >>>> implementation details. > >>>> > >>>> Some notes taken from etherpad and my remarks: > >>>> > >>>> 1. "Adding "mutable" parameter for each option." > >>>> "Do we have an option mutable=True on CfgOpt? Yes" > >>>> --------------------------------------------------------- > >>>> As I understood 'mutable' parameter must indicate whether service > >>>> contains > >>>> code responsible for reloading of this option or not. > >>>> And this parameter should be one of the arguments of cfg.Opt > >>>> constructor. > >>>> Problems: > >>>> 1. Library's options. > >>>> SSL options ca_file, cert_file, key_file taken from oslo.service library > >>>> could be reloaded in nova-api so these options should be mutable... > >>>> But for some projects that don't need SSL support reloading of SSL > >>>> options > >>>> doesn't make sense. For such projects this option should be non mutable. > >>>> Problem is that oslo.service - single and there are many different > >>>> projects > >>>> which use it in different way. > >>>> The same options could be mutable and non mutable in different contexts. > >>>> 2. Support of config options on some platforms. > >>>> Parameter "mutable" could be different for different platforms. Some > >>>> options > >>>> make sense only for specific platforms. If we mark such options as > >>>> mutable > >>>> it could be misleading on some platforms. > >>>> 3. Dependency of options. > >>>> There are many 'workers' options(osapi_compute_workers, ec2_workers, > >>>> metadata_workers, workers). These options specify number of workers for > >>>> OpenStack API services. > >>>> If value of the 'workers' option is greater than '1' instance of > >>>> ProcessLauncher is created otherwise instance of ServiceLauncher is > >>>> created. > >>>> When ProcessLauncher receives SIGHUP it reloads it own configuration, > >>>> gracefully terminates children and respawns new children. > >>>> This mechanism allows to reload many config options implicitly. > >>>> But if value of the 'workers' option equals '1' instance of > >>>> ServiceLauncher > >>>> is created. > >>>> ServiceLauncher starts everything in single process and in this case we > >>>> don't have such implicit reloading. > >>>> > >>>> I think that mutability of options is a complicated feature and I > >>>> think that > >>>> adding of 'mutable' parameter into cfg.Opt constructor could just add > >>>> mess. > >>>> > >>>> 2. "oslo.service catches SIGHUP and calls oslo.config" > >>>> --------------------------------------------------------- > >>>> From my point of view every service should register list of hooks to > >>>> reload > >>>> config options. oslo.service should catch SIGHUP and call list of > >>>> registered > >>>> hooks one by one with specified order. > >>>> Discussion of such implementation was started in ML: > >>>> http://lists.openstack.org/pipermail/openstack-dev/2015-September/074558.html. > >>>> > >>>> Raw reviews: > >>>> https://review.openstack.org/#/c/228892/, > >>>> https://review.openstack.org/#/c/223668/. > >>>> > >>>> 3. "oslo.config is responsible to log changes which were ignored on > >>>> SIGHUP" > >>>> --------------------------------------------------------- > >>>> Some config options could be changed using API(for example quotas) > >>>> that's why > >>>> oslo.config doesn't know actual configuration of service and can't log > >>>> changes of configuration. > >>>> > >>>> Regards, Marian Horban > >>>> > >>>> > >>>> __________________________________________________________________________ > >>>> > >>>> OpenStack Development Mailing List (not for usage questions) > >>>> Unsubscribe:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >>>> > >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>> -- > >>> gord > >>> > >>> __________________________________________________________________________ > >>> > >>> OpenStack Development Mailing List (not for usage questions) > >>> Unsubscribe: > >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> __________________________________________________________________________ > >> OpenStack Development Mailing List (not for usage questions) > >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev