Re: [openstack-dev] [oslo][cinder][nova][neutron] going forward to oslo-config-generator ...
Excerpts from Walter A. Boring IV's message of 2015-03-23 13:49:48 -0700: Maybe we can leverage Cinder's use of the abc in the drivers.py now. We could create an OptionsVD that drivers would add and implement. The config generator could inspect objects looking for OptionsVD and then call list_opts() on it. That way, driver maintainers don't also have to patch the setup.cfg. Just require new drivers to add OptionsVD and implement list_opts(). We could put a hacking check in for that as well. If I understand you correctly, you're proposing to add a list_opts() function to cinder somewhere that uses a separate set of entry points to find the drivers and then pulls all of their options from the driver class using a method that has to exist based on the abstract base class contract. If that's right, I like the idea. Doug My $0.02, Walt We could even further reduce the occurrence of such issues by moving the list_opts() function down into each driver and have an entry point for oslo.config.opts in setup.cfg for each of the drivers. As with the currently proposed solution, the developer doesn't have to edit a top level file for a new configuration option. This solution adds that the developer doesn't have to edit a top level file to add a new configuration item list to their driver. With this approach the change would happen in the driver's list_opts() function, rather than in cinder/opts.py . The only time that setup.cfg would needed to edited is when a new package is added or when a new driver is added. This would reduce some of the already minimal burden on the developer. We, however, would need to agree upon some method for aggregating together the options lists on a per package (i.e. cinder.scheduler, cinder.api) level. This approach, however, also has the advantage of providing a better indication in the sample config file of where the options are coming from. That is an improvement over what I have currently proposed. Does Doug's proposal sound more agreeable to everyone? It is important to note that the fact that some manual intervention is required to 'plumb' in the new configuration options was done by design. There is a little more work required to make options available to oslo-config-generator but the ability to use different namespaces, different sample configs, etc were added with the new generator. These additional capabilities were requested by other projects. So, moving to this design does have the potential for more long-term gain. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][cinder][nova][neutron] going forward to oslo-config-generator ...
Maybe we can leverage Cinder's use of the abc in the drivers.py now. We could create an OptionsVD that drivers would add and implement. The config generator could inspect objects looking for OptionsVD and then call list_opts() on it. That way, driver maintainers don't also have to patch the setup.cfg. Just require new drivers to add OptionsVD and implement list_opts(). We could put a hacking check in for that as well. My $0.02, Walt We could even further reduce the occurrence of such issues by moving the list_opts() function down into each driver and have an entry point for oslo.config.opts in setup.cfg for each of the drivers. As with the currently proposed solution, the developer doesn't have to edit a top level file for a new configuration option. This solution adds that the developer doesn't have to edit a top level file to add a new configuration item list to their driver. With this approach the change would happen in the driver's list_opts() function, rather than in cinder/opts.py . The only time that setup.cfg would needed to edited is when a new package is added or when a new driver is added. This would reduce some of the already minimal burden on the developer. We, however, would need to agree upon some method for aggregating together the options lists on a per package (i.e. cinder.scheduler, cinder.api) level. This approach, however, also has the advantage of providing a better indication in the sample config file of where the options are coming from. That is an improvement over what I have currently proposed. Does Doug's proposal sound more agreeable to everyone? It is important to note that the fact that some manual intervention is required to 'plumb' in the new configuration options was done by design. There is a little more work required to make options available to oslo-config-generator but the ability to use different namespaces, different sample configs, etc were added with the new generator. These additional capabilities were requested by other projects. So, moving to this design does have the potential for more long-term gain. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][cinder][nova][neutron] going forward to oslo-config-generator ...
Excerpts from Gary Kotton's message of 2015-03-21 16:36:07 +: Hi, One of the issues that we had in Nova was that when we moved to oslo libraries configuration options support by the libraries were no longer present in the generated configuration file. Is this something that is already supported or planned (sorry for being a little ignorant here). The new config generator uses entry points declared in the libraries to discover their options. This is one of the main reasons for moving away from the old generator that was scanning code in the local directory -- that code no longer contains all of the options. In neutron things may be a little more challenging as there are many different plugins and with the decomposition things may have additional challenges. The configuration binding is done via the external decomposed code and not in the neutron code base. So it is not clear how that code may be parsed to generate the sample configuration. The new generator expects the code to be installed. You then call it with a list of namespaces to check for options. This lets you add libraries, as mentioned above, as well as partitioning the options within the app based on which service actually uses them. You could, for example, have a different namespace for neutron.api and neutron.agent and generate different sample configs for those 2 daemons. Doug Thanks Gary On 3/21/15, 12:01 AM, Jay S. Bryant jsbry...@electronicjungle.net wrote: All, Let me start with the TLDR; Cinder, Nova and Neutron have lots of configuration options that need to be processed by oslo-config-generator to create the project.conf.sample file. There are a couple of different ways this could be done. I have one proposal out, which has raised concerns, there is a second approach that could be taken which I am proposing below. Please read on if you have a strong opinion on the precedent we will try to set in Cinder. :-) We discussed in the oslo meeting a couple of weeks ago a plan for how Cinder was going to blaze a trail to the new oslo-config-generator. The result of that discussion and work is here: [1] It needs some more work but has the bare bones pieces there to move to using oslo-config-generator. With the change I have written extensive hacking checks that ensure that any lists that are registered with register_opts() are included in the base cinder/opts.py file that is then a single entry point that pulls all of the options together to generate the cinder.conf.sample file. This has raised concern, however, that whenever a developer adds a new list of configuration options, they are going to have to know to go back to cinder/opts.py and add their module and option list there. The hacking check should catch this before code is submitted, but we are possibly setting ourselves up for cases where the patch will fail in the gate because updates are not made in all the correct places and because pep8 isn't run before the patch is pushed. It is important to note, that this will not happen every time a configuration option is changed or added, as was the case with the old check-uptodate.sh script. Only when a new list of configuration options is added which is a much less likely occurrence. To avoid this happening at all it was proposed by the Cinder team that we use the code I wrote for the hacking checks to dynamically go through the files and create cinder/opts.py whenever 'tox -egenconfig' is run. Doing this makes me uncomfortable as it is not consistent with anything else I am familiar with in OpenStack and is not consistent with what other projects are doing to handle this problem. In discussion with Doug Hellman, the approach also seemed to cause him concern. So, I don't believe that is the right solution. An alternative that may be a better solution was proposed by Doug: We could even further reduce the occurrence of such issues by moving the list_opts() function down into each driver and have an entry point for oslo.config.opts in setup.cfg for each of the drivers. As with the currently proposed solution, the developer doesn't have to edit a top level file for a new configuration option. This solution adds that the developer doesn't have to edit a top level file to add a new configuration item list to their driver. With this approach the change would happen in the driver's list_opts() function, rather than in cinder/opts.py . The only time that setup.cfg would needed to edited is when a new package is added or when a new driver is added. This would reduce some of the already minimal burden on the developer. We, however, would need to agree upon some method for aggregating together the options lists on a per package (i.e. cinder.scheduler, cinder.api) level. This approach, however, also has the advantage of providing a better indication in the sample config file of where the options are coming from. That is an improvement over
Re: [openstack-dev] [oslo][cinder][nova][neutron] going forward to oslo-config-generator ...
Hi, One of the issues that we had in Nova was that when we moved to oslo libraries configuration options support by the libraries were no longer present in the generated configuration file. Is this something that is already supported or planned (sorry for being a little ignorant here). In neutron things may be a little more challenging as there are many different plugins and with the decomposition things may have additional challenges. The configuration binding is done via the external decomposed code and not in the neutron code base. So it is not clear how that code may be parsed to generate the sample configuration. Thanks Gary On 3/21/15, 12:01 AM, Jay S. Bryant jsbry...@electronicjungle.net wrote: All, Let me start with the TLDR; Cinder, Nova and Neutron have lots of configuration options that need to be processed by oslo-config-generator to create the project.conf.sample file. There are a couple of different ways this could be done. I have one proposal out, which has raised concerns, there is a second approach that could be taken which I am proposing below. Please read on if you have a strong opinion on the precedent we will try to set in Cinder. :-) We discussed in the oslo meeting a couple of weeks ago a plan for how Cinder was going to blaze a trail to the new oslo-config-generator. The result of that discussion and work is here: [1] It needs some more work but has the bare bones pieces there to move to using oslo-config-generator. With the change I have written extensive hacking checks that ensure that any lists that are registered with register_opts() are included in the base cinder/opts.py file that is then a single entry point that pulls all of the options together to generate the cinder.conf.sample file. This has raised concern, however, that whenever a developer adds a new list of configuration options, they are going to have to know to go back to cinder/opts.py and add their module and option list there. The hacking check should catch this before code is submitted, but we are possibly setting ourselves up for cases where the patch will fail in the gate because updates are not made in all the correct places and because pep8 isn't run before the patch is pushed. It is important to note, that this will not happen every time a configuration option is changed or added, as was the case with the old check-uptodate.sh script. Only when a new list of configuration options is added which is a much less likely occurrence. To avoid this happening at all it was proposed by the Cinder team that we use the code I wrote for the hacking checks to dynamically go through the files and create cinder/opts.py whenever 'tox -egenconfig' is run. Doing this makes me uncomfortable as it is not consistent with anything else I am familiar with in OpenStack and is not consistent with what other projects are doing to handle this problem. In discussion with Doug Hellman, the approach also seemed to cause him concern. So, I don't believe that is the right solution. An alternative that may be a better solution was proposed by Doug: We could even further reduce the occurrence of such issues by moving the list_opts() function down into each driver and have an entry point for oslo.config.opts in setup.cfg for each of the drivers. As with the currently proposed solution, the developer doesn't have to edit a top level file for a new configuration option. This solution adds that the developer doesn't have to edit a top level file to add a new configuration item list to their driver. With this approach the change would happen in the driver's list_opts() function, rather than in cinder/opts.py . The only time that setup.cfg would needed to edited is when a new package is added or when a new driver is added. This would reduce some of the already minimal burden on the developer. We, however, would need to agree upon some method for aggregating together the options lists on a per package (i.e. cinder.scheduler, cinder.api) level. This approach, however, also has the advantage of providing a better indication in the sample config file of where the options are coming from. That is an improvement over what I have currently proposed. Does Doug's proposal sound more agreeable to everyone? It is important to note that the fact that some manual intervention is required to 'plumb' in the new configuration options was done by design. There is a little more work required to make options available to oslo-config-generator but the ability to use different namespaces, different sample configs, etc were added with the new generator. These additional capabilities were requested by other projects. So, moving to this design does have the potential for more long-term gain. Thanks for taking the time to consider this! Jay [1] https://review.openstack.org/#/c/165431/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe:
Re: [openstack-dev] [oslo][cinder][nova][neutron] going forward to oslo-config-generator ...
Jay, That sound reasonable. We will need to document in a guide for driver developers what to do when new option is added deprecated in conf file for a driver. Expect that nothing extra will need to be done beyond what we are doing now when new functionality added/deprecated from scheduler/default driver and perculates into drivers a release later. I can also comment directly on the patch if it make sense. Thanks, Arkady -Original Message- From: Jay S. Bryant [mailto:jsbry...@electronicjungle.net] Sent: Friday, March 20, 2015 5:02 PM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [oslo][cinder][nova][neutron] going forward to oslo-config-generator ... All, Let me start with the TLDR; Cinder, Nova and Neutron have lots of configuration options that need to be processed by oslo-config-generator to create the .conf.sample file. There are a couple of different ways this could be done. I have one proposal out, which has raised concerns, there is a second approach that could be taken which I am proposing below. Please read on if you have a strong opinion on the precedent we will try to set in Cinder. :-) We discussed in the oslo meeting a couple of weeks ago a plan for how Cinder was going to blaze a trail to the new oslo-config-generator. The result of that discussion and work is here: [1] It needs some more work but has the bare bones pieces there to move to using oslo-config-generator. With the change I have written extensive hacking checks that ensure that any lists that are registered with register_opts() are included in the base cinder/opts.py file that is then a single entry point that pulls all of the options together to generate the cinder.conf.sample file. This has raised concern, however, that whenever a developer adds a new list of configuration options, they are going to have to know to go back to cinder/opts.py and add their module and option list there. The hacking check should catch this before code is submitted, but we are possibly setting ourselves up for cases where the patch will fail in the gate because updates are not made in all the correct places and because pep8 isn't run before the patch is pushed. It is important to note, that this will not happen every time a configuration option is changed or added, as was the case with the old check-uptodate.sh script. Only when a new list of configuration options is added which is a much less likely occurrence. To avoid this happening at all it was proposed by the Cinder team that we use the code I wrote for the hacking checks to dynamically go through the files and create cinder/opts.py whenever 'tox -egenconfig' is run. Doing this makes me uncomfortable as it is not consistent with anything else I am familiar with in OpenStack and is not consistent with what other projects are doing to handle this problem. In discussion with Doug Hellman, the approach also seemed to cause him concern. So, I don't believe that is the right solution. An alternative that may be a better solution was proposed by Doug: We could even further reduce the occurrence of such issues by moving the list_opts() function down into each driver and have an entry point for oslo.config.opts in setup.cfg for each of the drivers. As with the currently proposed solution, the developer doesn't have to edit a top level file for a new configuration option. This solution adds that the developer doesn't have to edit a top level file to add a new configuration item list to their driver. With this approach the change would happen in the driver's list_opts() function, rather than in cinder/opts.py . The only time that setup.cfg would needed to edited is when a new package is added or when a new driver is added. This would reduce some of the already minimal burden on the developer. We, however, would need to agree upon some method for aggregating together the options lists on a per package (i.e. cinder.scheduler, cinder.api) level. This approach, however, also has the advantage of providing a better indication in the sample config file of where the options are coming from. That is an improvement over what I have currently proposed. Does Doug's proposal sound more agreeable to everyone? It is important to note that the fact that some manual intervention is required to 'plumb' in the new configuration options was done by design. There is a little more work required to make options available to oslo-config-generator but the ability to use different namespaces, different sample configs, etc were added with the new generator. These additional capabilities were requested by other projects. So, moving to this design does have the potential for more long-term gain. Thanks for taking the time to consider this! Jay [1] https://review.openstack.org/#/c/165431/ __ OpenStack Development Mailing List (not for usage
