Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares
It looks like the additional features added, in particular the 'oslo_config_project' property, needs to be documented. I have added some documentation into the keystonemiddleware too: https://review.openstack.org/#/c/208965/ --- Mehdi Abaakouk mail: sil...@sileht.net irc: sileht __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares
Hi, Also to support some of the newer services that don't use paste i think we should absolutely make it so that the CONF object is passed to middleware rather than sourced globally. I think gnochhi and zaqar both fit into this case. For example, Gnocchi doesn't use paste, deployer adds middlewares like this: [api] middlewares = oslo_middleware.request_id.RequestId,oslo_middleware.cors.CORS,keystonemiddleware.auth_token.AuthProtocol Of course because of the current issue, we browse the pipeline to pass the correct options to keystonemiddleware.auth_token.AuthProtocol, to make it works. My change allows to remove this workaround. The problem i see with what you are saying is that it is mixing deployment methodologies in a way that is unintended. Paste is designed to allow deployers to add and remove middleware independent of configuring the service. This means that at paste time there is no CONF object unless it's globally registered and this is why most middlewares allow taking options from paste.ini because if you don't have global CONF then it's the only way to actually get options into the middleware. My change adds a new way that doesn't use global CONF object but still read options from the application configuration file. Fixing this problem is always where i loose enthusiasm for removing global CONF files. With my solution, if all applications update they paste.ini configuration, we can remove the global CONF from keystonemiddleware and only relies on options loaded via paste or via the local oslo.cfg object. keystonemiddleware become like most middlewares and does not depend anymore on the application. (If you want that I can write a patch to deprecate usage of global CONF object once my change is merged, and update paste.ini of other projects) From a developer perspective I feel the solution is for us to reconsider how we deploy and configure middleware. If you are using paste those pieces of middleware should each be able to be configured without any integration with the service underneath it. Otherwise if your service needs a piece of middleware like auth_token middleware to operate or relies on oslo.config options like cors then that is not something that should be controlled by paste. From a deployer perspective there is no great answer i just want everything to be in oslo.config. Yeah, that's the goal, whatever the app uses global or not, uses paste or not, etc... for the deployer, the configuration of middlewares are in oslo.config. Equally from a deployer perspective this wasn't an issue until aodh (et al) decided to remove the global CONF object which developers from all the projects hate but live with. I don't mean to imply we shouldn't look at better ways to do things, but if you're main concern is deployers the easiest thing we can do for consistency is add back the global CONF object or remove paste from aodh. :) I will be sad to readd the global CONF object (removing paste is not really an option for us I think). Cheers, --- Mehdi Abaakouk mail: sil...@sileht.net irc: sileht __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares
On Tue, Aug 11 2015, Michael Krotscheck wrote: It looks like the additional features added, in particular the 'oslo_config_project' property, needs to be documented. It has been documented with one of the patch, you can see it here: http://docs.openstack.org/developer/oslo.middleware/oslo_config.html#configuration-with-paste-deploy-and-the-oslo-config Though if you think something is not clear I guess we can enhance it. :) A deeper read shows that you're right, existing functionality was not broken. Apologies for being heavy handed in my response. Cool, no problem! Best, -- Julien Danjou // Free Software hacker // http://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares
On Mon, Aug 10 2015, Michael Krotscheck wrote: Hi Michael, It appears that the patch related to this discussion were rushed through rather quickly, and without appropriate updates to the documentation. The documentation of the library no longer matches the actual functionality, and will need to be updated before we can release a new version of oslo_middleware. I'd be happy to fix the documentation, but I cannot find any point that does not work anymore or may have been broken by those pathes. Could you hint me in the right direction? Thanks! Best, -- Julien Danjou // Free Software hacker // http://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares
It looks like the additional features added, in particular the 'oslo_config_project' property, needs to be documented. A deeper read shows that you're right, existing functionality was not broken. Apologies for being heavy handed in my response. Michael On Tue, Aug 11, 2015 at 2:53 AM Julien Danjou jul...@danjou.info wrote: On Mon, Aug 10 2015, Michael Krotscheck wrote: Hi Michael, It appears that the patch related to this discussion were rushed through rather quickly, and without appropriate updates to the documentation. The documentation of the library no longer matches the actual functionality, and will need to be updated before we can release a new version of oslo_middleware. I'd be happy to fix the documentation, but I cannot find any point that does not work anymore or may have been broken by those pathes. Could you hint me in the right direction? Thanks! Best, -- Julien Danjou // Free Software hacker // http://julien.danjou.info __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares
It appears that the patch related to this discussion were rushed through rather quickly, and without appropriate updates to the documentation. The documentation of the library no longer matches the actual functionality, and will need to be updated before we can release a new version of oslo_middleware. Mehdi, if you would please update the documentation in oslo_middleware as quickly as possible? Some of us have features that we're trying to land in liberty, which we cannot do without a new release of this library. To the rest of the Oslo team, I'd like to propose that if we do not have appropriate updated documentation by the end of this week, we revert Medhi's patches so that we can unblock the library. In case you're curious, the related patch is here: https://review.openstack.org/#/c/209817/3 Michael Krotscheck On Sun, Aug 9, 2015 at 7:58 PM Jamie Lennox jamielen...@redhat.com wrote: - Original Message - From: Jamie Lennox jamielen...@redhat.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Monday, August 10, 2015 12:36:14 PM Subject: Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares - Original Message - From: Mehdi Abaakouk sil...@sileht.net To: openstack-dev@lists.openstack.org Sent: Friday, August 7, 2015 1:57:54 AM Subject: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares Hi, I want to share with you some problems I have recently encountered with openstack middlewares and oslo.config. The issues -- In project Gnocchi, I would use oslo.middleware.cors, I have expected to just put the name of the middleware to the wsgi pipeline, but I can't. The middlewares only works if you pass the oslo_config.cfg.ConfigOpts() object or via 'paste-deploy'... Gnocchi doesn't use paste-deploy, so I have to modify the code to load it... (For the keystonemiddleware, Gnocchi already have a special handling/hack to load it [1] and [2]). I don't want to write the same hack for each openstack middlewares. In project Aodh (ceilometer-alarm), we recently got an issue with keystonemiddleware since we remove the usage of the global object oslo_config.cfg.CONF. The middleware doesn't load its options from the config file of aodh anymore. Our authentication is broken. We can still pass them through paste-deploy configuration but this looks a method of the past. I still don't want to write a hack for each openstack middlewares. Then I have digged into other middlewares and applications to see how they handle their conf. oslo_middlewarre.sizelimit and oslo_middlewarre.ssl take options only via the global oslo_config.cfg.CONF. So they are unusable for application that doesn't use this global object. oslo_middleware.healthcheck take options as dict like any other python middleware. This is suitable for 'paste-deploy'. But doesn't allow configuration via oslo.config, doesn't have a strong config options type checking and co. Zaqar seems got same kind of issue about keystonemiddleware, and just write a hack to workaround the issue (monkeypatch the cfg.CONF of keystonemiddleware with their local version of the object [3] and then transform the loaded options into a dict to pass them via the legacy middleware dict options [4]) . Most applications, just still use the global object for the configuration and don't, yet, see those issues. All of that is really not consistent. This is confusing for developer to have some middlewares that need pre-setup, enforce them to rely on global python object, and some others not. This is confusing for deployer their can't do the configuration of middlewares in the same way for each middlewares and each projects. But keystonemiddleware, oslo.middleware.cors,... are supposed to be wsgi middlewares, something that is independant of the app. And this is not really the case. From my point of view and what wsgi looks like generally in python, the middleware object should be just MyMiddleware(app, options_as_dict), if the middleware want to rely to another configuration system it should do the setup/initialisation itself. So, how to solve that ? Do you agree: * all openstack middlewares should load their options with oslo.config ? this permits type checking and all other features it provides, it's cool :) configuration in paste-deploy conf is thing of past * we must support local AND global oslo.config object ? This is an application choice not something enforced by middleware. The deployer experience should be the same in both case. * the middleware must be responsible of the section name in the oslo.config ? Gnocchi/Zaqar hack have to hardcode
Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares
- Original Message -
From: Mehdi Abaakouk sil...@sileht.net
To: openstack-dev@lists.openstack.org
Sent: Friday, August 7, 2015 1:57:54 AM
Subject: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares
Hi,
I want to share with you some problems I have recently encountered with
openstack middlewares and oslo.config.
The issues
--
In project Gnocchi, I would use oslo.middleware.cors, I have expected to
just put the name of the middleware to the wsgi pipeline, but I can't.
The middlewares only works if you pass the oslo_config.cfg.ConfigOpts()
object or via 'paste-deploy'... Gnocchi doesn't use paste-deploy, so
I have to modify the code to load it...
(For the keystonemiddleware, Gnocchi already have a special
handling/hack to load it [1] and [2]).
I don't want to write the same hack for each openstack middlewares.
In project Aodh (ceilometer-alarm), we recently got an issue with
keystonemiddleware since we remove the usage of the global object
oslo_config.cfg.CONF. The middleware doesn't load its options from the
config file of aodh anymore. Our authentication is broken.
We can still pass them through paste-deploy configuration but this looks
a method of the past. I still don't want to write a hack for each
openstack middlewares.
Then I have digged into other middlewares and applications to see how
they handle their conf.
oslo_middlewarre.sizelimit and oslo_middlewarre.ssl take options only
via the global oslo_config.cfg.CONF. So they are unusable for application
that doesn't use this global object.
oslo_middleware.healthcheck take options as dict like any other python
middleware. This is suitable for 'paste-deploy'. But doesn't allow
configuration via oslo.config, doesn't have a strong config options
type checking and co.
Zaqar seems got same kind of issue about keystonemiddleware, and just
write a hack to workaround the issue (monkeypatch the cfg.CONF of
keystonemiddleware with their local version of the object [3] and then
transform the loaded options into a dict to pass them via the legacy
middleware dict options [4]) .
Most applications, just still use the global object for the
configuration and don't, yet, see those issues.
All of that is really not consistent.
This is confusing for developer to have some middlewares that need pre-setup,
enforce them to rely on global python object, and some others not.
This is confusing for deployer their can't do the configuration of
middlewares in the same way for each middlewares and each projects.
But keystonemiddleware, oslo.middleware.cors,... are supposed to be wsgi
middlewares, something that is independant of the app.
And this is not really the case.
From my point of view and what wsgi looks like generally in python, the
middleware object should be just MyMiddleware(app, options_as_dict),
if the middleware want to rely to another configuration system it should
do the setup/initialisation itself.
So, how to solve that ?
Do you agree:
* all openstack middlewares should load their options with oslo.config ?
this permits type checking and all other features it provides, it's cool :)
configuration in paste-deploy conf is thing of past
* we must support local AND global oslo.config object ?
This is an application choice not something enforced by middleware.
The deployer experience should be the same in both case.
* the middleware must be responsible of the section name in the oslo.config ?
Gnocchi/Zaqar hack have to hardcode the section name in their code,
this doesn't looks good.
* we must support legacy python signature for WSGI object,
MyMiddleware(app, options_as_dict) ? To be able to use paste for
application/deployer that want it and not break already deployed things.
I really think all our middlewares should be consistent:
* to be usable by all applications without enforcing them to write crap
around them.
* and to made the deployer life easier.
Possible solution:
--
I have already started to work on something that do all of that for all
middlewares [5], [6]
The idea is, the middleware should create a oslo_config.cfg.ConfigOpts()
(instead of rely on the global one) and load the configuration file of the
application in. oslo.config will discover the file location just with the
name of application as usual.
So the middleware can now be loaded like this:
code example:
app = MyMiddleware(app, {oslo_config_project: aodh})
paste-deploy example:
[filter:foobar]
paste.filter_factory = foobar:MyMiddleware.filter_factory
oslo_config_project = aodh
oslo_config.cfg.ConfigOpts() will easly find the /etc/aodh/aodh.conf,
This cut the hidden links between middleware and the application
(through the global object).
And of course if oslo_config_project is not provided, the middleware
fallback the global
Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares
- Original Message - From: Jamie Lennox jamielen...@redhat.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Monday, August 10, 2015 12:36:14 PM Subject: Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares - Original Message - From: Mehdi Abaakouk sil...@sileht.net To: openstack-dev@lists.openstack.org Sent: Friday, August 7, 2015 1:57:54 AM Subject: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares Hi, I want to share with you some problems I have recently encountered with openstack middlewares and oslo.config. The issues -- In project Gnocchi, I would use oslo.middleware.cors, I have expected to just put the name of the middleware to the wsgi pipeline, but I can't. The middlewares only works if you pass the oslo_config.cfg.ConfigOpts() object or via 'paste-deploy'... Gnocchi doesn't use paste-deploy, so I have to modify the code to load it... (For the keystonemiddleware, Gnocchi already have a special handling/hack to load it [1] and [2]). I don't want to write the same hack for each openstack middlewares. In project Aodh (ceilometer-alarm), we recently got an issue with keystonemiddleware since we remove the usage of the global object oslo_config.cfg.CONF. The middleware doesn't load its options from the config file of aodh anymore. Our authentication is broken. We can still pass them through paste-deploy configuration but this looks a method of the past. I still don't want to write a hack for each openstack middlewares. Then I have digged into other middlewares and applications to see how they handle their conf. oslo_middlewarre.sizelimit and oslo_middlewarre.ssl take options only via the global oslo_config.cfg.CONF. So they are unusable for application that doesn't use this global object. oslo_middleware.healthcheck take options as dict like any other python middleware. This is suitable for 'paste-deploy'. But doesn't allow configuration via oslo.config, doesn't have a strong config options type checking and co. Zaqar seems got same kind of issue about keystonemiddleware, and just write a hack to workaround the issue (monkeypatch the cfg.CONF of keystonemiddleware with their local version of the object [3] and then transform the loaded options into a dict to pass them via the legacy middleware dict options [4]) . Most applications, just still use the global object for the configuration and don't, yet, see those issues. All of that is really not consistent. This is confusing for developer to have some middlewares that need pre-setup, enforce them to rely on global python object, and some others not. This is confusing for deployer their can't do the configuration of middlewares in the same way for each middlewares and each projects. But keystonemiddleware, oslo.middleware.cors,... are supposed to be wsgi middlewares, something that is independant of the app. And this is not really the case. From my point of view and what wsgi looks like generally in python, the middleware object should be just MyMiddleware(app, options_as_dict), if the middleware want to rely to another configuration system it should do the setup/initialisation itself. So, how to solve that ? Do you agree: * all openstack middlewares should load their options with oslo.config ? this permits type checking and all other features it provides, it's cool :) configuration in paste-deploy conf is thing of past * we must support local AND global oslo.config object ? This is an application choice not something enforced by middleware. The deployer experience should be the same in both case. * the middleware must be responsible of the section name in the oslo.config ? Gnocchi/Zaqar hack have to hardcode the section name in their code, this doesn't looks good. * we must support legacy python signature for WSGI object, MyMiddleware(app, options_as_dict) ? To be able to use paste for application/deployer that want it and not break already deployed things. I really think all our middlewares should be consistent: * to be usable by all applications without enforcing them to write crap around them. * and to made the deployer life easier. Possible solution: -- I have already started to work on something that do all of that for all middlewares [5], [6] The idea is, the middleware should create a oslo_config.cfg.ConfigOpts() (instead of rely on the global one) and load the configuration file of the application in. oslo.config will discover the file location just with the name of application as usual. So the middleware can now be loaded like this: code example: app = MyMiddleware(app
Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares
On Fri, Aug 07 2015, Michael Krotscheck wrote: Here's the thing. If the middleware is _optional_ like keystone auth, then including it via paste.ini makes way more sense. In fact, keystone auth has gone to great lengths to have no dependencies for that very same reason. If, instead, the middleware is a feature that should ship with the service - like CORS, or a simple caching layer - then it should be baked into your application initialization directly. Except that AFAIU you cannot use keystonemiddleware neither without hardcoding its usage pattern (even if it's optional) in your code in the same way you're talking about with CORS. Which is then terrible. -- Julien Danjou # Free Software hacker # http://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares
Michael Krotscheck wrote: On Thu, Aug 6, 2015 at 10:08 AM Mehdi Abaakouk sil...@sileht.net mailto:sil...@sileht.net wrote: Yes, but you can't use oslo.config without hardcode the loading the middleware to pass the oslo.config object into the application. Yes, and that is intentional, because the use of global variables of any sort is bad. They're unconstrained, there's no access control to guarantee the thing you want hasn't been modified, and in the case of oslo.config, they require initialization before they can be used. Writing any kind of logic that assumes that a magic global instance has been initialized is brittle. The pastedeploy wsgi chain is a perfect example, because the application is only created after the middleware chain has been executed. This leaves you with - at best - a malfunctioning piece of middleware that breaks because the global oslo.config object isn't ready. At worst it's a security vulnerability that permits bypassing things like keystone. Passing the config object is a _good_ thing, because it doesn't rely on magic. Magic is bad. If someone looks at the code and says: I wonder how this piece of middleware gets its values, and they don't see the config object being passed, they have to dig into the middleware itself to figure out what's going on. It only relies on the rest of the config object to 'magically' fetch the values of attributes from somewhere, organize them into some grouping, and perform the right type checking/conversion (type checking in python, woah), and the magic of digging into help strings instead of docstrings (which means the generated code docs of config object using components either have to replicate the help string or do something else)... but point taken ;) (I've always preferred APIs that use the standard things u see in python to document arguments, parameter types, what there usage is; yes I know the history here, just saying this is just another different kind of magic). I'm clearly on the operator side too, and I just try to find a solution to be able to use all middlewares without having to write code for each in each application and use oslo.config. Zaqar, Gnocchi and Aodh are the first projects that do to not use cfg.CONF and can't load many middlewares without writing code for each. When middleware should be just something that deployer enabled and configuration. Our middleware looks more like a lib than a middleware) Sorry, but you're talking from the point of view of someone who wants to not have to write code for each. That's a developer. It's our job as developers to write code until it's as easy as possible, and passing in a config object is _dead simple_ in your application initialization. Here's the thing. If the middleware is _optional_ like keystone auth, then including it via paste.ini makes way more sense. In fact, keystone auth has gone to great lengths to have no dependencies for that very same reason. If, instead, the middleware is a feature that should ship with the service - like CORS, or a simple caching layer - then it should be baked into your application initialization directly. Michael __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares
On Thu, Aug 6, 2015 at 10:08 AM Mehdi Abaakouk sil...@sileht.net wrote: Yes, but you can't use oslo.config without hardcode the loading the middleware to pass the oslo.config object into the application. Yes, and that is intentional, because the use of global variables of any sort is bad. They're unconstrained, there's no access control to guarantee the thing you want hasn't been modified, and in the case of oslo.config, they require initialization before they can be used. Writing any kind of logic that assumes that a magic global instance has been initialized is brittle. The pastedeploy wsgi chain is a perfect example, because the application is only created after the middleware chain has been executed. This leaves you with - at best - a malfunctioning piece of middleware that breaks because the global oslo.config object isn't ready. At worst it's a security vulnerability that permits bypassing things like keystone. Passing the config object is a _good_ thing, because it doesn't rely on magic. Magic is bad. If someone looks at the code and says: I wonder how this piece of middleware gets its values, and they don't see the config object being passed, they have to dig into the middleware itself to figure out what's going on. I'm clearly on the operator side too, and I just try to find a solution to be able to use all middlewares without having to write code for each in each application and use oslo.config. Zaqar, Gnocchi and Aodh are the first projects that do to not use cfg.CONF and can't load many middlewares without writing code for each. When middleware should be just something that deployer enabled and configuration. Our middleware looks more like a lib than a middleware) Sorry, but you're talking from the point of view of someone who wants to not have to write code for each. That's a developer. It's our job as developers to write code until it's as easy as possible, and passing in a config object is _dead simple_ in your application initialization. Here's the thing. If the middleware is _optional_ like keystone auth, then including it via paste.ini makes way more sense. In fact, keystone auth has gone to great lengths to have no dependencies for that very same reason. If, instead, the middleware is a feature that should ship with the service - like CORS, or a simple caching layer - then it should be baked into your application initialization directly. Michael __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares
Hi there! The most recent version of the CORS middleware (~2.4) no longer requires the use of Oslo.config, and supports pastedeploy. While using oslo.config provides far better features - such as multiple origins - it doesn't prevent you from using it in the paste pipeline. The documentation has been updated to reflect this. As for the How do we proceed question, I feel your question can be better asked as: Do we support developers, or do we support operators? The developer stance is that of architectural purity, of minimizing dependencies, and of staying as true to default as possible. The Operator's stance is that of simplified configuration, performance, good documentation, and reliability. I fall on the operators side, and as a result feel that we should be using oslo.config for everything. One single configuration method across services, consistent naming conventions, autogenerated with sane options, with tooling and testing that makes it reliable. Special Snowflakes really just add cognitive friction, documentation overhead, and noise. I'm not saying oslo is perfect. I'm not saying that how oslo is used is always correct. But we're fixing it, and consistency, in my mind, always trumps ideology. Michael On Thu, Aug 6, 2015 at 9:02 AM Mehdi Abaakouk sil...@sileht.net wrote: Hi, I want to share with you some problems I have recently encountered with openstack middlewares and oslo.config. The issues -- In project Gnocchi, I would use oslo.middleware.cors, I have expected to just put the name of the middleware to the wsgi pipeline, but I can't. The middlewares only works if you pass the oslo_config.cfg.ConfigOpts() object or via 'paste-deploy'... Gnocchi doesn't use paste-deploy, so I have to modify the code to load it... (For the keystonemiddleware, Gnocchi already have a special handling/hack to load it [1] and [2]). I don't want to write the same hack for each openstack middlewares. In project Aodh (ceilometer-alarm), we recently got an issue with keystonemiddleware since we remove the usage of the global object oslo_config.cfg.CONF. The middleware doesn't load its options from the config file of aodh anymore. Our authentication is broken. We can still pass them through paste-deploy configuration but this looks a method of the past. I still don't want to write a hack for each openstack middlewares. Then I have digged into other middlewares and applications to see how they handle their conf. oslo_middlewarre.sizelimit and oslo_middlewarre.ssl take options only via the global oslo_config.cfg.CONF. So they are unusable for application that doesn't use this global object. oslo_middleware.healthcheck take options as dict like any other python middleware. This is suitable for 'paste-deploy'. But doesn't allow configuration via oslo.config, doesn't have a strong config options type checking and co. Zaqar seems got same kind of issue about keystonemiddleware, and just write a hack to workaround the issue (monkeypatch the cfg.CONF of keystonemiddleware with their local version of the object [3] and then transform the loaded options into a dict to pass them via the legacy middleware dict options [4]) . Most applications, just still use the global object for the configuration and don't, yet, see those issues. All of that is really not consistent. This is confusing for developer to have some middlewares that need pre-setup, enforce them to rely on global python object, and some others not. This is confusing for deployer their can't do the configuration of middlewares in the same way for each middlewares and each projects. But keystonemiddleware, oslo.middleware.cors,... are supposed to be wsgi middlewares, something that is independant of the app. And this is not really the case. From my point of view and what wsgi looks like generally in python, the middleware object should be just MyMiddleware(app, options_as_dict), if the middleware want to rely to another configuration system it should do the setup/initialisation itself. So, how to solve that ? Do you agree: * all openstack middlewares should load their options with oslo.config ? this permits type checking and all other features it provides, it's cool :) configuration in paste-deploy conf is thing of past * we must support local AND global oslo.config object ? This is an application choice not something enforced by middleware. The deployer experience should be the same in both case. * the middleware must be responsible of the section name in the oslo.config ? Gnocchi/Zaqar hack have to hardcode the section name in their code, this doesn't looks good. * we must support legacy python signature for WSGI object, MyMiddleware(app, options_as_dict) ? To be able to use paste for application/deployer that want it and not break already deployed things. I really think all our middlewares should be consistent: * to be
Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares
On Thu, Aug 06 2015, Mehdi Abaakouk wrote: This is confusing for developer to have some middlewares that need pre-setup, enforce them to rely on global python object, and some others not. This is confusing for deployer their can't do the configuration of middlewares in the same way for each middlewares and each projects. Agreed, this is a terrible design. Do you agree: * all openstack middlewares should load their options with oslo.config ? this permits type checking and all other features it provides, it's cool :) configuration in paste-deploy conf is thing of past That sounds like a good idea. * we must support local AND global oslo.config object ? This is an application choice not something enforced by middleware. The deployer experience should be the same in both case. Well, I guess if you support local you can anyway use the global as local. :) * the middleware must be responsible of the section name in the oslo.config ? Gnocchi/Zaqar hack have to hardcode the section name in their code, this doesn't looks good. I'd think so. * we must support legacy python signature for WSGI object, MyMiddleware(app, options_as_dict) ? To be able to use paste for application/deployer that want it and not break already deployed things. Definitely agreed. Possible solution: -- I have already started to work on something that do all of that for all middlewares [5], [6] […] I have already tested that in Gnocchi and Aodh, and that solves all of my issues. Remove all hacks, the application doesn't need special pre setup. All our middleware become normal middleware but still can use oslo.config. WDYT ? I already +2 some of the patches, I think the approach is good and sane, and I honestly don't have anything better so thank you Mehdi. :) Cheers, -- Julien Danjou ;; Free Software hacker ;; http://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares
On Thu, Aug 06, 2015 at 04:25:58PM +, Michael Krotscheck wrote: Hi there! The most recent version of the CORS middleware (~2.4) no longer requires the use of Oslo.config, and supports pastedeploy. While using oslo.config provides far better features - such as multiple origins - it doesn't prevent you from using it in the paste pipeline. The documentation has been updated to reflect this. Yes, but you can't use oslo.config without hardcode the loading the middleware to pass the oslo.config object into the application. I fall on the operators side, and as a result feel that we should be using oslo.config for everything. One single configuration method across services, consistent naming conventions, autogenerated with sane options, with tooling and testing that makes it reliable. Special Snowflakes really just add cognitive friction, documentation overhead, and noise. I'm clearly on the operator side too, and I just try to find a solution to be able to use all middlewares without having to write code for each in each application and use oslo.config. Zaqar, Gnocchi and Aodh are the first projects that do to not use cfg.CONF and can't load many middlewares without writing code for each. When middleware should be just something that deployer enabled and configuration. Our middleware looks more like a lib than a middleware) (Zaqar, Gnocchi and Aodh have written hack for keystonemiddleware because this is an essential piece but other middlewares are broken for them). Cheers, -- Mehdi Abaakouk mail: sil...@sileht.net irc: sileht __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo][keystone] oslo_config and wsgi middlewares
Hi,
I want to share with you some problems I have recently encountered with
openstack middlewares and oslo.config.
The issues
--
In project Gnocchi, I would use oslo.middleware.cors, I have expected to
just put the name of the middleware to the wsgi pipeline, but I can't.
The middlewares only works if you pass the oslo_config.cfg.ConfigOpts()
object or via 'paste-deploy'... Gnocchi doesn't use paste-deploy, so
I have to modify the code to load it...
(For the keystonemiddleware, Gnocchi already have a special
handling/hack to load it [1] and [2]).
I don't want to write the same hack for each openstack middlewares.
In project Aodh (ceilometer-alarm), we recently got an issue with
keystonemiddleware since we remove the usage of the global object
oslo_config.cfg.CONF. The middleware doesn't load its options from the
config file of aodh anymore. Our authentication is broken.
We can still pass them through paste-deploy configuration but this looks
a method of the past. I still don't want to write a hack for each
openstack middlewares.
Then I have digged into other middlewares and applications to see how
they handle their conf.
oslo_middlewarre.sizelimit and oslo_middlewarre.ssl take options only
via the global oslo_config.cfg.CONF. So they are unusable for application
that doesn't use this global object.
oslo_middleware.healthcheck take options as dict like any other python
middleware. This is suitable for 'paste-deploy'. But doesn't allow
configuration via oslo.config, doesn't have a strong config options
type checking and co.
Zaqar seems got same kind of issue about keystonemiddleware, and just
write a hack to workaround the issue (monkeypatch the cfg.CONF of
keystonemiddleware with their local version of the object [3] and then
transform the loaded options into a dict to pass them via the legacy
middleware dict options [4]) .
Most applications, just still use the global object for the
configuration and don't, yet, see those issues.
All of that is really not consistent.
This is confusing for developer to have some middlewares that need pre-setup,
enforce them to rely on global python object, and some others not.
This is confusing for deployer their can't do the configuration of
middlewares in the same way for each middlewares and each projects.
But keystonemiddleware, oslo.middleware.cors,... are supposed to be wsgi
middlewares, something that is independant of the app.
And this is not really the case.
From my point of view and what wsgi looks like generally in python, the
middleware object should be just MyMiddleware(app, options_as_dict),
if the middleware want to rely to another configuration system it should
do the setup/initialisation itself.
So, how to solve that ?
Do you agree:
* all openstack middlewares should load their options with oslo.config ?
this permits type checking and all other features it provides, it's cool :)
configuration in paste-deploy conf is thing of past
* we must support local AND global oslo.config object ?
This is an application choice not something enforced by middleware.
The deployer experience should be the same in both case.
* the middleware must be responsible of the section name in the oslo.config ?
Gnocchi/Zaqar hack have to hardcode the section name in their code,
this doesn't looks good.
* we must support legacy python signature for WSGI object,
MyMiddleware(app, options_as_dict) ? To be able to use paste for
application/deployer that want it and not break already deployed things.
I really think all our middlewares should be consistent:
* to be usable by all applications without enforcing them to write crap around
them.
* and to made the deployer life easier.
Possible solution:
--
I have already started to work on something that do all of that for all
middlewares [5], [6]
The idea is, the middleware should create a oslo_config.cfg.ConfigOpts()
(instead of rely on the global one) and load the configuration file of the
application in. oslo.config will discover the file location just with the
name of application as usual.
So the middleware can now be loaded like this:
code example:
app = MyMiddleware(app, {oslo_config_project: aodh})
paste-deploy example:
[filter:foobar]
paste.filter_factory = foobar:MyMiddleware.filter_factory
oslo_config_project = aodh
oslo_config.cfg.ConfigOpts() will easly find the /etc/aodh/aodh.conf,
This cut the hidden links between middleware and the application
(through the global object).
And of course if oslo_config_project is not provided, the middleware
fallback the global oslo.config object.
The middleware options can still be passed via the legacy dict.
The backward compatibility is conserved.
I have already tested that in Gnocchi and Aodh, and that solves all of
my issues. Remove all hacks, the application doesn't need special pre
setup. All our middleware become normal middleware but
