Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
Hi Julien, On Mon, 2013-12-02 at 16:45 +0100, Julien Danjou wrote: On Mon, Nov 18 2013, Julien Danjou wrote: https://blueprints.launchpad.net/oslo/+spec/messaging-decouple-cfg So I've gone through the code and started to write a plan on how I'd do things: https://wiki.openstack.org/wiki/Oslo/blueprints/messaging-decouple-cfg I don't think I missed too much, though I didn't run into all tiny details. Please feel free to tell me if I miss anything obvious, otherwise I'll try to start submitting patches, one at a time, to get this into shape step by step. Thanks for writing this up, I really appreciate it. I would like to spend more time getting to the bottom of what we're trying to solve here. If the goal is allow applications to use oslo.messaging without using oslo.config, then what's driving this? I'm guessing some possible answers: 1) I don't want to use a global CONF object This is a strawman - I think we all realize the conf object you pass to oslo.messaging doesn't have to be cfg.CONF. Just putting it here to make sure no-one's confused about that. 2) I don't want to have configuration files or command line options in order to use oslo.messaging But, even now, you don't actually have to parse the command line or any config files. See e.g. https://gist.github.com/markmc/7823230 3) Ok, but I want to be able to specify values for things like rpc_conn_pool_size without using any config files. We've talked about allowing the use of query parameters for stuff like this, but I don't love that. I think I'd restrict query parameters to those options which are fundamental to how you connect to a given transport. We could quite easily provide any API which would allow constructing a ConfigOpts object with a bunch of options set and without anyone having to use config files. Here's a PoC of how that might look: https://gist.github.com/markmc/7823420 (Now, if your reaction is OMG, you're using temporary config files on disk, that's awful then just bear with me an ignore the implementation details of get_config_from_dict(). We could very easily make oslo.config support a mode like this without needing to ever write anything to disk) The point of this example is that we could add an oslo.messaging API which takes a dict of config values and you never even know that oslo.config is involved. 4) But I want the API to be explicit about what config options are supported by the API This could be something useful to discuss, because right now the API hides configuration options rather than encoding them into the API. This is to give us a bit more flexibility about changing these over time (e.g. keeping backwards compat for old options for a short time than other aspects of the API). But actually, I'm assuming this isn't what you're thinking since your patch adds an free-form executor_kwargs dict. 5) But I want to avoid any dependency on oslo.config This could be fundamentally what we're talking about here, but I struggle to understand it - oslo.config is pretty tiny and it only requires argparse, so if it's just an implementation detail that you don't even notice if you're not using config files then what exactly is the problem? Basically, my thinking is that something like this example: https://gist.github.com/markmc/7823420 where you can use oslo.messaging with just a dict of config values (rather than having to parse config files) should handle any reasonable concern that I've understood so far ... without having to change much at all. Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
On Fri, Dec 06 2013, Mark McLoughlin wrote: Hi Mark, If the goal is allow applications to use oslo.messaging without using oslo.config, then what's driving this? I'm guessing some possible answers: 5) But I want to avoid any dependency on oslo.config I think that's the more important one to me. This could be fundamentally what we're talking about here, but I struggle to understand it - oslo.config is pretty tiny and it only requires argparse, so if it's just an implementation detail that you don't even notice if you're not using config files then what exactly is the problem? Basically, my thinking is that something like this example: https://gist.github.com/markmc/7823420 where you can use oslo.messaging with just a dict of config values (rather than having to parse config files) should handle any reasonable concern that I've understood so far ... without having to change much at all. I definitely agree with your arguments. There's a large number of technical solutions that can be used to bypass the usage of oslo.config and make it work with whatever you're using.. I just can't stop thinking that a library shouldn't impose any use of a configuration library. I can pick any library on PyPI, and, fortunately, most of them don't come with a dependency on the favorite configuration library of their author or related project, and its usage spread all over the code base. While I do respect the fact that this is a library to be consumed mainly in OpenStack (and I don't want to break that), I think we're also trying to not be the new Zope and contribute in a sane way to the Python ecosystem. And I think oslo.messaging doesn't do that right. Now if the consensus is to leave it that way, I honestly won't fight it over and over. As Mark proved, there's a lot of way to circumvent the oslo.config usage anyway. -- Julien Danjou ;; Free Software hacker ; independent consultant ;; http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
On Fri, 2013-12-06 at 15:41 +0100, Julien Danjou wrote: On Fri, Dec 06 2013, Mark McLoughlin wrote: Hi Mark, If the goal is allow applications to use oslo.messaging without using oslo.config, then what's driving this? I'm guessing some possible answers: 5) But I want to avoid any dependency on oslo.config I think that's the more important one to me. This could be fundamentally what we're talking about here, but I struggle to understand it - oslo.config is pretty tiny and it only requires argparse, so if it's just an implementation detail that you don't even notice if you're not using config files then what exactly is the problem? Basically, my thinking is that something like this example: https://gist.github.com/markmc/7823420 where you can use oslo.messaging with just a dict of config values (rather than having to parse config files) should handle any reasonable concern that I've understood so far ... without having to change much at all. I definitely agree with your arguments. There's a large number of technical solutions that can be used to bypass the usage of oslo.config and make it work with whatever you're using.. I just can't stop thinking that a library shouldn't impose any use of a configuration library. I can pick any library on PyPI, and, fortunately, most of them don't come with a dependency on the favorite configuration library of their author or related project, and its usage spread all over the code base. While I do respect the fact that this is a library to be consumed mainly in OpenStack (and I don't want to break that), I think we're also trying to not be the new Zope and contribute in a sane way to the Python ecosystem. And I think oslo.messaging doesn't do that right. Now if the consensus is to leave it that way, I honestly won't fight it over and over. As Mark proved, there's a lot of way to circumvent the oslo.config usage anyway. Ok, let's say oslo.messaging didn't use oslo.config at all and just took a free-form dict of configuration values. Then you'd have this separation whereby you can write code to retrieve those values from any number of possible configuration sources and pass them down to oslo.messaging. I think that's what you're getting at? However, what you lose with that is a consistent way of defining a schema for those configuration options in oslo.messaging. Should a given option be an int, bool or a list? What should it's default be? etc. etc. That stuff would live in the integration layer that maps from oslo.config to a dict, even though it's totally useful when you just supply a dict. I guess there's two sides to oslo.config - the option schemas and the code to retrieve values from various sources (command line, config files or overrides/defaults). I think the option schemas is a useful implementation detail in oslo.messaging, even if the values don't come from the usual oslo.config sources. Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
I really have to agree with this. It's especially important if oslo.messaging is also used in libraries like taskflow. If oslo.messaging imposes that users of it must use oslo.config then by using it in taskflow, taskflow then imposes the same oslo.config usage. This makes all libraries that use it inherently only useable in the openstack ecosystem which I think is very bad opensource behavior (not exactly open). There are other reasons to, a configuration dict means u can have many different active instances being simultaneously used (each with its own config), with oslo.config since it is a static configuration object u get 1 simultaneous instance. So this is yet another behavior that I as a library provider thing is very unhealthy restriction to impose on people that use taskflow. Sent from my really tiny device... On Dec 6, 2013, at 6:46 AM, Julien Danjou jul...@danjou.info wrote: On Fri, Dec 06 2013, Mark McLoughlin wrote: Hi Mark, If the goal is allow applications to use oslo.messaging without using oslo.config, then what's driving this? I'm guessing some possible answers: 5) But I want to avoid any dependency on oslo.config I think that's the more important one to me. This could be fundamentally what we're talking about here, but I struggle to understand it - oslo.config is pretty tiny and it only requires argparse, so if it's just an implementation detail that you don't even notice if you're not using config files then what exactly is the problem? Basically, my thinking is that something like this example: https://gist.github.com/markmc/7823420 where you can use oslo.messaging with just a dict of config values (rather than having to parse config files) should handle any reasonable concern that I've understood so far ... without having to change much at all. I definitely agree with your arguments. There's a large number of technical solutions that can be used to bypass the usage of oslo.config and make it work with whatever you're using.. I just can't stop thinking that a library shouldn't impose any use of a configuration library. I can pick any library on PyPI, and, fortunately, most of them don't come with a dependency on the favorite configuration library of their author or related project, and its usage spread all over the code base. While I do respect the fact that this is a library to be consumed mainly in OpenStack (and I don't want to break that), I think we're also trying to not be the new Zope and contribute in a sane way to the Python ecosystem. And I think oslo.messaging doesn't do that right. Now if the consensus is to leave it that way, I honestly won't fight it over and over. As Mark proved, there's a lot of way to circumvent the oslo.config usage anyway. -- Julien Danjou ;; Free Software hacker ; independent consultant ;; http://julien.danjou.info ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
Previous not precious, ha, durn autocorrect, lol. Sent from my really tiny device... On Dec 6, 2013, at 9:50 AM, Joshua Harlow harlo...@yahoo-inc.com wrote: Forgive me for not understanding your precious email (which I guess was confusing for me to understand). This one clears that up. If only we all had Vulcan mind meld capabilities, haha. Thanks for helping me understand, no need to get frustrated. Not everyone is able to decipher your email in the same way u wrote it, part of this ML should be about teaching others your viewpoints, not getting frustrated over simple things like misunderstandings... Sent from my really tiny device... On Dec 6, 2013, at 9:16 AM, Mark McLoughlin mar...@redhat.com wrote: On Fri, 2013-12-06 at 16:55 +, Joshua Harlow wrote: I really have to agree with this. It's especially important if oslo.messaging is also used in libraries like taskflow. If oslo.messaging imposes that users of it must use oslo.config then by using it in taskflow, taskflow then imposes the same oslo.config usage. You know, I think you either didn't read my (carefully considered) email or didn't take the time to understand it. That's incredibly frustrating. My proposal would mean that oslo.messaging could be used like this: from oslo import messaging conf = messaging.get_config_from_dict(dict(rpc_conn_pool_size=100)) transport = messaging.get_transport(conf, 'qpid:///test') server = Server(transport) server.start() server.wait() oslo.config is nothing but an implementation detail if you used oslo.messaging in this way. (Julien had a more subtle concern about this which I can actually relate more to) This makes all libraries that use it inherently only useable in the openstack ecosystem which I think is very bad opensource behavior (not exactly open). bad open-source behaviour? Seriously? Yeah, like gtk+ is only usable in the GNOME ecosystem because it uses glib and gtk+ authors are bad open-source people because they didn't allow an alternative to glib to be used. Bizarre statement, frankly. There are other reasons to, a configuration dict means u can have many different active instances being simultaneously used (each with its own config), with oslo.config since it is a static configuration object u get 1 simultaneous instance. So this is yet another behavior that I as a library provider thing is very unhealthy restriction to impose on people that use taskflow. 1 simultaneous instance ... you mean the cfg.CONF object? There's no requirement to use that and I explained that in my email too ... even though I actually thought it should need no explaining at this point. Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
Forgive me for not understanding your precious email (which I guess was confusing for me to understand). This one clears that up. If only we all had Vulcan mind meld capabilities, haha. Thanks for helping me understand, no need to get frustrated. Not everyone is able to decipher your email in the same way u wrote it, part of this ML should be about teaching others your viewpoints, not getting frustrated over simple things like misunderstandings... Sent from my really tiny device... On Dec 6, 2013, at 9:16 AM, Mark McLoughlin mar...@redhat.com wrote: On Fri, 2013-12-06 at 16:55 +, Joshua Harlow wrote: I really have to agree with this. It's especially important if oslo.messaging is also used in libraries like taskflow. If oslo.messaging imposes that users of it must use oslo.config then by using it in taskflow, taskflow then imposes the same oslo.config usage. You know, I think you either didn't read my (carefully considered) email or didn't take the time to understand it. That's incredibly frustrating. My proposal would mean that oslo.messaging could be used like this: from oslo import messaging conf = messaging.get_config_from_dict(dict(rpc_conn_pool_size=100)) transport = messaging.get_transport(conf, 'qpid:///test') server = Server(transport) server.start() server.wait() oslo.config is nothing but an implementation detail if you used oslo.messaging in this way. (Julien had a more subtle concern about this which I can actually relate more to) This makes all libraries that use it inherently only useable in the openstack ecosystem which I think is very bad opensource behavior (not exactly open). bad open-source behaviour? Seriously? Yeah, like gtk+ is only usable in the GNOME ecosystem because it uses glib and gtk+ authors are bad open-source people because they didn't allow an alternative to glib to be used. Bizarre statement, frankly. There are other reasons to, a configuration dict means u can have many different active instances being simultaneously used (each with its own config), with oslo.config since it is a static configuration object u get 1 simultaneous instance. So this is yet another behavior that I as a library provider thing is very unhealthy restriction to impose on people that use taskflow. 1 simultaneous instance ... you mean the cfg.CONF object? There's no requirement to use that and I explained that in my email too ... even though I actually thought it should need no explaining at this point. Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
Could jsonschema[1] be used here to do the options schema part? It works on dictionaries (and really isn't tied to json). But maybe I am missing some greater context/understanding (see other emails). [1] https://pypi.python.org/pypi/jsonschema Sent from my really tiny device... On Dec 6, 2013, at 7:12 AM, Mark McLoughlin mar...@redhat.com wrote: On Fri, 2013-12-06 at 15:41 +0100, Julien Danjou wrote: On Fri, Dec 06 2013, Mark McLoughlin wrote: Hi Mark, If the goal is allow applications to use oslo.messaging without using oslo.config, then what's driving this? I'm guessing some possible answers: 5) But I want to avoid any dependency on oslo.config I think that's the more important one to me. This could be fundamentally what we're talking about here, but I struggle to understand it - oslo.config is pretty tiny and it only requires argparse, so if it's just an implementation detail that you don't even notice if you're not using config files then what exactly is the problem? Basically, my thinking is that something like this example: https://gist.github.com/markmc/7823420 where you can use oslo.messaging with just a dict of config values (rather than having to parse config files) should handle any reasonable concern that I've understood so far ... without having to change much at all. I definitely agree with your arguments. There's a large number of technical solutions that can be used to bypass the usage of oslo.config and make it work with whatever you're using.. I just can't stop thinking that a library shouldn't impose any use of a configuration library. I can pick any library on PyPI, and, fortunately, most of them don't come with a dependency on the favorite configuration library of their author or related project, and its usage spread all over the code base. While I do respect the fact that this is a library to be consumed mainly in OpenStack (and I don't want to break that), I think we're also trying to not be the new Zope and contribute in a sane way to the Python ecosystem. And I think oslo.messaging doesn't do that right. Now if the consensus is to leave it that way, I honestly won't fight it over and over. As Mark proved, there's a lot of way to circumvent the oslo.config usage anyway. Ok, let's say oslo.messaging didn't use oslo.config at all and just took a free-form dict of configuration values. Then you'd have this separation whereby you can write code to retrieve those values from any number of possible configuration sources and pass them down to oslo.messaging. I think that's what you're getting at? However, what you lose with that is a consistent way of defining a schema for those configuration options in oslo.messaging. Should a given option be an int, bool or a list? What should it's default be? etc. etc. That stuff would live in the integration layer that maps from oslo.config to a dict, even though it's totally useful when you just supply a dict. I guess there's two sides to oslo.config - the option schemas and the code to retrieve values from various sources (command line, config files or overrides/defaults). I think the option schemas is a useful implementation detail in oslo.messaging, even if the values don't come from the usual oslo.config sources. Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
On Tue, Dec 03 2013, Joshua Harlow wrote: Another question that¹s come up recently. Python 3.3 support? Will oslo.messaging achieve that? Maybe its a later goal, but it seems like one that is required (and should almost be expected of new libraries imho). Thoughts? Seems to be mainly eventlet that is the blocker for oslo.messaging (so maybe it can be adjusted to work with things other than eventlet to make it python33 compat). I didn't dig into that yet, but I think the only blocker shall be eventlet indeed. However, the eventlet usage in oslo.messaging is smartly decoupled from the rest of the code. It's only used in an executor. That means only this executor module won't be available on Python 3. But there's still the generic one, and any other somebody would add. -- Julien Danjou -- Free Software hacker - independent consultant -- http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
On 04/12/13 10:14 +0100, Julien Danjou wrote: On Tue, Dec 03 2013, Joshua Harlow wrote: Another question that¹s come up recently. Python 3.3 support? Will oslo.messaging achieve that? Maybe its a later goal, but it seems like one that is required (and should almost be expected of new libraries imho). Thoughts? Seems to be mainly eventlet that is the blocker for oslo.messaging (so maybe it can be adjusted to work with things other than eventlet to make it python33 compat). I didn't dig into that yet, but I think the only blocker shall be eventlet indeed. However, the eventlet usage in oslo.messaging is smartly decoupled from the rest of the code. It's only used in an executor. That means only this executor module won't be available on Python 3. But there's still the generic one, and any other somebody would add. Correct. The idea there is be able to support different executors that will make it easier to port oslo.messaging to Py3K without depending on the support of it for eventlet / gevent. I just created this: https://blueprints.launchpad.net/oslo.messaging/+spec/asyncio-executor FF -- @flaper87 Flavio Percoco pgp55WNmmCkAb.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
On Mon, Dec 02 2013, Joshua Harlow wrote: Thanks for writing this up, looking forward to seeing this happen so that oslo.messaging can be used outside of the core openstack projects (and be used in libraries that do not want to force a oslo.cfg model onto users of said libraries). Any idea of a timeline as to when this would be reflected in https://github.com/openstack/oslo.messaging/ (even rough idea is fine). As fast as the code can be written in review. I'll start working on this now. Feel free to subscribe to the blueprint to receive notifications about upcoming patches so you can review. ;-) -- Julien Danjou ;; Free Software hacker ; independent consultant ;; http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
Of course ;-) Another question that¹s come up recently. Python 3.3 support? Will oslo.messaging achieve that? Maybe its a later goal, but it seems like one that is required (and should almost be expected of new libraries imho). Thoughts? Seems to be mainly eventlet that is the blocker for oslo.messaging (so maybe it can be adjusted to work with things other than eventlet to make it python33 compat). On 12/3/13 2:01 AM, Julien Danjou jul...@danjou.info wrote: On Mon, Dec 02 2013, Joshua Harlow wrote: Thanks for writing this up, looking forward to seeing this happen so that oslo.messaging can be used outside of the core openstack projects (and be used in libraries that do not want to force a oslo.cfg model onto users of said libraries). Any idea of a timeline as to when this would be reflected in https://github.com/openstack/oslo.messaging/ (even rough idea is fine). As fast as the code can be written in review. I'll start working on this now. Feel free to subscribe to the blueprint to receive notifications about upcoming patches so you can review. ;-) -- Julien Danjou ;; Free Software hacker ; independent consultant ;; http://julien.danjou.info ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
On Mon, Nov 18 2013, Julien Danjou wrote: https://blueprints.launchpad.net/oslo/+spec/messaging-decouple-cfg So I've gone through the code and started to write a plan on how I'd do things: https://wiki.openstack.org/wiki/Oslo/blueprints/messaging-decouple-cfg I don't think I missed too much, though I didn't run into all tiny details. Please feel free to tell me if I miss anything obvious, otherwise I'll try to start submitting patches, one at a time, to get this into shape step by step. -- Julien Danjou // Free Software hacker / independent consultant // http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
Thanks for writing this up, looking forward to seeing this happen so that oslo.messaging can be used outside of the core openstack projects (and be used in libraries that do not want to force a oslo.cfg model onto users of said libraries). Any idea of a timeline as to when this would be reflected in https://github.com/openstack/oslo.messaging/ (even rough idea is fine). -Josh On 12/2/13 7:45 AM, Julien Danjou jul...@danjou.info wrote: On Mon, Nov 18 2013, Julien Danjou wrote: https://blueprints.launchpad.net/oslo/+spec/messaging-decouple-cfg So I've gone through the code and started to write a plan on how I'd do things: https://wiki.openstack.org/wiki/Oslo/blueprints/messaging-decouple-cfg I don't think I missed too much, though I didn't run into all tiny details. Please feel free to tell me if I miss anything obvious, otherwise I'll try to start submitting patches, one at a time, to get this into shape step by step. -- Julien Danjou // Free Software hacker / independent consultant // http://julien.danjou.info ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Oslo] Layering olso.messaging usage of config
Hi Oslo developers, It seems my latest patch¹ on oslo.messaging scared Mark, so I'll try to discuss it a bit on this mailing list as it is more convenient. I've created a blueprint² as requested by Mark. That seems necessary since it will be spread on several patch. Now to the core. oslo.messaging API mixes usage of parameters and configuration file object. Such as you have to provide a configuration object for basic API usage, even if you don't have a configuration object. It seems to me that having this separation of concerns in oslo.messaging would be good idea. My plan is to move out the configuration object out of the basic object, like I did in the first patch. I don't plan to break the configuration handling or so, I just think it should be handled in a separate, individually testable part of the code. Ultimately, this would allow oslo.messaging to not be 'oslo' only, and just being friendly with oslo.config and therefore OpenStack. A goal I wish we'd had in more oslo library. :) If you think this is a bad idea, well, feel free to mark the blueprint as so and I'll drop the work. ¹ https://review.openstack.org/56068 ² https://blueprints.launchpad.net/ceilometer/+spec/messaging-decouple-cfg -- Julien Danjou ;; Free Software hacker ; independent consultant ;; http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
On 11/18/2013 05:05 AM, Julien Danjou wrote: Hi Oslo developers, It seems my latest patch¹ on oslo.messaging scared Mark, so I'll try to discuss it a bit on this mailing list as it is more convenient. I've created a blueprint² as requested by Mark. That seems necessary since it will be spread on several patch. Now to the core. oslo.messaging API mixes usage of parameters and configuration file object. Such as you have to provide a configuration object for basic API usage, even if you don't have a configuration object. It seems to me that having this separation of concerns in oslo.messaging would be good idea. My plan is to move out the configuration object out of the basic object, like I did in the first patch. I don't plan to break the configuration handling or so, I just think it should be handled in a separate, individually testable part of the code. Ultimately, this would allow oslo.messaging to not be 'oslo' only, and just being friendly with oslo.config and therefore OpenStack. A goal I wish we'd had in more oslo library. :) If you think this is a bad idea, well, feel free to mark the blueprint as so and I'll drop the work. I think it's a good idea to have as few dependencies between these libraries as possible. It encourages re-use and adoption of the individual libraries and doesn't force developers to use a particular library if they don't like the code or design or whatever in the dependent libs. Less coupling == win. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
Hi, On Mon, Nov 18, 2013 at 11:05:20AM +0100, Julien Danjou wrote: Hi Oslo developers, It seems my latest patch¹ on oslo.messaging scared Mark, so I'll try to discuss it a bit on this mailing list as it is more convenient. I've created a blueprint² as requested by Mark. That seems necessary since it will be spread on several patch. Now to the core. oslo.messaging API mixes usage of parameters and configuration file object. Such as you have to provide a configuration object for basic API usage, even if you don't have a configuration object. It seems to me that having this separation of concerns in oslo.messaging would be good idea. My plan is to move out the configuration object out of the basic object, like I did in the first patch. I don't plan to break the configuration handling or so, I just think it should be handled in a separate, individually testable part of the code. Ultimately, this would allow oslo.messaging to not be 'oslo' only, and just being friendly with oslo.config and therefore OpenStack. A goal I wish we'd had in more oslo library. :) I like the idea of separate the oslo.config stuffs. I have started to replace oslo.rpc by oslo.messaging in ceilometer. And I have two cases that this idea will helped. * Currently oslo.messaging register opts when a object that needs the opt is created, this have too side effets: - we cannot use import_opt to get this option outside oslo.messaging, - the tools generate_samples seems not catch this opts anymore. This force me to register all rpc needed opts in ceilometer and if oslo.messaging change one of them, ceilometer will be not broke automatically, like it does currently. * A more interesting one, this split will permits to easly create a piece of code that can build transport url from the legacy opt configuration (ie: rpc_backend) instead of rewritting the compatibility layer in each project. Regards, -- Mehdi Abaakouk mail: sil...@sileht.net irc: sileht signature.asc Description: Digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
It seems to me that having this separation of concerns in oslo.messaging would be good idea. My plan is to move out the configuration object out of the basic object, like I did in the first patch. I don't plan to break the configuration handling or so, I just think it should be handled in a separate, individually testable part of the code. Ultimately, this would allow oslo.messaging to not be 'oslo' only, and just being friendly with oslo.config and therefore OpenStack. A goal I wish we'd had in more oslo library. :) I'd like to see more decoupling here with oslo.config in a similar way that eventlet may be used, but is no longer necessary, when using oslo.messaging. That is to say, I agree with your premise, even if the patch and blueprint need work. I agree with Mark that a clear blueprint and plan should be outlined. I'm not happy with the blueprint as written, it is sparse, outlining your intention, rather than a plan of attack. I'd really like to know what the plan of attack here is, how this will affect the oslo.messaging API, what it will look like once complete. It shouldn't break the public API and if for some reason you don't think it is possible to keep that contract, it should be discussed. Personally, I'd be happy with a wiki page outlining these changes, linked to from the blueprint. Also, I've noticed that your blueprint is registered underneath Ceilometer. Once you move or recreate the blueprint, please email the list with the updated URL. -- Regards, Eric Windisch ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
On Mon, Nov 18 2013, Eric Windisch wrote: Hi Eric, I agree with Mark that a clear blueprint and plan should be outlined. I'm not happy with the blueprint as written, it is sparse, outlining your intention, rather than a plan of attack. I'd really like to know what the plan of attack here is, how this will affect the oslo.messaging API, what it will look like once complete. It shouldn't break the public API and if for some reason you don't think it is possible to keep that contract, it should be discussed. I think that some part of the API where a configuration object is expected will have to be changed to have proper keywords argument instead. It'll likely slightly break the current API though it should be easy to add a compatibility layer. Personally, I'd be happy with a wiki page outlining these changes, linked to from the blueprint. Sure, I'll try to describe that. Also, I've noticed that your blueprint is registered underneath Ceilometer. Once you move or recreate the blueprint, please email the list with the updated URL. Facepalm. Probably clicked in the wrong window. Recreated in the right project: https://blueprints.launchpad.net/oslo/+spec/messaging-decouple-cfg -- Julien Danjou # Free Software hacker # independent consultant # http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
On Mon, Nov 18, 2013 at 5:05 AM, Julien Danjou jul...@danjou.info wrote: Hi Oslo developers, It seems my latest patch¹ on oslo.messaging scared Mark, so I'll try to discuss it a bit on this mailing list as it is more convenient. I've created a blueprint² as requested by Mark. That seems necessary since it will be spread on several patch. Now to the core. oslo.messaging API mixes usage of parameters and configuration file object. Such as you have to provide a configuration object for basic API usage, even if you don't have a configuration object. IIRC, one of the concerns when oslo.messaging was split out was maintaining support for existing deployments with configuration files that worked with oslo.rpc. We had said that we would use URL query parameters for optional configuration values (with the required values going into other places in the URL), but that format wouldn't be backwards compatible if a deployment already had the other configuration settings listed individually. One way to address that difference is to make the use of oslo.config an explicit step, before instantiating a Transport using a URL. Given a URL for a transport and an oslo.config object, an application could ask oslo.messaging to build a new URL containing all of the settings from the config that apply. Something like: from oslo.config import cfg from oslo.messaging import config as om_config from oslo.messaging import transport base_url = cfg.CONF.rpc_url full_url = om_config.update_url_from_config(base_url, cfg.CONF) trans = transport.get_transport(full_url) update_url_from_config() would use the base_url to determine the driver, load the option definitions for that driver, then pull the values out of the config object and construct a new URL starting with the base and including the extra settings (thus preserving backwards-compatibility for the existing config files). Applications that do not use oslo.config would just call get_transport(). As an alternative, and to preserve API compatibility with the existing release of oslo.messaging, we could just change get_transport() to allow the first argument to be None (instead of removing it entirely). When it is a valid config object, we would go through the steps we do now to get configuration options. When it is None, we would skip those steps. Doug It seems to me that having this separation of concerns in oslo.messaging would be good idea. My plan is to move out the configuration object out of the basic object, like I did in the first patch. I don't plan to break the configuration handling or so, I just think it should be handled in a separate, individually testable part of the code. Ultimately, this would allow oslo.messaging to not be 'oslo' only, and just being friendly with oslo.config and therefore OpenStack. A goal I wish we'd had in more oslo library. :) If you think this is a bad idea, well, feel free to mark the blueprint as so and I'll drop the work. ¹ https://review.openstack.org/56068 ² https://blueprints.launchpad.net/ceilometer/+spec/messaging-decouple-cfg -- Julien Danjou ;; Free Software hacker ; independent consultant ;; http://julien.danjou.info ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
Hi Julien, On Mon, 2013-11-18 at 11:05 +0100, Julien Danjou wrote: Hi Oslo developers, It seems my latest patch¹ on oslo.messaging scared Mark, so I'll try to discuss it a bit on this mailing list as it is more convenient. Scared, heh :) I've created a blueprint² as requested by Mark. Ok, so it's a ceilometer blueprint and says: The goal of this blueprint is to be able to use oslo.messaging without using a configuration file/object, while keeping its usage possible and not breaking compatibility with OpenStack applications. Why is that important to ceilometer? Ceilometer heavily uses the RPC code already and uses the config object. Is this about allowing Ceilometer to consume from multiple brokers? Where will the configuration be stored for each broker connection? In the ceilometer database? Why won't transport URLs be sufficient for that use case given that we know it works fine for Nova cells? I'm struggling to care about this until I have some insight into why it's important. And it's a bit frustrating to have to guess the rationale for this. Like commit messages, blueprints should be as much about the why as the what. That seems necessary since it will be spread on several patch. It's not about multiple patches, it's about something which needs to be designed and thought through in advance. Now to the core. oslo.messaging API mixes usage of parameters and configuration file object. Such as you have to provide a configuration object for basic API usage, even if you don't have a configuration object. It seems to me that having this separation of concerns in oslo.messaging would be good idea. My plan is to move out the configuration object out of the basic object, like I did in the first patch. I don't plan to break the configuration handling or so, I just think it should be handled in a separate, individually testable part of the code. As I said in the review, I'm totally fine with the idea of allowing oslo.messaging to be used without a configuration object ... but I think the common use case is to use it with a configuration object and I don't want to undermine the usability of the library in the common use case. One way of approach this would be to describe how the new API would look like from the perspective of Nova[1] - i.e. if you are using a config object, what does the API look like? The other thing I want to figure out is how we expose the configuration options in the API in a way that allows us to maintain API compatibility as we move and rename the config options. That's doable, but needs some thought. Thanks, Mark. [1] - https://review.openstack.org/39929 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
On Mon, Nov 18 2013, Mark McLoughlin wrote: Ok, so it's a ceilometer blueprint and says: The goal of this blueprint is to be able to use oslo.messaging without using a configuration file/object, while keeping its usage possible and not breaking compatibility with OpenStack applications. Why is that important to ceilometer? Ceilometer heavily uses the RPC code already and uses the config object. It's a mistake, it should be under Oslo as Eric pointed out. I'm struggling to care about this until I have some insight into why it's important. And it's a bit frustrating to have to guess the rationale for this. Like commit messages, blueprints should be as much about the why as the what. Sure. I ought to think that having an application that wants to leverage oslo.messaging but is not using oslo.config and is retrieving its parameter from another way is a good enough argument. As I said in the review, I'm totally fine with the idea of allowing oslo.messaging to be used without a configuration object ... but I think the common use case is to use it with a configuration object and I don't want to undermine the usability of the library in the common use case. Understood. I know it's already a pain to transition from RPC to messaging, and I don't want to add more burden on that transition. -- Julien Danjou -- Free Software hacker - independent consultant -- http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
On Mon, 2013-11-18 at 17:37 +0100, Julien Danjou wrote: On Mon, Nov 18 2013, Mark McLoughlin wrote: I'm struggling to care about this until I have some insight into why it's important. And it's a bit frustrating to have to guess the rationale for this. Like commit messages, blueprints should be as much about the why as the what. Sure. I ought to think that having an application that wants to leverage oslo.messaging but is not using oslo.config and is retrieving its parameter from another way is a good enough argument. It's a theoretical benefit versus the very practical design an API for the use cases that are actually important to OpenStack projects. As I said in the review, I'm totally fine with the idea of allowing oslo.messaging to be used without a configuration object ... but I think the common use case is to use it with a configuration object and I don't want to undermine the usability of the library in the common use case. Understood. I know it's already a pain to transition from RPC to messaging, and I don't want to add more burden on that transition. Cool, thanks. Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
Hey Doug, On Mon, 2013-11-18 at 11:29 -0500, Doug Hellmann wrote: On Mon, Nov 18, 2013 at 5:05 AM, Julien Danjou jul...@danjou.info wrote: Hi Oslo developers, It seems my latest patch¹ on oslo.messaging scared Mark, so I'll try to discuss it a bit on this mailing list as it is more convenient. I've created a blueprint² as requested by Mark. That seems necessary since it will be spread on several patch. Now to the core. oslo.messaging API mixes usage of parameters and configuration file object. Such as you have to provide a configuration object for basic API usage, even if you don't have a configuration object. IIRC, one of the concerns when oslo.messaging was split out was maintaining support for existing deployments with configuration files that worked with oslo.rpc. We had said that we would use URL query parameters for optional configuration values (with the required values going into other places in the URL), but that format wouldn't be backwards compatible if a deployment already had the other configuration settings listed individually. I hadn't ever considered exposing all configuration options via the URL. We have a lot of fairly random options, that I don't think you need to configure per-connection if you have multiple connections in the one application. One way to address that difference is to make the use of oslo.config an explicit step, before instantiating a Transport using a URL. Given a URL for a transport and an oslo.config object, an application could ask oslo.messaging to build a new URL containing all of the settings from the config that apply. Something like: from oslo.config import cfg from oslo.messaging import config as om_config from oslo.messaging import transport base_url = cfg.CONF.rpc_url full_url = om_config.update_url_from_config(base_url, cfg.CONF) trans = transport.get_transport(full_url) I don't think anyone would thanks us for the horrific URLs that would result from this. But if we did this, I'd just make it: from oslo.config import cfg from oslo.messaging import config as om_config transport = om_config.get_transport_from_config(cfg.CONF) update_url_from_config() would use the base_url to determine the driver, load the option definitions for that driver, then pull the values out of the config object and construct a new URL starting with the base and including the extra settings (thus preserving backwards-compatibility for the existing config files). Applications that do not use oslo.config would just call get_transport(). As an alternative, and to preserve API compatibility with the existing release of oslo.messaging, we could just change get_transport() to allow the first argument to be None (instead of removing it entirely). When it is a valid config object, we would go through the steps we do now to get configuration options. When it is None, we would skip those steps. API compatibility isn't yet important. We haven't done a first release yet and no projects are using the library. Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Layering olso.messaging usage of config
I see it as important (and not theoretical). I'd like to use oslo.messaging (and oslo.messaging.rpc) in taskflow (to prototype how a distributed engine would work using it) and bringing in a library which requires global configuration is almost a non-starter for me. Although taskflow is targeted at being used in openstack it still is a generic library and a library shouldn't bring in dependencies which are only configurable in 1 manner (via a global config via oslo.cfg); to me that¹s bad style forces an ideology on users of taskflow (which limits adoption). So maybe this is a use-case that can be the rationale for oslo.messaging? On 11/18/13 8:46 AM, Mark McLoughlin mar...@redhat.com wrote: On Mon, 2013-11-18 at 17:37 +0100, Julien Danjou wrote: On Mon, Nov 18 2013, Mark McLoughlin wrote: I'm struggling to care about this until I have some insight into why it's important. And it's a bit frustrating to have to guess the rationale for this. Like commit messages, blueprints should be as much about the why as the what. Sure. I ought to think that having an application that wants to leverage oslo.messaging but is not using oslo.config and is retrieving its parameter from another way is a good enough argument. It's a theoretical benefit versus the very practical design an API for the use cases that are actually important to OpenStack projects. As I said in the review, I'm totally fine with the idea of allowing oslo.messaging to be used without a configuration object ... but I think the common use case is to use it with a configuration object and I don't want to undermine the usability of the library in the common use case. Understood. I know it's already a pain to transition from RPC to messaging, and I don't want to add more burden on that transition. Cool, thanks. Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
