Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins
Reviewing this thread to come to a conclusion (for myself at least - and hopefully so I can document something so reviewers know why I did it) For approach: 1. plugins should use stevedore with entry points (as stated by Russell) 2. the plugins should be explicitly selected through configuration For api stability: I'm not sure there was a consensus. Personally I would write a base class for the plugins and document in it that the interface is unstable. Sound good? BTW: this is one of those things that could be put in a place to make and record decisions (like the gerrit idea for blueprints). But now I am referring to another thread [http://lists.openstack.org/pipermail/openstack-dev/2014-March/029232.html ] Paul. -Original Message- From: Sandy Walsh [mailto:sandy.wa...@rackspace.com] Sent: 04 March 2014 21:25 To: Murray, Paul (HP Cloud Services) Cc: OpenStack Development Mailing List (not for usage questions); d...@danplanet.com Subject: Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins And sorry, as to your original problem, the loadables approach is kinda messy since only the classes that are loaded when *that* module are loaded are used (vs. explicitly specifying them in a config). You may get different results when the flow changes. Either entry-points or config would give reliable results. On 03/04/2014 03:21 PM, Murray, Paul (HP Cloud Services) wrote: In a chat with Dan Smith on IRC, he was suggesting that the important thing was not to use class paths in the config file. I can see that internal implementation should not be exposed in the config files - that way the implementation can change without impacting the nova users/operators. There's plenty of easy ways to deal with that problem vs. entry points. MyModule.get_my_plugin() ... which can point to anywhere in the module permanently. Also, we don't have any of the headaches of merging setup.cfg sections (as we see with oslo.* integration). Sandy, I'm not sure I really get the security argument. Python provides every means possible to inject code, not sure plugins are so different. Certainly agree on choosing which plugins you want to use though. The concern is that any compromised part of the python eco-system can get auto-loaded with the entry-point mechanism. Let's say Nova auto-loads all modules with entry-points the [foo] section. All I have to do is create a setup that has a [foo] section and my code is loaded. Explicit is better than implicit. So, assuming we don't auto-load modules ... what does the entry-point approach buy us? From: Russell Bryant [rbry...@redhat.com] We should be careful though. We need to limit what we expose as external plug points, even if we consider them unstable. If we don't want it to be public, it may not make sense for it to be a plugin interface at all. I'm not sure what the concern with introducing new extension points is? OpenStack is basically just a big bag of plugins. If it's optional, it's supposed to be a plugin (according to the design tenets). -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins
On Wed, 12 Mar 2014 14:31:13 + Murray, Paul (HP Cloud Services) pmur...@hp.com wrote: Reviewing this thread to come to a conclusion (for myself at least - and hopefully so I can document something so reviewers know why I did it) For approach: 1. plugins should use stevedore with entry points (as stated by Russell) 2. the plugins should be explicitly selected through configuration For api stability: I'm not sure there was a consensus. Personally I would write a base class for the plugins and document in it that the interface is unstable. Sound good? Even if you don't want to make any guarantees around API stability I'd suggest still putting some versioning info in at the start. So at least the various parts can detect and warn when they might be broken. Chris BTW: this is one of those things that could be put in a place to make and record decisions (like the gerrit idea for blueprints). But now I am referring to another thread [http://lists.openstack.org/pipermail/openstack-dev/2014-March/029232.html ] Paul. -Original Message- From: Sandy Walsh [mailto:sandy.wa...@rackspace.com] Sent: 04 March 2014 21:25 To: Murray, Paul (HP Cloud Services) Cc: OpenStack Development Mailing List (not for usage questions); d...@danplanet.com Subject: Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins And sorry, as to your original problem, the loadables approach is kinda messy since only the classes that are loaded when *that* module are loaded are used (vs. explicitly specifying them in a config). You may get different results when the flow changes. Either entry-points or config would give reliable results. On 03/04/2014 03:21 PM, Murray, Paul (HP Cloud Services) wrote: In a chat with Dan Smith on IRC, he was suggesting that the important thing was not to use class paths in the config file. I can see that internal implementation should not be exposed in the config files - that way the implementation can change without impacting the nova users/operators. There's plenty of easy ways to deal with that problem vs. entry points. MyModule.get_my_plugin() ... which can point to anywhere in the module permanently. Also, we don't have any of the headaches of merging setup.cfg sections (as we see with oslo.* integration). Sandy, I'm not sure I really get the security argument. Python provides every means possible to inject code, not sure plugins are so different. Certainly agree on choosing which plugins you want to use though. The concern is that any compromised part of the python eco-system can get auto-loaded with the entry-point mechanism. Let's say Nova auto-loads all modules with entry-points the [foo] section. All I have to do is create a setup that has a [foo] section and my code is loaded. Explicit is better than implicit. So, assuming we don't auto-load modules ... what does the entry-point approach buy us? From: Russell Bryant [rbry...@redhat.com] We should be careful though. We need to limit what we expose as external plug points, even if we consider them unstable. If we don't want it to be public, it may not make sense for it to be a plugin interface at all. I'm not sure what the concern with introducing new extension points is? OpenStack is basically just a big bag of plugins. If it's optional, it's supposed to be a plugin (according to the design tenets). -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins
On 03/04/2014 06:27 AM, Murray, Paul (HP Cloud Services) wrote: One of my patches has a query asking if I am using the agreed way to load plugins: https://review.openstack.org/#/c/71557/ I followed the same approach as filters/weights/metrics using nova.loadables. Was there an agreement to do it a different way? And if so, what is the agreed way of doing it? A pointer to an example or even documentation/wiki page would be appreciated. The short version is entry-point based plugins using stevedore. We should be careful though. We need to limit what we expose as external plug points, even if we consider them unstable. If we don't want it to be public, it may not make sense for it to be a plugin interface at all. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins
This brings up something that's been gnawing at me for a while now ... why use entry-point based loaders at all? I don't see the problem they're trying to solve. (I thought I got it for a while, but I was clearly fooling myself) 1. If you use the load all drivers in this category feature, that's a security risk since any compromised python library could hold a trojan. 2. otherwise you have to explicitly name the plugins you want (or don't want) anyway, so why have the extra indirection of the entry-point? Why not just name the desired modules directly? 3. the real value of a loader would be to also extend/manage the python path ... that's where the deployment pain is. Use fully qualified filename driver and take care of the pathing for me. Abstracting the module/class/function name isn't a great win. I don't see where the value is for the added pain (entry-point management/package metadata) it brings. CMV, -S From: Russell Bryant [rbry...@redhat.com] Sent: Tuesday, March 04, 2014 1:29 PM To: Murray, Paul (HP Cloud Services); OpenStack Development Mailing List Subject: Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins On 03/04/2014 06:27 AM, Murray, Paul (HP Cloud Services) wrote: One of my patches has a query asking if I am using the agreed way to load plugins: https://review.openstack.org/#/c/71557/ I followed the same approach as filters/weights/metrics using nova.loadables. Was there an agreement to do it a different way? And if so, what is the agreed way of doing it? A pointer to an example or even documentation/wiki page would be appreciated. The short version is entry-point based plugins using stevedore. We should be careful though. We need to limit what we expose as external plug points, even if we consider them unstable. If we don't want it to be public, it may not make sense for it to be a plugin interface at all. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins
In a chat with Dan Smith on IRC, he was suggesting that the important thing was not to use class paths in the config file. I can see that internal implementation should not be exposed in the config files - that way the implementation can change without impacting the nova users/operators. Sandy, I'm not sure I really get the security argument. Python provides every means possible to inject code, not sure plugins are so different. Certainly agree on choosing which plugins you want to use though. -Original Message- From: Sandy Walsh [mailto:sandy.wa...@rackspace.com] Sent: 04 March 2014 17:50 To: OpenStack Development Mailing List (not for usage questions); Murray, Paul (HP Cloud Services) Subject: RE: [openstack-dev] [Nova] What is the currently accepted way to do plugins This brings up something that's been gnawing at me for a while now ... why use entry-point based loaders at all? I don't see the problem they're trying to solve. (I thought I got it for a while, but I was clearly fooling myself) 1. If you use the load all drivers in this category feature, that's a security risk since any compromised python library could hold a trojan. 2. otherwise you have to explicitly name the plugins you want (or don't want) anyway, so why have the extra indirection of the entry-point? Why not just name the desired modules directly? 3. the real value of a loader would be to also extend/manage the python path ... that's where the deployment pain is. Use fully qualified filename driver and take care of the pathing for me. Abstracting the module/class/function name isn't a great win. I don't see where the value is for the added pain (entry-point management/package metadata) it brings. CMV, -S From: Russell Bryant [rbry...@redhat.com] Sent: Tuesday, March 04, 2014 1:29 PM To: Murray, Paul (HP Cloud Services); OpenStack Development Mailing List Subject: Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins On 03/04/2014 06:27 AM, Murray, Paul (HP Cloud Services) wrote: One of my patches has a query asking if I am using the agreed way to load plugins: https://review.openstack.org/#/c/71557/ I followed the same approach as filters/weights/metrics using nova.loadables. Was there an agreement to do it a different way? And if so, what is the agreed way of doing it? A pointer to an example or even documentation/wiki page would be appreciated. The short version is entry-point based plugins using stevedore. We should be careful though. We need to limit what we expose as external plug points, even if we consider them unstable. If we don't want it to be public, it may not make sense for it to be a plugin interface at all. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins
In a chat with Dan Smith on IRC, he was suggesting that the important thing was not to use class paths in the config file. I can see that internal implementation should not be exposed in the config files - that way the implementation can change without impacting the nova users/operators. Sandy, I'm not sure I really get the security argument. Python provides every means possible to inject code, not sure plugins are so different. Certainly agree on choosing which plugins you want to use though. Yeah, so I don't think there's any security reason why one is better than the other. I think that we've decided that providing a class path is ugly, and I agree, especially if we have entry points at our disposal. Now, the actual concern is not related to any of that, but about whether we're going to open this up as a new thing we support. In general, my reaction to adding new APIs people expect to be stable is no. However, I understand why things like the resource reporting and even my events mechanism are very useful for deployers to do some plumbing and monitoring of their environment -- things that don't belong upstream anyway. So I'm conflicted. I think that for these two cases, as long as we can say that it's not a stable interface, I think it's probably okay. However, things like we've had in the past, where we provide a clear plug point for something like Compute manager API class are clearly off the table, IMHO. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins
On Tue, 2014-03-04 at 12:11 -0800, Dan Smith wrote: Now, the actual concern is not related to any of that, but about whether we're going to open this up as a new thing we support. In general, my reaction to adding new APIs people expect to be stable is no. However, I understand why things like the resource reporting and even my events mechanism are very useful for deployers to do some plumbing and monitoring of their environment -- things that don't belong upstream anyway. So I'm conflicted. I think that for these two cases, as long as we can say that it's not a stable interface, I think it's probably okay. However, things like we've had in the past, where we provide a clear plug point for something like Compute manager API class are clearly off the table, IMHO. How about using 'unstable' as a component of the entrypoint group? E.g., nova.unstable.events… -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins
How about using 'unstable' as a component of the entrypoint group? E.g., nova.unstable.events… Well, this is a pretty heavy way to ensure that the admin gets the picture, but maybe appropriate :) What I don't think we want is the in-tree plugins having to hook into something called unstable. But, if we handle those one way and then clearly find and loop through any unstable $things then maybe that's a happy medium. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins
And sorry, as to your original problem, the loadables approach is kinda messy since only the classes that are loaded when *that* module are loaded are used (vs. explicitly specifying them in a config). You may get different results when the flow changes. Either entry-points or config would give reliable results. On 03/04/2014 03:21 PM, Murray, Paul (HP Cloud Services) wrote: In a chat with Dan Smith on IRC, he was suggesting that the important thing was not to use class paths in the config file. I can see that internal implementation should not be exposed in the config files - that way the implementation can change without impacting the nova users/operators. There's plenty of easy ways to deal with that problem vs. entry points. MyModule.get_my_plugin() ... which can point to anywhere in the module permanently. Also, we don't have any of the headaches of merging setup.cfg sections (as we see with oslo.* integration). Sandy, I'm not sure I really get the security argument. Python provides every means possible to inject code, not sure plugins are so different. Certainly agree on choosing which plugins you want to use though. The concern is that any compromised part of the python eco-system can get auto-loaded with the entry-point mechanism. Let's say Nova auto-loads all modules with entry-points the [foo] section. All I have to do is create a setup that has a [foo] section and my code is loaded. Explicit is better than implicit. So, assuming we don't auto-load modules ... what does the entry-point approach buy us? From: Russell Bryant [rbry...@redhat.com] We should be careful though. We need to limit what we expose as external plug points, even if we consider them unstable. If we don't want it to be public, it may not make sense for it to be a plugin interface at all. I'm not sure what the concern with introducing new extension points is? OpenStack is basically just a big bag of plugins. If it's optional, it's supposed to be a plugin (according to the design tenets). -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins
On 03/04/2014 05:00 PM, Kevin L. Mitchell wrote: On Tue, 2014-03-04 at 12:11 -0800, Dan Smith wrote: Now, the actual concern is not related to any of that, but about whether we're going to open this up as a new thing we support. In general, my reaction to adding new APIs people expect to be stable is no. However, I understand why things like the resource reporting and even my events mechanism are very useful for deployers to do some plumbing and monitoring of their environment -- things that don't belong upstream anyway. So I'm conflicted. I think that for these two cases, as long as we can say that it's not a stable interface, I think it's probably okay. However, things like we've had in the past, where we provide a clear plug point for something like Compute manager API class are clearly off the table, IMHO. How about using 'unstable' as a component of the entrypoint group? E.g., nova.unstable.events… Wouldn't that defeat the point of entry points ... immutable endpoints? What happens when an unstable event is deemed stable? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] What is the currently accepted way to do plugins
On Tue, 2014-03-04 at 17:31 -0400, Sandy Walsh wrote: How about using 'unstable' as a component of the entrypoint group? E.g., nova.unstable.events… Wouldn't that defeat the point of entry points ... immutable endpoints? What happens when an unstable event is deemed stable? Actually, the idea here is that the API that those entrypoints are expected to express is unstable; when the API stabilizes, you'd remove the unstable part, then *never* change the API again. This is as opposed to thinking that the entrypoint itself is unstable—nova shouldn't care, it should just use it… -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
