Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt
From: Doug Hellmann d...@doughellmann.com To: openstack-dev openstack-dev@lists.openstack.org Date: 07/28/2015 07:59 PM Subject: Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt snip I replied on both patches, but I'll repeat it here for a broader audience: Please set up an extras entry for each backend instead of just removing the dependencies. That will signal to users that you know what dependencies there are for a backend, but that they are optional, and still allow someone to do the equivalent of pip install glance[vmware] or pip install glance[swift] to get those dependencies. Nova and oslo.versionedobjects have examples in their setup.cfg if you need a template. I didn't mention in the reviews, but this will also make integration tests in our gate easier, since you can put .[vmware] or .[swift] in the tox.ini to pull in those dependencies. Doug snip I did this... but now I'm thinking I really shouldn't have. That makes perfect sense for glance_store, but shouldn't that only be in glance_store, and not also in glance? Matthew__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt
On 30 July 2015 at 02:48, Doug Hellmann d...@doughellmann.com wrote: Excerpts from Louis Taylor's message of 2015-07-29 15:25:33 +0100: On Tue, Jul 28, 2015 at 07:55:54PM -0400, Doug Hellmann wrote: I replied on both patches, but I'll repeat it here for a broader audience: Please set up an extras entry for each backend instead of just removing the dependencies. That will signal to users that you know what dependencies there are for a backend, but that they are optional, and still allow someone to do the equivalent of pip install glance[vmware] or pip install glance[swift] to get those dependencies. Nova and oslo.versionedobjects have examples in their setup.cfg if you need a template. I didn't mention in the reviews, but this will also make integration tests in our gate easier, since you can put .[vmware] or .[swift] in the tox.ini to pull in those dependencies. This sounds like the best option. What happens if you want to enable multiple backends? Can you do something like pip install glance[swift, vmware], or would you need to run a separate pip command to install each? I think you would say: pip install glance[swift] glance[vmware] That might work. pip install glance[swift,vmware] is the canonical form. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt
On 30 July 2015 at 09:38, Ian Cordasco ian.corda...@rackspace.com wrote: On 7/29/15, 13:27, William M Edmonds edmon...@us.ibm.com wrote: From: Doug Hellmann d...@doughellmann.com To: openstack-dev openstack-dev@lists.openstack.org Date: 07/28/2015 07:59 PM Subject: Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt snip I replied on both patches, but I'll repeat it here for a broader audience: Please set up an extras entry for each backend instead of just removing the dependencies. That will signal to users that you know what dependencies there are for a backend, but that they are optional, and still allow someone to do the equivalent of pip install glance[vmware] or pip install glance[swift] to get those dependencies. Nova and oslo.versionedobjects have examples in their setup.cfg if you need a template. I didn't mention in the reviews, but this will also make integration tests in our gate easier, since you can put .[vmware] or .[swift] in the tox.ini to pull in those dependencies. Doug snip I did this... but now I'm thinking I really shouldn't have. That makes perfect sense for glance_store, but shouldn't that only be in glance_store, and not also in glance? No, I think it belongs in glance too frankly. What most people care about is that Glance works with VMWare and Swift, not that they install glance and some library that interacts with those. For most people their concept of how glance works is that it talks to VMWare, Swift, etc. It makes sense, then (to not cause too much confusion) for Glance to set-up extras that merely then define glance_store[swift] or what have you. I'm willing to be convinced otherwise. Also, I haven't tested how something like $ pip install glance[swift,vmware] Would behave given we'll need glance_store in our requires no matter what, and then with the extras it'll be installing $ glance_store[swift] glance_store[vmware] I'll have to test this a bit. It may make more sense to simply have these be Glance dependencies since we're the only consumer of glance_store (for now). If pip doesn't merge the transitive extras then I know exactly where to fix it so it does, and we've multiple folk working fulltime around this space who can make it happen. Please do test: pip install glance[swift,vmware] should resolve to glance_store[swift,vmware] sanely, and that will bring in all the child deps happily. AFAIAA it all works now - and if it doesn't fixing it will be interrupt priority. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt
On 7/29/15, 13:27, William M Edmonds edmon...@us.ibm.com wrote: From: Doug Hellmann d...@doughellmann.com To: openstack-dev openstack-dev@lists.openstack.org Date: 07/28/2015 07:59 PM Subject: Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt snip I replied on both patches, but I'll repeat it here for a broader audience: Please set up an extras entry for each backend instead of just removing the dependencies. That will signal to users that you know what dependencies there are for a backend, but that they are optional, and still allow someone to do the equivalent of pip install glance[vmware] or pip install glance[swift] to get those dependencies. Nova and oslo.versionedobjects have examples in their setup.cfg if you need a template. I didn't mention in the reviews, but this will also make integration tests in our gate easier, since you can put .[vmware] or .[swift] in the tox.ini to pull in those dependencies. Doug snip I did this... but now I'm thinking I really shouldn't have. That makes perfect sense for glance_store, but shouldn't that only be in glance_store, and not also in glance? No, I think it belongs in glance too frankly. What most people care about is that Glance works with VMWare and Swift, not that they install glance and some library that interacts with those. For most people their concept of how glance works is that it talks to VMWare, Swift, etc. It makes sense, then (to not cause too much confusion) for Glance to set-up extras that merely then define glance_store[swift] or what have you. I'm willing to be convinced otherwise. Also, I haven't tested how something like $ pip install glance[swift,vmware] Would behave given we'll need glance_store in our requires no matter what, and then with the extras it'll be installing $ glance_store[swift] glance_store[vmware] I'll have to test this a bit. It may make more sense to simply have these be Glance dependencies since we're the only consumer of glance_store (for now). __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt
So helpful! Thank you. On Wed, Jul 29, 2015 at 7:48 AM, Doug Hellmann d...@doughellmann.com wrote: There is some documentation in the pbr manual (http://docs.openstack.org/developer/pbr/#extra-requirements). The feature is implemented throughout the packaging tool chain now. Ah, excellent! PEP 0426 seemed keen on standardizing, but I'm not seeing any recent movement and setuptools support [2] seems to indicate the ecosystem can move forward without it? 1. https://www.python.org/dev/peps/pep-0426/#extras-optional-dependencies 2. http://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies Here one would say pip install foo[testing] and on Python 2.7 would also get the quux library. I knew pip constantly telling me to upgrade it would pay off eventually. So cool. Thanks! -Clay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt
On 2015-07-28 19:25:04 -0700 (-0700), Clay Gerrard wrote: [...] I'd be more curious for the documentation if you have any idea where I might look for it? Is this a feature of pkg_resources, distutils, setuptools, pbr? What exactly does describing dependencies via this extras key afford? URL: https://www.python.org/dev/peps/pep-0426/#extras-optional-dependencies URL: https://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies URL: http://docs.openstack.org/developer/pbr/#extra-requirements URL: https://pip.pypa.io/en/latest/reference/pip_install.html#examples (see example #6) In short, the [extras] section of setup.cfg will be interpreted by PBR if you're using it, or for non-PBR projects you can define an extras_require dict in your setup.py instead. I'm not familiar with that syntax for pip or it's equivalent! That sounds awesome! Can you do like [extras:pluginname] in your setup.cfg and pip install project[pluginname] just works!? OMGBBQ! [...] Close. See the example in the PBR documentation linked above (note that the example for extra requirements is actually in the environment markers section, which is hopefully only a mildly confusing attempt to reduce duplication of examples). -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt
Excerpts from Clay Gerrard's message of 2015-07-28 19:25:04 -0700: Doug, I believe our glance friends are not the only project with some open questions on dealing with the required dependency for optional plugin use-case. You've made a recommendation to leverage some python tooling functionality that I'm not familiar with. I was hoping I could probe you to elaborate so I can try and educate myself more? ... inline On Tue, Jul 28, 2015 at 4:55 PM, Doug Hellmann d...@doughellmann.com wrote: Please set up an extras entry for each backend instead of just removing the dependencies. That will signal to users that you know what dependencies there are for a backend, You referenced nova [1], and oslo.versionedobjects [2] for examples - but I'd be more curious for the documentation if you have any idea where I might look for it? Is this a feature of pkg_resources, distutils, setuptools, pbr? What exactly does describing dependencies via this extras key afford? There is some documentation in the pbr manual (http://docs.openstack.org/developer/pbr/#extra-requirements). The feature is implemented throughout the packaging tool chain now. Extras is a way to provide, well, extra dependencies for cases where some items are optional. We can use it for test related items in libraries, so that for example the fixtures package is not dragged into production deployments. We can also use it in the case here in glance_store and in oslo.messaging, where a particular driver needs supporting libraries that are only needed if that driver is being used. Extras are a useful way to signal these sorts of dependencies for anyone installing via pip (which understands them) but also to downstream packagers. but that they are optional, and still allow someone to do the equivalent of pip install glance[vmware] or pip install glance[swift] to get those dependencies. I'm not familiar with that syntax for pip or it's equivalent! That sounds awesome! Can you do like [extras:pluginname] in your setup.cfg and pip install project[pluginname] just works!? OMGBBQ! The extras are listed under the extras section in setup.cfg. Each one has a name, and a list of the package specifications. The example from the pbr guide is: [extras] security = aleph bet :python_environment=='3.2' gimel :python_environment=='2.7' testing = quux :python_environment=='2.7' Here one would say pip install foo[testing] and on Python 2.7 would also get the quux library. Nova and oslo.versionedobjects have examples in their setup.cfg if you need a template. Hrm... I'm missing how either one of those setup.cfg's [1, 2] include an example relevant to this use-case (i.e. required dependency for optional backend plugin)? Oh, bother, that was a grep gone awry. Here's the example from oslo.versionedobjects: [extras] fixtures = mock=1.2 fixtures=1.3.1 If you're writing tests using oslo.versionedobjects in your project, you would add oslo.versionedobjects[fixtures] to your test-requirements.txt to get the supporting libraries needed by the fixtures module in the library. I didn't mention in the reviews, but this will also make integration tests in our gate easier, since you can put .[vmware] or .[swift] in the tox.ini to pull in those dependencies. Hrm... yes testing. So that's part just a new -e for the tox.ini - but I'm not quite sure I follow how each environment would specify different dependencies for the virtualenv? Set different values for the deps variable in the tox.ini. I hope you can point me to some more information on the subject. Thank you very much for pushing this out to a wider audience, clayg 1. https://github.com/openstack/nova/blob/master/setup.cfg 2. https://github.com/openstack/oslo.versionedobjects/blob/master/setup.cfg#L25 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt
Excerpts from Louis Taylor's message of 2015-07-29 15:25:33 +0100: On Tue, Jul 28, 2015 at 07:55:54PM -0400, Doug Hellmann wrote: I replied on both patches, but I'll repeat it here for a broader audience: Please set up an extras entry for each backend instead of just removing the dependencies. That will signal to users that you know what dependencies there are for a backend, but that they are optional, and still allow someone to do the equivalent of pip install glance[vmware] or pip install glance[swift] to get those dependencies. Nova and oslo.versionedobjects have examples in their setup.cfg if you need a template. I didn't mention in the reviews, but this will also make integration tests in our gate easier, since you can put .[vmware] or .[swift] in the tox.ini to pull in those dependencies. This sounds like the best option. What happens if you want to enable multiple backends? Can you do something like pip install glance[swift, vmware], or would you need to run a separate pip command to install each? I think you would say: pip install glance[swift] glance[vmware] Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt
On Tue, Jul 28, 2015 at 07:55:54PM -0400, Doug Hellmann wrote: I replied on both patches, but I'll repeat it here for a broader audience: Please set up an extras entry for each backend instead of just removing the dependencies. That will signal to users that you know what dependencies there are for a backend, but that they are optional, and still allow someone to do the equivalent of pip install glance[vmware] or pip install glance[swift] to get those dependencies. Nova and oslo.versionedobjects have examples in their setup.cfg if you need a template. I didn't mention in the reviews, but this will also make integration tests in our gate easier, since you can put .[vmware] or .[swift] in the tox.ini to pull in those dependencies. This sounds like the best option. What happens if you want to enable multiple backends? Can you do something like pip install glance[swift, vmware], or would you need to run a separate pip command to install each? Cheers, Louis signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt
I do agree, we don't depend or are cleaning the other clients out of the glance dependencies as well and I think swift should not be there either. The default store is filesystem store and if something is depending on the actual store clients it should be either glance_store or deployer (well for example our gate) glance itself should not have hard dependencies for 'em. - Erno From: William M Edmonds [mailto:edmon...@us.ibm.com] Sent: Monday, July 27, 2015 10:42 PM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt python-swiftclient is only needed by operators that are using the swift backend, so it really doesn't belong in requirements.txt. Listing it in requirements forces all operators to install it, even if they're not going to use the swift backend. When I proposed a change [1] to move this from requirements to test-requirements (would still be needed there because of tests using the swift backend), others raised concerns about the impact this could have on operators who use the swift backend today and would be upgrading to Liberty. I believe everyone agreed this should not be in requirements, but the fact is that it has been, so operators may have (incorrectly) been depending on that during upgrades. If we remove it in Liberty, and there are changes in Liberty that require a newer version of swiftclient, how would those operators know that they need to upgrade swiftclient? The optional dependencies spec [2] could definitely help here. I don't think we should have to wait for that, though. This is an issue we obviously already have today for other backends, which indicates folks can deal with it without an optional dependencies implementation. This would be a new concern for operators using the default swift backend, though. So how do we get the message out to those operators? And do we need to put out a message about this change in Liberty and then wait until Mitaka to actually remove this, or can we go ahead and remove in Liberty? [1] https://review.openstack.org/#/c/203242 [2] http://specs.openstack.org/openstack/oslo-specs/specs/liberty/optional-deps.html -Matthew __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt
On 07/27/2015 11:42 PM, William M Edmonds wrote: python-swiftclient is only needed by operators that are using the swift backend, so it really doesn't belong in requirements.txt. Listing it in requirements forces all operators to install it, even if they're not going to use the swift backend. When I proposed a change [1] to move this from requirements to test-requirements (would still be needed there because of tests using the swift backend), others raised concerns about the impact this could have on operators who use the swift backend today and would be upgrading to Liberty. I believe everyone agreed this should not be in requirements, but the fact is that it has been, so operators may have (incorrectly) been depending on that during upgrades. If we remove it in Liberty, and there are changes in Liberty that require a newer version of swiftclient, how would those operators know that they need to upgrade swiftclient? Even if swiftclient was removed from requirements.txt, I would still keep it as a hard Depends: in Debian, so it would not change anything for (Debian) users. Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt
From: Flavio Percoco fla...@redhat.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: 07/28/2015 07:36 AM Subject: Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt On 28/07/15 09:15 +, Kuvaja, Erno wrote: I do agree, we don’t depend or are cleaning the other clients out of the glance dependencies as well and I think swift should not be there either. The default store is filesystem store and if something is depending on the actual store clients it should be either glance_store or deployer (well for example our gate) glance itself should not have hard dependencies for ‘em. Agreed! William, would it be possible for you to spend some more time and create a single patch that removes all non-required dependencies? Cheers, Flavio This all started when I opened a bug [1] saying we needed to pull out oslo.vmware. Louis quickly threw up a patch [2] to remove that, but it was pointed out that swiftclient falls into the same category. So I created a separate patch to remove swiftclient [3]. Looking over what else is in requirements.txt and running a few searches, it looks like we may also be able to remove greenlet, kombu, and posix-ipc. Does anyone know if any of those (greenlet?) are needed for some reason other than a direct import? In which case I can add a comment to clarify that while I'm removing the others. I'd originally included the removal of oslo.vmware in [3], but I pulled that out thinking we could go ahead and merge [2] while we were having this discussion. But that didn't seem to fly, so I guess we want to make all these changes together under [3] and abandon [2]? Or should we go ahead and merge [2] while we're figuring out whether to remove greenlet, kombu, and posix-ipc as well under [3]? Just to be clear, it sounds to me like Flavio and Erno agree we should pull swiftclient out of requirements.txt immediately. I'd like to see a reply from Ian and Louis to round things out, make sure we're all on the same page and we won't be fighting over this in the review... [1] https://launchpad.net/bugs/1475737 [2] https://review.openstack.org/#/c/203200/ [3] https://review.openstack.org/#/c/203242/ - Erno From: William M Edmonds [mailto:edmon...@us.ibm.com] Sent: Monday, July 27, 2015 10:42 PM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt python-swiftclient is only needed by operators that are using the swift backend, so it really doesn't belong in requirements.txt. Listing it in requirements forces all operators to install it, even if they're not going to use the swift backend. When I proposed a change [1] to move this from requirements to test-requirements (would still be needed there because of tests using the swift backend), others raised concerns about the impact this could have on operators who use the swift backend today and would be upgrading to Liberty. I believe everyone agreed this should not be in requirements, but the fact is that it has been, so operators may have (incorrectly) been depending on that during upgrades. If we remove it in Liberty, and there are changes in Liberty that require a newer version of swiftclient, how would those operators know that they need to upgrade swiftclient? The optional dependencies spec [2] could definitely help here. I don't think we should have to wait for that, though. This is an issue we obviously already have today for other backends, which indicates folks can deal with it without an optional dependencies implementation. This would be a new concern for operators using the default swift backend, though. So how do we get the message out to those operators? And dowe need to put out a message about this change in Liberty and then wait until Mitaka to actually remove this, or can we go ahead and remove in Liberty? [1] https://review.openstack.org/#/c/203242 [2] http://specs.openstack.org/openstack/oslo-specs/specs/liberty/ optional-deps.html -Matthew __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco [attachment attdy18a.dat deleted by William M Edmonds/Raleigh/IBM] __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org
Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt
Excerpts from William M Edmonds's message of 2015-07-28 18:34:29 -0400: From: Flavio Percoco fla...@redhat.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: 07/28/2015 07:36 AM Subject: Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt On 28/07/15 09:15 +, Kuvaja, Erno wrote: I do agree, we don’t depend or are cleaning the other clients out of the glance dependencies as well and I think swift should not be there either. The default store is filesystem store and if something is depending on the actual store clients it should be either glance_store or deployer (well for example our gate) glance itself should not have hard dependencies for ‘em. Agreed! William, would it be possible for you to spend some more time and create a single patch that removes all non-required dependencies? Cheers, Flavio This all started when I opened a bug [1] saying we needed to pull out oslo.vmware. Louis quickly threw up a patch [2] to remove that, but it was pointed out that swiftclient falls into the same category. So I created a separate patch to remove swiftclient [3]. Looking over what else is in requirements.txt and running a few searches, it looks like we may also be able to remove greenlet, kombu, and posix-ipc. Does anyone know if any of those (greenlet?) are needed for some reason other than a direct import? In which case I can add a comment to clarify that while I'm removing the others. I'd originally included the removal of oslo.vmware in [3], but I pulled that out thinking we could go ahead and merge [2] while we were having this discussion. But that didn't seem to fly, so I guess we want to make all these changes together under [3] and abandon [2]? Or should we go ahead and merge [2] while we're figuring out whether to remove greenlet, kombu, and posix-ipc as well under [3]? Just to be clear, it sounds to me like Flavio and Erno agree we should pull swiftclient out of requirements.txt immediately. I'd like to see a reply from Ian and Louis to round things out, make sure we're all on the same page and we won't be fighting over this in the review... I replied on both patches, but I'll repeat it here for a broader audience: Please set up an extras entry for each backend instead of just removing the dependencies. That will signal to users that you know what dependencies there are for a backend, but that they are optional, and still allow someone to do the equivalent of pip install glance[vmware] or pip install glance[swift] to get those dependencies. Nova and oslo.versionedobjects have examples in their setup.cfg if you need a template. I didn't mention in the reviews, but this will also make integration tests in our gate easier, since you can put .[vmware] or .[swift] in the tox.ini to pull in those dependencies. Doug [1] https://launchpad.net/bugs/1475737 [2] https://review.openstack.org/#/c/203200/ [3] https://review.openstack.org/#/c/203242/ - Erno From: William M Edmonds [mailto:edmon...@us.ibm.com] Sent: Monday, July 27, 2015 10:42 PM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt python-swiftclient is only needed by operators that are using the swift backend, so it really doesn't belong in requirements.txt. Listing it in requirements forces all operators to install it, even if they're not going to use the swift backend. When I proposed a change [1] to move this from requirements to test-requirements (would still be needed there because of tests using the swift backend), others raised concerns about the impact this could have on operators who use the swift backend today and would be upgrading to Liberty. I believe everyone agreed this should not be in requirements, but the fact is that it has been, so operators may have (incorrectly) been depending on that during upgrades. If we remove it in Liberty, and there are changes in Liberty that require a newer version of swiftclient, how would those operators know that they need to upgrade swiftclient? The optional dependencies spec [2] could definitely help here. I don't think we should have to wait for that, though. This is an issue we obviously already have today for other backends, which indicates folks can deal with it without an optional dependencies implementation. This would be a new concern for operators using the default swift backend, though. So how do we get the message out to those operators? And dowe need to put out a message about this change in Liberty and then wait until Mitaka to actually remove this, or can we go ahead and remove in Liberty? [1] https://review.openstack.org/#/c/203242 [2] http://specs.openstack.org/openstack/oslo-specs/specs/liberty/ optional-deps.html
Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt
Doug, I believe our glance friends are not the only project with some open questions on dealing with the required dependency for optional plugin use-case. You've made a recommendation to leverage some python tooling functionality that I'm not familiar with. I was hoping I could probe you to elaborate so I can try and educate myself more? ... inline On Tue, Jul 28, 2015 at 4:55 PM, Doug Hellmann d...@doughellmann.com wrote: Please set up an extras entry for each backend instead of just removing the dependencies. That will signal to users that you know what dependencies there are for a backend, You referenced nova [1], and oslo.versionedobjects [2] for examples - but I'd be more curious for the documentation if you have any idea where I might look for it? Is this a feature of pkg_resources, distutils, setuptools, pbr? What exactly does describing dependencies via this extras key afford? but that they are optional, and still allow someone to do the equivalent of pip install glance[vmware] or pip install glance[swift] to get those dependencies. I'm not familiar with that syntax for pip or it's equivalent! That sounds awesome! Can you do like [extras:pluginname] in your setup.cfg and pip install project[pluginname] just works!? OMGBBQ! Nova and oslo.versionedobjects have examples in their setup.cfg if you need a template. Hrm... I'm missing how either one of those setup.cfg's [1, 2] include an example relevant to this use-case (i.e. required dependency for optional backend plugin)? I didn't mention in the reviews, but this will also make integration tests in our gate easier, since you can put .[vmware] or .[swift] in the tox.ini to pull in those dependencies. Hrm... yes testing. So that's part just a new -e for the tox.ini - but I'm not quite sure I follow how each environment would specify different dependencies for the virtualenv? I hope you can point me to some more information on the subject. Thank you very much for pushing this out to a wider audience, clayg 1. https://github.com/openstack/nova/blob/master/setup.cfg 2. https://github.com/openstack/oslo.versionedobjects/blob/master/setup.cfg#L25 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt
On 28/07/15 10:29 +0200, Thomas Goirand wrote: On 07/27/2015 11:42 PM, William M Edmonds wrote: python-swiftclient is only needed by operators that are using the swift backend, so it really doesn't belong in requirements.txt. Listing it in requirements forces all operators to install it, even if they're not going to use the swift backend. When I proposed a change [1] to move this from requirements to test-requirements (would still be needed there because of tests using the swift backend), others raised concerns about the impact this could have on operators who use the swift backend today and would be upgrading to Liberty. I believe everyone agreed this should not be in requirements, but the fact is that it has been, so operators may have (incorrectly) been depending on that during upgrades. If we remove it in Liberty, and there are changes in Liberty that require a newer version of swiftclient, how would those operators know that they need to upgrade swiftclient? Even if swiftclient was removed from requirements.txt, I would still keep it as a hard Depends: in Debian, so it would not change anything for (Debian) users. I think this is perfectly find and it's a good thing. Upstream packages don't need to follow everything that happens downstream. From a downstream perspective, it's a requirement that is not needed neither in the CI nor in most of the development environments. Cheers, Flavio Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco pgppnN17ZocFu.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt
On 28/07/15 09:15 +, Kuvaja, Erno wrote: I do agree, we don’t depend or are cleaning the other clients out of the glance dependencies as well and I think swift should not be there either. The default store is filesystem store and if something is depending on the actual store clients it should be either glance_store or deployer (well for example our gate) glance itself should not have hard dependencies for ‘em. Agreed! William, would it be possible for you to spend some more time and create a single patch that removes all non-required dependencies? Cheers, Flavio - Erno From: William M Edmonds [mailto:edmon...@us.ibm.com] Sent: Monday, July 27, 2015 10:42 PM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [glance] Removing python-swiftclient from requirements.txt python-swiftclient is only needed by operators that are using the swift backend, so it really doesn't belong in requirements.txt. Listing it in requirements forces all operators to install it, even if they're not going to use the swift backend. When I proposed a change [1] to move this from requirements to test-requirements (would still be needed there because of tests using the swift backend), others raised concerns about the impact this could have on operators who use the swift backend today and would be upgrading to Liberty. I believe everyone agreed this should not be in requirements, but the fact is that it has been, so operators may have (incorrectly) been depending on that during upgrades. If we remove it in Liberty, and there are changes in Liberty that require a newer version of swiftclient, how would those operators know that they need to upgrade swiftclient? The optional dependencies spec [2] could definitely help here. I don't think we should have to wait for that, though. This is an issue we obviously already have today for other backends, which indicates folks can deal with it without an optional dependencies implementation. This would be a new concern for operators using the default swift backend, though. So how do we get the message out to those operators? And do we need to put out a message about this change in Liberty and then wait until Mitaka to actually remove this, or can we go ahead and remove in Liberty? [1] https://review.openstack.org/#/c/203242 [2] http://specs.openstack.org/openstack/oslo-specs/specs/liberty/ optional-deps.html -Matthew __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco pgpAKKS5Gzx_m.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [glance] Removing python-swiftclient from requirements.txt
python-swiftclient is only needed by operators that are using the swift backend, so it really doesn't belong in requirements.txt. Listing it in requirements forces all operators to install it, even if they're not going to use the swift backend. When I proposed a change [1] to move this from requirements to test-requirements (would still be needed there because of tests using the swift backend), others raised concerns about the impact this could have on operators who use the swift backend today and would be upgrading to Liberty. I believe everyone agreed this should not be in requirements, but the fact is that it has been, so operators may have (incorrectly) been depending on that during upgrades. If we remove it in Liberty, and there are changes in Liberty that require a newer version of swiftclient, how would those operators know that they need to upgrade swiftclient? The optional dependencies spec [2] could definitely help here. I don't think we should have to wait for that, though. This is an issue we obviously already have today for other backends, which indicates folks can deal with it without an optional dependencies implementation. This would be a new concern for operators using the default swift backend, though. So how do we get the message out to those operators? And do we need to put out a message about this change in Liberty and then wait until Mitaka to actually remove this, or can we go ahead and remove in Liberty? [1] https://review.openstack.org/#/c/203242 [2] http://specs.openstack.org/openstack/oslo-specs/specs/liberty/optional-deps.html -Matthew__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
