Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On 30 September 2014 14:05, Robert Collins robe...@robertcollins.net wrote: I'm poking around now. It looks like pip install -e outside a venv writes to /usr/local/lib/python2.7/dist-packages, which is pure crack - dist-packages is for the distro, site-packages for pip. Sigh. Also writing to /usr/local/lib when the python prefix is /usr. More weirdness. Will be back with more soon :) So, I've pushed up an updated oslo.db patch that works in the wider set of situations we've identified. I've attached my test script (adjust as needed - be sure to use the updated pbr and the oslo tree from github and adjust paths as needed :)). The basic theory in this particular implementation boils down to this: - within distro packages, things are installed as normal, its a no-brainer. - within a venv, pip can happily ignore any dependent namespace packages it cannot import and just install them so the failure mode on things we can't import is that we install them in the venv as well - but we add the namespace site packages when working from the working directory, so that we can pick those things up [but we could just ignore them and pip install them] I think there is one remaining flaw, which is that I haven't written the code to *not* add system site (turns out python doesn't really expose that in a standard fashion) - but I think that can wait for some folk to beat on this and report whether it works for them too. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud test-oslo Description: Binary data ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On Sep 25, 2014, at 6:14 PM, Robert Collins robe...@robertcollins.net wrote: On 24 September 2014 11:03, Robert Collins robe...@robertcollins.net wrote: So... FWIW I think I've got a cleaner implementation of namespaces *for our context* - it takes inspiration from the PEP-420 discussion and final design. It all started when Mike reported issues with testr to me. https://bugs.launchpad.net/oslo.db/+bug/1372250 tl;dr: we should stop using pkg_resources style namespace packages and instead have an effectively empty oslo package that sets up the namespace, which all namespaced libraries would depend on. With a stub __init__ in local source trees that adds the site-packages path to itself automatically, and excluding that file in sdist, it should be entirely transparent to developers and packagers, with no file conflicts etc. This works with the existing pkg_resources namespace packages, lets us migrate away from the pkg_resources implementation one package at a time, and we don't need to rename any of the packages, and it works fine with uninstalled and install -e installed source trees. We need: - a new olso package to introduce a common oslo/__init__.py (recommended in the pre-PEP420 world) - a tiny pbr bugfix: https://review.openstack.org/123597 - and a patch like so to each project: https://review.openstack.org/123604 I have such an olso package https://github.com/rbtcollins/oslo, if this sounds reasonable I will push up an infra patch to create it. Doug raised on IRC a concern about system-site-packages. I have tested this, and I can make it work, but I'm not sure its needed: it is totally broken today: # Put oslo.config in the system site and oslo.18n not yet installed sudo apt-get install oslo.config sudo apt-get remove oslo.i18n # make a virtualenv with system site packages mkvirtualenv --system-site-packages test-system-site # install oslo.18n pip install oslo.i18n # now when I tested oslo.i18n doesn't depent on oslo.config, but lets be sure: python -c 'import oslo.config; print oslo.config.__file__' Traceback (most recent call last): File string, line 1, in module ImportError: No module named config # Oh look! can't import oslo.config. pip install oslo.db ... python -c 'import oslo.config; print oslo.config.__file__' /home/robertc/.virtualenvs/test-system-site/local/lib/python2.7/site-packages/oslo/config/__init__.pyc # Now we need it, it got pulled in. Now, as I say, I can fix this quite easily with a virtualenv aware pth file, but since its broken today and AFAIK there isn't a bug open about this, I think it will be fine. The breaking case was having a package installed editable in the global site-packages and then installing another sub-package non-editable in the virtualenv. We stopped doing that by making devstack install oslo libraries non-editable, but I think, based on what you’re saying, if the test environment has access to the global site-packages and only some of the oslo libs are installed then building the test environment will fail if it needs another lib because it won’t install the packages it can see globally. Is that right? Has anyone encountered that situation? When you install e.g. oslo.db which *does* depend on oslo.config, oslo.config is being installed within the venv. I'm not sure if thats strictly due to version constraints, or if its systemic. So - I'd like to say that its a separate preexisting issue and we can loop back and tackle it should it show up as a problem. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On Sep 23, 2014, at 7:03 PM, Robert Collins robe...@robertcollins.net wrote: On 29 August 2014 04:42, Sean Dague s...@dague.net wrote: On 08/28/2014 12:22 PM, Doug Hellmann wrote: ... The problem is that the setuptools implementation of namespace packages breaks in a way that is repeatable but difficult to debug when a common OpenStack installation pattern is used. So the fix is “don’t do that” where I thought “that” meant the installation pattern and Sean thought it meant “use namespace packages”. :-) Stupid english... be more specific! Yeh, Doug provides the most concise statement of where we failed on communication (I take a big chunk of that blame). Hopefully now it's a lot clearer what's going on, and why it hurts if you do it. So... FWIW I think I've got a cleaner implementation of namespaces *for our context* - it takes inspiration from the PEP-420 discussion and final design. It all started when Mike reported issues with testr to me. https://bugs.launchpad.net/oslo.db/+bug/1366869 tl;dr: we should stop using pkg_resources style namespace packages and instead have an effectively empty oslo package that sets up the namespace, which all namespaced libraries would depend on. With a stub __init__ in local source trees that adds the site-packages path to itself automatically, and excluding that file in sdist, it should be entirely transparent to developers and packagers, with no file conflicts etc. This works with the existing pkg_resources namespace packages, lets us migrate away from the pkg_resources implementation one package at a time, and we don't need to rename any of the packages, and it works fine with uninstalled and install -e installed source trees. We need: - a new olso package to introduce a common oslo/__init__.py (recommended in the pre-PEP420 world) - a tiny pbr bugfix: https://review.openstack.org/123597 - and a patch like so to each project: https://review.openstack.org/123604 I have such an olso package https://github.com/rbtcollins/oslo, if this sounds reasonable I will push up an infra patch to create it. You say this would work with uninstalled and install -e source trees”. Would this work if the base oslo package was installed with -e as well, as we might do in devstack? I’m not trying to shoot down the idea, I’m just trying to make sure we’ve thought of all of the cases we’ve seen things break. Doug -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On 24 September 2014 11:03, Robert Collins robe...@robertcollins.net wrote: So... FWIW I think I've got a cleaner implementation of namespaces *for our context* - it takes inspiration from the PEP-420 discussion and final design. It all started when Mike reported issues with testr to me. https://bugs.launchpad.net/oslo.db/+bug/1372250 tl;dr: we should stop using pkg_resources style namespace packages and instead have an effectively empty oslo package that sets up the namespace, which all namespaced libraries would depend on. With a stub __init__ in local source trees that adds the site-packages path to itself automatically, and excluding that file in sdist, it should be entirely transparent to developers and packagers, with no file conflicts etc. This works with the existing pkg_resources namespace packages, lets us migrate away from the pkg_resources implementation one package at a time, and we don't need to rename any of the packages, and it works fine with uninstalled and install -e installed source trees. We need: - a new olso package to introduce a common oslo/__init__.py (recommended in the pre-PEP420 world) - a tiny pbr bugfix: https://review.openstack.org/123597 - and a patch like so to each project: https://review.openstack.org/123604 I have such an olso package https://github.com/rbtcollins/oslo, if this sounds reasonable I will push up an infra patch to create it. Doug raised on IRC a concern about system-site-packages. I have tested this, and I can make it work, but I'm not sure its needed: it is totally broken today: # Put oslo.config in the system site and oslo.18n not yet installed sudo apt-get install oslo.config sudo apt-get remove oslo.i18n # make a virtualenv with system site packages mkvirtualenv --system-site-packages test-system-site # install oslo.18n pip install oslo.i18n # now when I tested oslo.i18n doesn't depent on oslo.config, but lets be sure: python -c 'import oslo.config; print oslo.config.__file__' Traceback (most recent call last): File string, line 1, in module ImportError: No module named config # Oh look! can't import oslo.config. pip install oslo.db ... python -c 'import oslo.config; print oslo.config.__file__' /home/robertc/.virtualenvs/test-system-site/local/lib/python2.7/site-packages/oslo/config/__init__.pyc # Now we need it, it got pulled in. Now, as I say, I can fix this quite easily with a virtualenv aware pth file, but since its broken today and AFAIK there isn't a bug open about this, I think it will be fine. When you install e.g. oslo.db which *does* depend on oslo.config, oslo.config is being installed within the venv. I'm not sure if thats strictly due to version constraints, or if its systemic. So - I'd like to say that its a separate preexisting issue and we can loop back and tackle it should it show up as a problem. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On 29 August 2014 04:42, Sean Dague s...@dague.net wrote: On 08/28/2014 12:22 PM, Doug Hellmann wrote: ... The problem is that the setuptools implementation of namespace packages breaks in a way that is repeatable but difficult to debug when a common OpenStack installation pattern is used. So the fix is “don’t do that” where I thought “that” meant the installation pattern and Sean thought it meant “use namespace packages”. :-) Stupid english... be more specific! Yeh, Doug provides the most concise statement of where we failed on communication (I take a big chunk of that blame). Hopefully now it's a lot clearer what's going on, and why it hurts if you do it. So... FWIW I think I've got a cleaner implementation of namespaces *for our context* - it takes inspiration from the PEP-420 discussion and final design. It all started when Mike reported issues with testr to me. https://bugs.launchpad.net/oslo.db/+bug/1366869 tl;dr: we should stop using pkg_resources style namespace packages and instead have an effectively empty oslo package that sets up the namespace, which all namespaced libraries would depend on. With a stub __init__ in local source trees that adds the site-packages path to itself automatically, and excluding that file in sdist, it should be entirely transparent to developers and packagers, with no file conflicts etc. This works with the existing pkg_resources namespace packages, lets us migrate away from the pkg_resources implementation one package at a time, and we don't need to rename any of the packages, and it works fine with uninstalled and install -e installed source trees. We need: - a new olso package to introduce a common oslo/__init__.py (recommended in the pre-PEP420 world) - a tiny pbr bugfix: https://review.openstack.org/123597 - and a patch like so to each project: https://review.openstack.org/123604 I have such an olso package https://github.com/rbtcollins/oslo, if this sounds reasonable I will push up an infra patch to create it. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On Sep 23, 2014, at 7:03 PM, Robert Collins robe...@robertcollins.net wrote: On 29 August 2014 04:42, Sean Dague s...@dague.net wrote: On 08/28/2014 12:22 PM, Doug Hellmann wrote: ... The problem is that the setuptools implementation of namespace packages breaks in a way that is repeatable but difficult to debug when a common OpenStack installation pattern is used. So the fix is “don’t do that” where I thought “that” meant the installation pattern and Sean thought it meant “use namespace packages”. :-) Stupid english... be more specific! Yeh, Doug provides the most concise statement of where we failed on communication (I take a big chunk of that blame). Hopefully now it's a lot clearer what's going on, and why it hurts if you do it. So... FWIW I think I've got a cleaner implementation of namespaces *for our context* - it takes inspiration from the PEP-420 discussion and final design. It all started when Mike reported issues with testr to me. https://bugs.launchpad.net/oslo.db/+bug/1366869 I think you’ve got the wrong bug linked in here and your review, seems like you meant this one: https://bugs.launchpad.net/oslo.db/+bug/1372250 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
Nuts, thank you. On 24 September 2014 14:28, Mike Bayer mba...@redhat.com wrote: On Sep 23, 2014, at 7:03 PM, Robert Collins robe...@robertcollins.net wrote: On 29 August 2014 04:42, Sean Dague s...@dague.net wrote: On 08/28/2014 12:22 PM, Doug Hellmann wrote: ... The problem is that the setuptools implementation of namespace packages breaks in a way that is repeatable but difficult to debug when a common OpenStack installation pattern is used. So the fix is “don’t do that” where I thought “that” meant the installation pattern and Sean thought it meant “use namespace packages”. :-) Stupid english... be more specific! Yeh, Doug provides the most concise statement of where we failed on communication (I take a big chunk of that blame). Hopefully now it's a lot clearer what's going on, and why it hurts if you do it. So... FWIW I think I've got a cleaner implementation of namespaces *for our context* - it takes inspiration from the PEP-420 discussion and final design. It all started when Mike reported issues with testr to me. https://bugs.launchpad.net/oslo.db/+bug/1366869 I think you’ve got the wrong bug linked in here and your review, seems like you meant this one: https://bugs.launchpad.net/oslo.db/+bug/1372250 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On 08/27/2014 05:52 PM, Doug Hellmann wrote: On Aug 27, 2014, at 11:14 AM, Flavio Percoco fla...@redhat.com wrote: On 08/27/2014 04:31 PM, Sean Dague wrote: So this change came in with adding glance.store - https://review.openstack.org/#/c/115265/5/lib/glance, which I think is a bad direction to be headed. Here is the problem when it comes to working with code from git, in python, that uses namespaces, it's kind of a hack that violates the principle of least surprise. For instance: cd /opt/stack/oslo.vmware pip install . cd /opt/stack/olso.config pip install -e . python -m olso.vmware /usr/bin/python: No module named olso.vmware In python 2.7 (using pip) namespaces are a bolt on because of the way importing modules works. And depending on how you install things in a namespace will overwrite the base __init__.py for the top level part of the namespace in such a way that you can't get access to the submodules. It's well known, and every conversation with dstuft that I've had in the past was don't use namespaces. A big reason we see this a lot is due to the fact that devstack does 'editable' pip installs for most things, because the point is it's a development environment, and you should be able to change code, and see if live without having to go through the install step again. If people remember the constant issues with oslo.config in unit tests 9 months ago, this was because of mismatch of editable vs. non editable libraries in the system and virtualenvs. This took months to get to a consistent workaround. The *workaround* that was done is we just gave up on installing olso libs in a development friendly way. I don't consider that a solution, it's a work around. But it has some big implications for the usefulness of the development environment. It also definitely violates the principle of least surprise, as changes to oslo.messaging in a devstack env don't immediately apply, you have to reinstall olso.messaging to get them to take. If this is just oslo, that's one thing (and still something I think should be revisted, because when the maintainer of pip says don't do this I'm inclined to go by that). But this change aims to start brining this pattern into other projects. Realistically I'm quite concerned that this will trigger more work arounds and confusion. It also means, for instance, that once we are in a namespace we can never decide to install some of the namespace from pypi and some of it from git editable (because it's a part that's under more interesting rapid development). So I'd like us to revisit using a namespace for glance, and honestly, for other places in OpenStack, because these kinds of violations of the principle of least surprise is something that I'd like us to be actively minimizing. Sean, Thanks for bringing this up. To be honest, I became familiar with these namespace issues when I started working on glance.store. That said, I realize how this can be an issue even with the current workaround. Unfortunately, it's already quite late in the release and starting the rename process will delay Glance's migration to glance.store leaving us with not enough time to test it and make sure things are working as expected. With full transparency, I don't have an counter-argument for what you're saying/proposing. I talked to Doug on IRC and he mentioned this is something that won't be fixed in py27 so there's not even hope on seeing it fixed/working soon. Based on that, I'm happy to rename glance.store but, I'd like us to think in a good way to make this rename happen without blocking the glance.store work in Juno. When you asked me about namespace packages, I thought using them was fine because we had worked out how to do it for Oslo and the same approach worked for you in glance. I didn’t realize that was still considered an unresolved issue, so I apologize that my mistake has ended up causing you more work, Flavio. NP. Good thing is we now have a plan and a reference for future cases if there happen to be any. You've been way too helpful in this whole process. Thanks for all your guidance and support, really. I've 2 ways to make this happen: 1. Do a partial rename and then complete it after the glance migration is done. If I'm not missing anything, we should be able to do something like: - Rename the project internally - Release a new version with the new name `glancestore` - Switch glance over to `glancestore` - Complete the rename process with support from infra This seems like the best approach. If nothing is using the library now, all of the name changes would need to happen within the library. You can release “glancestore” from a git repo called “glance.store” and we can rename that repo somewhere down the line, so you shouldn’t be blocked on the infra team (who I expect are going to be really busy keeping an eye on the gate as we get close to
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On 27/08/14 16:31, Sean Dague wrote: [snip] In python 2.7 (using pip) namespaces are a bolt on because of the way importing modules works. And depending on how you install things in a namespace will overwrite the base __init__.py for the top level part of the namespace in such a way that you can't get access to the submodules. It's well known, and every conversation with dstuft that I've had in the past was don't use namespaces. I think this is actually a solved problem. You just need a single line in your __init__.py files: https://bitbucket.org/thomaswaldmann/xstatic-jquery/src/tip/xstatic/__init__.py -- Radomir Dopieralski ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On 28/08/14 12:41, Radomir Dopieralski wrote: On 27/08/14 16:31, Sean Dague wrote: [snip] In python 2.7 (using pip) namespaces are a bolt on because of the way importing modules works. And depending on how you install things in a namespace will overwrite the base __init__.py for the top level part of the namespace in such a way that you can't get access to the submodules. It's well known, and every conversation with dstuft that I've had in the past was don't use namespaces. I think this is actually a solved problem. You just need a single line in your __init__.py files: https://bitbucket.org/thomaswaldmann/xstatic-jquery/src/tip/xstatic/__init__.py More on that at http://pythonhosted.org/setuptools/setuptools.html?namespace-packages#namespace-packages -- Radomir Dopieralski ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On 08/28/2014 12:50 PM, Radomir Dopieralski wrote: On 28/08/14 12:41, Radomir Dopieralski wrote: On 27/08/14 16:31, Sean Dague wrote: [snip] In python 2.7 (using pip) namespaces are a bolt on because of the way importing modules works. And depending on how you install things in a namespace will overwrite the base __init__.py for the top level part of the namespace in such a way that you can't get access to the submodules. It's well known, and every conversation with dstuft that I've had in the past was don't use namespaces. I think this is actually a solved problem. You just need a single line in your __init__.py files: https://bitbucket.org/thomaswaldmann/xstatic-jquery/src/tip/xstatic/__init__.py More on that at http://pythonhosted.org/setuptools/setuptools.html?namespace-packages#namespace-packages This is not solved. In fact, we already declare the namespace in the __init__ files. The problem, as Sean mentioned, is the way pip and setuptools will install this packages, which ends up overwriting the base __init__ file. Thanks for the links, though. Flavio -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On Aug 28, 2014, at 6:41 AM, Radomir Dopieralski openst...@sheep.art.pl wrote: On 27/08/14 16:31, Sean Dague wrote: [snip] In python 2.7 (using pip) namespaces are a bolt on because of the way importing modules works. And depending on how you install things in a namespace will overwrite the base __init__.py for the top level part of the namespace in such a way that you can't get access to the submodules. It's well known, and every conversation with dstuft that I've had in the past was don't use namespaces. I think this is actually a solved problem. You just need a single line in your __init__.py files: https://bitbucket.org/thomaswaldmann/xstatic-jquery/src/tip/xstatic/__init__.py The problem is that the setuptools implementation of namespace packages breaks in a way that is repeatable but difficult to debug when a common OpenStack installation pattern is used. So the fix is “don’t do that” where I thought “that” meant the installation pattern and Sean thought it meant “use namespace packages”. :-) Doug ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On 08/28/2014 12:22 PM, Doug Hellmann wrote: On Aug 28, 2014, at 6:41 AM, Radomir Dopieralski openst...@sheep.art.pl wrote: On 27/08/14 16:31, Sean Dague wrote: [snip] In python 2.7 (using pip) namespaces are a bolt on because of the way importing modules works. And depending on how you install things in a namespace will overwrite the base __init__.py for the top level part of the namespace in such a way that you can't get access to the submodules. It's well known, and every conversation with dstuft that I've had in the past was don't use namespaces. I think this is actually a solved problem. You just need a single line in your __init__.py files: https://bitbucket.org/thomaswaldmann/xstatic-jquery/src/tip/xstatic/__init__.py The problem is that the setuptools implementation of namespace packages breaks in a way that is repeatable but difficult to debug when a common OpenStack installation pattern is used. So the fix is “don’t do that” where I thought “that” meant the installation pattern and Sean thought it meant “use namespace packages”. :-) Stupid english... be more specific! Yeh, Doug provides the most concise statement of where we failed on communication (I take a big chunk of that blame). Hopefully now it's a lot clearer what's going on, and why it hurts if you do it. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
So this change came in with adding glance.store - https://review.openstack.org/#/c/115265/5/lib/glance, which I think is a bad direction to be headed. Here is the problem when it comes to working with code from git, in python, that uses namespaces, it's kind of a hack that violates the principle of least surprise. For instance: cd /opt/stack/oslo.vmware pip install . cd /opt/stack/olso.config pip install -e . python -m olso.vmware /usr/bin/python: No module named olso.vmware In python 2.7 (using pip) namespaces are a bolt on because of the way importing modules works. And depending on how you install things in a namespace will overwrite the base __init__.py for the top level part of the namespace in such a way that you can't get access to the submodules. It's well known, and every conversation with dstuft that I've had in the past was don't use namespaces. A big reason we see this a lot is due to the fact that devstack does 'editable' pip installs for most things, because the point is it's a development environment, and you should be able to change code, and see if live without having to go through the install step again. If people remember the constant issues with oslo.config in unit tests 9 months ago, this was because of mismatch of editable vs. non editable libraries in the system and virtualenvs. This took months to get to a consistent workaround. The *workaround* that was done is we just gave up on installing olso libs in a development friendly way. I don't consider that a solution, it's a work around. But it has some big implications for the usefulness of the development environment. It also definitely violates the principle of least surprise, as changes to oslo.messaging in a devstack env don't immediately apply, you have to reinstall olso.messaging to get them to take. If this is just oslo, that's one thing (and still something I think should be revisted, because when the maintainer of pip says don't do this I'm inclined to go by that). But this change aims to start brining this pattern into other projects. Realistically I'm quite concerned that this will trigger more work arounds and confusion. It also means, for instance, that once we are in a namespace we can never decide to install some of the namespace from pypi and some of it from git editable (because it's a part that's under more interesting rapid development). So I'd like us to revisit using a namespace for glance, and honestly, for other places in OpenStack, because these kinds of violations of the principle of least surprise is something that I'd like us to be actively minimizing. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On 08/27/2014 09:31 AM, Sean Dague wrote: So this change came in with adding glance.store - https://review.openstack.org/#/c/115265/5/lib/glance, which I think is a bad direction to be headed. Here is the problem when it comes to working with code from git, in python, that uses namespaces, it's kind of a hack that violates the principle of least surprise. For instance: cd /opt/stack/oslo.vmware pip install . cd /opt/stack/olso.config pip install -e . python -m olso.vmware /usr/bin/python: No module named olso.vmware In python 2.7 (using pip) namespaces are a bolt on because of the way importing modules works. And depending on how you install things in a namespace will overwrite the base __init__.py for the top level part of the namespace in such a way that you can't get access to the submodules. It's well known, and every conversation with dstuft that I've had in the past was don't use namespaces. A big reason we see this a lot is due to the fact that devstack does 'editable' pip installs for most things, because the point is it's a development environment, and you should be able to change code, and see if live without having to go through the install step again. If people remember the constant issues with oslo.config in unit tests 9 months ago, this was because of mismatch of editable vs. non editable libraries in the system and virtualenvs. This took months to get to a consistent workaround. The *workaround* that was done is we just gave up on installing olso libs in a development friendly way. I don't consider that a solution, it's a work around. But it has some big implications for the usefulness of the development environment. It also definitely violates the principle of least surprise, as changes to oslo.messaging in a devstack env don't immediately apply, you have to reinstall olso.messaging to get them to take. If this is just oslo, that's one thing (and still something I think should be revisted, because when the maintainer of pip says don't do this I'm inclined to go by that). But this change aims to start brining this pattern into other projects. Realistically I'm quite concerned that this will trigger more work arounds and confusion. It also means, for instance, that once we are in a namespace we can never decide to install some of the namespace from pypi and some of it from git editable (because it's a part that's under more interesting rapid development). So I'd like us to revisit using a namespace for glance, and honestly, for other places in OpenStack, because these kinds of violations of the principle of least surprise is something that I'd like us to be actively minimizing. -Sean +1. Just say no to namespaces. The only reason we didn't rip them out of oslo completely is that renaming libraries used by so many projects is painful in and of itself. If you can avoid making the namespace mistake up front that's absolutely the way to go. -Ben ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On 08/27/2014 04:31 PM, Sean Dague wrote: So this change came in with adding glance.store - https://review.openstack.org/#/c/115265/5/lib/glance, which I think is a bad direction to be headed. Here is the problem when it comes to working with code from git, in python, that uses namespaces, it's kind of a hack that violates the principle of least surprise. For instance: cd /opt/stack/oslo.vmware pip install . cd /opt/stack/olso.config pip install -e . python -m olso.vmware /usr/bin/python: No module named olso.vmware In python 2.7 (using pip) namespaces are a bolt on because of the way importing modules works. And depending on how you install things in a namespace will overwrite the base __init__.py for the top level part of the namespace in such a way that you can't get access to the submodules. It's well known, and every conversation with dstuft that I've had in the past was don't use namespaces. A big reason we see this a lot is due to the fact that devstack does 'editable' pip installs for most things, because the point is it's a development environment, and you should be able to change code, and see if live without having to go through the install step again. If people remember the constant issues with oslo.config in unit tests 9 months ago, this was because of mismatch of editable vs. non editable libraries in the system and virtualenvs. This took months to get to a consistent workaround. The *workaround* that was done is we just gave up on installing olso libs in a development friendly way. I don't consider that a solution, it's a work around. But it has some big implications for the usefulness of the development environment. It also definitely violates the principle of least surprise, as changes to oslo.messaging in a devstack env don't immediately apply, you have to reinstall olso.messaging to get them to take. If this is just oslo, that's one thing (and still something I think should be revisted, because when the maintainer of pip says don't do this I'm inclined to go by that). But this change aims to start brining this pattern into other projects. Realistically I'm quite concerned that this will trigger more work arounds and confusion. It also means, for instance, that once we are in a namespace we can never decide to install some of the namespace from pypi and some of it from git editable (because it's a part that's under more interesting rapid development). So I'd like us to revisit using a namespace for glance, and honestly, for other places in OpenStack, because these kinds of violations of the principle of least surprise is something that I'd like us to be actively minimizing. Sean, Thanks for bringing this up. To be honest, I became familiar with these namespace issues when I started working on glance.store. That said, I realize how this can be an issue even with the current workaround. Unfortunately, it's already quite late in the release and starting the rename process will delay Glance's migration to glance.store leaving us with not enough time to test it and make sure things are working as expected. With full transparency, I don't have an counter-argument for what you're saying/proposing. I talked to Doug on IRC and he mentioned this is something that won't be fixed in py27 so there's not even hope on seeing it fixed/working soon. Based on that, I'm happy to rename glance.store but, I'd like us to think in a good way to make this rename happen without blocking the glance.store work in Juno. I've 2 ways to make this happen: 1. Do a partial rename and then complete it after the glance migration is done. If I'm not missing anything, we should be able to do something like: - Rename the project internally - Release a new version with the new name `glancestore` - Switch glance over to `glancestore` - Complete the rename process with support from infra 2. Let this patch land, complete Glance's switch-over using namespaces and then do the rename all together. Do you have any other suggestion that would help avoiding namespaces without blocking glance.store? Thanks, Flavio -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On Aug 27, 2014, at 10:31 AM, Sean Dague s...@dague.net wrote: So this change came in with adding glance.store - https://review.openstack.org/#/c/115265/5/lib/glance, which I think is a bad direction to be headed. Here is the problem when it comes to working with code from git, in python, that uses namespaces, it's kind of a hack that violates the principle of least surprise. For instance: cd /opt/stack/oslo.vmware pip install . cd /opt/stack/olso.config pip install -e . python -m olso.vmware /usr/bin/python: No module named olso.vmware In python 2.7 (using pip) namespaces are a bolt on because of the way importing modules works. And depending on how you install things in a namespace will overwrite the base __init__.py for the top level part of the namespace in such a way that you can't get access to the submodules. It's well known, and every conversation with dstuft that I've had in the past was don't use namespaces”. I’ve been using namespace packages on and off for 10+ years, and OpenStack is the first project where I’ve encountered any issues. That doesn’t necessarily mean we shouldn’t change, but it’s also not fair to paint them as completely broken. Many projects continue to use them successfully. A big reason we see this a lot is due to the fact that devstack does 'editable' pip installs for most things, because the point is it's a development environment, and you should be able to change code, and see if live without having to go through the install step again. If people remember the constant issues with oslo.config in unit tests 9 months ago, this was because of mismatch of editable vs. non editable libraries in the system and virtualenvs. This took months to get to a consistent workaround. The *workaround* that was done is we just gave up on installing olso libs in a development friendly way. I don't consider that a solution, it's a work around. But it has some big implications for the usefulness of the development environment. It also definitely violates the principle of least surprise, as changes to oslo.messaging in a devstack env don't immediately apply, you have to reinstall olso.messaging to get them to take. We did make that change, but IIRC we also found and fixed some dependency management issues that were causing non-editable versions of libraries to be installed even though there was already another version present on the system. I thought at the time that not installing the oslo libs editable was a “belt and suspenders” change, rather than the final fix. If this is just oslo, that's one thing (and still something I think should be revisted, because when the maintainer of pip says don't do this I'm inclined to go by that). But this change aims to start brining this pattern into other projects. Realistically I'm quite concerned that this will trigger more work arounds and confusion. The reason we decided to leave oslo packages in the oslo namespace was due to the pain of renaming the existing libs and repackaging them in the distros. It might be possible to build a package that contains both oslo.config and oslo_config, though, so I’ll spend some time investigating that. If we can make that work, the Oslo team can look into applying that change to all of our libs in the next cycle. If not, we’ll need to get some feedback from our packagers about how much pain renaming things is going to cause for upgrades before we make any changes. It also means, for instance, that once we are in a namespace we can never decide to install some of the namespace from pypi and some of it from git editable (because it's a part that's under more interesting rapid development). So I'd like us to revisit using a namespace for glance, and honestly, for other places in OpenStack, because these kinds of violations of the principle of least surprise is something that I'd like us to be actively minimizing. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On Aug 27, 2014, at 11:14 AM, Flavio Percoco fla...@redhat.com wrote: On 08/27/2014 04:31 PM, Sean Dague wrote: So this change came in with adding glance.store - https://review.openstack.org/#/c/115265/5/lib/glance, which I think is a bad direction to be headed. Here is the problem when it comes to working with code from git, in python, that uses namespaces, it's kind of a hack that violates the principle of least surprise. For instance: cd /opt/stack/oslo.vmware pip install . cd /opt/stack/olso.config pip install -e . python -m olso.vmware /usr/bin/python: No module named olso.vmware In python 2.7 (using pip) namespaces are a bolt on because of the way importing modules works. And depending on how you install things in a namespace will overwrite the base __init__.py for the top level part of the namespace in such a way that you can't get access to the submodules. It's well known, and every conversation with dstuft that I've had in the past was don't use namespaces. A big reason we see this a lot is due to the fact that devstack does 'editable' pip installs for most things, because the point is it's a development environment, and you should be able to change code, and see if live without having to go through the install step again. If people remember the constant issues with oslo.config in unit tests 9 months ago, this was because of mismatch of editable vs. non editable libraries in the system and virtualenvs. This took months to get to a consistent workaround. The *workaround* that was done is we just gave up on installing olso libs in a development friendly way. I don't consider that a solution, it's a work around. But it has some big implications for the usefulness of the development environment. It also definitely violates the principle of least surprise, as changes to oslo.messaging in a devstack env don't immediately apply, you have to reinstall olso.messaging to get them to take. If this is just oslo, that's one thing (and still something I think should be revisted, because when the maintainer of pip says don't do this I'm inclined to go by that). But this change aims to start brining this pattern into other projects. Realistically I'm quite concerned that this will trigger more work arounds and confusion. It also means, for instance, that once we are in a namespace we can never decide to install some of the namespace from pypi and some of it from git editable (because it's a part that's under more interesting rapid development). So I'd like us to revisit using a namespace for glance, and honestly, for other places in OpenStack, because these kinds of violations of the principle of least surprise is something that I'd like us to be actively minimizing. Sean, Thanks for bringing this up. To be honest, I became familiar with these namespace issues when I started working on glance.store. That said, I realize how this can be an issue even with the current workaround. Unfortunately, it's already quite late in the release and starting the rename process will delay Glance's migration to glance.store leaving us with not enough time to test it and make sure things are working as expected. With full transparency, I don't have an counter-argument for what you're saying/proposing. I talked to Doug on IRC and he mentioned this is something that won't be fixed in py27 so there's not even hope on seeing it fixed/working soon. Based on that, I'm happy to rename glance.store but, I'd like us to think in a good way to make this rename happen without blocking the glance.store work in Juno. When you asked me about namespace packages, I thought using them was fine because we had worked out how to do it for Oslo and the same approach worked for you in glance. I didn’t realize that was still considered an unresolved issue, so I apologize that my mistake has ended up causing you more work, Flavio. I've 2 ways to make this happen: 1. Do a partial rename and then complete it after the glance migration is done. If I'm not missing anything, we should be able to do something like: - Rename the project internally - Release a new version with the new name `glancestore` - Switch glance over to `glancestore` - Complete the rename process with support from infra This seems like the best approach. If nothing is using the library now, all of the name changes would need to happen within the library. You can release “glancestore” from a git repo called “glance.store” and we can rename that repo somewhere down the line, so you shouldn’t be blocked on the infra team (who I expect are going to be really busy keeping an eye on the gate as we get close to feature freeze). Was there anything else you were worried about being blocked on by the rename? We also need to figure out how to avoid problems like this in the future. We can’t expect everyone to be intimately familiar with all of the work
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On Aug 27, 2014, at 11:45 AM, Doug Hellmann d...@doughellmann.com wrote: On Aug 27, 2014, at 10:31 AM, Sean Dague s...@dague.net mailto:s...@dague.net wrote: So this change came in with adding glance.store - https://review.openstack.org/#/c/115265/5/lib/glance, which I think is a bad direction to be headed. Here is the problem when it comes to working with code from git, in python, that uses namespaces, it's kind of a hack that violates the principle of least surprise. For instance: cd /opt/stack/oslo.vmware pip install . cd /opt/stack/olso.config pip install -e . python -m olso.vmware /usr/bin/python: No module named olso.vmware In python 2.7 (using pip) namespaces are a bolt on because of the way importing modules works. And depending on how you install things in a namespace will overwrite the base __init__.py for the top level part of the namespace in such a way that you can't get access to the submodules. It's well known, and every conversation with dstuft that I've had in the past was don't use namespaces”. I’ve been using namespace packages on and off for 10+ years, and OpenStack is the first project where I’ve encountered any issues. That doesn’t necessarily mean we shouldn’t change, but it’s also not fair to paint them as completely broken. Many projects continue to use them successfully. Just for the record, there are at least 3 different ways of installing a package using pip (under the cover ways), and there are two different ways for pip to tell setuptools to handle the namespace packages. Unfortunately both ways of namespace package handling only work on 2/3 of the ways to install things. Unfortunately theres not much that can be done about this, it’s a fundamental flaw in the way setuptools namespace packages work. The changes in Python 3 to enable real namespace packages should work without those problems, but you know, Python 3 only. Generally it’s my opinion that ``import foo_bar`` isn’t particularly any better or worse than ``import foo.bar``. The only real benefit is being able to iterate over ``foo.*``, however I’d just recommend using entry points instead of trying to do magic based on the name. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them
On Aug 27, 2014, at 11:55 AM, Donald Stufft don...@stufft.io wrote: On Aug 27, 2014, at 11:45 AM, Doug Hellmann d...@doughellmann.com wrote: On Aug 27, 2014, at 10:31 AM, Sean Dague s...@dague.net wrote: So this change came in with adding glance.store - https://review.openstack.org/#/c/115265/5/lib/glance, which I think is a bad direction to be headed. Here is the problem when it comes to working with code from git, in python, that uses namespaces, it's kind of a hack that violates the principle of least surprise. For instance: cd /opt/stack/oslo.vmware pip install . cd /opt/stack/olso.config pip install -e . python -m olso.vmware /usr/bin/python: No module named olso.vmware In python 2.7 (using pip) namespaces are a bolt on because of the way importing modules works. And depending on how you install things in a namespace will overwrite the base __init__.py for the top level part of the namespace in such a way that you can't get access to the submodules. It's well known, and every conversation with dstuft that I've had in the past was don't use namespaces”. I’ve been using namespace packages on and off for 10+ years, and OpenStack is the first project where I’ve encountered any issues. That doesn’t necessarily mean we shouldn’t change, but it’s also not fair to paint them as completely broken. Many projects continue to use them successfully. Just for the record, there are at least 3 different ways of installing a package using pip (under the cover ways), and there are two different ways for pip to tell setuptools to handle the namespace packages. Unfortunately both ways of namespace package handling only work on 2/3 of the ways to install things. Unfortunately theres not much that can be done about this, it’s a fundamental flaw in the way setuptools namespace packages work. The changes in Python 3 to enable real namespace packages should work without those problems, but you know, Python 3 only. Generally it’s my opinion that ``import foo_bar`` isn’t particularly any better or worse than ``import foo.bar``. The only real benefit is being able to iterate over ``foo.*``, however I’d just recommend using entry points instead of trying to do magic based on the name. Yeah, we’re not doing anything like that AFAIK, so that shouldn’t be a problem. I’m worried about ensuring that upgrades work, ensuring new versions of the existing libs don’t break stable releases, and not having to handle a lot of back ports to separate libraries for things that could otherwise be semver bumps. I’ll spend some time testing to see if we can create a shim layer with the namespace package to avoid some of those issues. Doug --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
