Re: [openstack-dev] openstack/requirements and tarball subdirs
On Mon, Jul 14, 2014 at 9:04 AM, Philipp Marek wrote: >> > It might be better to work with the python-dbus authors to get a >> > pip-installable package released to PyPI, so that it can be included >> > in the tox virtualenv (though for DevStack we'd still want to use >> > distro packages instead of PyPI, I think). >> I sent Simon an email about that now. > I talked to him today. > > The recommendation is to use the available (distribution) packages, because > he's had bad experiences with the distutils build system, so he wants to > stick to the Autotools - and the general issues (needing a C compiler, some > header files) would be the same with other libraries, too. We do depend on system libraries without python components, but for python libraries we want to manage the version we use by installing it from pypi so we can indicate to distros when we need a version they might not be packaging yet. >> > > You'll also need to modify cinder's tox.ini to set "sitepackages = >> > > True" so the virtualenvs created for the unit tests can see the global >> > > site-packages directory. Nova does the same thing for some of its >> > > dependencies. >> > ... I'm a little worried about taking on >> > sitepackages=True in more projects given the headaches it causes >> > (conflicts between versions in your virtualenv and system-installed >> > python modules which happen to be dependencies of the operating >> > system, for example the issues we ran into with Jinja2 on CentOS 6 >> > last year). >> But such a change would affect _all_ people, right? >> Hmmm... If you think such a change will be accepted? > So we're back to this question now. > > > While I don't have enough knowledge about the interactions to just change > the virtual-env setup in DevStack, I can surely create an issue on > https://github.com/openstack-dev/devstack. That change wouldn't be in devstack, it would be in every project that wants to use this dbus library. However, we shouldn't do that -- I wasn't aware of the changes to the way libvirt works and the fact that the precedent I cited was being eliminated. > How would this requirement be done for "production" setups? Should > installers read the requirements.txt and install matching distribution > packages? Pip needs to be able to install the entries in requirements.txt (that file is used by pbr to build the dependency list given to pip). > > Or is that out of scope of OpenStack/Cinder development anyway, and so > I can/should ignore that? > > > Regards, > > Phil > > -- > : Ing. Philipp Marek > : LINBIT | Your Way to High Availability > : DRBD/HA support and consulting http://www.linbit.com : > > DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] openstack/requirements and tarball subdirs
> It might be better to work with the python-dbus authors to get a > pip-installable package released to PyPI, so that it can be included > in the tox virtualenv (though for DevStack we'd still want to use > distro packages instead of PyPI, I think). I sent Simon an email about that now. -- : Ing. Philipp Marek : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com : DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] openstack/requirements and tarball subdirs
On Tue, Jul 8, 2014 at 10:18 AM, Jeremy Stanley wrote: > On 2014-07-08 09:09:35 -0400 (-0400), Doug Hellmann wrote: > [...] >> You'll also need to modify cinder's tox.ini to set "sitepackages = >> True" so the virtualenvs created for the unit tests can see the global >> site-packages directory. Nova does the same thing for some of its >> dependencies. > > Nova did this for python-libvirt I believe, but now that we finally > have platforms with new-enough libvirt to support the split-out > python-libvirt library on PyPI we can hopefully finally stop doing OK, I wasn't aware of that change. > this in master at least. I'm a little worried about taking on > sitepackages=True in more projects given the headaches it causes > (conflicts between versions in your virtualenv and system-installed > python modules which happen to be dependencies of the operating > system, for example the issues we ran into with Jinja2 on CentOS 6 > last year). > > It might be better to work with the python-dbus authors to get a > pip-installable package released to PyPI, so that it can be included > in the tox virtualenv (though for DevStack we'd still want to use > distro packages instead of PyPI, I think). I agree it would be better to have a pip-installable package. It's not clear if the dbus-python authors care about that, but I think asking them is the next step. A search on PyPI shows a couple of packages that look like alternatives, including at least one that claims to support asynchronous I/O. If the dbus-python authors won't package their lib for PyPI, we should consider looking at the other libraries more closely. Doug > -- > Jeremy Stanley > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] openstack/requirements and tarball subdirs
On 2014-07-08 09:09:35 -0400 (-0400), Doug Hellmann wrote: [...] > You'll also need to modify cinder's tox.ini to set "sitepackages = > True" so the virtualenvs created for the unit tests can see the global > site-packages directory. Nova does the same thing for some of its > dependencies. Nova did this for python-libvirt I believe, but now that we finally have platforms with new-enough libvirt to support the split-out python-libvirt library on PyPI we can hopefully finally stop doing this in master at least. I'm a little worried about taking on sitepackages=True in more projects given the headaches it causes (conflicts between versions in your virtualenv and system-installed python modules which happen to be dependencies of the operating system, for example the issues we ran into with Jinja2 on CentOS 6 last year). It might be better to work with the python-dbus authors to get a pip-installable package released to PyPI, so that it can be included in the tox virtualenv (though for DevStack we'd still want to use distro packages instead of PyPI, I think). -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] openstack/requirements and tarball subdirs
> >> The 1.2.0 release from > >> http://dbus.freedesktop.org/releases/dbus-python/ doesn't look like an > >> sdist, either (I see a configure script, so I think they've switched > >> to autoconf). Uploading that version to PyPI isn't going to give you > >> something you can install with pip. Are there system packages for > >> dbus-python for the distros we support directly? > > Yes; RHEL6 and Ubuntu 12.04 include python-dbus packages. > > How about SuSE and Debian? Ubuntu got the package from Debian AFAIK; it's available. A google search seems to indicate a dbus-1-python-devel-0.83.0-27.1.43.x86_64.rpm for SLES11-SP3. > >> That release also appears to be just over a year old. Do you know if > >> dbus-python is being actively maintained any more? Are there other > >> libraries for talking to dbus? > > AFAIK dbus-python is the most current and preferred one. > > > > http://dbus.freedesktop.org/doc/dbus-python/ lists two alternatives, but as > > these are not packaged (yet) I chose python-dbus instead. > > > > > > Can Jenkins use the pre-packaged versions instead of downloading and > > compiling the tarball? > > If dbus-python is indeed the best library, that may be the way to go. > System-level dependencies can be installed via devstack, so you could > submit a patch to devstack to install this library for cinder's use by > editing files/*/cinder. Within the devstack repository, I guess. > You'll also need to modify cinder's tox.ini to set "sitepackages = > True" so the virtualenvs created for the unit tests can see the global > site-packages directory. Nova does the same thing for some of its > dependencies. But such a change would affect _all_ people, right? Hmmm... If you think such a change will be accepted? Thank you for your help! Regards, Phil -- : Ing. Philipp Marek : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com : DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] openstack/requirements and tarball subdirs
On Tue, Jul 8, 2014 at 8:55 AM, Philipp Marek wrote: >> > The other tarballs in that hierarchy follow the same schema; perhaps the >> > cached download is broken? >> >> I downloaded the tarball and didn't find a setup.py at all. > Oh, that is the requirement? I'd have guessed that the directory name is at > fault here. > >> > The most current releases are available on >> > http://dbus.freedesktop.org/releases/dbus-python/ >> > though; perhaps the 1.2.0 release works better? >> > >> > But how could I specify to use _that_ source URL? >> >> Unfortunately, you can't. We only mirror PyPI, and we only download >> packages published there. >> >> The 1.2.0 release from >> http://dbus.freedesktop.org/releases/dbus-python/ doesn't look like an >> sdist, either (I see a configure script, so I think they've switched >> to autoconf). Uploading that version to PyPI isn't going to give you >> something you can install with pip. Are there system packages for >> dbus-python for the distros we support directly? > Yes; RHEL6 and Ubuntu 12.04 include python-dbus packages. How about SuSE and Debian? > >> That release also appears to be just over a year old. Do you know if >> dbus-python is being actively maintained any more? Are there other >> libraries for talking to dbus? > AFAIK dbus-python is the most current and preferred one. > > http://dbus.freedesktop.org/doc/dbus-python/ lists two alternatives, but as > these are not packaged (yet) I chose python-dbus instead. > > > Can Jenkins use the pre-packaged versions instead of downloading and > compiling the tarball? If dbus-python is indeed the best library, that may be the way to go. System-level dependencies can be installed via devstack, so you could submit a patch to devstack to install this library for cinder's use by editing files/*/cinder. You'll also need to modify cinder's tox.ini to set "sitepackages = True" so the virtualenvs created for the unit tests can see the global site-packages directory. Nova does the same thing for some of its dependencies. Doug > > > Regards, > > Phil > > -- > : Ing. Philipp Marek > : LINBIT | Your Way to High Availability > : DRBD/HA support and consulting http://www.linbit.com : > > DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] openstack/requirements and tarball subdirs
> > The other tarballs in that hierarchy follow the same schema; perhaps the > > cached download is broken? > > I downloaded the tarball and didn't find a setup.py at all. Oh, that is the requirement? I'd have guessed that the directory name is at fault here. > > The most current releases are available on > > http://dbus.freedesktop.org/releases/dbus-python/ > > though; perhaps the 1.2.0 release works better? > > > > But how could I specify to use _that_ source URL? > > Unfortunately, you can't. We only mirror PyPI, and we only download > packages published there. > > The 1.2.0 release from > http://dbus.freedesktop.org/releases/dbus-python/ doesn't look like an > sdist, either (I see a configure script, so I think they've switched > to autoconf). Uploading that version to PyPI isn't going to give you > something you can install with pip. Are there system packages for > dbus-python for the distros we support directly? Yes; RHEL6 and Ubuntu 12.04 include python-dbus packages. > That release also appears to be just over a year old. Do you know if > dbus-python is being actively maintained any more? Are there other > libraries for talking to dbus? AFAIK dbus-python is the most current and preferred one. http://dbus.freedesktop.org/doc/dbus-python/ lists two alternatives, but as these are not packaged (yet) I chose python-dbus instead. Can Jenkins use the pre-packaged versions instead of downloading and compiling the tarball? Regards, Phil -- : Ing. Philipp Marek : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com : DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] openstack/requirements and tarball subdirs
On Tue, Jul 8, 2014 at 1:46 AM, Philipp Marek wrote: > > Hello Doug, > > > thank you for your help. > >> > I guess the problem is that the subdirectory within that tarball includes >> > the version number, as in "dbus-python-0.84.0/". How can I tell the extract >> > script that it should look into that one? >> >> It looks like that package wasn't built correctly as an sdist, so pip >> won't install it. Have you contacted the author to report the problem >> as a bug? > No, not yet. > > I thought that it was okay, being hosted on python.org and so on. > > The other tarballs in that hierarchy follow the same schema; perhaps the > cached download is broken? I downloaded the tarball and didn't find a setup.py at all. > > > The most current releases are available on > http://dbus.freedesktop.org/releases/dbus-python/ > though; perhaps the 1.2.0 release works better? > > But how could I specify to use _that_ source URL? Unfortunately, you can't. We only mirror PyPI, and we only download packages published there. The 1.2.0 release from http://dbus.freedesktop.org/releases/dbus-python/ doesn't look like an sdist, either (I see a configure script, so I think they've switched to autoconf). Uploading that version to PyPI isn't going to give you something you can install with pip. Are there system packages for dbus-python for the distros we support directly? That release also appears to be just over a year old. Do you know if dbus-python is being actively maintained any more? Are there other libraries for talking to dbus? Doug > > > Thank you! > > > Regards, > > Phil > > > -- > : Ing. Philipp Marek > : LINBIT | Your Way to High Availability > : DRBD/HA support and consulting http://www.linbit.com : > > DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] openstack/requirements and tarball subdirs
Hello Doug, thank you for your help. > > I guess the problem is that the subdirectory within that tarball includes > > the version number, as in "dbus-python-0.84.0/". How can I tell the extract > > script that it should look into that one? > > It looks like that package wasn't built correctly as an sdist, so pip > won't install it. Have you contacted the author to report the problem > as a bug? No, not yet. I thought that it was okay, being hosted on python.org and so on. The other tarballs in that hierarchy follow the same schema; perhaps the cached download is broken? The most current releases are available on http://dbus.freedesktop.org/releases/dbus-python/ though; perhaps the 1.2.0 release works better? But how could I specify to use _that_ source URL? Thank you! Regards, Phil -- : Ing. Philipp Marek : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com : DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] openstack/requirements and tarball subdirs
On Mon, Jul 7, 2014 at 5:14 AM, Philipp Marek wrote: > Hi everybody, > > I'm trying to get > https://review.openstack.org/#/c/99013/ > through Jenkins, but keep failing. > > > The requirement I'm trying to add is >> dbus-python>=0.83 # MIT License > > > The logfile at > > http://logs.openstack.org/13/99013/2/check/check-requirements-integration-dsvm/d6e5418/console.html.gz > says this: > >> Downloading/unpacking dbus-python>=0.83 (from -r /tmp/tmpFt8D8L (line 13)) > Loads the tarball from > > https://pypi.python.org/packages/source/d/dbus-python/dbus-python-0.84.0.tar.gz. >> Using download cache from /tmp/tmp.JszD7LLXey/download/... >> Running setup.py (path:/tmp/...) egg_info for package dbus-python > > but then fails >>Traceback (most recent call last): >> File "", line 17, in >>IOError: [Errno 2] No such file or directory: >>'/tmp/tmpH1D5G3/build/dbus-python/setup.py' >>Complete output from command python setup.py egg_info: >>Traceback (most recent call last): >> >> File "", line 17, in >> >> IOError: [Errno 2] No such file or directory: >> '/tmp/tmpH1D5G3/build/dbus-python/setup.py' > > I guess the problem is that the subdirectory within that tarball includes > the version number, as in "dbus-python-0.84.0/". How can I tell the extract > script that it should look into that one? It looks like that package wasn't built correctly as an sdist, so pip won't install it. Have you contacted the author to report the problem as a bug? Doug > > > Thank you for your help! > > > Regards, > > Phil > > -- > : Ing. Philipp Marek > : LINBIT | Your Way to High Availability > : DRBD/HA support and consulting http://www.linbit.com : > > DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
