Re: [openstack-dev] openstack/requirements and tarball subdirs

2014-07-14 Thread Doug Hellmann
On Mon, Jul 14, 2014 at 9:04 AM, Philipp Marek philipp.ma...@linbit.com wrote:
  It might be better to work with the python-dbus authors to get a
  pip-installable package released to PyPI, so that it can be included
  in the tox virtualenv (though for DevStack we'd still want to use
  distro packages instead of PyPI, I think).
 I sent Simon an email about that now.
 I talked to him today.

 The recommendation is to use the available (distribution) packages, because
 he's had bad experiences with the distutils build system, so he wants to
 stick to the Autotools - and the general issues (needing a C compiler, some
 header files) would be the same with other libraries, too.

We do depend on system libraries without python components, but for
python libraries we want to manage the version we use by installing it
from pypi so we can indicate to distros when we need a version they
might not be packaging yet.

   You'll also need to modify cinder's tox.ini to set sitepackages =
   True so the virtualenvs created for the unit tests can see the global
   site-packages directory. Nova does the same thing for some of its
   dependencies.
  ... I'm a little worried about taking on
  sitepackages=True in more projects given the headaches it causes
  (conflicts between versions in your virtualenv and system-installed
  python modules which happen to be dependencies of the operating
  system, for example the issues we ran into with Jinja2 on CentOS 6
  last year).
 But such a change would affect _all_ people, right?
 Hmmm... If you think such a change will be accepted?
 So we're back to this question now.


 While I don't have enough knowledge about the interactions to just change
 the virtual-env setup in DevStack, I can surely create an issue on
 https://github.com/openstack-dev/devstack.

That change wouldn't be in devstack, it would be in every project that
wants to use this dbus library. However, we shouldn't do that -- I
wasn't aware of the changes to the way libvirt works and the fact that
the precedent I cited was being eliminated.

 How would this requirement be done for production setups? Should
 installers read the requirements.txt and install matching distribution
 packages?

Pip needs to be able to install the entries in requirements.txt (that
file is used by pbr to build the dependency list given to pip).


 Or is that out of scope of OpenStack/Cinder development anyway, and so
 I can/should ignore that?


 Regards,

 Phil

 --
 : Ing. Philipp Marek
 : LINBIT | Your Way to High Availability
 : DRBD/HA support and consulting http://www.linbit.com :

 DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] openstack/requirements and tarball subdirs

2014-07-09 Thread Philipp Marek
 It might be better to work with the python-dbus authors to get a
 pip-installable package released to PyPI, so that it can be included
 in the tox virtualenv (though for DevStack we'd still want to use
 distro packages instead of PyPI, I think).
I sent Simon an email about that now.


-- 
: Ing. Philipp Marek
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com :

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] openstack/requirements and tarball subdirs

2014-07-08 Thread Doug Hellmann
On Tue, Jul 8, 2014 at 1:46 AM, Philipp Marek philipp.ma...@linbit.com wrote:

 Hello Doug,


 thank you for your help.

  I guess the problem is that the subdirectory within that tarball includes
  the version number, as in dbus-python-0.84.0/. How can I tell the extract
  script that it should look into that one?

 It looks like that package wasn't built correctly as an sdist, so pip
 won't install it. Have you contacted the author to report the problem
 as a bug?
 No, not yet.

 I thought that it was okay, being hosted on python.org and so on.

 The other tarballs in that hierarchy follow the same schema; perhaps the
 cached download is broken?

I downloaded the tarball and didn't find a setup.py at all.



 The most current releases are available on
 http://dbus.freedesktop.org/releases/dbus-python/
 though; perhaps the 1.2.0 release works better?

 But how could I specify to use _that_ source URL?

Unfortunately, you can't. We only mirror PyPI, and we only download
packages published there.

The 1.2.0 release from
http://dbus.freedesktop.org/releases/dbus-python/ doesn't look like an
sdist, either (I see a configure script, so I think they've switched
to autoconf). Uploading that version to PyPI isn't going to give you
something you can install with pip. Are there system packages for
dbus-python for the distros we support directly?

That release also appears to be just over a year old. Do you know if
dbus-python is being actively maintained any more? Are there other
libraries for talking to dbus?

Doug



 Thank you!


 Regards,

 Phil


 --
 : Ing. Philipp Marek
 : LINBIT | Your Way to High Availability
 : DRBD/HA support and consulting http://www.linbit.com :

 DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] openstack/requirements and tarball subdirs

2014-07-08 Thread Doug Hellmann
On Tue, Jul 8, 2014 at 8:55 AM, Philipp Marek philipp.ma...@linbit.com wrote:
  The other tarballs in that hierarchy follow the same schema; perhaps the
  cached download is broken?

 I downloaded the tarball and didn't find a setup.py at all.
 Oh, that is the requirement? I'd have guessed that the directory name is at
 fault here.

  The most current releases are available on
  http://dbus.freedesktop.org/releases/dbus-python/
  though; perhaps the 1.2.0 release works better?
 
  But how could I specify to use _that_ source URL?

 Unfortunately, you can't. We only mirror PyPI, and we only download
 packages published there.

 The 1.2.0 release from
 http://dbus.freedesktop.org/releases/dbus-python/ doesn't look like an
 sdist, either (I see a configure script, so I think they've switched
 to autoconf). Uploading that version to PyPI isn't going to give you
 something you can install with pip. Are there system packages for
 dbus-python for the distros we support directly?
 Yes; RHEL6 and Ubuntu 12.04 include python-dbus packages.

How about SuSE and Debian?


 That release also appears to be just over a year old. Do you know if
 dbus-python is being actively maintained any more? Are there other
 libraries for talking to dbus?
 AFAIK dbus-python is the most current and preferred one.

 http://dbus.freedesktop.org/doc/dbus-python/ lists two alternatives, but as
 these are not packaged (yet) I chose python-dbus instead.


 Can Jenkins use the pre-packaged versions instead of downloading and
 compiling the tarball?

If dbus-python is indeed the best library, that may be the way to go.
System-level dependencies can be installed via devstack, so you could
submit a patch to devstack to install this library for cinder's use by
editing files/*/cinder.

You'll also need to modify cinder's tox.ini to set sitepackages =
True so the virtualenvs created for the unit tests can see the global
site-packages directory. Nova does the same thing for some of its
dependencies.

Doug



 Regards,

 Phil

 --
 : Ing. Philipp Marek
 : LINBIT | Your Way to High Availability
 : DRBD/HA support and consulting http://www.linbit.com :

 DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] openstack/requirements and tarball subdirs

2014-07-08 Thread Philipp Marek
  The 1.2.0 release from
  http://dbus.freedesktop.org/releases/dbus-python/ doesn't look like an
  sdist, either (I see a configure script, so I think they've switched
  to autoconf). Uploading that version to PyPI isn't going to give you
  something you can install with pip. Are there system packages for
  dbus-python for the distros we support directly?
  Yes; RHEL6 and Ubuntu 12.04 include python-dbus packages.
 
 How about SuSE and Debian?
Ubuntu got the package from Debian AFAIK; it's available.

A google search seems to indicate a 
dbus-1-python-devel-0.83.0-27.1.43.x86_64.rpm
for SLES11-SP3.


  That release also appears to be just over a year old. Do you know if
  dbus-python is being actively maintained any more? Are there other
  libraries for talking to dbus?
  AFAIK dbus-python is the most current and preferred one.
 
  http://dbus.freedesktop.org/doc/dbus-python/ lists two alternatives, but as
  these are not packaged (yet) I chose python-dbus instead.
 
 
  Can Jenkins use the pre-packaged versions instead of downloading and
  compiling the tarball?
 
 If dbus-python is indeed the best library, that may be the way to go.
 System-level dependencies can be installed via devstack, so you could
 submit a patch to devstack to install this library for cinder's use by
 editing files/*/cinder.
Within the devstack repository, I guess.


 You'll also need to modify cinder's tox.ini to set sitepackages =
 True so the virtualenvs created for the unit tests can see the global
 site-packages directory. Nova does the same thing for some of its
 dependencies.
But such a change would affect _all_ people, right? 

Hmmm... If you think such a change will be accepted?


Thank you for your help!


Regards,

Phil

-- 
: Ing. Philipp Marek
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com :

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] openstack/requirements and tarball subdirs

2014-07-08 Thread Jeremy Stanley
On 2014-07-08 09:09:35 -0400 (-0400), Doug Hellmann wrote:
[...]
 You'll also need to modify cinder's tox.ini to set sitepackages =
 True so the virtualenvs created for the unit tests can see the global
 site-packages directory. Nova does the same thing for some of its
 dependencies.

Nova did this for python-libvirt I believe, but now that we finally
have platforms with new-enough libvirt to support the split-out
python-libvirt library on PyPI we can hopefully finally stop doing
this in master at least. I'm a little worried about taking on
sitepackages=True in more projects given the headaches it causes
(conflicts between versions in your virtualenv and system-installed
python modules which happen to be dependencies of the operating
system, for example the issues we ran into with Jinja2 on CentOS 6
last year).

It might be better to work with the python-dbus authors to get a
pip-installable package released to PyPI, so that it can be included
in the tox virtualenv (though for DevStack we'd still want to use
distro packages instead of PyPI, I think).
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] openstack/requirements and tarball subdirs

2014-07-08 Thread Doug Hellmann
On Tue, Jul 8, 2014 at 10:18 AM, Jeremy Stanley fu...@yuggoth.org wrote:
 On 2014-07-08 09:09:35 -0400 (-0400), Doug Hellmann wrote:
 [...]
 You'll also need to modify cinder's tox.ini to set sitepackages =
 True so the virtualenvs created for the unit tests can see the global
 site-packages directory. Nova does the same thing for some of its
 dependencies.

 Nova did this for python-libvirt I believe, but now that we finally
 have platforms with new-enough libvirt to support the split-out
 python-libvirt library on PyPI we can hopefully finally stop doing

OK, I wasn't aware of that change.

 this in master at least. I'm a little worried about taking on
 sitepackages=True in more projects given the headaches it causes
 (conflicts between versions in your virtualenv and system-installed
 python modules which happen to be dependencies of the operating
 system, for example the issues we ran into with Jinja2 on CentOS 6
 last year).

 It might be better to work with the python-dbus authors to get a
 pip-installable package released to PyPI, so that it can be included
 in the tox virtualenv (though for DevStack we'd still want to use
 distro packages instead of PyPI, I think).

I agree it would be better to have a pip-installable package. It's not
clear if the dbus-python authors care about that, but I think asking
them is the next step.

A search on PyPI shows a couple of packages that look like
alternatives, including at least one that claims to support
asynchronous I/O. If the dbus-python authors won't package their lib
for PyPI, we should consider looking at the other libraries more
closely.

Doug

 --
 Jeremy Stanley

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] openstack/requirements and tarball subdirs

2014-07-07 Thread Doug Hellmann
On Mon, Jul 7, 2014 at 5:14 AM, Philipp Marek philipp.ma...@linbit.com wrote:
 Hi everybody,

 I'm trying to get
 https://review.openstack.org/#/c/99013/
 through Jenkins, but keep failing.


 The requirement I'm trying to add is
 dbus-python=0.83 # MIT License


 The logfile at
 
 http://logs.openstack.org/13/99013/2/check/check-requirements-integration-dsvm/d6e5418/console.html.gz
 says this:

 Downloading/unpacking dbus-python=0.83 (from -r /tmp/tmpFt8D8L (line 13))
 Loads the tarball from
   
 https://pypi.python.org/packages/source/d/dbus-python/dbus-python-0.84.0.tar.gz.
   Using download cache from /tmp/tmp.JszD7LLXey/download/...
   Running setup.py (path:/tmp/...) egg_info for package dbus-python

 but then fails
Traceback (most recent call last):
  File string, line 17, in module
IOError: [Errno 2] No such file or directory:
'/tmp/tmpH1D5G3/build/dbus-python/setup.py'
Complete output from command python setup.py egg_info:
Traceback (most recent call last):

  File string, line 17, in module

 IOError: [Errno 2] No such file or directory:
   '/tmp/tmpH1D5G3/build/dbus-python/setup.py'

 I guess the problem is that the subdirectory within that tarball includes
 the version number, as in dbus-python-0.84.0/. How can I tell the extract
 script that it should look into that one?

It looks like that package wasn't built correctly as an sdist, so pip
won't install it. Have you contacted the author to report the problem
as a bug?

Doug



 Thank you for your help!


 Regards,

 Phil

 --
 : Ing. Philipp Marek
 : LINBIT | Your Way to High Availability
 : DRBD/HA support and consulting http://www.linbit.com :

 DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] openstack/requirements and tarball subdirs

2014-07-07 Thread Philipp Marek

Hello Doug,


thank you for your help.

  I guess the problem is that the subdirectory within that tarball includes
  the version number, as in dbus-python-0.84.0/. How can I tell the extract
  script that it should look into that one?
 
 It looks like that package wasn't built correctly as an sdist, so pip
 won't install it. Have you contacted the author to report the problem
 as a bug?
No, not yet.

I thought that it was okay, being hosted on python.org and so on.

The other tarballs in that hierarchy follow the same schema; perhaps the 
cached download is broken?


The most current releases are available on
http://dbus.freedesktop.org/releases/dbus-python/
though; perhaps the 1.2.0 release works better?

But how could I specify to use _that_ source URL?


Thank you!


Regards,

Phil


-- 
: Ing. Philipp Marek
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com :

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev