Re: [openstack-dev] global-reqs on tooz pulls in worrisome transitive dep
On Thu, Sep 11, 2014 at 2:17 AM, Thomas Goirand z...@debian.org wrote: On my side (as the Debian package maintainer of OpenStack), I was more than happy to see that Ceilometer made the choice to use a Python module for memcache which supports Python 3. Currently python-memcache does *not* support Python 3. It's in fact standing in the way to add Python 3 compatibility to *a lot* of the OpenStack packages, because this directly impact python-keystoneclient, which is a (build-)dependency of almost everything. Thomas - python-keystoneclient should no longer have a hard dependency on python-memcache(d). The auth_token middleware which can use memcache has been moved into the keystonemiddleware repo (a copy is left in keystoneclient only for backwards-compatibility). If python-keystoneclient still has a dependency on python-memcache then we're doing something wrong and should be able to fix it. - Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] global-reqs on tooz pulls in worrisome transitive dep
I do not see python-memcache library in either keystone client’s requirements.txt[0] or test-requirements.txt[1]. For purposes of ensuring that we do not break people deploying auth_token in keystoneclient (for older releases) I don’t see the soft dependency on python-memcache from going away. Even for keystonemiddleware we do not have a hard-dependency on python-memcache in requirements.txt[2] or test-requirements.txt[3] as we gate on py33. —Morgan [0] http://git.openstack.org/cgit/openstack/python-keystoneclient/tree/requirements.txt?id=0.10.1 [1] http://git.openstack.org/cgit/openstack/python-keystoneclient/tree/test-requirements.txt?id=0.10.1 [2] http://git.openstack.org/cgit/openstack/keystonemiddleware/tree/requirements.txt?id=1.1.1 [3] http://git.openstack.org/cgit/openstack/keystonemiddleware/tree/test-requirements.txt?id=1.1.1 — Morgan Fainberg -Original Message- From: Brant Knudson b...@acm.org Reply: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: September 12, 2014 at 08:33:15 To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] global-reqs on tooz pulls in worrisome transitive dep On Thu, Sep 11, 2014 at 2:17 AM, Thomas Goirand wrote: On my side (as the Debian package maintainer of OpenStack), I was more than happy to see that Ceilometer made the choice to use a Python module for memcache which supports Python 3. Currently python-memcache does *not* support Python 3. It's in fact standing in the way to add Python 3 compatibility to *a lot* of the OpenStack packages, because this directly impact python-keystoneclient, which is a (build-)dependency of almost everything. Thomas - python-keystoneclient should no longer have a hard dependency on python-memcache(d). The auth_token middleware which can use memcache has been moved into the keystonemiddleware repo (a copy is left in keystoneclient only for backwards-compatibility). If python-keystoneclient still has a dependency on python-memcache then we're doing something wrong and should be able to fix it. - Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] global-reqs on tooz pulls in worrisome transitive dep
Hi Matt, On 09/10/2014 04:30 AM, Matt Riedemann wrote: It took me a while to untangle this so prepare for links. :) I noticed this change [1] today for global-requirements to require tooz [2] for a ceilometer blueprint [3]. The sad part is that tooz requires pymemcache [4] which is, from what I can tell, a memcached client that is not the same as python-memcached [5]. Note that python-memcached is listed in global-requirements already [6]. The problem I have with this is it doesn't appear that RHEL/Fedora package pymemcache (they do package python-memcached). I see that openSUSE builds separate packages for each. It looks like Ubuntu also has separate packages. My question is, is this a problem? I'm assuming RDO will just have to package python-pymemcache themselves but what about people not using RDO (SOL? Don't care? Other?). Reverting the requirements change would probably mean reverting the ceilometer blueprint (or getting a version of tooz out that works with python-memcached which is probably too late for that right now). Given the point in the schedule that seems pretty drastic. Maybe I'm making more of this than it's worth but wanted to bring it up in case anyone else has concerns. [1] https://review.openstack.org/#/c/93443/ [2] https://github.com/stackforge/tooz/blob/master/requirements.txt#L6 [3] http://specs.openstack.org/openstack/ceilometer-specs/specs/juno/central-agent-partitioning.html [4] https://pypi.python.org/pypi/pymemcache [5] https://pypi.python.org/pypi/python-memcached/ [6] https://github.com/openstack/requirements/blob/master/global-requirements.txt#L108 On my side (as the Debian package maintainer of OpenStack), I was more than happy to see that Ceilometer made the choice to use a Python module for memcache which supports Python 3. Currently python-memcache does *not* support Python 3. It's in fact standing in the way to add Python 3 compatibility to *a lot* of the OpenStack packages, because this directly impact python-keystoneclient, which is a (build-)dependency of almost everything. This situation has been very frustrating for me. I really would like this to be solved. I see 2 ways to have it solved: 1- Complete the Python 3 support for python-memcache or 2- Switch all OpenStack packages to pymemcache There has been a few attempts with 1- already, with even a complete fork of the project which you can see on both github and pypi. But it's a real problem that it hasn't been upstreamed, and that the fork doesn't support *both* Python 2 and 3. I've tried to work out the port to Python 3 myself, but lamely failed. It's not that easy, as one has to deal with unicode string from and to the memcache server, and I have to admin I lack knowledge of how this works. Also, the unit tests of python-memcache would have to be re-written in a much better way than it is right now. It would be really cool if someone could work on porting python-memcache to Python 3, regardless if we decide to switch to pymemcache: that way, we'd addressed the Python 3 compatibility quickly. If one wants to work out a Python 3 compat in python-memcache, it'd be really great to have it before Debian Jessie is frozen on the 5th of Nov, so that I get a chance to package it. I asked several people, it seems that none really wants to do it (is the code too ugly? probably...). As for 2-, well, that would be a lot of work, I guess. I haven't compared both APIs yet, but they should be different. And we have calls to python-memcache in a lot of our projects. But this seems to be the best way forward, as pymemcache seems to be written in a much better way: smaller method, and the code is easier to understand., with more extensive unit tests. As for the issue with Redhat packaging, well, I'm sorry if that's a problem for you, though really, pymemcache is a good choice, and I support Julien on that one. Cheers, Thomas Goirand (zigo) P.S: It's to be noted that python-memcached is called python-memcache in Debian, because that's the name of the module when doing imports... ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] global-reqs on tooz pulls in worrisome transitive dep
On Tue, Sep 09 2014, Matt Riedemann wrote: I noticed this change [1] today for global-requirements to require tooz [2] for a ceilometer blueprint [3]. The sad part is that tooz requires pymemcache [4] which is, from what I can tell, a memcached client that is not the same as python-memcached [5]. Note that python-memcached is listed in global-requirements already [6]. You're not going to control the entire full list of dependency of things we use in OpenStack, so this kind of situation is going to arise anyway. The problem I have with this is it doesn't appear that RHEL/Fedora package pymemcache (they do package python-memcached). I see that openSUSE builds separate packages for each. It looks like Ubuntu also has separate packages. My question is, is this a problem? I'm assuming RDO will just have to package python-pymemcache themselves but what about people not using RDO (SOL? Don't care? Other?). Reverting the requirements change would probably mean reverting the ceilometer blueprint (or getting a version of tooz out that works with python-memcached which is probably too late for that right now). Given the point in the schedule that seems pretty drastic. python-memcached is a terrible memcache client, which does not support Python 3. pymemcache is way better than python-memcached, and everybody should switch to it. When we started tooz from scratch a year ago, there was no point starting to use a non-Python 3 compatible and crappy memcache client. pymemcache shouldn't be a problem to package anyway. :) -- Julien Danjou /* Free Software hacker http://julien.danjou.info */ signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] global-reqs on tooz pulls in worrisome transitive dep
It took me a while to untangle this so prepare for links. :) I noticed this change [1] today for global-requirements to require tooz [2] for a ceilometer blueprint [3]. The sad part is that tooz requires pymemcache [4] which is, from what I can tell, a memcached client that is not the same as python-memcached [5]. Note that python-memcached is listed in global-requirements already [6]. The problem I have with this is it doesn't appear that RHEL/Fedora package pymemcache (they do package python-memcached). I see that openSUSE builds separate packages for each. It looks like Ubuntu also has separate packages. My question is, is this a problem? I'm assuming RDO will just have to package python-pymemcache themselves but what about people not using RDO (SOL? Don't care? Other?). Reverting the requirements change would probably mean reverting the ceilometer blueprint (or getting a version of tooz out that works with python-memcached which is probably too late for that right now). Given the point in the schedule that seems pretty drastic. Maybe I'm making more of this than it's worth but wanted to bring it up in case anyone else has concerns. [1] https://review.openstack.org/#/c/93443/ [2] https://github.com/stackforge/tooz/blob/master/requirements.txt#L6 [3] http://specs.openstack.org/openstack/ceilometer-specs/specs/juno/central-agent-partitioning.html [4] https://pypi.python.org/pypi/pymemcache [5] https://pypi.python.org/pypi/python-memcached/ [6] https://github.com/openstack/requirements/blob/master/global-requirements.txt#L108 -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] global-reqs on tooz pulls in worrisome transitive dep
For the future == IMHO, we shouldn't be pulling in duplicate dependencies when we can control it. Since tooz is part of stackforge, it's somewhat part of OpenStack. We should strive to make all OpenStack projects use one memcached client. That being said, a quick Google search indicates that pymemcache has some benefits over python-memcache, the latter not being python 3 compatible. Additionally, pymemcache was written by the Pinterest people, so I'd imaging it stands up fairly well under stress. Perhaps we should consider porting the existing OpenStack projects from python-memcache to pymemcache. In the future, though, we need to make sure to avoid getting into a situation like this. For the present === Probably, we'll just have to get pymemcache packaged for Fedora in some form. Like you said, it can be bundled in RDO. If you're not using RDO I think it's safe to just tell people to install it from other sources (pip install) until we can get the package into Fedora. Best Regards, Solly Ross - Original Message - From: Matt Riedemann mrie...@linux.vnet.ibm.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Tuesday, September 9, 2014 4:30:02 PM Subject: [openstack-dev] global-reqs on tooz pulls in worrisome transitive dep It took me a while to untangle this so prepare for links. :) I noticed this change [1] today for global-requirements to require tooz [2] for a ceilometer blueprint [3]. The sad part is that tooz requires pymemcache [4] which is, from what I can tell, a memcached client that is not the same as python-memcached [5]. Note that python-memcached is listed in global-requirements already [6]. The problem I have with this is it doesn't appear that RHEL/Fedora package pymemcache (they do package python-memcached). I see that openSUSE builds separate packages for each. It looks like Ubuntu also has separate packages. My question is, is this a problem? I'm assuming RDO will just have to package python-pymemcache themselves but what about people not using RDO (SOL? Don't care? Other?). Reverting the requirements change would probably mean reverting the ceilometer blueprint (or getting a version of tooz out that works with python-memcached which is probably too late for that right now). Given the point in the schedule that seems pretty drastic. Maybe I'm making more of this than it's worth but wanted to bring it up in case anyone else has concerns. [1] https://review.openstack.org/#/c/93443/ [2] https://github.com/stackforge/tooz/blob/master/requirements.txt#L6 [3] http://specs.openstack.org/openstack/ceilometer-specs/specs/juno/central-agent-partitioning.html [4] https://pypi.python.org/pypi/pymemcache [5] https://pypi.python.org/pypi/python-memcached/ [6] https://github.com/openstack/requirements/blob/master/global-requirements.txt#L108 -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
