Re: [openstack-dev] [requirements] Do we care about pypy for clients (broken by cryptography)
Thanks John for the intro. I believe cryptography is supported by pypy today. Just did a "pip install cryptography" using an older version of pypy, pypy2-v5.6.0, with no error. This package has been downloaded > 13 millions of times based on pypy package tracking site: http://packages.pypy.org/##cryptography. I'd be interested in knowing the exact errors where it's failing, and help out. As John said, we've observed 2x boost in throughput, and 78% reduction in latency from the proxy node in Swift lab setups. We've also seen perf gain from Cinder, Keystone, Nova, Glance, Neutron, with most significant gains seen (22x) from Oslo.i18n from our lab setup running benchmarks such as Rally. Switching from CPython to PyPy for OpenStack is not hard. However, I found it really challenging to identify real world (customers) OpenStack examples where performance is hindered mainly on Python codes/or interpreter. Thanks, Peter -Original Message- From: John Dickinson [mailto:m...@not.mn] Sent: Wednesday, May 31, 2017 9:22 AM To: OpenStack Development Mailing List <openstack-dev@lists.openstack.org> Cc: Wang, Peter Xihong <peter.xihong.w...@intel.com> Subject: Re: [openstack-dev] [requirements] Do we care about pypy for clients (broken by cryptography) On 31 May 2017, at 5:34, Monty Taylor wrote: > On 05/31/2017 06:39 AM, Sean McGinnis wrote: >> On Wed, May 31, 2017 at 06:37:02AM -0500, Sean McGinnis wrote: >>> We had a discussion a few months back around what to do for >>> cryptography since pycrypto is basically dead [1]. After some >>> discussion, at least on the Cinder project, we decided the best way >>> forward was to use the cryptography package instead, and work has >>> been done to completely remove pycrypto usage. >>> >>> It all seemed like a good plan at the time. >>> >>> I now notice that for the python-cinderclient jobs, there is a pypy >>> job >>> (non-voting!) that is failing because the cryptography package is >>> not supported with pypy. >>> >>> So this leaves us with two options I guess. Change the cryto library >>> again, or drop support for pypy. >>> >>> I am not aware of anyone using pypy, and there are other valid >>> working alternatives. I would much rather just drop support for it >>> than redo our crypto functions again. >>> >>> Thoughts? I'm sure the Grand Champion of the Clients (Monty) >>> probably has some input? > > There was work a few years ago to get pypy support going - but it never > really seemed to catch on. The chance that we're going to start a new push > and be successful at this point seems low at best. > > I'd argue that pypy is already not supported, so dropping the non-voting job > doesn't seem like losing very much to me. Reworking cryptography libs again, > otoh, seems like a lot of work. > > Monty On the other hand, I've been working with Intel on getting PyPy support in Swift (it works, just need to reenable that gate job), and I know they were working on support in other projects. Summary is that for Swift, we got around 2x improvement (lower latency + higher throughput) by simply replacing CPython with PyPy. I believe similar gains in other projects were expected. I've cc'd Peter from Intel who's been working on PyPy quite a bit. I know he'll be very interested in this discussion and will have valuable input. --john > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] Do we care about pypy for clients (broken by cryptography)
> Just a couple things (I don't think it changes the decision made). > > Cryptography does at least claim to support PyPy (see > https://pypi.python.org/pypi/cryptography/ trove identifiers), so > possibly a bug on their end that should get filed? > > Also, our user clients should probably run under as many interpreters as > possible to make life easier for end users; however, they currently > depend on oslo and if oslo doesn't support pypy then likely not > reasonable to do in user clients. > > Clark > This looks like it may be due to the version of pypy packaged for Xenial. The pypi info for cryptography does state PyPy 5.3+ is supported. The latest package I see for 16.04 is pypy/xenial-updates 5.1.2+dfsg-1~16.04. Is there a way we can get a newer version loaded on our images? I would be fine testing against it, though the lack of oslo support does give a high risk that something else may break once we get past the cryptography issue. Sean __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] Do we care about pypy for clients (broken by cryptography)
On 31 May 2017, at 5:34, Monty Taylor wrote: > On 05/31/2017 06:39 AM, Sean McGinnis wrote: >> On Wed, May 31, 2017 at 06:37:02AM -0500, Sean McGinnis wrote: >>> We had a discussion a few months back around what to do for cryptography >>> since pycrypto is basically dead [1]. After some discussion, at least on >>> the Cinder project, we decided the best way forward was to use the >>> cryptography package instead, and work has been done to completely remove >>> pycrypto usage. >>> >>> It all seemed like a good plan at the time. >>> >>> I now notice that for the python-cinderclient jobs, there is a pypy job >>> (non-voting!) that is failing because the cryptography package is not >>> supported with pypy. >>> >>> So this leaves us with two options I guess. Change the cryto library again, >>> or drop support for pypy. >>> >>> I am not aware of anyone using pypy, and there are other valid working >>> alternatives. I would much rather just drop support for it than redo our >>> crypto functions again. >>> >>> Thoughts? I'm sure the Grand Champion of the Clients (Monty) probably has >>> some input? > > There was work a few years ago to get pypy support going - but it never > really seemed to catch on. The chance that we're going to start a new push > and be successful at this point seems low at best. > > I'd argue that pypy is already not supported, so dropping the non-voting job > doesn't seem like losing very much to me. Reworking cryptography libs again, > otoh, seems like a lot of work. > > Monty On the other hand, I've been working with Intel on getting PyPy support in Swift (it works, just need to reenable that gate job), and I know they were working on support in other projects. Summary is that for Swift, we got around 2x improvement (lower latency + higher throughput) by simply replacing CPython with PyPy. I believe similar gains in other projects were expected. I've cc'd Peter from Intel who's been working on PyPy quite a bit. I know he'll be very interested in this discussion and will have valuable input. --john > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] Do we care about pypy for clients (broken by cryptography)
On Wed, May 31, 2017, at 07:39 AM, Sean McGinnis wrote: > On Wed, May 31, 2017 at 09:47:37AM -0400, Doug Hellmann wrote: > > Excerpts from Monty Taylor's message of 2017-05-31 07:34:03 -0500: > > > On 05/31/2017 06:39 AM, Sean McGinnis wrote: > > > > On Wed, May 31, 2017 at 06:37:02AM -0500, Sean McGinnis wrote: > > > >> > > > >> I am not aware of anyone using pypy, and there are other valid working > > > >> alternatives. I would much rather just drop support for it than redo > > > >> our > > > >> crypto functions again. > > > >> > > > > > > > This question came up recently for the Oslo libraries, and I think we > > also agreed that pypy support was not being actively maintained. > > > > Doug > > > > Thanks Doug. If oslo does not support pypy, then I think that makes the > decision for me. I will put up a patch to get rid of that job and stop > wasting infra resources on it. Just a couple things (I don't think it changes the decision made). Cryptography does at least claim to support PyPy (see https://pypi.python.org/pypi/cryptography/ trove identifiers), so possibly a bug on their end that should get filed? Also, our user clients should probably run under as many interpreters as possible to make life easier for end users; however, they currently depend on oslo and if oslo doesn't support pypy then likely not reasonable to do in user clients. Clark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] Do we care about pypy for clients (broken by cryptography)
On Wed, May 31, 2017 at 09:47:37AM -0400, Doug Hellmann wrote: > Excerpts from Monty Taylor's message of 2017-05-31 07:34:03 -0500: > > On 05/31/2017 06:39 AM, Sean McGinnis wrote: > > > On Wed, May 31, 2017 at 06:37:02AM -0500, Sean McGinnis wrote: > > >> > > >> I am not aware of anyone using pypy, and there are other valid working > > >> alternatives. I would much rather just drop support for it than redo our > > >> crypto functions again. > > >> > > > > This question came up recently for the Oslo libraries, and I think we > also agreed that pypy support was not being actively maintained. > > Doug > Thanks Doug. If oslo does not support pypy, then I think that makes the decision for me. I will put up a patch to get rid of that job and stop wasting infra resources on it. I was hoping this would be the answer. ;) Sean __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] Do we care about pypy for clients (broken by cryptography)
Excerpts from Monty Taylor's message of 2017-05-31 07:34:03 -0500: > On 05/31/2017 06:39 AM, Sean McGinnis wrote: > > On Wed, May 31, 2017 at 06:37:02AM -0500, Sean McGinnis wrote: > >> We had a discussion a few months back around what to do for cryptography > >> since pycrypto is basically dead [1]. After some discussion, at least on > >> the Cinder project, we decided the best way forward was to use the > >> cryptography package instead, and work has been done to completely remove > >> pycrypto usage. > >> > >> It all seemed like a good plan at the time. > >> > >> I now notice that for the python-cinderclient jobs, there is a pypy job > >> (non-voting!) that is failing because the cryptography package is not > >> supported with pypy. > >> > >> So this leaves us with two options I guess. Change the cryto library again, > >> or drop support for pypy. > >> > >> I am not aware of anyone using pypy, and there are other valid working > >> alternatives. I would much rather just drop support for it than redo our > >> crypto functions again. > >> > >> Thoughts? I'm sure the Grand Champion of the Clients (Monty) probably has > >> some input? > > There was work a few years ago to get pypy support going - but it never > really seemed to catch on. The chance that we're going to start a new > push and be successful at this point seems low at best. > > I'd argue that pypy is already not supported, so dropping the non-voting > job doesn't seem like losing very much to me. Reworking cryptography > libs again, otoh, seems like a lot of work. > > Monty > This question came up recently for the Oslo libraries, and I think we also agreed that pypy support was not being actively maintained. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] Do we care about pypy for clients (broken by cryptography)
On 05/31/2017 06:39 AM, Sean McGinnis wrote: On Wed, May 31, 2017 at 06:37:02AM -0500, Sean McGinnis wrote: We had a discussion a few months back around what to do for cryptography since pycrypto is basically dead [1]. After some discussion, at least on the Cinder project, we decided the best way forward was to use the cryptography package instead, and work has been done to completely remove pycrypto usage. It all seemed like a good plan at the time. I now notice that for the python-cinderclient jobs, there is a pypy job (non-voting!) that is failing because the cryptography package is not supported with pypy. So this leaves us with two options I guess. Change the cryto library again, or drop support for pypy. I am not aware of anyone using pypy, and there are other valid working alternatives. I would much rather just drop support for it than redo our crypto functions again. Thoughts? I'm sure the Grand Champion of the Clients (Monty) probably has some input? There was work a few years ago to get pypy support going - but it never really seemed to catch on. The chance that we're going to start a new push and be successful at this point seems low at best. I'd argue that pypy is already not supported, so dropping the non-voting job doesn't seem like losing very much to me. Reworking cryptography libs again, otoh, seems like a lot of work. Monty __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] Do we care about pypy for clients (broken by cryptography)
On Wed, May 31, 2017 at 06:37:02AM -0500, Sean McGinnis wrote: > We had a discussion a few months back around what to do for cryptography > since pycrypto is basically dead [1]. After some discussion, at least on > the Cinder project, we decided the best way forward was to use the > cryptography package instead, and work has been done to completely remove > pycrypto usage. > > It all seemed like a good plan at the time. > > I now notice that for the python-cinderclient jobs, there is a pypy job > (non-voting!) that is failing because the cryptography package is not > supported with pypy. > > So this leaves us with two options I guess. Change the cryto library again, > or drop support for pypy. > > I am not aware of anyone using pypy, and there are other valid working > alternatives. I would much rather just drop support for it than redo our > crypto functions again. > > Thoughts? I'm sure the Grand Champion of the Clients (Monty) probably has > some input? > > Sean > That missing reference: [1] http://lists.openstack.org/pipermail/openstack-dev/2017-March/113568.html > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [requirements] Do we care about pypy for clients (broken by cryptography)
We had a discussion a few months back around what to do for cryptography since pycrypto is basically dead [1]. After some discussion, at least on the Cinder project, we decided the best way forward was to use the cryptography package instead, and work has been done to completely remove pycrypto usage. It all seemed like a good plan at the time. I now notice that for the python-cinderclient jobs, there is a pypy job (non-voting!) that is failing because the cryptography package is not supported with pypy. So this leaves us with two options I guess. Change the cryto library again, or drop support for pypy. I am not aware of anyone using pypy, and there are other valid working alternatives. I would much rather just drop support for it than redo our crypto functions again. Thoughts? I'm sure the Grand Champion of the Clients (Monty) probably has some input? Sean __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
