Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
Excerpts from Sean Dague's message of 2015-04-15 15:10:54 -0400: > On 04/15/2015 01:48 PM, Doug Hellmann wrote: > > Excerpts from Clint Byrum's message of 2015-04-15 10:15:11 -0700: > >> Excerpts from Sean Dague's message of 2015-04-14 16:54:30 -0700: > >>> > >>> It's time to be honest about the level of support that comes with those > >>> other backends, deprecate the plugability, and move on to more > >>> interesting problems. We do have plenty of them to solve. :) Perhaps in > >>> doing so we could get a better Rabbit implementation and make life > >>> easier for everyone. > >>> > >> > >> I think you're right about most of this, so +1* > >> > >> *I want to suggest that having this pluggable isn't the problem. Merging > >> drivers without integration testing and knowledgeable resources from > >> interested parties is the problem. If there isn't a well defined gate > >> test, and a team of people willing to respond to any and all issues with > >> that infrastructure committed, then the driver should not be shipped > >> with oslo.messaging. > > > > I tend to agree, although it's up to the oslo-messaging-core team to > > decide what they want to support. > > I do feel like decisions like this need to uplevel from just the library > maintainers, because it's a think that as a community we need to all be > willing to stand up and support the way OpenStack works, for some > definition of that. Otherwise we're going to get into a situation where > a lot of projects are just going to say: oh, not rabbit, go talk to > those folks instead. Nothing infuriates people more than support > telephone tag. > > I don't think that's a situation we want to put the oslo-messaging team > in, and I don't think it's the way we want to work as a community. Sure. My point was that the messaging team needs to have a say in what they support, and under what conditions. It's no more appropriate for the rest of the community to expect that we'll support everything, without help, than it is for the messaging team to decide to drop a driver without having the broader discussion. So, yes, let's have the conversation and make sure everyone is involved. Clint's spec is a good place to do that. Doug > > -Sean > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
- Original Message - > From: "Ken Giusti" > To: "OpenStack Development Mailing List (not for usage questions)" > > Sent: Thursday, April 16, 2015 4:47:50 PM > Subject: Re: [openstack-dev] [all] QPID incompatible with python 3 and > untested in gate -- what to do? > > On Wed, Apr 15, 2015 at 8:18 PM, Joshua Harlow wrote: > > Ken Giusti wrote: > >> > >> On Wed, Apr 15, 2015 at 1:33 PM, Doug Hellmann > >> wrote: > >>> > >>> Excerpts from Ken Giusti's message of 2015-04-15 09:31:18 -0400: > >>>> > >>>> On Tue, Apr 14, 2015 at 6:23 PM, Joshua Harlow > >>>> wrote: > >>>>> > >>>>> Ken Giusti wrote: > >>>>>> > >>>>>> Just to be clear: you're asking specifically about the 0-10 based > >>>>>> impl_qpid.py driver, correct? This is the driver that is used for > >>>>>> the "qpid://" transport (aka rpc_backend). > >>>>>> > >>>>>> I ask because I'm maintaining the AMQP 1.0 driver (transport > >>>>>> "amqp://") that can also be used with qpidd. > >>>>>> > >>>>>> However, the AMQP 1.0 driver isn't yet Python 3 compatible due to its > >>>>>> dependency on Proton, which has yet to be ported to python 3 - though > >>>>>> that's currently being worked on [1]. > >>>>>> > >>>>>> I'm planning on porting the AMQP 1.0 driver once the dependent > >>>>>> libraries are available. > >>>>>> > >>>>>> [1]: https://issues.apache.org/jira/browse/PROTON-490 > >>>>> > >>>>> > >>>>> What's the expected date on this as it appears this also blocks python > >>>>> 3 > >>>>> work as well... Seems like that hasn't been updated since nov 2014 > >>>>> which > >>>>> doesn't inspire that much confidence (especially for what appears to be > >>>>> mostly small patches). > >>>>> > >>>> Good point. I reached out to the bug owner. He got it 'mostly > >>>> working' but got hung up on porting the proton unit tests. I've > >>>> offered to help this along and he's good with that. I'll make this a > >>>> priority to move this along. > >>>> > >>>> In terms of availability - proton tends to do releases about every 4-6 > >>>> months. They just released 0.9, so the earliest availability would be > >>>> in that 4-6 month window (assuming that should be enough time to > >>>> complete the work). Then there's the time it will take for the > >>>> various distros to pick it up... > >>>> > >>>> so, definitely not 'real soon now'. :( > >>> > >>> This seems like a case where if we can get the libs we need to a point > >>> where they install via pip, we can let the distros catch up instead of > >>> waiting for them. > >>> > >> > >> Sadly just the python wrappers are available via pip. Its C extension > >> requires that the native proton shared library (libqpid-proton) is > >> available. To date we've relied on the distro to provide that > >> library. > > > > > > How does that (c extension) work with eventlet? Does it? > > > > I haven't experienced any issues in my testing. > > To be clear - the libqpid-proton library is non-blocking and > non-threading. It's simply an protocol processing engine - the driver > hands it raw network data and messages magically pop out (and vise > versa). > > All I/O, blocking, threading etc is done in the python driver itself. > I suspect there's nothing eventlet needs to do that requires > overloading functionality provided by the binary proton library, but > my knowledge of eventlet is pretty slim. > Usually to make a C code I/O lib eventlet friendly you need to use python sockets. It is possible to explicitly use python socket on the C code. For. Ex.: https://github.com/esnme/ultramysql/blob/master/python/io_cpython.c#L100 If the driver using python sockets and just passing the data to the C code as you said, it is fine! > > > >> > >>> Similarly, if we have *an* approach for Python 3 on oslo.messaging, that > >>> means
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On Wed, Apr 15, 2015 at 8:18 PM, Joshua Harlow wrote: > Ken Giusti wrote: >> >> On Wed, Apr 15, 2015 at 1:33 PM, Doug Hellmann >> wrote: >>> >>> Excerpts from Ken Giusti's message of 2015-04-15 09:31:18 -0400: On Tue, Apr 14, 2015 at 6:23 PM, Joshua Harlow wrote: > > Ken Giusti wrote: >> >> Just to be clear: you're asking specifically about the 0-10 based >> impl_qpid.py driver, correct? This is the driver that is used for >> the "qpid://" transport (aka rpc_backend). >> >> I ask because I'm maintaining the AMQP 1.0 driver (transport >> "amqp://") that can also be used with qpidd. >> >> However, the AMQP 1.0 driver isn't yet Python 3 compatible due to its >> dependency on Proton, which has yet to be ported to python 3 - though >> that's currently being worked on [1]. >> >> I'm planning on porting the AMQP 1.0 driver once the dependent >> libraries are available. >> >> [1]: https://issues.apache.org/jira/browse/PROTON-490 > > > What's the expected date on this as it appears this also blocks python > 3 > work as well... Seems like that hasn't been updated since nov 2014 > which > doesn't inspire that much confidence (especially for what appears to be > mostly small patches). > Good point. I reached out to the bug owner. He got it 'mostly working' but got hung up on porting the proton unit tests. I've offered to help this along and he's good with that. I'll make this a priority to move this along. In terms of availability - proton tends to do releases about every 4-6 months. They just released 0.9, so the earliest availability would be in that 4-6 month window (assuming that should be enough time to complete the work). Then there's the time it will take for the various distros to pick it up... so, definitely not 'real soon now'. :( >>> >>> This seems like a case where if we can get the libs we need to a point >>> where they install via pip, we can let the distros catch up instead of >>> waiting for them. >>> >> >> Sadly just the python wrappers are available via pip. Its C extension >> requires that the native proton shared library (libqpid-proton) is >> available. To date we've relied on the distro to provide that >> library. > > > How does that (c extension) work with eventlet? Does it? > I haven't experienced any issues in my testing. To be clear - the libqpid-proton library is non-blocking and non-threading. It's simply an protocol processing engine - the driver hands it raw network data and messages magically pop out (and vise versa). All I/O, blocking, threading etc is done in the python driver itself. I suspect there's nothing eventlet needs to do that requires overloading functionality provided by the binary proton library, but my knowledge of eventlet is pretty slim. > >> >>> Similarly, if we have *an* approach for Python 3 on oslo.messaging, that >>> means the library isn't blocking us from testing applications with >>> Python 3. If some of the drivers lag, their test jobs may need to be >>> removed or disabled if the apps start testing under Python 3. >>> >>> Doug >>> >>> >>> __ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> >> > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Ken Giusti (kgiu...@gmail.com) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On 15/04/15 09:09 -0700, Joshua Harlow wrote: Flavio Percoco wrote: On 14/04/15 19:54 -0400, Sean Dague wrote: On 04/14/2015 07:26 PM, Flavio Percoco wrote: On 14/04/15 23:18 +, Jeremy Stanley wrote: On 2015-04-15 01:10:03 +0200 (+0200), Flavio Percoco wrote: [...] I'd recommend sending this email to the ops mailing list And I'd recommend subscribing to it... it's really quite good! He did (twice apparently, I expect the second by mistake): http://lists.openstack.org/pipermail/openstack-operators/2015-April/006735.html It'd have been useful to have this linked in this thread... and the users mailing list too. [...] The general mailing list seems a little less focused on this sort of thing, but I suppose it can't hurt. I disagree, they are still users and we get feedback from them. There is a problem with sending out an "is anyone using this?" email and deciding whether or not to do this based on that. You're always going to find a few voices that pop up. We've gotten a ton of feedback from operators, both via survey, and meetups. And the answer is that they are all running Rabbit. Many have tried to run one of the other backends because of Rabbit bugs, and have largely found them worse, and moved back. The operator community has gathered around this backend. Even though it's got it's issues, there are best practices that people have come to develop in dealing with them. Making this pluggable doesn't provide a service to our users, because it doesn't make it clear that there is 1 backend you'll get help from others with, and the rest, well you are pretty much on your own, good luck, and you get to keep all the parts. Writing a "seemingly correct" driver for oslo.messaging doesn't mean that it's seen the kind of field abuse that's really needed to work out where the hard bugs are. It's time to be honest about the level of support that comes with those other backends, deprecate the plugability, and move on to more interesting problems. We do have plenty of them to solve. :) Perhaps in doing so we could get a better Rabbit implementation and make life easier for everyone. The only reason I proposed to move it in a separate repo is to provide sort of a deprecation path that won't block our work towards Py3K. In the stripped part of my previous email I also mentioned that we could mark it as deprecated to make clear what our intentions going forward are. I don't agree that "just killing it" is the right thing to do here. Doing this will give us a bit more work to do since we'll have to go through the repo creation process but at least we don't risk to be blamed for killing people's deployments out of the blue. If it isn't working in the gate and/or maintained, exactly whose deployments are working (for some definition of working) to 'kill' in the first place? Seems like it has to work at least in the gate for people to have deployments that work in the first place, otherwise they likely don't have deployments that could be 'killed' in the first place, right? The problem bein raised in this thread is that it doesn't work with py3k and we want to move forward to have a fully py3k compatible oslo.messaging. Therefore, we need to either fix the non-py3k compatible drivers or pull them out of the code base. I don't think anyone said it's not working in the gate and we're not 100% sure it's not being used. The proposal is to not kill it now but later. Therefore, pull it out of the repo, let people know it's going to be killed and then kill it. Flavio /me not saying that we should 'kill' it as the best way, just if it doesn't work then it doesn't seem to do much harm to 'kill' it... Cheers, Flavio __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco pgpacGX6SOmGB.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
Ken Giusti wrote: On Wed, Apr 15, 2015 at 1:33 PM, Doug Hellmann wrote: Excerpts from Ken Giusti's message of 2015-04-15 09:31:18 -0400: On Tue, Apr 14, 2015 at 6:23 PM, Joshua Harlow wrote: Ken Giusti wrote: Just to be clear: you're asking specifically about the 0-10 based impl_qpid.py driver, correct? This is the driver that is used for the "qpid://" transport (aka rpc_backend). I ask because I'm maintaining the AMQP 1.0 driver (transport "amqp://") that can also be used with qpidd. However, the AMQP 1.0 driver isn't yet Python 3 compatible due to its dependency on Proton, which has yet to be ported to python 3 - though that's currently being worked on [1]. I'm planning on porting the AMQP 1.0 driver once the dependent libraries are available. [1]: https://issues.apache.org/jira/browse/PROTON-490 What's the expected date on this as it appears this also blocks python 3 work as well... Seems like that hasn't been updated since nov 2014 which doesn't inspire that much confidence (especially for what appears to be mostly small patches). Good point. I reached out to the bug owner. He got it 'mostly working' but got hung up on porting the proton unit tests. I've offered to help this along and he's good with that. I'll make this a priority to move this along. In terms of availability - proton tends to do releases about every 4-6 months. They just released 0.9, so the earliest availability would be in that 4-6 month window (assuming that should be enough time to complete the work). Then there's the time it will take for the various distros to pick it up... so, definitely not 'real soon now'. :( This seems like a case where if we can get the libs we need to a point where they install via pip, we can let the distros catch up instead of waiting for them. Sadly just the python wrappers are available via pip. Its C extension requires that the native proton shared library (libqpid-proton) is available. To date we've relied on the distro to provide that library. How does that (c extension) work with eventlet? Does it? Similarly, if we have *an* approach for Python 3 on oslo.messaging, that means the library isn't blocking us from testing applications with Python 3. If some of the drivers lag, their test jobs may need to be removed or disabled if the apps start testing under Python 3. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
As a result of all of the excellent discussions here, I have produced a spec for policy which I think will assist us all in moving forward on improving the quality in this space: https://review.openstack.org/174105 Thanks everyone for your excellent replies! Excerpts from Clint Byrum's message of 2015-04-14 10:22:56 -0700: > Hello! There's been some recent progress on python3 compatibility for > core libraries that OpenStack depends on[1], and this is likely to open > the flood gates for even more python3 problems to be found and fixed. > > Recently a proposal was made to make oslo.messaging start to run python3 > tests[2], and it was found that qpid-python is not python3 compatible yet. > > This presents us with questions: Is anyone using QPID, and if so, should > we add gate testing for it? If not, can we deprecate the driver? In the > most recent survey results I could find [3] I don't even see message > broker mentioned, whereas Databases in use do vary somewhat. > > Currently it would appear that only oslo.messaging runs functional tests > against QPID. I was unable to locate integration testing for it, but I > may not know all of the places to dig around to find that. > > So, please let us know if QPID is important to you. Otherwise it may be > time to unburden ourselves of its maintenance. > > [1] https://pypi.python.org/pypi/eventlet/0.17.3 > [2] https://review.openstack.org/#/c/172135/ > [3] > http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On 04/15/2015 01:48 PM, Doug Hellmann wrote: > Excerpts from Clint Byrum's message of 2015-04-15 10:15:11 -0700: >> Excerpts from Sean Dague's message of 2015-04-14 16:54:30 -0700: >>> >>> It's time to be honest about the level of support that comes with those >>> other backends, deprecate the plugability, and move on to more >>> interesting problems. We do have plenty of them to solve. :) Perhaps in >>> doing so we could get a better Rabbit implementation and make life >>> easier for everyone. >>> >> >> I think you're right about most of this, so +1* >> >> *I want to suggest that having this pluggable isn't the problem. Merging >> drivers without integration testing and knowledgeable resources from >> interested parties is the problem. If there isn't a well defined gate >> test, and a team of people willing to respond to any and all issues with >> that infrastructure committed, then the driver should not be shipped >> with oslo.messaging. > > I tend to agree, although it's up to the oslo-messaging-core team to > decide what they want to support. I do feel like decisions like this need to uplevel from just the library maintainers, because it's a think that as a community we need to all be willing to stand up and support the way OpenStack works, for some definition of that. Otherwise we're going to get into a situation where a lot of projects are just going to say: oh, not rabbit, go talk to those folks instead. Nothing infuriates people more than support telephone tag. I don't think that's a situation we want to put the oslo-messaging team in, and I don't think it's the way we want to work as a community. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
Excerpts from Ken Giusti's message of 2015-04-15 14:08:57 -0400: > On Wed, Apr 15, 2015 at 1:33 PM, Doug Hellmann wrote: > > Excerpts from Ken Giusti's message of 2015-04-15 09:31:18 -0400: > >> On Tue, Apr 14, 2015 at 6:23 PM, Joshua Harlow > >> wrote: > >> > Ken Giusti wrote: > >> >> > >> >> Just to be clear: you're asking specifically about the 0-10 based > >> >> impl_qpid.py driver, correct? This is the driver that is used for > >> >> the "qpid://" transport (aka rpc_backend). > >> >> > >> >> I ask because I'm maintaining the AMQP 1.0 driver (transport > >> >> "amqp://") that can also be used with qpidd. > >> >> > >> >> However, the AMQP 1.0 driver isn't yet Python 3 compatible due to its > >> >> dependency on Proton, which has yet to be ported to python 3 - though > >> >> that's currently being worked on [1]. > >> >> > >> >> I'm planning on porting the AMQP 1.0 driver once the dependent > >> >> libraries are available. > >> >> > >> >> [1]: https://issues.apache.org/jira/browse/PROTON-490 > >> > > >> > > >> > What's the expected date on this as it appears this also blocks python 3 > >> > work as well... Seems like that hasn't been updated since nov 2014 which > >> > doesn't inspire that much confidence (especially for what appears to be > >> > mostly small patches). > >> > > >> > >> Good point. I reached out to the bug owner. He got it 'mostly > >> working' but got hung up on porting the proton unit tests. I've > >> offered to help this along and he's good with that. I'll make this a > >> priority to move this along. > >> > >> In terms of availability - proton tends to do releases about every 4-6 > >> months. They just released 0.9, so the earliest availability would be > >> in that 4-6 month window (assuming that should be enough time to > >> complete the work). Then there's the time it will take for the > >> various distros to pick it up... > >> > >> so, definitely not 'real soon now'. :( > > > > This seems like a case where if we can get the libs we need to a point > > where they install via pip, we can let the distros catch up instead of > > waiting for them. > > > > Sadly just the python wrappers are available via pip. Its C extension > requires that the native proton shared library (libqpid-proton) is > available. To date we've relied on the distro to provide that > library. OK, that may pose more of a problem. It is possible to put C extensions into a Python library and make them pip installable, so that might be our path out. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
Excerpts from Clint Byrum's message of 2015-04-15 11:02:34 -0700: > Excerpts from Doug Hellmann's message of 2015-04-15 10:48:30 -0700: > > Excerpts from Clint Byrum's message of 2015-04-15 10:15:11 -0700: > > > Excerpts from Sean Dague's message of 2015-04-14 16:54:30 -0700: > > > > > > > > It's time to be honest about the level of support that comes with those > > > > other backends, deprecate the plugability, and move on to more > > > > interesting problems. We do have plenty of them to solve. :) Perhaps in > > > > doing so we could get a better Rabbit implementation and make life > > > > easier for everyone. > > > > > > > > > > I think you're right about most of this, so +1* > > > > > > *I want to suggest that having this pluggable isn't the problem. Merging > > > drivers without integration testing and knowledgeable resources from > > > interested parties is the problem. If there isn't a well defined gate > > > test, and a team of people willing to respond to any and all issues with > > > that infrastructure committed, then the driver should not be shipped > > > with oslo.messaging. > > > > I tend to agree, although it's up to the oslo-messaging-core team to > > decide what they want to support. > > > > A general note on these sorts of conversations: > > > > It's very easy to look at the state of OpenStack testing now and > > say, "we must have integration test jobs for oslo.messaging!" Don't > > forget that most of the work in this repo came out of Nova at a > > time when there was no such thing, and we've only just settled on > > good processes for managing third-party testing of that sort in > > Nova, Cinder, and Neutron. We've been watching that work with > > interest, but given the small size of the team currently maintaining > > the library, it wasn't necessarily the highest priority. > > > > That said, I know Mehdi and others have been working on setting up > > integration test jobs, and I expect that at some point in the > > not-too-distant future we'll need to discuss putting a rule into > > place for these drivers just like the other projects have for their > > drivers. We don't yet have a sufficiently strong test suite to do > > that, though, so requiring test jobs now would be premature. > > Great points Doug. > > A devstack-gate job that is pointed at the major consumers of > oslo.messaging would be enough I think. The library sits at the core > of nearly everything, so I don't think we necessarily need to have > a split-out gate that just tests oslo.messaging narrowly with each > backend. Somewhere we would need _something_ in the gate using a > particular driver to be able to say that one should use it. > Sure, that makes sense. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On Wed, Apr 15, 2015 at 1:33 PM, Doug Hellmann wrote: > Excerpts from Ken Giusti's message of 2015-04-15 09:31:18 -0400: >> On Tue, Apr 14, 2015 at 6:23 PM, Joshua Harlow wrote: >> > Ken Giusti wrote: >> >> >> >> Just to be clear: you're asking specifically about the 0-10 based >> >> impl_qpid.py driver, correct? This is the driver that is used for >> >> the "qpid://" transport (aka rpc_backend). >> >> >> >> I ask because I'm maintaining the AMQP 1.0 driver (transport >> >> "amqp://") that can also be used with qpidd. >> >> >> >> However, the AMQP 1.0 driver isn't yet Python 3 compatible due to its >> >> dependency on Proton, which has yet to be ported to python 3 - though >> >> that's currently being worked on [1]. >> >> >> >> I'm planning on porting the AMQP 1.0 driver once the dependent >> >> libraries are available. >> >> >> >> [1]: https://issues.apache.org/jira/browse/PROTON-490 >> > >> > >> > What's the expected date on this as it appears this also blocks python 3 >> > work as well... Seems like that hasn't been updated since nov 2014 which >> > doesn't inspire that much confidence (especially for what appears to be >> > mostly small patches). >> > >> >> Good point. I reached out to the bug owner. He got it 'mostly >> working' but got hung up on porting the proton unit tests. I've >> offered to help this along and he's good with that. I'll make this a >> priority to move this along. >> >> In terms of availability - proton tends to do releases about every 4-6 >> months. They just released 0.9, so the earliest availability would be >> in that 4-6 month window (assuming that should be enough time to >> complete the work). Then there's the time it will take for the >> various distros to pick it up... >> >> so, definitely not 'real soon now'. :( > > This seems like a case where if we can get the libs we need to a point > where they install via pip, we can let the distros catch up instead of > waiting for them. > Sadly just the python wrappers are available via pip. Its C extension requires that the native proton shared library (libqpid-proton) is available. To date we've relied on the distro to provide that library. > Similarly, if we have *an* approach for Python 3 on oslo.messaging, that > means the library isn't blocking us from testing applications with > Python 3. If some of the drivers lag, their test jobs may need to be > removed or disabled if the apps start testing under Python 3. > > Doug > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Ken Giusti (kgiu...@gmail.com) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
Excerpts from Doug Hellmann's message of 2015-04-15 10:48:30 -0700: > Excerpts from Clint Byrum's message of 2015-04-15 10:15:11 -0700: > > Excerpts from Sean Dague's message of 2015-04-14 16:54:30 -0700: > > > > > > It's time to be honest about the level of support that comes with those > > > other backends, deprecate the plugability, and move on to more > > > interesting problems. We do have plenty of them to solve. :) Perhaps in > > > doing so we could get a better Rabbit implementation and make life > > > easier for everyone. > > > > > > > I think you're right about most of this, so +1* > > > > *I want to suggest that having this pluggable isn't the problem. Merging > > drivers without integration testing and knowledgeable resources from > > interested parties is the problem. If there isn't a well defined gate > > test, and a team of people willing to respond to any and all issues with > > that infrastructure committed, then the driver should not be shipped > > with oslo.messaging. > > I tend to agree, although it's up to the oslo-messaging-core team to > decide what they want to support. > > A general note on these sorts of conversations: > > It's very easy to look at the state of OpenStack testing now and > say, "we must have integration test jobs for oslo.messaging!" Don't > forget that most of the work in this repo came out of Nova at a > time when there was no such thing, and we've only just settled on > good processes for managing third-party testing of that sort in > Nova, Cinder, and Neutron. We've been watching that work with > interest, but given the small size of the team currently maintaining > the library, it wasn't necessarily the highest priority. > > That said, I know Mehdi and others have been working on setting up > integration test jobs, and I expect that at some point in the > not-too-distant future we'll need to discuss putting a rule into > place for these drivers just like the other projects have for their > drivers. We don't yet have a sufficiently strong test suite to do > that, though, so requiring test jobs now would be premature. Great points Doug. A devstack-gate job that is pointed at the major consumers of oslo.messaging would be enough I think. The library sits at the core of nearly everything, so I don't think we necessarily need to have a split-out gate that just tests oslo.messaging narrowly with each backend. Somewhere we would need _something_ in the gate using a particular driver to be able to say that one should use it. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On 04/14/2015 08:21 PM, Chris Dent wrote: > On Tue, 14 Apr 2015, Sean Dague wrote: > >> It's time to be honest about the level of support that comes with those >> other backends, deprecate the plugability, and move on to more >> interesting problems. We do have plenty of them to solve. :) Perhaps in >> doing so we could get a better Rabbit implementation and make life >> easier for everyone. > > Etch this in stone about all kinds of pluggability. +1000 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
Excerpts from Clint Byrum's message of 2015-04-15 10:15:11 -0700: > Excerpts from Sean Dague's message of 2015-04-14 16:54:30 -0700: > > > > It's time to be honest about the level of support that comes with those > > other backends, deprecate the plugability, and move on to more > > interesting problems. We do have plenty of them to solve. :) Perhaps in > > doing so we could get a better Rabbit implementation and make life > > easier for everyone. > > > > I think you're right about most of this, so +1* > > *I want to suggest that having this pluggable isn't the problem. Merging > drivers without integration testing and knowledgeable resources from > interested parties is the problem. If there isn't a well defined gate > test, and a team of people willing to respond to any and all issues with > that infrastructure committed, then the driver should not be shipped > with oslo.messaging. I tend to agree, although it's up to the oslo-messaging-core team to decide what they want to support. A general note on these sorts of conversations: It's very easy to look at the state of OpenStack testing now and say, "we must have integration test jobs for oslo.messaging!" Don't forget that most of the work in this repo came out of Nova at a time when there was no such thing, and we've only just settled on good processes for managing third-party testing of that sort in Nova, Cinder, and Neutron. We've been watching that work with interest, but given the small size of the team currently maintaining the library, it wasn't necessarily the highest priority. That said, I know Mehdi and others have been working on setting up integration test jobs, and I expect that at some point in the not-too-distant future we'll need to discuss putting a rule into place for these drivers just like the other projects have for their drivers. We don't yet have a sufficiently strong test suite to do that, though, so requiring test jobs now would be premature. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
Excerpts from Ken Giusti's message of 2015-04-15 09:31:18 -0400: > On Tue, Apr 14, 2015 at 6:23 PM, Joshua Harlow wrote: > > Ken Giusti wrote: > >> > >> Just to be clear: you're asking specifically about the 0-10 based > >> impl_qpid.py driver, correct? This is the driver that is used for > >> the "qpid://" transport (aka rpc_backend). > >> > >> I ask because I'm maintaining the AMQP 1.0 driver (transport > >> "amqp://") that can also be used with qpidd. > >> > >> However, the AMQP 1.0 driver isn't yet Python 3 compatible due to its > >> dependency on Proton, which has yet to be ported to python 3 - though > >> that's currently being worked on [1]. > >> > >> I'm planning on porting the AMQP 1.0 driver once the dependent > >> libraries are available. > >> > >> [1]: https://issues.apache.org/jira/browse/PROTON-490 > > > > > > What's the expected date on this as it appears this also blocks python 3 > > work as well... Seems like that hasn't been updated since nov 2014 which > > doesn't inspire that much confidence (especially for what appears to be > > mostly small patches). > > > > Good point. I reached out to the bug owner. He got it 'mostly > working' but got hung up on porting the proton unit tests. I've > offered to help this along and he's good with that. I'll make this a > priority to move this along. > > In terms of availability - proton tends to do releases about every 4-6 > months. They just released 0.9, so the earliest availability would be > in that 4-6 month window (assuming that should be enough time to > complete the work). Then there's the time it will take for the > various distros to pick it up... > > so, definitely not 'real soon now'. :( This seems like a case where if we can get the libs we need to a point where they install via pip, we can let the distros catch up instead of waiting for them. Similarly, if we have *an* approach for Python 3 on oslo.messaging, that means the library isn't blocking us from testing applications with Python 3. If some of the drivers lag, their test jobs may need to be removed or disabled if the apps start testing under Python 3. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
Excerpts from Sean Dague's message of 2015-04-14 16:54:30 -0700: > On 04/14/2015 07:26 PM, Flavio Percoco wrote: > > On 14/04/15 23:18 +, Jeremy Stanley wrote: > >> On 2015-04-15 01:10:03 +0200 (+0200), Flavio Percoco wrote: > >> [...] > >>> I'd recommend sending this email to the ops mailing list > >> > >> And I'd recommend subscribing to it... it's really quite good! He > >> did (twice apparently, I expect the second by mistake): > >> > >> http://lists.openstack.org/pipermail/openstack-operators/2015-April/006735.html > >> > > > > It'd have been useful to have this linked in this thread... > > > >> > >>> and the users mailing list too. > >> [...] > >> > >> The general mailing list seems a little less focused on this sort of > >> thing, but I suppose it can't hurt. > > > > I disagree, they are still users and we get feedback from them. > > There is a problem with sending out an "is anyone using this?" email and > deciding whether or not to do this based on that. You're always going to > find a few voices that pop up. > > We've gotten a ton of feedback from operators, both via survey, and > meetups. And the answer is that they are all running Rabbit. Many have > tried to run one of the other backends because of Rabbit bugs, and have > largely found them worse, and moved back. > > The operator community has gathered around this backend. Even though > it's got it's issues, there are best practices that people have come to > develop in dealing with them. Making this pluggable doesn't provide a > service to our users, because it doesn't make it clear that there is 1 > backend you'll get help from others with, and the rest, well you are > pretty much on your own, good luck, and you get to keep all the parts. > Writing a "seemingly correct" driver for oslo.messaging doesn't mean > that it's seen the kind of field abuse that's really needed to work out > where the hard bugs are. > > It's time to be honest about the level of support that comes with those > other backends, deprecate the plugability, and move on to more > interesting problems. We do have plenty of them to solve. :) Perhaps in > doing so we could get a better Rabbit implementation and make life > easier for everyone. > I think you're right about most of this, so +1* *I want to suggest that having this pluggable isn't the problem. Merging drivers without integration testing and knowledgeable resources from interested parties is the problem. If there isn't a well defined gate test, and a team of people willing to respond to any and all issues with that infrastructure committed, then the driver should not be shipped with oslo.messaging. We've been through this already with the virt drivers and databases. Having the ability to move into a space that a different backend serves well is a good feature. We *will* hit the limits of Rabbit. Encouraging users to submit every possible backend has costs though. This neighborhood has been gentrified, and it's time to evict anyone not willing or able to pay the rent. This thread has convinced me that the right path is to make an announcement, and deprecate the QPID driver as of the next release of oslo.messaging. We can always reverse that decision if users actually show up. Then the usual 2 cycle dance and we're relieved of, oddly enough, 666 SLOC: $ radon raw oslo_messaging/_drivers/impl_qpid.py oslo_messaging/_drivers/impl_qpid.py LOC: 794 LLOC: 430 SLOC: 666 Comments: 50 Multi: 72 Blank: 128 - Comment Stats (C % L): 6% (C % S): 8% (C + M % L): 15% __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
Flavio Percoco wrote: On 14/04/15 19:54 -0400, Sean Dague wrote: On 04/14/2015 07:26 PM, Flavio Percoco wrote: On 14/04/15 23:18 +, Jeremy Stanley wrote: On 2015-04-15 01:10:03 +0200 (+0200), Flavio Percoco wrote: [...] I'd recommend sending this email to the ops mailing list And I'd recommend subscribing to it... it's really quite good! He did (twice apparently, I expect the second by mistake): http://lists.openstack.org/pipermail/openstack-operators/2015-April/006735.html It'd have been useful to have this linked in this thread... and the users mailing list too. [...] The general mailing list seems a little less focused on this sort of thing, but I suppose it can't hurt. I disagree, they are still users and we get feedback from them. There is a problem with sending out an "is anyone using this?" email and deciding whether or not to do this based on that. You're always going to find a few voices that pop up. We've gotten a ton of feedback from operators, both via survey, and meetups. And the answer is that they are all running Rabbit. Many have tried to run one of the other backends because of Rabbit bugs, and have largely found them worse, and moved back. The operator community has gathered around this backend. Even though it's got it's issues, there are best practices that people have come to develop in dealing with them. Making this pluggable doesn't provide a service to our users, because it doesn't make it clear that there is 1 backend you'll get help from others with, and the rest, well you are pretty much on your own, good luck, and you get to keep all the parts. Writing a "seemingly correct" driver for oslo.messaging doesn't mean that it's seen the kind of field abuse that's really needed to work out where the hard bugs are. It's time to be honest about the level of support that comes with those other backends, deprecate the plugability, and move on to more interesting problems. We do have plenty of them to solve. :) Perhaps in doing so we could get a better Rabbit implementation and make life easier for everyone. The only reason I proposed to move it in a separate repo is to provide sort of a deprecation path that won't block our work towards Py3K. In the stripped part of my previous email I also mentioned that we could mark it as deprecated to make clear what our intentions going forward are. I don't agree that "just killing it" is the right thing to do here. Doing this will give us a bit more work to do since we'll have to go through the repo creation process but at least we don't risk to be blamed for killing people's deployments out of the blue. If it isn't working in the gate and/or maintained, exactly whose deployments are working (for some definition of working) to 'kill' in the first place? Seems like it has to work at least in the gate for people to have deployments that work in the first place, otherwise they likely don't have deployments that could be 'killed' in the first place, right? /me not saying that we should 'kill' it as the best way, just if it doesn't work then it doesn't seem to do much harm to 'kill' it... Cheers, Flavio __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On Tue, Apr 14, 2015 at 6:23 PM, Joshua Harlow wrote: > Ken Giusti wrote: >> >> Just to be clear: you're asking specifically about the 0-10 based >> impl_qpid.py driver, correct? This is the driver that is used for >> the "qpid://" transport (aka rpc_backend). >> >> I ask because I'm maintaining the AMQP 1.0 driver (transport >> "amqp://") that can also be used with qpidd. >> >> However, the AMQP 1.0 driver isn't yet Python 3 compatible due to its >> dependency on Proton, which has yet to be ported to python 3 - though >> that's currently being worked on [1]. >> >> I'm planning on porting the AMQP 1.0 driver once the dependent >> libraries are available. >> >> [1]: https://issues.apache.org/jira/browse/PROTON-490 > > > What's the expected date on this as it appears this also blocks python 3 > work as well... Seems like that hasn't been updated since nov 2014 which > doesn't inspire that much confidence (especially for what appears to be > mostly small patches). > Good point. I reached out to the bug owner. He got it 'mostly working' but got hung up on porting the proton unit tests. I've offered to help this along and he's good with that. I'll make this a priority to move this along. In terms of availability - proton tends to do releases about every 4-6 months. They just released 0.9, so the earliest availability would be in that 4-6 month window (assuming that should be enough time to complete the work). Then there's the time it will take for the various distros to pick it up... so, definitely not 'real soon now'. :( > >> >> On Tue, Apr 14, 2015 at 1:22 PM, Clint Byrum wrote: >>> >>> Hello! There's been some recent progress on python3 compatibility for >>> core libraries that OpenStack depends on[1], and this is likely to open >>> the flood gates for even more python3 problems to be found and fixed. >>> >>> Recently a proposal was made to make oslo.messaging start to run python3 >>> tests[2], and it was found that qpid-python is not python3 compatible >>> yet. >>> >>> This presents us with questions: Is anyone using QPID, and if so, should >>> we add gate testing for it? If not, can we deprecate the driver? In the >>> most recent survey results I could find [3] I don't even see message >>> broker mentioned, whereas Databases in use do vary somewhat. >>> >>> Currently it would appear that only oslo.messaging runs functional tests >>> against QPID. I was unable to locate integration testing for it, but I >>> may not know all of the places to dig around to find that. >>> >>> So, please let us know if QPID is important to you. Otherwise it may be >>> time to unburden ourselves of its maintenance. >>> >>> [1] https://pypi.python.org/pypi/eventlet/0.17.3 >>> [2] https://review.openstack.org/#/c/172135/ >>> [3] >>> http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014 >>> >>> >>> __ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> >> > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Ken Giusti (kgiu...@gmail.com) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On 2015-04-15 11:53:28 +0200 (+0200), Flavio Percoco wrote: [...] > When I proposed removing the GridFS driver from glance_store, I asked > for feedback in other mailing lists and then came back here proposing > de dev removal. Got it--so the recommendation is to not ask the developer community until the operator community has been polled for input first. I suppose I can see the logic there. We do already get plenty of E-mail on the dev ML. > The point I tried to make in my previous email is that, whenever we > propose removing something important - like support for a broker - the > broader the audience we try to get feedback from is, the better. You > can argue saying that it's very unlikely that there are ops in the > "OpenStack General mailing list" that are not in the ops m-l, but we > don't know that. [...] On the other hand, limiting these sorts of discussions to only the most appropriate venues encourages those who didn't get a say in the discussion to join the mailing lists where they take place so that they can participate more effectively in the future. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On 15/04/15 00:04 +, Jeremy Stanley wrote: On 2015-04-15 01:26:21 +0200 (+0200), Flavio Percoco wrote: On 14/04/15 23:18 +, Jeremy Stanley wrote: >On 2015-04-15 01:10:03 +0200 (+0200), Flavio Percoco wrote: >[...] >>I'd recommend sending this email to the ops mailing list > >And I'd recommend subscribing to it... it's really quite good! He >did (twice apparently, I expect the second by mistake): > >http://lists.openstack.org/pipermail/openstack-operators/2015-April/006735.html It'd have been useful to have this linked in this thread... Er, I just did? You could make the argument that he should have posted there first and then linked that thread in his post to the dev ML, or posted here first and linked the post here to the ops ML. Or are you saying he should have mentioned in both messages that he was going to post to both lists but not actually "linked" them because of the chicken-and-egg requirement there? Not going to make a big deal out of this. You came back saying I should be subscribed to that ml (which I am) and I said that a link to the ops thread in the original email would've been useful. When I proposed removing the GridFS driver from glance_store, I asked for feedback in other mailing lists and then came back here proposing de dev removal. >>and the users mailing list too. >[...] > >The general mailing list seems a little less focused on this sort of >thing, but I suppose it can't hurt. I disagree, they are still users and we get feedback from them. I'm just trying to figure out where in OpenStack any use of QPID is end-user facing. If there are users interacting with OpenStack clouds via QPID then I'm probably misunderstanding its use cases since I assumed it was entirely for inter-service communication (so would have bearing on people writing those services and people deploying them). Regardless, I don't think we have a "users mailing list" we have "The OpenStack General mailing list" which includes in its mandate "community announcements" so this might qualify, but seems excessive and at that point we might as well just post questions to every OpenStack mailing list because there might be someone using something? The point I tried to make in my previous email is that, whenever we propose removing something important - like support for a broker - the broader the audience we try to get feedback from is, the better. You can argue saying that it's very unlikely that there are ops in the "OpenStack General mailing list" that are not in the ops m-l, but we don't know that. That's it. -- @flaper87 Flavio Percoco pgpmTkhLFyWDe.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On 14/04/15 19:54 -0400, Sean Dague wrote: On 04/14/2015 07:26 PM, Flavio Percoco wrote: On 14/04/15 23:18 +, Jeremy Stanley wrote: On 2015-04-15 01:10:03 +0200 (+0200), Flavio Percoco wrote: [...] I'd recommend sending this email to the ops mailing list And I'd recommend subscribing to it... it's really quite good! He did (twice apparently, I expect the second by mistake): http://lists.openstack.org/pipermail/openstack-operators/2015-April/006735.html It'd have been useful to have this linked in this thread... and the users mailing list too. [...] The general mailing list seems a little less focused on this sort of thing, but I suppose it can't hurt. I disagree, they are still users and we get feedback from them. There is a problem with sending out an "is anyone using this?" email and deciding whether or not to do this based on that. You're always going to find a few voices that pop up. We've gotten a ton of feedback from operators, both via survey, and meetups. And the answer is that they are all running Rabbit. Many have tried to run one of the other backends because of Rabbit bugs, and have largely found them worse, and moved back. The operator community has gathered around this backend. Even though it's got it's issues, there are best practices that people have come to develop in dealing with them. Making this pluggable doesn't provide a service to our users, because it doesn't make it clear that there is 1 backend you'll get help from others with, and the rest, well you are pretty much on your own, good luck, and you get to keep all the parts. Writing a "seemingly correct" driver for oslo.messaging doesn't mean that it's seen the kind of field abuse that's really needed to work out where the hard bugs are. It's time to be honest about the level of support that comes with those other backends, deprecate the plugability, and move on to more interesting problems. We do have plenty of them to solve. :) Perhaps in doing so we could get a better Rabbit implementation and make life easier for everyone. The only reason I proposed to move it in a separate repo is to provide sort of a deprecation path that won't block our work towards Py3K. In the stripped part of my previous email I also mentioned that we could mark it as deprecated to make clear what our intentions going forward are. I don't agree that "just killing it" is the right thing to do here. Doing this will give us a bit more work to do since we'll have to go through the repo creation process but at least we don't risk to be blamed for killing people's deployments out of the blue. Cheers, Flavio -- @flaper87 Flavio Percoco pgpTJCV0xh7L4.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On Tue, 14 Apr 2015, Sean Dague wrote: It's time to be honest about the level of support that comes with those other backends, deprecate the plugability, and move on to more interesting problems. We do have plenty of them to solve. :) Perhaps in doing so we could get a better Rabbit implementation and make life easier for everyone. Etch this in stone about all kinds of pluggability. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On 2015-04-15 01:26:21 +0200 (+0200), Flavio Percoco wrote: > On 14/04/15 23:18 +, Jeremy Stanley wrote: > >On 2015-04-15 01:10:03 +0200 (+0200), Flavio Percoco wrote: > >[...] > >>I'd recommend sending this email to the ops mailing list > > > >And I'd recommend subscribing to it... it's really quite good! He > >did (twice apparently, I expect the second by mistake): > > > >http://lists.openstack.org/pipermail/openstack-operators/2015-April/006735.html > > It'd have been useful to have this linked in this thread... Er, I just did? You could make the argument that he should have posted there first and then linked that thread in his post to the dev ML, or posted here first and linked the post here to the ops ML. Or are you saying he should have mentioned in both messages that he was going to post to both lists but not actually "linked" them because of the chicken-and-egg requirement there? > >>and the users mailing list too. > >[...] > > > >The general mailing list seems a little less focused on this sort of > >thing, but I suppose it can't hurt. > > I disagree, they are still users and we get feedback from them. I'm just trying to figure out where in OpenStack any use of QPID is end-user facing. If there are users interacting with OpenStack clouds via QPID then I'm probably misunderstanding its use cases since I assumed it was entirely for inter-service communication (so would have bearing on people writing those services and people deploying them). Regardless, I don't think we have a "users mailing list" we have "The OpenStack General mailing list" which includes in its mandate "community announcements" so this might qualify, but seems excessive and at that point we might as well just post questions to every OpenStack mailing list because there might be someone using something? -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On 04/14/2015 07:26 PM, Flavio Percoco wrote: > On 14/04/15 23:18 +, Jeremy Stanley wrote: >> On 2015-04-15 01:10:03 +0200 (+0200), Flavio Percoco wrote: >> [...] >>> I'd recommend sending this email to the ops mailing list >> >> And I'd recommend subscribing to it... it's really quite good! He >> did (twice apparently, I expect the second by mistake): >> >> http://lists.openstack.org/pipermail/openstack-operators/2015-April/006735.html >> > > It'd have been useful to have this linked in this thread... > >> >>> and the users mailing list too. >> [...] >> >> The general mailing list seems a little less focused on this sort of >> thing, but I suppose it can't hurt. > > I disagree, they are still users and we get feedback from them. There is a problem with sending out an "is anyone using this?" email and deciding whether or not to do this based on that. You're always going to find a few voices that pop up. We've gotten a ton of feedback from operators, both via survey, and meetups. And the answer is that they are all running Rabbit. Many have tried to run one of the other backends because of Rabbit bugs, and have largely found them worse, and moved back. The operator community has gathered around this backend. Even though it's got it's issues, there are best practices that people have come to develop in dealing with them. Making this pluggable doesn't provide a service to our users, because it doesn't make it clear that there is 1 backend you'll get help from others with, and the rest, well you are pretty much on your own, good luck, and you get to keep all the parts. Writing a "seemingly correct" driver for oslo.messaging doesn't mean that it's seen the kind of field abuse that's really needed to work out where the hard bugs are. It's time to be honest about the level of support that comes with those other backends, deprecate the plugability, and move on to more interesting problems. We do have plenty of them to solve. :) Perhaps in doing so we could get a better Rabbit implementation and make life easier for everyone. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On 14/04/15 23:18 +, Jeremy Stanley wrote: On 2015-04-15 01:10:03 +0200 (+0200), Flavio Percoco wrote: [...] I'd recommend sending this email to the ops mailing list And I'd recommend subscribing to it... it's really quite good! He did (twice apparently, I expect the second by mistake): http://lists.openstack.org/pipermail/openstack-operators/2015-April/006735.html It'd have been useful to have this linked in this thread... and the users mailing list too. [...] The general mailing list seems a little less focused on this sort of thing, but I suppose it can't hurt. I disagree, they are still users and we get feedback from them. -- @flaper87 Flavio Percoco pgpzpbC7dEIoQ.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On 2015-04-15 01:10:03 +0200 (+0200), Flavio Percoco wrote: [...] > I'd recommend sending this email to the ops mailing list And I'd recommend subscribing to it... it's really quite good! He did (twice apparently, I expect the second by mistake): http://lists.openstack.org/pipermail/openstack-operators/2015-April/006735.html > and the users mailing list too. [...] The general mailing list seems a little less focused on this sort of thing, but I suppose it can't hurt. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On 14/04/15 10:22 -0700, Clint Byrum wrote: Hello! There's been some recent progress on python3 compatibility for core libraries that OpenStack depends on[1], and this is likely to open the flood gates for even more python3 problems to be found and fixed. Recently a proposal was made to make oslo.messaging start to run python3 tests[2], and it was found that qpid-python is not python3 compatible yet. This presents us with questions: Is anyone using QPID, and if so, should we add gate testing for it? If not, can we deprecate the driver? In the most recent survey results I could find [3] I don't even see message broker mentioned, whereas Databases in use do vary somewhat. Currently it would appear that only oslo.messaging runs functional tests against QPID. I was unable to locate integration testing for it, but I may not know all of the places to dig around to find that. So, please let us know if QPID is important to you. Otherwise it may be time to unburden ourselves of its maintenance. [1] https://pypi.python.org/pypi/eventlet/0.17.3 [2] https://review.openstack.org/#/c/172135/ [3] http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014 My understanding is that there are deployments using QPID. I'd recommend sending this email to the ops mailing list and the users mailing list too. This should give it more visibility. That said, in case suppor tfor python 3 is not done in reasonable times, I'd recommend moving this driver into its own project instead of just killing it. If necessary, we can also mark it as un-maintained. This will allow for us to keep working on the py3k support and it'll also give time to people for migrating to a different messaging solution. Cheers, Flavio -- @flaper87 Flavio Percoco pgp2XMDwYCaEy.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
rdo packstack deploys rabbit these days too. Thanks, Kevin From: Clint Byrum [cl...@fewbar.com] Sent: Tuesday, April 14, 2015 3:20 PM To: openstack-dev Subject: Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do? Excerpts from Matt Riedemann's message of 2015-04-14 13:13:47 -0700: > > On 4/14/2015 12:22 PM, Clint Byrum wrote: > > Hello! There's been some recent progress on python3 compatibility for > > core libraries that OpenStack depends on[1], and this is likely to open > > the flood gates for even more python3 problems to be found and fixed. > > > > Recently a proposal was made to make oslo.messaging start to run python3 > > tests[2], and it was found that qpid-python is not python3 compatible yet. > > > > This presents us with questions: Is anyone using QPID, and if so, should > > we add gate testing for it? If not, can we deprecate the driver? In the > > most recent survey results I could find [3] I don't even see message > > broker mentioned, whereas Databases in use do vary somewhat. > > > > Currently it would appear that only oslo.messaging runs functional tests > > against QPID. I was unable to locate integration testing for it, but I > > may not know all of the places to dig around to find that. > > > > So, please let us know if QPID is important to you. Otherwise it may be > > time to unburden ourselves of its maintenance. > > > > [1] https://pypi.python.org/pypi/eventlet/0.17.3 > > [2] https://review.openstack.org/#/c/172135/ > > [3] > > http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014 > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > FWIW IBM Cloud Manager with OpenStack is still shipping qpid 0.30. We > switched to the default deployment being RabbitMQ in Kilo though (maybe > even Juno). But we do have a support matrix tested with qpid as the rpc > backend. Our mainline paths are tested with rabbitmq though since > that's the default backend for us now. > So, I think we can count this as another point toward removing impl_qpid. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
Ken Giusti wrote: Just to be clear: you're asking specifically about the 0-10 based impl_qpid.py driver, correct? This is the driver that is used for the "qpid://" transport (aka rpc_backend). I ask because I'm maintaining the AMQP 1.0 driver (transport "amqp://") that can also be used with qpidd. However, the AMQP 1.0 driver isn't yet Python 3 compatible due to its dependency on Proton, which has yet to be ported to python 3 - though that's currently being worked on [1]. I'm planning on porting the AMQP 1.0 driver once the dependent libraries are available. [1]: https://issues.apache.org/jira/browse/PROTON-490 What's the expected date on this as it appears this also blocks python 3 work as well... Seems like that hasn't been updated since nov 2014 which doesn't inspire that much confidence (especially for what appears to be mostly small patches). On Tue, Apr 14, 2015 at 1:22 PM, Clint Byrum wrote: Hello! There's been some recent progress on python3 compatibility for core libraries that OpenStack depends on[1], and this is likely to open the flood gates for even more python3 problems to be found and fixed. Recently a proposal was made to make oslo.messaging start to run python3 tests[2], and it was found that qpid-python is not python3 compatible yet. This presents us with questions: Is anyone using QPID, and if so, should we add gate testing for it? If not, can we deprecate the driver? In the most recent survey results I could find [3] I don't even see message broker mentioned, whereas Databases in use do vary somewhat. Currently it would appear that only oslo.messaging runs functional tests against QPID. I was unable to locate integration testing for it, but I may not know all of the places to dig around to find that. So, please let us know if QPID is important to you. Otherwise it may be time to unburden ourselves of its maintenance. [1] https://pypi.python.org/pypi/eventlet/0.17.3 [2] https://review.openstack.org/#/c/172135/ [3] http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
Excerpts from Matt Riedemann's message of 2015-04-14 13:13:47 -0700: > > On 4/14/2015 12:22 PM, Clint Byrum wrote: > > Hello! There's been some recent progress on python3 compatibility for > > core libraries that OpenStack depends on[1], and this is likely to open > > the flood gates for even more python3 problems to be found and fixed. > > > > Recently a proposal was made to make oslo.messaging start to run python3 > > tests[2], and it was found that qpid-python is not python3 compatible yet. > > > > This presents us with questions: Is anyone using QPID, and if so, should > > we add gate testing for it? If not, can we deprecate the driver? In the > > most recent survey results I could find [3] I don't even see message > > broker mentioned, whereas Databases in use do vary somewhat. > > > > Currently it would appear that only oslo.messaging runs functional tests > > against QPID. I was unable to locate integration testing for it, but I > > may not know all of the places to dig around to find that. > > > > So, please let us know if QPID is important to you. Otherwise it may be > > time to unburden ourselves of its maintenance. > > > > [1] https://pypi.python.org/pypi/eventlet/0.17.3 > > [2] https://review.openstack.org/#/c/172135/ > > [3] > > http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014 > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > FWIW IBM Cloud Manager with OpenStack is still shipping qpid 0.30. We > switched to the default deployment being RabbitMQ in Kilo though (maybe > even Juno). But we do have a support matrix tested with qpid as the rpc > backend. Our mainline paths are tested with rabbitmq though since > that's the default backend for us now. > So, I think we can count this as another point toward removing impl_qpid. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
Excerpts from Ken Giusti's message of 2015-04-14 12:54:20 -0700: > Just to be clear: you're asking specifically about the 0-10 based > impl_qpid.py driver, correct? This is the driver that is used for > the "qpid://" transport (aka rpc_backend). > > I ask because I'm maintaining the AMQP 1.0 driver (transport > "amqp://") that can also be used with qpidd. > > However, the AMQP 1.0 driver isn't yet Python 3 compatible due to its > dependency on Proton, which has yet to be ported to python 3 - though > that's currently being worked on [1]. > > I'm planning on porting the AMQP 1.0 driver once the dependent > libraries are available. > > [1]: https://issues.apache.org/jira/browse/PROTON-490 > Thanks Ken. Yes that's what I mean. That there is an alternative already that has active work toward Python 3 seems like a point toward removal of impl qpid. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On 04/14/2015 01:22 PM, Clint Byrum wrote: > Hello! There's been some recent progress on python3 compatibility for > core libraries that OpenStack depends on[1], and this is likely to open > the flood gates for even more python3 problems to be found and fixed. > > Recently a proposal was made to make oslo.messaging start to run python3 > tests[2], and it was found that qpid-python is not python3 compatible yet. > > This presents us with questions: Is anyone using QPID, and if so, should > we add gate testing for it? If not, can we deprecate the driver? In the > most recent survey results I could find [3] I don't even see message > broker mentioned, whereas Databases in use do vary somewhat. > > Currently it would appear that only oslo.messaging runs functional tests > against QPID. I was unable to locate integration testing for it, but I > may not know all of the places to dig around to find that. > > So, please let us know if QPID is important to you. Otherwise it may be > time to unburden ourselves of its maintenance. Kill it. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
On 4/14/2015 12:22 PM, Clint Byrum wrote: Hello! There's been some recent progress on python3 compatibility for core libraries that OpenStack depends on[1], and this is likely to open the flood gates for even more python3 problems to be found and fixed. Recently a proposal was made to make oslo.messaging start to run python3 tests[2], and it was found that qpid-python is not python3 compatible yet. This presents us with questions: Is anyone using QPID, and if so, should we add gate testing for it? If not, can we deprecate the driver? In the most recent survey results I could find [3] I don't even see message broker mentioned, whereas Databases in use do vary somewhat. Currently it would appear that only oslo.messaging runs functional tests against QPID. I was unable to locate integration testing for it, but I may not know all of the places to dig around to find that. So, please let us know if QPID is important to you. Otherwise it may be time to unburden ourselves of its maintenance. [1] https://pypi.python.org/pypi/eventlet/0.17.3 [2] https://review.openstack.org/#/c/172135/ [3] http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev FWIW IBM Cloud Manager with OpenStack is still shipping qpid 0.30. We switched to the default deployment being RabbitMQ in Kilo though (maybe even Juno). But we do have a support matrix tested with qpid as the rpc backend. Our mainline paths are tested with rabbitmq though since that's the default backend for us now. -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
Just to be clear: you're asking specifically about the 0-10 based impl_qpid.py driver, correct? This is the driver that is used for the "qpid://" transport (aka rpc_backend). I ask because I'm maintaining the AMQP 1.0 driver (transport "amqp://") that can also be used with qpidd. However, the AMQP 1.0 driver isn't yet Python 3 compatible due to its dependency on Proton, which has yet to be ported to python 3 - though that's currently being worked on [1]. I'm planning on porting the AMQP 1.0 driver once the dependent libraries are available. [1]: https://issues.apache.org/jira/browse/PROTON-490 On Tue, Apr 14, 2015 at 1:22 PM, Clint Byrum wrote: > Hello! There's been some recent progress on python3 compatibility for > core libraries that OpenStack depends on[1], and this is likely to open > the flood gates for even more python3 problems to be found and fixed. > > Recently a proposal was made to make oslo.messaging start to run python3 > tests[2], and it was found that qpid-python is not python3 compatible yet. > > This presents us with questions: Is anyone using QPID, and if so, should > we add gate testing for it? If not, can we deprecate the driver? In the > most recent survey results I could find [3] I don't even see message > broker mentioned, whereas Databases in use do vary somewhat. > > Currently it would appear that only oslo.messaging runs functional tests > against QPID. I was unable to locate integration testing for it, but I > may not know all of the places to dig around to find that. > > So, please let us know if QPID is important to you. Otherwise it may be > time to unburden ourselves of its maintenance. > > [1] https://pypi.python.org/pypi/eventlet/0.17.3 > [2] https://review.openstack.org/#/c/172135/ > [3] > http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014 > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Ken Giusti (kgiu...@gmail.com) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] QPID incompatible with python 3 and untested in gate -- what to do?
Hello! There's been some recent progress on python3 compatibility for core libraries that OpenStack depends on[1], and this is likely to open the flood gates for even more python3 problems to be found and fixed. Recently a proposal was made to make oslo.messaging start to run python3 tests[2], and it was found that qpid-python is not python3 compatible yet. This presents us with questions: Is anyone using QPID, and if so, should we add gate testing for it? If not, can we deprecate the driver? In the most recent survey results I could find [3] I don't even see message broker mentioned, whereas Databases in use do vary somewhat. Currently it would appear that only oslo.messaging runs functional tests against QPID. I was unable to locate integration testing for it, but I may not know all of the places to dig around to find that. So, please let us know if QPID is important to you. Otherwise it may be time to unburden ourselves of its maintenance. [1] https://pypi.python.org/pypi/eventlet/0.17.3 [2] https://review.openstack.org/#/c/172135/ [3] http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
