Re: [openstack-dev] [oslo.messaging] Request to include AMQP 1.0 support in Juno-3
On 08/28/2014 12:34 PM, Doug Hellmann wrote: > > On Aug 28, 2014, at 8:36 AM, Mark McLoughlin wrote: > >> On Thu, 2014-08-28 at 13:24 +0200, Flavio Percoco wrote: >>> On 08/27/2014 03:35 PM, Ken Giusti wrote: Hi All, I believe Juno-3 is our last chance to get this feature [1] included into olso.messaging. I honestly believe this patch is about as low risk as possible for a change that introduces a whole new transport into oslo.messaging. The patch shouldn't affect the existing transports at all, and doesn't come into play unless the application specifically turns on the new 'amqp' transport, which won't be the case for existing applications. The patch includes a set of functional tests which exercise all the messaging patterns, timeouts, and even broker failover. These tests do not mock out any part of the driver - a simple test broker is included which allows the full driver codepath to be executed and verified. IFAIK, the only remaining technical block to adding this feature, aside from core reviews [2], is sufficient infrastructure test coverage. We discussed this a bit at the last design summit. The root of the issue is that this feature is dependent on a platform-specific library (proton) that isn't in the base repos for most of the CI platforms. But it is available via EPEL, and the Apache QPID team is actively working towards getting the packages into Debian (a PPA is available in the meantime). In the interim I've proposed a non-voting CI check job that will sanity check the new driver on EPEL based systems [3]. I'm also working towards adding devstack support [4], which won't be done in time for Juno but nevertheless I'm making it happen. I fear that this feature's inclusion is stuck in a chicken/egg deadlock: the driver won't get merged until there is CI support, but the CI support won't run correctly (and probably won't get merged) until the driver is available. The driver really has to be merged first, before I can continue with CI/devstack development. [1] https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation [2] https://review.openstack.org/#/c/75815/ [3] https://review.openstack.org/#/c/115752/ [4] https://review.openstack.org/#/c/109118/ >>> >>> >>> Hi Ken, >>> >>> Thanks a lot for your hard work here. As I stated in my last comment on >>> the driver's review, I think we should let this driver land and let >>> future patches improve it where/when needed. >>> >>> I agreed on letting the driver land as-is based on the fact that there >>> are patches already submitted ready to enable the gates for this driver. >> >> I feel bad that the driver has been in a pretty complete state for quite >> a while but hasn't received a whole lot of reviews. There's a lot of >> promise to this idea, so it would be ideal if we could unblock it. >> >> One thing I've been meaning to do this cycle is add concrete advice for >> operators on the state of each driver. I think we'd be a lot more >> comfortable merging this in Juno if we could somehow make it clear to >> operators that it's experimental right now. My idea was: >> >> - Write up some notes which discusses the state of each driver e.g. >> >> - RabbitMQ - the default, used by the majority of OpenStack >>deployments, perhaps list some of the known bugs, particularly >>around HA. >> >> - Qpid - suitable for production, but used in a limited number of >>deployments. Again, list known issues. Mention that it will >>probably be removed with the amqp10 driver matures. >> >> - Proton/AMQP 1.0 - experimental, in active development, will >>support multiple brokers and topologies, perhaps a pointer to a >>wiki page with the current TODO list >> >> - ZeroMQ - unmaintained and deprecated, planned for removal in >>Kilo >> >> - Propose this addition to the API docs and ask the operators list >>for feedback >> >> - Propose a patch which adds a load-time deprecation warning to the >>ZeroMQ driver >> >> - Include a load-time experimental warning in the proton driver >> >> Thoughts on that? > > By "API docs" do you mean the ones in the oslo.messaging repository? Would it > be better to put this information in the operator’s guide? I was talking to Ken a little about this today and came up with http://docs.openstack.org/icehouse/config-reference/content/configuring-rpc.html That seems like a reasonable place to put information like this (in fact, there's already some there about rabbit being the default). I wasn't sure exactly where those docs are generated from, so I suggested he talk to Anne Gentle about it. -Ben > > Other than the question of where to put it, I definitely think this is the > sort of guidance we should document, incl
Re: [openstack-dev] [oslo.messaging] Request to include AMQP 1.0 support in Juno-3
On Thu, 28 Aug 2014 13:36:46 +0100, Mark McLoughlin wrote: > On Thu, 2014-08-28 at 13:24 +0200, Flavio Percoco wrote: > > On 08/27/2014 03:35 PM, Ken Giusti wrote: > > > Hi All, > > > > > > I believe Juno-3 is our last chance to get this feature [1] included > > > into olso.messaging. > > > > > > > > > Hi Ken, > > > > Thanks a lot for your hard work here. As I stated in my last comment on > > the driver's review, I think we should let this driver land and let > > future patches improve it where/when needed. > > > > I agreed on letting the driver land as-is based on the fact that there > > are patches already submitted ready to enable the gates for this driver. > > I feel bad that the driver has been in a pretty complete state for quite > a while but hasn't received a whole lot of reviews. There's a lot of > promise to this idea, so it would be ideal if we could unblock it. > > One thing I've been meaning to do this cycle is add concrete advice for > operators on the state of each driver. I think we'd be a lot more > comfortable merging this in Juno if we could somehow make it clear to > operators that it's experimental right now. My idea was: > > - Write up some notes which discusses the state of each driver e.g. > > - RabbitMQ - the default, used by the majority of OpenStack > deployments, perhaps list some of the known bugs, particularly > around HA. > > - Qpid - suitable for production, but used in a limited number of > deployments. Again, list known issues. Mention that it will > probably be removed with the amqp10 driver matures. > > - Proton/AMQP 1.0 - experimental, in active development, will > support multiple brokers and topologies, perhaps a pointer to a > wiki page with the current TODO list > > - ZeroMQ - unmaintained and deprecated, planned for removal in > Kilo Sounds like a plan - I'll take on the Qpid and Proton notes. I've been (trying) to keep the status of the Proton stuff up to date on the blueprint page: https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation Is there a more appropriate home for these notes? Etherpad? > > - Propose this addition to the API docs and ask the operators list > for feedback > > - Propose a patch which adds a load-time deprecation warning to the > ZeroMQ driver > > - Include a load-time experimental warning in the proton driver Done! > > Thoughts on that? > > (I understand the ZeroMQ situation needs further discussion - I don't > think that's on-topic for the thread, I was just using it as example of > what kind of advice we'd be giving in these docs) > > Mark. > > - Ken Giusti (kgiu...@gmail.com) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.messaging] Request to include AMQP 1.0 support in Juno-3
On Aug 28, 2014, at 8:36 AM, Mark McLoughlin wrote: > On Thu, 2014-08-28 at 13:24 +0200, Flavio Percoco wrote: >> On 08/27/2014 03:35 PM, Ken Giusti wrote: >>> Hi All, >>> >>> I believe Juno-3 is our last chance to get this feature [1] included >>> into olso.messaging. >>> >>> I honestly believe this patch is about as low risk as possible for a >>> change that introduces a whole new transport into oslo.messaging. The >>> patch shouldn't affect the existing transports at all, and doesn't >>> come into play unless the application specifically turns on the new >>> 'amqp' transport, which won't be the case for existing applications. >>> >>> The patch includes a set of functional tests which exercise all the >>> messaging patterns, timeouts, and even broker failover. These tests do >>> not mock out any part of the driver - a simple test broker is included >>> which allows the full driver codepath to be executed and verified. >>> >>> IFAIK, the only remaining technical block to adding this feature, >>> aside from core reviews [2], is sufficient infrastructure test coverage. >>> We discussed this a bit at the last design summit. The root of the >>> issue is that this feature is dependent on a platform-specific library >>> (proton) that isn't in the base repos for most of the CI platforms. >>> But it is available via EPEL, and the Apache QPID team is actively >>> working towards getting the packages into Debian (a PPA is available >>> in the meantime). >>> >>> In the interim I've proposed a non-voting CI check job that will >>> sanity check the new driver on EPEL based systems [3]. I'm also >>> working towards adding devstack support [4], which won't be done in >>> time for Juno but nevertheless I'm making it happen. >>> >>> I fear that this feature's inclusion is stuck in a chicken/egg >>> deadlock: the driver won't get merged until there is CI support, but >>> the CI support won't run correctly (and probably won't get merged) >>> until the driver is available. The driver really has to be merged >>> first, before I can continue with CI/devstack development. >>> >>> [1] >>> https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation >>> [2] https://review.openstack.org/#/c/75815/ >>> [3] https://review.openstack.org/#/c/115752/ >>> [4] https://review.openstack.org/#/c/109118/ >> >> >> Hi Ken, >> >> Thanks a lot for your hard work here. As I stated in my last comment on >> the driver's review, I think we should let this driver land and let >> future patches improve it where/when needed. >> >> I agreed on letting the driver land as-is based on the fact that there >> are patches already submitted ready to enable the gates for this driver. > > I feel bad that the driver has been in a pretty complete state for quite > a while but hasn't received a whole lot of reviews. There's a lot of > promise to this idea, so it would be ideal if we could unblock it. > > One thing I've been meaning to do this cycle is add concrete advice for > operators on the state of each driver. I think we'd be a lot more > comfortable merging this in Juno if we could somehow make it clear to > operators that it's experimental right now. My idea was: > > - Write up some notes which discusses the state of each driver e.g. > > - RabbitMQ - the default, used by the majority of OpenStack >deployments, perhaps list some of the known bugs, particularly >around HA. > > - Qpid - suitable for production, but used in a limited number of >deployments. Again, list known issues. Mention that it will >probably be removed with the amqp10 driver matures. > > - Proton/AMQP 1.0 - experimental, in active development, will >support multiple brokers and topologies, perhaps a pointer to a >wiki page with the current TODO list > > - ZeroMQ - unmaintained and deprecated, planned for removal in >Kilo > > - Propose this addition to the API docs and ask the operators list >for feedback > > - Propose a patch which adds a load-time deprecation warning to the >ZeroMQ driver > > - Include a load-time experimental warning in the proton driver > > Thoughts on that? By "API docs" do you mean the ones in the oslo.messaging repository? Would it be better to put this information in the operator’s guide? Other than the question of where to put it, I definitely think this is the sort of guidance we should document, including through the logged warnings. Doug > > (I understand the ZeroMQ situation needs further discussion - I don't > think that's on-topic for the thread, I was just using it as example of > what kind of advice we'd be giving in these docs) > > Mark. > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenSt
Re: [openstack-dev] [oslo.messaging] Request to include AMQP 1.0 support in Juno-3
On Thu, 2014-08-28 at 13:24 +0200, Flavio Percoco wrote: > On 08/27/2014 03:35 PM, Ken Giusti wrote: > > Hi All, > > > > I believe Juno-3 is our last chance to get this feature [1] included > > into olso.messaging. > > > > I honestly believe this patch is about as low risk as possible for a > > change that introduces a whole new transport into oslo.messaging. The > > patch shouldn't affect the existing transports at all, and doesn't > > come into play unless the application specifically turns on the new > > 'amqp' transport, which won't be the case for existing applications. > > > > The patch includes a set of functional tests which exercise all the > > messaging patterns, timeouts, and even broker failover. These tests do > > not mock out any part of the driver - a simple test broker is included > > which allows the full driver codepath to be executed and verified. > > > > IFAIK, the only remaining technical block to adding this feature, > > aside from core reviews [2], is sufficient infrastructure test coverage. > > We discussed this a bit at the last design summit. The root of the > > issue is that this feature is dependent on a platform-specific library > > (proton) that isn't in the base repos for most of the CI platforms. > > But it is available via EPEL, and the Apache QPID team is actively > > working towards getting the packages into Debian (a PPA is available > > in the meantime). > > > > In the interim I've proposed a non-voting CI check job that will > > sanity check the new driver on EPEL based systems [3]. I'm also > > working towards adding devstack support [4], which won't be done in > > time for Juno but nevertheless I'm making it happen. > > > > I fear that this feature's inclusion is stuck in a chicken/egg > > deadlock: the driver won't get merged until there is CI support, but > > the CI support won't run correctly (and probably won't get merged) > > until the driver is available. The driver really has to be merged > > first, before I can continue with CI/devstack development. > > > > [1] > > https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation > > [2] https://review.openstack.org/#/c/75815/ > > [3] https://review.openstack.org/#/c/115752/ > > [4] https://review.openstack.org/#/c/109118/ > > > Hi Ken, > > Thanks a lot for your hard work here. As I stated in my last comment on > the driver's review, I think we should let this driver land and let > future patches improve it where/when needed. > > I agreed on letting the driver land as-is based on the fact that there > are patches already submitted ready to enable the gates for this driver. I feel bad that the driver has been in a pretty complete state for quite a while but hasn't received a whole lot of reviews. There's a lot of promise to this idea, so it would be ideal if we could unblock it. One thing I've been meaning to do this cycle is add concrete advice for operators on the state of each driver. I think we'd be a lot more comfortable merging this in Juno if we could somehow make it clear to operators that it's experimental right now. My idea was: - Write up some notes which discusses the state of each driver e.g. - RabbitMQ - the default, used by the majority of OpenStack deployments, perhaps list some of the known bugs, particularly around HA. - Qpid - suitable for production, but used in a limited number of deployments. Again, list known issues. Mention that it will probably be removed with the amqp10 driver matures. - Proton/AMQP 1.0 - experimental, in active development, will support multiple brokers and topologies, perhaps a pointer to a wiki page with the current TODO list - ZeroMQ - unmaintained and deprecated, planned for removal in Kilo - Propose this addition to the API docs and ask the operators list for feedback - Propose a patch which adds a load-time deprecation warning to the ZeroMQ driver - Include a load-time experimental warning in the proton driver Thoughts on that? (I understand the ZeroMQ situation needs further discussion - I don't think that's on-topic for the thread, I was just using it as example of what kind of advice we'd be giving in these docs) Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.messaging] Request to include AMQP 1.0 support in Juno-3
On 08/27/2014 03:35 PM, Ken Giusti wrote: > Hi All, > > I believe Juno-3 is our last chance to get this feature [1] included > into olso.messaging. > > I honestly believe this patch is about as low risk as possible for a > change that introduces a whole new transport into oslo.messaging. The > patch shouldn't affect the existing transports at all, and doesn't > come into play unless the application specifically turns on the new > 'amqp' transport, which won't be the case for existing applications. > > The patch includes a set of functional tests which exercise all the > messaging patterns, timeouts, and even broker failover. These tests do > not mock out any part of the driver - a simple test broker is included > which allows the full driver codepath to be executed and verified. > > IFAIK, the only remaining technical block to adding this feature, > aside from core reviews [2], is sufficient infrastructure test coverage. > We discussed this a bit at the last design summit. The root of the > issue is that this feature is dependent on a platform-specific library > (proton) that isn't in the base repos for most of the CI platforms. > But it is available via EPEL, and the Apache QPID team is actively > working towards getting the packages into Debian (a PPA is available > in the meantime). > > In the interim I've proposed a non-voting CI check job that will > sanity check the new driver on EPEL based systems [3]. I'm also > working towards adding devstack support [4], which won't be done in > time for Juno but nevertheless I'm making it happen. > > I fear that this feature's inclusion is stuck in a chicken/egg > deadlock: the driver won't get merged until there is CI support, but > the CI support won't run correctly (and probably won't get merged) > until the driver is available. The driver really has to be merged > first, before I can continue with CI/devstack development. > > [1] > https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation > [2] https://review.openstack.org/#/c/75815/ > [3] https://review.openstack.org/#/c/115752/ > [4] https://review.openstack.org/#/c/109118/ Hi Ken, Thanks a lot for your hard work here. As I stated in my last comment on the driver's review, I think we should let this driver land and let future patches improve it where/when needed. I agreed on letting the driver land as-is based on the fact that there are patches already submitted ready to enable the gates for this driver. Lets see what others think about this, Flavio -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo.messaging] Request to include AMQP 1.0 support in Juno-3
Hi All, I believe Juno-3 is our last chance to get this feature [1] included into olso.messaging. I honestly believe this patch is about as low risk as possible for a change that introduces a whole new transport into oslo.messaging. The patch shouldn't affect the existing transports at all, and doesn't come into play unless the application specifically turns on the new 'amqp' transport, which won't be the case for existing applications. The patch includes a set of functional tests which exercise all the messaging patterns, timeouts, and even broker failover. These tests do not mock out any part of the driver - a simple test broker is included which allows the full driver codepath to be executed and verified. IFAIK, the only remaining technical block to adding this feature, aside from core reviews [2], is sufficient infrastructure test coverage. We discussed this a bit at the last design summit. The root of the issue is that this feature is dependent on a platform-specific library (proton) that isn't in the base repos for most of the CI platforms. But it is available via EPEL, and the Apache QPID team is actively working towards getting the packages into Debian (a PPA is available in the meantime). In the interim I've proposed a non-voting CI check job that will sanity check the new driver on EPEL based systems [3]. I'm also working towards adding devstack support [4], which won't be done in time for Juno but nevertheless I'm making it happen. I fear that this feature's inclusion is stuck in a chicken/egg deadlock: the driver won't get merged until there is CI support, but the CI support won't run correctly (and probably won't get merged) until the driver is available. The driver really has to be merged first, before I can continue with CI/devstack development. [1] https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation [2] https://review.openstack.org/#/c/75815/ [3] https://review.openstack.org/#/c/115752/ [4] https://review.openstack.org/#/c/109118/ -- Ken Giusti (kgiu...@gmail.com) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev