Re: [openstack-dev] AMQP Version upgrade plans?
- Original Message - On 07/11/2013 12:06 PM, William Henry wrote: - Original Message - On 07/08/2013 10:51 AM, Ted Ross wrote: If someone from the Qpid community were to work on integrating the new AMQP 1.0 technology into OpenStack, where would be the right place to start? Would it be to add a new transport to oslo.messaging? I think so, yes. oslo.messaging is new, but it will deprecate the existing 'rpc' library in oslo-incubator. All projects will need to move to oslo.messaging, so for something new I would focus efforts there. I think that one of the important points that Ted brought up is that AMQP 1.0 doesn't have the concepts of broker artifacts like exchanges etc. A recent change I proposed to the existing impl_qpid.py which focuses more on addressing and not exchanges is a very important first step to solve several issues: a recent qpidd leak issue, transitioning to AMQP 1.0 (addressing), and possible HA solutions. This is an area I'd really like to continue to help out in. I'm back from some vacation and would like to get stuck in soon. Regarding the qpid exchange leak, that issue is mitigated largely by the fact that the only time we declare a direct exchange is for replies to an rpc. In previous versions there was a new one of these for *every* method call, which made this problem really bad. In the current code, we only create a single one. However, we're still left with a leak. The fact that RabbitMQ supports auto-delete on exchanges and Qpid doesn't is what got us into this spot, since this code works just like the kombu driver with respect to all of this. Right you still have the leak. Right auto-delete on exchanges in Qpid would mitigate. Qpid never supported this because, to my knowledge, auto-delete on exchanges didn't mean anything or have any value in Qpid. I'm not sure it was ever mentioned in the old pre AMQP 1.0 spec. (I'd need to check). And anyway exchanges don't mean anything in AMQP anymore. And don't mean anything in nonAMQP messaging - though AMQP implementations are not prohibited from implementing the queue/topic/fanout using exchange patterns. As for migration to AMQP 1.0, how do these changes help? Supporting AMQP 1.0 requires an entirely new driver that uses Qpid Proton, right? How does changing addressing in the current Qpid driver (that will never do 1.0) help? Actually Qpid project saw the pending AMQP 1.0 ramifications early on and created an addressing mechanism that supported AMQP 1.0 addressing in Qpid long before Qpid/Proton. So the changes I made recently would mean a simplified driver migration process. But you are correct, it would still need that migration to a new driver. It's my opinion that moving from a driver I recently submitted to the new driver would be very simple because of the way I've restructured the addressing. All great points you raise Russell. William I'm curious what you mean by possible HA solutions. Can you elaborate? -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] AMQP Version upgrade plans?
On 07/11/2013 12:06 PM, William Henry wrote: - Original Message - On 07/08/2013 10:51 AM, Ted Ross wrote: If someone from the Qpid community were to work on integrating the new AMQP 1.0 technology into OpenStack, where would be the right place to start? Would it be to add a new transport to oslo.messaging? I think so, yes. oslo.messaging is new, but it will deprecate the existing 'rpc' library in oslo-incubator. All projects will need to move to oslo.messaging, so for something new I would focus efforts there. I think that one of the important points that Ted brought up is that AMQP 1.0 doesn't have the concepts of broker artifacts like exchanges etc. A recent change I proposed to the existing impl_qpid.py which focuses more on addressing and not exchanges is a very important first step to solve several issues: a recent qpidd leak issue, transitioning to AMQP 1.0 (addressing), and possible HA solutions. This is an area I'd really like to continue to help out in. I'm back from some vacation and would like to get stuck in soon. Regarding the qpid exchange leak, that issue is mitigated largely by the fact that the only time we declare a direct exchange is for replies to an rpc. In previous versions there was a new one of these for *every* method call, which made this problem really bad. In the current code, we only create a single one. However, we're still left with a leak. The fact that RabbitMQ supports auto-delete on exchanges and Qpid doesn't is what got us into this spot, since this code works just like the kombu driver with respect to all of this. As for migration to AMQP 1.0, how do these changes help? Supporting AMQP 1.0 requires an entirely new driver that uses Qpid Proton, right? How does changing addressing in the current Qpid driver (that will never do 1.0) help? I'm curious what you mean by possible HA solutions. Can you elaborate? -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] AMQP Version upgrade plans?
On 06/24/2013, Russell Bryant wrote:
On 06/24/2013 02:01 PM, Sunjeet Singh wrote:
/
/ / OpenStack currently uses AMQP 0.8. Are there any plans to upgrade?
/
OpenStack is not specifically tied to an AMQP version. It's more about
what clients we have support for, what versions they speak, and then
what message brokers support that.
Right now we have support for using the kombu (for RabbitMQ) and qpid
(for Qpid) clients.
I actually have a big interest in adding support for AMQP 1.0. One of
the *really* interesting things with AMQP 1.0 that is relevant for
OpenStack is that it is not tied to using a message broker. You can use
it in both a peer-to-peer and a brokered way. This is interesting,
because depending on the messaging pattern we're using, we may want one
vs the other in different parts of OpenStack.
As for the specifics to doing this, the only AMQP 1.0 client I know of
is Proton. It does have Python bindings.
http://qpid.apache.org/proton/
As for the server side, ActiveMQ recently added AMQP 1.0 support using
this same library (proton).
http://activemq.apache.org/
--
Russell Bryant
AMQP 0-{8,9,10} are asymmetric client/server protocols with a
well-defined server/broker behavior. AMQP 1.0, by contrast, is a
symmetric wire-line protocol, without a requirement for a broker at
all. As Russell points out, this opens the possibility of brokerless
peer-to-peer messaging.
The problem with peer-to-peer at the transport is that it doesn't scale
well, doesn't provide fanout, requires complicated URL management, and
causes problems with firewalls. It might be appropriate for a small
number of use cases, but a broker or some other form of intermediary is
still needed.
There is a sub-project of Apache Qpid, built on the Proton libraries,
called Qpid Dispatch. Dispatch is a lightweight intermediary that is
not a broker but can be used to build a messaging bus that provides
peer-to-peer and peer-to-multi-peer connectivity at the messaging level
(i.e. based on messaging addresses, not host/port urls). It can also be
used to provide connectivity to brokers for the cases where brokers are
necessary (persistent store-and-forward for example).
If someone from the Qpid community were to work on integrating the new
AMQP 1.0 technology into OpenStack, where would be the right place to
start? Would it be to add a new transport to oslo.messaging?
-Ted
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] AMQP Version upgrade plans?
If you have a recent version of kombu, and amqp[1] rather than amqplib installed, things will just start using AMQP 0.9.1. Ubuntu doesn't package a new enough Kombu, and they don't package amqp at all.. Not sure about other distro's. Thanks, Kiall [1]: https://pypi.python.org/pypi/amqp On 24/06/13 19:12, Sunjeet Singh wrote: OpenStack currently uses AMQP 0.8. Are there any plans to upgrade? Thank you, Sunjeet ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] AMQP Version upgrade plans?
Hi On Mon, Jul 08, 2013 at 03:11:02PM +, Mac Innes, Kiall wrote: If you have a recent version of kombu, and amqp[1] rather than amqplib installed, things will just start using AMQP 0.9.1. Ubuntu doesn't package a new enough Kombu, and they don't package amqp at all.. Not sure about other distro's. Actually saucy has the latest and greatest and it will be backported to the cloud archive when havana-2 is out. https://launchpad.net/ubuntu/+source/kombu Thanks, Kiall [1]: https://pypi.python.org/pypi/amqp On 24/06/13 19:12, Sunjeet Singh wrote: OpenStack currently uses AMQP 0.8. Are there any plans to upgrade? Thank you, Sunjeet ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] AMQP Version upgrade plans?
On 07/08/2013 10:51 AM, Ted Ross wrote: If someone from the Qpid community were to work on integrating the new AMQP 1.0 technology into OpenStack, where would be the right place to start? Would it be to add a new transport to oslo.messaging? I think so, yes. oslo.messaging is new, but it will deprecate the existing 'rpc' library in oslo-incubator. All projects will need to move to oslo.messaging, so for something new I would focus efforts there. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] AMQP Version upgrade plans?
OpenStack currently uses AMQP 0.8. Are there any plans to upgrade? Thank you, Sunjeet ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] AMQP Version upgrade plans?
On 06/24/2013 02:01 PM, Sunjeet Singh wrote: OpenStack currently uses AMQP 0.8. Are there any plans to upgrade? OpenStack is not specifically tied to an AMQP version. It's more about what clients we have support for, what versions they speak, and then what message brokers support that. Right now we have support for using the kombu (for RabbitMQ) and qpid (for Qpid) clients. I actually have a big interest in adding support for AMQP 1.0. One of the *really* interesting things with AMQP 1.0 that is relevant for OpenStack is that it is not tied to using a message broker. You can use it in both a peer-to-peer and a brokered way. This is interesting, because depending on the messaging pattern we're using, we may want one vs the other in different parts of OpenStack. As for the specifics to doing this, the only AMQP 1.0 client I know of is Proton. It does have Python bindings. http://qpid.apache.org/proton/ As for the server side, ActiveMQ recently added AMQP 1.0 support using this same library (proton). http://activemq.apache.org/ -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
