Since the start of AMQP 1.0 implementation work I've posted several
times on the design of the work as it's evolved, and a bit on the source
code layout as that work was pulled into qpid. After recent discussions
I thought it would be good to post a bit more about the motivation and
strategy behind the work and try to clear up any confusion or
uncertainty about what Proton is and what it is trying to do.
So here goes, apologies for the long and rambling nature, and please
view in a fixed width font if you want the diagram to look pretty.
--Rafael
== Proton History & Mission ==
The idea behind Proton originated from one of the fundamental
conflicts within the AMQP working group. In one camp there are those
who believe that the protocol itself needs to be simple enough that
people can "write to the wire", in other words the wire format itself
forms a suitable API for people to directly use from their
applications.
In the other camp there are those who believe that certain features,
e.g. acknowledgment and flow control, are a requirement in enough
messaging use cases that any general purpose messaging protocol must
support them, and that in either case applications ultimately won't be
developed against the wire protocol itself, but will be built on top
of libraries that happen to speak the protocol underneath, and a
library operating on top of a sophisticated protocol can be much more
powerful.
In many ways this is the classic worse is better debate. A
sophisticated protocol with a good library on top can provide both the
simplest and most capable tool, whereas a simplistic protocol while
less capable can be ready to use right away and spread quickly without
waiting for suitable library support to be built.
Once it was understood that a) this was at the core of much of the
ongoing debate within the working group, and b) a simplistic protocol
could never satisfy the user requirements of the working group, it
seemed clear that a protocol alone was not sufficient for the success
of AMQP. We also needed a strategy around jump starting
implementations. A way to let a sophisticated protocol spread (almost)
as quickly as a simplistic one.
This was the origin of the notion to create a neutral library with the
sole mission to enable widespread internet scale adoption of AMQP by
making it as easy as possible to integrate into a wide variety of
environments, and this is the core mission of Proton today.
While it's core mission has remained the same, the means by which it
seeks to acheive that mission have evolved over time. Providing simple
codecs to give people a head start in parsing the protocol was quickly
evolved into providing a more complete engine that encapsulates the
protocol state and implements much of the low level logic for you.
>From there it was an easy step to provide optional drivers for that
engine to make it easier to integrate into environments that did not
mandate the use of their own I/O subsystem.
With the engine and driver in place it became fairly straightforward
to build clients and servers directly on top of Proton, and after
creating a few examples, the need for a common message representation
was keenly felt.
The latest evolution in thinking was motivated by a confluence of
several initiatives:
1) The use of Proton to build a widely available and very simple
client library where the "hello world" of sending and receiving
messages was literaly only one or two lines of code.
2) The use of Proton to integrate AMQP support into OpenMAMA.
3) The prospect of using Proton to integrate AMQP support into a
variety of open source frameworks.
What all these have in common is that while they each potentially have
the need for the flexible programming models permitted by the engine
architecture, none of them require the level of protocol control
provided by the engine, and in fact in all these cases the level of
fine grained control that the engine provides is a barrier to entry
because it requires understanding the lion's share of the protocol
model of AMQP in order to use.
This realization lead to the notion of classifying integration use
cases according to dimensions of protocol control and programming
model.
Protocol Control
P |
r Limited | Full
o +------------------+---------------------+
g Blocking | messenger | engine + driver |
---------------+------------------+---------------------+
M Non Blocking | ? | engine |
o +------------------+---------------------+
d
e
l
To date the evolution of Proton has consisted of steadily filling in
and refining the integration support offered in each corner of the
above matrix.
Looking to the future it is tempting to predict that there will be a
demand for functionality in the as yet unfilled lower left quadrant of
the matrix. Indeed in retrospect it should perhaps seem obvious that
many if not most systems integrating support for AMQP will do so with
no detailed knowledge of the protocol model.
== Proton Scope ==
Put simply, the scope of Proton is first and formost defined as
whatever is necessary in order to acheive its mission: "to enable
widespread internet scale adoption of AMQP by making it as easy as
possible to integrate into a wide variety of environments."
This should make it clear that by its very nature it is an open ended
effort that will evolve over time, however everything needs a starting
point, and so its near and medium term scope is very much moderated by
the strategy and tactics chosen to persue its mission.
In order to be attractive to integrators and to remove any obstacles
to adoption, a deliberate effort has been made to minimize the number
of required dependencies on the target platforms, and to keep the
footprint of the project small and well understood.
To reach the maximum number of environments with the minimum number of
implentations necessary, the core functionality is developed in C with
bindings made available in a wide array of popular languages via swig.
It is an important design goal to minimize the effort to produce a new
swig binding for the core set of APIs.
Because bindings are not an option for many java and javascript
deployments, the same core functionality is maintained in parallel
implementations in these languages. It is important that all three
implementations behave identically, and to ensure that is the case,
they are all validated against a common suite of tests written in
python. (The C implementation is driven via the python binding, the
Java via use of the jython interpreter, and the javascript
implementation will be driven via jython in conjunction with Rhino.)
== Proton within Qpid ==
The fact that Proton is an independent sub project with its own
clearly defined mission raises some natural questions about how it
fits together with the rest of Qpid.
The original mission of Qpid is something along the lines of "an open
source AMQP implementation". On the surface this might sound like a
close match to the Proton mission, however at the time of its
inception, AMQP was a very different specification, and an open source
implementation of AMQP 0-x meant building a server that implements the
0-x broker model and provides services like transactions and
persistence. This is further compounded by the fact that the 0-x
broker model is not a complete middleware specification and the Qpid
brokers have extended it significantly. In fact the meat of most Qpid
development for quite a while has been things like HA, LVQs, message
grouping, etc, none of which appear in any AMQP specifcation
regardless of the version.
In contrast AMQP 1.0 is a specification for a general purpose
messaging protocol. While this can easily serve as the protocol for an
0-x broker, an 0-x broker that happens to speak AMQP 1.0 is very
different from a general purpose AMQP 1.0 implementation.
So while the Proton mission is in many ways compatible with the
original Qpid charter, the de facto Qpid mission of today is really
quite different from Proton's.
Understanding both missions well and understanding the full scope of
AMQP 1.0 is key to understanding how they might evolve and relate to
each other in the future. The reality is that these sorts of things
are difficult to predict and there is a whole spectrum of in betweens,
however it is still useful to look at some extreme points in order to
understand the full space of possibilities. The following three
hypotheticals are worth considering:
1) The incubation route.
Proton is an indepenent sub project hosted by Qpid. Qpid makes use
of Proton when there is a good fit, although some overlap might
occur and would be tolerated. Eventually proton graduates into its
own TLP.
2) The reverse incubation route.
Proton and "The Brokers" are independent sub projects hosted by
Qpid. Proton becomes the primary focus of the Qpid project, i.e.
Proton becomes simply a codename for Qpid 2.0. The Brokers make use
of proton where there is a good fit, and ultimately The Brokers are
phased out or split off into their own TLP.
3) The two headed project route.
Proton and "The Brokers" both live within Qpid indefinitely. The
Brokers integrate Proton as the basis of their AMQP 1.0 protocol
support. As the focus of Proton is to make it easy to speak AMQP,
ultimately all Qpid users access "The Brokers" by using Proton.
Proton is also integrated by other brokers/servers, and is also used
by their users.
It's worth noting that in each of these substantially different
scenarios, the technical relationship between the two software
components can hypothetically be the same. The important difference
between scenarios is really in the governance of the project(s), and
the degree of independence aforded to each component, and this can
make a significant difference in how the technical relationship
evolves in practice.
Consider a hypothetical scenario in which Proton is embedded in the
server and/or client side of several popular messaging solutions, only
one of which is qpid. Suppose there are two bugs, one effects 80% of
Proton users but happens to have no impact on Qpid, whereas the other
bug impacts 20% of Proton users, including Qpid. If Proton is merely a
component of Qpid that happens to be packaged separately, then
priority would go to the bug that impacts Qpid. If Proton is an
independent sub project, then prioirity would go to the bug that
impacts 80% of Proton users. This gets even more interesting when you
consider the clear conflict of interest that arises when the other 80%
of Proton users includes brokers that are direct competitors to
Qpid's.
This illustrates the important part of what it means to be an
*independent* (sub) project: A governance model and environment where
all Proton users have an equal voice, not only when it comes to
prioritizing bug fixes, but also in the evolution of the software
itself.
While it's not necessary to decide today where things will end up in
the distant future, hopefully it is useful to lay out a few of the
possibilities in order to inform some of the more immediate
discussions.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
