On Mon, Jul 7, 2014 at 9:06 AM, Gordon Sim <[email protected]> wrote: > On 07/02/2014 08:24 PM, Rafael Schloming wrote: > >> My goal with this demo has been to explore a couple of distinct themes. >> The >> toolkit is intended in part to illustrate how the events API allows use of >> proton in a reactive programming style. It does this by building a >> select/nio based Driver that functions as a simple reactor. This is useful >> in its own right, however it is also intended to serve as an example of >> how >> you can integrate proton's events into an existing selector based >> implementation. >> >> Another big theme is providing the ability to encapsulate different >> aspects >> of protocol behaviour into distinct but composable chunks. This is >> achieved >> because the Driver dispatches each event to a configurable set of >> handlers. >> This allows the Collector to serve as a central "event bus" and allows the >> various handlers to all tie into this bus and work together to provide >> useful combinations of behaviour. >> > > I really like this composable approach, especially based on a reactive > style with which it seems to fit well. > > > Ultimately I hope this work will address the gap that we've had between >> the >> simple/easy to use Messenger API and the experts-only Engine API. >> > > I personally don't think of the gap in those terms. I didn't find either > of the APIs particularly simple or easy to use and I'm not convinced that > the issue with the engine API was that it was intended for 'experts' >
It wasn't, there were supposed to be scare quotes on that, i.e. 'experts-only' seems to be how people have classified it ex post facto. The intention behind the engine API was/is to provide easily embeddable access to the full capabilites of AMQP, which somehow ended up as "you need to be an expert in AMQP in order to use it" which might be coincidentally true, but was certainly not a goal. > > My issue with the engine API was that using it involved lots of duplicated > code and the API itself was not always entirely obvious. This was less an > intrinsic issue, more of a lack of focus on making it easy to use. Efforts > like pyngus did a good job of closing that gap. The approach you > demonstrate here shows how that could be taken even further. > > My issue with the Messenger API is that I don't understand its purpose. I've been struggling a bit with how to explain this, so apologies in advance for rambling a bit. I can understand at some level its purpose is just to send/recv messages, and this doesn't necessarily seem distinct from what the engine API is providing, however I think there are some important differences to recognize. If you consider Connection and Messenger as simply communication endpoints, e.g. ignore the difference in programming style for the moment (blocking vs non-blocking/etc), then I think the key differences come down to lifecycle and coupling. The lifecycle of a Connection object is geared towards the lifecycle of a TCP Connection, and as a data structure it is very much coupled to a single peer at the other end of said TCP connection. The Messenger object as a communication endpoint is quite different. It's lifecycle is defined by the application, not by the TCP Connection. It *could* be a single TCP connection if that is what the application configures/desires, but it could also be the process lifetime of the application, or even beyond that with suitable support for persistence. As a communication endpoint it is also much more loosely coupled than the Connection objects provided by the engine API. The objects provided by the engine API are designed to be used in local/remote pairs to track the communication state relevant to the on-the-wire exchange of information between two peers, but once the wire goes away, that data structure has limited utility for tracking state relevant to the application level communication model. Any unresolved state needs to be extracted from the unsettled Deliveries on the Connection and reflected back into the application level communication model, e.g. a broker might, depending on policy, need to requeue certain messages. A Messenger endpoint on the other hand can store/track messages even when it has no peers, and is perfectly capable of delivering/distributing those messages to any number of peers when they become available. Of course for a broker, whose entire domain is tracking application level communication info, the engine API is probably all you would ever need, however for most other applications that are trying to be resilient to TCP failures, I believe there is a gap without some sort of Messenger-like communication endpoint. Once the connection goes away, there is no library level construct to track the relevant communication state for any unsettled deliveries, and this means that lots of applications will need to build their own little mini-queues over and over again just to be resilient to TCP connection failure. The messag*ing* API does venture a little bit into this area with its auto reconnect feature, however I would argue that the approach it takes is limited. While it does address some of the lifecycle issues by extending the lifecycle of the communication endpoint to match the application process, it's not as clear how you would extend this beyond a single process (i.e. add client side persistence), and it doesn't address the coupling issues at all. The reconnect feature depends on being able to reconnect to a replica (more or less) of its original peer. So to summarize, I would say the core idea behind Messenger which differentiates it from other offerings is that it is a loosely coupled communication endpoint whose lifecycle is defined/controlled by the application. I don't think any other API we have is really designed around those principles, yet I would argue those two things are what most distributed applications are actually built upon. I hope the above ramblings help clarify things a bit. It's important to note that a lot of what I'm talking about is really focused around the ideas behind Messenger rather than how it is currently realized. The current implementation suffers from starting out as a pure blocking API and having various non blocking extensions bolted on. This (and various unrelated things) have sucked up time and made it difficult to actually fully develop the core ideas, so I'm not surprised there is confusion and lack of confidence around it, but I do believe the core idea is still highly relevant. > > I believe >> with a relatively small amount of work on a toolkit like this, the >> internals of Messenger can be fairly trivially expressed as a composition >> of different protocol behaviours, thereby allowing a simple and easy >> starting point that can smoothly transition to any level of sophistication >> desired. >> > > I think this ability for users to transition for simple usage to more > complex handling is very valuable. I think there is a related benefit to > that approach in that it allows the API - or toolkit more properly - to > evolve in a better way. Rather than an API that ring-fences the > functionality, you offer a core API and a set of utilties. Different > utilities can be provided for different sets of use cases. In fact another > aspect I like about your demo is the example driven nature. > > I would be very much in favour of refocusing proton on the engine and > events API and this expanding toolkit and refining it based on a growing > set of concrete examples. I think that would make proton more > comprehensible and more useful. That in turn would have a positive impact > on AMQP adoption. > > I think it also neatly avoids the confusion with regards to existing APIs > such as qpid::messaging. > I agree it would be good to focus on the events API as a way to build out/explore, and I'm 100% in favor of driving this sort of thing via examples. For the event API itself I think an important next step is to ensure that it is extensible enough that we can add events without breaking people's code. This includes both having new kinds of things that generate events, and also ensuring that existing things can generate new types of events. This should also cover Bozo's case of having some way to inject user defined events. I believe once we have this we will have a very strong framework within which to build/evolve. --Rafael
