----- Original Message ----- > I'd like to understand the use case more - why would you want to do > this > exactly (use AMQP + Kafka) other than just because?
Good point. See my previous email. It's not a just because. I think it is precisely the kind of synergy that projects like Qpid/Proton are working towards. We'd love to see AMQP 1.0 be ubiquitous. However we understand that at a higher level there are frameworks and architectures that have specific value to users. I've recently been working on integrating Qpid/Proton with OpenMAMA as an example. And I'm working on interoperability with Microsoft's and other middleware products. Again, the goal of AMQP was to handle all messaging use cases and if it can't handle Kafka's workload then AMQP needs to address that. So I could see: Kafka using Proton which provides AMQP 1.0 Best, William > > On Tue, Oct 2, 2012 at 9:22 AM, Jay Kreps <jay.kr...@gmail.com> > wrote: > > > I looked into AMQP when I was first starting Kafka work. I see the > > crux of > > the issue as this: if you have a bunch of systems that essentially > > expose > > the same functionality there is value in standardizing the protocol > > by > > which they are accessed to help decouple interface from > > implementation. Of > > course I think it is better still to end up with a single good > > implementation (e.g. Linux rather than Posix). But invariably the > > protocol > > dictates the feature set, which dictates the implementation, and so > > this > > only really works if the systems have the same feature set and > > similar > > enough implementations. This becomes true in a domain over time as > > people > > learn the best way to build that kind of system, and all the > > systems > > converge to that. > > > > The reason we have not been pursuing this is that I think the set > > of > > functionality we are aiming for is a little different than what > > most > > message brokers have. Basically the idea we have is to attempt to > > re-imagine "messaging" or asynchronous processing infrastructure as > > a > > distributed, replicated, partitioned "commit log". This is > > different enough > > from what other system do that attempting to support a standardized > > protocol is unlikely to work out well. For example, the consumer > > balancing > > we do is not modeled in AMQP, and there are many AMQP features that > > Kafka > > doesn't have. > > > > Basically I don't really see other messaging systems as being fully > > formed > > distributed systems that acts as a *cluster* (rather than an > > ensemble of > > brokers). Conceptually when people program to, say, HDFS, you > > largely > > forget that under the covers it is a collection of data nodes and > > you think > > about it as a single entity. There are a number of points in the > > design > > that make this possible (and a number of areas where HDFS falls > > short). I > > think there is a lot to be gained by bringing to bear this modern > > style of > > distributed systems design in this space. Needless to say people > > who work > > on these other systems totally disagree with this assessment, so it > > is a > > bit of an experiment. > > > > I think an interesting analogy is to databases. Relational > > databases took > > this path to some extent. They started out with a very diverse > > feature set, > > and eventually converged to a fairly standard set of functionality > > with > > reasonable compatibility protocols (ODBC, JDBC). Distributed > > databases, > > though, are much more constrained and virtually always fail when > > they > > attempt to be compatible with centralized RDBMS's because they just > > can't > > do all the same stuff (but can do other things). I think as the > > distributed > > database space settles down it will become clear how to provide > > some kind > > of general protocol to standardize access, but trying to do that > > too soon > > wouldn't really help. > > > > Another option, instead of making Kafka an AMPQ system, would be to > > try to > > make Kafka a multi-protocol system that supported many protocol's > > natively, > > sharing basic socket infrastructure. I have been down this path and > > it is a > > very hard road. I would not like to do that again. > > > > That said it would be very interesting to see how well AMQP could > > be mapped > > to Kafka semantics, and there is nothing that prevents this > > experiment from > > happening outside the main codebase. It is totally possible to just > > call > > new KafkaServer(), access all the business logic from there, and > > wrap that > > in AMQP, REST, or any other protocol. That might be a good way to > > conduct > > the experiment if anyone is interested in trying it. > > > > Cheers, > > > > -Jay > > > > > > > > > > > > > > > > On Mon, Oct 1, 2012 at 12:07 PM, William Henry <whe...@redhat.com> > > wrote: > > > > > Hi, > > > > > > Has anyone looked at this email? Anyone care to express an > > > opinion? > > > > > > It seems like Apache has ActiveMQ and Qpid, which are already > > > working on > > > integrating, and now Kafka. Kafka might benefit by using > > > Qpid/Proton just > > > as ActiveMQ is trying to integrate with Qpid/Proton. > > > > > > If folks are interested I'd be willing to take a look at the > > > integration > > > and help out. > > > > > > Best regards, > > > William > > > > > > ----- Original Message ----- > > > > Hi, > > > > > > > > > > > > Has anyone looked at integrating kafka with Apache Qpid to get > > > > AMQP > > > > support? > > > > > > > > > > > > Best, > > > > William > > > > > >
