I'd like to understand the use case more - why would you want to do this exactly (use AMQP + Kafka) other than just because?
On Tue, Oct 2, 2012 at 9:22 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > I looked into AMQP when I was first starting Kafka work. I see the crux of > the issue as this: if you have a bunch of systems that essentially expose > the same functionality there is value in standardizing the protocol by > which they are accessed to help decouple interface from implementation. Of > course I think it is better still to end up with a single good > implementation (e.g. Linux rather than Posix). But invariably the protocol > dictates the feature set, which dictates the implementation, and so this > only really works if the systems have the same feature set and similar > enough implementations. This becomes true in a domain over time as people > learn the best way to build that kind of system, and all the systems > converge to that. > > The reason we have not been pursuing this is that I think the set of > functionality we are aiming for is a little different than what most > message brokers have. Basically the idea we have is to attempt to > re-imagine "messaging" or asynchronous processing infrastructure as a > distributed, replicated, partitioned "commit log". This is different enough > from what other system do that attempting to support a standardized > protocol is unlikely to work out well. For example, the consumer balancing > we do is not modeled in AMQP, and there are many AMQP features that Kafka > doesn't have. > > Basically I don't really see other messaging systems as being fully formed > distributed systems that acts as a *cluster* (rather than an ensemble of > brokers). Conceptually when people program to, say, HDFS, you largely > forget that under the covers it is a collection of data nodes and you think > about it as a single entity. There are a number of points in the design > that make this possible (and a number of areas where HDFS falls short). I > think there is a lot to be gained by bringing to bear this modern style of > distributed systems design in this space. Needless to say people who work > on these other systems totally disagree with this assessment, so it is a > bit of an experiment. > > I think an interesting analogy is to databases. Relational databases took > this path to some extent. They started out with a very diverse feature set, > and eventually converged to a fairly standard set of functionality with > reasonable compatibility protocols (ODBC, JDBC). Distributed databases, > though, are much more constrained and virtually always fail when they > attempt to be compatible with centralized RDBMS's because they just can't > do all the same stuff (but can do other things). I think as the distributed > database space settles down it will become clear how to provide some kind > of general protocol to standardize access, but trying to do that too soon > wouldn't really help. > > Another option, instead of making Kafka an AMPQ system, would be to try to > make Kafka a multi-protocol system that supported many protocol's natively, > sharing basic socket infrastructure. I have been down this path and it is a > very hard road. I would not like to do that again. > > That said it would be very interesting to see how well AMQP could be mapped > to Kafka semantics, and there is nothing that prevents this experiment from > happening outside the main codebase. It is totally possible to just call > new KafkaServer(), access all the business logic from there, and wrap that > in AMQP, REST, or any other protocol. That might be a good way to conduct > the experiment if anyone is interested in trying it. > > Cheers, > > -Jay > > > > > > > > On Mon, Oct 1, 2012 at 12:07 PM, William Henry <whe...@redhat.com> wrote: > > > Hi, > > > > Has anyone looked at this email? Anyone care to express an opinion? > > > > It seems like Apache has ActiveMQ and Qpid, which are already working on > > integrating, and now Kafka. Kafka might benefit by using Qpid/Proton just > > as ActiveMQ is trying to integrate with Qpid/Proton. > > > > If folks are interested I'd be willing to take a look at the integration > > and help out. > > > > Best regards, > > William > > > > ----- Original Message ----- > > > Hi, > > > > > > > > > Has anyone looked at integrating kafka with Apache Qpid to get AMQP > > > support? > > > > > > > > > Best, > > > William > > >
