Also FYI. I plan to be at ApacheCon Eu and would love to talk to Kafka folks there perhaps over a pint.
:) William ----- Original Message ----- > > > ----- Original Message ----- > > I'd like to understand the use case more - why would you want to do > > this > > exactly (use AMQP + Kafka) other than just because? > > Good point. > > See my previous email. It's not a just because. I think it is > precisely the kind of synergy that projects like Qpid/Proton are > working towards. We'd love to see AMQP 1.0 be ubiquitous. However we > understand that at a higher level there are frameworks and > architectures that have specific value to users. > > I've recently been working on integrating Qpid/Proton with OpenMAMA > as an example. And I'm working on interoperability with Microsoft's > and other middleware products. > > Again, the goal of AMQP was to handle all messaging use cases and if > it can't handle Kafka's workload then AMQP needs to address that. > > > So I could see: Kafka using Proton which provides AMQP 1.0 > > Best, > William > > > > > > On Tue, Oct 2, 2012 at 9:22 AM, Jay Kreps <jay.kr...@gmail.com> > > wrote: > > > > > I looked into AMQP when I was first starting Kafka work. I see > > > the > > > crux of > > > the issue as this: if you have a bunch of systems that > > > essentially > > > expose > > > the same functionality there is value in standardizing the > > > protocol > > > by > > > which they are accessed to help decouple interface from > > > implementation. Of > > > course I think it is better still to end up with a single good > > > implementation (e.g. Linux rather than Posix). But invariably the > > > protocol > > > dictates the feature set, which dictates the implementation, and > > > so > > > this > > > only really works if the systems have the same feature set and > > > similar > > > enough implementations. This becomes true in a domain over time > > > as > > > people > > > learn the best way to build that kind of system, and all the > > > systems > > > converge to that. > > > > > > The reason we have not been pursuing this is that I think the set > > > of > > > functionality we are aiming for is a little different than what > > > most > > > message brokers have. Basically the idea we have is to attempt to > > > re-imagine "messaging" or asynchronous processing infrastructure > > > as > > > a > > > distributed, replicated, partitioned "commit log". This is > > > different enough > > > from what other system do that attempting to support a > > > standardized > > > protocol is unlikely to work out well. For example, the consumer > > > balancing > > > we do is not modeled in AMQP, and there are many AMQP features > > > that > > > Kafka > > > doesn't have. > > > > > > Basically I don't really see other messaging systems as being > > > fully > > > formed > > > distributed systems that acts as a *cluster* (rather than an > > > ensemble of > > > brokers). Conceptually when people program to, say, HDFS, you > > > largely > > > forget that under the covers it is a collection of data nodes and > > > you think > > > about it as a single entity. There are a number of points in the > > > design > > > that make this possible (and a number of areas where HDFS falls > > > short). I > > > think there is a lot to be gained by bringing to bear this modern > > > style of > > > distributed systems design in this space. Needless to say people > > > who work > > > on these other systems totally disagree with this assessment, so > > > it > > > is a > > > bit of an experiment. > > > > > > I think an interesting analogy is to databases. Relational > > > databases took > > > this path to some extent. They started out with a very diverse > > > feature set, > > > and eventually converged to a fairly standard set of > > > functionality > > > with > > > reasonable compatibility protocols (ODBC, JDBC). Distributed > > > databases, > > > though, are much more constrained and virtually always fail when > > > they > > > attempt to be compatible with centralized RDBMS's because they > > > just > > > can't > > > do all the same stuff (but can do other things). I think as the > > > distributed > > > database space settles down it will become clear how to provide > > > some kind > > > of general protocol to standardize access, but trying to do that > > > too soon > > > wouldn't really help. > > > > > > Another option, instead of making Kafka an AMPQ system, would be > > > to > > > try to > > > make Kafka a multi-protocol system that supported many protocol's > > > natively, > > > sharing basic socket infrastructure. I have been down this path > > > and > > > it is a > > > very hard road. I would not like to do that again. > > > > > > That said it would be very interesting to see how well AMQP could > > > be mapped > > > to Kafka semantics, and there is nothing that prevents this > > > experiment from > > > happening outside the main codebase. It is totally possible to > > > just > > > call > > > new KafkaServer(), access all the business logic from there, and > > > wrap that > > > in AMQP, REST, or any other protocol. That might be a good way to > > > conduct > > > the experiment if anyone is interested in trying it. > > > > > > Cheers, > > > > > > -Jay > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 1, 2012 at 12:07 PM, William Henry > > > <whe...@redhat.com> > > > wrote: > > > > > > > Hi, > > > > > > > > Has anyone looked at this email? Anyone care to express an > > > > opinion? > > > > > > > > It seems like Apache has ActiveMQ and Qpid, which are already > > > > working on > > > > integrating, and now Kafka. Kafka might benefit by using > > > > Qpid/Proton just > > > > as ActiveMQ is trying to integrate with Qpid/Proton. > > > > > > > > If folks are interested I'd be willing to take a look at the > > > > integration > > > > and help out. > > > > > > > > Best regards, > > > > William > > > > > > > > ----- Original Message ----- > > > > > Hi, > > > > > > > > > > > > > > > Has anyone looked at integrating kafka with Apache Qpid to > > > > > get > > > > > AMQP > > > > > support? > > > > > > > > > > > > > > > Best, > > > > > William > > > > > > > > > >
