Also FYI. I plan to be at ApacheCon Eu and would love to talk to Kafka folks 
there perhaps over a pint.

:)

William

----- Original Message -----
> 
> 
> ----- Original Message -----
> > I'd like to understand the use case more - why would you want to do
> > this
> > exactly (use AMQP + Kafka) other than just because?
> 
> Good point.
> 
> See my previous email. It's not a just because. I think it is
> precisely the kind of synergy that projects like Qpid/Proton are
> working towards. We'd love to see AMQP 1.0 be ubiquitous. However we
> understand that at a higher level there are frameworks and
> architectures that have specific value to users.
> 
> I've recently been working on integrating Qpid/Proton with OpenMAMA
> as an example. And I'm working on interoperability with Microsoft's
> and other middleware products.
> 
> Again, the goal of AMQP was to handle all messaging use cases and if
> it can't handle Kafka's workload then AMQP needs to address that.
> 
> 
> So    I could see:  Kafka using Proton which provides AMQP 1.0
> 
> Best,
> William
> 
> 
> > 
> > On Tue, Oct 2, 2012 at 9:22 AM, Jay Kreps <jay.kr...@gmail.com>
> > wrote:
> > 
> > > I looked into AMQP when I was first starting Kafka work. I see
> > > the
> > > crux of
> > > the issue as this: if you have a bunch of systems that
> > > essentially
> > > expose
> > > the same functionality there is value in standardizing the
> > > protocol
> > > by
> > > which they are accessed to help decouple interface from
> > > implementation. Of
> > > course I think it is better still to end up with a single good
> > > implementation (e.g. Linux rather than Posix). But invariably the
> > > protocol
> > > dictates the feature set, which dictates the implementation, and
> > > so
> > > this
> > > only really works if the systems have the same feature set and
> > > similar
> > > enough implementations. This becomes true in a domain over time
> > > as
> > > people
> > > learn the best way to build that kind of system, and all the
> > > systems
> > > converge to that.
> > >
> > > The reason we have not been pursuing this is that I think the set
> > > of
> > > functionality we are aiming for is a little different than what
> > > most
> > > message brokers have. Basically the idea we have is to attempt to
> > > re-imagine "messaging" or asynchronous processing infrastructure
> > > as
> > > a
> > > distributed, replicated, partitioned "commit log". This is
> > > different enough
> > > from what other system do that attempting to support a
> > > standardized
> > > protocol is unlikely to work out well. For example, the consumer
> > > balancing
> > > we do is not modeled in AMQP, and there are many AMQP features
> > > that
> > > Kafka
> > > doesn't have.
> > >
> > > Basically I don't really see other messaging systems as being
> > > fully
> > > formed
> > > distributed systems that acts as a *cluster* (rather than an
> > > ensemble of
> > > brokers). Conceptually when people program to, say, HDFS, you
> > > largely
> > > forget that under the covers it is a collection of data nodes and
> > > you think
> > > about it as a single entity. There are a number of points in the
> > > design
> > > that make this possible (and a number of areas where HDFS falls
> > > short). I
> > > think there is a lot to be gained by bringing to bear this modern
> > > style of
> > > distributed systems design in this space. Needless to say people
> > > who work
> > > on these other systems totally disagree with this assessment, so
> > > it
> > > is a
> > > bit of an experiment.
> > >
> > > I think an interesting analogy is to databases. Relational
> > > databases took
> > > this path to some extent. They started out with a very diverse
> > > feature set,
> > > and eventually converged to a fairly standard set of
> > > functionality
> > > with
> > > reasonable compatibility protocols (ODBC, JDBC). Distributed
> > > databases,
> > > though, are much more constrained and virtually always fail when
> > > they
> > > attempt to be compatible with centralized RDBMS's because they
> > > just
> > > can't
> > > do all the same stuff (but can do other things). I think as the
> > > distributed
> > > database space settles down it will become clear how to provide
> > > some kind
> > > of general protocol to standardize access, but trying to do that
> > > too soon
> > > wouldn't really help.
> > >
> > > Another option, instead of making Kafka an AMPQ system, would be
> > > to
> > > try to
> > > make Kafka a multi-protocol system that supported many protocol's
> > > natively,
> > > sharing basic socket infrastructure. I have been down this path
> > > and
> > > it is a
> > > very hard road. I would not like to do that again.
> > >
> > > That said it would be very interesting to see how well AMQP could
> > > be mapped
> > > to Kafka semantics, and there is nothing that prevents this
> > > experiment from
> > > happening outside the main codebase. It is totally possible to
> > > just
> > > call
> > > new KafkaServer(), access all the business logic from there, and
> > > wrap that
> > > in AMQP, REST, or any other protocol. That might be a good way to
> > > conduct
> > > the experiment if anyone is interested in trying it.
> > >
> > > Cheers,
> > >
> > > -Jay
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Oct 1, 2012 at 12:07 PM, William Henry
> > > <whe...@redhat.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Has anyone looked at this email?  Anyone care to express an
> > > > opinion?
> > > >
> > > > It seems like Apache has ActiveMQ and Qpid, which are already
> > > > working on
> > > > integrating, and now Kafka. Kafka might benefit by using
> > > > Qpid/Proton just
> > > > as ActiveMQ is trying to integrate with Qpid/Proton.
> > > >
> > > > If folks are interested I'd be willing to take a look at the
> > > > integration
> > > > and help out.
> > > >
> > > > Best regards,
> > > > William
> > > >
> > > > ----- Original Message -----
> > > > > Hi,
> > > > >
> > > > >
> > > > > Has anyone looked at integrating kafka with Apache Qpid to
> > > > > get
> > > > > AMQP
> > > > > support?
> > > > >
> > > > >
> > > > > Best,
> > > > > William
> > > >
> > >
> > 
> 

Reply via email to