----- Original Message ----- > I'm not exactly sure about why talking the same at the wire level is > an > explicit goal; if raw blazing speed is also equally important. > > Removing the wireformat from Kafka could be done through abstraction; > but > that would incur reinterpretation costs(talking native Kafka) and > take a > performance hit. > > I could make a similar argument over the second goal as well. It is > not > apparent that solving ALL problems through abstraction and then > universally > accepting a performance hit is that ideal. It may make > purchasing/acquiring > easier; but by adhering to the lowest common denominator. > > Wouldn't it be better to make explicit tradeoffs for one-offs based > on > specific needs? i.e. if your architecture doesnt require zookeeper in > Kafka > for coordination, reduce the complexity. Don't force complexity in > all > cases.
Sure but if AMQP solves all the goals then why do so-called best-of-breed "proprietary"? Who said lowest common-denominator? If AMQP is lowest common-denominator then it has failed. Not sure how this is complexity either. Surely reusing an existing tried and tested protocol instead of developing and maintaining a new one is less complex. > > In other words, Kafka is different enough from other messaging > systems that > to enforce a common contract (aka amqp) ,without incurring a > significant > performance hit, would be very challenging. There is no metrics to say this would be a performance hit at all. Again I'm not saying it is the right fit but I don't think we can conclude that without investigating. And I haven't seen anything that would suggest it can't fit. William > On Oct 2, 2012 3:22 PM, "William Henry" <whe...@redhat.com> wrote: > > > > > > > ----- Original Message ----- > > > I looked into AMQP when I was first starting Kafka work. I see > > > the > > > crux of > > > the issue as this: if you have a bunch of systems that > > > essentially > > > expose > > > the same functionality there is value in standardizing the > > > protocol > > > by > > > which they are accessed to help decouple interface from > > > implementation. Of > > > course I think it is better still to end up with a single good > > > implementation (e.g. Linux rather than Posix). But invariably the > > > protocol > > > dictates the feature set, which dictates the implementation, and > > > so > > > this > > > only really works if the systems have the same feature set and > > > similar > > > enough implementations. This becomes true in a domain over time > > > as > > > people > > > learn the best way to build that kind of system, and all the > > > systems > > > converge to that. > > > > +1 > > > > > > > > The reason we have not been pursuing this is that I think the set > > > of > > > functionality we are aiming for is a little different than what > > > most > > > message brokers have. Basically the idea we have is to attempt to > > > re-imagine "messaging" or asynchronous processing infrastructure > > > as a > > > distributed, replicated, partitioned "commit log". This is > > > different > > > enough > > > from what other system do that attempting to support a > > > standardized > > > protocol is unlikely to work out well. For example, the consumer > > > balancing > > > we do is not modeled in AMQP, and there are many AMQP features > > > that > > > Kafka > > > doesn't have. > > > > I need to understand your consumer balancing a bit more but AMQP is > > designed not to be another MOM like traditional broker based > > messaging > > systems, though it does support that model. > > > > I like to explain the goals of AMQP to be threefold (some may argue > > differently): > > > > 1) A Standard wire protocol for interoperability. i.e. have all > > messaging > > systems speak the same on the wire. > > 2) Handle all messaging use cases well - i.e. not just asynch, not > > just > > fanout, not just pub/sub but instead do it all so that AMQP is > > applicable > > to all use cases. Let's not have a "we do AMQP everywhere except X > > because > > it does do X very well. > > 3) Must be fast. Even if it does 1 and 2 very well it will not be > > adopted > > by a wide range of applications. > > > > So if by consumer balancing you mean multiple consumers feeding off > > a > > particular address/source/publisher/producer etc. then AMQP does > > manage > > that model. > > > > > > > > > > Basically I don't really see other messaging systems as being > > > fully > > > formed > > > distributed systems that acts as a *cluster* (rather than an > > > ensemble > > > of > > > brokers). > > > > This is exactly what we in the Qpid community are working towards > > right > > now. I think AMQP as a protocol under Kafka and exploiting Kafka's > > framework is a great idea. > > > > Please look at the new Qpid/Proton work and some of Ted Ross's > > (cc-ed) > > router work. > > > > > Conceptually when people program to, say, HDFS, you largely > > > forget that under the covers it is a collection of data nodes and > > > you > > > think > > > about it as a single entity. There are a number of points in the > > > design > > > that make this possible (and a number of areas where HDFS falls > > > short). I > > > think there is a lot to be gained by bringing to bear this modern > > > style of > > > distributed systems design in this space. Needless to say people > > > who > > > work > > > on these other systems totally disagree with this assessment, so > > > it > > > is a > > > bit of an experiment. > > > > This is very interesting to me and some of the customers (at least > > 2) I > > work with. > > > > > > > > I think an interesting analogy is to databases. Relational > > > databases > > > took > > > this path to some extent. They started out with a very diverse > > > feature set, > > > and eventually converged to a fairly standard set of > > > functionality > > > with > > > reasonable compatibility protocols (ODBC, JDBC). Distributed > > > databases, > > > though, are much more constrained and virtually always fail when > > > they > > > attempt to be compatible with centralized RDBMS's because they > > > just > > > can't > > > do all the same stuff (but can do other things). I think as the > > > distributed > > > database space settles down it will become clear how to provide > > > some > > > kind > > > of general protocol to standardize access, but trying to do that > > > too > > > soon > > > wouldn't really help. > > > > > > Another option, instead of making Kafka an AMPQ system, would be > > > to > > > try to > > > make Kafka a multi-protocol system that supported many protocol's > > > natively, > > > sharing basic socket infrastructure. I have been down this path > > > and > > > it is a > > > very hard road. I would not like to do that again. > > > > I understand that. > > > > > > > > That said it would be very interesting to see how well AMQP could > > > be > > > mapped > > > to Kafka semantics, and there is nothing that prevents this > > > experiment from > > > happening outside the main codebase. It is totally possible to > > > just > > > call > > > new KafkaServer(), access all the business logic from there, and > > > wrap > > > that > > > in AMQP, REST, or any other protocol. That might be a good way to > > > conduct > > > the experiment if anyone is interested in trying it. > > > > > > > I would love to take a look at this. Any pointer on where an > > integration > > point might be would be welcome. There is so much work in the AMQP > > and > > Qpid communities that Kafka could benefit from. You could > > concentrate on > > the "cluster" model and let Qpid/Proton handle the payload > > distribution on > > the wire. > > > > I'm willing to take the risk that I might be wrong but right now I > > don't > > see where AMQP would fall down in this case. > > > > Best regards, > > William > > > > > Cheers, > > > > > > -Jay > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 1, 2012 at 12:07 PM, William Henry > > > <whe...@redhat.com> > > > wrote: > > > > > > > Hi, > > > > > > > > Has anyone looked at this email? Anyone care to express an > > > > opinion? > > > > > > > > It seems like Apache has ActiveMQ and Qpid, which are already > > > > working on > > > > integrating, and now Kafka. Kafka might benefit by using > > > > Qpid/Proton just > > > > as ActiveMQ is trying to integrate with Qpid/Proton. > > > > > > > > If folks are interested I'd be willing to take a look at the > > > > integration > > > > and help out. > > > > > > > > Best regards, > > > > William > > > > > > > > ----- Original Message ----- > > > > > Hi, > > > > > > > > > > > > > > > Has anyone looked at integrating kafka with Apache Qpid to > > > > > get > > > > > AMQP > > > > > support? > > > > > > > > > > > > > > > Best, > > > > > William > > > > > > > > > >
