+1. I can help you in the same timeframe. William
On Oct 2, 2012, at 8:32 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > I think a first step would be to do a detailed comparison of kafka > semantics and the AMQP model and see how good the fit is. If no one else is > game I would be willing to do this, though i probably wouldn't start for > about a month. I think this would be more meaningful since unless we know > the explicit drawbacks it is hard to really think concretely about the pros > and cons. > > -Jay > > On Tue, Oct 2, 2012 at 5:27 PM, William Henry <whe...@redhat.com> wrote: > >> >> >> ----- Original Message ----- >>> I'm not exactly sure about why talking the same at the wire level is >>> an >>> explicit goal; if raw blazing speed is also equally important. >>> >>> Removing the wireformat from Kafka could be done through abstraction; >>> but >>> that would incur reinterpretation costs(talking native Kafka) and >>> take a >>> performance hit. >>> >>> I could make a similar argument over the second goal as well. It is >>> not >>> apparent that solving ALL problems through abstraction and then >>> universally >>> accepting a performance hit is that ideal. It may make >>> purchasing/acquiring >>> easier; but by adhering to the lowest common denominator. >>> >>> Wouldn't it be better to make explicit tradeoffs for one-offs based >>> on >>> specific needs? i.e. if your architecture doesnt require zookeeper in >>> Kafka >>> for coordination, reduce the complexity. Don't force complexity in >>> all >>> cases. >> >> Sure but if AMQP solves all the goals then why do so-called best-of-breed >> "proprietary"? >> >> Who said lowest common-denominator? If AMQP is lowest common-denominator >> then it has failed. >> >> Not sure how this is complexity either. Surely reusing an existing tried >> and tested protocol instead of developing and maintaining a new one is less >> complex. >> >>> >>> In other words, Kafka is different enough from other messaging >>> systems that >>> to enforce a common contract (aka amqp) ,without incurring a >>> significant >>> performance hit, would be very challenging. >> >> There is no metrics to say this would be a performance hit at all. >> >> Again I'm not saying it is the right fit but I don't think we can conclude >> that without investigating. And I haven't seen anything that would suggest >> it can't fit. >> >> William >> >> >>> On Oct 2, 2012 3:22 PM, "William Henry" <whe...@redhat.com> wrote: >>> >>>> >>>> >>>> ----- Original Message ----- >>>>> I looked into AMQP when I was first starting Kafka work. I see >>>>> the >>>>> crux of >>>>> the issue as this: if you have a bunch of systems that >>>>> essentially >>>>> expose >>>>> the same functionality there is value in standardizing the >>>>> protocol >>>>> by >>>>> which they are accessed to help decouple interface from >>>>> implementation. Of >>>>> course I think it is better still to end up with a single good >>>>> implementation (e.g. Linux rather than Posix). But invariably the >>>>> protocol >>>>> dictates the feature set, which dictates the implementation, and >>>>> so >>>>> this >>>>> only really works if the systems have the same feature set and >>>>> similar >>>>> enough implementations. This becomes true in a domain over time >>>>> as >>>>> people >>>>> learn the best way to build that kind of system, and all the >>>>> systems >>>>> converge to that. >>>> >>>> +1 >>>> >>>>> >>>>> The reason we have not been pursuing this is that I think the set >>>>> of >>>>> functionality we are aiming for is a little different than what >>>>> most >>>>> message brokers have. Basically the idea we have is to attempt to >>>>> re-imagine "messaging" or asynchronous processing infrastructure >>>>> as a >>>>> distributed, replicated, partitioned "commit log". This is >>>>> different >>>>> enough >>>>> from what other system do that attempting to support a >>>>> standardized >>>>> protocol is unlikely to work out well. For example, the consumer >>>>> balancing >>>>> we do is not modeled in AMQP, and there are many AMQP features >>>>> that >>>>> Kafka >>>>> doesn't have. >>>> >>>> I need to understand your consumer balancing a bit more but AMQP is >>>> designed not to be another MOM like traditional broker based >>>> messaging >>>> systems, though it does support that model. >>>> >>>> I like to explain the goals of AMQP to be threefold (some may argue >>>> differently): >>>> >>>> 1) A Standard wire protocol for interoperability. i.e. have all >>>> messaging >>>> systems speak the same on the wire. >>>> 2) Handle all messaging use cases well - i.e. not just asynch, not >>>> just >>>> fanout, not just pub/sub but instead do it all so that AMQP is >>>> applicable >>>> to all use cases. Let's not have a "we do AMQP everywhere except X >>>> because >>>> it does do X very well. >>>> 3) Must be fast. Even if it does 1 and 2 very well it will not be >>>> adopted >>>> by a wide range of applications. >>>> >>>> So if by consumer balancing you mean multiple consumers feeding off >>>> a >>>> particular address/source/publisher/producer etc. then AMQP does >>>> manage >>>> that model. >>>> >>>> >>>>> >>>>> Basically I don't really see other messaging systems as being >>>>> fully >>>>> formed >>>>> distributed systems that acts as a *cluster* (rather than an >>>>> ensemble >>>>> of >>>>> brokers). >>>> >>>> This is exactly what we in the Qpid community are working towards >>>> right >>>> now. I think AMQP as a protocol under Kafka and exploiting Kafka's >>>> framework is a great idea. >>>> >>>> Please look at the new Qpid/Proton work and some of Ted Ross's >>>> (cc-ed) >>>> router work. >>>> >>>>> Conceptually when people program to, say, HDFS, you largely >>>>> forget that under the covers it is a collection of data nodes and >>>>> you >>>>> think >>>>> about it as a single entity. There are a number of points in the >>>>> design >>>>> that make this possible (and a number of areas where HDFS falls >>>>> short). I >>>>> think there is a lot to be gained by bringing to bear this modern >>>>> style of >>>>> distributed systems design in this space. Needless to say people >>>>> who >>>>> work >>>>> on these other systems totally disagree with this assessment, so >>>>> it >>>>> is a >>>>> bit of an experiment. >>>> >>>> This is very interesting to me and some of the customers (at least >>>> 2) I >>>> work with. >>>> >>>>> >>>>> I think an interesting analogy is to databases. Relational >>>>> databases >>>>> took >>>>> this path to some extent. They started out with a very diverse >>>>> feature set, >>>>> and eventually converged to a fairly standard set of >>>>> functionality >>>>> with >>>>> reasonable compatibility protocols (ODBC, JDBC). Distributed >>>>> databases, >>>>> though, are much more constrained and virtually always fail when >>>>> they >>>>> attempt to be compatible with centralized RDBMS's because they >>>>> just >>>>> can't >>>>> do all the same stuff (but can do other things). I think as the >>>>> distributed >>>>> database space settles down it will become clear how to provide >>>>> some >>>>> kind >>>>> of general protocol to standardize access, but trying to do that >>>>> too >>>>> soon >>>>> wouldn't really help. >>>>> >>>>> Another option, instead of making Kafka an AMPQ system, would be >>>>> to >>>>> try to >>>>> make Kafka a multi-protocol system that supported many protocol's >>>>> natively, >>>>> sharing basic socket infrastructure. I have been down this path >>>>> and >>>>> it is a >>>>> very hard road. I would not like to do that again. >>>> >>>> I understand that. >>>> >>>>> >>>>> That said it would be very interesting to see how well AMQP could >>>>> be >>>>> mapped >>>>> to Kafka semantics, and there is nothing that prevents this >>>>> experiment from >>>>> happening outside the main codebase. It is totally possible to >>>>> just >>>>> call >>>>> new KafkaServer(), access all the business logic from there, and >>>>> wrap >>>>> that >>>>> in AMQP, REST, or any other protocol. That might be a good way to >>>>> conduct >>>>> the experiment if anyone is interested in trying it. >>>> >>>> I would love to take a look at this. Any pointer on where an >>>> integration >>>> point might be would be welcome. There is so much work in the AMQP >>>> and >>>> Qpid communities that Kafka could benefit from. You could >>>> concentrate on >>>> the "cluster" model and let Qpid/Proton handle the payload >>>> distribution on >>>> the wire. >>>> >>>> I'm willing to take the risk that I might be wrong but right now I >>>> don't >>>> see where AMQP would fall down in this case. >>>> >>>> Best regards, >>>> William >>>> >>>>> Cheers, >>>>> >>>>> -Jay >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Mon, Oct 1, 2012 at 12:07 PM, William Henry >>>>> <whe...@redhat.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Has anyone looked at this email? Anyone care to express an >>>>>> opinion? >>>>>> >>>>>> It seems like Apache has ActiveMQ and Qpid, which are already >>>>>> working on >>>>>> integrating, and now Kafka. Kafka might benefit by using >>>>>> Qpid/Proton just >>>>>> as ActiveMQ is trying to integrate with Qpid/Proton. >>>>>> >>>>>> If folks are interested I'd be willing to take a look at the >>>>>> integration >>>>>> and help out. >>>>>> >>>>>> Best regards, >>>>>> William >>>>>> >>>>>> ----- Original Message ----- >>>>>>> Hi, >>>>>>> >>>>>>> >>>>>>> Has anyone looked at integrating kafka with Apache Qpid to >>>>>>> get >>>>>>> AMQP >>>>>>> support? >>>>>>> >>>>>>> >>>>>>> Best, >>>>>>> William >>
