Hi Daniel, Thanks for sharing this. Looks like a great project. I probably don't know enough to give a great answer but will throw in my 2c anyway.
I think Kafka prioritizes throughput and aeron prioritizes latency. As you mentioned, maybe Aeron could replace the current Kafka TCP protocol. The issues I foresee are with things like batching and compression, supported by Kafka. These features trade latency for throughput to great effect but may be something that Aeron protocol would never support because of it's focus on latency first. Cheers, Roger On Mon, Nov 17, 2014 at 11:12 AM, Daniel Compton <d...@danielcompton.net> wrote: > Recently there has been some interesting discussion on the Mechanical > Sympathy mailing list about Martin Thompson’s new extremely high > performance, extremely low latency, open source messaging system called > Aeron. For those of you that don’t know, Martin Thompson is an expert on > wringing as much throughput and as little latency as possible out of > hardware for financial firms so his background made me very interested in > this. > > The high level gist is that Aeron has publishers that write messages to a > partitioned, replicated, immutable log, and subscribers that are delivered > messages from the immutable log using message offsets. Sound familiar? > There are a few differences ( > https://groups.google.com/d/msg/mechanical-sympathy/fr7moLcsuiI/ZfUMk4QUbN8J), > the key ones being that there are no brokers in this system, and that Aeron > is a pure OSI layer 4 implementation. > > At first it seems like Aeron is a competing product to Kafka, and indeed > for some cases it could be competing. However because it is implemented as > a layer 4 transport, after thinking about it for a while, I think it could > be implemented into Kafka as an alternative to the TCP transport that Kafka > uses. The semantics would need to be thought through, but I think you could > implement the brokered model on top of this, as well as supporting point to > point subscriptions bypassing the brokers. > > I wanted to bring this to the groups attention and see what others > thought. Aeron is still pretty new, and there’s a lot to digest in this so > I wouldn’t recommend trying to land it in 0.8.2 :), but I thought it’s > worth considering it for future development. > > The Youtube video where he talks about it is at > https://www.youtube.com/watch?v=tM4YskS94b0 and the GitHub repo is at > https://github.com/real-logic/Aeron. Discussion is spread across a few > threads but starts here: > https://groups.google.com/d/msg/mechanical-sympathy/fr7moLcsuiI/04unxUfIJ8sJ > and > https://groups.google.com/forum/#!topic/mechanical-sympathy/GrEce0gP7RU. > > Daniel. > >
