On 28 May 2013, at 23:38, Mark <[email protected]> wrote: > I actually looked into Kafka quite some time ago and I think we passed on it > because it didn't have much ruby support (That may have changed by now).
Ruby support unfortunately continues to be weak. I actually wrote a brand new Kafka producer client for Ruby (not yet open source), but from Kafka 0.7 to 0.8 the wire protocol is changing (as replication & high availability features are added), which means I'm going to have to re-do it. The Kafka project officially supports producers using the JVM or the C client (the latter could be embedded in Ruby using FFI), and consumers using the JVM client only. However, that doesn't rule out other languages. LinkedIn actually operates a large service that consumes Kafka using the JVM client, writes the Avro messages to stdout as JSON, and then pipes that into a python process' stdin. Works surprisingly well! Martin
