Re: Data Structure abstractions over kafka

2015-07-13 Thread Ewen Cheslack-Postava
Tim, Kafka can be used as a key-value store if you turn on log compaction: http://kafka.apache.org/documentation.html#compaction You need to be careful with that since it's purely last-writer-wins and doesn't have anything like CAS that might help you manage concurrent writers, but the basic

Data Structure abstractions over kafka

2015-07-13 Thread Tim Smith
Hi, In the big data ecosystem, I have started to use kafka, essentially, as a: - unordered list/array, and - a cluster-wide pipe I guess you could argue that any message bus product is a simple array/pipe but kafka's scale and model make things so easy :) I am wondering if there are any