Hi, In the big data ecosystem, I have started to use kafka, essentially, as a: - unordered list/array, and - a cluster-wide pipe
I guess you could argue that any message bus product is a simple array/pipe but kafka's scale and model make things so easy :) I am wondering if there are any abstractions on top of kafka that will let me use kafka to store/organize other simple data structures like a linked-list? I have a use case for massive linked list that can easily grow to tens of gigabytes and could easily use - (1) redundancy (2) multiple producers/consumers working on processing the list (implemented over spark, storm etc). Any ideas? Maybe maintain a linked-list of offsets in another store like ZooKeeper or a NoSQL DB while store the messages on kafka? Thanks, - Tim
