Re: Consuming from Kafka but don't need to save offsets

2015-07-20 Thread tao xiao
James, You can reference confluent IO schema registry implementation. http://docs.confluent.io/1.0/schema-registry/docs/index.html It does similar thing as what you described. A REST front end that serves data from a compacted topic and HA is also provided in the solution. On Tue, 21 Jul 2015 at

Consuming from Kafka but don't need to save offsets

2015-07-20 Thread James Cheng
Hi, I have a web service that serves up some data that it obtains from a kafka topic. When the process starts up, it wants to load the entire kafka topic into memory, and serve the data up from an in-memory hashtable. The data in the topic has primary keys and is log compacted, and so the total