I think we are going to go with Kafka itself hopefully if the code isn't too hard to update - it already has everything we need, we just change it's receive message operation from:
receive msg => write into log file receive msg => write into log file and get offset, write offset into index log file where index log file is just another topic that contains 64 bit offsets of the original log file. of course with batching and sendfile calls this may be trickier than we anticipate…. On Thu, Sep 22, 2011 at 8:17 AM, Jeffrey Damick <jeffreydam...@gmail.com>wrote: > This was something i've asked for in the past as well. To neha's comment, > sounds like you'd need some kind of table to maintain the list of offsets > and which segment they live in? > > > > On Thu, Sep 22, 2011 at 9:07 AM, Chris Burroughs > <chris.burrou...@gmail.com>wrote: > > > On 09/21/2011 10:06 PM, Taylor Gautier wrote: > > > I see that kafka-87 addresses this with a request for having a time > based > > > index, this would be relatively useful, but I also would like to have a > > way > > > to go back say 1,000 messages. Other than walking backwards one > segment > > at > > > a time, can then scanning forward from there, do you have any > suggestions > > > how this might be done or is it also a feature request? > > > > Could you elaborate a little on your use case where you need to rewind > > by a fixed number of messages? > > >
