We thought of another idea - if you treat the messages as a reverse linked list, then you write the offset of the previous message at the end of each message. Thus a message format would look like:
header msg offset off msg This would allow for walking backwards at the cost of linear scans - if you assume that you don't do this often enough this wouldn't have any cost at write or read time except a tiny (negligible) penalty in space. On Thu, Sep 22, 2011 at 8:22 AM, Taylor Gautier <tgaut...@tagged.com> wrote: > I think we are going to go with Kafka itself hopefully if the code isn't > too hard to update - it already has everything we need, we just change it's > receive message operation from: > > receive msg => write into log file > > receive msg => write into log file and get offset, write offset into index > log file > > where index log file is just another topic that contains 64 bit offsets of > the original log file. > > of course with batching and sendfile calls this may be trickier than we > anticipate…. > > On Thu, Sep 22, 2011 at 8:17 AM, Jeffrey Damick > <jeffreydam...@gmail.com>wrote: > >> This was something i've asked for in the past as well. To neha's comment, >> sounds like you'd need some kind of table to maintain the list of offsets >> and which segment they live in? >> >> >> >> On Thu, Sep 22, 2011 at 9:07 AM, Chris Burroughs >> <chris.burrou...@gmail.com>wrote: >> >> > On 09/21/2011 10:06 PM, Taylor Gautier wrote: >> > > I see that kafka-87 addresses this with a request for having a time >> based >> > > index, this would be relatively useful, but I also would like to have >> a >> > way >> > > to go back say 1,000 messages. Other than walking backwards one >> segment >> > at >> > > a time, can then scanning forward from there, do you have any >> suggestions >> > > how this might be done or is it also a feature request? >> > >> > Could you elaborate a little on your use case where you need to rewind >> > by a fixed number of messages? >> > >> > >
