We have in fact built something like this. No plans as yet to release, but I think we would like to eventually.
On Wed, Nov 2, 2016 at 2:43 AM, Khurrum Nasim <khurrumnas...@gmail.com> wrote: > As part of implementing the kafka subscriber interface, I am wondering is > there anyone use DL stream for storing the offsets? > > For example, if I have N streams (0..N-1), I need to track the read offset > for each stream and store them somewhere. I can probably use other external > services (like any key/value store) to store the offset. But it would > introduce extra dependencies. I am thinking if I have a map of <stream, > offset> and periodically flush the map into a separate stream (let's say > __offset_ stream). With proper truncation/checkpoint mechanism, it would be > very fast. > > This use case here is a very standard replicated state machine. I am also > wondering do you guys think of providing some common library on > distributedlog to simply implementing state machines. > > - KN >
