Good point about compressed message sets. I think that that works and is simpler. We might still need the txn id to be able to do the application to the hashmap atomically, but this depends on a number of details that aren't really spec'd out, in particular how the replicas keep their hashmap fed and how we fail over when mastership changes.
-Jay On Tue, Dec 18, 2012 at 8:05 AM, Jun Rao <jun...@gmail.com> wrote: > Thanks for the proposal. Added a couple of comments to the wiki. > > Thanks, > > Jun > > On Mon, Dec 17, 2012 at 10:45 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > Hey Guys, > > > > David has made a bunch of progress on the offset commit api > implementation. > > > > Since this is a public API it would be good to do as much thinking > up-front > > as possible to minimize future iterations. > > > > It would be great if folks could do the following: > > 1. Read the wiki here: > > https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management > > 2. Check out the code David wrote here: > > https://issues.apache.org/jira/browse/KAFKA-657 > > > > In particular our hope is that this API can act as the first step in > > scaling the way we store offsets (ZK is not really very appropriate for > > this). This of course requires having some plan in mind for offset > storage. > > I have written (and then after getting some initial feedback, rewritten) > a > > section in the above wiki on how this might work. > > > > If no one says anything I will be taking a slightly modified patch that > > adds this functionality on trunk as soon as David gets in a few minor > > tweaks. > > > > -Jay > > >
