Rohit, Thanks for the feedback!
I don't understand your assertion that the two things need to be done atomically. As I see it, as long as the data partition can provide it's end offset to the producer after the last message in the transaction has been written, there should be no problem. Since writing the transaction to the control partition essentially IS the commit, I don't see a need for any separate atomic transactions. Of course, I could be looking at it wrong. --Tom On Thu, Nov 15, 2012 at 4:22 PM, Rohit Prasad <rohit.prasa...@gmail.com> wrote: > Thanks Tom for sharing your idea. > > I see the below condition as necessary for the solution to be full proof - > These two things need to be done atomically by server. > After writing the data into data partition, > 1. Updating offset of data partition, and > 2. Writing the updated offset to control partition. > > This "may" not be possible to do atomically. (since we need to write to two > different files.) I have glimpsed over server code but dont have expertise. > Can someone who is more versed with the server code ack/nack ? > > Also, can someone validate this: > 1. Write to partition as usual. The data should contain a uid based on > timestamp/counter. > 2. On producer re-connects/timeouts or server crashes, the producer should > ask for last committed message, and extract the uid to decide if last > message went through. Or may be get this from Zookeeper. > > > Thanks, > Rohit
