The whole problem Google is trying to avoid is the delay associated with persisting data. If you insert the persist operation serially into the delta processing process, then it introduce delays and increased load on the persistence mechanism. Being able to queue up several deltas and persist them in a single bulk operation is much more efficient than requiring each delta to be persisted before proceeding (i.e. 1 milllion 512B write vs 1000 512KB writes).
What I'm proposing allows the Host to persist and process the deltas in parallel. The acknowledgment of a delta, original or applied, would simply indicate reliable storage of that delta at the distant end and not that the delta had been processed. The commit notice is then an indication of reliable storage of the processing result. I agree that, if the Host persisted the application result prior to sending the applied deltas to the remotes, there would be no need for a commit notice. However, doing so would introduce delays between when a delta is created and when it is propagated. The total delay with commit notices is (<time to send original delta> + <time to transform the delta> + <time to transmit the applied delta>). The total delay without commit notices is (<time to send original delta> + <time to transform the delta> + <time to persist original and applied delta> + <time to transmit the applied delta>). What I'm proposing allows us to avoid the persistence delay and avoid the need to rollback a remote server if a host dies before it could commit applied deltas. There is a little added complexity in the propagation logic but it seems manageable to me. The one thing that is missing is a mechanism for error recovery in the case where a delta is lost after persistence due to storage failure. If the host suffers a storage failure and is unable to recover everything from backup, then there is no way to regain a consistent state between hosts and remotes. Personally, I think this is a provider problem and not a protocol problem. -Tad On Thu, Feb 25, 2010 at 11:27 AM, Torben Weis <[email protected]> wrote: > Hi, > > if you persist data on receipt (state 2) you can simply drop the whole > commit notice. In this case the delta is persisted before it is distributed > to any remotes. That is what I proposed in my mail some days ago. As I > understood the Google guys, the sole point of the commit notice is to skip > step 2, complete 3 and 4 and persist after deltas have been sent to the > remotes. This reduces delay, because the delay is not influenced by I/O (if > and only if the server is not constantly on full I/O load). > > Greetings > Torben > > > > > 2010/2/25 Tad Glines <[email protected]> > >> I've been thinking about how federation should work with regard to the >> commit notice. >> >> Here's a scenario to explain my thinking: >> >> 1. Remote sends three deltas to Host. It persists the deltas and does >> not remove them from the outbound queue until the Host acknowledges >> receipt. >> 2. Host persists the deltas to a FIFO input queue before acknowledging >> them to the remote. >> 3. Host applies the deltas and associates the applied delta with the >> original that is still on the input queue. >> 4. Host sends applied deltas to the Remote >> 5. Remote persists applied deltas to a FIFO input queue before >> acknowledging receipt. >> 6. Remote applies the deltas and marks them as processed, in memory, >> but does not delete them from the input queue. >> 7. When the Host commits the applied deltas it deletes from the input >> queue the associated original deltas, all in a single transaction, then >> sends commit notice. >> 8. Remote receives commit notice. It persists applied deltas and >> removes them form the input queue in a single transaction, then >> acknowledges >> the commit notice. >> >> >> If the Host dies after applying and sending the deltas to Remote, but >> before committing them: >> >> 1. When Host restarts, it will begin processing all deltas it finds on >> the input queue. >> 2. After application it will send out new applied deltas. >> 3. When the Remote receives these "duplicate" deltas, it will compare >> them to the previously received set of applied deltas. >> 4. If the new applied deltas is identical to the original applied >> delta with the exception of the timestamps, then the Remote will update >> the >> timestamp on the original but do nothing else. >> 5. If the new applied deltas does not match the original the Remote >> will mark the Wavelet as broken. This should not happen unless the Host >> fails to reapply the deltas from the input queue in the order in which >> they >> where received. >> >> If the remote dies before it has committed the deltas, then it will >> reprocess items from the input queue. >> >> In order to ensure that the persisting of messages in inbound queues does >> not slow things down, the need to allow for multiple "in flight" messages. >> Basically a server would send all new messages before old ones have been >> acknowledged, but would have some upper limit on the number of allowed "in >> flight" messages. It would also back off on transmission rate based on the >> delay between sending the message and when it is acknowledged. If the delay >> between oldest sent message exceeds some amount, then all furthor >> transmission is halted and the oldest message is re-transmitted repeatedly >> until it is acknowledged. The retry rate should back off logarithmically but >> the lower limit should probably not go below once per day. >> >> -Tad >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Wave Protocol" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]<wave-protocol%[email protected]> >> . >> For more options, visit this group at >> http://groups.google.com/group/wave-protocol?hl=en. >> > > > > -- > --------------------------- > Prof. Torben Weis > Universitaet Duisburg-Essen > [email protected] > > -- > You received this message because you are subscribed to the Google Groups > "Wave Protocol" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<wave-protocol%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/wave-protocol?hl=en. > -- You received this message because you are subscribed to the Google Groups "Wave Protocol" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/wave-protocol?hl=en.
