Oh, now I'm getting you point! The issue is that a filtered replication doesn't periodically checkpoint if no changes match the filter. This looks like a gap in the continuous replication "protocol". The filtered _changes feed that is used by the continuous replication should emit "empty" changes (only seq) when used with the heartbeat parameter.
Your idea of a "synchronization" document that passes all filters seems like a pretty straight fwd workaround. - mathias On Jul 13, 2012, at 2:33 , Andreas Kemkes wrote: > Mathias: > > I had planned to not allow new entries into the source to let the continuous > replications catch up, but I don't see how your approach changes the > conundrum that "source_seq" - "checkpointed_source_seq" will only be 0 for > the exceptional case that the last entry into the source gets through the > filter. > > Given full coverage, there will be at least one replication globally where > this value is indeed zero. Maybe a document that doesn't get filtered by any > of the replications is the workaround (do-it-yourself boundary > synchronization). > > I like your idea of graphing the "lag of changes". That may come in handy in > other replication patterns. > > Thanks, > > Andreas > > From: Mathias Leppich <[email protected]> > To: [email protected]; Andreas Kemkes <[email protected]> > Sent: Thursday, July 12, 2012 12:13 AM > Subject: Re: Replication and checkpoints - what to expect? > > Hi Andreas, > > with continuous replications and an ever changing dataset there is no point > where you can tell your replication is "up-to-date" in terms of "100% > replicated" as replication always happens after the data has been written to > the source database. (which is a good thing) > > You need to change the way you measure the up-to-date-ness. Instead of > measuring the percentage of completion you should better me sure the lag of > changes. e.g. targetDB is N changes behind sourceDB. > > With couchdb 1.2 you get this number with a single request to /_active_tasks > by calculating a replications "source_seq" - "checkpointed_source_seq". Prior > to 1.2 you can get this number too but its a more difficult because you have > to know the replications _local ID and check the "source_last_seq" field in > the replications session document… > > Once you have the "lag of changes" for your continuous replications its a > good thing to graph it with some monitoring tool to get a big picture of how > replication performance is going through the day. > > - mathias > > On Jul 12, 2012, at 1:31 , Andreas Kemkes wrote: > > > I wanted to follow up on this thread as I'm still experience difficulties > > using the feature and would like some advise how to best deal with the > > situation. > > > > The goal is to break up a monolithic database into multiple, which was > > achieved after a lot of trial and error. Now the quest is to keep it in > > sync for a while by using filtered, continuous replications. Yet the > > replication gets stuck on the last sequence number that passes the filter. > > In the Futon UI, I see: > > > > Checkpointed source sequence 165850, current source sequence 166253, > > progress 99% > > > > If I start a non-continuous replication with the exact same parameters, it > > returns: > > > > > > { > > "ok": true, > > "no_changes": true, > > "session_id": ..., > > "source_last_seq": 165850, > > "replication_id_version": 2, > > ... > > } > > It apparently knows that there are no changes and it knows the current > > source sequence. Why could it not move the checkpointed source sequence > > forward to match the current source sequence? What am I missing? > > > > Unless there is an exact match between checkpointed and current source > > sequence, how would one ever know if a replication is up-to-date? > > > > -- Andreas > > > > > > ________________________________ > > From: Filipe David Manana <[email protected]> > > To: [email protected]; Andreas Kemkes <[email protected]> > > Sent: Thursday, June 21, 2012 12:40 PM > > Subject: Re: Replication and checkpoints - what to expect? > > > >> The same should be true for filtered replications if there is no > >> applicable document between the current source sequence and the last > >> checkpoint. Otherwise you would be always wondering if it has been > >> replicated entirely. > > > > That's harder. With filtered replication, we only know about sequence > > numbers of changes that pass the filter. > > >
