Hi,
On 19/01/17 15:35, Vikas Saurabh wrote:
Just to add a bit of disclaimer - external observation also would be
affected in similar fashion.. but afaik, we've never claimed lossless
delivery of observation events - it's a best case service.
This is correct, but the async index update does not only depend on
observation but also uses checkpoints.
With CN1 (leader) and CN2 (other cluster node) participating in the
cluster, here's a rough timeline that I think can lead to issues:
# cluster is idle and index is updated till this point
# CN2 changes some indexable data at /tree/node1
# CN2->bkWrite works
# CN2 changes some indexable data at /tree/node2
# CN2 crashes
# CN1 changes some indexable data at /tree/node3
# CN1 bkReads
# Async index cycle runs and indexes /tree/node1 and /tree/node3
# CN1 notices that CN2 lease has timed out and recovers changes in /tree/node2
BUT, /tree/node2 never goes into index.
That would be a bug. As part of the recovery for CN2, its _lastRev entry
will be updated to reflect the /tree/node2 change. Let's say previously
it was r7-0-2 and after the recovery is r9-0-2. The current
checkpoint for the async index update on CN1 does not yet see the
updated _lastRev. E.g. it could be at r10-0-1,r7-0-2. On the next index
update (after the recovery), the head revision used to compare against
the checkpoint will be r10-0-1,r9-0-2. Now CN1 will update the index
with changed that happened between r7-0-2 and r9-0-2.
Btw, recovery does push journal entry so, if the timing is right, then
things would work fine. But it's hard to imagine that bkRead+async
cycle wouldn't have run before lease timeout of crashed node.
It doesn't have to. See above.
Please do file an issue, if you have a test that shows an issue with the
scenario you described. I may be missing something and the index update
is indeed broken.
Regards
Marcel
