It's an invariant of the system that the txn logs should contain txns at least up to zxid, where zxid is the value tagged on the snapshot name. If we don't check that this invariant holds, I think we should. In this case, say you didn't backup the logs, the system would refuse to start because it conservatively assumes that its state is inconsistent.
-Flavio On Thu, Jul 16, 2015 at 1:21 PM, Adam Milne-Smith <[email protected]> wrote: > So it seems that this is fine as long as we always backup the tranlogs as > well. > > Although I'm not sure whether there is a scenario that might lead to an > error whilst replaying the log file, where the transaction fails due to the > partially updated/erroneous state that the snapshot initialises. > > With this in mind, our ZOOKEEPER-2141 patch may have to ignore some > unexpected state rather than throw an exception. We'll follow up on this at > a later date. > > Thanks, > Adam > > On 16 Jul 2015 12:30, Flavio Junqueira wrote: > > (moving discussion to > dev) > > Hi Adam, > > I can't see a problem with your description about the > snapshot generation, but I would expect that replaying the transaction log > would bring back the missing transactions. We replay from the zxid in the > snapshot name, which is taken before the snapshot starts > (FileTxnSnapLog.save(...)). > > -Flavio > > > On 16 Jul 2015, at 12:02, > Adam Milne-Smith wrote: > > > > I've created a jira ticket here: > > > https://issues.apache.org/jira/browse/ZOOKEEPER-2234 > > > > Thanks, > > > Adam > > > > On 15 Jul 2015 16:07, Adam Milne-Smith wrote: > >> > >> Whilst > writing a patch for ZOOKEEPER-2141 (3.4.6 branch), we spotted an ephemeral > node that had not been deleted despite its session having expired. Its ACL > long did not exist in the ACL cache so any operation against this node will > fail. > >> > >> This could lead to things like curator locks never being > deleted (even after the timeout) and deadlocking applications. > >> > >> We > inspected the code and are reasonably certain that there are no bugs in > updating the in-memory data tree that could cause this. However serialising > the snapshot happens asynchronously and follows these 4 steps: > >> > >> > -copy the sessions map > >> -serialise the sessions map copy > >> > -serialise the ACL map (synchronised) > >> -serialise the data tree > (synchronised at the individual node level) > >> > >> We suspect the issue > we are seeing is a new session and ephemeral node being created during the > data tree serialisation hence the corresponding session and acl are missing > from the snapshot but the node is present. This means the snapshot contains > a partial transaction. > >> > >> If we were to deserialise from this > snapshot then the data in-memory would be invalid. If one member of the > quorum were to reboot and restore from this snapshot, it would contain this > node where the other hosts had removed it. If this host were to become the > leader and send its snapshot to other members of the quorum, those would > have the invalid data too. > >> > >> As far as we can see, the only way to > delete this node when this happens in production would be to perform manual > surgery on the snapshot. > >> > >> Can anyone confirm that they agree this > to be the case or let us know if we've misunderstood something? > >> > >> > Thanks, > >> Adam >
