Andrew Wong has submitted this change and it was merged. (
Change subject: KUDU-2293 fix cleanup after failed copies
KUDU-2293 fix cleanup after failed copies
Before, when a tablet server failed to tablet copy, Kudu would perform a
best-effort cleanup of the partially-copied replica and leave the tablet
tombstoned. If this tombstoning were to fail due to disk issues (e.g.
out of space), Kudu would allow this and tablet would remain in
TABLET_DATA_COPYING both in-memory and on-disk. This would lead to a
FATAL error if another tablet copy were started for the same replica, as
the server would attempt to copy over a replica with data already marked
This behavior arose from trying to balance two invariants:
- keep on-disk state consistent with in-memory state when possible
- when a tablet copy fails, leave it in as dead of a state as we can
(i.e. TABLET_DATA_TOMBSTONED with no transitions in progress)
This patch updates the Abort() logic to lean towards the latter
invariant: if a tablet copy fails, at least its in-memory state will be
set as TABLET_DATA_TOMBSTONED. This may not be true on-disk, but that's
okay because either 1) the tablet server will eventually overwrite it
via another tablet copy (at which time its data must _not_ be in the
TABLET_DATA_COPYING state), or 2) the server will be restarted and the
tablet will be tombstoned upon seeing a non-TABLET_DATA_READY state
We use the tablet copy internal state machine to determine whether to do
anything during Abort(). This wasn't always right before, since the
state machine didn't accurately reflect when cleanup was necessary (e.g.
the copy would be set to kStarted only at the end of Start(), so if
Abort() were called due to a failure in Start(), no cleanup was done).
This patch updates the state machine to account for this by introducing a
new state kStarting that indicates that there exists state to clean up
even if Start() has not completed, and by moving the setting of
kFinished such that cleanup is done even if Finish() fails.
A test is added to tablet_copy-itest that tests failures to copy,
ensuring that the tablet is left in such an state that further copies
are possible without crashing. The test uses EIO injection to fail the
copies, but the logic is the same as if full disks were used instead.
Another is added to tablet_copy_client-test that tests getting into the
various states in the updated tablet copy state machine.
Reviewed-by: Mike Percy <mpe...@apache.org>
Tested-by: Andrew Wong <aw...@cloudera.com>
6 files changed, 271 insertions(+), 41 deletions(-)
Mike Percy: Looks good to me, approved
Andrew Wong: Verified
To view, visit http://gerrit.cloudera.org:8080/9452
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mpe...@apache.org>
Gerrit-Reviewer: Tidy Bot