Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8859 )
Change subject: KUDU-2115: avoid compacting already-compacted rowsets ...................................................................... KUDU-2115: avoid compacting already-compacted rowsets Tablets will perform compaction selection on a copy of the currently available rowsets in order to avoid holding the component_lock_ for the duration of rowset selection. It is then verified that nothing else compacted the selected rowsets by iterating over the selected rowsets and checking that they still exist to be compacted. However, the initial selection of rowsets is performed on a snapshotted copy of the available rowsets, and in between the snapshot of the rowsets and the verification, the component_lock_ is dropped, allowing the following race: T1: runs PickRowSetsToCompact, picks {A, B}, begins compaction T2: runs PickRowSetsToCompact: T2: snapshots the rowset tree (which still contains {A, B}) T2: drops component_lock_ after taking the snapshot T1: finishes compaction and removes {A, B} from the rowset tree, unlocking each rowset's compact_flush_lock_ T2: iterates over the rowset tree and sees {A, B} as IsAvailableForCompaction(), since they have been unlocked T2: selects {A, B} as a part of its compaction T2: grabs the component_lock_ to verify that the selected rowsets are still available T2: sees that some of the selected rowsets would not be returned because they are missing from the currently available rowsets (DFATALs and aborts the compaction) I verified the diagnosis by placing a random sleep just after making the copy in PickRowSetsToCompact() and seeing TabletServerDiskFailureTest.TestRandomOpSequence, which schedules many compactions, fail consistently with the failed verification. Upon making the fix, this passes. This can lead to scheduling a fairly useless compaction. This patch adds an API to RowSets to mark itself as compacted, and ensure that when selecting rowsets to compact, check that rowset candidates have not already been compacted. To verify the solution, I've added a small test that schedules several concurrent compactions. Upon adding the aforementioned randomized delay and removing a couple sanity D/CHECKs (including the above verification), I saw the test fail, leading to an unexpected number of rows left in the tablet. Change-Id: I4dab330d61facb18717f6faf179f9b94a9e55236 Reviewed-on: http://gerrit.cloudera.org:8080/8859 Tested-by: Kudu Jenkins Reviewed-by: Todd Lipcon <t...@apache.org> --- M src/kudu/tablet/compaction-test.cc M src/kudu/tablet/diskrowset.cc M src/kudu/tablet/diskrowset.h M src/kudu/tablet/memrowset-test.cc M src/kudu/tablet/memrowset.cc M src/kudu/tablet/memrowset.h M src/kudu/tablet/mock-rowsets.h M src/kudu/tablet/rowset.h M src/kudu/tablet/tablet.cc 9 files changed, 170 insertions(+), 77 deletions(-) Approvals: Kudu Jenkins: Verified Todd Lipcon: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/8859 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4dab330d61facb18717f6faf179f9b94a9e55236 Gerrit-Change-Number: 8859 Gerrit-PatchSet: 8 Gerrit-Owner: Andrew Wong <aw...@cloudera.com> Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy <mpe...@apache.org> Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon <t...@apache.org>