Ashwani Raina has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24452
Change subject: KUDU-3783 Extend clean-superblock guarantee into LoadFromSuperBlock ...................................................................... KUDU-3783 Extend clean-superblock guarantee into LoadFromSuperBlock Patches under KUDU-1060 only guarantee only when a superblock cleanup happens through DeleteTabletData, ignoring crash-recovery path: 1. A crash occurs after a superblock was written with orphaned block entries, but before cleanup Flush() that re-syncs a superblock with empty list of orphaned block entries. 2. On the next startup LoadFromSuperBlock() calls DeleteOrphanedBlocks() that deletes the orphaned blocks from disk as well as in-memory list. But it doesn't sync a clean superblock afterwards. 3. For a tombstoned tablet that doesn't receive any writes(i.e., flush), on-disk superblock always has those stale orphaned block entries. Every subsequent restart would trigger parsing of those block entries and repeated call to DeleteOrphanedBlocks() that is a per-restart cost proportional to: No. of tombstoned tablets x No. of orphaned blocks per tablet The patch adds a fix by calling Flush() after DeleteOrphanedBlocks, during load-time to persist a clean superblock (gated by --enable_orphaned_block_cleanup_on_load). This converts the recurring per-restart reprocessing into a one-time cost on the first restart, which is precisely what the newly added unit test i.e., BenchmarkCleanStartupAfterOrphanCleanup demonstrates. Some benchmarking stats: === KUDU-3783 Startup Overhead Benchmark === Tablets : 1000 Orphaned blocks / tablet : 1000000 With cleanup + Flush() : 613387 ms Without cleanup + Flush() : 253331 ms Extra overhead (Flush x N) : 360056 ms Avg overhead per tablet : 360 ms === KUDU-3783 One-Time-Cost Benchmark === Tablets : 1000 Orphaned blocks / tablet : 1000000 Pass 1 - 1st restart (fix ON, dirty) : 629216 ms [cleanup+Flush for all tablets] Pass 2 - 2nd restart (fix ON, clean) : 114 ms [superblocks already clean] Pass 3 - clean baseline (fix OFF, clean) : 112 ms [no orphaned blocks to parse] Pass 4 - legacy baseline (fix OFF, dirty) : 262499 ms [parses orphan IDs, no Flush] Pass1 overhead vs Pass4 legacy : 366717 ms Pass2 vs Pass3 clean baseline : 2 ms (should be ~0) Change-Id: I729395085fb845daa667b1374dc1ac4291a0c30b --- M src/kudu/tablet/tablet_metadata-test.cc M src/kudu/tablet/tablet_metadata.cc M src/kudu/tablet/tablet_metadata.h 3 files changed, 505 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/52/24452/1 -- To view, visit http://gerrit.cloudera.org:8080/24452 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I729395085fb845daa667b1374dc1ac4291a0c30b Gerrit-Change-Number: 24452 Gerrit-PatchSet: 1 Gerrit-Owner: Ashwani Raina <[email protected]>
