Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12532 )
Change subject: KUDU-2701 Fix compaction loop due to using wrong rowset size ...................................................................... KUDU-2701 Fix compaction loop due to using wrong rowset size Compaction policy originally evaluated the size of a rowset to be the size of the rowset's base data and its REDOs. This size is used to calculate the probability mass of the rowset and the weight of the rowset in the compaction knapsack problem. Mistakenly, it was also used as the size of a rowset for KUDU-1400 small rowset compaction policy. This is wrong- the size of the whole rowset should be used. The reason for this is the following: small rowset compaction compacts rowsets when the number of rowsets is reduced. The number of rowsets produced depends on the total size of the data when written. If a partial size of the rowset is used, this can lead to bad decisions being made about compaction. For example, I discovered a case where a tablet had 8 rowsets of "size" 16MB. In reality, this size was the base data size plus REDOs. Small rowset compaction policy determined that it would be good to compact these 8 rowsets, believing that it would produce 4 rowsets of size 32MB. However, the rowsets were actually 32MB in size, and compacting them produced 8 rowsets of size 32MB identical to the previous 8, and therefore 8 rowsets that appeared to be of size 16MB to compaction policy. Thus these 8 were chosen to be compacted, and so on... This patch changes the small rowset compaction policy code to use the full size of the rowsets. All other uses of size remain the same. It's not totally clear to me why just base data and REDOs are used, but in any case changing it would change CDFs and change knapsack calculations, which could lead to pretty big differences in compaction behavior. Since, outside this bug, compaction is working fine, and since this patch is targeted to unblock 1.9, I opted to make a minimal change to fix the bug and not evaluate any larger chance to compaction policy. I tested this code on the cluster where I originally encountered the compaction loop. The loop did not reoccur and the updated compaction policy successfully compacted small rowsets together for several hours without becoming stuck. It finally converged to a stable end state. I also ran a number of sequential and random insert workloads on this cluster and saw no problems with compactions looping or getting stuck, even under high load or when insertions ended and the tablet was free to catch up on compactions. Change-Id: I21b7ff6333137aaf1e98ef4849691dd08e24e007 Reviewed-on: http://gerrit.cloudera.org:8080/12488 Reviewed-by: Andrew Wong <[email protected]> Tested-by: Andrew Wong <[email protected]> Reviewed-by: Alexey Serbin <[email protected]> Reviewed-on: http://gerrit.cloudera.org:8080/12532 Tested-by: Kudu Jenkins --- M src/kudu/tablet/compaction_policy.cc M src/kudu/tablet/rowset_info.cc M src/kudu/tablet/rowset_info.h 3 files changed, 28 insertions(+), 17 deletions(-) Approvals: Alexey Serbin: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12532 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.9.x Gerrit-MessageType: merged Gerrit-Change-Id: I21b7ff6333137aaf1e98ef4849691dd08e24e007 Gerrit-Change-Number: 12532 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Wong <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley <[email protected]>
