[kudu-CR] tablet: change default bloom filter FP rate to 0.01%
Todd Lipcon has submitted this change and it was merged. Change subject: tablet: change default bloom filter FP rate to 0.01% .. tablet: change default bloom filter FP rate to 0.01% The old default, 1%, was high enough that in a uniform random write workload, we ended up needing to read in most of the key blocks even with bloom filters enabled. On a 5 node cluster, after inserting a few billion rows, the write throughput dropped dramatically as every batch of writes was seeking and reading keys off disk. In testing on the same cluster, changing the FP rate to 0.01% improved the throughput dramatically (>2x) by reducing the random reads coming off disk. The cost is a 2x increase in bloom filter size (20 bits per key vs 10) but 20 bits is still a small percentage compared to typical row key sizes in target applications. Of course if an application has no random write characteristics and really cares about disk space, this can always be flipped back. Screenshots of the inserts/second graph (1hr rolling average) for these tests are at: https://gist.github.com/toddlipcon/1ab9b36b7fbae10b635d3a905e1fe55a Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293 Reviewed-on: http://gerrit.cloudera.org:8080/3517 Tested-by: Kudu Jenkins Reviewed-by: Jean-Daniel Cryans Reviewed-by: David Ribeiro Alves --- M docs/release_notes.adoc M src/kudu/tablet/tablet.cc 2 files changed, 10 insertions(+), 1 deletion(-) Approvals: David Ribeiro Alves: Looks good to me, approved Jean-Daniel Cryans: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/3517 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293 Gerrit-PatchSet: 3 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon
[kudu-CR] tablet: change default bloom filter FP rate to 0.01%
Todd Lipcon has posted comments on this change. Change subject: tablet: change default bloom filter FP rate to 0.01% .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/3517/2//COMMIT_MSG Commit Message: PS2, Line 25: https://gist.github.com/toddlipcon/1ab9b36b7fbae10b635d3a905e1fe55a > interesting how the changed graph seems to have a second life towards the e yea, guess I should have mentioned that -- that was me hacking out block cache memory tracking (KUDU-1502) -- To view, visit http://gerrit.cloudera.org:8080/3517 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-HasComments: Yes
[kudu-CR] tablet: change default bloom filter FP rate to 0.01%
David Ribeiro Alves has posted comments on this change. Change subject: tablet: change default bloom filter FP rate to 0.01% .. Patch Set 2: Code-Review+2 thanks for adding the additional info -- To view, visit http://gerrit.cloudera.org:8080/3517 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-HasComments: No
[kudu-CR] tablet: change default bloom filter FP rate to 0.01%
Jean-Daniel Cryans has posted comments on this change. Change subject: tablet: change default bloom filter FP rate to 0.01% .. Patch Set 2: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/3517/2//COMMIT_MSG Commit Message: PS2, Line 25: https://gist.github.com/toddlipcon/1ab9b36b7fbae10b635d3a905e1fe55a interesting how the changed graph seems to have a second life towards the end. -- To view, visit http://gerrit.cloudera.org:8080/3517 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-HasComments: Yes
[kudu-CR] tablet: change default bloom filter FP rate to 0.01%
Kudu Jenkins has posted comments on this change. Change subject: tablet: change default bloom filter FP rate to 0.01% .. Patch Set 2: Build Started http://104.196.14.100/job/kudu-gerrit/2079/ -- To view, visit http://gerrit.cloudera.org:8080/3517 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-HasComments: No
[kudu-CR] tablet: change default bloom filter FP rate to 0.01%
Hello Jean-Daniel Cryans, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/3517 to look at the new patch set (#2). Change subject: tablet: change default bloom filter FP rate to 0.01% .. tablet: change default bloom filter FP rate to 0.01% The old default, 1%, was high enough that in a uniform random write workload, we ended up needing to read in most of the key blocks even with bloom filters enabled. On a 5 node cluster, after inserting a few billion rows, the write throughput dropped dramatically as every batch of writes was seeking and reading keys off disk. In testing on the same cluster, changing the FP rate to 0.01% improved the throughput dramatically (>2x) by reducing the random reads coming off disk. The cost is a 2x increase in bloom filter size (20 bits per key vs 10) but 20 bits is still a small percentage compared to typical row key sizes in target applications. Of course if an application has no random write characteristics and really cares about disk space, this can always be flipped back. Screenshots of the inserts/second graph (1hr rolling average) for these tests are at: https://gist.github.com/toddlipcon/1ab9b36b7fbae10b635d3a905e1fe55a Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293 --- M docs/release_notes.adoc M src/kudu/tablet/tablet.cc 2 files changed, 10 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/17/3517/2 -- To view, visit http://gerrit.cloudera.org:8080/3517 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] tablet: change default bloom filter FP rate to 0.01%
David Ribeiro Alves has posted comments on this change. Change subject: tablet: change default bloom filter FP rate to 0.01% .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/3517/1/docs/release_notes.adoc File docs/release_notes.adoc: PS1, Line 72: substantially how substantantially? PS1, Line 74: incremental which increase? Can you provide rough numbers for both? -- To view, visit http://gerrit.cloudera.org:8080/3517 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-HasComments: Yes
[kudu-CR] tablet: change default bloom filter FP rate to 0.01%
Jean-Daniel Cryans has posted comments on this change. Change subject: tablet: change default bloom filter FP rate to 0.01% .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/3517 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-HasComments: No
[kudu-CR] tablet: change default bloom filter FP rate to 0.01%
Kudu Jenkins has posted comments on this change. Change subject: tablet: change default bloom filter FP rate to 0.01% .. Patch Set 1: Build Started http://104.196.14.100/job/kudu-gerrit/2077/ -- To view, visit http://gerrit.cloudera.org:8080/3517 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-HasComments: No
[kudu-CR] tablet: change default bloom filter FP rate to 0.01%
Hello Jean-Daniel Cryans, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/3517 to review the following change. Change subject: tablet: change default bloom filter FP rate to 0.01% .. tablet: change default bloom filter FP rate to 0.01% The old default, 1%, was high enough that in a uniform random write workload, we ended up needing to read in most of the key blocks even with bloom filters enabled. On a 5 node cluster, after inserting a few billion rows, the write throughput dropped dramatically as every batch of writes was seeking and reading keys off disk. In testing on the same cluster, changing the FP rate to 0.01% improved the throughput dramatically by reducing the random reads coming off disk. The cost is a 2x increase in bloom filter size (20 bits per key vs 10) but 20 bits is still a small percentage compared to typical row key sizes in target applications. Of course if an application has no random write characteristics and really cares about disk space, this can always be flipped back. Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293 --- M docs/release_notes.adoc M src/kudu/tablet/tablet.cc 2 files changed, 10 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/17/3517/1 -- To view, visit http://gerrit.cloudera.org:8080/3517 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Jean-Daniel Cryans