[kudu-CR](branch-0.9.x) java: use truncated randomized exponential backoff for retries
Todd Lipcon has submitted this change and it was merged. Change subject: java: use truncated randomized exponential backoff for retries .. java: use truncated randomized exponential backoff for retries This changes the Java client to use a randomized exponential backoff, starting at 1ms, and going up to 4096ms, rather than the previous linear backoff of (500ms * retry_count). The old backoff was far too aggressive, and resulted in the Java client regularly having high percentile latencies of 500 or 1000ms even when there was spare capacity in the system. I tested this on a 70 node cluster using a read-only YCSB workload, and the 99.99th percentile latency dropped from >1s to 390ms. Combined with further in-flight changes, the 99.99th percentile dropped further to ~56ms. Change-Id: I79256e00cf35c072c60cd2b9034e6c1b10c18b38 Reviewed-on: http://gerrit.cloudera.org:8080/3184 Tested-by: Kudu Jenkins Reviewed-by: Jean-Daniel Cryans Reviewed-by: Adar Dembo (cherry picked from commit 8094b73147e0238cdfd96a141554b5344c801cdb) Reviewed-on: http://gerrit.cloudera.org:8080/3198 --- M java/kudu-client/src/main/java/org/kududb/client/AsyncKuduClient.java 1 file changed, 3 insertions(+), 2 deletions(-) Approvals: Jean-Daniel Cryans: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/3198 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I79256e00cf35c072c60cd2b9034e6c1b10c18b38 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: branch-0.9.x Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon
[kudu-CR](branch-0.9.x) java: use truncated randomized exponential backoff for retries
Jean-Daniel Cryans has posted comments on this change. Change subject: java: use truncated randomized exponential backoff for retries .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/3198 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I79256e00cf35c072c60cd2b9034e6c1b10c18b38 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: branch-0.9.x Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-HasComments: No
[kudu-CR](branch-0.9.x) java: use truncated randomized exponential backoff for retries
Todd Lipcon has uploaded a new change for review. http://gerrit.cloudera.org:8080/3198 Change subject: java: use truncated randomized exponential backoff for retries .. java: use truncated randomized exponential backoff for retries This changes the Java client to use a randomized exponential backoff, starting at 1ms, and going up to 4096ms, rather than the previous linear backoff of (500ms * retry_count). The old backoff was far too aggressive, and resulted in the Java client regularly having high percentile latencies of 500 or 1000ms even when there was spare capacity in the system. I tested this on a 70 node cluster using a read-only YCSB workload, and the 99.99th percentile latency dropped from >1s to 390ms. Combined with further in-flight changes, the 99.99th percentile dropped further to ~56ms. Change-Id: I79256e00cf35c072c60cd2b9034e6c1b10c18b38 Reviewed-on: http://gerrit.cloudera.org:8080/3184 Tested-by: Kudu Jenkins Reviewed-by: Jean-Daniel Cryans Reviewed-by: Adar Dembo (cherry picked from commit 8094b73147e0238cdfd96a141554b5344c801cdb) --- M java/kudu-client/src/main/java/org/kududb/client/AsyncKuduClient.java 1 file changed, 3 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/98/3198/1 -- To view, visit http://gerrit.cloudera.org:8080/3198 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I79256e00cf35c072c60cd2b9034e6c1b10c18b38 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: branch-0.9.x Gerrit-Owner: Todd Lipcon