[kudu-CR] java: use truncated randomized exponential backoff for retries

[Todd Lipcon (Code Review)]

Todd Lipcon has submitted this change and it was merged.

Change subject: java: use truncated randomized exponential backoff for retries
..


java: use truncated randomized exponential backoff for retries

This changes the Java client to use a randomized exponential backoff,
starting at 1ms, and going up to 4096ms, rather than the previous linear
backoff of (500ms * retry_count). The old backoff was far too
aggressive, and resulted in the Java client regularly having high
percentile latencies of 500 or 1000ms even when there was spare capacity
in the system.

I tested this on a 70 node cluster using a read-only YCSB workload, and
the 99.99th percentile latency dropped from >1s to 390ms.  Combined with
further in-flight changes, the 99.99th percentile dropped further to
~56ms.

Change-Id: I79256e00cf35c072c60cd2b9034e6c1b10c18b38
Reviewed-on: http://gerrit.cloudera.org:8080/3184
Tested-by: Kudu Jenkins
Reviewed-by: Jean-Daniel Cryans
Reviewed-by: Adar Dembo 
---
M java/kudu-client/src/main/java/org/kududb/client/AsyncKuduClient.java
1 file changed, 3 insertions(+), 2 deletions(-)

Approvals:
  Jean-Daniel Cryans: Looks good to me, approved
  Adar Dembo: Looks good to me, approved
  Kudu Jenkins: Verified



-- 
To view, visit http://gerrit.cloudera.org:8080/3184
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I79256e00cf35c072c60cd2b9034e6c1b10c18b38
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Jean-Daniel Cryans
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 

[kudu-CR] java: use truncated randomized exponential backoff for retries

[Adar Dembo (Code Review)]

Adar Dembo has posted comments on this change.

Change subject: java: use truncated randomized exponential backoff for retries
..


Patch Set 1: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/3184/1/java/kudu-client/src/main/java/org/kududb/client/AsyncKuduClient.java
File java/kudu-client/src/main/java/org/kududb/client/AsyncKuduClient.java:

Line 924: // Randomized exponential backoff, truncated at 4096ms.
> The Java client uses this somewhat indiscriminately including things like w
> Looking at the C++ client, there are a number of different strategies 
> sprinkled around.

Yeah, and it's made me rather grumpy. Makes thinking about backoff a very 
unpredictable affair.

Anyway, I guess this isn't any worse, since the Java client did its own thing 
previously too.


-- 
To view, visit http://gerrit.cloudera.org:8080/3184
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I79256e00cf35c072c60cd2b9034e6c1b10c18b38
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Jean-Daniel Cryans
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: Yes

[kudu-CR] java: use truncated randomized exponential backoff for retries

[Todd Lipcon (Code Review)]

Todd Lipcon has posted comments on this change.

Change subject: java: use truncated randomized exponential backoff for retries
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/3184/1/java/kudu-client/src/main/java/org/kududb/client/AsyncKuduClient.java
File java/kudu-client/src/main/java/org/kududb/client/AsyncKuduClient.java:

Line 924: // Randomized exponential backoff, truncated at 4096ms.
> Why not use the same backoff scheme as the C++ client? I think that's linea
The Java client uses this somewhat indiscriminately including things like 
waiting for a table to be created, etc. Exponential backoff seems like a more 
reasonable strategy than linear.

Looking at the C++ client, there are a number of different strategies sprinkled 
around. For scanners, we use:

// Exponential backoff with jitter anchored between 10ms and 20ms, and an
// upper bound between 2.5s and 5s.
MonoDelta sleep = MonoDelta::FromMilliseconds(
(10 + rand() % 10) * static_cast(std::pow(2.0, std::min(8, 
scan_attempts_ - 1;

so this is approximately the same, except starting at 10ms instead of 1ms


-- 
To view, visit http://gerrit.cloudera.org:8080/3184
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I79256e00cf35c072c60cd2b9034e6c1b10c18b38
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Jean-Daniel Cryans
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon 
Gerrit-HasComments: Yes

[kudu-CR] java: use truncated randomized exponential backoff for retries

[Adar Dembo (Code Review)]

Adar Dembo has posted comments on this change.

Change subject: java: use truncated randomized exponential backoff for retries
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/3184/1/java/kudu-client/src/main/java/org/kududb/client/AsyncKuduClient.java
File java/kudu-client/src/main/java/org/kududb/client/AsyncKuduClient.java:

Line 924: // Randomized exponential backoff, truncated at 4096ms.
Why not use the same backoff scheme as the C++ client? I think that's linear 
backoff, 1 ms per attempt + 0-4 additional ms.


-- 
To view, visit http://gerrit.cloudera.org:8080/3184
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I79256e00cf35c072c60cd2b9034e6c1b10c18b38
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon 
Gerrit-Reviewer: Adar Dembo 
Gerrit-Reviewer: Jean-Daniel Cryans
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: Yes