SebastianGruza commented on code in PR #3204:
URL: https://github.com/apache/hugegraph/pull/3204#discussion_r3995606936
##########
hugegraph-store/hg-store-client/src/main/java/org/apache/hugegraph/store/client/NodeTxExecutor.java:
##########
@@ -373,35 +398,70 @@ boolean ifAnyTrue(Supplier<Stream<HgPair<HgStoreNode,
NodeTkv>>> nodeStreamSuppl
}
<T> Optional<T> retryingInvoke(Supplier<T> supplier) {
+ boolean[] deadlineRetried = {false};
return IntStream.rangeClosed(0, NODE_MAX_RETRYING_TIMES).boxed()
.map(
i -> {
+ if
(Thread.currentThread().isInterrupted()) {
+ // The caller (e.g. a REST worker
hitting
+ // restserver.request_timeout) gave
up: stop
+ // retrying instead of holding its
thread.
+ // InterruptedException as the root
cause: the
+ // server's task cancel path
recognises it
+ // (HugeException.isInterrupted()).
+ throw HgStoreClientException.of(
+ "Interrupted before retry " +
i,
+ new InterruptedException());
+ }
T buffer = null;
try {
buffer = supplier.get();
} catch (Throwable t) {
- if (i + 1 <= NODE_MAX_RETRYING_TIMES) {
- try {
- int sleepTime;
- // The first three times try
once every second
- if (i < 3) {
- sleepTime = 1;
- } else {
- // Subsequent incremental
- sleepTime = i - 1;
- }
- log.info("Waiting {} seconds "
+
- "for the next try.",
- sleepTime);
- Thread.sleep(sleepTime *
1000L);
- } catch (InterruptedException e) {
- log.error("Failed to sleep",
e);
+ Failure failure = classify(t);
+ if (failure == Failure.FATAL) {
+ // The caller's thread was
interrupted or the
+ // call was cancelled: fail fast.
+ log.warn("Not retrying after: {}",
+ t.getMessage(), t);
+ throw HgStoreClientException.of(
+ t.getMessage(), t);
+ }
+ if (failure == Failure.DEADLINE) {
+ // One retry: the NOT_WORK notice
sent for the
+ // failed RPC reloads the
partition leaders, so
+ // the next attempt can reach a
new leader. A
+ // second deadline in a row would
only wait the
+ // full deadline again on the same
stalled store.
+ if (deadlineRetried[0]) {
+ log.warn("Not retrying a
second deadline: {}",
+ t.getMessage(), t);
+ throw
HgStoreClientException.of(
+ t.getMessage(), t);
}
- } else {
+ deadlineRetried[0] = true;
+ log.warn("Deadline exceeded,
retrying once in " +
Review Comment:
Done — the description now describes `classify()` / `Failure` with the
single deadline retry (and the reset after a different failure from 35e0a6d),
and I re-ran the SIGSTOP measurement at `35e0a6d` instead of annotating: with
one deadline retry the REST outage under the 1 request/s load is 128 of 300 s
(in 20 short windows) against 12 s for the never-retry head and 431 of 503 s
for master, the slowest write is 45 s (two deadlines plus backoff) and the
server is still healthy 0 s after the writer stops. The table in the body has
all three columns and states that the old column is superseded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
