SebastianGruza opened a new pull request, #3204:
URL: https://github.com/apache/hugegraph/pull/3204
## Purpose of the PR
- close #3199
When a store node stops answering, `NodeTxExecutor.retryingInvoke()`
(hg-store-client) retries the failing call up to `NODE_MAX_RETRYING_TIMES` (10)
with a 1, 1, 1, 2, 3, 4, 5, 6, 7, 8 s sleep schedule, and every attempt is a
blocking gRPC call bounded only by `grpc.timeout.seconds` (default 100 s).
After a `DEADLINE_EXCEEDED` the next attempt just waits the full deadline
again, so one commit or one point lookup on a stalled partition holds its
calling thread for `11 × grpc.timeout.seconds + 38 s` (about 19 minutes on
defaults). The loop also catches the `InterruptedException` from
`Thread.sleep`, logs `Failed to sleep` and continues, so the interrupt sent by
`restserver.request_timeout` (REST worker) or by the Gremlin Server
`evaluationTimeout` is swallowed and neither limit can free the thread. With a
writer at 1 request/s, a single stalled store node exhausts the REST worker
pool within tens of seconds and the server answers 503 to everything for
minutes after the writer stops
.
## Main Changes
- `NodeTxExecutor.retryingInvoke()`: abort the loop when the calling thread
is interrupted (before an attempt, or while sleeping between attempts),
restoring the interrupt flag and surfacing an `HgStoreClientException`.
- Do not retry a failure whose cause chain carries
`Status.Code.DEADLINE_EXCEEDED`, `Status.Code.CANCELLED` or an
`InterruptedException` (`isRetryable()`); a second attempt cannot succeed
faster than the deadline that just expired. `UNAVAILABLE` and other transport
failures are retried exactly as before, which is what the store-replacement
recovery of #3130 relies on.
- No change to the attempt count or the sleep schedule; making them
configurable is a possible follow-up.
## Verifying these changes
- New unit tests in `NodeTxExecutorTest` (`hg-store-test`,
`store-client-test` profile): `isRetryable()` classification, a
`DEADLINE_EXCEEDED` failure makes exactly one attempt, an interrupt during a
retryable failure stops the loop after one attempt with the interrupt flag
restored, and an `UNAVAILABLE` failure is still retried and succeeds on the
second attempt.
```
mvn test -pl hugegraph-store/hg-store-test -am -P store-client-test
-Dtest=NodeTxExecutorTest -Dsurefire.failIfNoSpecifiedTests=false
```
6/6 pass on Temurin 17.
- Before/after on a live PD + 3-store cluster (master `36811483`, one store
frozen with `SIGSTOP`, one `POST /graph/vertices` per second for 300 s, a probe
`GET` every 2 s, `grpc.timeout.seconds=20` so that a run fits in minutes):
| | before | after |
|---|---|---|
| REST unavailable (probe gets 503) | 431 of 503 s | 12 of 300 s |
| REST still unavailable after the writer stopped | 200 s | 0 s |
| 300 writes | 27 × 201, 243 × 503, 23 × 500 | 131 × 201, 152 × 500 after
20.0 s, 7 × 503 |
| slowest write | 257 s (= 11 × 20 + 38) | 20.1 s |
| server log | `Failed to sleep` ×30, `reached the upper limit` ×30 |
`Failed to sleep` 0, `Not retrying after` ×152 |
Full logs and the reproduction script:
https://github.com/SebastianGruza/hugegraph-validation/blob/master/docs/findings.md#f15
- [ ] Trivial rework / code cleanup without any test coverage. (No Need)
- [ ] Already covered by existing tests, such as *(please modify tests
here)*.
- [x] Need tests and can be verified as shown above.
## Does this PR potentially affect the following parts?
- [ ] Dependencies (add/update license info)
- [ ] Modify configurations
- [ ] The public API
- [x] Other affects: retry semantics of the HStore client —
`DEADLINE_EXCEEDED`/`CANCELLED` are no longer retried, interrupts are honoured
- [ ] Nope
## Documentation Status
- [ ] `Doc - TODO`
- [ ] `Doc - Done`
- [x] `Doc - No Need`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
