SEPURI-SAI-KRISHNA opened a new pull request, #23212:
URL: https://github.com/apache/kafka/pull/23212
`RetryWithToleranceOperator.backoff()` computes the exponential backoff with
an unchecked long shift:
```java
void backoff(int attempt, long deadline) {
int numRetry = attempt - 1;
long delay = RETRIES_DELAY_MIN_MS << numRetry; // 300L <<
numRetry
if (delay > errorMaxDelayInMillis) {
delay = ThreadLocalRandom.current().nextLong(errorMaxDelayInMillis);
}
...
stopRequestedLatch.await(delay, TimeUnit.MILLISECONDS);
}
```
`errors.retry.timeout = -1` is a documented value — `ConnectorConfig`
describes it as "Use -1 for infinite retries." With it, `execAndRetry()` sets
`deadline = Long.MAX_VALUE`, so `attempt` grows without bound while an
operation keeps throwing `RetriableException`. That is the scenario infinite
retries exists for: a sink connector whose downstream system is unavailable for
an extended outage.
Because `numRetry` is unbounded, `300L << numRetry` eventually overflows,
with two consequences.
**The delay becomes non-positive and the jitter cap is bypassed.** At
`numRetry = 55` the shift overflows to `-7638104968020361216`. A negative delay
is not greater than `errorMaxDelayInMillis`, so the branch that applies jitter
is skipped; with an infinite deadline the second clamp is skipped too, and the
negative value reaches `CountDownLatch.await()`, which returns immediately for
a non-positive timeout. The backoff disappears and the connector retries in a
tight loop against the already-failing resource. The same occurs at `numRetry =
58, 60, 61` (negative) and `numRetry = 62, 63` (exactly zero).
**Beyond `numRetry = 63` the ramp silently restarts.** Java masks the shift
distance for a long to its six lowest-order bits, so at `numRetry = 64` the
shift is 0 and the delay returns to 300 ms, then 600 ms, and so on. The backoff
cycles with period 64 rather than remaining capped, repeatedly dropping back to
sub-second retries however long the outage lasts.
Both outcomes defeat the protection `errors.retry.delay.max.ms` is
documented to provide: "Jitter will be added to the delay once this limit is
reached to prevent thundering herd issues."
This is reached in ordinary operating time. With the default
`errors.retry.delay.max.ms` of 60000, attempts 1–8 use the real exponential
values (300 ms through 38400 ms) and attempts 9–55 each wait a jittered
interval uniform in [0, 60000) ms, averaging 30 seconds — so a connector
retrying an unavailable downstream system reaches the first zero-backoff
attempt after roughly 25 minutes, then re-enters the degraded region every 64
attempts.
### Changes
When the shift would overflow, the exponential term is already far beyond
the maximum delay, so it is treated as unbounded (`Long.MAX_VALUE`) and falls
into the existing jitter branch. This keeps the structure of the method intact
and confines the change to the one expression that overflows, so the delay
stays within `[0, errors.retry.delay.max.ms]` for every attempt number.
The threshold is derived from the constant itself via
`Long.numberOfLeadingZeros(RETRIES_DELAY_MIN_MS)` rather than hard-coded, so it
stays correct if `RETRIES_DELAY_MIN_MS` is ever changed.
### Testing
Added
`RetryWithToleranceOperatorTest#testBackoffStaysWithinLimitForLargeAttemptCounts`,
which configures infinite retries and drives the attempt count to 130 — past
both the overflow point and the point where the shift distance wraps —
capturing every `await()` timeout and asserting each falls within `[0,
errors.retry.delay.max.ms]`.
The test was confirmed to fail against the unfixed code, reporting `Backoff
delay -7638104968020361216 is outside the expected range [0, 60000]`, and to
pass with the fix applied. The full `RetryWithToleranceOperatorTest` suite
passes (31 tests, 0 failures), including the existing `testBackoffLimit` case
that pins the exact delays for the first several attempts, confirming normal
backoff behaviour is unchanged.
Co-Authored-By: Claude Sonnet 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
