1991santhu commented on PR #11654:
URL: https://github.com/apache/seatunnel/pull/11654#issuecomment-5183753678
Pushed a correction to my own patch — the first revision missed the case
where the backoff matters most.
I bounded the added jitter by the headroom to `MAX_BACKOFF_MILLIS`. At the
maximum that headroom is zero, so the wait came back unjittered, and every
retry past the point where the sequence tops out put concurrent readers and
writers back into lockstep — precisely when the rate limit is at its most
persistent. The original patch spread the early retries and left the late ones
exactly as they were.
The wait is now spread downwards once there is nothing left to add into:
```java
long floor = Math.max(waitMillis / 2, rateLimitBackoffMs);
if (floor >= waitMillis) {
return waitMillis;
}
return waitMillis - ThreadLocalRandom.current().nextLong(waitMillis - floor
+ 1);
```
On the direction: `MAX_BACKOFF_MILLIS` is an upper bound rather than a
target, so drawing below it breaks nothing. `rate_limit_backoff_ms` is the
opposite — it is the wait Airtable's 429 handling is configured around, and the
comment above this method promises the wait is never shorter than it asked for.
So the downward spread is floored there rather than at half the interval. That
floor binds whenever `rate_limit_backoff_ms` is above half the maximum, and
there is a test for it.
Reachability at the defaults: `rate_limit_backoff_ms=30000` and
`rate_limit_max_retries=3` top out at 120s, below the 300s maximum, so the
default configuration never reaches the saturated case. It needs
`rate_limit_max_retries >= 5`. Configurable, so worth fixing, but I want to be
straight that this is not a default-path bug.
Applied to both `AirtableSourceReader` and `AirtableSinkWriter`, which carry
the same method.
Verified with JDK 11:
```
./mvnw -o test -pl
seatunnel-connectors-v2/connector-http/connector-http-airtable \
-Dtest='AirtableSourceReaderTest,AirtableSinkWriterTest'
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0 - AirtableSourceReaderTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0 - AirtableSinkWriterTest
BUILD SUCCESS
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
