Bharath Reddy Gunapati created FLINK-41220:
----------------------------------------------
Summary: Honour Retry-After and add jitter to HTTP sink retry
delays
Key: FLINK-41220
URL: https://issues.apache.org/jira/browse/FLINK-41220
Project: Flink
Issue Type: Sub-task
Components: Connectors / HTTP
Reporter: Bharath Reddy Gunapati
The sink retries with a fixed or exponential delay (FLINK-40277) and ignores
the {{Retry-After}} header that most rate-limited APIs send with 429 and 503. A
throttled request is therefore retried too early and throttled again. Also,
after an outage all parallel subtasks retry at the same instant because the
exponential backoff has no randomisation, which is the classic thundering-herd
pattern.
h3. Proposed scope
* When a retryable response carries {{Retry-After}} (seconds or HTTP date), use
it as the delay for that retry, capped by a configurable maximum.
* Add {{http.sink.retry-strategy.exponential-delay.jitter}} exposing
Resilience4j's randomisation factor; consider the same option for the lookup
source for parity.
* Defaults keep today's behaviour: no jitter, header ignored only if the option
is off.
h3. Acceptance criteria
* A 429 with {{Retry-After: 2}} delays the retry by about two seconds
regardless of the configured strategy; without the header the configured
strategy applies.
* With jitter configured, retry delays vary within the expected range.
* Documented in the retry strategy section for sink (and lookup if added).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
