1991santhu commented on PR #11654: URL: https://github.com/apache/seatunnel/pull/11654#issuecomment-5184517576
Adding a measurement rather than leaving this as an argument. I simulated the arrival times this backoff produces for 100 parallel readers and writers that hit the Airtable rate limit together — which is the normal case, since a parallel job's subtasks share one quota and so cross it at the same moment. The metric is the peak number of requests landing inside any 1s window. Parameters are the connector's defaults: `rate_limit_backoff_ms=30000`, `rate_limit_max_retries=3`, `MAX_BACKOFF_MILLIS=300000`, giving 30s, 60s, 120s. ``` attempt none equal added 1 100.0 13.2 8.4 2 100.0 8.7 6.1 3 100.0 5.8 4.3 ``` Without jitter all 100 subtasks retry in the same second at every attempt, and the peak never decays — a deterministic delay preserves the correlation that caused the 429 in the first place, so each retry round re-forms the burst. With the jitter in this PR the first retry peaks at ~8 and falls to ~4, a **12x reduction rising to 23x**. The `equal` column is equal jitter. It decorrelates less, and it would retry sooner than `rate_limit_backoff_ms` was set to — which is the substantive problem when the delay exists to respect a rate limiter, so the jitter is added rather than centred. Being straight about what this is: a simulation of the arrival process, not a production incident report. I have not observed this against Airtable and am not claiming anyone has — it quantifies the mechanism the code implies. Dependency-free Go, happy to attach. ``` go run . -clients=100 -schedule=30s,60s,120s -cap=300s -floor=30s -window=1s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
