potiuk edited a comment on pull request #21852:
URL: https://github.com/apache/airflow/pull/21852#issuecomment-1055258678
> > Exponential backoff still tries too early in most situations. A client
receiving 429 is supposed to wait at least until the date specified in the
`Retry-After` response header before any retries.
>
> Sure. Retry-After should be the source of first retry time - but
exponential back-off after that does not hurt.
Just to add a bit more reasoning (I thought a bit about it).
The problem is that Retry-After is only a hint, not a "source of truth". It
relies on the fact the server "knows" what it is doing. Which is not necessary
valid:
a) it might be based on past information (which might be outdated and might
not include the mounting spike traffic properly) - this happens often
b) it might be simply not there. often you will not get 429 but 5XX in
similar situations because it's not only the server that gets flooded but also
some gateways on the way or simply the server might timeout or run out of
memory or other resources.
So IMHO it needs to be client to decide how to behave. One added value of
exponential backoff is that it is still helpful in all retriable conditions
that are "unknown" - i.e. 5XX. Those do not contain "Retry-After" to base your
decision on.
So I think exponential back-off with some initial timeout (if Retry-After is
available - it should be a starting point) should be the right approach.
Additionally if even including exponential back-off, you get 429 with
Retry-After where your exponential back-off next step is not long enough - the
timeout should be re-adjusted to the "Retry-After" received. But if it is
longer, then we should continue exponential back-off (because server
information might be already out-dated and not include mounting traffic spike).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
