vincbeck commented on code in PR #31388:
URL: https://github.com/apache/airflow/pull/31388#discussion_r1198950724
##########
airflow/providers/amazon/aws/utils/__init__.py:
##########
@@ -47,6 +53,42 @@ def get_airflow_version() -> tuple[int, ...]:
return tuple(int(x) for x in val.split("."))
+def retry_with_backoff(retries=5, backoff_in_seconds=1):
+ """
+ Decorator to retry a function when an exception occurs.
+ Use exponential backoff to spread the retries over time.
+
+ :param retries: the number of retries
+ :param backoff_in_seconds: backoff number in seconds
+ """
+
+ def rwb(func: Callable):
+ def wrapper(*args, **kwargs):
+ x = 0
+ while True:
+ try:
+ return func(*args, **kwargs)
+ except Exception as e:
+ if x == retries:
+ raise
+
+ method_name = f"{func.__module__}.{func.__qualname__}"
+ log.warning(
+ "The method %s failed. Retrying with exponential
backoff. Retry %s. Exception %s.",
+ method_name,
+ retries,
+ e,
+ )
+
+ sleep = backoff_in_seconds * 2**x + random.uniform(0, 1)
Review Comment:
You can find multiple documentation/specifications on Google:
-
[Example](https://cloud.google.com/iot/docs/how-tos/exponential-backoff#:~:text=An%20exponential%20backoff%20algorithm%20retries,seconds%20and%20retry%20the%20request.)
- [Example](https://en.wikipedia.org/wiki/Exponential_backoff)
I am just trying to use an existing algorithm and have an implementation for
it. I dont think we should modify the algorithm here. But if this is something
you are really against I can remove it, I just dont see the harm here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
