vandonr-amz commented on code in PR #31388:
URL: https://github.com/apache/airflow/pull/31388#discussion_r1202681028
##########
airflow/providers/amazon/aws/utils/__init__.py:
##########
@@ -47,6 +53,42 @@ def get_airflow_version() -> tuple[int, ...]:
return tuple(int(x) for x in val.split("."))
+def retry_with_backoff(retries=5, backoff_in_seconds=1):
+ """
+ Decorator to retry a function when an exception occurs.
+ Use exponential backoff to spread the retries over time.
+
+ :param retries: the number of retries
+ :param backoff_in_seconds: backoff number in seconds
+ """
+
+ def rwb(func: Callable):
+ def wrapper(*args, **kwargs):
+ x = 0
+ while True:
+ try:
+ return func(*args, **kwargs)
+ except Exception as e:
+ if x == retries:
+ raise
+
+ method_name = f"{func.__module__}.{func.__qualname__}"
+ log.warning(
+ "The method %s failed. Retrying with exponential
backoff. Retry %s. Exception %s.",
+ method_name,
+ retries,
+ e,
+ )
+
+ sleep = backoff_in_seconds * 2**x + random.uniform(0, 1)
Review Comment:
Yeah I read the wikipedia page, and the random part is only mentioned in
"collision avoidance", which does not seem relevant here. It's not problematic
per say as it will work properly like this, but it's just unnecessary.
Using the python package would be nice too, but it means adding a new
dependency, which may be more trouble ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
