shubhamraj-git commented on code in PR #45121:
URL: https://github.com/apache/airflow/pull/45121#discussion_r1895045605
##########
task_sdk/tests/api/test_client.py:
##########
@@ -82,6 +83,122 @@ def handle_request(request: httpx.Request) ->
httpx.Response:
assert err.value.args == ("Not found",)
assert err.value.detail is None
+ @mock.patch("time.sleep", return_value=None)
+ def test_retry_handling_unrecoverable_error(self, mock_sleep):
+ responses: list[httpx.Response] = [
+ *[httpx.Response(500, text="Internal Server Error")] * 11,
+ httpx.Response(200, json={"detail": "Recovered from error - but
will fail before"}),
+ httpx.Response(400, json={"detail": "Should not get here"}),
+ ]
+
+ def mock_handle_request(request: httpx.Request) -> httpx.Response:
+ return responses.pop(0)
+
+ client = Client(
+ base_url=None,
+ dry_run=True,
+ token="",
+ mounts={"'http://": httpx.MockTransport(mock_handle_request)},
+ )
+
Review Comment:
The section of code has been duplicated multiple times. Should we consider
getting them out?
```
@pytest.fixture
def mock_client():
"""Helper fixture to create a mock client with custom responses."""
def _create_client(responses):
def mock_handle_request(request: httpx.Request) -> httpx.Response:
return responses.pop(0)
return Client(
base_url=None,
dry_run=True,
token="",
mounts={"'http://": httpx.MockTransport(mock_handle_request)},
)
return _create_client
```
and use `client = mock_client(responses)`
##########
task_sdk/src/airflow/sdk/api/client.py:
##########
@@ -263,6 +267,14 @@ def noop_handler(request: httpx.Request) -> httpx.Response:
return httpx.Response(200, json={"text": "Hello, world!"})
+# Config options for SDK how retries on HTTP requests should be handled
+# Note: Given defaults make attempts after 1, 3, 7, 15, 31seconds, 1:03, 2:07,
3:37 and fails after 5:07min
+# As long as there is no other config facility in SDK we use ENV for the moment
+API_RETRIES = int(os.getenv("AIRFLOW__WORKERS__API_RETRIES", 10))
+API_RETRY_WAIT_MIN = int(os.getenv("AIRFLOW__WORKERS__API_RETRY_WAIT_MIN", 1))
+API_RETRY_WAIT_MAX = int(os.getenv("AIRFLOW__WORKERS__API_RETRY_WAIT_MAX", 90))
+
Review Comment:
Two things here:
1. Currently, the API_RETRIES, API_RETRY_WAIT_MIN, and API_RETRY_WAIT_MAX
are directly cast to integers. This can raise a ValueError if the environment
variables are not set correctly. Can we add validation or some fallback
defaults (In case of error, if we don't want to fail)?
2. Let's also have a check, `API_RETRY_WAIT_MIN > API_RETRY_WAIT_MAX` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
