jscheffl commented on code in PR #45121:
URL: https://github.com/apache/airflow/pull/45121#discussion_r1895763398
##########
task_sdk/src/airflow/sdk/api/client.py:
##########
@@ -263,6 +267,14 @@ def noop_handler(request: httpx.Request) -> httpx.Response:
return httpx.Response(200, json={"text": "Hello, world!"})
+# Config options for SDK how retries on HTTP requests should be handled
+# Note: Given defaults make attempts after 1, 3, 7, 15, 31seconds, 1:03, 2:07,
3:37 and fails after 5:07min
+# As long as there is no other config facility in SDK we use ENV for the moment
+API_RETRIES = int(os.getenv("AIRFLOW__WORKERS__API_RETRIES", 10))
+API_RETRY_WAIT_MIN = int(os.getenv("AIRFLOW__WORKERS__API_RETRY_WAIT_MIN", 1))
+API_RETRY_WAIT_MAX = int(os.getenv("AIRFLOW__WORKERS__API_RETRY_WAIT_MAX", 90))
Review Comment:
Do you have a proposal where to document? At the moment there is no config
facility for Task SDK and also no section. Yes, the ENVs are new, ususally they
are derived from airflow/config_templates/config.yml but for task SDK we
explicitly don't want to force all backend config to be present in task SDK.
Therefore the config mechanism was not used.
I just introduced these ENVs to have an option to configure and prevent
hard-coding the retry values. More details are to be elaborated in
https://github.com/apache/airflow/issues/44352
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
