Jeoffreybauvin opened a new issue, #64212: URL: https://github.com/apache/airflow/issues/64212
### Apache Airflow version 3.1.8 ### If "Other Airflow 3 version" selected, which one? _No response_ ### What happened? I have a dag scheduled every 10 minutes. The 3rd one always fails .... <img width="242" height="192" alt="Image" src="https://github.com/user-attachments/assets/3175f032-6678-406d-93a0-33761264ace0"; /> On worker side I can see some timeouts : ``` 2026-03-25T10:30:01.055984Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:01.056166Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 --- Supervised process Last chance exception handler --- Traceback (most recent call last): File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1112, in run result = _execute_task(context=context, ti=ti, log=log) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1523, in _execute_task result = ctx.run(execute, context=context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 417, in wrapper return func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/decorator.py", line 252, in execute return_value = super().execute(context) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 417, in wrapper return func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 228, in execute return_value = self.execute_callable() ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 251, in execute_callable return runner.run(*self.op_args, **self.op_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/callback_runner.py", line 82, in run return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/airflow/dags/live/dags/dag_netbox_devices_maintenance.py", line 69, in link_vms xcom_value = context["ti"].xcom_pull( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 366, in xcom_pull values = XCom.get_all( ^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/xcom.py", line 304, in get_all msg = SUPERVISOR_COMMS.send( ^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", line 206, in send return self._get_response() ^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", line 269, in _get_response frame = self._read_frame() ^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", line 230, in _read_frame len_bytes = self.socket.recv(4) ^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/timeout.py", line 37, in handle_timeout self.log.error("Process timed out", pid=os.getpid()) File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/_shared/logging/structlog.py", line 95, in meth return self._proxy_to_logger(name, event, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_base.py", line 223, in _proxy_to_logger return getattr(self._logger, method_name)(*args, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_output.py", line 321, in msg self._write(message + b"\n") BrokenPipeError: [Errno 32] Broken pipe During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1186, in run log.exception("Task failed with exception") File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_native.py", line 47, in exception return self.error(event, *args, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/_shared/logging/structlog.py", line 95, in meth return self._proxy_to_logger(name, event, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_base.py", line 223, in _proxy_to_logger return getattr(self._logger, method_name)(*args, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_output.py", line 321, in msg self._write(message + b"\n") BrokenPipeError: [Errno 32] Broken pipe During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1697, in main state, _, error = run(ti, context, log) ^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1191, in run f"ti.finish.{ti.dag_id}.{ti.task_id}.{state.value}", ^^^^^ UnboundLocalError: cannot access local variable 'state' where it is not associated with a value During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 388, in _fork_main target() File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 206, in _subprocess_main main() File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1704, in main log.exception("Top level error") File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_native.py", line 47, in exception return self.error(event, *args, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/_shared/logging/structlog.py", line 95, in meth return self._proxy_to_logger(name, event, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_base.py", line 223, in _proxy_to_logger return getattr(self._logger, method_name)(*args, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_output.py", line 321, in msg self._write(message + b"\n") BrokenPipeError: [Errno 32] Broken pipe 2026-03-25T10:30:01.425867Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:02.130218Z [debug ] heartbeat_tick : for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:730 2026-03-25T10:30:02.130481Z [debug ] heartbeat_tick : Prev sent/recv: 4993/1325006, now - 4993/1325174, monotonic - 420551.394657526, last_heartbeat_sent - 420531.392953123, heartbeat int. - 60 for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:748 2026-03-25T10:30:02.130650Z [debug ] heartbeat_tick: sending heartbeat for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:764 2026-03-25T10:30:02.197240Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:06.425046Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:07.196062Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:10.981992Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:10.982286Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:10.982729Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:10.983217Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client] loc=before.py:40 2026-03-25T10:30:11.057156Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:11.057421Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:11.057769Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:11.058240Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client] loc=before.py:40 2026-03-25T10:30:11.425105Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:12.197500Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:14.092457Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': '569d87ec-8b31-3e7b-8124-30a33342604d'} ticket:4198e42b-4207-4725-9e0d-f1eba84c01bb] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:16.425168Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:17.196067Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:20.984262Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47 2026-03-25T10:30:20.996950Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bcd4a090> [httpcore.connection] loc=_trace.py:47 2026-03-25T10:30:20.997314Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0fd8d50> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47 2026-03-25T10:30:21.000576Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc65aa50> [httpcore.connection] loc=_trace.py:47 2026-03-25T10:30:21.000826Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:21.001151Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:21.001307Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:21.001478Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:21.001610Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:21.059241Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47 2026-03-25T10:30:21.080066Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58af044f80> [httpcore.connection] loc=_trace.py:47 2026-03-25T10:30:21.080298Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0faedd0> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47 2026-03-25T10:30:21.083181Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58af023d10> [httpcore.connection] loc=_trace.py:47 2026-03-25T10:30:21.083411Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:21.083728Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:21.083900Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:21.084075Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:21.084207Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:21.425544Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:22.132032Z [debug ] heartbeat_tick : for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:730 2026-03-25T10:30:22.132399Z [debug ] heartbeat_tick : Prev sent/recv: 4993/1325174, now - 4994/1325333, monotonic - 420571.396551106, last_heartbeat_sent - 420571.396549525, heartbeat int. - 60 for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:748 2026-03-25T10:30:22.197758Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:26.427229Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:27.196171Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:27.871188Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'ec90397f-5794-3113-80d2-7121be96b7b6'} ticket:88299c2e-82b9-4e03-98fd-0323ac37d8fe] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:31.012018Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:31.012272Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:31.012521Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:31.012828Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client] loc=before.py:40 2026-03-25T10:30:31.094629Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:31.094944Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:31.095310Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:31.095613Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client] loc=before.py:40 2026-03-25T10:30:31.425628Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:32.197584Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:36.425520Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:37.196474Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:41.014457Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47 2026-03-25T10:30:41.018476Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc673cb0> [httpcore.connection] loc=_trace.py:47 2026-03-25T10:30:41.018686Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0fd8d50> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47 2026-03-25T10:30:41.021958Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc672ff0> [httpcore.connection] loc=_trace.py:47 2026-03-25T10:30:41.022183Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:41.022500Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:41.022699Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:41.022894Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:41.023032Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:41.096446Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47 2026-03-25T10:30:41.099030Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58aee0a090> [httpcore.connection] loc=_trace.py:47 2026-03-25T10:30:41.099257Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0faedd0> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47 2026-03-25T10:30:41.102609Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58aee0ae70> [httpcore.connection] loc=_trace.py:47 2026-03-25T10:30:41.102853Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:41.103176Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:41.103333Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:41.103505Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:41.103636Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:41.413066Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'b96a3d63-ef49-35b5-87e7-8a7c5f29634b'} ticket:9d6bfcc8-db46-42d9-94df-52b8394f82e9] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:41.425611Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:42.134000Z [debug ] heartbeat_tick : for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:730 2026-03-25T10:30:42.134280Z [debug ] heartbeat_tick : Prev sent/recv: 4994/1325333, now - 4994/1325498, monotonic - 420591.398436219, last_heartbeat_sent - 420571.396549525, heartbeat int. - 60 for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:748 2026-03-25T10:30:42.134482Z [debug ] heartbeat_tick: sending heartbeat for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:764 2026-03-25T10:30:42.197596Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:46.426644Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:47.197483Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:51.024749Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:51.025049Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:51.025313Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:51.025579Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client] loc=before.py:40 2026-03-25T10:30:51.114102Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:51.114455Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:51.114779Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:30:51.115225Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client] loc=before.py:40 2026-03-25T10:30:51.425463Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:52.197710Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:55.024679Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': '24170662-5cfd-39ac-8495-ffa8773da31f'} ticket:4a9d24f7-1bd0-4491-b1bc-1fea82954d40] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:56.426185Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:30:57.196374Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:31:01.026491Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47 2026-03-25T10:31:01.029477Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc6816a0> [httpcore.connection] loc=_trace.py:47 2026-03-25T10:31:01.029706Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0fd8d50> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47 2026-03-25T10:31:01.033288Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc680dd0> [httpcore.connection] loc=_trace.py:47 2026-03-25T10:31:01.033580Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:01.034072Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:01.034313Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:01.034606Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:01.034778Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:01.116463Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47 2026-03-25T10:31:01.118559Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58aee27800> [httpcore.connection] loc=_trace.py:47 2026-03-25T10:31:01.118876Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0faedd0> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47 2026-03-25T10:31:01.123325Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58aee26840> [httpcore.connection] loc=_trace.py:47 2026-03-25T10:31:01.123614Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:01.124083Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:01.124279Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:01.124484Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:01.124695Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:01.425581Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:31:02.136276Z [debug ] heartbeat_tick : for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:730 2026-03-25T10:31:02.136610Z [debug ] heartbeat_tick : Prev sent/recv: 4994/1325498, now - 4995/1325657, monotonic - 420611.400776736, last_heartbeat_sent - 420611.400775552, heartbeat int. - 60 for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:748 2026-03-25T10:31:02.197911Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:31:06.425600Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:31:07.196376Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:31:08.547138Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'ad3a4b26-88db-32b9-8b72-c6257848e8b6'} ticket:02fcf9a7-2d2e-467c-b411-9e45e8eb1413] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:31:11.045252Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:11.045532Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:11.045762Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:11.046066Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client] loc=before.py:40 2026-03-25T10:31:11.135225Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:11.135532Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:11.135781Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:11.136095Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client] loc=before.py:40 2026-03-25T10:31:11.425077Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:31:12.197613Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:31:16.425981Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:31:17.197156Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:31:21.046940Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47 2026-03-25T10:31:21.049988Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc6820c0> [httpcore.connection] loc=_trace.py:47 2026-03-25T10:31:21.050205Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0fd8d50> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47 2026-03-25T10:31:21.053581Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc681a00> [httpcore.connection] loc=_trace.py:47 2026-03-25T10:31:21.053835Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:21.054143Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:21.054293Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:21.054460Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:21.054587Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:21.137066Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47 2026-03-25T10:31:21.139022Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58af206ed0> [httpcore.connection] loc=_trace.py:47 2026-03-25T10:31:21.139237Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0faedd0> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47 2026-03-25T10:31:21.142489Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58af205b20> [httpcore.connection] loc=_trace.py:47 2026-03-25T10:31:21.142716Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:21.143039Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:21.143200Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:21.143370Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:21.143501Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:21.426376Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:31:22.110421Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'b8885937-29a4-366c-8887-e2a0601f26f9'} ticket:667cb126-3953-471d-8711-07996e013455] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:31:22.137485Z [debug ] heartbeat_tick : for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:730 2026-03-25T10:31:22.137742Z [debug ] heartbeat_tick : Prev sent/recv: 4995/1325657, now - 4995/1325822, monotonic - 420631.401909165, last_heartbeat_sent - 420611.400775552, heartbeat int. - 60 for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:748 2026-03-25T10:31:22.137919Z [debug ] heartbeat_tick: sending heartbeat for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:764 2026-03-25T10:31:22.196920Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:31:26.426201Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:31:27.196677Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:31:31.065029Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:31.065299Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:31.065533Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:31.118060Z [error ] Task execute_workload[51cc03c0-8c9c-4afc-aad0-bd5527c889b1] raised unexpected: ReadTimeout('The read operation timed out') [celery.app.trace] loc=trace.py:285 Traceback (most recent call last): File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions yield File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 250, in handle_request resp = self._pool.handle_request(req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request raise exc from None File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request response = connection.handle_request( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 103, in handle_request return self._connection.handle_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 136, in handle_request raise exc File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 106, in handle_request ) = self._receive_response_headers(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 177, in _receive_response_headers event = self._receive_event(timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 217, in _receive_event data = self._network_stream.read( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_backends/sync.py", line 126, in read with map_exceptions(exc_map): ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions raise to_exc(exc) from exc httpcore.ReadTimeout: The read operation timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/trace.py", line 479, in trace_task R = retval = fun(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/trace.py", line 779, in __protected_call__ return self.run(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/celery/executors/celery_executor_utils.py", line 202, in execute_workload supervise( File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1993, in supervise exit_code = process.wait() ^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 996, in wait self._monitor_subprocess() File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1061, in _monitor_subprocess alive = self._service_subprocess(max_wait_time=max_wait_time) is None ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 773, in _service_subprocess need_more = socket_handler(key.fileobj) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1770, in cb gen.send(request) File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 614, in handle_requests self._handle_request(msg, log, request.id) File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1247, in _handle_request xcoms = self.client.xcoms.get_sequence_slice( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 570, in get_sequence_slice resp = self.client.get(f"xcoms/{dag_id}/{run_id}/{task_id}/{key}/slice", params=params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1053, in get return self.request( ^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 331, in wrapped_f return copy(f, *args, **kw) ^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 470, in __call__ do = self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 371, in iter result = action(retry_state) ^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 413, in exc_check raise retry_exc.reraise() ^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 184, in reraise raise self.last_attempt.result() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 473, in __call__ result = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 887, in request return super().request(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in request return self.send(request, auth=auth, follow_redirects=follow_redirects) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send response = self._send_handling_auth( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth response = self._send_handling_redirects( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 979, in _send_handling_redirects response = self._send_single_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1014, in _send_single_request response = transport.handle_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 249, in handle_request with map_httpcore_exceptions(): ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.ReadTimeout: The read operation timed out 2026-03-25T10:31:31.153931Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:31.154179Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:31.154394Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47 2026-03-25T10:31:31.206917Z [error ] Task execute_workload[eedcb4b9-06ba-4366-8e07-cf16ad60bd25] raised unexpected: ReadTimeout('The read operation timed out') [celery.app.trace] loc=trace.py:285 Traceback (most recent call last): File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions yield File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 250, in handle_request resp = self._pool.handle_request(req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request raise exc from None File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request response = connection.handle_request( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 103, in handle_request return self._connection.handle_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 136, in handle_request raise exc File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 106, in handle_request ) = self._receive_response_headers(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 177, in _receive_response_headers event = self._receive_event(timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 217, in _receive_event data = self._network_stream.read( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_backends/sync.py", line 126, in read with map_exceptions(exc_map): ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions raise to_exc(exc) from exc httpcore.ReadTimeout: The read operation timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/trace.py", line 479, in trace_task R = retval = fun(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/trace.py", line 779, in __protected_call__ return self.run(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/celery/executors/celery_executor_utils.py", line 202, in execute_workload supervise( File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1993, in supervise exit_code = process.wait() ^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 996, in wait self._monitor_subprocess() File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1061, in _monitor_subprocess alive = self._service_subprocess(max_wait_time=max_wait_time) is None ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 773, in _service_subprocess need_more = socket_handler(key.fileobj) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1770, in cb gen.send(request) File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 614, in handle_requests self._handle_request(msg, log, request.id) File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1247, in _handle_request xcoms = self.client.xcoms.get_sequence_slice( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 570, in get_sequence_slice resp = self.client.get(f"xcoms/{dag_id}/{run_id}/{task_id}/{key}/slice", params=params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1053, in get return self.request( ^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 331, in wrapped_f return copy(f, *args, **kw) ^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 470, in __call__ do = self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 371, in iter result = action(retry_state) ^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 413, in exc_check raise retry_exc.reraise() ^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 184, in reraise raise self.last_attempt.result() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 473, in __call__ result = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 887, in request return super().request(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in request return self.send(request, auth=auth, follow_redirects=follow_redirects) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send response = self._send_handling_auth( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth response = self._send_handling_redirects( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 979, in _send_handling_redirects response = self._send_single_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1014, in _send_single_request response = transport.handle_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 249, in handle_request with map_httpcore_exceptions(): ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.ReadTimeout: The read operation timed out 2026-03-25T10:31:31.425181Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 2026-03-25T10:31:32.197312Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98 ``` On webserver side, nothing at all, evenr with DEBUG log level :( . 90% on my dags works fine, but sometimes, I have some timeout on the webserver. I don't know how to debug it properly. ### What you think should happen instead? no timeout :D. ### How to reproduce No sure, the dag is pretty straight forward, the dag starts logging things, and "nothing". ### Operating System Ubuntu 24.04.4 LTS ### Versions of Apache Airflow Providers apache-airflow-providers-amazon==9.22.0 apache-airflow-providers-celery==3.17.0 apache-airflow-providers-cncf-kubernetes==10.13.0 apache-airflow-providers-common-compat==1.14.0 apache-airflow-providers-common-io==1.7.1 apache-airflow-providers-common-messaging==2.0.2 apache-airflow-providers-common-sql==1.32.0 apache-airflow-providers-docker==4.5.2 apache-airflow-providers-elasticsearch==6.5.0 apache-airflow-providers-fab==3.4.0 apache-airflow-providers-ftp==3.14.1 apache-airflow-providers-git==0.2.4 apache-airflow-providers-google==20.0.0 apache-airflow-providers-grpc==3.9.2 apache-airflow-providers-hashicorp==4.5.0 apache-airflow-providers-http==6.0.0 apache-airflow-providers-microsoft-azure==13.0.0 apache-airflow-providers-mysql==6.5.0 apache-airflow-providers-odbc==4.12.0 apache-airflow-providers-openlineage==2.11.0 apache-airflow-providers-postgres==6.6.0 apache-airflow-providers-redis==4.4.2 apache-airflow-providers-sendgrid==4.2.1 apache-airflow-providers-sftp==5.7.0 apache-airflow-providers-slack==9.7.0 apache-airflow-providers-smtp==2.4.2 apache-airflow-providers-snowflake==6.10.0 apache-airflow-providers-ssh==4.3.1 apache-airflow-providers-standard==1.12.0 ### Deployment Docker-Compose ### Deployment details _No response_ ### Anything else? ``` AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: mysql+mysqldb://.... AIRFLOW__CELERY__RESULT_BACKEND: db+mysql://.... AIRFLOW__CORE__EXECUTOR: CeleryExecutor AIRFLOW__LOGGING__LOGGING_LEVEL: DEBUG AIRFLOW__CORE__DAGS_FOLDER: "/opt/airflow/dags/live/dags" AIRFLOW__CORE__LAZY_LOAD_PLUGINS: 'false' AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true' AIRFLOW__CORE__LOAD_EXAMPLES: 'False' AIRFLOW__CORE__DEFAULT_TASK_EXECUTION_TIMEOUT: 43200 AIRFLOW__CORE__DEFAULT_TIMEZONE: Europe/Paris AIRFLOW__EMAIL__DEFAULT_EMAIL_ON_FAILURE: 'true' AIRFLOW__EMAIL__DEFAULT_EMAIL_ON_RETRY: 'true' AIRFLOW__SMTP__SMTP_HOST: prod.all.mailhub.dc01 AIRFLOW__SMTP__SMTP_STARTTLS: 'false' AIRFLOW__SMTP__SMTP_SSL: 'false' AIRFLOW__API__INSTANCE_NAME: Airflow INFRA PROD DC01 AIRFLOW__API__EXPOSE_CONFIG: 'True' AIRFLOW__API__SSL_CERT: "/opt/airflow/api.pem" AIRFLOW__API__SSL_KEY: "/opt/airflow/api.key" AIRFLOW__CELERY__BROKER_URL: amqp://app_inf_airflow01_prod:4hd6633efb5qpqh6a31que...@prod.inf-airflowbroker-mq-amqp.service.consul:5672/inf-airflow3-broker01-prod AIRFLOW__CELERY__BROKER_USE_SSL: 'False' AIRFLOW__CELERY__FLOWER_URL_PREFIX: "/flower" AIRFLOW__CELERY__BROKER_CONNECTION_MAX_RETRIES: 10 AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL: '20' AIRFLOW__DAG_PROCESSOR__MIN_FILE_PROCESS_INTERVAL: '300' AIRFLOW__DAG_PROCESSOR__STALE_DAG_THRESHOLD: '300' AIRFLOW__DAG_PROCESSOR__DAG_FILE_PROCESSOR_TIMEOUT: '300' AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT: '300' AIRFLOW__SCHEDULER__CATCHUP_BY_DEFAULT: 'False' TZ: Europe/Paris HTTP_TIMEOUT: 60 REQUESTS_CA_BUNDLE: "/etc/ssl/certs/ca-certificates.crt" PYTHONPATH: "/opt/airflow/dags/live/dags" AIRFLOW__METRICS__STATSD_ON: 'True' ``` AIRFLOW__METRICS__STATSD_PREFIX: inf_airflow02.prod AIRFLOW__METRICS__STATSD_HOST: 10.25.30.35 AIRFLOW__METRICS__STATSD_PORT: '9125' AIRFLOW__METRICS__STATSD_INFLUXDB_ENABLED: 'True' AIRFLOW__KERBEROS__KINIT_PATH: "/usr/bin/kinit.heimdal" AIRFLOW__KERBEROS__REINIT_FREQUENCY: '3600' AIRFLOW__KERBEROS__KEYTABS_PATH: "/opt/airflow/security" AIRFLOW__KERBEROS__CCACHE_PATH: "/opt/airflow/kerberos_cache" AIRFLOW__FAB__ENABLE_PROXY_FIX: 'True' AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager AIRFLOW__LOGGING__REMOTE_LOGGING: 'True' AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: s3://prod-airflow01-logs/logs AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: s3_infra AIRFLOW__EXECUTION_API__JWT_EXPIRATION_TIME: '86400' AIRFLOW__WORKERS__SOCKET_CLEANUP_TIMEOUT: '120' AIRFLOW__WORKERS__EXECUTION_API_RETRY_WAIT_MIN: '10' AIRFLOW__WORKERS__MAX_FAILED_HEARTBEATS: '5' AIRFLOW__WORKERS__EXECUTION_API_TIMEOUT: '10' AIRFLOW__CORE__XCOM_BACKEND: airflow.providers.common.io.xcom.backend.XComObjectStorageBackend AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH: s3://s3_infra@prod-airflow01-logs/xcom AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD: '0' AIRFLOW__API__BASE_URL: https://prod-scheduler-inf.int.test.net AIRFLOW__CORE__EXECUTION_API_SERVER_URL: https://prodv3.inf-airflow-webserver-http.service.consul:8080/execution/ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
