alkismavridis commented on issue #62218:
URL: https://github.com/apache/airflow/issues/62218#issuecomment-3943147252
@ashb sorry I forgot to add them. Here they are. This is the moment that the
task dies. This happens consistently at exactly 24h after starting.
```
[2026-02-19T12:10:34.423556Z] {celery_executor_utils.py:154} INFO -
[a6915c74-3b86-46df-a0f2-2bd3848a98f4] Executing workload in Celery:
token='eyJ***' ti=TaskInstance(id=UUID('019c70a7-fb7c-7480-84d0-7232151770ed'),
dag_version_id=UUID('019c4721-09cb-7d79-8402-e3925d3ab5b2'),
task_id='mt68-gbr-crawler', dag_id='selbstaendige-gbr-crawler',
run_id='manual__2026-02-18T12:09:40+00:00', try_number=1, map_index=-1,
pool_slots=1, queue='default', priority_weight=1, executor_config=None,
parent_context_carrier={}, context_carrier={})
dag_rel_path=PurePosixPath('selbstaendige/gbr_crawler.py')
bundle_info=BundleInfo(name='dags-folder', version=None)
log_path='dag_id=selbstaendige-gbr-crawler/run_id=manual__2026-02-18T12:09:40+00:00/task_id=mt68-gbr-crawler/attempt=1.log'
type='ExecuteTask'
[2026-02-19T12:10:34.425926Z] {supervisor.py:1975} INFO - Secrets backends
loaded for worker count=1 backend_classes=['EnvironmentVariablesBackend']
[2026-02-19T12:10:34.448869Z] {supervisor.py:710} INFO - Process exited
pid=123088 exit_code=-9 signal_sent=SIGKILL
[2026-02-19T12:10:34.462386Z] {trace.py:285} ERROR - Task
execute_workload[a6915c74-3b86-46df-a0f2-2bd3848a98f4] raised unexpected:
ServerResponseError('Invalid auth token: Signature has expired')
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.12/site-packages/celery/app/trace.py", line
479, in trace_task
R = retval = fun(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/celery/app/trace.py", line
779, in __protected_call__
return self.run(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/celery/executors/celery_executor_utils.py",
line 162, in execute_workload
supervise(
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py",
line 1984, in supervise
process = ActivitySubprocess.start(
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py",
line 955, in start
proc._on_child_started(ti=what, dag_rel_path=dag_rel_path,
bundle_info=bundle_info)
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py",
line 966, in _on_child_started
ti_context = self.client.task_instances.start(ti.id, self.pid,
start_date)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py",
line 215, in start
resp = self.client.patch(f"task-instances/{id}/run",
content=body.model_dump_json())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 1218, in patch
return self.request(
^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
338, in wrapped_f
return copy(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
477, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
378, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
400, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449,
in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401,
in __get_result
raise self._exception
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
480, in __call__
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py",
line 887, in request
return super().request(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 825, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 914, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 942, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 999, in _send_handling_redirects
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 982, in _send_handling_redirects
hook(response)
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py",
line 186, in raise_on_4xx_5xx_with_note
return get_json_error(response) or response.raise_for_status()
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py",
line 176, in get_json_error
raise err
airflow.sdk.api.client.ServerResponseError: Invalid auth token: Signature
has expired
Correlation-id=019c75cf-2041-7fc7-a554-970c6a610cb3
INFO: 172.25.0.5:52116 - "GET
/log/dag_id%3Dselbstaendige-gbr-crawler/run_id%3Dmanual__2026-02-18T12%3A09%3A40%2B00%3A00/task_id%3Dmt68-gbr-crawler/attempt%3D1.log
HTTP/1.1" 200 OK
[2026-02-19T12:10:36.186345Z] {supervisor.py:1115} ERROR - Server indicated
the task shouldn't be running anymore detail={'detail': {'reason':
'not_running', 'message': 'TI is no longer in the running state and task should
terminate', 'current_state': 'failed'}} status_code=409
ti_id=UUID('019c70a7-fb7c-7480-84d0-7232151770ed')
[2026-02-19T12:10:41.192109Z] {supervisor.py:718} WARNING - Process did not
terminate in time; escalating pid=3873 signal=SIGTERM
[2026-02-19T12:10:41.201091Z] {supervisor.py:710} INFO - Process exited
pid=3873 exit_code=-9 signal_sent=SIGKILL
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
