GitHub user Neon-Face added a comment to the discussion: [Bug]
`standard.operators.python` fails the task randomly
The full scheduler return error:
```
[2025-08-11T11:46:06.659+0000] {dagrun.py:1039} INFO - Found span_status
'ended', while updating state for dag_run
'manual__2025-08-11T11:20:19.498701+00:00'
2025-08-11 11:46:06 [debug ] Received message from task runner [supervisor]
msg=SucceedTask(state='success', end_date=datetime.datetime(2025, 8, 11, 11,
46, 6, 660306, tzinfo=datetime.timezone.utc), task_outlets=[],
outlet_events=[], rendered_map_index=None, type='SucceedTask')
[2025-08-11T11:46:06.719+0000] {_client.py:1026} INFO - HTTP Request: PATCH
http://localhost:8080/execution/task-instances/019898ef-0cfd-7ba2-a315-cb5caf28eb36/state
"HTTP/1.1 204 No Content"
[2025-08-11T11:46:07.714+0000] {dagrun.py:1176} INFO - Marking run <DagRun
Insights_IPv6_Adoption @ 2025-08-11 11:20:18.505000+00:00:
manual__2025-08-11T11:20:19.498701+00:00, state:running, queued_at: 2025-08-11
11:41:00.039941+00:00. run_type: manual> successful
[2025-08-11T11:46:07.715+0000] {dagrun.py:1226} INFO - DagRun Finished:
dag_id=Insights_IPv6_Adoption, logical_date=2025-08-11 11:20:18.505000+00:00,
run_id=manual__2025-08-11T11:20:19.498701+00:00, run_start_date=2025-08-11
11:41:00.180309+00:00, run_end_date=2025-08-11 11:46:07.715110+00:00,
run_duration=307.534801, state=success, run_type=manual,
data_interval_start=2025-08-11 11:20:18.505000+00:00,
data_interval_end=2025-08-11 11:20:18.505000+00:00,
2025-08-11 11:46:11 [info ] Task finished [supervisor]
duration=310.4626057599671 exit_code=0 final_state=success
[2025-08-11T11:46:12.779+0000] {scheduler_job_runner.py:811} INFO - Received
executor event with state success for task instance
TaskInstanceKey(dag_id='Insights_IPv6_Adoption',
task_id='download_cloudflare_ipv6_data',
run_id='manual__2025-08-11T11:20:19.498701+00:00', try_number=5, map_index=-1)
[2025-08-11T11:46:12.783+0000] {scheduler_job_runner.py:853} INFO -
TaskInstance Finished: dag_id=Insights_IPv6_Adoption,
task_id=download_cloudflare_ipv6_data,
run_id=manual__2025-08-11T11:20:19.498701+00:00, map_index=-1,
run_start_date=2025-08-11 11:41:01.324087+00:00, run_end_date=2025-08-11
11:46:06.660306+00:00, run_duration=310.336, state=success,
executor=LocalExecutor(parallelism=64), executor_state=success, try_number=5,
max_tries=4, pool=default_pool, queue=default, priority_weight=1,
operator=_PythonDecoratedOperator, queued_dttm=2025-08-11
11:41:01.254731+00:00, scheduled_dttm=2025-08-11
11:41:01.224247+00:00,queued_by_job_id=7, pid=1415904
[2025-08-11T11:47:30.971+0000] {dagrun.py:1039} INFO - Found span_status
'ended', while updating state for dag_run
'manual__2025-08-11T11:20:19.498701+00:00'
[2025-08-11T11:47:30.999+0000] {scheduler_job_runner.py:451} INFO - 1 tasks up
for execution:
<TaskInstance: Insights_IPv6_Adoption.download_cloudflare_ipv6_data
manual__2025-08-11T11:20:19.498701+00:00 [scheduled]>
[2025-08-11T11:47:30.999+0000] {scheduler_job_runner.py:523} INFO - DAG
Insights_IPv6_Adoption has 0/32 running and queued tasks
[2025-08-11T11:47:30.999+0000] {scheduler_job_runner.py:662} INFO - Setting the
following tasks to queued state:
<TaskInstance: Insights_IPv6_Adoption.download_cloudflare_ipv6_data
manual__2025-08-11T11:20:19.498701+00:00 [scheduled]>
[2025-08-11T11:47:31.000+0000] {scheduler_job_runner.py:768} INFO - Trying to
enqueue tasks: [<TaskInstance:
Insights_IPv6_Adoption.download_cloudflare_ipv6_data
manual__2025-08-11T11:20:19.498701+00:00 [scheduled]>] for executor:
LocalExecutor(parallelism=64)
2025-08-11 11:47:31 [info ] Secrets backends loaded for worker [supervisor]
backend_classes=['EnvironmentVariablesBackend'] count=1
[2025-08-11T11:47:31.038+0000] {_client.py:1026} INFO - HTTP Request: PATCH
http://localhost:8080/execution/task-instances/019898f5-02a4-73c2-9e98-baf7be52acee/run
"HTTP/1.1 409 Conflict"
2025-08-11 11:47:31 [warning ] Server error
[airflow.sdk.api.client] detail={'detail': {'reason': 'invalid_state',
'message': 'TI was not in a state where it could be marked as running',
'previous_state': 'scheduled'}}
2025-08-11 11:47:31 [info ] Process exited [supervisor]
exit_code=<Negsignal.SIGKILL: -9> pid=1419080 signal_sent=SIGKILL
[2025-08-11T11:47:31.060+0000] {local_executor.py:96} ERROR - uhoh
Traceback (most recent call last):
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/airflow/executors/local_executor.py",
line 92, in _run_worker
_execute_work(log, workload)
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/airflow/executors/local_executor.py",
line 120, in _execute_work
supervise(
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/airflow/sdk/execution_time/supervisor.py",
line 1793, in supervise
process = ActivitySubprocess.start(
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/airflow/sdk/execution_time/supervisor.py",
line 854, in start
proc._on_child_started(ti=what, dag_rel_path=dag_rel_path,
bundle_info=bundle_info)
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/airflow/sdk/execution_time/supervisor.py",
line 865, in _on_child_started
ti_context = self.client.task_instances.start(ti.id, self.pid, start_date)
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/airflow/sdk/api/client.py",
line 151, in start
resp = self.client.patch(f"task-instances/{id}/run",
content=body.model_dump_json())
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/httpx/_client.py",
line 1219, in patch
return self.request(
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/tenacity/__init__.py",
line 338, in wrapped_f
return copy(f, *args, **kw)
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/tenacity/__init__.py",
line 477, in __call__
do = self.iter(retry_state=retry_state)
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/tenacity/__init__.py",
line 378, in iter
result = action(retry_state)
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/tenacity/__init__.py",
line 400, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in
__get_result
raise self._exception
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/tenacity/__init__.py",
line 480, in __call__
result = fn(*args, **kwargs)
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/airflow/sdk/api/client.py",
line 714, in request
return super().request(*args, **kwargs)
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/httpx/_client.py",
line 827, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/httpx/_client.py",
line 914, in send
response = self._send_handling_auth(
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/httpx/_client.py",
line 942, in _send_handling_auth
response = self._send_handling_redirects(
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/httpx/_client.py",
line 999, in _send_handling_redirects
raise exc
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/httpx/_client.py",
line 982, in _send_handling_redirects
hook(response)
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/airflow/sdk/api/client.py",
line 116, in raise_on_4xx_5xx
return get_json_error(response) or response.raise_for_status()
File
"/home/zifengma/Airflow/.venv/lib64/python3.9/site-packages/airflow/sdk/api/client.py",
line 112, in get_json_error
raise err
airflow.sdk.api.client.ServerResponseError: Server returned error
```
GitHub link:
https://github.com/apache/airflow/discussions/54534#discussioncomment-14113662
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
