prayaagmahajan commented on issue #53905:
URL: https://github.com/apache/airflow/issues/53905#issuecomment-4108934200
Hello, I am also facing the same issue.
I am using **airflow 3.1.8** with 3 schedulers and 3 celery nodes. Below is
my api server error log. and my celery log.
PS: This is an random failure happens only some times.
`{"timestamp":"2026-03-23T04:10:25.385669Z","level":"error","event":"Task
failed with
exception","logger":"task","filename":"task_runner.py","lineno":1164,"error_detail":[{"exc_type":"AirflowRuntimeError","exc_value":"API_SERVER_ERROR:
{'status_code': 409, 'message': 'Server returned error', 'detail': {'detail':
{'reason': 'Unique constraint violation', 'statement': 'INSERT INTO
rendered_task_instance_fields (dag_id, task_id, run_id, map_index,
rendered_fields, k8s_pod_yaml) VALUES (%s, %s, %s, %s, %s, %s)', 'orig_error':
'(1062, \"Duplicate entry \\'{{uuid}}_RUN_ONCE-DOS_END_NODE_19149\\' for key
\\'rendered_task_instance_fields.PRIMARY\\'\")', 'message': 'Serious error when
handling your request. Check logs for more details - you will find it in api
server when you look for ID
**E6XWUa37**'}}}","exc_notes":[],"syntax_error":null,"is_cause":false,"frames":[{"filename":"/usr/local/lib/python3.11/site-packages/airflow/sdk/execution_time/task_runner.py","lineno":1106,"name":"run"},{
"filename":"/usr/local/lib/python3.11/site-packages/airflow/sdk/execution_time/task_runner.py","lineno":995,"name":"_prepare"},{"filename":"/usr/local/lib/python3.11/site-packages/airflow/sdk/execution_time/comms.py","lineno":206,"name":"send"},{"filename":"/usr/local/lib/python3.11/site-packages/airflow/sdk/execution_time/comms.py","lineno":270,"name":"_get_response"},{"filename":"/usr/local/lib/python3.11/site-packages/airflow/sdk/execution_time/comms.py","lineno":257,"name":"_from_frame"}],"is_group":false,"exceptions":[]}]}`
^^^^
`82913811:2026-03-23_04:10:25.38245 [2026-03-23T04:10:25.382312Z]
{{exceptions.py:78}} ERROR - Error with id **E6XWUa37**
82913919-2026-03-23_04:10:25.38248 File
"/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line
42, in wrapped_app
82914051-2026-03-23_04:10:25.38249 await app(scope, receive, sender)
82914115-2026-03-23_04:10:25.38249 File
"/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
82914228-2026-03-23_04:10:25.38250 response = await f(request)
82914286-2026-03-23_04:10:25.38251 ^^^^^^^^^^^^^^^^
82914344-2026-03-23_04:10:25.38251 File
"/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 308, in app
82914456-2026-03-23_04:10:25.38252 raw_response = await
run_endpoint_function(
82914530-2026-03-23_04:10:25.38253
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
82914604-2026-03-23_04:10:25.38253 File
"/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 219, in
run_endpoint_function
82914734-2026-03-23_04:10:25.38254 return await dependant.call(**values)
82914802-2026-03-23_04:10:25.38255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
82914870-2026-03-23_04:10:25.38255 File
"/usr/local/lib/python3.11/site-packages/cadwyn/structure/versions.py", line
474, in decorator
82914998-2026-03-23_04:10:25.38256 response = await
self._convert_endpoint_response_to_version(
82915089-2026-03-23_04:10:25.38257
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
82915180-2026-03-23_04:10:25.38257 File
"/usr/local/lib/python3.11/site-packages/cadwyn/structure/versions.py", line
520, in _convert_endpoint_response_to_version
82915336-2026-03-23_04:10:25.38261 response_or_response_body:
Union[FastapiResponse, object] = await run_in_threadpool(
82915451-2026-03-23_04:10:25.38262
^^^^^^^^^^^^^^^^^^^^^^^^
82915566-2026-03-23_04:10:25.38262 File
"/usr/local/lib/python3.11/site-packages/starlette/concurrency.py", line 38, in
run_in_threadpool
82915697-2026-03-23_04:10:25.38263 return await
anyio.to_thread.run_sync(func)
82915771-2026-03-23_04:10:25.38264
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
82915845-2026-03-23_04:10:25.38264 File
"/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 63, in
run_sync
82915961-2026-03-23_04:10:25.38270 return await
get_async_backend().run_sync_in_worker_thread(
82916051-2026-03-23_04:10:25.38271
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
82916141-2026-03-23_04:10:25.38272 File
"/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line
2502, in run_sync_in_worker_thread
82916285-2026-03-23_04:10:25.38273 return await future
82916335-2026-03-23_04:10:25.38273 ^^^^^^^^^^^^
82916385-2026-03-23_04:10:25.38274 File
"/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line
986, in run
82916506-2026-03-23_04:10:25.38275 result = context.run(func, *args)
82916570-2026-03-23_04:10:25.38277 ^^^^^^^^^^^^^^^^^^^^^^^^
82916634-2026-03-23_04:10:25.38278 File
"/usr/local/lib/python3.11/site-packages/cadwyn/schema_generation.py", line
515, in __call__
82916760-2026-03-23_04:10:25.38280 return self._original_callable(*args,
**kwargs)
82916838-2026-03-23_04:10:25.38280
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
82916916-2026-03-23_04:10:25.38281 File
"/usr/local/lib/python3.11/site-packages/airflow/api_fastapi/execution_api/routes/task_instances.py",
line 713, in ti_put_rtif
82917076-2026-03-23_04:10:25.38282
task_instance.update_rtif(put_rtif_payload, session)
82917159-2026-03-23_04:10:25.38282 File
"/usr/local/lib/python3.11/site-packages/airflow/utils/session.py", line 98, in
wrapper
82917280-2026-03-23_04:10:25.38283 return func(*args, **kwargs)
82917339-2026-03-23_04:10:25.38284 ^^^^^^^^^^^^^^^^^^^^^
82917398-2026-03-23_04:10:25.38284 File
"/usr/local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line
1447, in update_rtif
82917531-2026-03-23_04:10:25.38285 session.flush()
82917577-2026-03-23_04:10:25.38286 File
"/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/session.py", line
4331, in flush
82917701-2026-03-23_04:10:25.38286 self._flush(objects)
82917752-2026-03-23_04:10:25.38287 File
"/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/session.py", line
4466, in _flush
82917877-2026-03-23_04:10:25.38288 with util.safe_reraise():
82917933-2026-03-23_04:10:25.38288 File
"/usr/local/lib64/python3.11/site-packages/sqlalchemy/util/langhelpers.py",
line 121, in __exit__
82918064-2026-03-23_04:10:25.38290 raise exc_value.with_traceback(exc_tb)
82918133-2026-03-23_04:10:25.38291 File
"/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/session.py", line
4427, in _flush
82918258-2026-03-23_04:10:25.38292 flush_context.execute()
82918312-2026-03-23_04:10:25.38292 File
"/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line
466, in execute
82918440-2026-03-23_04:10:25.38293 rec.execute(self)
82918488-2026-03-23_04:10:25.38294 File
"/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line
642, in execute
82918616-2026-03-23_04:10:25.38294
util.preloaded.orm_persistence.save_obj(
82918687-2026-03-23_04:10:25.38295 File
"/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/persistence.py", line
93, in save_obj
82918816-2026-03-23_04:10:25.38296 _emit_insert_statements(
82918871-2026-03-23_04:10:25.38296 File
"/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/persistence.py", line
1048, in _emit_insert_statements
82919017-2026-03-23_04:10:25.38297 result = connection.execute(
82919076-2026-03-23_04:10:25.38298 ^^^^^^^^^^^^^^^^^^^
82919135-2026-03-23_04:10:25.38298 File
"/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line
1419, in execute
82919261-2026-03-23_04:10:25.38299 return meth(
82919304-2026-03-23_04:10:25.38300 ^^^^^
82919347-2026-03-23_04:10:25.38300 File
"/usr/local/lib64/python3.11/site-packages/sqlalchemy/sql/elements.py", line
527, in _execute_on_connection
82919488-2026-03-23_04:10:25.38302 return
connection._execute_clauseelement(
82919560-2026-03-23_04:10:25.38304
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
82919632-2026-03-23_04:10:25.38305 File
"/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line
1641, in _execute_clauseelement
82919773-2026-03-23_04:10:25.38305 ret = self._execute_context(
82919832-2026-03-23_04:10:25.38306 ^^^^^^^^^^^^^^^^^^^^^^
82919891-2026-03-23_04:10:25.38307 File
"/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line
1846, in _execute_context
82920026-2026-03-23_04:10:25.38307 return self._exec_single_context(
82920090-2026-03-23_04:10:25.38308 ^^^^^^^^^^^^^^^^^^^^^^^^^^
82920154-2026-03-23_04:10:25.38309 File
"/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line
1986, in _exec_single_context
82920293-2026-03-23_04:10:25.38309 self._handle_dbapi_exception(
82920353-2026-03-23_04:10:25.38310 File
"/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line
2363, in _handle_dbapi_exception
82920495-2026-03-23_04:10:25.38311 raise
sqlalchemy_exception.with_traceback(exc_info[2]) from e
82920587-2026-03-23_04:10:25.38312 File
"/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line
1967, in _exec_single_context
82920726-2026-03-23_04:10:25.38312 self.dialect.do_execute(
82920781-2026-03-23_04:10:25.38313 File
"/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/default.py", line
952, in do_execute
82920912-2026-03-23_04:10:25.38315 cursor.execute(statement, parameters)
82920980-2026-03-23_04:10:25.38316 File
"/usr/local/lib64/python3.11/site-packages/MySQLdb/cursors.py", line 179, in
execute
82921098-2026-03-23_04:10:25.38317 res = self._query(mogrified_query)
82921163-2026-03-23_04:10:25.38317 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
82921228-2026-03-23_04:10:25.38321 File
"/usr/local/lib64/python3.11/site-packages/MySQLdb/cursors.py", line 330, in
_query
82921345-2026-03-23_04:10:25.38322 db.query(q)
82921387-2026-03-23_04:10:25.38323 File
"/usr/local/lib64/python3.11/site-packages/MySQLdb/connections.py", line 286,
in query
82921507-2026-03-23_04:10:25.38324 _mysql.connection.query(self, query)
82921574-2026-03-23_04:10:25.38324
82921601-2026-03-23_04:10:26.56985 Terminating Task`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
