sunnylevelai commented on issue #53905:
URL: https://github.com/apache/airflow/issues/53905#issuecomment-3318770392
Hey folks š,
Iām running into the same issue with rendered_task_instance_fields duplicate
key violations.
Full error:
API_SERVER_ERROR: {'status_code': 409, 'message': 'Server returned error',
'detail': {'reason': 'Unique constraint violation',
'statement': 'INSERT INTO rendered_task_instance_fields
(dag_id, task_id, run_id, map_index, rendered_fields, k8s_pod_yaml)
VALUES (%(dag_id)s, %(task_id)s, %(run_id)s, %(map_index)s,
%(rendered_fields)s, %(k8s_pod_yaml)s)',
'orig_error': 'duplicate key value violates unique constraint
"rendered_task_instance_fields_pkey"',
DETAIL: Key (dag_id, task_id, run_id, map_index)=(silver_production,
build_dbt_command, scheduled__2025-09-14T05:50:00+00:00, 36) already
exists.',
'message': 'Serious error when handling your request.
Check logs for more details - you will find it in api server when you look
for ID SG9WN9zJ'}}
It seems like when multiple tasks are rendering fields in parallel, the
insert conflicts and raises the error above.
A couple of questions for clarification:
Is this a known concurrency issue in Airflow 3.0.4 (with CeleryExecutor)?
Would enabling [core] max_active_runs_per_dag or adjusting scheduler
settings help reduce this?
Is there any recommended workaround (retry logic, DB constraint handling, or
disabling rendered task instance persistence) until a fix lands?
Any guidance or confirmation would be super helpful š
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
