anmolxlight opened a new pull request, #71074: URL: https://github.com/apache/airflow/pull/71074
closes #71070 ## Summary When two **different producer assets** map to the same downstream partition key (e.g. via `IdentityMapper`), each asset event could create its own `AssetPartitionDagRun` row. The scheduler then held two Pending Dag runs for the same partition key and never triggered the consumer Dag. ## Root cause `AssetManager._get_or_create_apdr` serialized APDR find-or-create with `_lock_asset_model`, which locks the **producer** `AssetModel` row. But APDR dedup is keyed on `(target_dag_id, partition_key)`. Two events from different producer assets that resolve to the same target key therefore took locks on **two different asset rows**, neither blocked the other, and both observed "no existing APDR" and inserted a duplicate. The reporter's two rows created ~1ms apart with one carrying `partition_date` and the other empty match this exactly: the two events came from different producers, and only one carried a date. ## Fix Lock the **target** `DagModel` row instead (`_lock_target_dag`). All APDR find-or-create calls for a given consumer Dag now serialize on the same resource regardless of which producer asset triggered the event, so the second event finds the APDR created by the first and dedups onto it. SQLite still uses the global writer lock (unchanged semantics), and the existing "work on the latest matching APDR" fallback in the UI route and scheduler cleanup is preserved. ## Test Added `test_queue_partitioned_dags_dedups_across_different_producer_assets`: two `register_asset_change` calls from different producer assets with the same partition key must both lock the same target Dag and yield exactly one `AssetPartitionDagRun`. Verified locally with breeze: `6 passed` (including the new test and all existing `_get_or_create_apdr` partition-date tests), ruff clean, formatting clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
