uranusjr opened a new pull request, #75943: URL: https://github.com/apache/airflow/pull/75943
Fix issue mentioned in https://github.com/apache/airflow/issues/75865#issuecomment-6483614606 Each `asset_dag_run_queue` row now references a single asset event, so a Dag can have any number of queued rows. The scheduler sent them in a literal IN, and the query would be too long for a database to receive if there are too many rows. Those queries now use subqueries bounded by the largest event ID so the query string length is predictable. As a result, a run may consume queued events whose rows were skipped by SKIP LOCKED if their ids are within the bound. Since only one scheduler can hold an even row (the Dag row itself is locked), this should only affect rows being dequeued concurrently elsewhere (e.g. with the API). Even in this scenario, the end result should still be correct, and the scheduler's existing retry mechanism should recover potential deadlock issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
