dzhigimont opened a new pull request, #30372:
URL: https://github.com/apache/airflow/pull/30372
The improvement contains two cases:
- Update status for ready for scheduling tasks in batches to get rid of the
stack depth limit exceeded error when mapped task instances > 4k
- speed up the scheduling of mapped tasks:
Calling _revise_map_indexes_if_mapped for each task instance was
significantly slowing down the scheduling, but it's enough to call the method
only once per task id. The improvement reduced the time for scheduling 10k
mapped tasks in my case from 7 - 10 minutes to seconds.
Closes: https://github.com/apache/airflow/issues/29959.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
