vandonr-amz opened a new pull request, #32572:
URL: https://github.com/apache/airflow/pull/32572
duplicating #29602 as it seems a bit abandoned 🏚️
my changes:
- merged recent main
- removed try/except that created a special case for tests (pointed by
@uranusjr in a comment on the original PR)
---
### Original description:
This PR changes the default value for configure scheduler.max_tis_per_query
to 16 and update related document.
Before this change scheduler.max_tis_per_query=512 by default.
scheduler.max_tis_per_query should not be greater than core.parallelism
according to related code
[[1]](https://github.com/apache/airflow/blob/49867b660b6231c1319969217bc61917f7cf9829/airflow/jobs/scheduler_job.py#L581)
and
[[2]](https://github.com/apache/airflow/blob/49867b660b6231c1319969217bc61917f7cf9829/airflow/executors/base_executor.py#L331).
If both configurations use default value, the default 512 of max_tis_per_query
is never used.
After this PR, default will set scheduler.max_tis_per_query=16 and
core.parallelism=32. This will make the executor can be populated in two
iteration of scheduler main loop.
The actual change applied in Airflow runtime by this PR is to reduce the
max_tis from 32 to 16. The query batch size reduced. This can help scheduler
finish the iteration faster. Thus make scheduler more responsive and healthy.
This PR also updated related document about scheduler.max_tis_per_query to
help users understand how to set this configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
