vsilviaMO opened a new issue, #50276:
URL: https://github.com/apache/airflow/issues/50276
### Apache Airflow version
3.0.0
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I have a task defined as:
`@task(task_id="transform_load_fusion", max_active_tis_per_dag=4,
map_index_template="{{ custom_name }}", retries=3,
retry_delay=timedelta(seconds=120),
on_retry_callback=sl.task_retry_alert,
on_failure_callback=sl.task_failure_alert)`
It should be limiting the number of concurrent tasks running (so as to
minimize the RAM footprint and avoid OOMs). In Airflow 2.10.0, I use
`max_active_tis_per_dagrun` and it works fine, limiting the task to 4 active
tasks executing concurrently. However, in 3.0.0, it seems like the value is
being ignored. Below is a screenshot showing 9 task instances with the same
start time. The image shows two in a "running" state and 7 in "success", but at
the outset, all 9 were in a "running" state, which would seemingly violate the
`max_active_tis_per_dag` setting. It also similarly failed when using the
`max_active_tis_per_dagrun` parameter I was using in 2.10.0.
The values in airflow.cfg relating to concurrency appear to match, and they
only address total tasks, not task instances: `max_active_tasks_per_dag = 16`
### What you think should happen instead?
As it does in 2.10.0, I would expect the DAG execution to limit the number
of concurrent task instances as defined by either the `max_active_tis_per_dag`
or `max_active_tis_per_dagrun` task-level parameter, keeping the others in
scheduled state until slots open up below the threshold set at the task level.
### How to reproduce
Deploy via sample docker-compose.yaml file with default values for
concurrency. Create a DAG with a task that can run concurrently (say, "process"
a file for 60 seconds), set the `max_active_tis_per_dag/dagrun` parameter at
the task level, and execute the DAG against more files than the parameter is
set to. In my case, I ran against 14 files with a max of 4, and all 14 task
instances were running concurrently.
### Operating System
Debian GNU/Linux 12 (bookworm)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
