GitHub user deanleggo created a discussion: Add max_active_tis_per_dag[run] to
Task_Group?
I have a folder with a varying number of files I need to load into my database.
Each file will need to go through an extract, transform, load (ETL) pipeline.
So, I need to load each file one at a time though the ETL pipeline.
I have a dynamic task group that runs each group concurrently. But because the
way we have structured the database, Airflow is triggering the same procedure
multiple times, and the data is being overwritten. So, I need to run each group
sequentially - with 1 file going through the 3 tasks before Airflow starts the
next group.
But I cannot add max_active_tis_per_dag or max_active_tis_per_dagrun to the
`@task_group()` or `.partial()` on the task group.
Would it be possible to add max_active_tis_per_dag or max_active_tis_per_dagrun
to the task_group?
Or is there another way to achieve my goal?
**Example**
```
@task()
def get_files():
....
@task_group()
def process_file(name):
@task()
def extract():
<run SQL command>
@task()
def transform():
<run SQL command>
@task()
def load():
<run SQL command>
extract(name) >> transform(name) >> load(name)
process_file.partial(max_active_tis_per_dag=1).expand(name=get_files) # This
line fails
```
Our Airflow version is currently 2.10.1 in some sort of docker environment.
This [discussions 50297](https://github.com/apache/airflow/discussions/50297)
may be similar.
GitHub link: https://github.com/apache/airflow/discussions/53956
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
