blcksrx opened a new issue, #25265:
URL: https://github.com/apache/airflow/issues/25265
### Description
catchup and backfill provided good flexibility to run `Dags` in pasts. the
only thing that might concerning on those context is that they might produces a
lot of `DagRun`. this situation could leads to some performance issues such as
remaining slot availability and long time to finish.
Assume that there is a DAG with `SqlOperator` using this sql template that
runs every 15 mins:
```SQL
SELECT * FROM table WHERE created_at BETWEEN {{
prev_data_interval_start_success }} AND {{ ts }}
```
In case of any interruption on the scheduler level such as 1 day, this
`catchup` process would creates 96 `DagRun`.
or also assume running this `Dag` with backfill like this:
```BASH
airflow backfill DAG --start-date=today --end-date=prev_week
```
that would creates 672 `DagRun`!
with simple feature that only runs 1 instance of `DagRun` and fulfils these
parameters by the starts of the gap and end of the gap, this issue will be
solve.
### Use case/motivation
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
