Yup that's true @Tzu-ping Chung <t...@astronomer.io> . There will need to be something in the database. I think a natural choice for the behavior would be like pausing a dag -- anything already scheduled would continue to run but nothing new would be scheduled.
On Tue, Jul 9, 2024 at 7:08 PM Tzu-ping Chung <t...@astronomer.io.invalid> wrote: > How does the user cancel or pause the entire backfill process? The > proposal only says this should be possible, but does not touch on how > exactly. > > My intuition while reading the document was to have a flag on BackfillRun, > but that does not seem to be the case in your illustrative code. > > TP > > > > On Jul 9, 2024, at 22:12, Daniel Standish > <daniel.stand...@astronomer.io.INVALID> wrote: > > > > I put up a draft AIP for scheduler-managed backfill here: > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-78+Scheduler-managed+backfill > > > > Quick summary: > > > > TLDR: move backfill from CLI process to the scheduler > > > > Backfill currently is a CLI-only feature that in effect runs a scheduler > > locally in the CLI process. We don't have good visibility of backfill > jobs > > in the web UI, and users without CLI access cannot access the feature. > > Additionally, it's not ideal to have a "second scheduler" from a project > > maintenance perspective. > > > > This AIP focuses specifically on moving management of backfill jobs to > the > > scheduler. This will take something away from users. Previously you > could > > run backfill in local mode which would not only schedule the backfill > > locally but run all the tasks locally as well. This will go away. And > the > > scheduler will of course have more to do, to the extent that backfill is > > used. The scheduler will become somewhat more complex since it will have > > to manage backfill runs too. > > > > There are some interactions with other AIPs. E.g. backfill is > > fundamentally about data completeness. And the data awareness AIPs may > > change what that can mean in Airflow. > > > > I look forward to your feedback. > > > > Thanks > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
