Seems valid for default behaviour, but if I backfill for a year and realize there was something wrong with the code, I don't want to manually fail each dag run that is running. How about a force kill option?
On Wed, Jul 10, 2024 at 9:28 AM Daniel Standish <daniel.stand...@astronomer.io.invalid> wrote: > Yup that's true @Tzu-ping Chung <t...@astronomer.io> . There will need to > be > something in the database. I think a natural choice for the behavior > would be like pausing a dag -- anything already scheduled would continue to > run but nothing new would be scheduled. > > On Tue, Jul 9, 2024 at 7:08 PM Tzu-ping Chung <t...@astronomer.io.invalid> > wrote: > > > How does the user cancel or pause the entire backfill process? The > > proposal only says this should be possible, but does not touch on how > > exactly. > > > > My intuition while reading the document was to have a flag on > BackfillRun, > > but that does not seem to be the case in your illustrative code. > > > > TP > > > > > > > On Jul 9, 2024, at 22:12, Daniel Standish > > <daniel.stand...@astronomer.io.INVALID> wrote: > > > > > > I put up a draft AIP for scheduler-managed backfill here: > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-78+Scheduler-managed+backfill > > > > > > Quick summary: > > > > > > TLDR: move backfill from CLI process to the scheduler > > > > > > Backfill currently is a CLI-only feature that in effect runs a > scheduler > > > locally in the CLI process. We don't have good visibility of backfill > > jobs > > > in the web UI, and users without CLI access cannot access the feature. > > > Additionally, it's not ideal to have a "second scheduler" from a > project > > > maintenance perspective. > > > > > > This AIP focuses specifically on moving management of backfill jobs to > > the > > > scheduler. This will take something away from users. Previously you > > could > > > run backfill in local mode which would not only schedule the backfill > > > locally but run all the tasks locally as well. This will go away. And > > the > > > scheduler will of course have more to do, to the extent that backfill > is > > > used. The scheduler will become somewhat more complex since it will > have > > > to manage backfill runs too. > > > > > > There are some interactions with other AIPs. E.g. backfill is > > > fundamentally about data completeness. And the data awareness AIPs may > > > change what that can mean in Airflow. > > > > > > I look forward to your feedback. > > > > > > Thanks > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >
