I think it makes sense to orchestrate backfills in a more managed way than through a CLI command, especially that execution of tasks would happen through regular executors configured in the Airflow deployment. Once concern I have, also called out in the AIP, is the increased load on the scheduler. I think having some reliability/separation measures so that execution of backfill does not impact/starve regular, day-to-day scheduler's job, will be critical.
On Wed, Jul 10, 2024 at 9:06 PM Daniel Standish <daniel.stand...@astronomer.io.invalid> wrote: > > > > Seems valid for default behaviour, but if I backfill for a year and > realize > > there was something wrong with the code, I don't want to manually fail > each > > dag run that is running. How about a force kill option? > > > Yes I would not expect users to need to have to go in and manually fail > each dag run if they wanted to cancel a backfill job. TP was just talking > about pausing. Here's how I phrased it in the doc: > > > - We should be able to view backfill jobs in the webserver and observe > progress and status, and cancel or pause them > > > There will be some details that will need to be sorted through but that's > the high level goal. >
