Hey Tomasz, We discussed on a ticket in November about rolling in other backfill improvements: https://github.com/apache/airflow/issues/12654. Unfortunately I've been unable to work on that since then, but I have the time to do it now. Should I add this to your AIP document, or do you think it would be better to have a separate document (or separate AIP)?
Thanks On Mon, 4 Jan 2021 at 14:45, Ash Berlin-Taylor <[email protected]> wrote: > I agree with Jarek -- running a backfill is essentially another scheduler > (albeit it one with a lot of specialised logic -- how much of it is needed > is unclear) so it would be nice to have that all rolled up in to the > scheduler, and `airflow backfill` and the new API just sets "something" > that the scheduler than looks at to start running the backfill. > > (The scheduling of backfill should not be very resource intensive, no more > so than normal dag runs) > > -ash > > > > On Mon, 28 Dec, 2020 at 13:23, Jarek Potiuk <[email protected]> > wrote: > > Made some comments. Summarising: I think we need it, and the proposal is > reasonable, I only have one serious question where the Backjob should be > running, My guts feeling tell me that scheduler is a better "entity" for > such a job than workers as proposed in the original document, but I am > happy to discuss it. > > On Wed, Dec 23, 2020 at 2:45 PM Tomasz Urbaszek <[email protected]> > wrote: > >> Hello all, >> >> I would like to discuss the new Airflow Improvement Proposal which >> aims to give Airflow users the possibility to trigger backfill >> externally (via API or UI). >> >> In short this AIP proposes: >> - create new API endpoint to trigger backfill job (and same mechanism >> for web ui) >> - create new celery task to run backfill on worker machines (in case >> of Celery-like executors) >> - extend scheduler mechanism of removing zombies to take care of >> backfill-triggered tasks and dagruns >> - improve UI so users can see difference between scheduled and backfilled >> runs >> >> I drafted a doc with the proposal: https://s.apache.org/backfill-aip >> so we can discuss it there before moving it to cwiki. >> >> Happy to hear your opinion on this. And have a peaceful and warm holidays! >> >> Best, >> Tomek >> > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jacob Ward | Graduate Data Infrastructure Engineer [email protected] NEW YORK | BOSTON | BRIGHTON | LONDON | BERLIN | STUTTGART | PARIS | SINGAPORE | SYDNEY
