Made some comments. Summarising: I think we need it, and the proposal is reasonable, I only have one serious question where the Backjob should be running, My guts feeling tell me that scheduler is a better "entity" for such a job than workers as proposed in the original document, but I am happy to discuss it.
On Wed, Dec 23, 2020 at 2:45 PM Tomasz Urbaszek <[email protected]> wrote: > Hello all, > > I would like to discuss the new Airflow Improvement Proposal which > aims to give Airflow users the possibility to trigger backfill > externally (via API or UI). > > In short this AIP proposes: > - create new API endpoint to trigger backfill job (and same mechanism > for web ui) > - create new celery task to run backfill on worker machines (in case > of Celery-like executors) > - extend scheduler mechanism of removing zombies to take care of > backfill-triggered tasks and dagruns > - improve UI so users can see difference between scheduled and backfilled > runs > > I drafted a doc with the proposal: https://s.apache.org/backfill-aip > so we can discuss it there before moving it to cwiki. > > Happy to hear your opinion on this. And have a peaceful and warm holidays! > > Best, > Tomek > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
