Any discoveries from this thread should probably be added to https://airflow.apache.org/concepts.html#core-ideas
FWIW, I'm not sure if it's still this way on the master branch, but at one point a couple useful features (e.g. was it accessing task logs from the UI and maybe running from the UI?) were supported only on the CeleryExecutor, which made me consider that as the de-facto BaseExecutor. Clarifying this implementation will make it way easier to extend airflow to other types of executors which could be preferable to local/celery in some installations. On Fri, Apr 6, 2018 at 7:32 PM Kristoffer Sjögren <[email protected]> wrote: > Hi > > We have been running airflow for several years and decided to create a > executor plugin. > > After studying the code quite a bit (both code interacting with the > executor and some of the implementations), I still find it hard to distil > concrete requirements for an BaseExecutor implementation. > > Specifically, there's not much documentation around high availability and > how task states are concluded in the event of failures and other corner > cases, which I believe is at the core of any scheduling mechanism. > > I would be very excited to hear if anyone would be so kind to elaborate a > bit more in depth on the principles used and how the interaction between > the executor and tasks are designed in airflow. > > Cheers, > -Kristoffer >
