I should probably mention that our implementation will be running multiple executors/schedulers/workers for the sake of resilience and recovery. We are more concerned about task state correctness than performance if that makes sense, i.e. the same task should not run twice by accident etc.
On Sat, Apr 7, 2018 at 4:32 AM, Kristoffer Sjögren <sto...@gmail.com> wrote: > Hi > > We have been running airflow for several years and decided to create a > executor plugin. > > After studying the code quite a bit (both code interacting with the > executor and some of the implementations), I still find it hard to distil > concrete requirements for an BaseExecutor implementation. > > Specifically, there's not much documentation around high availability and > how task states are concluded in the event of failures and other corner > cases, which I believe is at the core of any scheduling mechanism. > > I would be very excited to hear if anyone would be so kind to elaborate a > bit more in depth on the principles used and how the interaction between > the executor and tasks are designed in airflow. > > Cheers, > -Kristoffer >