Looking ahead forward to a not too distant future: I’m currently finishing up the foundations for a Rest API. It would be trivial to add a call that returns the state of a task in this API. Then having a federated or even fully decoupled HA becomes a possibility. Obviously it is experimental at the moment cause it will need to settle a bit.
https://github.com/apache/incubator-airflow/pull/1783 Bolke > Op 17 nov. 2016, om 22:47 heeft Rob Froetscher <[email protected]> > het volgende geschreven: > > Hey Michael, > > If you have separate installations with separate operational databases, > then the ExternalTaskSensor will not do what you need. However, I'm not > sure I understand why you would need separate Airflow installations that > rely upon each other. Using Celery, you should be able to scale a single > installation of Airflow as much as you need. > > Regarding setting up celery worker nodes. There is some information on that > in the Airflow Documentation > <https://airflow.incubator.apache.org/configuration.html#scaling-out-with-celery> > . > > On Thu, Nov 17, 2016 at 11:48 AM, Michael Gong <[email protected]> wrote: > >> Rob, >> >> thanks for reply. >> >> >> If I have multiple installations of airflow on different servers, will >> this dependency be done ? >> >> >> For the second setup, ie. 1 installation of airflow and it is distributed, >> can you advise how to set it up ? I am not familiar with that. >> >> >> Thanks. >> >> >> >> ________________________________ >> From: Rob Froetscher <[email protected]> >> Sent: Thursday, November 17, 2016 6:52 PM >> To: [email protected] >> Subject: Re: external task dependency across 2 different servers >> >> Are you saying you have multiple installations of Airflow, or simply that >> your airflow setup is distributed, with celery workers on their own nodes? >> >> If the latter, ExternalTaskSensor should still do the trick for you, as I >> believe it just checks airflow's operational database for the status of >> this external task. >> >> On Thu, Nov 17, 2016 at 10:40 AM, Michael Gong <[email protected]> wrote: >> >>> Hi, >>> >>> >>> Is it possible that 1 task can depend on another task running on another >>> DAG on another sever? >>> >>> >>> For example, I have task1 in dag1, and task2 in dag2. >>> >>> >>> If both dag1 and dag2 running on the same airflow instance on the same >>> server, airflow provides the ExternalTaskSensor operator. >>> >>> >>> But if dag2 runs on another airflow instance on another sever, can >> airflow >>> allow dag1's task1 depend on dag2's task2? >>> >>> >>> >>> Any suggestions are welcomed. >>> >>> >>> thanks >>> >>> >>> >>
