We are thinking about the same thing. We are going to write a custom plugins similar to the ExternalTaskSensor and check for the task success on the different backend database.
Please let me know if you have better idea. Thanks, Catherine On 11/17/16, 11:48 AM, "Michael Gong" <[email protected]> wrote: Rob, thanks for reply. If I have multiple installations of airflow on different servers, will this dependency be done ? For the second setup, ie. 1 installation of airflow and it is distributed, can you advise how to set it up ? I am not familiar with that. Thanks. ________________________________ From: Rob Froetscher <[email protected]> Sent: Thursday, November 17, 2016 6:52 PM To: [email protected] Subject: Re: external task dependency across 2 different servers Are you saying you have multiple installations of Airflow, or simply that your airflow setup is distributed, with celery workers on their own nodes? If the latter, ExternalTaskSensor should still do the trick for you, as I believe it just checks airflow's operational database for the status of this external task. On Thu, Nov 17, 2016 at 10:40 AM, Michael Gong <[email protected]> wrote: > Hi, > > > Is it possible that 1 task can depend on another task running on another > DAG on another sever? > > > For example, I have task1 in dag1, and task2 in dag2. > > > If both dag1 and dag2 running on the same airflow instance on the same > server, airflow provides the ExternalTaskSensor operator. > > > But if dag2 runs on another airflow instance on another sever, can airflow > allow dag1's task1 depend on dag2's task2? > > > > Any suggestions are welcomed. > > > thanks > > >
