Hey Michael, If you have separate installations with separate operational databases, then the ExternalTaskSensor will not do what you need. However, I'm not sure I understand why you would need separate Airflow installations that rely upon each other. Using Celery, you should be able to scale a single installation of Airflow as much as you need.
Regarding setting up celery worker nodes. There is some information on that in the Airflow Documentation <https://airflow.incubator.apache.org/configuration.html#scaling-out-with-celery> . On Thu, Nov 17, 2016 at 11:48 AM, Michael Gong <[email protected]> wrote: > Rob, > > thanks for reply. > > > If I have multiple installations of airflow on different servers, will > this dependency be done ? > > > For the second setup, ie. 1 installation of airflow and it is distributed, > can you advise how to set it up ? I am not familiar with that. > > > Thanks. > > > > ________________________________ > From: Rob Froetscher <[email protected]> > Sent: Thursday, November 17, 2016 6:52 PM > To: [email protected] > Subject: Re: external task dependency across 2 different servers > > Are you saying you have multiple installations of Airflow, or simply that > your airflow setup is distributed, with celery workers on their own nodes? > > If the latter, ExternalTaskSensor should still do the trick for you, as I > believe it just checks airflow's operational database for the status of > this external task. > > On Thu, Nov 17, 2016 at 10:40 AM, Michael Gong <[email protected]> wrote: > > > Hi, > > > > > > Is it possible that 1 task can depend on another task running on another > > DAG on another sever? > > > > > > For example, I have task1 in dag1, and task2 in dag2. > > > > > > If both dag1 and dag2 running on the same airflow instance on the same > > server, airflow provides the ExternalTaskSensor operator. > > > > > > But if dag2 runs on another airflow instance on another sever, can > airflow > > allow dag1's task1 depend on dag2's task2? > > > > > > > > Any suggestions are welcomed. > > > > > > thanks > > > > > > >
