You can simply use SqlSensor poking at the Airflow metadata database of your other cluster. https://airflow.apache.org/code.html?highlight=sqlsensor#airflow.operators.SqlSensor
Max On Fri, Nov 18, 2016 at 7:26 AM, Michael Gong <[email protected]> wrote: > Hey Rob, > > > Thanks for the valuable information. > > > Currently, our airflow configuration does use Celery executor and Redis > backend. > > But the worker processes are all created locally. > > > I assume you mean that Celery support distributed worker process to > different servers/nodes. > > If that's the case, do you know any guideline of how to setup the Celery > with distributed worker ? > > > Any advise is welcomed. > > > thanks. > > Michael > > > > > > ________________________________ > From: Rob Froetscher <[email protected]> > Sent: Thursday, November 17, 2016 9:47 PM > To: [email protected] > Subject: Re: external task dependency across 2 different servers > > Hey Michael, > > If you have separate installations with separate operational databases, > then the ExternalTaskSensor will not do what you need. However, I'm not > sure I understand why you would need separate Airflow installations that > rely upon each other. Using Celery, you should be able to scale a single > installation of Airflow as much as you need. > > Regarding setting up celery worker nodes. There is some information on that > in the Airflow Documentation > <https://airflow.incubator.apache.org/configuration.html# > scaling-out-with-celery> > . > > On Thu, Nov 17, 2016 at 11:48 AM, Michael Gong <[email protected]> wrote: > > > Rob, > > > > thanks for reply. > > > > > > If I have multiple installations of airflow on different servers, will > > this dependency be done ? > > > > > > For the second setup, ie. 1 installation of airflow and it is > distributed, > > can you advise how to set it up ? I am not familiar with that. > > > > > > Thanks. > > > > > > > > ________________________________ > > From: Rob Froetscher <[email protected]> > > Sent: Thursday, November 17, 2016 6:52 PM > > To: [email protected] > > Subject: Re: external task dependency across 2 different servers > > > > Are you saying you have multiple installations of Airflow, or simply that > > your airflow setup is distributed, with celery workers on their own > nodes? > > > > If the latter, ExternalTaskSensor should still do the trick for you, as I > > believe it just checks airflow's operational database for the status of > > this external task. > > > > On Thu, Nov 17, 2016 at 10:40 AM, Michael Gong <[email protected]> > wrote: > > > > > Hi, > > > > > > > > > Is it possible that 1 task can depend on another task running on > another > > > DAG on another sever? > > > > > > > > > For example, I have task1 in dag1, and task2 in dag2. > > > > > > > > > If both dag1 and dag2 running on the same airflow instance on the same > > > server, airflow provides the ExternalTaskSensor operator. > > > > > > > > > But if dag2 runs on another airflow instance on another sever, can > > airflow > > > allow dag1's task1 depend on dag2's task2? > > > > > > > > > > > > Any suggestions are welcomed. > > > > > > > > > thanks > > > > > > > > > > > >
