Here's the issue I was thinking of https://issues.apache.org/jira/browse/AIRFLOW-5177
On May 7 2020, at 10:43 am, Ash Berlin-Taylor <[email protected]> wrote: > Right now the location of the dags folder needs to be identical on both the > scheduler and all the workers. Sorry. > > This is a bug, and it shouldn't need to be the case. (Apache Jira is having a > hiccough right now so I can't find the issue at the moment.) > -ash > On May 7 2020, at 8:53 am, devang pandey <[email protected]> wrote: > > Hi All, > > > > I am pretty new to Airflow and we are planning to use it at organisation > > level for automations (mainly data-extraction pipelines). > > > > I have setup an airflow cluster with one server and 2 VDI machines. Using > > Airflow 1.10.9 , postgres , celery 4.4.0 and redis as message broker. > > > > Thing is I am successfully able to run example dags on my cluster but when > > trying to run my own dag - its gets scheduled and picked up by worker > > machine but then fails with an error message "DAG ID could not found". > > > > On taking a closer look I observed my worker machine while picking up the > > task is passing a - sd parameter which is pointing to dag folder of my > > scheduler machine / server . > > > > Things I can confirm: > > > > 1- DAG is present on all machines , master and worker machines inside > > AIRFLOW_HOME/dags > > 2- Airflow worker config file points to local correct dag location. > > > > I tried multiple things but could not understand the issue. Please if you > > provide suggestions. > > > > > > Thank you, > > Devang > > >
