Hi Tamara,
do you happen to use the celery executor for deploying your DAG factory?
We've got a setup where webserver, scheduler, and workers all run in one
Docker container each.
Our DAG factory is akin to this:
# factory.py
from airflow import DAG
def callable_():
dag_id = ...
parameters = (fetch parameters from a separate metadata database)
dag = DAG(dag_id=dag_id)
operators = (dag=dag, parameters=parameters...)
globals[dag_id] = dag
The above callable_ is the Python callable of the PythonOperator of one
classically defined DAG in the `dags_folder` defined in our airflow.cfg.
We do have the required `from airport import DAG` statement in our DAG
factory, however the above `dag` object is not executed.
Thanks,
Georg
---
Dr. Georg Walther
Co-Founder of Markovian
Skype: [email protected]
WWW: https://markovian.com/
On Thu, Nov 17, 2016 at 11:45 AM, Tamara Mendt <[email protected]> wrote:
> Thanks very much for the detailed reply Maxime,
>
> Like you suggested, I had solved the issue by adding an unused import of an
> airflow class. It took me a while to get to this solution though. I had to
> look into the airflow code to the place you pointed to realize that this
> import was necessary to add the dag to the DagBag. Perhaps its worth it to
> add something about this in the Airflow documentation? Or perhaps adding an
> INFO log stating this sanity check, so that on running the airflow
> list_dags
> command, it is obvious that DAG files must contain airflow imports.
>
> Cheers,
>
> Tamara
>
> --
>
> [image: logo]
> <http://www.facebook.com/hellofreshde> <http://twitter.com/
> HelloFreshde>
> <http://instagram.com/hellofreshde/> <http://blog.hellofresh.de/>
> <https://app.adjust.com/ayje08?campaign=Hellofresh&;
> deep_link=hellofresh%3A%2F%2F&post_deep_link=https%3A%2F%
> 2Fwww.hellofresh.com%2Fapp%2F%3Futm_medium%3Demail%26utm_
> source%3Demail_signature&fallback=https%3A%2F%2Fwww.
> hellofresh.com%2Fapp%2F%3Futm_medium%3Demail%26utm_source%
> 3Demail_signature>
>
> *HelloFresh App –Download Now!*
> <https://app.adjust.com/ayje08?campaign=Hellofresh&;
> deep_link=hellofresh%3A%2F%2F&post_deep_link=https%3A%2F%
> 2Fwww.hellofresh.com%2Fapp%2F%3Futm_medium%3Demail%26utm_
> source%3Demail_signature&fallback=https%3A%2F%2Fwww.
> hellofresh.com%2Fapp%2F%3Futm_medium%3Demail%26utm_source%
> 3Demail_signature>
> *We're active in:*
> US <https://www.hellofresh.com/?utm_medium=email&utm_source=
> email_signature>
> | DE
> <https://www.hellofresh.de/?utm_medium=email&utm_source=email_signature> |
> UK
> <https://www.hellofresh.co.uk/?utm_medium=email&utm_source=email_signature
> >
> | NL
> <https://www.hellofresh.nl/?utm_medium=email&utm_source=email_signature> |
> AU
> <https://www.hellofresh.com.au/?utm_medium=email&utm_
> source=email_signature>
> | BE
> <https://www.hellofresh.be/?utm_medium=email&utm_source=email_signature> |
> AT <https://www.hellofresh.at/?utm_medium=email&utm_source=email_signature
> >
> | CH
> <https://www.hellofresh.ch/?utm_medium=email&utm_source=email_signature> |
> CA <https://www.hellofresh.ca/?utm_medium=email&utm_source=email_signature
> >
>
> www.HelloFreshGroup.com
> <http://www.hellofreshgroup.com/?utm_medium=email&utm_
> source=email_signature>
>
> We are hiring around the world – Click here to join us
> <https://www.hellofresh.com/jobs/?utm_medium=email&utm_
> source=email_signature>
>
> --
>
> <https://www.hellofresh.com/jobs/?utm_medium=email&utm_
> source=email_signature>
> HelloFresh AG, Berlin (Sitz der Gesellschaft) | Vorstände: Dominik S.
> Richter (Vorsitzender), Thomas W. Griesel, Christian Gärtner | Vorsitzender
> des Aufsichtsrats: Jeffrey Lieberman | Eingetragen beim Amtsgericht
> Charlottenburg, HRB 171666 B | USt-Id Nr.: DE 302210417
>
> *CONFIDENTIALITY NOTICE:* This message (including any attachments) is
> confidential and may be privileged. It may be read, copied and used only by
> the intended recipient. If you have received it in error please contact the
> sender (by return e-mail) immediately and delete this message. Any
> unauthorized use or dissemination of this message in whole or in parts is
> strictly prohibited.
>
