Hi all,
I have a probably rather special use case scenario:
Using Airflow 1.10.10, I would like to physically decouple the Webserver and
the Scheduler for some secure access reasons. According to
https://airflow.apache.org/docs/stable/dag-serialization.html, this should be a
piece of cake with Airflow 1.10.10, since DAGs are stored in the Metadata
database and the webserver does not need to access the DAG files anymore. The
metadata database is of course reachable by both Airflow instances:
Docker Image Instance: Airflow Webserver <----> Docker Image Instance: Metadata
database <----> Physical Machine: Airflow Scheduler
However, every time I start a DAG manually, it crashes with a rather lengthy
error message:
File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line
1255, in trigger
external_trigger=True
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74,
in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/models/dag.py", line
1818, in create_dagrun
return self.get_dag().create_dagrun(run_id=run_id,
AttributeError: 'NoneType' object has no attribute 'create_dagrun'
This basically boils down to self.get_dag() not having set the Boolean flag
store_serialized_flags to True (or whatever value the config is set to), but
always using False (the default value).
This then leads to Airflow attempting to read the DAG file, ignoring the DAG
database entry and returning None, which obviously has no attribute
create_dagrun.
I’ve got several questions here now:
1. Is my scenario even possible or am I overlooking something rather obvious?
2. Is the crashing DAG behavior intended like that? It rather seems like a
bug to me.
3. Is it worth fixing this issue (if it is one) for Airflow 1.10.x,
considering that Airflow 2.0.0 does not even contain the corresponding classes
anymore and takes a different path?
Mit freundlichen Grüßen / Best regards
Dr. Thomas Niebler
Data Scientist
Sales Data Lab, Analytics DC-IH/SDL1
Tel. +49 9352 18-2392
Fax +49 9352 18-0
thomas.nieb...@boschrexroth.de<mailto:thomas.nieb...@boschrexroth.de>
www.boschrexroth.com
Bosch Rexroth AG
Partensteiner Straße 23
97816 Lohr am Main
GERMANY
[BOSCH REXROTH]<http://www.boschrexroth.com/>
Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192
Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner Lang,
Reinhard Schäfer, Dr. Marc Wucherer
Vorsitzender des Aufsichtsrats: Christoph Kübel
