Hi Kaxil, Awesome, thank you 😊 I should have checked the closed bugs more thoroughly :-X
Mit freundlichen Grüßen / Best regards Dr. Thomas Niebler Data Scientist Sales Data Lab, Analytics DC-IH/SDL1 Tel. +49 9352 18-2392 Fax +49 9352 18-0 thomas.nieb...@boschrexroth.de<mailto:thomas.nieb...@boschrexroth.de> www.boschrexroth.com Bosch Rexroth AG Partensteiner Straße 23 97816 Lohr am Main GERMANY [BOSCH REXROTH]<http://www.boschrexroth.com/> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192 Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner Lang, Reinhard Schäfer, Dr. Marc Wucherer Vorsitzender des Aufsichtsrats: Christoph Kübel ​ Von: Kaxil Naik <kaxiln...@gmail.com> Gesendet: Monday, May 4, 2020 11:38 AM An: dev@airflow.apache.org Cc: Niebler Thomas (DC-IH/SDL1) <thomas.nieb...@boschrexroth.de> Betreff: Re: Potential issue with serialized DAGs in decoupled Webserver and Scheduler And there was a wider issue, the trigger method didn't actually used the DagBag already created for the Webserver but instead created a new one which wasn't necessary. And the existing DagBag in views.py already passed all the flags correctly :) Regards, Kaxil On Mon, May 4, 2020, 10:30 Kaxil Naik <kaxiln...@gmail.com<mailto:kaxiln...@gmail.com>> wrote: Hi Niebler, Like Ash mentioned this is fixed for 1.10.11 already. This was first reported in following Github issue: - https://github.com/apache/airflow/issues/8247 And PR that fixes it for v1-10-test: https://github.com/apache/airflow/pull/8411 And https://github.com/apache/airflow/pull/8501 also fixes it in Master. Regards, Kaxil On Mon, May 4, 2020, 09:21 Ash Berlin-Taylor <a...@apache.org<mailto:a...@apache.org>> wrote: I think this was fixed by this PR - aiming for 1.10.11 https://github.com/apache/airflow/pull/8501 On 4 May 2020 07:41:55 BST, "Niebler Thomas (DC-IH/SDL1)" <thomas.nieb...@boschrexroth.de.INVALID<mailto:thomas.nieb...@boschrexroth.de.INVALID>> wrote: >Hi all, > >I have a probably rather special use case scenario: >Using Airflow 1.10.10, I would like to physically decouple the >Webserver and the Scheduler for some secure access reasons. According >to https://airflow.apache.org/docs/stable/dag-serialization.html, this >should be a piece of cake with Airflow 1.10.10, since DAGs are stored >in the Metadata database and the webserver does not need to access the >DAG files anymore. The metadata database is of course reachable by both >Airflow instances: > >Docker Image Instance: Airflow Webserver <----> Docker Image Instance: >Metadata database <----> Physical Machine: Airflow Scheduler > >However, every time I start a DAG manually, it crashes with a rather >lengthy error message: > >File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", >line 1255, in trigger > external_trigger=True >File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line >74, in wrapper > return func(*args, **kwargs) >File "/usr/local/lib/python3.7/site-packages/airflow/models/dag.py", >line 1818, in create_dagrun > return self.get_dag().create_dagrun(run_id=run_id, >AttributeError: 'NoneType' object has no attribute 'create_dagrun' > >This basically boils down to self.get_dag() not having set the Boolean >flag store_serialized_flags to True (or whatever value the config is >set to), but always using False (the default value). >This then leads to Airflow attempting to read the DAG file, ignoring >the DAG database entry and returning None, which obviously has no >attribute create_dagrun. > >I’ve got several questions here now: > >1. Is my scenario even possible or am I overlooking something rather >obvious? >2. Is the crashing DAG behavior intended like that? It rather seems >like a bug to me. >3. Is it worth fixing this issue (if it is one) for Airflow 1.10.x, >considering that Airflow 2.0.0 does not even contain the corresponding >classes anymore and takes a different path? > >Mit freundlichen Grüßen / Best regards > >Dr. Thomas Niebler >Data Scientist >Sales Data Lab, Analytics DC-IH/SDL1 > >Tel. +49 9352 18-2392 >Fax +49 9352 18-0 >thomas.nieb...@boschrexroth.de<mailto:thomas.nieb...@boschrexroth.de><mailto:thomas.nieb...@boschrexroth.de<mailto:thomas.nieb...@boschrexroth.de>> >www.boschrexroth.com<http://www.boschrexroth.com> > >Bosch Rexroth AG >Partensteiner Straße 23 >97816 Lohr am Main >GERMANY > >[BOSCH REXROTH]<http://www.boschrexroth.com/> > > > >Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192 >Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner >Lang, Reinhard Schäfer, Dr. Marc Wucherer >Vorsitzender des Aufsichtsrats: Christoph Kübel >​
