Hello again, So I have created a PR: https://github.com/apache/airflow/pull/8718
Some context. Cloud Composer supports store_serialized_dags and store_dag_code for Airflow 1.10.3 and 1.10.6 (but without rendering templates) to support VPC SC ( https://cloud.google.com/composer/docs/release-notes#March_20_2020). Some changes were made that did not make into open source Apache Airflow on time to make it work. But after looking at the changes I do not think there are many major issues left that have not been fixed in Airflow 1.10.10 already. The PR is a version that merges into master not the one from Airflow 1.10.6. Anita pon., 4 maj 2020 o 12:09 Niebler Thomas (DC-IH/SDL1) <thomas.nieb...@boschrexroth.de.invalid> napisał(a): > Hi Kaxil, > > Awesome, thank you 😊 I should have checked the closed bugs more > thoroughly :-X > > Mit freundlichen Grüßen / Best regards > > Dr. Thomas Niebler > Data Scientist > Sales Data Lab, Analytics DC-IH/SDL1 > > Tel. +49 9352 18-2392 > Fax +49 9352 18-0 > thomas.nieb...@boschrexroth.de<mailto:thomas.nieb...@boschrexroth.de> > www.boschrexroth.com > > Bosch Rexroth AG > Partensteiner Straße 23 > 97816 Lohr am Main > GERMANY > > [BOSCH REXROTH]<http://www.boschrexroth.com/> > > > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192 > Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner > Lang, Reinhard Schäfer, Dr. Marc Wucherer > Vorsitzender des Aufsichtsrats: Christoph Kübel > > Von: Kaxil Naik <kaxiln...@gmail.com> > Gesendet: Monday, May 4, 2020 11:38 AM > An: dev@airflow.apache.org > Cc: Niebler Thomas (DC-IH/SDL1) <thomas.nieb...@boschrexroth.de> > Betreff: Re: Potential issue with serialized DAGs in decoupled Webserver > and Scheduler > > And there was a wider issue, the trigger method didn't actually used the > DagBag already created for the Webserver but instead created a new one > which wasn't necessary. And the existing DagBag in views.py already passed > all the flags correctly :) > > Regards, > Kaxil > > On Mon, May 4, 2020, 10:30 Kaxil Naik <kaxiln...@gmail.com<mailto: > kaxiln...@gmail.com>> wrote: > Hi Niebler, > > Like Ash mentioned this is fixed for 1.10.11 already. > > This was first reported in following Github issue: > > - https://github.com/apache/airflow/issues/8247 > > And PR that fixes it for v1-10-test: > https://github.com/apache/airflow/pull/8411 > > And https://github.com/apache/airflow/pull/8501 also fixes it in Master. > > Regards, > Kaxil > > On Mon, May 4, 2020, 09:21 Ash Berlin-Taylor <a...@apache.org<mailto: > a...@apache.org>> wrote: > I think this was fixed by this PR - aiming for 1.10.11 > > https://github.com/apache/airflow/pull/8501 > > On 4 May 2020 07:41:55 BST, "Niebler Thomas (DC-IH/SDL1)" > <thomas.nieb...@boschrexroth.de.INVALID<mailto: > thomas.nieb...@boschrexroth.de.INVALID>> wrote: > >Hi all, > > > >I have a probably rather special use case scenario: > >Using Airflow 1.10.10, I would like to physically decouple the > >Webserver and the Scheduler for some secure access reasons. According > >to https://airflow.apache.org/docs/stable/dag-serialization.html, this > >should be a piece of cake with Airflow 1.10.10, since DAGs are stored > >in the Metadata database and the webserver does not need to access the > >DAG files anymore. The metadata database is of course reachable by both > >Airflow instances: > > > >Docker Image Instance: Airflow Webserver <----> Docker Image Instance: > >Metadata database <----> Physical Machine: Airflow Scheduler > > > >However, every time I start a DAG manually, it crashes with a rather > >lengthy error message: > > > >File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", > >line 1255, in trigger > > external_trigger=True > >File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line > >74, in wrapper > > return func(*args, **kwargs) > >File "/usr/local/lib/python3.7/site-packages/airflow/models/dag.py", > >line 1818, in create_dagrun > > return self.get_dag().create_dagrun(run_id=run_id, > >AttributeError: 'NoneType' object has no attribute 'create_dagrun' > > > >This basically boils down to self.get_dag() not having set the Boolean > >flag store_serialized_flags to True (or whatever value the config is > >set to), but always using False (the default value). > >This then leads to Airflow attempting to read the DAG file, ignoring > >the DAG database entry and returning None, which obviously has no > >attribute create_dagrun. > > > >I’ve got several questions here now: > > > >1. Is my scenario even possible or am I overlooking something rather > >obvious? > >2. Is the crashing DAG behavior intended like that? It rather seems > >like a bug to me. > >3. Is it worth fixing this issue (if it is one) for Airflow 1.10.x, > >considering that Airflow 2.0.0 does not even contain the corresponding > >classes anymore and takes a different path? > > > >Mit freundlichen Grüßen / Best regards > > > >Dr. Thomas Niebler > >Data Scientist > >Sales Data Lab, Analytics DC-IH/SDL1 > > > >Tel. +49 9352 18-2392 > >Fax +49 9352 18-0 > >thomas.nieb...@boschrexroth.de<mailto:thomas.nieb...@boschrexroth.de > ><mailto:thomas.nieb...@boschrexroth.de<mailto: > thomas.nieb...@boschrexroth.de>> > >www.boschrexroth.com<http://www.boschrexroth.com> > > > >Bosch Rexroth AG > >Partensteiner Straße 23 > >97816 Lohr am Main > >GERMANY > > > >[BOSCH REXROTH]<http://www.boschrexroth.com/> > > > > > > > >Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192 > >Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner > >Lang, Reinhard Schäfer, Dr. Marc Wucherer > >Vorsitzender des Aufsichtsrats: Christoph Kübel > > >
