Hi Anita, Yes, we fixed all the bugs that were found or reported during 1.10.10 rc voting.
There were some bugs in the DagCode that were found in 1.10.10rc's (have tagged you in few of the PR with fixes) but fixed before 1.10.10 was released. Some of Astronomer's (https://www.astronomer.io/cloud/) customers use Dag Serialization in production (with Airflow >=1.10.5) and haven't reported any critical/major issues yet even with 1.10.10. But after looking at the changes I do not think there are many major issues > left that have not been fixed in Airflow 1.10.10 already. I looked at the PR and I am not sure it fixes anything. "conf.getboolean" is an expensive operation and moving the store_serialized_dags config from settings.py to all places just means that conf.getboolean is called every time and if it is run inside a loop, that would increase the time taken by the process running the loop. store_serialized_dags should be set to false everywhere except UI, experimental > API and dag import That is the case now too, I have left comments on the PR. Regards, Kaxil On Tue, May 5, 2020 at 4:58 PM Anita Fronczak <akar...@gmail.com> wrote: > Hello again, > > So I have created a PR: https://github.com/apache/airflow/pull/8718 > > Some context. Cloud Composer supports store_serialized_dags and > store_dag_code for Airflow 1.10.3 and 1.10.6 (but without rendering > templates) to support VPC SC ( > https://cloud.google.com/composer/docs/release-notes#March_20_2020). Some > changes were made that did not make into open source Apache Airflow on time > to make it work. > But after looking at the changes I do not think there are many major > issues left that have not been fixed in Airflow 1.10.10 already. > The PR is a version that merges into master not the one from Airflow > 1.10.6. > > Anita > > pon., 4 maj 2020 o 12:09 Niebler Thomas (DC-IH/SDL1) > <thomas.nieb...@boschrexroth.de.invalid> napisał(a): > >> Hi Kaxil, >> >> Awesome, thank you 😊 I should have checked the closed bugs more >> thoroughly :-X >> >> Mit freundlichen Grüßen / Best regards >> >> Dr. Thomas Niebler >> Data Scientist >> Sales Data Lab, Analytics DC-IH/SDL1 >> >> Tel. +49 9352 18-2392 >> Fax +49 9352 18-0 >> thomas.nieb...@boschrexroth.de<mailto:thomas.nieb...@boschrexroth.de> >> www.boschrexroth.com >> >> Bosch Rexroth AG >> Partensteiner Straße 23 >> 97816 Lohr am Main >> GERMANY >> >> [BOSCH REXROTH]<http://www.boschrexroth.com/> >> >> >> >> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192 >> Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner >> Lang, Reinhard Schäfer, Dr. Marc Wucherer >> Vorsitzender des Aufsichtsrats: Christoph Kübel >> >> Von: Kaxil Naik <kaxiln...@gmail.com> >> Gesendet: Monday, May 4, 2020 11:38 AM >> An: dev@airflow.apache.org >> Cc: Niebler Thomas (DC-IH/SDL1) <thomas.nieb...@boschrexroth.de> >> Betreff: Re: Potential issue with serialized DAGs in decoupled Webserver >> and Scheduler >> >> And there was a wider issue, the trigger method didn't actually used the >> DagBag already created for the Webserver but instead created a new one >> which wasn't necessary. And the existing DagBag in views.py already passed >> all the flags correctly :) >> >> Regards, >> Kaxil >> >> On Mon, May 4, 2020, 10:30 Kaxil Naik <kaxiln...@gmail.com<mailto: >> kaxiln...@gmail.com>> wrote: >> Hi Niebler, >> >> Like Ash mentioned this is fixed for 1.10.11 already. >> >> This was first reported in following Github issue: >> >> - https://github.com/apache/airflow/issues/8247 >> >> And PR that fixes it for v1-10-test: >> https://github.com/apache/airflow/pull/8411 >> >> And https://github.com/apache/airflow/pull/8501 also fixes it in Master. >> >> Regards, >> Kaxil >> >> On Mon, May 4, 2020, 09:21 Ash Berlin-Taylor <a...@apache.org<mailto: >> a...@apache.org>> wrote: >> I think this was fixed by this PR - aiming for 1.10.11 >> >> https://github.com/apache/airflow/pull/8501 >> >> On 4 May 2020 07:41:55 BST, "Niebler Thomas (DC-IH/SDL1)" >> <thomas.nieb...@boschrexroth.de.INVALID<mailto: >> thomas.nieb...@boschrexroth.de.INVALID>> wrote: >> >Hi all, >> > >> >I have a probably rather special use case scenario: >> >Using Airflow 1.10.10, I would like to physically decouple the >> >Webserver and the Scheduler for some secure access reasons. According >> >to https://airflow.apache.org/docs/stable/dag-serialization.html, this >> >should be a piece of cake with Airflow 1.10.10, since DAGs are stored >> >in the Metadata database and the webserver does not need to access the >> >DAG files anymore. The metadata database is of course reachable by both >> >Airflow instances: >> > >> >Docker Image Instance: Airflow Webserver <----> Docker Image Instance: >> >Metadata database <----> Physical Machine: Airflow Scheduler >> > >> >However, every time I start a DAG manually, it crashes with a rather >> >lengthy error message: >> > >> >File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", >> >line 1255, in trigger >> > external_trigger=True >> >File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line >> >74, in wrapper >> > return func(*args, **kwargs) >> >File "/usr/local/lib/python3.7/site-packages/airflow/models/dag.py", >> >line 1818, in create_dagrun >> > return self.get_dag().create_dagrun(run_id=run_id, >> >AttributeError: 'NoneType' object has no attribute 'create_dagrun' >> > >> >This basically boils down to self.get_dag() not having set the Boolean >> >flag store_serialized_flags to True (or whatever value the config is >> >set to), but always using False (the default value). >> >This then leads to Airflow attempting to read the DAG file, ignoring >> >the DAG database entry and returning None, which obviously has no >> >attribute create_dagrun. >> > >> >I’ve got several questions here now: >> > >> >1. Is my scenario even possible or am I overlooking something rather >> >obvious? >> >2. Is the crashing DAG behavior intended like that? It rather seems >> >like a bug to me. >> >3. Is it worth fixing this issue (if it is one) for Airflow 1.10.x, >> >considering that Airflow 2.0.0 does not even contain the corresponding >> >classes anymore and takes a different path? >> > >> >Mit freundlichen Grüßen / Best regards >> > >> >Dr. Thomas Niebler >> >Data Scientist >> >Sales Data Lab, Analytics DC-IH/SDL1 >> > >> >Tel. +49 9352 18-2392 >> >Fax +49 9352 18-0 >> >thomas.nieb...@boschrexroth.de<mailto:thomas.nieb...@boschrexroth.de >> ><mailto:thomas.nieb...@boschrexroth.de<mailto: >> thomas.nieb...@boschrexroth.de>> >> >www.boschrexroth.com<http://www.boschrexroth.com> >> > >> >Bosch Rexroth AG >> >Partensteiner Straße 23 >> >97816 Lohr am Main >> >GERMANY >> > >> >[BOSCH REXROTH]<http://www.boschrexroth.com/> >> > >> > >> > >> >Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart HRB 23192 >> >Vorstand: Rolf Najork (Vorsitzender), Dr. Markus Forschner, Dr. Heiner >> >Lang, Reinhard Schäfer, Dr. Marc Wucherer >> >Vorsitzender des Aufsichtsrats: Christoph Kübel >> > >> >
