Thanks Kaxil, Ash, Anita and Zhou! This is a big step forward :) T.
On Sat, Mar 14, 2020 at 12:03 PM Sumit Maheshwari <[email protected]> wrote: > Great stuff!! Thanks to everyone who contributed 👏 > > On Sat, Mar 14, 2020 at 2:26 PM Deng Xiaodong <[email protected]> wrote: > > > It is really great news and would be a big plus. > > > > Thanks Kaxil, Ash, Anita, and Zhou Fang! > > > > > > XD > > > > > On 14 Mar 2020, at 2:31 AM, Kevin Yang <[email protected]> wrote: > > > > > > This is thrilling news! Completely upleveling the experience using and > > > maintaining the webserver. Thank you so much everyone who contributed > to > > > this initiative! > > > > > > > > > Cheers, > > > Kevin Y > > > > > > On Fri, Mar 13, 2020 at 6:24 PM Kaxil Naik <[email protected]> > wrote: > > > > > >> Hi all, > > >> > > >> Happy to tell you all that we have completed the first phase of DAG > > >> Serialisation i.e. the Webserver is stateless and can now run without > > >> access to DAG Files. > > >> > > >> The 2 limitations we had in 1.10.7-1.10.9 ( > > >> > > > https://airflow.apache.org/docs/1.10.7/dag-serialization.html#limitations) > > >> have been resolved. > > >> > > >> Special thanks to @ash for his continuous guidance and contributions. > > >> > > >> Also a special mention to Anita Fronczak and Zhou Fang for their > > >> contributions along the way. > > >> > > >> The next step is to remove SimpleDag representation in the Scheduler > and > > >> replace it with Serialized DAG (WIP PR: > > >> https://github.com/apache/airflow/pull/7694) > > >> > > >> *Advantages*: > > >> > > >> - *Reduction in Webserver startup time* for large number of DAGs. > > >> Without DAG Serialization all the DAGs are loaded in the DagBag > during > > >> the > > >> Webserver startup. With DAG Serialization, an empty DagBag is > created > > >> and > > >> Dags are loaded from DB only when needed (i.e. when a particular DAG > > is > > >> clicked on in the home page) > > >> - *No DAG Parsing / Consistency*: Webserver would load DAGs from DB > > and > > >> won't even need the DAG Files when DAG Serialization is turned on. > > DAGs > > >> are > > >> parsed, serialized and stored in DB by the Scheduler. > > >> - Rendered Templates for TasksInstances that have already run will > now > > >> correctly display their value which was true at the time of the run > > >> instead > > >> of the current value. > > >> - Paves way for* DAG Versioning* (more details on it when I create a > > >> separate AIP / update an existing AIP for it) and *Scheduler HA > > *(AIP-15 > > >> < > > >> > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651 > > >>> > > >> ). > > >> > > >> > > >> I will create new JIRA issues for further steps with DAG Serialization > > and > > >> DAG Versioning and would discuss them in our next > sig-dag-serialization > > >> call (later this month). > > >> > > >> Regards, > > >> Kaxil > > >> > > > > >
