kaxil commented on pull request #14792: URL: https://github.com/apache/airflow/pull/14792#issuecomment-800225965
> Hey @kaxil @ashb -> I am still chasing the remaining "count" problems and while I have not solved them all, I think most of them come from the serialized_dag missing in the SerializedDag table. > > Is there a way we can make sure the serialized DAG will be immediately written to the database immediately? > > I have a suspicion it is around min_update/fetch interval and I am trying this one: > > [4991009](https://github.com/apache/airflow/commit/49910097ec88fd4298bc5fe8c9e14d3cbc8f13eb) > > But I am not sure if this is the right fix. > > More info/good example here: > > https://github.com/apache/airflow/runs/2117788392?check_suite_focus=true#step:6:9982 > > ``` > E scheduler_job.py:_do_scheduling>retries.py:wrapped_function>scheduler_job.py:_create_dagruns_for_dags:1573: 1 > E retries.py:wrapped_function>scheduler_job.py:_create_dagruns_for_dags>scheduler_job.py:_create_dag_runs:1592: 1 > E dagbag.py:get_dag>dagbag.py:_add_dag_from_db>serialized_dag.py:get:229: 3 > E dagbag.py:get_dag>dagbag.py:_add_dag_from_db>serialized_dag.py:get:235: 2 > E dagbag.py:get_dag>dagbag.py:_add_dag_from_db>serialized_dag.py:get:237: 2 > ``` > > The last two lines never appear when you run the tests locally. > > Lines of code: > > ``` > 229 row = session.query(cls).filter(cls.dag_id == dag_id).one_or_none() > if row: > return row > > # If we didn't find a matching DAG id then ask the DAG table to find > # out the root dag > 235 root_dag_id = session.query(DagModel.root_dag_id).filter(DagModel.dag_id == dag_id).scalar() > > 237 return session.query(cls).filter(cls.dag_id == root_dag_id).one_or_none() > ``` Setting `min_serialized_dag_update_interval` to `0` should do it -- shouldn't need any code change for that -- only tests (I think) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
