potiuk commented on pull request #14792: URL: https://github.com/apache/airflow/pull/14792#issuecomment-800112254
Hey @kaxil @ashb -> I am still chasing the remaining "count" problems and while I have not solved them all, I think most of them come from the serialized_dag missing in the SerializedDag table. Is there a way we can make sure the serialized DAG will be immediately written to the database immediately? I have a suspicion it is around min_update/fetch interval and I am trying this one: https://github.com/apache/airflow/pull/14792/commits/49910097ec88fd4298bc5fe8c9e14d3cbc8f13eb But I am not sure if this is the right fix. More info/good example here: https://github.com/apache/airflow/runs/2117788392?check_suite_focus=true#step:6:9982 ``` E scheduler_job.py:_do_scheduling>retries.py:wrapped_function>scheduler_job.py:_create_dagruns_for_dags:1573: 1 E retries.py:wrapped_function>scheduler_job.py:_create_dagruns_for_dags>scheduler_job.py:_create_dag_runs:1592: 1 E dagbag.py:get_dag>dagbag.py:_add_dag_from_db>serialized_dag.py:get:229: 3 E dagbag.py:get_dag>dagbag.py:_add_dag_from_db>serialized_dag.py:get:235: 2 E dagbag.py:get_dag>dagbag.py:_add_dag_from_db>serialized_dag.py:get:237: 2 ``` The last two lines never appear when you run the tests locally. Lines of code: ``` 229 row = session.query(cls).filter(cls.dag_id == dag_id).one_or_none() if row: return row # If we didn't find a matching DAG id then ask the DAG table to find # out the root dag 235 root_dag_id = session.query(DagModel.root_dag_id).filter(DagModel.dag_id == dag_id).scalar() 237 return session.query(cls).filter(cls.dag_id == root_dag_id).one_or_none() ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
