anitakar opened a new pull request #8718: URL: https://github.com/apache/airflow/pull/8718
store_serialized_dags should be set to false everywhere except UI, experimental API and dag import Added missing store_serialized_dags to the _DagBag class. When I have applied store_serialized_dags from config everywhere I did not know/understand how this parameter works. It caused some problems: * migrations failed as they have used DagBag with SerializedDag before serialized_dags table was created * code that runs after all migrations and clean up dags took much longer as it was creating and saving serialized representations of dags * and, first of all, scheduler stopped parsing dags because DagBag::collect_dags method that it uses, returned no dags, as it does not collect dags when stored_serialized_dags is on. Generally, even though store_serialized_dags parameter is used in models and throughout common code, it should only be set to True in web server. If it is set to true in other parts of the system, they will stop working. CLI will stop working. Scheduler is the element that creates serialized dags and it should always read from disk to keep other parts of the system up to date. But there is no way to check whether dag_bag was called by web server and store_serialized_dags should be applied or if it was called from scheduler and dags should be read from files on the disk not from the db. --- Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Target Github ISSUE in description if exists - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
