jason810496 commented on code in PR #74249:
URL: https://github.com/apache/airflow/pull/74249#discussion_r4193551018
##########
airflow-core/src/airflow/dag_processing/dagbag.py:
##########
@@ -327,17 +327,21 @@ def _process_definition(
self.captured_warnings.pop(fileloc, None)
self._import_warnings.pop(fileloc, None)
- self.dag_source_codes.pop(fileloc, None)
+ # Keyed by dag_id, not fileloc: a bundle can embed several Dags' own
source under one
+ # fileloc, so a stale entry is found by which dag_ids this fileloc
previously bagged.
+ for dag_id, dag in list(self.dags.items()):
Review Comment:
Here're some notes in my mind to solve the further write amplification if
there's anyone complain in the future.
```suggestion
# TODO: store the file content more efficient (e.g. if multiple
dag_ids will store the exact same DagCode content)
# There're two parts: 1) the Dagbag in-memory path 2) the DB
persistent layer
# The bigger picture might be normalized the DagCode table, so we
won't store the exact same DagCode content twice.
for dag_id, dag in list(self.dags.items()):
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
