ephraimbuddy commented on code in PR #74789:
URL: https://github.com/apache/airflow/pull/74789#discussion_r4358605704
##########
airflow-core/src/airflow/dag_processing/manager.py:
##########
@@ -926,17 +949,28 @@ def _refresh_dag_bundles(self, known_files: dict[str,
set[DagFileInfo]]):
current_version_matches_db = True
previously_seen = bundle.name in self._bundle_versions
- if self.should_skip_refresh(
- bundle=bundle,
- elapsed_time_since_refresh=elapsed_time_since_refresh,
- current_version_matches_db=current_version_matches_db,
- previously_seen=previously_seen,
+ if (
+ bundle.name not in self._bundles_waiting_for_refresh
+ and bundle.name in self._bundle_parse_sources
+ and self.should_skip_refresh(
+ bundle=bundle,
+ elapsed_time_since_refresh=elapsed_time_since_refresh,
+ current_version_matches_db=current_version_matches_db,
+ previously_seen=previously_seen,
+ )
):
self.log.debug("Not time to refresh bundle %s", bundle.name)
continue
+ if bundle.name in active_bundles:
+ # Forced refreshes wait too: refreshing under an import would
discard its result.
+ self._bundles_waiting_for_refresh.add(bundle.name)
+ continue
+
+ self._bundles_waiting_for_refresh.discard(bundle.name)
self.log.info("Refreshing bundle %s", bundle.name)
+ self._invalidate_bundle_parse_source(bundle.name)
Review Comment:
I agree that a failed fetch should not stall parsing of an otherwise usable
checkout. An unchanged HEAD does not prove that a failed reset or submodule
update left the source files unchanged, though. I've implemented recovery using
an explicit provider guarantee that the failure occurred before checkout
mutation. Git fetch failures can now retain the previous parse source and retry
the refresh, while reset or submodule failures still pause parsing until a
successful refresh establishes a consistent source. Regression tests cover both
safe fetch failures and partial mutations with unchanged HEAD.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
