ephraimbuddy commented on code in PR #74789:
URL: https://github.com/apache/airflow/pull/74789#discussion_r4490969822


##########
airflow-core/docs/administration-and-deployment/dag-bundles.rst:
##########
@@ -261,6 +261,20 @@ Some caveats:
 * A failing refresh looks like one that is not due, since a refresh that 
raises is logged and leaves
   ``last_refreshed`` at its last success. Check the Dag processor logs, and
   ``/api/v2/jobs?job_type=DagProcessorJob`` for a live processor.
+  If the refresh may have changed files, parsing pauses until a refresh 
succeeds. The processor
+  logs the affected bundle and reports the total number of deferred files 
through
+  ``dag_processing.files_awaiting_source``. On metrics backends that support 
tags,
+  ``dag_processing.files_awaiting_source_by_bundle`` provides counts tagged 
with ``bundle_name``, and
+  with ``team_name`` in multi-team mode.
+  A provider that reports an unchanged source after failure lets ordinary 
parsing continue against
+  the previous source while refresh retries continue. Currently only 
``GitDagBundle`` reports this,
+  for credential and fetch failures, in git provider releases that support it; 
the S3, GCS and
+  Azure Blob Storage bundles pause on any refresh failure. Explicit reparse 
requests wait for a
+  successful refresh. A Dag processor started with ``--num-runs`` does not 
exit while files wait for
+  a paused bundle, so it finishes only after that bundle refreshes 
successfully.
+  If listing the bundle's Dag files or recording the refresh fails after a 
successful refresh, the
+  processor keeps parsing the files it listed before, from the refreshed 
source, and retries the
+  refresh no later than the bundle's 
:ref:`config:dag_processor__refresh_interval`.

Review Comment:
   Agreed on both. The retry waits until `refresh_interval` has passed, and 
only a failed listing keeps the earlier file list; when just 
`update_bundle_state` fails, the new listing is used. 4552a6bc20 splits the two 
cases and says "after", noting that a reparse request still skips the wait.
   



##########
airflow-core/docs/administration-and-deployment/dag-bundles.rst:
##########
@@ -261,6 +261,20 @@ Some caveats:
 * A failing refresh looks like one that is not due, since a refresh that 
raises is logged and leaves
   ``last_refreshed`` at its last success. Check the Dag processor logs, and
   ``/api/v2/jobs?job_type=DagProcessorJob`` for a live processor.
+  If the refresh may have changed files, parsing pauses until a refresh 
succeeds. The processor
+  logs the affected bundle and reports the total number of deferred files 
through
+  ``dag_processing.files_awaiting_source``. On metrics backends that support 
tags,
+  ``dag_processing.files_awaiting_source_by_bundle`` provides counts tagged 
with ``bundle_name``, and
+  with ``team_name`` in multi-team mode.
+  A provider that reports an unchanged source after failure lets ordinary 
parsing continue against
+  the previous source while refresh retries continue. Currently only 
``GitDagBundle`` reports this,
+  for credential and fetch failures, in git provider releases that support it; 
the S3, GCS and
+  Azure Blob Storage bundles pause on any refresh failure. Explicit reparse 
requests wait for a
+  successful refresh. A Dag processor started with ``--num-runs`` does not 
exit while files wait for
+  a paused bundle, so it finishes only after that bundle refreshes 
successfully.

Review Comment:
   Yes, added `74789.significant.rst` in 4552a6bc20. It says a failed refresh 
now pauses that bundle's parsing until a refresh succeeds (any failure for S3, 
GCS, Azure Blob Storage and older git provider releases; Git keeps parsing 
after credential or fetch failures), and that a `--num-runs` processor waits 
for that refresh before it exits.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to