ephraimbuddy commented on code in PR #74789: URL: https://github.com/apache/airflow/pull/74789#discussion_r4490969822
########## airflow-core/docs/administration-and-deployment/dag-bundles.rst: ########## @@ -261,6 +261,20 @@ Some caveats: * A failing refresh looks like one that is not due, since a refresh that raises is logged and leaves ``last_refreshed`` at its last success. Check the Dag processor logs, and ``/api/v2/jobs?job_type=DagProcessorJob`` for a live processor. + If the refresh may have changed files, parsing pauses until a refresh succeeds. The processor + logs the affected bundle and reports the total number of deferred files through + ``dag_processing.files_awaiting_source``. On metrics backends that support tags, + ``dag_processing.files_awaiting_source_by_bundle`` provides counts tagged with ``bundle_name``, and + with ``team_name`` in multi-team mode. + A provider that reports an unchanged source after failure lets ordinary parsing continue against + the previous source while refresh retries continue. Currently only ``GitDagBundle`` reports this, + for credential and fetch failures, in git provider releases that support it; the S3, GCS and + Azure Blob Storage bundles pause on any refresh failure. Explicit reparse requests wait for a + successful refresh. A Dag processor started with ``--num-runs`` does not exit while files wait for + a paused bundle, so it finishes only after that bundle refreshes successfully. + If listing the bundle's Dag files or recording the refresh fails after a successful refresh, the + processor keeps parsing the files it listed before, from the refreshed source, and retries the + refresh no later than the bundle's :ref:`config:dag_processor__refresh_interval`. Review Comment: Agreed on both. The retry waits until `refresh_interval` has passed, and only a failed listing keeps the earlier file list; when just `update_bundle_state` fails, the new listing is used. 4552a6bc20 splits the two cases and says "after", noting that a reparse request still skips the wait. ########## airflow-core/docs/administration-and-deployment/dag-bundles.rst: ########## @@ -261,6 +261,20 @@ Some caveats: * A failing refresh looks like one that is not due, since a refresh that raises is logged and leaves ``last_refreshed`` at its last success. Check the Dag processor logs, and ``/api/v2/jobs?job_type=DagProcessorJob`` for a live processor. + If the refresh may have changed files, parsing pauses until a refresh succeeds. The processor + logs the affected bundle and reports the total number of deferred files through + ``dag_processing.files_awaiting_source``. On metrics backends that support tags, + ``dag_processing.files_awaiting_source_by_bundle`` provides counts tagged with ``bundle_name``, and + with ``team_name`` in multi-team mode. + A provider that reports an unchanged source after failure lets ordinary parsing continue against + the previous source while refresh retries continue. Currently only ``GitDagBundle`` reports this, + for credential and fetch failures, in git provider releases that support it; the S3, GCS and + Azure Blob Storage bundles pause on any refresh failure. Explicit reparse requests wait for a + successful refresh. A Dag processor started with ``--num-runs`` does not exit while files wait for + a paused bundle, so it finishes only after that bundle refreshes successfully. Review Comment: Yes, added `74789.significant.rst` in 4552a6bc20. It says a failed refresh now pauses that bundle's parsing until a refresh succeeds (any failure for S3, GCS, Azure Blob Storage and older git provider releases; Git keeps parsing after credential or fetch failures), and that a `--num-runs` processor waits for that refresh before it exits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
