kjh0623 commented on PR #70003: URL: https://github.com/apache/airflow/pull/70003#issuecomment-5007669611
@jedcunningham Good question — pinning doesn't delay picking up new commits: the very next refresh() repoints to the new target, so the update cadence is unchanged. What it prevents is a swap landing mid-round, where one parse round (or one task run) reads files from two different commits — e.g. a DAG file from the new commit importing a shared helper still resolved from the old one. For parsing that means transient import errors that mark DAGs broken and vanish on the next round; for a task run it's a real, non-self-healing failure. We hit both in production with a ~3,600-DAG repo, where long parse rounds make the window wide. Another way to look at it: GitDagBundle also only moves its working tree forward at refresh() — nobody expects it to jump to a newer commit halfway through a parse. This flag just gives the git-sync/symlink deployment style the same refresh-boundary semantics; today it's the odd one out, with code changing underneath a round at arbitrary times. For context on why we run LocalDagBundle + git-sync rather than GitDagBundle: with KubernetesExecutor, GitDagBundle initializes a clone per pod, which for our repo adds ~15s of cold-start to every single task. Shallow/sparse cloning shaves that down but doesn't change the structure of the cost — every task pod still pays a network fetch on startup. A single git-sync deployment syncing to a shared RWX PVC removes that fetch entirely; pods just mount the volume. It's the same dags-folder + git-sync pattern from Airflow 2 that this flag targets. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
