piotrlinski opened a new pull request, #71713:
URL: https://github.com/apache/airflow/pull/71713
A Git DAG bundle mirrors the whole repository — full history, every branch
and tag — into a bare clone before checking out a version. That amortises well
for a long-lived DAG processor or Celery worker that reuses the clone, but a
**KubernetesExecutor task pod is ephemeral**: its bundle cache starts empty
every time, so each task re-clones the entire repository just to check out the
one commit it was pinned to. With many tasks and large repositories this is a
large, repeated waste of network and disk. (By contrast, the legacy Helm
git-sync path defaults to a shallow `depth: 1`.)
When running a pinned version, the worker already knows the exact commit SHA
(`DagRun.bundle_version`), so the bundle can fetch only that commit. This adds
an opt-in `shallow` option to `GitDagBundle`:
- When a version is pinned **and** `shallow` is set, the bundle does `git
fetch --depth 1 <sha>` straight into the version directory, **skipping the
full-history bare mirror**.
- It requires the remote to allow fetching an unadvertised commit
(`uploadpack.allowAnySHA1InWant` / `allowReachableSHA1InWant`, which GitHub,
GitLab and Bitbucket Cloud enable). If the server refuses, the bundle
**transparently falls back to the existing full clone**, so it is safe to
enable anywhere.
- It is **off by default** and has **no effect on the unpinned
branch-tracking path** used by the DAG processor (which legitimately needs the
full history for versioning).
Enable it per bundle:
```json
{
"name": "my_dags",
"classpath": "airflow.providers.git.bundles.git.GitDagBundle",
"kwargs": {"tracking_ref": "main", "git_conn_id": "my_git", "shallow":
true}
}
```
Tests cover: default off; shallow fetches only the pinned commit and skips
the bare mirror; the default `.git`-pruned worktree is produced and reused;
`sparse_dirs` still applies; fallback to full clone when the shallow fetch is
rejected; and cleanup (repo handle closed, partial dir removed) when finalizing
the checkout fails.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.8)
Generated-by: Claude Code (Opus 4.8) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
