hkc-8010 opened a new issue, #72922:
URL: https://github.com/apache/airflow/issues/72922
### Apache Airflow Provider(s)
git
### Versions of Apache Airflow Providers
apache-airflow-providers-git 0.4.x (code path unchanged on main)
### Apache Airflow version
3.3.0
### Operating System
Linux (containerized)
### Deployment
Other 3rd-party Helm chart
### Deployment details
Long-lived worker pods (a worker Deployment rather than one pod per task),
with the bundle storage path left at its default under `/tmp`, so it is
pod-local and lives as long as the pod.
### What happened
A task was marked failed from the UI while its worker was cloning the DAG
bundle's tracking repository. The clone started at 06:04:11.472800 and the
supervisor sent SIGTERM at 06:04:11.787280, about 315 ms in:
```
{"timestamp":"2026-09-01T06:04:11.775140Z","level":"error","event":"Server
indicated the task shouldn't be running
anymore","detail":{"reason":"not_running","message":"TI is no longer in the
running state and task should
terminate","current_state":"failed"},"status_code":409,"logger":"supervisor"}
{"timestamp":"2026-09-01T06:04:11.787280Z","level":"info","event":"Process
exited","pid":16,"exit_code":-15,"signal_sent":"SIGTERM","logger":"supervisor"}
```
From then on every task that landed on that worker failed at parse time:
```
Top level error source=task loc=task_runner.py:2347 GitCommandError:
Cmd('git') failed due to: exit code(1)
cmdline: git checkout master
stderr: 'error: The following untracked working tree files would be
overwritten by checkout:
.agents/instructions/airflow-patterns.md
.agents/instructions/column_naming_conventions.md
.agents/instructions/data-modeling-conventions.md
<...>
Aborting'
File .../airflow/sdk/execution_time/task_runner.py, line 1005 in parse
File .../airflow/providers/git/bundles/git.py, line 244 in initialize
File .../airflow/providers/git/bundles/git.py, line 215 in _initialize
```
Thirty task instances ran on that pod over the following 23 minutes and all
thirty failed. Sibling pods in the same ReplicaSet were unaffected, one of them
completed 1,180 tasks the same day, and it was the only worker in three weeks
with a 100% failure rate. The killed task was the first task that pod ever ran,
so it was poisoned on its first bundle initialization and never served a
successful task. Recovery only came when the workers rolled to a new ReplicaSet
20 minutes later.
The leftover directory here is different from the one in #72759. There the
working tree is empty and the refs are missing, so the checkout fails with
`pathspec ... did not match any file(s) known to git`. Here `git clone` had got
far enough to write files into the working tree but not far enough to record
them in the index, so those files are untracked relative to the incomplete
index and the checkout refuses to overwrite them.
Why it never self-heals is the same as in #72759:
1. `_is_pruned_worktree()` treats "no `.git`" as a completed version, and a
`.git` is present, so it returns `False`.
2. `_local_repo_has_version()` returns `False` for the truncated repo.
3. `_clone_repo_if_required()` only clones when `repo_path` does not exist.
It does exist, so the clone is skipped and the truncated repository is reopened.
4. `self.repo.git.checkout(self.tracking_ref)` then fails. It sits outside
both the `try/except` and the `@retry` / `shutil.rmtree` block inside
`_clone_repo_if_required()`, so nothing cleans the directory up and nothing
retries.
Two things make this worse than a one-off task failure. The failure is
raised in `task_runner.parse()`, before the DAG is parsed, so task-level
`on_failure_callback` and `on_retry_callback` never run and the failures are
invisible to DAG-level alerting. And on a long-lived worker the pod-local
bundle directory survives for the pod's lifetime, so a transient fault becomes
permanent. On KubernetesExecutor, where every task gets a fresh pod, the same
bug is invisible.
### Which of the open fixes cover this
#72787 does. Moving the checkout inside `_clone_repo_if_required()`'s `try`
means the `GitCommandError` hits the existing `except
(InvalidGitRepositoryError, GitCommandError)` handler,
`shutil.rmtree(repo_path)` runs, and the single retry produces a clean clone.
#71535 does not. Fetching the missing tracking refs before the checkout does
nothing about untracked files in the working tree, so `git checkout master`
still aborts with the error above. So this state is a concrete reason to prefer
the broader recovery in #72787 over the narrow fetch, rather than a matter of
taste.
### What is still uncovered by either
Both fixes recover from the broken directory after the fact. Neither stops
it being created. `Repo.clone_from` still writes straight into `repo_path`, so
any interruption can still leave a directory there that the next initialization
has to detect and discard. #72759 already noted one residual case this leaves
open: a process killed in the narrow window before `git clone` writes the
remote configuration leaves a repository with no `origin`, where
`self.repo.remotes.origin` raises `AttributeError` rather than anything the
recovery path catches.
Cloning into a staging directory next to the final path and renaming it into
place once the clone is complete removes the whole class. `rename` is atomic,
so `repo_path` either does not exist or holds a finished clone, and an
interrupted clone leaves only an orphaned staging directory that the next
attempt discards before it starts.
### What you think should happen instead
An interrupted clone should not be able to poison a bundle directory at all,
and if a poisoned directory does exist (from an earlier provider version, or
from anything else that damaged it), initialization should discard and re-clone
it rather than failing forever.
### How to reproduce
The end state reproduces in seconds without racing a real signal:
```bash
git clone --no-checkout <bare-repo> tracking_repo # valid .git, empty
index and working tree
# copy the repository's tracked files into tracking_repo/ so they are
untracked
git -C tracking_repo checkout master
# error: The following untracked working tree files would be overwritten by
checkout: ...
# Aborting
```
Point a `GitDagBundle` at that directory and every `initialize()` fails with
the traceback above.
For the crash-consistency half, patch `Repo.clone_from` to write a `.git`
skeleton into its target and then raise, and check whether `repo_path` exists
afterwards. On main it does.
### Anything else
Happy to take the crash-consistency change; I have it working with tests.
The recovery half is already covered by #72787 and I would rather that landed
than have a competing PR for it.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
