DoDiODev opened a new pull request, #9218:
URL: https://github.com/apache/devlake/pull/9218
### Summary
Incremental git collection fails sporadically in `Collect Commits` with:
```
could not find
'/tmp/gitextractorXXXX/objects/pack/.tmp-26800-pack-<sha>.idx' to open: No such
file or directory
```
Root cause: since git 2.29, `git fetch` ends with `git maintenance run
--auto`
(older versions: `git gc --auto`), and that runs **detached** in the
background
by default. For an incremental collection `GitcliCloner` runs several fetches
(shallow fetch, `--shallow-since`, `--deepen` rounds). The last one may
leave a
background maintenance or gc process repacking the repository while libgit2
is
already scanning `objects/pack/`. libgit2 sees the temporary
`.tmp-<pid>-pack-*.idx` file, which git renames or removes a moment later,
and
the subtask aborts. Because this is a race, the failure is intermittent.
This PR passes
`-c gc.auto=0 -c gc.autoDetach=false -c maintenance.auto=false` to every git
command run by `GitcliCloner`. The clone is a short-lived temporary directory
that is removed after extraction, so automatic housekeeping has no benefit
there. The explicit `git repack -d` in `deepen()` is still executed
synchronously as before.
A regression test runs an incremental clone with `GIT_TRACE` and asserts that
git does not start `maintenance run` / `gc --auto`. Without the fix the test
fails with
`run_command: git maintenance run --auto --no-quiet --detach` after each
fetch.
### Does this close any open issues?
N/A
### Screenshots
N/A
### Other Information
- Unknown config keys are ignored by git, so `maintenance.auto` is safe on
git
versions older than 2.29; `gc.auto=0` covers those versions.
- Checked on a local instance (git 2.56.0): before the change an incremental
run failed in `Collect Commits` with the error above; with the change, 47
gitextractor tasks across two further pipeline runs completed without it.
- Full CI mirror on my fork, including unit tests (the new test passes
there),
lint and e2e on MySQL and PostgreSQL:
[run](https://github.com/DoDiODev/devlake/actions/runs/38557507450)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
