chenwei791129 opened a new issue, #9189: URL: https://github.com/apache/devlake/issues/9189
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/devlake/issues?q=is%3Aissue) and found no similar issues. Related, but closed as stale without a root cause: #8434 (same `skip commit ... because it has no parent commit` log and the same Lead Time impact), #8818, #8857. This issue adds the root cause and a small self-contained reproduction. It also explains why "increase the time range" (suggested in #8434) does not help: the lost commits are **newer** than the incremental start time, not older than `timeAfter`. ### What happened **In short:** after the first run, `gitextractor` collects commits incrementally. In some common merge situations, it silently and permanently loses commits that are well inside the collection window. Their child commits are stored, and `commit_parents` even points to them, but the commits themselves never appear in `commits` or `repo_commits`, and later runs never pick them up. **Why it matters:** when a lost commit is a deployment head, `refdiff` compares the next deployment against an incomplete commit graph. Very old PRs are then attributed to a recent deployment, and DORA Lead Time for Changes jumps (we see samples of about 10,000 hours). #9187 describes how those wrong diffs then stay cached. **What we see on our instance** (self-managed GitLab, daily blueprints, merge-commit MR workflow): - In one repository, 7 of 91 default-branch commits over six weeks were never collected. They are scattered, not in one time range. - At least 35 production deployment head commits have exactly this signature: missing from `commits`, but referenced as `parent_commit_sha` in `commit_parents`. - The pipeline log shows `skip commit <sha> because it has no parent commit` for them. Most other skipped SHAs really are older than `timeAfter` (the intended case), so the log line alone does not reveal the problem. #### How it happens, step by step Take this history on the default branch. The previous incremental run started at 12:00, so this run fetches with `since = 12:00`: ``` 12:00 = since (previous run start) | R ------------ | -- Q ------ P ------ M main (01-01) | 13:00 14:00 16:00 (merge commit) \ | / F ---------- | --------------------+ feature (head F at 10:00, 10:00 | collected by the previous run) ``` `Q` and `P` landed on `main` after the previous run. Then a merge request merged `feature` (whose head `F` is older than `since`) with the merge commit `M`. So this run should store `Q`, `P` and `M`. Incremental clone does this (`GitcliCloner.shallowClone()` + `deepen()` in `plugins/gitextractor/parser/clone_gitcli.go`): ``` git clone --depth=1 --bare <url> git fetch --depth=1 origin git fetch --shallow-since=<since> git repack -d git fetch --deepen=1 ``` 1. **`--shallow-since` stops at `M`.** Git decides "shallow or not" per commit: because one parent of `M` (`F`) is older than `since`, git cuts `M` off from **all** its parents. `P` and `Q` are newer than `since`, but they are only reachable through `M`, so they are not fetched. ``` local repo after --shallow-since: M (shallow: no parents) F (branch head) ``` 2. **`--deepen=1` adds one generation**, so `P` arrives. `Q`, the parent of `P`, is still missing. ``` local repo after --deepen=1: P (parent Q missing) -- M F ``` 3. **The collector drops `P` entirely.** `Libgit2RepoCollector.CollectCommits()` (the same code is in `GogitRepoCollector`) returns early for any commit whose first parent is not in the local repo: ```go // Skip calculating commit statistics when there are parent commits, but the first one cannot be fetched from the ODB. ... if parent == nil { r.logger.Info("skip commit %s because it has no parent commit", commit.Id().String()) return nil } ``` This `return nil` runs before `store.Commits`, `store.RepoCommits` and `storeParentCommits`, so `P` is not stored at all. The comment (added in #7720) says only the statistics should be skipped. `Q` was never fetched. 4. **The gap is permanent.** The next run uses `since` = the start of this run. `P` and `Q` are now older than that, so no incremental run fetches them again. Only a full sync recovers them. **When it triggers:** between two runs, the default branch first gets a new commit or merge (`P`), and then a merge request whose source-branch head is older than the previous run is merged with a merge commit (`M`). This is common in merge-commit MR/PR workflows. Linear histories (fast-forward or squash only) are not affected. ### What do you expect to happen Incremental collection stores every commit that is reachable from the collected branches and newer than the previous run, together with its `repo_commits` and `commit_parents` rows. In the example, this run stores `Q`, `P` and `M`. ### How to reproduce The script below builds the history above in a temporary git repository (git only, no network). It runs the same git commands as `GitcliCloner.shallowClone()` + `deepen()` for two consecutive incremental runs. Then it applies the `CollectCommits()` rule ("skip if the first parent is not in the local repo") to show what would be stored. <details> <summary>repro.sh</summary> ```bash #!/usr/bin/env bash # Synthetic, self-contained reproduction of the gitextractor incremental-clone gap. # Replays GitcliCloner.shallowClone()+deepen() (plugins/gitextractor/parser/clone_gitcli.go) # and applies the Libgit2RepoCollector.CollectCommits() "no parent commit" rule, for two # consecutive incremental runs. set -euo pipefail work="$(mktemp -d)" trap 'rm -rf "${work}"' EXIT cd "${work}" commit() { # commit <date> <message> [parents...] local date="$1" msg="$2"; shift 2 local args=(); for p in "$@"; do args+=(-p "$p"); done GIT_AUTHOR_DATE="${date}" GIT_COMMITTER_DATE="${date}" \ git -C origin.git commit-tree "${tree}" "${args[@]}" -m "${msg}" } # incremental_run <since> <label>: one gitextractor incremental run into a fresh dir. incremental_run() { local since="$1" label="$2" dir="${work}/$2.git" git clone -q --depth=1 --bare "file://${work}/origin.git" "${dir}" printf '\tfetch = +refs/heads/*:refs/remotes/origin/*\n' >>"${dir}/config" git -C "${dir}" fetch -q --depth=1 origin git -C "${dir}" fetch -q --shallow-since="${since}" git -C "${dir}" repack -q -d git -C "${dir}" fetch -q --deepen=1 echo "${label} (since=${since}):" for c in $(git -C "${dir}" rev-list --all); do local subject first_parent subject="$(git -C "${dir}" log -1 --format=%s "$c" | cut -d: -f1)" first_parent="$(git -C origin.git rev-list --parents -n1 "$c" | cut -s -d' ' -f2)" if [[ -n "${first_parent}" ]] && ! git -C "${dir}" cat-file -e "${first_parent}" 2>/dev/null; then echo " ${subject}: SKIPPED (first parent not in local ODB) -> not stored" else echo " ${subject}: stored" fi done } git init -q --bare -b main origin.git tree="$(git -C origin.git hash-object -t tree -w --stdin </dev/null)" # Run 0 (previous incremental run) started at 2024-01-02T12:00:00Z; R and F were collected then. R=$(commit "2024-01-01T00:00:00Z" "R: base on main") F=$(commit "2024-01-02T10:00:00Z" "F: feature branch head, pushed before run 0" "$R") # After run 0: two changes land on main. Q=$(commit "2024-01-02T13:00:00Z" "Q: main commit after run 0" "$R") P=$(commit "2024-01-02T14:00:00Z" "P: main commit after run 0 (deployed)" "$Q") M=$(commit "2024-01-02T16:00:00Z" "M: merge request merges feature into main" "$P" "$F") git -C origin.git update-ref refs/heads/main "$M" git -C origin.git update-ref refs/heads/feature "$F" echo "git $(git --version | cut -d' ' -f3)" echo "expected new commits for run 1: Q, P, M" incremental_run "2024-01-02T12:00:00Z" run1 # Run 2 (next day): since = start of run 1; a new commit lands on main. N=$(commit "2024-01-03T09:00:00Z" "N: next main commit" "$M") git -C origin.git update-ref refs/heads/main "$N" incremental_run "2024-01-02T18:00:00Z" run2 echo "=> Q and P were never stored by any run." ``` </details> Output (the same with git 2.39.5, the version in the `apache/devlake:v1.0.3-beta15` image, and with git 2.56.0): ``` expected new commits for run 1: Q, P, M run1 (since=2024-01-02T12:00:00Z): M: stored P: SKIPPED (first parent not in local ODB) -> not stored F: stored R: stored run2 (since=2024-01-02T18:00:00Z): N: stored M: SKIPPED (first parent not in local ODB) -> not stored F: stored R: stored => Q and P were never stored by any run. ``` In run 2, `M` is skipped too, but that is harmless because `M` was stored in run 1. This is the case #7720 was designed for. The script replays DevLake's git commands and skip rule; we have not run this exact history through a fresh DevLake instance. Our production data matches its prediction: the commits the script says are lost are the ones missing from our database. To reproduce end to end: 1. Push `R`, create branch `feature` with `F`, and run a blueprint once. 2. Land `Q` and then `P` on the default branch, and merge `feature` with a merge commit (`M`). 3. Run the blueprint again (incremental). 4. Check `commits` / `repo_commits`: `Q` and `P` should be missing, while `commit_parents` has `(M, P)`. ### Anything else - **Frequency:** every incremental run where the trigger condition happens. The gaps pile up silently. - **Workaround:** a full sync. Per #8818, "Full Refresh" collects the missing commits. - **Proposed fix (I plan to open a PR):** after `--deepen=1`, keep running `git fetch --deepen=1` while any shallow boundary commit is still newer than `since`, with an iteration cap. When the loop ends, every boundary commit is older than `since`, so it was stored by an earlier run, and the existing skip rule becomes harmless again. The first full sync with `timeAfter` uses the same path and benefits too. In the script above this stores `Q`, `P` and `M` in run 1, with both git versions. Already-lost commits still need one full sync after upgrading. - **Environment:** `USE_GO_GIT_IN_GIT_EXTRACTOR=false` (libgit2), `SkipCommitStat=false`, `SkipCommitFiles=true`, `NoShallowClone=false`. - The fetch sequence and the skip rule are unchanged in `v1.0.3-beta18` and on `main` (`e355317df`). ### Version v1.0.3-beta15 ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
