klesh opened a new pull request, #9293: URL: https://github.com/apache/devlake/pull/9293
### Summary Replace the two git-library based collectors in `gitextractor` — libgit2 (git2go) and go-git — with a single collector that runs the `git` command line. This removes the last `import "C"` in the backend, so the server now builds with `CGO_ENABLED=0`. Cross-compiling the single binary no longer needs a C toolchain, libgit2, libssh2 or per-architecture sysroots, and the CI/devcontainer images no longer have to build a pinned libgit2 1.5. The clone path already used the git CLI (auth, proxy, private keys, shallow deepening, partial clones). This PR moves the read path over as well, so the plugin has one way of talking to git. ### What changed 1. **Tests first** — replaced the go-git based fixtures with deterministic repositories built by the git CLI and added a collector contract suite (counts, refs, commit metadata, parents, stats, commit files, components, exclusions, `SkipCommitStat`, `SkipCommitFiles`, shallow boundaries, empty repo, cancellation, `CollectAll`/`Close`, snapshot vs `git blame`, line changes). The suite recorded the deviations of the two old implementations before they were removed. 2. **New collector** — `GitcliRepoCollector` streams `rev-list`/`cat-file --batch`/`log --numstat` in bounded batches (1,024 commits), rebuilds the snapshot from the first-parent patch stream, and honours context cancellation. Added unit tests for the `cat-file`, numstat and unified-diff parsers. 3. **Removal** — deleted `repo_libgit2.go`, `repo_gogit.go`, the git2go sort adapter, the `useGoGit` option/flag, `install-libgit2.sh`, the libgit2 build/copy steps in the Dockerfiles and devcontainer, the CI install steps and the dependabot ignore. Dropped `go-git` and `git2go` from `go.mod`/`go.sum`. ### Behaviour changes - Only commits reachable from refs are collected (dangling ODB objects left by force-pushes are ignored). - Shallow boundary commits are always skipped. libgit2 reported 0 parents for them and stored a full-tree diff; go-git skipped them only when the parent object was missing. The new collector reads `$(git rev-parse --git-path shallow)` and skips them deterministically, including when the parent object is present. - `commit_file_components` rows are only written when the repository has components configured. Nothing in DevLake populates the `components` table, so the previous per-file "Default" rows were pure write amplification (67,520 rows for the 8,383 commits of `apache/devlake`). - **git >= 2.31 is required** (for `--diff-merges=first-parent`) and is checked at startup with a clear error. The DB schema is unchanged. ### Performance `apache/devlake`, 8,383 reachable commits, full stats and commit files: | | git CLI | libgit2 | |---|---|---| | `CollectCommits` | **2.9 s** | 21.3 s | | `CollectDiffLine` | 3.4 s (559k snapshot rows, 2.7M line changes) | — | ### Tests - `go test ./plugins/gitextractor/...` - `go vet ./plugins/gitextractor/... ./plugins ./plugins/schema_e2e/... ./plugins/all/...` - `CGO_ENABLED=0 go build ./server/... ./plugins/all/...` - `go build -buildmode=plugin ./plugins/gitextractor` - `go mod tidy` is clean The contract suite includes a `git blame` oracle for the snapshot, a shallow-boundary test whose parent object is still present (it fails if the shallow-file guard is disabled), and parser unit tests for binary/empty/unicode/newline paths and C-quoted patch paths. ### Does this close any open issues? N/A ### Screenshots N/A ### Other Information - Full CI will run on this PR; nothing was skipped locally. - The website PR for the docs is separate: apache/devlake-website#(to be linked). - Follow-up (not in this PR): drop the now-unused `components`/`commit_file_components` tables if nothing populates them externally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
