erikbogado-nstech commented on issue #5966: URL: https://github.com/apache/datafusion-comet/issues/5966#issuecomment-5769686568
Two contributions, both measured in a digest-pinned container. **Pre-commit (opt-in, ~1-5 s):** the seven `dev/ci` checks (<0.3 s each, offline), `cargo fmt --all -- --check`, RAT/prettier when present, scalafix *syntactic* only. Installed by `dev/install-hooks.sh`. **Pre-push (opt-in, aggressive — 5-15 min warm):** runs the deterministic CI layers in the pinned image, selecting by changed paths — config checks; `cargo check --locked --profile ci` + `cargo test -p datafusion-comet`; `./mvnw -B -Pspark-4.1 -pl spark -Dsuites=<selected> test`, or the merged `dev/local-ci.sh`. Each prevented red push saves the ~175-min PR-tier run. Measured (28 vCPU / 125 GiB host, image `comet-ci-gate:2026-09-19`): config <0.3 s each; `cargo check` 6m32s; `cargo test` 9.1 s run; JVM compile 2m26s; one suite 5m13s; full CI suite set in four JVMs 51m54s sequential (~20 min parallel; each JVM capped at 7 vCPU / 12 GB). A 64 GB / 28 vCPU container was used for the resource-constraint check. Container traps found while doing this: glibc must match between the native build and the test JVM (else `GLIBC_2.38 not found` and a silent fallback); `--pids-limit=-1` (container default 2048 causes `pthread_create EAGAIN`); clean `jacoco.exec`; mount `.git`; mount cargo/sbt caches; set `SPARK_HOME`. Four separate JVMs are the faithful CI shape — the combined all-suite single JVM aborts nondeterministically (#6096). Recommendation: pre-commit on by default; pre-push opt-in and documented. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
