comphead opened a new pull request, #5974: URL: https://github.com/apache/datafusion-comet/pull/5974
## Which issue does this PR close? Closes #5966. That issue asks for two things: a way to run the heavy suites locally, and a way for agents to run local checks before a merge. This PR covers the first. See "Not covered" at the bottom for the second. ## Rationale for this change The Spark SQL and Iceberg suites do not run on an unlabeled pull request. Spark 4.1 and Iceberg 1.11 first report in the merge queue, where a failure evicts the pull request and stalls everyone else's merges. Spark 3.5, Spark 4.0 and Iceberg 1.8/1.9/1.10 first report in the nightly run, after the change has already landed. Measured from queue run [35022044196](https://github.com/apache/datafusion-comet/actions/runs/35022044196), those two workflows are 522 of the 891 runner-minutes a queue pipeline costs: | Workflow | Runner-minutes | Jobs | Slowest job | | --------------------- | -------------- | ---- | -------------------------------- | | Spark SQL Tests (4.1) | 314 | 8 | `sql_hive-1` 57 min | | Iceberg 1.11 | 208 | 8 | `extensions` 55 min (unsharded) | The commands to run these locally are already documented in `docs/source/contributor-guide/spark-sql-tests.md` and `iceberg-spark-tests.md`, but nothing executes them, so every local invocation is retyped by hand. The parts that are easy to get wrong are exactly the parts that decide whether a local pass means anything: the `-l`/`-n` tag splits between the `sql_core` and `sql_hive` shards, the per-shard `HEAP_SIZE` and `METASPACE_SIZE` caps, the Iceberg shard init script and its index/count properties, the `DEDICATED_JVM_SBT_TESTS` gate, and the Parquet cache purge both workflows depend on. ## What changes are included in this PR? One script, `dev/local-ci.sh`, that builds the same sandbox a runner builds and then runs either workflow: ```sh dev/local-ci.sh spark # every Spark SQL matrix row dev/local-ci.sh spark sql_core-1 # one row, or all/core/hive dev/local-ci.sh iceberg # 4 shards + extensions + runtime dev/local-ci.sh iceberg shard-2 # one target ``` Prepare runs first (`cargo build --profile ci`, stage the library where `-Prelease` looks for it, clone and patch, `mvnw install -Prelease -DskipTests`, and for Spark the `catalyst`/`sql`/`hive` `Test/compile`), then the tests. `SKIP_PREPARE=1` goes straight to the tests when the sandbox is already current. Clones live under `$COMET_LOCAL_CI_HOME`, default `~/comet-local-ci`, outside the repository because `apache-spark/` is not in `.gitignore`. Nothing version-shaped or row-shaped is restated in the script. It is read from the existing sources at run time: | What | Read from | | ---------------------------------------- | ------------------------------------------------ | | Default version (the queue-tier one) | `POLICY` in `dev/ci/compute-changes.py` | | Spark version, JDK, Iceberg Spark/Scala | the `with:` blocks of `.github/workflows/ci.yml` | | Spark SQL matrix rows, heap caps | `dev/ci/spark-sql-modules.py` | | Iceberg shard count | `dev/ci/check-iceberg-shards.py` | | `DEDICATED_JVM_SBT_TESTS` and its gate | `.github/workflows/spark_sql_test_reusable.yml` | So a version bump, a new matrix row or a change to the shard count needs no edit here. Older versions are still reachable for reproducing a nightly failure, with `dev/local-ci.sh spark 3.5 sql_core-1`. Docs: a "Reproducing a suite failure locally" rewrite in `contributor-guide/ci.md`, and a pointer at the top of the Spark SQL and Iceberg test guides. The manual steps in those guides stay as the reference, since they are also the diff-regeneration workflow. ## How are these changes tested? No new CI job. This is a developer script, and `dev/ci/compute-changes.py` confirms a PR touching only `dev/local-ci.sh` and `docs/**` triggers nothing beyond preflight, which is intended. Checked locally: - `shellcheck` and `bash -n` clean. Written for bash 3.2, which is what `/bin/bash` is on macOS. - `dev/ci/check-ci-config.py`, `dev/ci/check-suites.py`, `prettier --check "**/*.md"` and `apache-rat:check` all pass. - Every argument shape resolves correctly: default and explicit versions for both suites, version versus target disambiguation, all 7 Spark rows and the `all`/`core`/`hive` groups against the counts `spark-sql-modules.py` reports, all 4 Iceberg shards plus `extensions` and `runtime`, and the `DEDICATED_JVM_SBT_TESTS` gate firing on 4.0 and not 4.1. - The emitted sbt and Gradle command lines were diffed against the workflow steps they mirror. What is **not** verified end to end is a full green suite run. I could not complete one: a Spark sandbox needs more free disk than the machine I wrote this on has, and the environment could not reach Maven Central to finish the Comet install. The clone, patch and sbt compile path is therefore verified by construction and by inspection of the emitted commands, not by execution. Worth a second pair of eyes on an actual run before this is relied on. ## Not covered The second half of #5966, making agents run these checks before a merge, is not in this PR. The tooling is the prerequisite for it. A follow-up could add guidance to `AGENTS.md` or a skill under `.ai/skills/`. Happy to fold that in here instead if reviewers prefer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
