This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-5871-e16461e0c6b79fd5da3599e17c19a293673b3136 in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 460f0fcaac7f0fd03579454046fc4f21b4d94aef Author: Andy Grove <[email protected]> AuthorDate: Sat Sep 12 21:00:11 2026 +0000 ci: move the Spark 4.1 sql_hive shards behind the merge queue (#5871) Keep catalyst and the three sql_core shards in the PR tier and run the three sql_hive shards only in the merge queue, or on a pull request that carries the new run-spark-4.1-hive-tests label. Over Aug 12 to Sep 11 the hive shards cost about 65 runner-hours a day on pull requests and were the only failing job on 7 PR runs, against 33 for the sql_core shards, and their 67-minute shard set the PR tier's wall clock at about 110 minutes. The reusable workflow gains a `modules` input (all, core or hive). The matrix rows move from a workflow literal to dev/ci/spark-sql-modules.py, which the build job runs to publish the selected rows as a job output, because a job-level `if:` cannot see the matrix context. ci.yml folds two POLICY outputs, spark_4_1 and the new queue-only spark_4_1_hive, into that one input so the queue still gets every module from a single build. Part of #5870. --- .github/workflows/README.md | 47 +++++----- .github/workflows/ci.yml | 21 +++-- .github/workflows/spark_sql_test_reusable.yml | 32 ++++--- dev/ci/check-ci-config.py | 47 +++++++++- dev/ci/compute-changes.py | 17 +++- dev/ci/spark-sql-modules.py | 122 ++++++++++++++++++++++++++ docs/source/contributor-guide/ci.md | 22 +++-- 7 files changed, 259 insertions(+), 49 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 34765f74a3..de4717b00b 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -15,11 +15,11 @@ Merging goes through GitHub's merge queue, configured by the `Merge Queue` ruleset in `.asf.yaml`. That splits CI into two tiers: - **PR tier** (`pr`): fast feedback while a change is being iterated on. - The Linux build, Spark 4.1 and Iceberg 1.11. + The Linux build, Spark 4.1 (catalyst and `sql_core` only) and Iceberg 1.11. - **Queue tier** (`queue`): the authoritative gate. Everything the PR tier - runs, plus the macOS build, the benchmark compile check, Spark 3.4/3.5/4.0 - and Iceberg 1.8/1.9/1.10, evaluated against the merge result rather than - against the PR head. + runs, plus the macOS build, the benchmark compile check, the Spark 4.1 + `sql_hive` shards, Spark 3.4/3.5/4.0 and Iceberg 1.8/1.9/1.10, evaluated + against the merge result rather than against the PR head. Every queue-only job has a `run-*` label that opts a pull request into it early, listed in the diagram below. @@ -59,8 +59,9 @@ and TPC-H/TPC-DS caches on `main` stale until the next unrelated change. PR + queue tier push to main only queue tier, or PR with label --------------- ----------------- --------------------------- pr_build_linux (+ push, for cache) docs pr_build_macos run-macos-tests - spark_4_1 pr_benchmark_check run-benchmark-check - iceberg_1_11 spark_3_4 run-spark-3.4-tests + spark_4_1 (catalyst + sql_core) pr_benchmark_check run-benchmark-check + iceberg_1_11 spark_4_1 sql_hive run-spark-4.1-hive-tests + spark_3_4 run-spark-3.4-tests spark_3_5 run-spark-3.5-tests spark_4_0 run-spark-4.0-tests iceberg_1_8 run-iceberg-tests @@ -85,23 +86,23 @@ and TPC-H/TPC-DS caches on `main` stale until the next unrelated change. ## What runs when -| Job in `ci.yml` | Triggered by | Routing rule | -| -------------------- | ------------------------------------------------- | ----------------------------------- | -| `preflight` | every PR / merge group / push / dispatch / label | none (always runs) | -| `changes` | every PR / merge group / push / dispatch / label | runs `dev/ci/compute-changes.py` | -| `pr_build_linux` | PR, merge group or push to main, paths matched | `dev/ci/compute-changes.py` | -| `pr_build_macos` | merge group, **or** PR with `run-macos-tests` | `dev/ci/compute-changes.py` | -| `pr_benchmark_check` | merge group, **or** PR with `run-benchmark-check` | benchmark sources only | -| `docs` | push to main, paths matched | `.asf.yaml`, `docs/**`, `docs.yaml` | -| `spark_3_5` | merge group, **or** PR with `run-spark-3.5-tests` | Spark 3.5 sources | -| `spark_4_1` | PR or merge group, paths matched | Spark 4.1 sources | -| `spark_3_4` | merge group, **or** PR with `run-spark-3.4-tests` | Spark 3.4 sources | -| `spark_4_0` | merge group, **or** PR with `run-spark-4.0-tests` | Spark 4.0 sources | -| `iceberg_1_11` | PR or merge group, paths matched | Iceberg sources | -| `iceberg_1_8` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources | -| `iceberg_1_9` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources | -| `iceberg_1_10` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources | -| `required_checks` | always, after every job above except `docs` | none (always runs) | +| Job in `ci.yml` | Triggered by | Routing rule | +| -------------------- | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | +| `preflight` | every PR / merge group / push / dispatch / label | none (always runs) | +| `changes` | every PR / merge group / push / dispatch / label | runs `dev/ci/compute-changes.py` | +| `pr_build_linux` | PR, merge group or push to main, paths matched | `dev/ci/compute-changes.py` | +| `pr_build_macos` | merge group, **or** PR with `run-macos-tests` | `dev/ci/compute-changes.py` | +| `pr_benchmark_check` | merge group, **or** PR with `run-benchmark-check` | benchmark sources only | +| `docs` | push to main, paths matched | `.asf.yaml`, `docs/**`, `docs.yaml` | +| `spark_3_5` | merge group, **or** PR with `run-spark-3.5-tests` | Spark 3.5 sources | +| `spark_4_1` | PR or merge group, paths matched; the `sql_hive` shards only in the merge group **or** with `run-spark-4.1-hive-tests` | Spark 4.1 sources | +| `spark_3_4` | merge group, **or** PR with `run-spark-3.4-tests` | Spark 3.4 sources | +| `spark_4_0` | merge group, **or** PR with `run-spark-4.0-tests` | Spark 4.0 sources | +| `iceberg_1_11` | PR or merge group, paths matched | Iceberg sources | +| `iceberg_1_8` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources | +| `iceberg_1_9` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources | +| `iceberg_1_10` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources | +| `required_checks` | always, after every job above except `docs` | none (always runs) | A heavy job appears in the PR's checks list as a `skipped` entry whenever its path filter or event criteria don't match. Skipped checks count as diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9754b89d4d..bd3a0aa99c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -22,11 +22,12 @@ # Merging goes through GitHub's merge queue (see `rulesets` in `.asf.yaml`), so # there are two tiers: # -# pull_request fast feedback. Linux build + Spark 4.1 + Iceberg 1.11. +# pull_request fast feedback. Linux build + Spark 4.1 (catalyst and +# sql_core only) + Iceberg 1.11. # merge_group the authoritative gate. Everything the PR tier runs, plus -# the macOS build, the benchmark compile check, Spark -# 3.4/3.5/4.0 and Iceberg 1.8/1.9/1.10, evaluated against -# the merge result rather than the PR head. +# the macOS build, the benchmark compile check, the Spark 4.1 +# sql_hive shards, Spark 3.4/3.5/4.0 and Iceberg 1.8/1.9/1.10, +# evaluated against the merge result rather than the PR head. # # Which tier a job sits in is POLICY in dev/ci/compute-changes.py, not an # expression here. Heavy jobs deliberately have no `push` tier: the queue @@ -160,6 +161,7 @@ jobs: spark_3_5: ${{ steps.compute.outputs.spark_3_5 }} spark_4_0: ${{ steps.compute.outputs.spark_4_0 }} spark_4_1: ${{ steps.compute.outputs.spark_4_1 }} + spark_4_1_hive: ${{ steps.compute.outputs.spark_4_1_hive }} iceberg_1_8: ${{ steps.compute.outputs.iceberg_1_8 }} iceberg_1_9: ${{ steps.compute.outputs.iceberg_1_9 }} iceberg_1_10: ${{ steps.compute.outputs.iceberg_1_10 }} @@ -288,12 +290,21 @@ jobs: spark_4_1: name: Spark SQL Tests (Spark 4.1) needs: changes - if: needs.changes.outputs.spark_4_1 == 'true' + # Two POLICY outputs feed one call, so the queue gets every module from a + # single 40-minute build instead of two. `spark_4_1` (PR tier) covers + # catalyst and the sql_core shards; `spark_4_1_hive` (queue-only, or the + # `run-spark-4.1-hive-tests` label) adds the sql_hive shards. A `labeled` + # run sets only the second, and then runs only the hive rows. + if: needs.changes.outputs.spark_4_1 == 'true' || needs.changes.outputs.spark_4_1_hive == 'true' uses: ./.github/workflows/spark_sql_test_reusable.yml with: spark-short: '4.1' spark-full: '4.1.3' java: 17 + modules: >- + ${{ needs.changes.outputs.spark_4_1 != 'true' && 'hive' + || needs.changes.outputs.spark_4_1_hive != 'true' && 'core' + || 'all' }} iceberg_1_8: name: Iceberg Spark SQL Tests (Iceberg 1.8) diff --git a/.github/workflows/spark_sql_test_reusable.yml b/.github/workflows/spark_sql_test_reusable.yml index 2790855332..9fa254f9b6 100644 --- a/.github/workflows/spark_sql_test_reusable.yml +++ b/.github/workflows/spark_sql_test_reusable.yml @@ -41,6 +41,17 @@ on: required: false default: false type: boolean + modules: + # Which matrix rows to run. `core` is catalyst plus the three sql_core + # shards, `hive` is the three sql_hive shards, `all` is both. The + # umbrella uses this to keep the Spark 4.1 hive shards out of the PR + # tier without a second call (and a second 40-minute build) for the + # queue: one call, and the rows the tier does not want are left out + # of the matrix by dev/ci/spark-sql-modules.py. + description: 'Which test modules to run: all, core (catalyst + sql_core) or hive (sql_hive)' + required: false + default: 'all' + type: string env: RUST_VERSION: stable @@ -71,9 +82,15 @@ jobs: runs-on: ubuntu-24.04 container: image: amd64/rust + outputs: + module-matrix: ${{ steps.modules.outputs.matrix }} steps: - uses: actions/checkout@v7 + - name: Select test modules + id: modules + run: python3 dev/ci/spark-sql-modules.py --modules "${{ inputs.modules }}" --github-output "$GITHUB_OUTPUT" + - name: Setup Rust & Java toolchain uses: ./.github/actions/setup-builder with: @@ -173,16 +190,11 @@ jobs: spark-sql-test: needs: build strategy: - matrix: - module: - - {name: "catalyst", args1: "catalyst/test", args2: ""} - # sql_core-* set HEAP_SIZE / METASPACE_SIZE so SparkBuild.scala caps - - {name: "sql_core-1", args1: "", args2: "sql/testOnly * -- -l org.apache.spark.tags.ExtendedSQLTest -l org.apache.spark.tags.SlowSQLTest", heap: "3g", metaspace: "1g"} - - {name: "sql_core-2", args1: "", args2: "sql/testOnly * -- -n org.apache.spark.tags.ExtendedSQLTest", heap: "3g", metaspace: "1g"} - - {name: "sql_core-3", args1: "", args2: "sql/testOnly * -- -n org.apache.spark.tags.SlowSQLTest", heap: "3g", metaspace: "1g"} - - {name: "sql_hive-1", args1: "", args2: "hive/testOnly * -- -l org.apache.spark.tags.ExtendedHiveTest -l org.apache.spark.tags.SlowHiveTest"} - - {name: "sql_hive-2", args1: "", args2: "hive/testOnly * -- -n org.apache.spark.tags.ExtendedHiveTest"} - - {name: "sql_hive-3", args1: "", args2: "hive/testOnly * -- -n org.apache.spark.tags.SlowHiveTest"} + # The rows are defined in dev/ci/spark-sql-modules.py and selected by + # the `modules` input; the build job runs the script and publishes the + # result. A job-level `if:` cannot read `matrix`, so a row the caller + # does not want has to be left out of the matrix rather than skipped. + matrix: ${{ fromJSON(needs.build.outputs.module-matrix) }} fail-fast: false name: spark-sql-${{ matrix.module.name }}/spark-${{ inputs.spark-full }}-jdk${{ inputs.java }} runs-on: ubuntu-24.04 diff --git a/dev/ci/check-ci-config.py b/dev/ci/check-ci-config.py index 0d30910f29..8e8a2f0ab9 100644 --- a/dev/ci/check-ci-config.py +++ b/dev/ci/check-ci-config.py @@ -86,6 +86,7 @@ BUILD_JOBS = { "spark_3_5", "spark_4_0", "spark_4_1", + "spark_4_1_hive", "iceberg_1_8", "iceberg_1_9", "iceberg_1_10", @@ -113,7 +114,7 @@ ROUTING_CASES = [ # derived from POLICY, so that a change to the routing has to be stated twice # and cannot be made by accident. PR_TIER = {"build_linux", "spark_4_1", "iceberg_1_11"} -SPARK_OPT_IN = {"spark_3_4", "spark_3_5", "spark_4_0"} +SPARK_OPT_IN = {"spark_3_4", "spark_3_5", "spark_4_0", "spark_4_1_hive"} ICEBERG_OPT_IN = {"iceberg_1_8", "iceberg_1_9", "iceberg_1_10"} BUILD_OPT_IN = {"build_macos", "benchmark"} QUEUE_TIER = PR_TIER | SPARK_OPT_IN | ICEBERG_OPT_IN | BUILD_OPT_IN @@ -157,6 +158,21 @@ POLICY_CASES = [ }, {"build_macos"}, ), + # The Spark 4.1 hive shards are queue-only with their own label. The label + # adds them to the PR tier's Spark 4.1 call rather than starting a second. + ( + {"name": "pull_request", "action": "synchronize", "labels": ["run-spark-4.1-hive-tests"]}, + PR_TIER | {"spark_4_1_hive"}, + ), + ( + { + "name": "pull_request", + "action": "labeled", + "label": "run-spark-4.1-hive-tests", + "labels": ["run-spark-4.1-hive-tests"], + }, + {"spark_4_1_hive"}, + ), # An opt-in label present on a pushed commit adds just that suite. ( {"name": "pull_request", "action": "synchronize", "labels": ["run-spark-3.4-tests"]}, @@ -229,6 +245,34 @@ def load_filters(): return module +def check_spark_sql_modules(): + """`--modules core` and `--modules hive` must partition `--modules all`. + + ci.yml maps its two Spark 4.1 POLICY outputs onto these three values, so a + row that lands in no group, or in both, would either never run or run + twice in the queue, and nothing else would notice. + """ + spec = importlib.util.spec_from_file_location("spark_sql_modules", "dev/ci/spark-sql-modules.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + failures = [] + names = lambda rows: [row["name"] for row in rows] + everything = names(module.select("all")) + core, hive = names(module.select("core")), names(module.select("hive")) + if not core or not hive: + failures.append("a module group is empty (see MODULES in dev/ci/spark-sql-modules.py)") + if sorted(core + hive) != sorted(everything): + failures.append( + f"core {core} + hive {hive} does not partition all {everything} " + f"(see MODULES in dev/ci/spark-sql-modules.py)" + ) + if len(set(everything)) != len(everything): + failures.append(f"duplicate module names in {everything}") + for failure in failures: + print(f"spark sql modules: {failure}") + return not failures + + def check_change_filters(): module = load_filters() failures = [] @@ -545,6 +589,7 @@ def check_required_checks(): if __name__ == "__main__": ok = check_change_filters() ok = check_event_policy() and ok + ok = check_spark_sql_modules() and ok ok = check_artifact_names() and ok ok = check_local_actions_have_checkout() and ok ok = check_required_checks() and ok diff --git a/dev/ci/compute-changes.py b/dev/ci/compute-changes.py index 9716ced234..40d1797023 100644 --- a/dev/ci/compute-changes.py +++ b/dev/ci/compute-changes.py @@ -125,6 +125,7 @@ FILTERS = { "rust-toolchain.toml", ".github/workflows/ci.yml", ".github/workflows/spark_sql_test_reusable.yml", + "dev/ci/spark-sql-modules.py", ".github/actions/setup-builder/**", ".github/actions/setup-spark-builder/**", ".github/actions/upload-artifact-retry/**", @@ -151,6 +152,7 @@ FILTERS = { "rust-toolchain.toml", ".github/workflows/ci.yml", ".github/workflows/spark_sql_test_reusable.yml", + "dev/ci/spark-sql-modules.py", ".github/actions/setup-builder/**", ".github/actions/setup-spark-builder/**", ".github/actions/upload-artifact-retry/**", @@ -177,6 +179,7 @@ FILTERS = { "rust-toolchain.toml", ".github/workflows/ci.yml", ".github/workflows/spark_sql_test_reusable.yml", + "dev/ci/spark-sql-modules.py", ".github/actions/setup-builder/**", ".github/actions/setup-spark-builder/**", ".github/actions/upload-artifact-retry/**", @@ -203,6 +206,7 @@ FILTERS = { "rust-toolchain.toml", ".github/workflows/ci.yml", ".github/workflows/spark_sql_test_reusable.yml", + "dev/ci/spark-sql-modules.py", ".github/actions/setup-builder/**", ".github/actions/setup-spark-builder/**", ".github/actions/upload-artifact-retry/**", @@ -210,6 +214,11 @@ FILTERS = { ".mvn/**", "mvnw", ], + # Same inputs as spark_4_1: this is not a separate job but a second + # POLICY decision for the same call, selecting the sql_hive matrix rows. + # ci.yml folds the two outputs into the reusable workflow's `modules` + # input. Populated below, after the dict, so the two lists cannot drift. + "spark_4_1_hive": [], "iceberg_1_8": [ "native/**/src/**", "native/**/Cargo.toml", @@ -307,6 +316,7 @@ FILTERS = { "mvnw", ], } +FILTERS["spark_4_1_hive"] = FILTERS["spark_4_1"] # Which events may run each job, independent of the path filters above. # @@ -349,8 +359,13 @@ POLICY = { "spark_3_5": ["queue", "label:run-spark-3.5-tests"], "spark_4_0": ["queue", "label:run-spark-4.0-tests"], # Spark 4.1 is the default build profile, so it is the cheapest early - # warning that a change is wrong and stays in the PR tier. + # warning that a change is wrong and stays in the PR tier. Only the + # catalyst and sql_core shards, though: over Aug 12 to Sep 11 2026 the + # three sql_hive shards cost about 65 runner-hours a day on pull requests + # and were the only failing job on 7 PR runs, against 33 for sql_core, and + # their 67-minute shard set the PR tier's wall clock. See issue #5870. "spark_4_1": ["pr", "queue"], + "spark_4_1_hive": ["queue", "label:run-spark-4.1-hive-tests"], "iceberg_1_8": ["queue", "label:run-iceberg-tests"], "iceberg_1_9": ["queue", "label:run-iceberg-tests"], "iceberg_1_10": ["queue", "label:run-iceberg-tests"], diff --git a/dev/ci/spark-sql-modules.py b/dev/ci/spark-sql-modules.py new file mode 100644 index 0000000000..f31ac25c48 --- /dev/null +++ b/dev/ci/spark-sql-modules.py @@ -0,0 +1,122 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# The test matrix for .github/workflows/spark_sql_test_reusable.yml. +# +# The rows used to be a literal `strategy.matrix.module` list in the workflow. +# They live here so that a caller can ask for a subset: the umbrella keeps the +# Spark 4.1 `sql_hive` shards out of the PR tier (see POLICY in +# dev/ci/compute-changes.py and issue #5870) and a job-level `if:` cannot see +# `matrix`, so the selection has to happen before the matrix is expanded. The +# `build` job runs this script and publishes the result as a job output that +# `spark-sql-test` reads with `fromJSON`, the same way the Iceberg reusable +# workflow sizes its shards. +# +# Usage: +# spark-sql-modules.py --modules all|core|hive --github-output $GITHUB_OUTPUT +# spark-sql-modules.py --modules core (prints the matrix JSON) + +import argparse +import json +import sys +from pathlib import Path + +# `group` is what --modules selects on. `heap` and `metaspace` are read by the +# "Run Spark tests" step as per-row forked-test-JVM caps for SparkBuild.scala; +# the sql_core rows set them because those shards were the ones hitting the +# 7 GB runner budget. +MODULES = [ + {"name": "catalyst", "group": "core", "args1": "catalyst/test", "args2": ""}, + { + "name": "sql_core-1", + "group": "core", + "args1": "", + "args2": "sql/testOnly * -- -l org.apache.spark.tags.ExtendedSQLTest -l org.apache.spark.tags.SlowSQLTest", + "heap": "3g", + "metaspace": "1g", + }, + { + "name": "sql_core-2", + "group": "core", + "args1": "", + "args2": "sql/testOnly * -- -n org.apache.spark.tags.ExtendedSQLTest", + "heap": "3g", + "metaspace": "1g", + }, + { + "name": "sql_core-3", + "group": "core", + "args1": "", + "args2": "sql/testOnly * -- -n org.apache.spark.tags.SlowSQLTest", + "heap": "3g", + "metaspace": "1g", + }, + { + "name": "sql_hive-1", + "group": "hive", + "args1": "", + "args2": "hive/testOnly * -- -l org.apache.spark.tags.ExtendedHiveTest -l org.apache.spark.tags.SlowHiveTest", + }, + { + "name": "sql_hive-2", + "group": "hive", + "args1": "", + "args2": "hive/testOnly * -- -n org.apache.spark.tags.ExtendedHiveTest", + }, + { + "name": "sql_hive-3", + "group": "hive", + "args1": "", + "args2": "hive/testOnly * -- -n org.apache.spark.tags.SlowHiveTest", + }, +] + +GROUPS = ("all", "core", "hive") + + +def select(modules): + """Return the matrix rows for a --modules value, or raise ValueError.""" + if modules not in GROUPS: + raise ValueError(f"--modules must be one of {', '.join(GROUPS)}, got {modules!r}") + rows = [row for row in MODULES if modules == "all" or row["group"] == modules] + # Every row carries every key, so a `${{ matrix.module.heap }}` lookup on a + # row without a cap is an empty string rather than a template error. + keys = sorted({key for row in MODULES for key in row}) + return [{key: row.get(key, "") for key in keys} for row in rows] + + +def main(argv): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--modules", default="all", help="all, core (catalyst + sql_core) or hive (sql_hive)") + parser.add_argument("--github-output", type=Path, help="append matrix=<json> to this $GITHUB_OUTPUT file") + args = parser.parse_args(argv) + try: + rows = select(args.modules) + except ValueError as e: + print(f"error: {e}", file=sys.stderr) + return 2 + matrix = json.dumps({"module": rows}) + if args.github_output: + with args.github_output.open("a", encoding="utf-8") as out: + out.write(f"matrix={matrix}\n") + else: + print(matrix) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/docs/source/contributor-guide/ci.md b/docs/source/contributor-guide/ci.md index 9bcd275b38..e6adb1c746 100644 --- a/docs/source/contributor-guide/ci.md +++ b/docs/source/contributor-guide/ci.md @@ -35,7 +35,8 @@ Which jobs run also depends on the event: | Suite | Pull request | Merge queue | | ------------------------------------------------- | ------------ | ----------- | | Linux build, lint, and Comet test suites | yes | yes | -| Spark SQL tests, Spark 4.1 | yes | yes | +| Spark SQL tests, Spark 4.1, catalyst and sql_core | yes | yes | +| Spark SQL tests, Spark 4.1, sql_hive | with label | yes | | Iceberg Spark SQL tests, Iceberg 1.11 | yes | yes | | macOS build and Comet test suites | with label | yes | | Benchmark compile and lint check | with label | yes | @@ -62,14 +63,15 @@ cases that pin both down. Each queue-only suite has a label that runs it on a pull request: -| Label | Runs | -| --------------------- | ---------------------------------------------------- | -| `run-macos-tests` | macOS build and Comet test suites | -| `run-benchmark-check` | Benchmark compile and lint check | -| `run-spark-3.4-tests` | Spark SQL tests against Spark 3.4 | -| `run-spark-3.5-tests` | Spark SQL tests against Spark 3.5 | -| `run-spark-4.0-tests` | Spark SQL tests against Spark 4.0 | -| `run-iceberg-tests` | Iceberg Spark SQL tests against Iceberg 1.8/1.9/1.10 | +| Label | Runs | +| -------------------------- | ---------------------------------------------------- | +| `run-macos-tests` | macOS build and Comet test suites | +| `run-benchmark-check` | Benchmark compile and lint check | +| `run-spark-4.1-hive-tests` | Spark SQL hive tests against Spark 4.1 | +| `run-spark-3.4-tests` | Spark SQL tests against Spark 3.4 | +| `run-spark-3.5-tests` | Spark SQL tests against Spark 3.5 | +| `run-spark-4.0-tests` | Spark SQL tests against Spark 4.0 | +| `run-iceberg-tests` | Iceberg Spark SQL tests against Iceberg 1.8/1.9/1.10 | Apply a label from the pull request sidebar, or from the command line: @@ -92,6 +94,8 @@ does not cover. Some examples: - code under `spark/src/main/spark-3.4/`, `spark-3.5/`, `spark-4.0/` or the shared `spark-3.x/` directory, or any change to `CometExprShim` and friends - a change to a Spark SQL diff under `dev/diffs/` for a version other than 4.1 +- anything that touches Hive table support, `InsertIntoHiveTable`, or the `sql/hive` parts of + the 4.1 diff - anything touching Iceberg reflection or the Iceberg diffs - native code with platform-specific behavior, or a dependency bump that changes what is compiled on macOS --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
