This is an automated email from the ASF dual-hosted git repository.
zhouyuan pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 7b193da3fd [GLUTEN-6887][VL] Daily Update Velox Version (2026_10_23)
(#13418)
7b193da3fd is described below
commit 7b193da3fd462db56c0b66a11762d748d145269f
Author: Gluten Performance Bot
<[email protected]>
AuthorDate: Fri Oct 23 17:57:08 2026 +0100
[GLUTEN-6887][VL] Daily Update Velox Version (2026_10_23) (#13418)
* [GLUTEN-6887][VL] Daily Update Velox Version (dft-2026_10_23)
Upstream Velox's New Commits:
72ad30f8f by Maria Basmanova, fix(exec): Support broadcast in FixedPoint
body plans (#19753)
1afcf9079 by Pedro Pedreira, refactor(functions): Dispatch MakeRowFromMap
on the map encoding (#19674)
09bced8ca by GALI PREM SAGAR, feat(cudf): Run from_unixtime and to_unixtime
on the GPU
1831f3590 by Ke, fix(nimble): Free previous chunk before decoding the next
e1cefcc92 by Ke, perf(nimble): Skip value decoding for null filters (#19669)
8d6a4ad19 by Hongze Zhang, fix!: CountDistinctAggregate should override
initializeNewGroupsInternal
cbe4bec92 by inf / Hazmi, feat(iceberg): Add split offsets to commit
metadata
58d01dc99 by Mariam AlMesfer, feat(spark): Add support for
unix_timestamp/to_unix_timestamp(timestamp_utc)
f40aa0c9a by Wechar Yu, fix(spill): Delete spill inputs after successful
pre-merge
f0e1a32fd by Apurva Kumar, fix(iceberg): Omit unknown null counts from DWRF
data file metrics (#19738)
0068a37cb by GALI PREM SAGAR, fix(fuzzer): Keep nulls out of the k-RANGE
bound sent to Presto
07c29ee4e by oerling, feat(torchwave): Faster scatter_add and compiler
fixes (#19679)
b8982c7f6 by dependabot[bot], build(deps): Bump launch-editor from 2.14.1
to 2.14.2 in /website
7ff007df4 by Misiu Godfrey, fix(cudf): Preserve 64-bit offsets when
restoring VARBINARY format
7fbd57fde by Maria Basmanova, fix(exec)!: Fail when a convergence plan
produces no row
d499fb3df by Apurva Kumar, fix(iceberg): Commit a deletion vector under its
data file's partition spec
54c5d740b by tanjialiang, refactor(nimble): Let CachedDataInput run on a
cache other than AsyncDataCache (#19663)
cca087339 by Dmitry Spikhalsky, fix(exec): Clear FixedPointLoop next-row
link with an unaligned store
3f397aff6 by Dmitry Spikhalsky, fix(dwio): Use unaligned loads in
DecoderUtil bit helpers
22ac4a306 by Dmitry Spikhalsky, fix(functions): Use unaligned loads in
Murmur3 32-bit string hashes
4c900b305 by Natasha Sehgal, test(fuzzer): Update padding skip issue
references (#19647)
241ed544d by mohsaka, feat(type): Parse Hive "void" type as Velox UNKNOWN
d645ae590 by Xin Zhang, fix(docs): Fix broken TLSF paper link
34403efaa by Krishna Pai, build(ci): Make stash saves and restores best
effort
b4957001e by Jimmy Lu, fix(exec): Default-initialize NestedFixedPoint
members
b2a92df83 by Wechar Yu, fix: Spill writer should respect writeBufferSize
for large batches
3fcdb2b33 by Anshul Baliga, docs(sparksql): Document approx_percentile
968ec27fa by Deepak Majeti, fix(build): Prompt Boost build and fix
wget_opts on MacOS
f9327757a by peterenescu, feat(fuzzer): Add support for floating point edge
cases (#19649)
552ec51e2 by Zhenyuan Zhao, fix(nimble): Keep file read options on
small-file preload
8501e6f4a by Pedro Pedreira, feat(benchmark): Benchmark make_row_from_map
over flat maps
c57fd8592 by Wei He, fix(row): Pad fixed-width UnsafeRow arrays (#19668)
13d9d32e3 by Apurva Kumar, fix(iceberg): Count an UPDATE once in the MERGE
row count
51d70eda8 by fzxu, feat: Add flat map support for map_subset (#18629)
399f9d753 by Shruti Shivakumar, feat(arrow): Import 32-bit decimal arrays
e257582f4 by Pedrum Jalali, feat(spark): Implement native transform_keys
11c27c404 by dependabot[bot], build(deps): Bump follow-redirects from
1.16.0 to 1.16.1 in /website
6eeb70ac7 by shanhao-203, docs(nimble): Add Hybrid FlatMap blog post
(#19532)
447e432c9 by Maria Basmanova, feat(exec)!: Simplify fixed point worker
addressing (#19688)
4a4679f80 by Maria Basmanova, test(exec): Fix data race in fixed point test
cursors
c52ff97b4 by Ke, perf(nimble): Add bulkScan to ConstantEncoding (#19581)
3c9fd367a by Apurva Kumar, fix(iceberg): Report NIMBLE data files as NIMBLE
in Iceberg commit messages
42b5ef84a by Apurva Kumar, fix(iceberg): Keep partition data on MERGE
deletion vectors
0cba44b3c by Matt Karrmann, fix(exec): Avoid division by zero in
AdaptivePrefetch (#19614)
d0f64a95e by Xiaoxuan, feat(nimble): Add synchronous row access and point
seeks to index iterators
959199f65 by Ke, perf(nimble): Bulk-decode hooks and filter-only reads in
FBW, BBP, SubIntSplit (#19572)
0235167b2 by Maria Basmanova, feat(exec)!: Decide fixed point convergence
on worker 0 (#19622)
aa55e9e6e by PRASHANT GOLASH, feat(nimble): Simplify ALP-RD size estimation
and selection (#19497)
7fdce0438 by Maria Basmanova, fix(core)!: Check the plans inside a fixed
point (#19676)
Signed-off-by: glutenperfbot <[email protected]>
* optimize CI uage
Signed-off-by: Yuan <[email protected]>
---------
Signed-off-by: glutenperfbot <[email protected]>
Signed-off-by: Yuan <[email protected]>
Co-authored-by: glutenperfbot <[email protected]>
Co-authored-by: Yuan <[email protected]>
---
.../workflows/util/delta-spark-ut/flaky-tests.txt | 7 +
.github/workflows/util/parallel-ut/Discover.java | 52 +++
.github/workflows/util/parallel-ut/mvn-test.sh | 30 ++
.github/workflows/util/parallel-ut/record-fork.sh | 25 ++
.github/workflows/util/parallel-ut/run.py | 481 +++++++++++++++++++++
.github/workflows/velox_backend_enhanced.yml | 14 +-
.github/workflows/velox_backend_x86.yml | 73 ++--
ep/build-velox/src/get-velox.sh | 4 +-
pom.xml | 4 +
9 files changed, 653 insertions(+), 37 deletions(-)
diff --git a/.github/workflows/util/delta-spark-ut/flaky-tests.txt
b/.github/workflows/util/delta-spark-ut/flaky-tests.txt
index e06a65b2bd..ea2d2d79dc 100644
--- a/.github/workflows/util/delta-spark-ut/flaky-tests.txt
+++ b/.github/workflows/util/delta-spark-ut/flaky-tests.txt
@@ -20,3 +20,10 @@
# whack-a-mole), quarantine it by ERROR SIGNATURE in flaky-error-patterns.txt
# instead.
+# Timing-sensitive: the test gives an AvailableNow-triggered CDC stream 10s to
+# terminate (query.awaitTermination(10000L)). With 4 forked JVMs near the
runner's
+# memory limit, the batch can still be in Gluten's Parquet metadata validation
+# (ParquetMetadataUtils listing files via RawLocalFileSystem, which forks `ls`
per
+# file for permissions), so it times out. Failed once in 25 runs, on the
+# Velox-bump PR #13418 (run 39902045267), which changed no Java code.
+org.apache.spark.sql.delta.DeltaCDCStream*Suite#maxFilesPerTrigger with
Trigger.AvailableNow respects read limits
diff --git a/.github/workflows/util/parallel-ut/Discover.java
b/.github/workflows/util/parallel-ut/Discover.java
new file mode 100644
index 0000000000..82a0661509
--- /dev/null
+++ b/.github/workflows/util/parallel-ut/Discover.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.lang.annotation.Annotation;
+
+/**
+ * Lists the ScalaTest suites ScalaTest's Runner would discover on the given
runpath, one per line
+ * as {@code <suite>\t<comma-separated class annotations>}. Launched by run.py
in source-file mode
+ * with the recorded fork's classpath, so it works for both Scala 2.12 and
2.13.
+ */
+public class Discover {
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ public static void main(String[] args) throws Exception {
+ scala.collection.immutable.List runpath =
scala.collection.immutable.Nil$.MODULE$;
+ for (int i = args.length - 1; i >= 0; i--) {
+ runpath = new scala.collection.immutable.$colon$colon(args[i], runpath);
+ }
+ ClassLoader loader = Thread.currentThread().getContextClassLoader();
+ // The same (package-private in Scala, public in bytecode) helper Runner
uses for discovery.
+ scala.collection.Iterator suites =
+ org.scalatest.tools.SuiteDiscoveryHelper.discoverSuiteNames(
+ runpath, loader, scala.Option.empty())
+ .iterator();
+ StringBuilder out = new StringBuilder();
+ while (suites.hasNext()) {
+ String suite = (String) suites.next();
+ StringBuilder annotations = new StringBuilder();
+ for (Annotation annotation : Class.forName(suite, false,
loader).getAnnotations()) {
+ if (annotations.length() > 0) {
+ annotations.append(',');
+ }
+ annotations.append(annotation.annotationType().getName());
+ }
+ out.append(suite).append('\t').append(annotations).append('\n');
+ }
+ System.out.print(out);
+ }
+}
diff --git a/.github/workflows/util/parallel-ut/mvn-test.sh
b/.github/workflows/util/parallel-ut/mvn-test.sh
new file mode 100755
index 0000000000..e16c64670e
--- /dev/null
+++ b/.github/workflows/util/parallel-ut/mvn-test.sh
@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Usage: mvn-test.sh <maven test command...>
+#
+# Runs the Maven command with every ScalaTest fork recorded instead of run
(Maven still
+# compiles and runs the JUnit tests), then runs the recorded suites in
${UT_PARALLELISM:-3}
+# concurrent JVMs with run.py. Extra run.py options can be passed in
$PARALLEL_UT_OPTS.
+set -euo pipefail
+here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+GLUTEN_UT_FORK_DIR=$(mktemp -d "${TMPDIR:-/tmp}/gluten-ut-forks-XXXXXX")
+export GLUTEN_UT_FORK_DIR
+"$@" -Dscalatest.jvm="$here/record-fork.sh"
+# CentOS 8 images without a python3 module still ship platform-python.
+python=$(command -v python3 || echo /usr/libexec/platform-python)
+# shellcheck disable=SC2086
+"$python" "$here/run.py" --jobs "${UT_PARALLELISM:-3}" ${PARALLEL_UT_OPTS:-}
diff --git a/.github/workflows/util/parallel-ut/record-fork.sh
b/.github/workflows/util/parallel-ut/record-fork.sh
new file mode 100755
index 0000000000..539f8a12b7
--- /dev/null
+++ b/.github/workflows/util/parallel-ut/record-fork.sh
@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Stand-in for the java executable of scalatest-maven-plugin
(-Dscalatest.jvm=<this file>).
+# Instead of running the suites, it saves the fork's working directory,
environment and
+# arguments under $GLUTEN_UT_FORK_DIR so run.py can replay them across
parallel JVMs.
+set -euo pipefail
+mkdir -p "${GLUTEN_UT_FORK_DIR:?GLUTEN_UT_FORK_DIR must be set}"
+out=$(mktemp -d "$GLUTEN_UT_FORK_DIR/fork-XXXXXX")
+pwd > "$out/cwd"
+env -0 > "$out/env"
+printf '%s\0' "$@" > "$out/args"
diff --git a/.github/workflows/util/parallel-ut/run.py
b/.github/workflows/util/parallel-ut/run.py
new file mode 100644
index 0000000000..5ae55ff634
--- /dev/null
+++ b/.github/workflows/util/parallel-ut/run.py
@@ -0,0 +1,481 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Replays recorded ScalaTest forks across parallel JVMs.
+
+The Spark UT jobs run `mvn test` with -Dscalatest.jvm=record-fork.sh, so every
+module's ScalaTest fork is saved (working directory, environment and arguments,
+which already include the module's classpath, tags, -DwildcardSuites and
+argLine) instead of being run serially inside Maven. This script discovers the
+suites each fork would run, splits them into chunks and runs the chunks in
+--jobs concurrent JVMs. Chunks of one module reuse its recorded command with
the
+suite selection replaced by -s, so the result matches the serial Maven run.
+
+Many suites depend on JVM-wide state left by the suites before them, e.g.
Gluten's
+once-per-JVM static initialization or a leftover default SparkSession. So a
chunk
+is a contiguous slice of the order the serial run uses, and the span of suites
+matching a --sticky-pattern, plus the suite before it, is never split. Suites
+matching a --colocate-pattern write the same fixed paths, so they share a
chunk.
+
+Every JVM keeps the module basedir as working directory, which some suites
+resolve paths against, and gets its own java.io.tmpdir, derby.system.home and
+gluten.test.dir so concurrent suites do not share scratch space. Chunk logs go
+to <module>/target/parallel-ut-<chunk>.log; JUnit XML reports go where Maven
+would have written them.
+"""
+
+import argparse
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+import threading
+import time
+from concurrent.futures import ThreadPoolExecutor
+
+RUNNER = "org.scalatest.tools.Runner"
+DISCOVER = os.path.join(os.path.dirname(os.path.abspath(__file__)),
"Discover.java")
+# Runner options that take a value. Reporter options (-f, -h, -u, -C) may carry
+# glued configuration letters, e.g. -fW <file>.
+VALUE_OPTS = set("-R -l -n -s -w -m -z -t -q -j -b -M -A -F -T -y -i".split())
+REPORTER_OPTS = ("-f", "-h", "-u", "-C")
+FLAG_OPTS = ("-o", "-e", "-P")
+SELECTION_OPTS = {"-s", "-w", "-m"}
+# Selections the replay cannot split; such forks run unchanged in one JVM.
+UNSPLITTABLE_OPTS = {"-z", "-t", "-i", "-j", "-b", "-A"}
+TESTS_LINE = re.compile(
+ r"Tests: succeeded (\d+), failed (\d+), canceled (\d+), "
+ r"ignored (\d+), pending (\d+)"
+)
+FAILURE_LINE = re.compile(r"\*\*\* (FAILED|ABORTED|RUN ABORTED) \*\*\*")
+
+print_lock = threading.Lock()
+
+
+def log(message):
+ with print_lock:
+ print(message, flush=True)
+
+
+class Fork(object):
+ """One recorded scalatest-maven-plugin fork, i.e. one Maven module."""
+
+ def __init__(self, path):
+ with open(os.path.join(path, "cwd")) as f:
+ self.cwd = f.read().strip()
+ with open(os.path.join(path, "env"), "rb") as f:
+ entries = [e.decode() for e in f.read().split(b"\0") if e]
+ self.env = dict(e.split("=", 1) for e in entries if "=" in e)
+ with open(os.path.join(path, "args"), "rb") as f:
+ args = [a.decode() for a in f.read().split(b"\0")[:-1]]
+ split = args.index(RUNNER)
+ self.jvm_args = args[:split]
+ self.options = parse_options(args[split + 1 :])
+ self.module = os.path.relpath(self.cwd, os.getcwd())
+
+ def values(self, name):
+ return [value for opt, value in self.options if opt == name]
+
+ def splittable(self):
+ return not any(opt in UNSPLITTABLE_OPTS for opt, _ in self.options)
+
+ def java(self):
+ java_home = self.env.get("JAVA_HOME")
+ return os.path.join(java_home, "bin", "java") if java_home else "java"
+
+
+def parse_options(args):
+ options = []
+ i = 0
+ while i < len(args):
+ opt = args[i]
+ if opt in VALUE_OPTS or opt.startswith(REPORTER_OPTS):
+ if i + 1 >= len(args):
+ raise ValueError("Missing value for ScalaTest option " + opt)
+ options.append((opt, args[i + 1]))
+ i += 2
+ elif opt.startswith(FLAG_OPTS):
+ options.append((opt, None))
+ i += 1
+ else:
+ raise ValueError("Unsupported ScalaTest option " + opt)
+ return options
+
+
+def runpath(fork):
+ entries = []
+ for value in fork.values("-R"):
+ entries += [
+ e.replace("\\ ", " ") for e in re.split(r"(?<!\\) +",
value.strip()) if e
+ ]
+ return entries
+
+
+def discover(fork):
+ """Returns [(suite, weight)] for the suites the recorded fork would run,
in the
+ order the serial run uses."""
+ paths = runpath(fork)
+ out = subprocess.run(
+ [fork.java(), DISCOVER] + paths,
+ cwd=fork.cwd,
+ env=fork.env,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ universal_newlines=True,
+ )
+ if out.returncode != 0:
+ raise RuntimeError(
+ "Suite discovery failed for {}:\n{}".format(fork.module,
out.stderr)
+ )
+ explicit = fork.values("-s")
+ wildcards = fork.values("-w")
+ members = fork.values("-m")
+ include = set(t for v in fork.values("-n") for t in v.split())
+ exclude = set(t for v in fork.values("-l") for t in v.split())
+ discovered = []
+ for line in out.stdout.splitlines():
+ name, _, annotations = line.partition("\t")
+ discovered.append((name, set(annotations.split(",")) if annotations
else set()))
+ # ScalaTest's Runner runs -s suites first, then one DiscoverySuite per -w
and per
+ # -m in argument order, each in discovery order
(DiscoverySuite.wildcardList /
+ # narrowList). A suite matched twice runs once here.
+ if explicit or wildcards or members:
+ ordered = list(explicit)
+ for w in wildcards:
+ ordered += [name for name, _ in discovered if name.startswith(w)]
+ for m in members:
+ ordered += [
+ name
+ for name, _ in discovered
+ if name.startswith(m + ".") and "." not in name[len(m) + 1 :]
+ ]
+ else:
+ ordered = [name for name, _ in discovered]
+ tags_of = dict(discovered)
+ suites, seen = [], set()
+ for name in ordered:
+ if name in seen:
+ continue
+ seen.add(name)
+ weight = class_size(paths, name)
+ # Class-level tags decide most suites up front. Those that run nothing
still
+ # start, so keep a small weight rather than dropping them.
+ tags = tags_of.get(name, set())
+ if (include and not tags & include) or tags & exclude:
+ weight = max(1, weight // 50)
+ suites.append((name, weight))
+ return suites
+
+
+def class_size(paths, suite):
+ """Bytecode size of the suite and its nested classes, a proxy for its
runtime."""
+ relative = suite.replace(".", "/")
+ size = 0
+ for path in paths:
+ directory = os.path.join(path, os.path.dirname(relative))
+ if not os.path.isdir(directory):
+ continue
+ base = os.path.basename(relative)
+ for f in os.listdir(directory):
+ if f == base + ".class" or (
+ f.startswith(base + "$") and f.endswith(".class")
+ ):
+ size += os.path.getsize(os.path.join(directory, f))
+ return max(size, 1)
+
+
+class Chunk(object):
+ def __init__(self, fork, suites):
+ self.fork = fork
+ self.suites = [s for s, _ in suites]
+ self.weight = sum(w for _, w in suites)
+ self.id = None
+ self.returncode = None
+ self.seconds = 0
+ self.log = None
+
+
+def make_chunks(fork, suites, target, sticky_patterns, colocate_patterns):
+ """Cuts the serially ordered suites into contiguous chunks of about
`target`
+ weight.
+
+ For each sticky pattern, the span from the suite before its first match to
its
+ last match stays in one chunk, so those suites see the same predecessors
as in
+ the serial run. Suites matching one colocate pattern share fixed paths but
not
+ state: they move next to the first of them, so they run one after another.
+ """
+ if not fork.splittable():
+ return [Chunk(fork, suites)]
+ for pattern in colocate_patterns:
+ members = [s for s in suites if pattern.search(s[0])]
+ if len(members) > 1:
+ first = suites.index(members[0])
+ rest = [s for s in suites if s not in members]
+ suites = rest[:first] + members + rest[first:]
+ # no_cut[i]: suite i must stay in the same chunk as suite i - 1.
+ no_cut = [False] * len(suites)
+ for pattern in sticky_patterns + colocate_patterns:
+ hits = [i for i, s in enumerate(suites) if pattern.search(s[0])]
+ if hits:
+ for i in range(hits[0], hits[-1] + 1):
+ no_cut[i] = True
+ chunks, current = [], []
+ for i, suite in enumerate(suites):
+ if current and not no_cut[i] and sum(w for _, w in current) >= target:
+ chunks.append(Chunk(fork, current))
+ current = []
+ current.append(suite)
+ if current:
+ chunks.append(Chunk(fork, current))
+ return chunks
+
+
+def command(chunk, work, heap, jvm_opts):
+ fork = chunk.fork
+ jvm_args = list(fork.jvm_args)
+ if heap and not any(a.startswith("-Xmx") for a in jvm_args):
+ jvm_args.append("-Xmx" + heap)
+ jvm_args += jvm_opts.split()
+ jvm_args += [
+ "-Djava.io.tmpdir=" + os.path.join(work, "tmp"),
+ "-Dderby.system.home=" + os.path.join(work, "derby"),
+ "-Dgluten.test.dir=" + os.path.join(work, "gluten"),
+ ]
+ runner_args = []
+ for opt, value in fork.options:
+ if fork.splittable() and opt in SELECTION_OPTS:
+ continue
+ if opt.startswith("-f") and value:
+ # File reporters would overwrite each other; give every chunk its
own file.
+ root, ext = os.path.splitext(value)
+ value = "{}-{}{}".format(root, chunk.id, ext)
+ runner_args.append(opt)
+ if value is not None:
+ runner_args.append(value)
+ if fork.splittable():
+ for suite in chunk.suites:
+ runner_args += ["-s", suite]
+ return [fork.java()] + jvm_args + [RUNNER] + runner_args
+
+
+def run_chunk(chunk, args, total):
+ fork = chunk.fork
+ work = tempfile.mkdtemp(prefix="chunk-{}-".format(chunk.id),
dir=args.work_dir)
+ for sub in ("tmp", "derby", "gluten"):
+ os.makedirs(os.path.join(work, sub))
+ target = os.path.join(fork.cwd, "target")
+ os.makedirs(target, exist_ok=True)
+ chunk.log = os.path.join(target, "parallel-ut-{}.log".format(chunk.id))
+ cmd = command(chunk, work, args.heap, args.jvm_opts)
+ start = time.time()
+ with open(chunk.log, "w") as out:
+ out.write("# " + " ".join(cmd) + "\n")
+ out.flush()
+ try:
+ chunk.returncode = subprocess.run(
+ cmd,
+ cwd=fork.cwd,
+ env=fork.env,
+ stdout=out,
+ stderr=subprocess.STDOUT,
+ timeout=args.timeout or None,
+ ).returncode
+ except subprocess.TimeoutExpired:
+ out.write(
+ "\n*** RUN ABORTED *** chunk timed out after
{}s\n".format(args.timeout)
+ )
+ chunk.returncode = -1
+ chunk.seconds = time.time() - start
+ with open(chunk.log, errors="replace") as f:
+ text = f.read()
+ chunk.passed = (
+ chunk.returncode == 0
+ and "Run completed in" in text
+ and "*** RUN ABORTED ***" not in text
+ )
+ # Keep a failed chunk's scratch directory: some suites write the actual
output
+ # they compared (e.g. plan stability's *.actual.*.txt) to java.io.tmpdir.
+ if chunk.passed:
+ shutil.rmtree(work, ignore_errors=True)
+ else:
+ text += "\nScratch directory kept for inspection: {}\n".format(work)
+ with print_lock:
+ print(
+ "::group::[{}] chunk {}/{} {} ({} suites, {:.0f}s)".format(
+ "PASS" if chunk.passed else "FAIL",
+ chunk.id,
+ total,
+ fork.module,
+ len(chunk.suites),
+ chunk.seconds,
+ )
+ )
+ print(text)
+ print("::endgroup::", flush=True)
+ chunk.counts = [
+ sum(int(m.group(i)) for m in TESTS_LINE.finditer(text)) for i in
range(1, 6)
+ ]
+ chunk.failures = [
+ line.strip() for line in text.splitlines() if FAILURE_LINE.search(line)
+ ]
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter
+ )
+ parser.add_argument(
+ "--forks-dir",
+ default=os.environ.get("GLUTEN_UT_FORK_DIR"),
+ help="Directory record-fork.sh wrote to (default:
$GLUTEN_UT_FORK_DIR)",
+ )
+ parser.add_argument("--jobs", type=int, default=3, help="Concurrent test
JVMs")
+ parser.add_argument(
+ "--chunks-per-job",
+ type=int,
+ default=6,
+ help="Chunks per JVM slot; more balances better, fewer starts fewer
JVMs",
+ )
+ parser.add_argument(
+ "--heap",
+ default="3g",
+ help="-Xmx for each test JVM unless the fork already sets one",
+ )
+ parser.add_argument(
+ "--jvm-opts",
+ # Fewer GC threads per JVM, since the JVMs share few cores. Do not add
+ # -XX:TieredStopAtLevel=1: C1-only code made suites several times
slower and
+ # its larger frames overflowed the stack in Spark's variant tests.
+ default="-XX:ParallelGCThreads=2 -XX:ConcGCThreads=1",
+ help="Extra options for each test JVM",
+ )
+ parser.add_argument(
+ "--sticky-pattern",
+ action="append",
+ help="Regex of suites that must run after the same suites as in the
serial "
+ "run (repeatable). Default: Hive suites, which share the TestHive
session, "
+ "its added jars and a warehouse; and VeloxTransitionSuite, which
initializes "
+ "Gluten from a mock executor context and breaks a later real driver
start "
+ "unless a Gluten session already ran in the JVM",
+ )
+ parser.add_argument(
+ "--colocate-pattern",
+ action="append",
+ help="Regex of suites that write the same fixed paths and so must not
run "
+ "concurrently (repeatable). Default: the Velox TPC-H Iceberg suites,
which "
+ "share test-classes/tpch-data-iceberg-velox",
+ )
+ parser.add_argument(
+ "--timeout", type=int, default=0, help="Per-chunk timeout in seconds"
+ )
+ parser.add_argument(
+ "--work-dir", default=None, help="Parent of per-JVM scratch
directories"
+ )
+ args = parser.parse_args()
+ if not args.forks_dir or not os.path.isdir(args.forks_dir):
+ sys.exit(
+ "No recorded forks in {!r}; was Maven run with
-Dscalatest.jvm=record-fork.sh?".format(
+ args.forks_dir
+ )
+ )
+ args.work_dir = args.work_dir or
tempfile.mkdtemp(prefix="gluten-parallel-ut-")
+ os.makedirs(args.work_dir, exist_ok=True)
+
+ forks = sorted(
+ (Fork(os.path.join(args.forks_dir, d)) for d in
os.listdir(args.forks_dir)),
+ key=lambda fork: fork.cwd,
+ )
+ start = time.time()
+ with ThreadPoolExecutor(max_workers=args.jobs) as pool:
+ discovered = list(pool.map(discover, forks))
+ total_weight = sum(w for suites in discovered for _, w in suites)
+ target = max(1, total_weight // max(1, args.jobs * args.chunks_per_job))
+ sticky_patterns = [
+ re.compile(p)
+ for p in args.sticky_pattern
+ or [r"^org\.apache\.spark\.sql\.hive\.", r"\.VeloxTransitionSuite$"]
+ ]
+ colocate_patterns = [
+ re.compile(p)
+ for p in args.colocate_pattern
+ or [r"\.Velox(PartitionedTable)?TPCHIcebergSuite$"]
+ ]
+ chunks = []
+ for fork, suites in zip(forks, discovered):
+ log("{}: {} suites".format(fork.module, len(suites)))
+ if suites:
+ chunks += make_chunks(
+ fork, suites, target, sticky_patterns, colocate_patterns
+ )
+ # Heaviest first, so the long chunks do not end up as the tail.
+ chunks.sort(key=lambda chunk: -chunk.weight)
+ for i, chunk in enumerate(chunks, 1):
+ chunk.id = i
+ log(
+ "Discovered {} suites in {} modules in {:.0f}s; running {} chunks in
{} JVMs".format(
+ sum(len(c.suites) for c in chunks),
+ len(forks),
+ time.time() - start,
+ len(chunks),
+ args.jobs,
+ )
+ )
+ with ThreadPoolExecutor(max_workers=args.jobs) as pool:
+ for future in [
+ pool.submit(run_chunk, chunk, args, len(chunks)) for chunk in
chunks
+ ]:
+ future.result()
+
+ failed = [c for c in chunks if not c.passed]
+ counts = [sum(c.counts[i] for c in chunks) for i in range(5)]
+ print("\nParallel ScalaTest summary ({:.0f}s):".format(time.time() -
start))
+ for chunk in sorted(chunks, key=lambda c: c.id):
+ print(
+ " [{}] chunk {:3d} {:6.0f}s {:4d} suites {} {}".format(
+ "PASS" if chunk.passed else "FAIL",
+ chunk.id,
+ chunk.seconds,
+ len(chunk.suites),
+ chunk.fork.module,
+ os.path.relpath(chunk.log),
+ )
+ )
+ print(
+ "Tests: succeeded {}, failed {}, canceled {}, ignored {}, pending
{}".format(
+ *counts
+ )
+ )
+ if failed:
+ print("\n{} of {} chunks failed:".format(len(failed), len(chunks)))
+ for chunk in failed:
+ print(
+ " chunk {} ({}, exit code {}), log: {}".format(
+ chunk.id,
+ chunk.fork.module,
+ chunk.returncode,
+ os.path.relpath(chunk.log),
+ )
+ )
+ for line in chunk.failures or ["(no failed test reported; see the
log)"]:
+ print(" " + line)
+ sys.exit(1)
+ print("All chunks passed.")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/.github/workflows/velox_backend_enhanced.yml
b/.github/workflows/velox_backend_enhanced.yml
index a38e890066..6520430bf8 100644
--- a/.github/workflows/velox_backend_enhanced.yml
+++ b/.github/workflows/velox_backend_enhanced.yml
@@ -19,6 +19,7 @@ on:
pull_request:
paths:
- '.github/workflows/velox_backend_enhanced.yml'
+ - '.github/workflows/util/parallel-ut/**'
- 'pom.xml'
- 'backends-velox/**'
- 'gluten-uniffle/**'
@@ -45,6 +46,13 @@ env:
CCACHE_DIR: "${{ github.workspace }}/.ccache"
# spark.sql.ansi.enabled defaults to false.
SPARK_ANSI_SQL_MODE: false
+ # The spark-test-* jobs prefix their `mvn test` with this wrapper: Maven
compiles and runs
+ # the JUnit tests, then the ScalaTest suites run in UT_PARALLELISM
concurrent JVMs instead
+ # of one after another inside Maven. See
.github/workflows/util/parallel-ut/run.py.
+ PARALLEL_UT: '.github/workflows/util/parallel-ut/mvn-test.sh'
+ # The runners have 4 vCPUs and a single test JVM already keeps several busy:
with 4
+ # JVMs each suite ran ~5x slower than serially, so total throughput dropped.
+ UT_PARALLELISM: 3
concurrency:
group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{
github.workflow }}
@@ -160,7 +168,7 @@ jobs:
java -version
export SPARK_HOME=/opt/shims/spark35/spark_home/
ls -l $SPARK_HOME
- $MVN_CMD clean test -Pspark-3.5 -Pjava-17 -Pbackends-velox -Piceberg
-Pdelta -Phudi \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-3.5 -Pjava-17
-Pbackends-velox -Piceberg -Pdelta -Phudi \
-DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowHiveTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.CudfTest,org.apache.gluten.tags.SkipTest
\
-DargLine="-Dspark.test.home=$SPARK_HOME"
- name: Upload test report
@@ -207,10 +215,10 @@ jobs:
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
export PATH=$JAVA_HOME/bin:$PATH
java -version
- $MVN_CMD clean test -Pspark-3.5 -Pjava-17 -Pbackends-velox -Piceberg
-Pdelta -Phudi -Ppaimon -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-3.5 -Pjava-17
-Pbackends-velox -Piceberg -Pdelta -Phudi -Ppaimon -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/" \
-DtagsToInclude=org.apache.spark.tags.ExtendedSQLTest
- $MVN_CMD clean test -Pspark-3.5 -Pjava-17 -Pbackends-velox -Piceberg
-Pdelta -Phudi -Ppaimon -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-3.5 -Pjava-17
-Pbackends-velox -Piceberg -Pdelta -Phudi -Ppaimon -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/" \
-DtagsToInclude=org.apache.spark.tags.SlowHiveTest
- name: Upload test report
diff --git a/.github/workflows/velox_backend_x86.yml
b/.github/workflows/velox_backend_x86.yml
index 48dc7d7a71..40ca515c0c 100644
--- a/.github/workflows/velox_backend_x86.yml
+++ b/.github/workflows/velox_backend_x86.yml
@@ -19,6 +19,7 @@ on:
pull_request:
paths:
- '.github/workflows/velox_backend_x86.yml'
+ - '.github/workflows/util/parallel-ut/**'
- 'pom.xml'
- 'backends-velox/**'
- 'gluten-uniffle/**'
@@ -57,6 +58,13 @@ env:
CCACHE_DIR: "${{ github.workspace }}/.ccache"
# spark.sql.ansi.enabled defaults to false.
SPARK_ANSI_SQL_MODE: false
+ # The spark-test-* jobs prefix their `mvn test` with this wrapper: Maven
compiles and runs
+ # the JUnit tests, then the ScalaTest suites run in UT_PARALLELISM
concurrent JVMs instead
+ # of one after another inside Maven. See
.github/workflows/util/parallel-ut/run.py.
+ PARALLEL_UT: '.github/workflows/util/parallel-ut/mvn-test.sh'
+ # The runners have 4 vCPUs and a single test JVM already keeps several busy:
with 4
+ # JVMs each suite ran ~5x slower than serially, so total throughput dropped.
+ UT_PARALLELISM: 3
concurrency:
group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{
github.workflow }}
@@ -769,15 +777,15 @@ jobs:
export SPARK_HOME=/opt/shims/spark34/spark_home/
TAGS_EXCLUDE="org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowHiveTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.EnhancedFeaturesTest,org.apache.gluten.tags.CudfTest,org.apache.gluten.tags.SkipTest"
if [ "${{ matrix.group }}" = "1" ]; then
- $MVN_CMD clean test -Pspark-3.4 -Pjava-17 -Pbackends-velox
-Piceberg -Piceberg-test -Pdelta -Phudi -Ppaimon -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-3.4 -Pjava-17
-Pbackends-velox -Piceberg -Piceberg-test -Pdelta -Phudi -Ppaimon -Pspark-ut \
-DtagsToExclude="$TAGS_EXCLUDE"
-DargLine="-Dspark.test.home=$SPARK_HOME" \
-DwildcardSuites="org.apache.spark.sql.streaming,org.apache.spark.GlutenSortShuffleSuite,org.apache.gluten"
elif [ "${{ matrix.group }}" = "2" ]; then
- $MVN_CMD clean test -Pspark-3.4 -Pjava-17 -Pbackends-velox
-Piceberg -Piceberg-test -Pdelta -Phudi -Ppaimon -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-3.4 -Pjava-17
-Pbackends-velox -Piceberg -Piceberg-test -Pdelta -Phudi -Ppaimon -Pspark-ut \
-DtagsToExclude="$TAGS_EXCLUDE"
-DargLine="-Dspark.test.home=$SPARK_HOME" \
-DwildcardSuites="org.apache.spark.sql.execution,org.apache.spark.sql.catalyst,org.apache.spark.sql.errors,org.apache.spark.sql.extension"
else
- $MVN_CMD clean test -Pspark-3.4 -Pjava-17 -Pbackends-velox
-Piceberg -Piceberg-test -Pdelta -Phudi -Ppaimon -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-3.4 -Pjava-17
-Pbackends-velox -Piceberg -Piceberg-test -Pdelta -Phudi -Ppaimon -Pspark-ut \
-DtagsToExclude="$TAGS_EXCLUDE"
-DargLine="-Dspark.test.home=$SPARK_HOME" \
-DwildcardSuites="org.apache.spark.sql.GlutenSQL,org.apache.spark.sql.Gluten,org.apache.spark.sql.connector,org.apache.spark.sql.sources,org.apache.spark.sql.hive,org.apache.spark.sql.gluten,org.apache.spark.sql.shim"
fi
@@ -831,11 +839,11 @@ jobs:
java -version
export SPARK_HOME=/opt/shims/spark34/spark_home/
if [ "${{ matrix.suite }}" = "extended" ]; then
- $MVN_CMD clean test -Pspark-3.4 -Pjava-17 -Pbackends-velox
-Piceberg -Pdelta -Phudi -Ppaimon -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-3.4 -Pjava-17
-Pbackends-velox -Piceberg -Pdelta -Phudi -Ppaimon -Pspark-ut \
-DtagsToInclude=org.apache.spark.tags.ExtendedSQLTest \
-DargLine="-Dspark.test.home=$SPARK_HOME"
else
- $MVN_CMD clean test -Pspark-3.4 -Pjava-17 -Pbackends-velox
-Piceberg -Pdelta -Phudi -Ppaimon -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-3.4 -Pjava-17
-Pbackends-velox -Piceberg -Pdelta -Phudi -Ppaimon -Pspark-ut \
-DtagsToInclude=org.apache.spark.tags.SlowHiveTest \
-DargLine="-Dspark.test.home=$SPARK_HOME"
fi
@@ -893,15 +901,15 @@ jobs:
java -version
TAGS_EXCLUDE="org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowHiveTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.EnhancedFeaturesTest,org.apache.gluten.tags.CudfTest,org.apache.gluten.tags.SkipTest"
if [ "${{ matrix.group }}" = "1" ]; then
- $MVN_CMD clean test -Pspark-3.5 -Pjava-17 -Pbackends-velox
-Piceberg -Pdelta -Phudi -Ppaimon -Plance -Plance-test -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-3.5 -Pjava-17
-Pbackends-velox -Piceberg -Pdelta -Phudi -Ppaimon -Plance -Plance-test
-Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
-DwildcardSuites="org.apache.spark.sql.streaming,org.apache.spark.GlutenSortShuffleSuite,org.apache.gluten"
elif [ "${{ matrix.group }}" = "2" ]; then
- $MVN_CMD clean test -Pspark-3.5 -Pjava-17 -Pbackends-velox
-Piceberg -Pdelta -Phudi -Ppaimon -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-3.5 -Pjava-17
-Pbackends-velox -Piceberg -Pdelta -Phudi -Ppaimon -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
-DwildcardSuites="org.apache.spark.sql.execution,org.apache.spark.sql.catalyst,org.apache.spark.sql.errors,org.apache.spark.sql.extension"
else
- $MVN_CMD clean test -Pspark-3.5 -Pjava-17 -Pbackends-velox
-Piceberg -Pdelta -Phudi -Ppaimon -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-3.5 -Pjava-17
-Pbackends-velox -Piceberg -Pdelta -Phudi -Ppaimon -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
-DwildcardSuites="org.apache.spark.sql.GlutenSQL,org.apache.spark.sql.Gluten,org.apache.spark.sql.connector,org.apache.spark.sql.sources,org.apache.spark.sql.hive,org.apache.spark.sql.gluten,org.apache.spark.sql.shim"
fi
@@ -929,10 +937,11 @@ jobs:
spark-test-spark35-scala213:
needs: [detect-changes, build-native-lib-centos-8]
- if: >-
- needs.detect-changes.outputs.java == 'true' ||
- needs.detect-changes.outputs.shims35 == 'true' ||
- needs.detect-changes.outputs.cpp == 'true'
+ # Nightly only: the spark40/41/42 jobs already build and test with Scala
2.13 on
+ # every PR, so per-PR this Spark 3.5 + Scala 2.13 combo adds ~90
runner-minutes for
+ # little PR-specific signal. The nightly full_run (or a workflow_dispatch
run)
+ # still covers it daily.
+ if: needs.detect-changes.outputs.full_run == 'true'
runs-on: ubuntu-22.04
strategy:
fail-fast: false
@@ -965,15 +974,15 @@ jobs:
java -version
TAGS_EXCLUDE="org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowHiveTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.EnhancedFeaturesTest,org.apache.gluten.tags.CudfTest,org.apache.gluten.tags.SkipTest"
if [ "${{ matrix.group }}" = "1" ]; then
- $MVN_CMD clean test -Pspark-3.5 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Piceberg -Pdelta -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-3.5 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Piceberg -Pdelta -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark35-scala-2.13/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
-DwildcardSuites="org.apache.spark.sql.streaming,org.apache.spark.GlutenSortShuffleSuite,org.apache.gluten"
elif [ "${{ matrix.group }}" = "2" ]; then
- $MVN_CMD clean test -Pspark-3.5 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Piceberg -Pdelta -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-3.5 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Piceberg -Pdelta -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark35-scala-2.13/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
-DwildcardSuites="org.apache.spark.sql.execution,org.apache.spark.sql.catalyst,org.apache.spark.sql.errors,org.apache.spark.sql.extension"
else
- $MVN_CMD clean test -Pspark-3.5 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Piceberg -Pdelta -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-3.5 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Piceberg -Pdelta -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark35-scala-2.13/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
-DwildcardSuites="org.apache.spark.sql.GlutenSQL,org.apache.spark.sql.Gluten,org.apache.spark.sql.connector,org.apache.spark.sql.sources,org.apache.spark.sql.hive,org.apache.spark.sql.gluten,org.apache.spark.sql.shim"
fi
@@ -1022,11 +1031,11 @@ jobs:
export PATH=$JAVA_HOME/bin:$PATH
java -version
if [ "${{ matrix.suite }}" = "extended" ]; then
- $MVN_CMD clean test -Pspark-3.5 -Pjava-17 -Pbackends-velox
-Piceberg -Pdelta -Phudi -Ppaimon -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-3.5 -Pjava-17
-Pbackends-velox -Piceberg -Pdelta -Phudi -Ppaimon -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/" \
-DtagsToInclude=org.apache.spark.tags.ExtendedSQLTest
else
- $MVN_CMD clean test -Pspark-3.5 -Pjava-17 -Pbackends-velox
-Piceberg -Pdelta -Phudi -Ppaimon -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-3.5 -Pjava-17
-Pbackends-velox -Piceberg -Pdelta -Phudi -Ppaimon -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/" \
-DtagsToInclude=org.apache.spark.tags.SlowHiveTest
fi
@@ -1299,15 +1308,15 @@ jobs:
java -version
TAGS_EXCLUDE="org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowHiveTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.EnhancedFeaturesTest,org.apache.gluten.tags.CudfTest,org.apache.gluten.tags.SkipTest"
if [ "${{ matrix.group }}" = "1" ]; then
- $MVN_CMD clean test -Pspark-4.0 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pdelta -Ppaimon -Plance -Plance-test -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.0 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pdelta -Ppaimon -Plance -Plance-test -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark40/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
-DwildcardSuites="org.apache.spark.sql.streaming,org.apache.spark.GlutenSortShuffleSuite,org.apache.gluten"
elif [ "${{ matrix.group }}" = "2" ]; then
- $MVN_CMD clean test -Pspark-4.0 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pdelta -Ppaimon -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.0 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pdelta -Ppaimon -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark40/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
-DwildcardSuites="org.apache.spark.sql.execution,org.apache.spark.sql.catalyst,org.apache.spark.sql.errors,org.apache.spark.sql.extension"
else
- $MVN_CMD clean test -Pspark-4.0 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pdelta -Ppaimon -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.0 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pdelta -Ppaimon -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark40/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
-DwildcardSuites="org.apache.spark.sql.GlutenSQL,org.apache.spark.sql.Gluten,org.apache.spark.sql.connector,org.apache.spark.sql.sources,org.apache.spark.sql.hive,org.apache.spark.sql.gluten,org.apache.spark.sql.shim"
fi
@@ -1356,11 +1365,11 @@ jobs:
export PATH=$JAVA_HOME/bin:$PATH
java -version
if [ "${{ matrix.suite }}" = "extended" ]; then
- $MVN_CMD clean test -Pspark-4.0 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pdelta -Ppaimon -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.0 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pdelta -Ppaimon -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark40/spark_home/" \
-DtagsToInclude=org.apache.spark.tags.ExtendedSQLTest
else
- $MVN_CMD clean test -Pspark-4.0 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pdelta -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.0 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pdelta -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark40/spark_home/" \
-DtagsToInclude=org.apache.spark.tags.SlowHiveTest
fi
@@ -1425,17 +1434,17 @@ jobs:
TAGS_EXCLUDE="org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowHiveTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.EnhancedFeaturesTest,org.apache.gluten.tags.CudfTest,org.apache.gluten.tags.SkipTest"
if [ "${{ matrix.group }}" = "1" ]; then
# Group 1: streaming + gluten utils + top-level spark tests (~55
classes)
- $MVN_CMD clean test -Pspark-4.1 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pspark-ut -Pdelta \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.1 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pspark-ut -Pdelta \
-DargLine="-Dspark.test.home=/opt/shims/spark41/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
-DwildcardSuites="org.apache.spark.sql.streaming,org.apache.spark.GlutenSortShuffleSuite,org.apache.gluten"
elif [ "${{ matrix.group }}" = "2" ]; then
# Group 2: execution + catalyst + errors + extension tests (~140
classes)
- $MVN_CMD clean test -Pspark-4.1 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pspark-ut -Pdelta \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.1 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pspark-ut -Pdelta \
-DargLine="-Dspark.test.home=/opt/shims/spark41/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
-DwildcardSuites="org.apache.spark.sql.execution,org.apache.spark.sql.catalyst,org.apache.spark.sql.errors,org.apache.spark.sql.extension"
else
# Group 3: top-level sql, connector, sources, hive and remaining
tests (~205 classes)
- $MVN_CMD clean test -Pspark-4.1 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pspark-ut -Pdelta \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.1 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pspark-ut -Pdelta \
-DargLine="-Dspark.test.home=/opt/shims/spark41/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
-DwildcardSuites="org.apache.spark.sql.GlutenSQL,org.apache.spark.sql.Gluten,org.apache.spark.sql.connector,org.apache.spark.sql.sources,org.apache.spark.sql.hive,org.apache.spark.sql.gluten,org.apache.spark.sql.shim"
fi
@@ -1485,11 +1494,11 @@ jobs:
export PATH=$JAVA_HOME/bin:$PATH
java -version
if [ "${{ matrix.suite }}" = "extended" ]; then
- $MVN_CMD clean test -Pspark-4.1 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pdelta -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.1 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pdelta -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark41/spark_home/" \
-DtagsToInclude=org.apache.spark.tags.ExtendedSQLTest
else
- $MVN_CMD clean test -Pspark-4.1 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.1 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark41/spark_home/" \
-DtagsToInclude=org.apache.spark.tags.SlowHiveTest
fi
@@ -1557,17 +1566,17 @@ jobs:
$MVN_CMD clean install -Pspark-4.2 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pspark-ut -Pdelta -DskipTests
if [ "${{ matrix.group }}" = "1" ]; then
# Group 1: streaming + gluten utils + top-level spark tests (~55
classes)
- $MVN_CMD clean test -Pspark-4.2 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pspark-ut -Pdelta \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.2 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pspark-ut -Pdelta \
-DargLine="-Dspark.test.home=/opt/shims/spark42/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
-DwildcardSuites="org.apache.spark.sql.streaming,org.apache.spark.GlutenSortShuffleSuite,org.apache.gluten"
elif [ "${{ matrix.group }}" = "2" ]; then
# Group 2: execution + catalyst + errors + extension tests (~140
classes)
- $MVN_CMD clean test -Pspark-4.2 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pspark-ut -Pdelta \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.2 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pspark-ut -Pdelta \
-DargLine="-Dspark.test.home=/opt/shims/spark42/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
-DwildcardSuites="org.apache.spark.sql.execution,org.apache.spark.sql.catalyst,org.apache.spark.sql.errors,org.apache.spark.sql.extension"
else
# Group 3: top-level sql, connector, sources, hive and remaining
tests (~205 classes)
- $MVN_CMD clean test -Pspark-4.2 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pspark-ut -Pdelta \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.2 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pspark-ut -Pdelta \
-DargLine="-Dspark.test.home=/opt/shims/spark42/spark_home/"
-DtagsToExclude="$TAGS_EXCLUDE" \
-DwildcardSuites="org.apache.spark.sql.GlutenSQL,org.apache.spark.sql.Gluten,org.apache.spark.sql.connector,org.apache.spark.sql.sources,org.apache.spark.sql.hive,org.apache.spark.sql.gluten,org.apache.spark.sql.shim"
fi
@@ -1620,11 +1629,11 @@ jobs:
# sibling Gluten modules (gluten-substrait etc.) before testing.
$MVN_CMD clean install -Pspark-4.2 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pdelta -Pspark-ut -DskipTests
if [ "${{ matrix.suite }}" = "extended" ]; then
- $MVN_CMD clean test -Pspark-4.2 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pdelta -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.2 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pdelta -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark42/spark_home/" \
-DtagsToInclude=org.apache.spark.tags.ExtendedSQLTest
else
- $MVN_CMD clean test -Pspark-4.2 -Pscala-2.13 -Pjava-17
-Pbackends-velox -Pspark-ut \
+ $PARALLEL_UT $MVN_CMD clean test -Pspark-4.2 -Pscala-2.13
-Pjava-17 -Pbackends-velox -Pspark-ut \
-DargLine="-Dspark.test.home=/opt/shims/spark42/spark_home/" \
-DtagsToInclude=org.apache.spark.tags.SlowHiveTest
fi
diff --git a/ep/build-velox/src/get-velox.sh b/ep/build-velox/src/get-velox.sh
index 37529d31ec..1fbc9efa9e 100755
--- a/ep/build-velox/src/get-velox.sh
+++ b/ep/build-velox/src/get-velox.sh
@@ -18,8 +18,8 @@ set -exu
CURRENT_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd)
VELOX_REPO=https://github.com/IBM/velox.git
-VELOX_BRANCH=dft-2026_10_21
-VELOX_ENHANCED_BRANCH=ibm-2026_10_21
+VELOX_BRANCH=dft-2026_10_23
+VELOX_ENHANCED_BRANCH=ibm-2026_10_23
VELOX_HOME=""
RUN_SETUP_SCRIPT=ON
ENABLE_ENHANCED_FEATURES=OFF
diff --git a/pom.xml b/pom.xml
index ab555f8c2d..4d21411157 100644
--- a/pom.xml
+++ b/pom.xml
@@ -201,6 +201,9 @@
-Dfile.encoding=UTF-8
</extraJavaTestArgs>
<log4j.conf>file:src/test/resources/log4j2.properties</log4j.conf>
+ <!-- Java executable for ScalaTest forks; empty means the JDK running
Maven. CI points
+ it at .github/workflows/util/parallel-ut/record-fork.sh to run suites
in parallel. -->
+ <scalatest.jvm/>
</properties>
<dependencyManagement>
@@ -623,6 +626,7 @@
<testFailureIgnore>${maven.test.failure.ignore}</testFailureIgnore>
<junitxml>.</junitxml>
<argLine>${argLine} ${extraJavaTestArgs}</argLine>
+ <jvm>${scalatest.jvm}</jvm>
<systemProperties>
<log4j.configurationFile>${log4j.conf}</log4j.configurationFile>
</systemProperties>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
