This is an automated email from the ASF dual-hosted git repository.
changchen pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 48ca0ba9ce [GLUTEN-11559][Build] Improve incremental build time for
test-compile phase (#11560)
48ca0ba9ce is described below
commit 48ca0ba9ce7af44c0d96f6d75da0ce77784535ba
Author: Chang Chen <[email protected]>
AuthorDate: Tue Feb 10 19:11:10 2026 +0800
[GLUTEN-11559][Build] Improve incremental build time for test-compile phase
(#11560)
* Upgrade protobuf-maven-plugin from 0.5.1 to 0.6.1 and enable
checkStaleness
Enable <checkStaleness>true</checkStaleness> in all protobuf-maven-plugin
executions (gluten-core, gluten-substrait, backends-velox) so protobuf
compilation is skipped when .proto files haven't changed, improving
incremental build speed.
* Enable Scala incremental compilation
1. Upgrade scala-maven-plugin from 4.8.0 to 4.9.2 (aligned with Spark)
2. Change scala.recompile.mode from 'all' to 'incremental'
3. Skip javac compilation - Zinc already handles Java sources in
incremental mode (same approach as Apache Spark)
4. Add -Ybackend-parallelism 8 for both Scala 2.12 and 2.13 profiles
5. Update gluten-it to use incremental mode and 4.9.2 (hardcoded since
it's a standalone third-party module without parent POM properties)
* Consolidate build-info generation into a single location
Merge build-info and build-info-with-backends into a single execution
in gluten-core, eliminating the separate call from gluten-substrait:
- Remove build-info-with-backends execution from gluten-substrait/pom.xml
- Remove redundant backend profile definitions from gluten-substrait
- Add --backend parameter to gluten-core's build-info execution
- Modify gluten-build-info.sh to compute backend paths internally
based on backend_type (no longer needs external path argument)
- Remove DO_REMOVAL flag; always regenerate the file from scratch
* Add dev tooling: run-scala-test.sh, mvnd wrapper, and build profiler
- dev/run-scala-test.sh: Run ScalaTest like IntelliJ IDEA from CLI
with auto classpath resolution, profiler support, and mvnd integration
- build/mvnd: Maven Daemon wrapper (auto-downloads mvnd 1.0.3)
for persistent JVM that keeps Zinc's JIT caches across builds
- build/mvn: Increase ReservedCodeCacheSize from 1g to 2g
- dev/analyze-build-profile.py: Analyze Maven profiler JSON reports
- .gitignore: Add build/mvnd, .run-scala-test-cache/, .profiler/, .mvn/
* Move test-jar from test-compile to package phase
Remove the <phase>test-compile</phase> override from the prepare-test-jar
execution in all 18 modules. The test-jar goal defaults to the package
phase, so test-jars are no longer rebuilt during mvn test-compile.
This eliminates a Zinc cascade recompilation issue: previously, test-jars
were repackaged at every test-compile invocation (even with no changes),
causing downstream modules to detect classpath changes and triggering
full recompilation of their test sources.
Trade-off: cross-module test-jar dependencies in the reactor are now
resolved from the local repository (~/.m2) during test-compile. Run
'mvn install -DskipTests' after changing upstream test APIs.
* Split -Wconf combined rules into separate flags for Scala 2.13.15+
Since Scala 2.13.15 (scala/scala#10708), the semantics of combined
-Wconf rules changed: in '-Wconf:x,y', y now takes priority over x
(last-match-wins), whereas before 2.13.15, x took priority
(first-match-wins). This means '-Wconf:cat=deprecation:wv,any:e' now
treats deprecation warnings as errors (any:e overrides
cat=deprecation:wv), breaking Scala 2.13 compilation when -Pdelta is
enabled.
Split into separate -Wconf flags where later flags have higher
priority:
-Wconf:any:e (baseline)
-Wconf:msg=While parsing annotations in:silent (override)
-Wconf:cat=deprecation:wv (override)
This aligns with Apache Spark's approach in SPARK-49746 (983f6f43).
Gluten uses Scala 2.13.17 which is affected by this change.
Reference:
- https://github.com/scala/scala/pull/10708
- https://github.com/apache/spark/pull/48192
* Remove analyze-build-profile.py
AI tools can perform build profile analysis on-demand without
requiring a committed script. Moved to fix/improve-incremental-build-tmp
branch for reference.
---
.gitignore | 12 +-
backends-clickhouse/pom.xml | 1 -
backends-velox/pom.xml | 2 +-
build/mvn | 2 +-
build/mvnd | 179 +++++++++++
dev/gluten-build-info.sh | 19 +-
dev/run-scala-test.sh | 673 +++++++++++++++++++++++++++++++++++++++++
gluten-core/pom.xml | 5 +-
gluten-delta/pom.xml | 1 -
gluten-hudi/pom.xml | 1 -
gluten-iceberg/pom.xml | 1 -
gluten-kafka/pom.xml | 1 -
gluten-paimon/pom.xml | 1 -
gluten-ras/pom.xml | 1 -
gluten-substrait/pom.xml | 43 +--
gluten-ut/common/pom.xml | 1 -
gluten-ut/spark32/pom.xml | 1 -
gluten-ut/spark33/pom.xml | 1 -
gluten-ut/spark34/pom.xml | 1 -
gluten-ut/spark35/pom.xml | 1 -
gluten-ut/spark40/pom.xml | 1 -
gluten-ut/spark41/pom.xml | 1 -
gluten-ut/test/pom.xml | 1 -
pom.xml | 27 +-
tools/gluten-it/common/pom.xml | 6 +-
25 files changed, 900 insertions(+), 83 deletions(-)
diff --git a/.gitignore b/.gitignore
index 8a794e3865..a679fc3002 100644
--- a/.gitignore
+++ b/.gitignore
@@ -27,8 +27,9 @@ CMakeFiles/
CMakeCache.txt
CTestTestfile.cmake
cmake_install.cmake
-!build/mvn
build/
+!build/mvn
+!build/mvnd
*-build/
Testing/
cmake-build-*/
@@ -76,3 +77,12 @@ metastore_db/
# GitHub Copilot config (allow local customization)
.github/copilot-instructions.md
.github/copilot-setup-steps.yml
+
+# Dev script build cache
+.run-scala-test-cache/
+
+# Maven profiler reports
+.profiler/
+
+# Maven extensions (local config)
+.mvn/
diff --git a/backends-clickhouse/pom.xml b/backends-clickhouse/pom.xml
index 765dfdd762..4347deac18 100644
--- a/backends-clickhouse/pom.xml
+++ b/backends-clickhouse/pom.xml
@@ -457,7 +457,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
diff --git a/backends-velox/pom.xml b/backends-velox/pom.xml
index fa4e52817e..428bfce03c 100644
--- a/backends-velox/pom.xml
+++ b/backends-velox/pom.xml
@@ -278,7 +278,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
@@ -298,6 +297,7 @@
<protocArtifact>com.google.protobuf:protoc:${protobuf.version}:exe:${os.detected.classifier}</protocArtifact>
<protoSourceRoot>src/main/resources/org/apache/gluten/proto</protoSourceRoot>
<clearOutputDirectory>false</clearOutputDirectory>
+ <checkStaleness>true</checkStaleness>
</configuration>
</execution>
</executions>
diff --git a/build/mvn b/build/mvn
index f20ea7f7b8..550f1d1435 100755
--- a/build/mvn
+++ b/build/mvn
@@ -127,7 +127,7 @@ if [ ! -f "${MVN_BIN}" ]; then
exit 1
fi
-_COMPILE_JVM_OPTS="-Xss128m -Xmx4g -XX:ReservedCodeCacheSize=1g"
+_COMPILE_JVM_OPTS="-Xss128m -Xmx4g -XX:ReservedCodeCacheSize=2g"
# Set any `mvn` options if not already present
export MAVEN_OPTS=${MAVEN_OPTS:-"$_COMPILE_JVM_OPTS"}
diff --git a/build/mvnd b/build/mvnd
new file mode 100755
index 0000000000..166c505f81
--- /dev/null
+++ b/build/mvnd
@@ -0,0 +1,179 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This script downloads and installs Maven Daemon (mvnd) if it's not already
installed.
+# mvnd keeps a persistent JVM that preserves Zinc's JIT-optimized code and
+# classloader caches across builds, significantly speeding up incremental
compilation.
+# Usage: build/mvnd <maven-args>
+#
+
+# Determine the current working directory
+GLUTEN_HOME="$(cd "$(dirname "$0")"/.. && pwd)" || exit 1
+DOWNLOAD_DIR="${GLUTEN_HOME}/build"
+MVND_DOWNLOAD_DIR="${DOWNLOAD_DIR}/.mvnd"
+
+# mvnd version to download if not found on system
+MVND_VERSION="1.0.3"
+
+# Global variable for mvnd binary path
+MVND_BIN=""
+
+# Detect OS and architecture for mvnd download
+detect_platform() {
+ local os arch
+ os="$(uname -s)"
+ arch="$(uname -m)"
+
+ case "$os" in
+ Linux) os="linux" ;;
+ Darwin) os="darwin" ;;
+ *)
+ echo "ERROR: Unsupported OS: $os. mvnd binaries are available for
Linux and macOS." >&2
+ exit 1
+ ;;
+ esac
+
+ case "$arch" in
+ x86_64|amd64) arch="amd64" ;;
+ aarch64|arm64) arch="aarch64" ;;
+ *)
+ echo "ERROR: Unsupported architecture: $arch. mvnd binaries are
available for amd64 and aarch64." >&2
+ exit 1
+ ;;
+ esac
+
+ echo "${os}-${arch}"
+}
+
+install_mvnd() {
+ # Check for system mvnd first
+ local SYSTEM_MVND
+ SYSTEM_MVND="$(command -v mvnd)"
+
+ if [ -n "$SYSTEM_MVND" ]; then
+ local MVND_DETECTED_VERSION
+ MVND_DETECTED_VERSION="$(mvnd --version 2>/dev/null | grep -i 'mvnd' |
head -1 | awk '{print $NF}')"
+ echo "Using system mvnd: $SYSTEM_MVND (version
$MVND_DETECTED_VERSION)" >&2
+ MVND_BIN="$SYSTEM_MVND"
+ return 0
+ fi
+
+ # Detect platform
+ local PLATFORM
+ PLATFORM="$(detect_platform)"
+
+ local MVND_DIR_NAME="maven-mvnd-${MVND_VERSION}-${PLATFORM}"
+ local MVND_LOCAL_BIN="${MVND_DOWNLOAD_DIR}/${MVND_DIR_NAME}/bin/mvnd"
+
+ if [ ! -f "${MVND_LOCAL_BIN}" ]; then
+ echo "mvnd ${MVND_VERSION} not found locally. Downloading..." >&2
+
+ # Create download directory
+ mkdir -p "${MVND_DOWNLOAD_DIR}"
+
+ local MVND_TAR="${MVND_DOWNLOAD_DIR}/${MVND_DIR_NAME}.tar.gz"
+
+ if [ ! -f "${MVND_TAR}" ]; then
+ # Download from Apache
+ local
DOWNLOAD_URL="https://downloads.apache.org/maven/mvnd/${MVND_VERSION}/${MVND_DIR_NAME}.tar.gz";
+
+ echo "Downloading mvnd ${MVND_VERSION} for ${PLATFORM}..." >&2
+ echo "URL: ${DOWNLOAD_URL}" >&2
+
+ if command -v curl > /dev/null 2>&1; then
+ curl -f -L --retry 3 --retry-delay 3 \
+ --connect-timeout 30 --max-time 600 \
+ -o "${MVND_TAR}" "${DOWNLOAD_URL}" || {
+ echo "ERROR: Failed to download mvnd from ${DOWNLOAD_URL}"
>&2
+ rm -f "${MVND_TAR}"
+ exit 1
+ }
+ elif command -v wget > /dev/null 2>&1; then
+ wget --tries=3 --waitretry=3 \
+ --connect-timeout=30 --read-timeout=600 \
+ -O "${MVND_TAR}" "${DOWNLOAD_URL}" || {
+ echo "ERROR: Failed to download mvnd from ${DOWNLOAD_URL}"
>&2
+ rm -f "${MVND_TAR}"
+ exit 1
+ }
+ else
+ echo "ERROR: Neither curl nor wget found. Please install one
of them or install mvnd manually." >&2
+ exit 1
+ fi
+
+ echo "Download completed successfully" >&2
+ fi
+
+ # Extract mvnd
+ echo "Extracting mvnd to ${MVND_DOWNLOAD_DIR}..." >&2
+ if ! tar -xzf "${MVND_TAR}" -C "${MVND_DOWNLOAD_DIR}"; then
+ echo "ERROR: Failed to extract mvnd" >&2
+ rm -f "${MVND_TAR}"
+ exit 1
+ fi
+
+ # Clean up tar file
+ rm -f "${MVND_TAR}"
+
+ # Configure mvnd daemon JVM settings for Scala compilation
+ # ReservedCodeCacheSize=2g: Scala compiler generates enormous JIT code;
+ # default 240M fills up, causing ~10x slowdown in interpreted mode.
+ local
MVND_PROPS="${MVND_DOWNLOAD_DIR}/${MVND_DIR_NAME}/conf/mvnd.properties"
+ if [ -f "${MVND_PROPS}" ]; then
+ echo "" >> "${MVND_PROPS}"
+ echo "# Gluten: tuned for Scala compilation workloads" >>
"${MVND_PROPS}"
+ echo "mvnd.maxHeapSize = 30G" >> "${MVND_PROPS}"
+ echo "mvnd.threadStackSize = 128M" >> "${MVND_PROPS}"
+ echo "mvnd.jvmArgs = -XX:ReservedCodeCacheSize=2g" >>
"${MVND_PROPS}"
+ echo "Configured mvnd daemon JVM settings in ${MVND_PROPS}" >&2
+ fi
+
+ echo "mvnd ${MVND_VERSION} installed successfully to
${MVND_DOWNLOAD_DIR}/${MVND_DIR_NAME}" >&2
+ else
+ echo "Using downloaded mvnd: ${MVND_LOCAL_BIN} (version
${MVND_VERSION})" >&2
+ fi
+
+ # Set global variable
+ MVND_BIN="${MVND_LOCAL_BIN}"
+}
+
+# Install mvnd if needed
+install_mvnd
+
+# Verify mvnd binary is set
+if [ -z "${MVND_BIN}" ]; then
+ echo "ERROR: mvnd binary not found. Please install mvnd or check your
installation." >&2
+ exit 1
+fi
+
+# Verify mvnd binary exists
+if [ ! -f "${MVND_BIN}" ]; then
+ echo "ERROR: mvnd binary does not exist: ${MVND_BIN}" >&2
+ exit 1
+fi
+
+_COMPILE_JVM_OPTS="-Xss128m -Xmx4g -XX:ReservedCodeCacheSize=2g"
+# Set any `mvn` options if not already present
+export MAVEN_OPTS=${MAVEN_OPTS:-"$_COMPILE_JVM_OPTS"}
+
+echo "MAVEN_OPTS: ${MAVEN_OPTS}" >&2
+
+"${MVND_BIN}" "$@"
+
+MVND_RETCODE=$?
+
+exit $MVND_RETCODE
diff --git a/dev/gluten-build-info.sh b/dev/gluten-build-info.sh
index d4bd424d67..22db643728 100755
--- a/dev/gluten-build-info.sh
+++ b/dev/gluten-build-info.sh
@@ -21,10 +21,9 @@ GLUTEN_ROOT=$(cd $(dirname -- $0)/..; pwd -P)
EXTRA_RESOURCE_DIR=$GLUTEN_ROOT/gluten-core/target/generated-resources
BUILD_INFO="$EXTRA_RESOURCE_DIR"/gluten-build-info.properties
-DO_REMOVAL="$1" && shift
-if [ "true" = "$DO_REMOVAL" ]; then
- rm -rf "$BUILD_INFO"
-fi
+
+# Delete old build-info file before regenerating
+rm -f "$BUILD_INFO"
mkdir -p "$EXTRA_RESOURCE_DIR"
function echo_revision_info() {
@@ -56,13 +55,15 @@ while (( "$#" )); do
echo gluten_version="$2" >> "$BUILD_INFO"
;;
--backend)
- echo backend_type="$2" >> "$BUILD_INFO"
- if [ "velox" = "$2" ]; then
- echo_velox_revision_info "$3" >> "$BUILD_INFO"
- elif [ "ch" = "$2" ]; then
+ BACKEND_TYPE="$2"
+ echo backend_type="$BACKEND_TYPE" >> "$BUILD_INFO"
+ # Compute backend home path based on type
+ if [ "velox" = "$BACKEND_TYPE" ]; then
+ BACKEND_HOME="$GLUTEN_ROOT/ep/build-velox/build/velox_ep"
+ echo_velox_revision_info "$BACKEND_HOME" >> "$BUILD_INFO"
+ elif [ "ch" = "$BACKEND_TYPE" ] || [ "clickhouse" = "$BACKEND_TYPE" ];
then
echo_clickhouse_revision_info >> "$BUILD_INFO"
fi
- shift
;;
--java)
echo java_version="$2" >> "$BUILD_INFO"
diff --git a/dev/run-scala-test.sh b/dev/run-scala-test.sh
new file mode 100755
index 0000000000..4cc1908085
--- /dev/null
+++ b/dev/run-scala-test.sh
@@ -0,0 +1,673 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# =============================================================================
+# run-scala-test.sh - Run ScalaTest like IntelliJ IDEA
+# =============================================================================
+#
+# This script simulates IntelliJ IDEA's ScalaTest execution to allow running
+# individual test methods, which is not possible with standard Maven commands
+# due to the conflict between -Dsuites and -am parameters.
+#
+# Usage:
+# ./dev/run-scala-test.sh [options] -P<profiles> -pl <module> -s <suite> [-t
"test name"]
+#
+# Examples:
+# # Run entire suite
+# ./dev/run-scala-test.sh \
+# -Pjava-17,spark-4.0,scala-2.13,backends-velox,hadoop-3.3,spark-ut \
+# -pl gluten-ut/spark40 \
+# -s org.apache.spark.sql.GlutenDeprecatedDatasetAggregatorSuite
+#
+# # Run single test method
+# ./dev/run-scala-test.sh \
+# -Pjava-17,spark-4.0,scala-2.13,backends-velox,hadoop-3.3,spark-ut \
+# -pl gluten-ut/spark40 \
+# -s org.apache.spark.sql.GlutenDeprecatedDatasetAggregatorSuite \
+# -t "typed aggregation: class input with reordering"
+#
+# # With Maven Daemon (mvnd) for faster builds
+# ./dev/run-scala-test.sh \
+# -Pjava-17,spark-4.0,scala-2.13,backends-velox,hadoop-3.3,spark-ut \
+# -pl gluten-ut/spark40 \
+# -s org.apache.spark.sql.GlutenDeprecatedDatasetAggregatorSuite \
+# --mvnd
+#
+# # With Maven profiler enabled
+# ./dev/run-scala-test.sh \
+# -Pjava-17,spark-4.0,scala-2.13,backends-velox,hadoop-3.3,spark-ut \
+# -pl gluten-ut/spark40 \
+# -s org.apache.spark.sql.GlutenDeprecatedDatasetAggregatorSuite \
+# --profile
+#
+# # Export classpath only (no test execution)
+# ./dev/run-scala-test.sh \
+# -Pjava-17,spark-4.0,scala-2.13,backends-velox,hadoop-3.3,spark-ut \
+# -pl gluten-ut/spark40 \
+# -s org.apache.spark.sql.GlutenDeprecatedDatasetAggregatorSuite \
+# --export-only
+#
+# =============================================================================
+
+set -e
+
+# Get script directory and project root
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+GLUTEN_HOME="$(cd "${SCRIPT_DIR}/.." && pwd)"
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m'
+CYAN='\033[0;36m'
+MAGENTA='\033[0;35m'
+
+# =============================================================================
+# Timing helpers
+# =============================================================================
+
+timer_now() {
+ date +%s%N
+}
+
+# Returns elapsed time in seconds (with milliseconds)
+timer_elapsed() {
+ local start=$1 end=$2
+ echo "scale=1; ($end - $start) / 1000000000" | bc
+}
+
+format_duration() {
+ local secs=$1
+ local mins=$(echo "$secs / 60" | bc)
+ local remaining=$(echo "scale=1; $secs - $mins * 60" | bc)
+ if [[ "$mins" -gt 0 ]]; then
+ printf "%dm %.1fs" "$mins" "$remaining"
+ else
+ printf "%.1fs" "$remaining"
+ fi
+}
+
+log_timing() {
+ [[ "$ENABLE_PROFILER" != "true" ]] && return
+ local label=$1 elapsed=$2
+ local formatted=$(format_duration "$elapsed")
+ echo -e "${CYAN}[TIME]${NC} ${label}: ${MAGENTA}${formatted}${NC}"
+}
+
+# Print timing summary table. Args: include_step4 (true/false)
+print_timing_summary() {
+ [[ "$ENABLE_PROFILER" != "true" ]] && return
+ local include_step4=${1:-false}
+ local mvn_total=$(echo "$TIMING_STEP1 + $TIMING_STEP2" | bc)
+ local overall=$(echo "$mvn_total + $TIMING_STEP3" | bc)
+ local cached_tag=""
+ [[ "$CACHE_HIT" == "true" ]] && cached_tag=" (cached)"
+
+ echo ""
+ echo -e "${CYAN}==========================================${NC}"
+ echo -e "${CYAN} ⏱ Timing Summary${NC}"
+ echo -e "${CYAN}==========================================${NC}"
+ printf " %-40s %s\n" "Step 1 - Compile + Classpath (mvn):"
"$(format_duration $TIMING_STEP1)${cached_tag}"
+ printf " %-40s %s\n" "Step 2 - Evaluate JVM args (mvn):" "$(format_duration
$TIMING_STEP2)${cached_tag}"
+ printf " %-40s %s\n" "Step 3 - Resolve classpath:" "$(format_duration
$TIMING_STEP3)"
+ if [[ "$include_step4" == "true" ]]; then
+ printf " %-40s %s\n" "Step 4 - ScalaTest execution:" "$(format_duration
$TIMING_STEP4)"
+ overall=$(echo "$overall + $TIMING_STEP4" | bc)
+ fi
+ echo " ------------------------------------------"
+ printf " %-40s %s\n" "Maven total:" "$(format_duration $mvn_total)"
+ if [[ "$include_step4" == "true" ]]; then
+ printf " %-40s %s\n" "Test execution:" "$(format_duration $TIMING_STEP4)"
+ fi
+ printf " %-40s %s\n" "Overall:" "$(format_duration $overall)"
+ echo -e "${CYAN}==========================================${NC}"
+}
+
+# =============================================================================
+# Module Mapping: artifactId -> directory path
+# =============================================================================
+# Format: "artifactId:directory:type"
+# type: "scala" for target/scala-X.XX/classes, "java" for target/classes
+declare -A MODULE_MAP=(
+ # Core modules
+ ["gluten-core"]="gluten-core:scala"
+ ["gluten-substrait"]="gluten-substrait:scala"
+ ["gluten-ui"]="gluten-ui:scala"
+ ["gluten-arrow"]="gluten-arrow:scala"
+
+ # Backend modules
+ ["backends-velox"]="backends-velox:scala"
+ ["backends-clickhouse"]="backends-clickhouse:scala"
+
+ # RAS modules
+ ["gluten-ras-common"]="gluten-ras/common:scala"
+ ["gluten-ras-planner"]="gluten-ras/planner:scala"
+
+ # Shims modules (Java only, no scala subdirectory)
+ ["spark-sql-columnar-shims-common"]="shims/common:java"
+ ["spark-sql-columnar-shims-spark32"]="shims/spark32:java"
+ ["spark-sql-columnar-shims-spark33"]="shims/spark33:java"
+ ["spark-sql-columnar-shims-spark34"]="shims/spark34:java"
+ ["spark-sql-columnar-shims-spark35"]="shims/spark35:java"
+ ["spark-sql-columnar-shims-spark40"]="shims/spark40:java"
+ ["spark-sql-columnar-shims-spark41"]="shims/spark41:java"
+
+ # Unit test modules
+ ["gluten-ut-common"]="gluten-ut/common:scala"
+ ["gluten-ut-test"]="gluten-ut/test:scala"
+ ["gluten-ut-spark32"]="gluten-ut/spark32:scala"
+ ["gluten-ut-spark33"]="gluten-ut/spark33:scala"
+ ["gluten-ut-spark34"]="gluten-ut/spark34:scala"
+ ["gluten-ut-spark35"]="gluten-ut/spark35:scala"
+ ["gluten-ut-spark40"]="gluten-ut/spark40:scala"
+ ["gluten-ut-spark41"]="gluten-ut/spark41:scala"
+
+ # Data lake modules
+ ["gluten-delta"]="gluten-delta:scala"
+ ["gluten-iceberg"]="gluten-iceberg:scala"
+ ["gluten-hudi"]="gluten-hudi:scala"
+ ["gluten-paimon"]="gluten-paimon:scala"
+
+ # Shuffle modules
+ ["gluten-celeborn"]="gluten-celeborn:scala"
+ ["gluten-uniffle"]="gluten-uniffle:scala"
+
+ # Other modules
+ ["gluten-kafka"]="gluten-kafka:scala"
+)
+
+# =============================================================================
+# Functions
+# =============================================================================
+
+print_usage() {
+ cat << EOF
+Usage: $0 [options] -P<profiles> -pl <module> -s <suite> [-t "test name"]
+
+Required:
+ -P<profiles> Maven profiles (e.g.,
-Pjava-17,spark-4.0,scala-2.13,backends-velox)
+ -pl <module> Target module (e.g., gluten-ut/spark40)
+ -s <suite> Full suite class name
+
+Optional:
+ -t "test name" Specific test method name to run
+ --mvnd Use Maven Daemon (mvnd) instead of ./build/mvn
+ --clean Run 'mvn clean' before compiling
+ --force Force Maven rebuild, bypass build cache
+ --profile Enable Maven profiler (reports in .profiler/)
+ --export-only Export classpath and exit (no test execution)
+ --help Show this help message
+
+Examples:
+ # Run entire suite
+ $0 -Pjava-17,spark-4.0,scala-2.13,backends-velox,hadoop-3.3,spark-ut \\
+ -pl gluten-ut/spark40 \\
+ -s org.apache.spark.sql.GlutenDeprecatedDatasetAggregatorSuite
+
+ # Run single test method
+ $0 -Pjava-17,spark-4.0,scala-2.13,backends-velox,hadoop-3.3,spark-ut \\
+ -pl gluten-ut/spark40 \\
+ -s org.apache.spark.sql.GlutenDeprecatedDatasetAggregatorSuite \\
+ -t "typed aggregation: class input with reordering"
+EOF
+}
+
+log_info() {
+ echo -e "${GREEN}[INFO]${NC} $1"
+}
+
+log_warn() {
+ echo -e "${YELLOW}[WARN]${NC} $1"
+}
+
+log_error() {
+ echo -e "${RED}[ERROR]${NC} $1"
+}
+
+log_step() {
+ echo -e "${BLUE}[STEP]${NC} $1"
+}
+
+# Detect Scala version from profiles
+detect_scala_version() {
+ local profiles="$1"
+ if [[ "$profiles" == *"scala-2.13"* ]]; then
+ echo "2.13"
+ else
+ echo "2.12"
+ fi
+}
+
+# Get target classes directory for a module
+get_target_dir() {
+ local artifact_id="$1"
+ local scala_version="$2"
+ local class_type="$3"
+
+ local mapping="${MODULE_MAP[$artifact_id]}"
+ if [[ -z "$mapping" ]]; then
+ return 1
+ fi
+
+ local dir_path="${mapping%%:*}"
+ local module_type="${mapping##*:}"
+
+ local target_path="${GLUTEN_HOME}/${dir_path}/target"
+
+ if [[ "$module_type" == "scala" ]]; then
+ echo "${target_path}/scala-${scala_version}/${class_type}"
+ else
+ echo "${target_path}/${class_type}"
+ fi
+}
+
+# Replace gluten jar paths with target/classes directories
+replace_gluten_paths() {
+ local classpath="$1"
+ local scala_version="$2"
+ local result=""
+ local added_paths=""
+
+ IFS=':' read -ra CP_ENTRIES <<< "$classpath"
+ for entry in "${CP_ENTRIES[@]}"; do
+ if [[ "$entry" != *"/org/apache/gluten/"* && "$entry" !=
*"/gluten-"*"/target/"* ]]; then
+ result="${result}:${entry}"
+ continue
+ fi
+
+ # Local target directories are already in the desired form from reactor
builds
+ if [[ -d "$entry" && "$entry" == *"/target/"* ]]; then
+ result="${result}:${entry}"
+ continue
+ fi
+
+ local filename=$(basename "$entry")
+ local artifact_id=""
+ for known_artifact in "${!MODULE_MAP[@]}"; do
+ if [[ "$filename" == "${known_artifact}-"* ]]; then
+ artifact_id="$known_artifact"
+ break
+ fi
+ done
+
+ if [[ -z "$artifact_id" ]]; then
+ log_error "Unknown gluten module in classpath: $entry" >&2
+ log_error "Please add it to MODULE_MAP in this script." >&2
+ exit 1
+ fi
+
+ local class_type="classes"
+ [[ "$filename" == *"-tests.jar" ]] && class_type="test-classes"
+
+ local target_dir=$(get_target_dir "$artifact_id" "$scala_version"
"$class_type")
+ if [[ -d "$target_dir" && "$added_paths" != *"$target_dir"* ]]; then
+ result="${result}:${target_dir}"
+ added_paths="${added_paths}:${target_dir}"
+ fi
+ done
+
+ echo "${result#:}"
+}
+
+# =============================================================================
+# Parse Arguments
+# =============================================================================
+
+PROFILES=""
+MODULE=""
+SUITE=""
+TEST_METHOD=""
+EXTRA_MVN_ARGS=""
+ENABLE_PROFILER=false
+EXPORT_ONLY=false
+ENABLE_CLEAN=false
+FORCE_BUILD=false
+USE_MVND=false
+
+while [[ $# -gt 0 ]]; do
+ case $1 in
+ -P*)
+ PROFILES="${1#-P}"
+ shift
+ ;;
+ -pl)
+ MODULE="$2"
+ shift 2
+ ;;
+ -s)
+ SUITE="$2"
+ shift 2
+ ;;
+ -t)
+ TEST_METHOD="$2"
+ shift 2
+ ;;
+ --profile)
+ ENABLE_PROFILER=true
+ shift
+ ;;
+ --clean)
+ ENABLE_CLEAN=true
+ shift
+ ;;
+ --force)
+ FORCE_BUILD=true
+ shift
+ ;;
+ --mvnd)
+ USE_MVND=true
+ shift
+ ;;
+ --export-only)
+ EXPORT_ONLY=true
+ shift
+ ;;
+ --help)
+ print_usage
+ exit 0
+ ;;
+ *)
+ # Collect other arguments for Maven
+ EXTRA_MVN_ARGS="${EXTRA_MVN_ARGS} $1"
+ shift
+ ;;
+ esac
+done
+
+# Validate required arguments
+if [[ -z "$PROFILES" ]]; then
+ log_error "Missing required argument: -P<profiles>"
+ print_usage
+ exit 1
+fi
+
+if [[ -z "$MODULE" ]]; then
+ log_error "Missing required argument: -pl <module>"
+ print_usage
+ exit 1
+fi
+
+if [[ -z "$SUITE" ]]; then
+ log_error "Missing required argument: -s <suite>"
+ print_usage
+ exit 1
+fi
+
+# Detect Scala version
+SCALA_VERSION=$(detect_scala_version "$PROFILES")
+log_info "Detected Scala version: ${SCALA_VERSION}"
+
+# =============================================================================
+# Build Cache - Skip Maven when source files haven't changed
+# =============================================================================
+#
+# After a successful Maven build, a sentinel file is touched. On subsequent
+# runs, if no .scala/.java/pom.xml files are newer than the sentinel, Maven
+# is skipped entirely and cached classpath/JVM args are reused. This saves
+# ~52s of Zinc analysis loading overhead per run.
+#
+# Use --force to bypass the cache, or --clean which implicitly bypasses it.
+# =============================================================================
+
+CACHE_DIR="${GLUTEN_HOME}/.run-scala-test-cache"
+mkdir -p "$CACHE_DIR"
+CACHE_KEY=$(echo "${PROFILES}__${MODULE}" | md5sum | cut -d' ' -f1)
+BUILD_SENTINEL="${CACHE_DIR}/sentinel_${CACHE_KEY}"
+CLASSPATH_CACHE="${CACHE_DIR}/classpath_${CACHE_KEY}.txt"
+JVM_ARGS_CACHE="${CACHE_DIR}/jvm_args_${CACHE_KEY}.txt"
+
+can_skip_maven() {
+ [[ "$ENABLE_CLEAN" == "true" ]] && return 1
+ [[ "$FORCE_BUILD" == "true" ]] && return 1
+ [[ ! -f "$BUILD_SENTINEL" || ! -f "$CLASSPATH_CACHE" || ! -f
"$JVM_ARGS_CACHE" ]] && return 1
+
+ # Check if any source file or pom.xml changed since last successful build
+ local changed
+ changed=$(find "${GLUTEN_HOME}" \
+ \( -path "*/target" -o -path "*/.git" -o -path "*/.run-scala-test-cache" \
+ -o -path "*/.profiler" -o -path "*/.mvn" \) -prune -o \
+ -newer "$BUILD_SENTINEL" \( \
+ -name "pom.xml" -o \
+ \( -path "*/src/*" \( -name "*.scala" -o -name "*.java" \) \) \
+ \) -print 2>/dev/null | head -1)
+
+ [[ -z "$changed" ]]
+}
+
+save_build_cache() {
+ echo "$RAW_CLASSPATH" > "$CLASSPATH_CACHE"
+ echo "$JVM_ARGS" > "$JVM_ARGS_CACHE"
+ touch "$BUILD_SENTINEL"
+ log_info "Build cache saved"
+}
+
+if can_skip_maven; then
+ log_step "Steps 1-2: Using cached build (no source changes detected)"
+ RAW_CLASSPATH=$(cat "$CLASSPATH_CACHE")
+ JVM_ARGS=$(cat "$JVM_ARGS_CACHE")
+ TIMING_STEP1=0
+ TIMING_STEP2=0
+ CACHE_HIT=true
+ log_info "Cached classpath (${#RAW_CLASSPATH} chars)"
+ log_info "Cached JVM args: ${JVM_ARGS:0:100}..."
+ log_info "Maven compilation skipped — use --force to rebuild"
+else
+
+# =============================================================================
+# Step 0: Ensure maven-profiler extension is installed (optional)
+# =============================================================================
+
+MVN_EXTENSIONS_FILE="${GLUTEN_HOME}/.mvn/extensions.xml"
+if [[ ! -f "${MVN_EXTENSIONS_FILE}" ]]; then
+ log_info "Creating .mvn/extensions.xml for maven-profiler..."
+ mkdir -p "${GLUTEN_HOME}/.mvn"
+ cat > "${MVN_EXTENSIONS_FILE}" << 'EXTENSIONS_EOF'
+<?xml version="1.0" encoding="UTF-8"?>
+<extensions>
+ <extension>
+ <groupId>fr.jcgay.maven</groupId>
+ <artifactId>maven-profiler</artifactId>
+ <version>3.3</version>
+ </extension>
+</extensions>
+EXTENSIONS_EOF
+fi
+
+# =============================================================================
+# Step 1: Compile and get classpath
+# =============================================================================
+
+log_step "Step 1: Compiling and getting classpath..."
+
+TIMER_STEP1_START=$(timer_now)
+
+CLASSPATH_FILE="/tmp/gluten-test-classpath-$$.txt"
+
+cd "${GLUTEN_HOME}"
+
+log_info "Running: ./build/mvn test-compile dependency:build-classpath -pl
${MODULE} -am -P${PROFILES} ..."
+
+# Enable Maven profiler if requested
+if [[ "$ENABLE_PROFILER" == "true" ]]; then
+ EXTRA_MVN_ARGS="${EXTRA_MVN_ARGS} -Dprofile -DprofileFormat=JSON"
+ log_info "Profiler enabled - timing summary will be printed; JSON reports in
.profiler/"
+fi
+
+# Build Maven goals
+MVN_GOALS="test-compile dependency:build-classpath"
+if [[ "$ENABLE_CLEAN" == "true" ]]; then
+ MVN_GOALS="clean ${MVN_GOALS}"
+ log_info "Clean build requested"
+fi
+
+# Select Maven command: --mvnd uses Maven Daemon (persistent JVM that
+# keeps Zinc's JIT-optimized code and classloader caches across builds).
+if [[ "$USE_MVND" == "true" ]]; then
+ MVN_CMD="./build/mvnd"
+ log_info "Using Maven Daemon (mvnd) for faster builds"
+else
+ MVN_CMD="./build/mvn"
+fi
+
+${MVN_CMD} ${MVN_GOALS} \
+ -T 1C -q \
+ -pl "${MODULE}" -am \
+ -P"${PROFILES}" \
+ -DincludeScope=test \
+ -Dmdep.outputFile="${CLASSPATH_FILE}" \
+ -Dspotless.check.skip=true \
+ -Dscalastyle.skip=true \
+ -Dcheckstyle.skip=true \
+ -Dmaven.gitcommitid.skip=true \
+ -Dremoteresources.skip=true \
+ ${EXTRA_MVN_ARGS}
+
+if [[ ! -f "${CLASSPATH_FILE}" ]]; then
+ log_error "Failed to generate classpath file"
+ exit 1
+fi
+
+RAW_CLASSPATH=$(cat "${CLASSPATH_FILE}")
+log_info "Got raw classpath (${#RAW_CLASSPATH} chars)"
+
+TIMER_STEP1_END=$(timer_now)
+TIMING_STEP1=$(timer_elapsed $TIMER_STEP1_START $TIMER_STEP1_END)
+log_timing "Step 1 - Compile + Classpath (mvn)" "$TIMING_STEP1"
+
+# =============================================================================
+# Step 2: Get JVM arguments
+# =============================================================================
+
+log_step "Step 2: Getting JVM arguments from pom.xml..."
+
+TIMER_STEP2_START=$(timer_now)
+
+JVM_ARGS_RAW=$(${MVN_CMD} help:evaluate \
+ -Dexpression=extraJavaTestArgs \
+ -q -DforceStdout \
+ -P"${PROFILES}" 2>/dev/null || echo "")
+
+# Clean up JVM args: remove comments, trim whitespace, convert to single line
+JVM_ARGS=$(echo "$JVM_ARGS_RAW" | \
+ grep -v '<!--' | \
+ grep -v '^\s*$' | \
+ tr '\n' ' ' | \
+ tr -s ' ')
+
+log_info "JVM args: ${JVM_ARGS:0:100}..."
+
+TIMER_STEP2_END=$(timer_now)
+TIMING_STEP2=$(timer_elapsed $TIMER_STEP2_START $TIMER_STEP2_END)
+log_timing "Step 2 - Evaluate JVM args (mvn)" "$TIMING_STEP2"
+# Save to cache for next run
+save_build_cache
+
+fi # end of can_skip_maven else branch
+# =============================================================================
+# Step 3: Replace gluten paths with target/classes
+# =============================================================================
+
+log_step "Step 3: Replacing gluten jar paths with target/classes..."
+
+TIMER_STEP3_START=$(timer_now)
+
+RESOLVED_CLASSPATH=$(replace_gluten_paths "$RAW_CLASSPATH" "$SCALA_VERSION")
+
+# Also add the target module's test-classes at the beginning (highest priority)
+MODULE_TEST_CLASSES="${GLUTEN_HOME}/${MODULE}/target/scala-${SCALA_VERSION}/test-classes"
+MODULE_CLASSES="${GLUTEN_HOME}/${MODULE}/target/scala-${SCALA_VERSION}/classes"
+
+if [[ -d "$MODULE_TEST_CLASSES" ]]; then
+ RESOLVED_CLASSPATH="${MODULE_TEST_CLASSES}:${RESOLVED_CLASSPATH}"
+fi
+if [[ -d "$MODULE_CLASSES" ]]; then
+ RESOLVED_CLASSPATH="${MODULE_CLASSES}:${RESOLVED_CLASSPATH}"
+fi
+
+log_info "Resolved classpath ready"
+
+# Sanity check: no non-.m2 jar files should remain in the classpath
+IFS=':' read -ra FINAL_ENTRIES <<< "$RESOLVED_CLASSPATH"
+for entry in "${FINAL_ENTRIES[@]}"; do
+ if [[ "$entry" == *.jar && "$entry" != *"/.m2/"* ]]; then
+ log_error "Non-.m2 jar found in resolved classpath: $entry"
+ log_error "All gluten jars should be replaced with target/classes or
target/test-classes directories."
+ exit 1
+ fi
+done
+
+TIMER_STEP3_END=$(timer_now)
+TIMING_STEP3=$(timer_elapsed $TIMER_STEP3_START $TIMER_STEP3_END)
+log_timing "Step 3 - Resolve classpath" "$TIMING_STEP3"
+
+# =============================================================================
+# Step 3.5: Export-only mode (if requested)
+# =============================================================================
+
+if [[ "$EXPORT_ONLY" == "true" ]]; then
+ EXPORT_FILE="/tmp/gluten-classpath-exported.txt"
+ echo "$RESOLVED_CLASSPATH" > "$EXPORT_FILE"
+ log_info "✓ Classpath exported to: ${EXPORT_FILE}"
+ log_info " Use with: java <jvm-args> -cp \"\$(cat ${EXPORT_FILE})\"
org.scalatest.tools.Runner -s <suite>"
+ print_timing_summary false
+ exit 0
+fi
+
+# =============================================================================
+# Step 4: Run ScalaTest
+# =============================================================================
+
+log_step "Step 4: Running ScalaTest..."
+
+# Find Java
+JAVA_CMD="${JAVA_HOME}/bin/java"
+if [[ ! -x "$JAVA_CMD" ]]; then
+ JAVA_CMD="java"
+fi
+
+log_info "Suite: ${SUITE}"
+[[ -n "$TEST_METHOD" ]] && log_info "Test method: ${TEST_METHOD}"
+
+echo ""
+echo "=========================================="
+echo "Running ScalaTest"
+echo "=========================================="
+echo ""
+
+# Build test method args conditionally
+TEST_METHOD_ARGS=()
+[[ -n "$TEST_METHOD" ]] && TEST_METHOD_ARGS=(-t "${TEST_METHOD}")
+
+TIMER_STEP4_START=$(timer_now)
+
+TEST_EXIT_CODE=0
+${JAVA_CMD} \
+ ${JVM_ARGS} \
+
-Dlog4j.configurationFile=file:${GLUTEN_HOME}/${MODULE}/src/test/resources/log4j2.properties
\
+ -cp "${RESOLVED_CLASSPATH}" \
+ org.scalatest.tools.Runner \
+ -s "${SUITE}" \
+ "${TEST_METHOD_ARGS[@]}" \
+ -oDF || TEST_EXIT_CODE=$?
+
+TIMER_STEP4_END=$(timer_now)
+TIMING_STEP4=$(timer_elapsed $TIMER_STEP4_START $TIMER_STEP4_END)
+log_timing "Step 4 - ScalaTest execution" "$TIMING_STEP4"
+
+print_timing_summary true
+exit $TEST_EXIT_CODE
diff --git a/gluten-core/pom.xml b/gluten-core/pom.xml
index 0f817143dc..800b7fc21c 100644
--- a/gluten-core/pom.xml
+++ b/gluten-core/pom.xml
@@ -171,6 +171,7 @@
<protocArtifact>com.google.protobuf:protoc:${protobuf.version}:exe:${os.detected.classifier}</protocArtifact>
<protoSourceRoot>src/main/resources/org/apache/gluten/proto</protoSourceRoot>
<clearOutputDirectory>false</clearOutputDirectory>
+ <checkStaleness>true</checkStaleness>
</configuration>
</execution>
</executions>
@@ -189,7 +190,6 @@
<target>
<exec executable="bash" osfamily="unix">
<arg value="${project.basedir}/../dev/gluten-build-info.sh"/>
- <arg value="true"/>
<arg value="--version"/>
<arg value="${project.version}"/>
<arg value="--java"/>
@@ -200,6 +200,8 @@
<arg value="${spark.version}"/>
<arg value="--hadoop"/>
<arg value="${hadoop.version}"/>
+ <arg value="--backend"/>
+ <arg value="${backend_type}"/>
<arg value="--revision"/>
<arg value="true"/>
</exec>
@@ -253,7 +255,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
diff --git a/gluten-delta/pom.xml b/gluten-delta/pom.xml
index a4196d0447..7a8f3f09eb 100755
--- a/gluten-delta/pom.xml
+++ b/gluten-delta/pom.xml
@@ -156,7 +156,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
diff --git a/gluten-hudi/pom.xml b/gluten-hudi/pom.xml
index 4ba2447832..186791856c 100755
--- a/gluten-hudi/pom.xml
+++ b/gluten-hudi/pom.xml
@@ -133,7 +133,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
diff --git a/gluten-iceberg/pom.xml b/gluten-iceberg/pom.xml
index 45b13feb24..1d98909eac 100644
--- a/gluten-iceberg/pom.xml
+++ b/gluten-iceberg/pom.xml
@@ -136,7 +136,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
diff --git a/gluten-kafka/pom.xml b/gluten-kafka/pom.xml
index ec3ca3a61a..f240e9b4ba 100644
--- a/gluten-kafka/pom.xml
+++ b/gluten-kafka/pom.xml
@@ -139,7 +139,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
diff --git a/gluten-paimon/pom.xml b/gluten-paimon/pom.xml
index 21d3b0479e..b16e268041 100644
--- a/gluten-paimon/pom.xml
+++ b/gluten-paimon/pom.xml
@@ -132,7 +132,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
diff --git a/gluten-ras/pom.xml b/gluten-ras/pom.xml
index f52c41ba00..d5f7349d02 100644
--- a/gluten-ras/pom.xml
+++ b/gluten-ras/pom.xml
@@ -103,7 +103,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
diff --git a/gluten-substrait/pom.xml b/gluten-substrait/pom.xml
index 7ee22dd127..0b8d75ddfc 100644
--- a/gluten-substrait/pom.xml
+++ b/gluten-substrait/pom.xml
@@ -216,30 +216,6 @@
<build>
<plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-antrun-plugin</artifactId>
- <executions>
- <execution>
- <id>build-info-with-backends</id>
- <goals>
- <goal>run</goal>
- </goals>
- <phase>generate-resources</phase>
- <configuration>
- <target>
- <exec executable="bash" osfamily="unix">
- <arg value="${project.basedir}/../dev/gluten-build-info.sh"/>
- <arg value="false"/>
- <arg value="--backend"/>
- <arg value="${backend.type}"/>
- <arg value="${backend.home}"/>
- </exec>
- </target>
- </configuration>
- </execution>
- </executions>
- </plugin>
<!-- compile proto buffer files using copied protoc binary -->
<plugin>
<groupId>org.xolstice.maven.plugins</groupId>
@@ -256,6 +232,7 @@
<protocArtifact>com.google.protobuf:protoc:${protobuf.version}:exe:${os.detected.classifier}</protocArtifact>
<protoSourceRoot>src/main/resources/substrait/proto</protoSourceRoot>
<clearOutputDirectory>false</clearOutputDirectory>
+ <checkStaleness>true</checkStaleness>
</configuration>
</execution>
</executions>
@@ -305,7 +282,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
@@ -313,21 +289,4 @@
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
</build>
-
- <profiles>
- <profile>
- <id>backends-velox</id>
- <properties>
- <backend.type>velox</backend.type>
-
<backend.home>${project.basedir}/../ep/build-velox/build/velox_ep</backend.home>
- </properties>
- </profile>
- <profile>
- <id>backends-clickhouse</id>
- <properties>
- <backend.type>ch</backend.type>
- <backend.home>${project.basedir}/../cpp-ch/ClickHouse</backend.home>
- </properties>
- </profile>
- </profiles>
</project>
diff --git a/gluten-ut/common/pom.xml b/gluten-ut/common/pom.xml
index c5fab50454..d08253fe31 100644
--- a/gluten-ut/common/pom.xml
+++ b/gluten-ut/common/pom.xml
@@ -93,7 +93,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
diff --git a/gluten-ut/spark32/pom.xml b/gluten-ut/spark32/pom.xml
index f45b49cf27..d13ae92e81 100644
--- a/gluten-ut/spark32/pom.xml
+++ b/gluten-ut/spark32/pom.xml
@@ -89,7 +89,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
diff --git a/gluten-ut/spark33/pom.xml b/gluten-ut/spark33/pom.xml
index 04ba31be9e..0e1274c3fc 100644
--- a/gluten-ut/spark33/pom.xml
+++ b/gluten-ut/spark33/pom.xml
@@ -95,7 +95,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
diff --git a/gluten-ut/spark34/pom.xml b/gluten-ut/spark34/pom.xml
index 32ca19e99f..dc5c578290 100644
--- a/gluten-ut/spark34/pom.xml
+++ b/gluten-ut/spark34/pom.xml
@@ -95,7 +95,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
diff --git a/gluten-ut/spark35/pom.xml b/gluten-ut/spark35/pom.xml
index e82d8d9086..41fd483dd2 100644
--- a/gluten-ut/spark35/pom.xml
+++ b/gluten-ut/spark35/pom.xml
@@ -102,7 +102,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
diff --git a/gluten-ut/spark40/pom.xml b/gluten-ut/spark40/pom.xml
index 05ecac5967..895f74abec 100644
--- a/gluten-ut/spark40/pom.xml
+++ b/gluten-ut/spark40/pom.xml
@@ -175,7 +175,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
diff --git a/gluten-ut/spark41/pom.xml b/gluten-ut/spark41/pom.xml
index c3de20267d..297602b2a2 100644
--- a/gluten-ut/spark41/pom.xml
+++ b/gluten-ut/spark41/pom.xml
@@ -175,7 +175,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
diff --git a/gluten-ut/test/pom.xml b/gluten-ut/test/pom.xml
index 5867434c74..496f86bb36 100644
--- a/gluten-ut/test/pom.xml
+++ b/gluten-ut/test/pom.xml
@@ -88,7 +88,6 @@
<goals>
<goal>test-jar</goal>
</goals>
- <phase>test-compile</phase>
</execution>
</executions>
</plugin>
diff --git a/pom.xml b/pom.xml
index 21145c3d7d..46638817f0 100644
--- a/pom.xml
+++ b/pom.xml
@@ -97,7 +97,7 @@
-->
<os.full.name>unknown</os.full.name>
<!-- To build built-in backend c++ codes -->
- <scala.recompile.mode>all</scala.recompile.mode>
+ <scala.recompile.mode>incremental</scala.recompile.mode>
<!-- For unit tests -->
<fasterxml.version>2.15.0</fasterxml.version>
<junit.version>4.13.1</junit.version>
@@ -140,7 +140,7 @@
<!-- plugin version-->
<build-helper-maven-plugin.version>3.6.0</build-helper-maven-plugin.version>
- <scala.compiler.version>4.8.0</scala.compiler.version>
+ <scala.compiler.version>4.9.2</scala.compiler.version>
<maven.compiler.plugin>3.14.1</maven.compiler.plugin>
<maven.jar.plugin>3.2.2</maven.jar.plugin>
<scalastyle.version>1.0.0</scalastyle.version>
@@ -570,10 +570,11 @@
<configuration>
<encoding>UTF-8</encoding>
<maxmem>1024m</maxmem>
- <fork>true</fork>
- <compilerArgs>
- <arg>-Xlint:all,-serial,-path</arg>
- </compilerArgs>
+ <!-- Skip javac entirely. In incremental mode
(recompileMode=incremental),
+ Zinc compiles both Scala and Java sources together, making
+ maven-compiler-plugin redundant. Same approach as Apache
Spark. -->
+ <skipMain>true</skipMain>
+ <skip>true</skip>
</configuration>
</plugin>
<plugin>
@@ -628,12 +629,15 @@
</dependencies>
<recompileMode>${scala.recompile.mode}</recompileMode>
<args>
- <arg>-Wconf:msg=While parsing annotations in:silent,any:e</arg>
+ <arg>-Wconf:any:e</arg>
+ <arg>-Wconf:msg=While parsing annotations in:silent</arg>
+ <arg>-Wconf:cat=deprecation:wv</arg>
<arg>-Ywarn-unused:imports</arg>
<arg>-deprecation</arg>
<arg>-feature</arg>
- <arg>-Wconf:cat=deprecation:wv,any:e</arg>
<arg>-P:wartremover:traverser:io.github.zhztheplayer.scalawarts.InheritFromCaseClass</arg>
+ <arg>-Ybackend-parallelism</arg>
+ <arg>8</arg>
</args>
</configuration>
<executions>
@@ -901,7 +905,7 @@
<plugin>
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
- <version>0.5.1</version>
+ <version>0.6.1</version>
</plugin>
<plugin>
<groupId>org.antlr</groupId>
@@ -1023,7 +1027,8 @@
<arg>-feature</arg>
<arg>-explaintypes</arg>
<arg>-release:${java.version}</arg>
- <arg>-Wconf:cat=deprecation:wv,any:e</arg>
+ <arg>-Wconf:any:e</arg>
+ <arg>-Wconf:cat=deprecation:wv</arg>
<arg>-Wunused:imports</arg>
<!--
TODO(SPARK-33805): Undo the corresponding deprecated usage
suppression rule after fixed
@@ -1057,6 +1062,8 @@
<arg>-Wconf:cat=unchecked&msg=eliminated by
erasure:s</arg>
<arg>-Wconf:msg=^(?=.*?a value of type)(?=.*?cannot also
be).+$:s</arg>
<arg>-P:wartremover:traverser:io.github.zhztheplayer.scalawarts.InheritFromCaseClass</arg>
+ <arg>-Ybackend-parallelism</arg>
+ <arg>8</arg>
</args>
</configuration>
<executions>
diff --git a/tools/gluten-it/common/pom.xml b/tools/gluten-it/common/pom.xml
index ba1485a17b..dd8caa1b4f 100644
--- a/tools/gluten-it/common/pom.xml
+++ b/tools/gluten-it/common/pom.xml
@@ -113,9 +113,11 @@
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
- <version>4.8.0</version>
+ <!-- Hardcoded since gluten-it is a standalone third-party module
+ that doesn't inherit parent POM properties -->
+ <version>4.9.2</version>
<configuration>
- <recompileMode>all</recompileMode>
+ <recompileMode>incremental</recompileMode>
</configuration>
<executions>
<execution>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
