This is an automated email from the ASF dual-hosted git repository.
FrankChen021 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new 997a2a95683 ci: cut Docker test job from 40m to 25m by building a lean
distribution (#20297)
997a2a95683 is described below
commit 997a2a9568376e7f88c37af509981263fe2237d0
Author: Frank Chen <[email protected]>
AuthorDate: Tue Sep 15 13:20:19 2026 +0800
ci: cut Docker test job from 40m to 25m by building a lean distribution
(#20297)
* ci: persist Maven build cache for Docker tests
* ci: avoid rebuilding Docker test reactor dependencies
* ci: remove duplicate distribution installs
* ci: address Docker cache review feedback
* ci: build a lean distribution in the Docker job, move release checks to
packaging-check
The Docker test job spent 18 minutes in build-dist because it ran the full
release build (apache-release, rat, and all static checks). Log analysis of
a
master run shows that cyclonedx (13 CPU-min), javadoc (7.7), the license
dependency reports (4), and checkstyle (5) dominated the build, and the
distribution module ran alone for the last 5.6 minutes. None of that is
needed
to produce the tarball for the Docker image, and all of it is already
validated
on the same commit by the Static Checks CI workflow.
- build-dist: build with
dist,bundle-contrib-exts,skip-static-checks,skip-tests,
the same flags as the Dockerfile builder stage.
- packaging-check.sh: enable apache-release (with gpg and dependency-check
skipped) so javadoc, source jars, the source-release assembly and the
license
dependency reports keep a CI run. This job is off the critical path.
- static-checks-maven.sh: comment out license_checks_script.sh since RAT
and the
license checks now run in packaging-check.
- run_docker-tests: drop -am; build-dist has already installed the reactor
in
the same job, and -am re-ran 38 modules including web-console.
* ci: address review: keep -am in Docker tests, keep repo-wide RAT in
static checks
- run_docker-tests: restore -am. On a Maven build-cache hit the cached
segment
includes the install execution, so restored modules such as web-console
are
not published to ~/.m2/repository; -am resolves them from the reactor.
- static-checks-maven.sh: keep a standalone repo-wide apache-rat:check. The
packaging-check job excludes benchmarks from its reactor, so it alone
would
leave that module without license-header validation. Only the license
dependency report generation stays delegated to packaging-check.
* ci: drop the persisted Maven build cache from the Docker job
The build-cache restore/save steps are hard to verify from a PR branch since
the cache is only written on master pushes. Remove them and keep the Docker
job
speedup that is verifiable: the lean build-dist. With no persisted build
cache,
build-dist always installs every reactor module on the fresh runner, so
run_docker-tests can build embedded-tests alone without -am.
* ci: inline the Docker distribution build into the workflow, drop
build-dist
build-dist was only used by the Docker test workflow (it was validate-dist
before #20270) and the name suggests a general-purpose or release build,
which
it no longer is. Inline the single mvn command into docker-tests.yml with an
explanatory comment and delete the script.
* ci: exclude benchmarks from the Docker distribution build
Nothing in the distribution or the Docker tests depends on druid-benchmarks.
* ci: skip the web console build in the Docker test job
No Docker test uses the console, the router tolerates missing console assets
(the unit-test shards already run embedded routers with
web.console.skip=true),
and packaging-check still builds and packages the console on every commit.
web-console was the critical path of the distribution build (3m49s of
5m04s).
---
.github/scripts/build-dist | 21 ---------------------
.github/scripts/openrewrite.sh | 1 -
.github/scripts/packaging-check.sh | 16 ++++++++++++----
.github/scripts/run_docker-tests | 8 +++++++-
.github/scripts/static-checks-maven.sh | 12 ++++++++++--
.github/workflows/docker-tests.yml | 15 +++++++++++++--
6 files changed, 42 insertions(+), 31 deletions(-)
diff --git a/.github/scripts/build-dist b/.github/scripts/build-dist
deleted file mode 100755
index fb77bb1b484..00000000000
--- a/.github/scripts/build-dist
+++ /dev/null
@@ -1,21 +0,0 @@
-#!/bin/bash
-
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-set -e
-set -x
-
-mvn -B -T1C clean install -Papache-release,dist,rat,bundle-contrib-exts
-DskipTests -Ddependency-check.skip -Dgpg.skip
diff --git a/.github/scripts/openrewrite.sh b/.github/scripts/openrewrite.sh
index af0173a43d1..587601267c3 100755
--- a/.github/scripts/openrewrite.sh
+++ b/.github/scripts/openrewrite.sh
@@ -22,6 +22,5 @@ export MAVEN_OPTS=-Xmx8g
echo 'Running Maven install...'
mvn -B clean install -q -ff -pl '!distribution' -P skip-tests
-Dweb.console.skip=true -T1C
-mvn -B install -q -ff -pl 'distribution' -P skip-tests -Dweb.console.skip=true
mvn -B rewrite:dryRun -Dweb.console.skip=true
diff --git a/.github/scripts/packaging-check.sh
b/.github/scripts/packaging-check.sh
index fe9ec3eb0e1..6952b891bd3 100755
--- a/.github/scripts/packaging-check.sh
+++ b/.github/scripts/packaging-check.sh
@@ -19,7 +19,15 @@ set -e
set -x
./.github/scripts/setup_generate_license.sh
-mvn -B clean install -Prat --fail-at-end \
- -pl '!benchmarks, !distribution' -P skip-tests -Dweb.console.skip=false -T1C
-mvn -B install -Prat -Pdist -Pbundle-contrib-exts --fail-at-end \
- -pl 'distribution' -P skip-tests -Dweb.console.skip=false -T1C
+# This job is the single place in CI that validates everything a release build
+# produces: RAT license headers, javadoc and source jars, the binary and source
+# distribution assemblies, and the license dependency reports. The
apache-release
+# profile is enabled here so that the Docker test job only needs to build the
+# binary tarball. GPG signing and the OWASP dependency check are skipped as
they
+# are not meaningful in CI.
+mvn -B clean install -Prat -Papache-release --fail-at-end \
+ -pl '!benchmarks, !distribution' -P skip-tests -Dweb.console.skip=false -T1C
\
+ -Dgpg.skip -Ddependency-check.skip
+mvn -B install -Prat -Papache-release -Pdist -Pbundle-contrib-exts
--fail-at-end \
+ -pl 'distribution' -P skip-tests -Dweb.console.skip=false -T1C \
+ -Dgpg.skip -Ddependency-check.skip
diff --git a/.github/scripts/run_docker-tests b/.github/scripts/run_docker-tests
index 02bce778792..4ce8a6c0fe8 100755
--- a/.github/scripts/run_docker-tests
+++ b/.github/scripts/run_docker-tests
@@ -35,4 +35,10 @@ fi
# No snapshot updates
OPTS+=" -nsu"
-mvn -B -pl embedded-tests -am $OPTS verify -Pdocker-tests,skip-static-checks
-DskipUTs -D$DRUID_IMAGE_SYS_PROPERTY=$DRUID_IMAGE_NAME
"-DjfrProfilerArgLine=$JFR_PROFILER_ARG_LINE" "$@"
+# Only embedded-tests is built here. The Docker workflow builds the
distribution
+# first (mvn install of the whole reactor), which installs every module
+# (including web-console, a test-scoped dependency of embedded-tests) into the
+# local Maven repository on a fresh runner, and no Maven build cache is
persisted
+# across jobs. Adding -am would re-run the whole reactor, including
web-console,
+# a second time.
+mvn -B -pl embedded-tests $OPTS verify -Pdocker-tests,skip-static-checks
-DskipUTs -D$DRUID_IMAGE_SYS_PROPERTY=$DRUID_IMAGE_NAME
"-DjfrProfilerArgLine=$JFR_PROFILER_ARG_LINE" "$@"
diff --git a/.github/scripts/static-checks-maven.sh
b/.github/scripts/static-checks-maven.sh
index 0b643a8b3df..81a79853954 100755
--- a/.github/scripts/static-checks-maven.sh
+++ b/.github/scripts/static-checks-maven.sh
@@ -20,11 +20,19 @@ set -x
echo 'Running Maven install...'
mvn -B clean install -q -ff -pl '!distribution' -P skip-tests
-Dweb.console.skip=true -Dmaven.javadoc.skip=true -T1C
-mvn -B install -q -ff -pl 'distribution' -P skip-tests -Dweb.console.skip=true
-Dmaven.javadoc.skip=true
mvn -B checkstyle:checkstyle --fail-at-end
-./.github/scripts/license_checks_script.sh
+# Repo-wide RAT check. packaging-check also runs RAT, but it excludes the
+# benchmarks module from its reactor, so keep this standalone pass here.
+mvn -B apache-rat:check -Prat --fail-at-end \
+
-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn
\
+ -Drat.consoleOutput=true
+
+# The license dependency reports and check-licenses.py that used to run via
+# license_checks_script.sh are covered by the packaging-check job, which builds
+# the distribution with the apache-release profile. Not repeated here.
+# ./.github/scripts/license_checks_script.sh
./.github/scripts/analyze_dependencies_script.sh
diff --git a/.github/workflows/docker-tests.yml
b/.github/workflows/docker-tests.yml
index ff4580755cd..2f816bb70fd 100644
--- a/.github/workflows/docker-tests.yml
+++ b/.github/workflows/docker-tests.yml
@@ -31,8 +31,19 @@ jobs:
distribution: 'zulu'
java-version: 25
cache: 'maven'
- - name: Build the Druid distribution
- run: .github/scripts/build-dist
+ - name: Build the Druid distribution for the Docker image
+ # Build only what the Docker image needs: the binary distribution
tarball.
+ # Release-only work (javadoc and source jars, source-release assembly,
+ # license dependency reports) and the static checks are validated by
the
+ # Static Checks CI workflow (packaging-check and static-checks-maven),
so
+ # they are skipped here. Same flags as the builder stage in
+ # distribution/docker/Dockerfile. This also installs every reactor
module
+ # into the local Maven repository, which run_docker-tests relies on.
+ # benchmarks is excluded: nothing in the distribution or the Docker
tests
+ # depends on it. The web console assets are skipped too: no Docker test
+ # uses the console, the router tolerates the missing assets, and the
+ # console is fully built and packaged by packaging-check on every
commit.
+ run: mvn -B -T1C clean install -pl '!benchmarks'
-Pdist,bundle-contrib-exts,skip-static-checks,skip-tests -Dweb.console.skip=true
- name: Build the Docker image
run: |
set -o pipefail
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
