This is an automated email from the ASF dual-hosted git repository.
CalvinKirs pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1a3b141824b [fix](build) Run copy-dependencies on a build-cache hit so
a cache-restored BE plugin still deploys its dependencies (#68000)
1a3b141824b is described below
commit 1a3b141824bf4c8a0d77b95338a6978360282f24
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Tue Sep 15 17:04:22 2026 +0800
[fix](build) Run copy-dependencies on a build-cache hit so a cache-restored
BE plugin still deploys its dependencies (#68000)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #66729 (the plugin layout this breaks under the build
cache), #60964 (the hadoop-deps-only fallback this removes)
Problem Summary:
Since #66729 merged (2026-09-15 10:52 CST), the TeamCity **Compile**
step fails for one PR after another with a few hundred
`closure-self-contained` problems from `check_plugin_layout.py`, e.g.
build 121702 (pull/67966), 121701 / 121695 (pull/67551), 121700 / 121689
(pull/67675):
```
checking hudi 1 jars
checking iceberg 1 jars
checking java-udf 28 jars
checking java-writer 1 jars
checking jdbc 1 jars
checking max-compute 1 jars
checking paimon 1 jars
checking trino-connector 1 jars
322 problem(s):
[closure-self-contained] plugin 'trino-connector':
io.trino.SystemSessionProperties is referenced by
org.apache.doris.trinoconnector.TrinoConnectorJniScanner but is in no jar of
this plugin directory. ...
Error: the plugin tree just deployed breaks the isolation rules; see above.
```
(The `output/ms missing` error the CI wrapper prints afterwards is a
symptom: `build.sh` exits at the plugin check, before it copies
`cloud/output` to `output/ms`.)
**Root cause.** Every BE Java extension now deploys as a thin module jar
plus the runtime closure that
`maven-dependency-plugin:copy-dependencies` writes to `target/lib` at
the `package` phase, and `build.sh` copies that directory verbatim into
`plugins/jni/<name>`. The Maven build cache restores a module's jar on a
hit and nothing else — the log says so outright:
```
Found cached build, restoring org.apache.doris:hadoop-hudi-scanner from
cache by checksum c6118376d1a7c373
Skipping plugin execution (cached): clean:clean
Skipping plugin execution (cached): jar:jar
Skipping plugin execution (cached): dependency:copy-dependencies
```
On a working copy that is harmless, because `target/lib` is still there
from the module's last real build — which is what the comment in
`be-java-extensions/pom.xml` relied on. CI is the opposite case:
TeamCity runs `git clean -f -d -x` before every build, so `target/` is
always fresh, while `/home/work/.m2` (build cache included) is mounted
from the host and survives. So the first build on each agent after
#66729 built the plugins for real and seeded the cache (e.g. 121674 on
agent 172.16.0.115: hudi 238 / iceberg 148 / jdbc 13 / paimon 155 /
trino 99 jars, green), and the very next build on the same agent
(121702, same checksums, `Found cached build, restoring ...`) deployed
each plugin as a single jar and failed. Only a module whose inputs the
PR happened to touch — `java-udf` behind a `Config.java` change, since
it depends on `fe-common` — was rebuilt and came out whole, which is why
the count is 322 problems for some PRs and 352 for others.
**Fix.** Force the goal under `runAlways` in
`fe/.mvn/maven-build-cache-config.xml`, the same mechanism that already
keeps checkstyle running on a cache hit. It is matched by goal rather
than execution id so the plugins' `copy-plugin-dependencies` and
hadoop-deps' `copy-dependencies` are covered alike, and so is any module
that adds the goal later. The `wipe-plugin-lib` clean execution that
precedes it is forced too: the two exist as a pair, and a copy without
the wipe would deploy the union of every dependency set the directory
has ever held. Forcing the goal costs a copy out of the local
repository; the alternative — keeping the closure in the cache as
attached outputs — would store hundreds of jars per plugin per checksum.
`build.sh` had a fallback for exactly this, for hadoop-deps only
(#60964): re-run the goal from the command line when `target/lib` was
missing. It goes, for two reasons. The forced execution covers
hadoop-deps as well; and the fallback ran the goal without the pom's
runtime-scope filter, so the `be/lib/hadoop_hdfs/lib` it produced
carried the test-scope closure — the JUnit 5 stack, hamcrest,
awaitility, opentest4j, plus provided-scope zookeeper and the netty QUIC
natives — in every CI build where hadoop-deps hit the cache (167 jars,
where the pom's execution yields 149; build 121702 shows the junit jars
being copied into `output/be/lib/hadoop_hdfs/lib/`).
The comment in `be-java-extensions/pom.xml` that described the old
(wrong) assumption is corrected to point at the forced executions.
### Release note
---
build.sh | 13 ++++++-------
fe/.mvn/maven-build-cache-config.xml | 30 +++++++++++++++++++++++++++++-
fe/be-java-extensions/pom.xml | 7 +++++--
3 files changed, 40 insertions(+), 10 deletions(-)
diff --git a/build.sh b/build.sh
index d2f701bb808..55c5bacb919 100755
--- a/build.sh
+++ b/build.sh
@@ -1591,13 +1591,12 @@ if [[ "${OUTPUT_BE_BINARY}" -eq 1 ]]; then
mkdir "${BE_HADOOP_HDFS_DIR}"
HADOOP_DEPS_JAR_DIR="${DORIS_HOME}/fe/be-java-extensions/${HADOOP_DEPS_NAME}/target"
echo "HADOOP_DEPS_JAR_DIR: ${HADOOP_DEPS_JAR_DIR}"
- if [[ "${BUILD_BE_JAVA_EXTENSIONS}" -eq 1 && ! -d
"${HADOOP_DEPS_JAR_DIR}/lib" ]]; then
- echo "WARN: lib directory missing (likely due to Maven cache).
Regenerating..."
- pushd "${DORIS_HOME}/fe/be-java-extensions/${HADOOP_DEPS_NAME}"
- "${MVN_CMD}" dependency:copy-dependencies -DskipTests
-Dcheckstyle.skip=true
- mv target/dependency target/lib
- popd
- fi
+ # target/lib is present even when the Maven build cache restored
this module instead of
+ # building it: fe/.mvn/maven-build-cache-config.xml forces
copy-dependencies to run on a
+ # cache hit, for hadoop-deps and every plugin alike. There used to
be a fallback here
+ # that re-ran the goal from the command line when the directory
was missing; besides
+ # covering only this one module, it ran the goal without the pom's
runtime-scope
+ # filter and so put the test-scope closure (JUnit and friends)
into lib/hadoop_hdfs.
if [[ -f "${HADOOP_DEPS_JAR_DIR}/${HADOOP_DEPS_NAME}.jar" ]]; then
echo "Copy Be Extensions hadoop deps jar to
${BE_HADOOP_HDFS_DIR}"
cp "${HADOOP_DEPS_JAR_DIR}/${HADOOP_DEPS_NAME}.jar"
"${BE_HADOOP_HDFS_DIR}"
diff --git a/fe/.mvn/maven-build-cache-config.xml
b/fe/.mvn/maven-build-cache-config.xml
index 59c73b6dafb..3643d9a32b1 100644
--- a/fe/.mvn/maven-build-cache-config.xml
+++ b/fe/.mvn/maven-build-cache-config.xml
@@ -63,14 +63,42 @@ under the License.
</input>
<executionControl>
<runAlways>
- <!-- Checkstyle is a quality gate that must always execute; never
use a cached result. -->
<executions>
+ <!-- Checkstyle is a quality gate that must always execute;
never use a cached result. -->
<execution artifactId="maven-checkstyle-plugin">
<execIds>
<execId>validate</execId>
</execIds>
</execution>
+ <!-- The wipe of target/lib that be-java-extensions/pom.xml
binds right before the
+ copy-dependencies forced below. The two run as a pair or
not at all: a copy
+ without the wipe deploys the union of every dependency
set the directory has
+ ever held, which is the drift that wipe exists to
prevent. -->
+ <execution artifactId="maven-clean-plugin">
+ <execIds>
+ <execId>wipe-plugin-lib</execId>
+ </execIds>
+ </execution>
</executions>
+ <goalsLists>
+ <!-- A cache hit restores a module's jar and nothing else:
whatever a skipped mojo
+ wrote elsewhere under target/ stays missing. Every BE
Java plugin (and
+ hadoop-deps) deploys as a thin jar plus the runtime
closure copy-dependencies
+ writes to target/lib, which build.sh copies verbatim, so
a cache-restored
+ plugin would ship with no dependencies at all and fail
the closure rule of
+ tools/be-java-plugins/check_plugin_layout.py. Harmless on
a working copy, which
+ still holds the target/lib of its last real build; fatal
in CI, which wipes
+ target/ before every build (git clean -fdx) while
~/.m2/build-cache survives -
+ the first build on an agent seeds the cache and every
later one deploys empty
+ plugins. Forcing the goal only copies jars out of the
local repository, far
+ cheaper than keeping the closure (hundreds of jars per
plugin) in the cache as
+ attached outputs. -->
+ <goalsList artifactId="maven-dependency-plugin">
+ <goals>
+ <goal>copy-dependencies</goal>
+ </goals>
+ </goalsList>
+ </goalsLists>
</runAlways>
</executionControl>
</cache>
diff --git a/fe/be-java-extensions/pom.xml b/fe/be-java-extensions/pom.xml
index 6758f919a8a..5b61be24269 100644
--- a/fe/be-java-extensions/pom.xml
+++ b/fe/be-java-extensions/pom.xml
@@ -106,8 +106,11 @@ under the License.
copy-dependencies refills it makes an incremental build's plugin
directory equal to a
clean one's.
- A cache-restored module skips both executions and keeps the
target/lib its own last
- real build wrote, which is consistent for the same reason.
+ Both executions are also forced under runAlways in
fe/.mvn/maven-build-cache-config.xml.
+ A build-cache hit restores the module jar and nothing else, so
without that a cache-
+ restored module has no target/lib at all: harmless on a working
copy, which still
+ holds the directory its last real build wrote, but CI wipes
target/ before every
+ build while the cache survives, and deployed every plugin empty.
-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
