morningman opened a new pull request, #68000:
URL: https://github.com/apache/doris/pull/68000
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #66729 (the plugin layout this breaks under the build cache),
#60964 (the hadoop-deps-only fallback this removes)
Problem Summary:
Since #66729 merged (2026-09-15 10:52 CST), the TeamCity **Compile** step
fails for one PR after another with a few hundred `closure-self-contained`
problems from `check_plugin_layout.py`, e.g. build 121702 (pull/67966), 121701
/ 121695 (pull/67551), 121700 / 121689 (pull/67675):
```
checking hudi 1 jars
checking iceberg 1 jars
checking java-udf 28 jars
checking java-writer 1 jars
checking jdbc 1 jars
checking max-compute 1 jars
checking paimon 1 jars
checking trino-connector 1 jars
322 problem(s):
[closure-self-contained] plugin 'trino-connector':
io.trino.SystemSessionProperties is referenced by
org.apache.doris.trinoconnector.TrinoConnectorJniScanner but is in no jar of
this plugin directory. ...
Error: the plugin tree just deployed breaks the isolation rules; see above.
```
(The `output/ms missing` error the CI wrapper prints afterwards is a
symptom: `build.sh` exits at the plugin check, before it copies `cloud/output`
to `output/ms`.)
**Root cause.** Every BE Java extension now deploys as a thin module jar
plus the runtime closure that `maven-dependency-plugin:copy-dependencies`
writes to `target/lib` at the `package` phase, and `build.sh` copies that
directory verbatim into `plugins/jni/<name>`. The Maven build cache restores a
module's jar on a hit and nothing else — the log says so outright:
```
Found cached build, restoring org.apache.doris:hadoop-hudi-scanner from
cache by checksum c6118376d1a7c373
Skipping plugin execution (cached): clean:clean
Skipping plugin execution (cached): jar:jar
Skipping plugin execution (cached): dependency:copy-dependencies
```
On a working copy that is harmless, because `target/lib` is still there from
the module's last real build — which is what the comment in
`be-java-extensions/pom.xml` relied on. CI is the opposite case: TeamCity runs
`git clean -f -d -x` before every build, so `target/` is always fresh, while
`/home/work/.m2` (build cache included) is mounted from the host and survives.
So the first build on each agent after #66729 built the plugins for real and
seeded the cache (e.g. 121674 on agent 172.16.0.115: hudi 238 / iceberg 148 /
jdbc 13 / paimon 155 / trino 99 jars, green), and the very next build on the
same agent (121702, same checksums, `Found cached build, restoring ...`)
deployed each plugin as a single jar and failed. Only a module whose inputs the
PR happened to touch — `java-udf` behind a `Config.java` change, since it
depends on `fe-common` — was rebuilt and came out whole, which is why the count
is 322 problems for some PRs and 352 for others.
**Fix.** Force the goal under `runAlways` in
`fe/.mvn/maven-build-cache-config.xml`, the same mechanism that already keeps
checkstyle running on a cache hit. It is matched by goal rather than execution
id so the plugins' `copy-plugin-dependencies` and hadoop-deps'
`copy-dependencies` are covered alike, and so is any module that adds the goal
later. The `wipe-plugin-lib` clean execution that precedes it is forced too:
the two exist as a pair, and a copy without the wipe would deploy the union of
every dependency set the directory has ever held. Forcing the goal costs a copy
out of the local repository; the alternative — keeping the closure in the cache
as attached outputs — would store hundreds of jars per plugin per checksum.
`build.sh` had a fallback for exactly this, for hadoop-deps only (#60964):
re-run the goal from the command line when `target/lib` was missing. It goes,
for two reasons. The forced execution covers hadoop-deps as well; and the
fallback ran the goal without the pom's runtime-scope filter, so the
`be/lib/hadoop_hdfs/lib` it produced carried the test-scope closure — the JUnit
5 stack, hamcrest, awaitility, opentest4j, plus provided-scope zookeeper and
the netty QUIC natives — in every CI build where hadoop-deps hit the cache (167
jars, where the pom's execution yields 149; build 121702 shows the junit jars
being copied into `output/be/lib/hadoop_hdfs/lib/`).
The comment in `be-java-extensions/pom.xml` that described the old (wrong)
assumption is corrected to point at the forced executions.
### Release note
None
### Check List (For Author)
- Test
- [x] Manual test (add detailed scripts or steps below)
On master@846d9b2ebfc, `cd fe && mvn -pl
be-java-extensions/jdbc-scanner,be-java-extensions/hadoop-deps -am package
-DskipTests -Dcheckstyle.skip=true`, simulating CI's `git clean -fdx` with `rm
-rf` of the modules' `target/` between runs:
| run | config | jdbc-scanner | hadoop-deps |
|---|---|---|---|
| 1 – real build (seeds cache) | old | `Local build was not found by
checksum 2c03b13c59c59c7b` → `target/lib` 12 jars | — |
| 2 – rm target, rebuild | old | `Found cached build, restoring ...
2c03b13c59c59c7b`, `Skipping plugin execution (cached):
dependency:copy-dependencies` → `target/` holds only `jdbc-scanner.jar`, **no
lib** (the CI failure) | — |
| 3 – rm target, rebuild | **new** | same cache hit, `Mojo execution
is forced by project property: clean:clean` / `dependency:copy-dependencies` →
`target/lib` 12 jars, byte-for-byte the same list as run 1 | real build, 149
jars |
| 4 – rm target, rebuild | **new** | cache hit, 12 jars, same list |
`Found cached build, restoring ... 7df78f352270b07a`, forced copy → 149 jars,
same list as run 3 |
The config change does not alter module checksums (run 3 hits the
entry run 1 wrote), so existing cache entries on the CI agents keep being used
and simply regain their `target/lib`.
- Behavior changed:
- [x] No.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01Ja986K4PEm2LD8u44L9jd2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
