morningman opened a new pull request, #67770:
URL: https://github.com/apache/doris/pull/67770
### What problem does this PR solve?
Issue Number: None
Related PR: #66770
Problem Summary:
Groundwork for an FE kernel that ships no Hadoop classes at all — the end
state
`ConnectorPluginManager` already names in its parent-first comment ("the
intended end state is
an FE kernel with no hadoop classes at all, every plugin bringing its own"),
and the one the
authorization plugin family already reached: `org.apache.hadoop.` is
child-first there and the
Ranger plugins bundle their own Hadoop.
**Nothing is removed from `fe/lib` here.** This PR only makes the remaining
coupling honest, so
that the step which does remove it cannot silently take unrelated libraries
with it.
#### 1. fe-common stops depending on Hadoop
fe-common declared `hadoop-common` (provided) and `hadoop-aws` while not
naming a single Hadoop
class — the module has zero `org.apache.hadoop` references in source, only
two configuration-key
strings in `Config`.
Dropping them exposed one real user: `Hll` uses
`org.apache.commons.codec.binary.StringUtils` and
had been riding on hadoop-common's `commons-codec`. That is now declared
where it is used (the
version is already managed in `fe/pom.xml`, and fe-core has long declared
the same artifact).
`hadoop-aws` was compile scope, so it also stops being re-exported to
fe-type, fe-catalog,
hive-udf and `be-java-extensions/java-common` — none of which reference
`org.apache.hadoop.fs.s3a`
either. There is no `import org.apache.hadoop.fs.s3a...` anywhere in the
repository; every mention
is a configuration-value string such as
`"org.apache.hadoop.fs.s3a.S3AFileSystem"`, produced by
filesystem plugins that declare `hadoop-aws` themselves. fe-core is
unaffected: it declares
`hadoop-aws` directly.
#### 2. fe-core's hidden Hadoop passengers are declared
fe-core likewise names no Hadoop class in `src/main` — its six
`org.apache.hadoop` occurrences are
comments and the two parent-first prefix lists. But two libraries it
compiles against directly
reach it *only* as Hadoop transitives:
| Library | Arrives via | Used by |
|---|---|---|
| `io.dropwizard.metrics:metrics-core` (`com.codahale.metrics`) |
`hadoop-auth` | the whole metric layer: `MetricRepo`, every `MetricVisitor`,
`HistogramMetric`, `CloudMetrics`, `SqlBlockRule` |
| `org.bouncycastle:bcprov-jdk18on` (`org.bouncycastle.util`) |
`hadoop-common` | `TableScanParams` |
Both are now declared in fe-core. They are already on this classpath at
these versions, so no jar
is added to `fe/lib`; the point is that the FE metric layer should not be a
passenger of a
dependency that says nothing about it.
#### 3. The Hadoop auth property names move to fe-foundation
fe-core read three property names from
`org.apache.doris.kerberos.AuthenticationConfig`, a class
that imports `org.apache.hadoop.conf.Configuration`. The names are plain
strings; the code that
turns them into a `UserGroupInformation` is what needs Hadoop.
They move to a new hadoop-free holder,
`org.apache.doris.foundation.security.HadoopAuthConfigKeys`.
`AuthenticationConfig` re-exports every constant declared there, so both
spellings name the same
property and no existing caller changes meaning.
This is not only a compile-time tidy-up. The old copies were `public static
String` — not
compile-time constants — so reading one emitted a `getstatic`, and
initialising
`HdfsStorageVault.PropertyKey` or running `HdfsResource.generateHdfsParam`
loaded
`AuthenticationConfig`, and with it Hadoop's `Configuration`, just to obtain
a string. They no
longer do. fe-core keeps its fe-kerberos dependency for
`ExecutionAuthenticator`, which is
hadoop-free.
### Release note
None
### Check List (For Author)
- Test
- [x] No need to test or manual test. Explain why:
- [x] This is a refactor/code format and no logic has been changed.
- [x] Previous test can cover this change.
Verification performed:
- `mvn -pl fe-core -am package` — all 67 modules build, fe-core's 4470
main sources compile,
**0 Checkstyle violations** in every module.
- `mvn -pl hive-udf,be-java-extensions/java-common -am package` — BUILD
SUCCESS, confirming the
other fe-common consumers do not need the re-exported `hadoop-aws`.
- `mvn -pl fe-core dependency:list -DincludeScope=runtime` is
**byte-for-byte identical** before
and after this change — 395 artifacts — so `fe/lib` does not change.
fe-common, fe-type and
fe-catalog lose `hadoop-aws` and its two transitives
(`wildfly-openssl`,
`analyticsaccelerator-s3`) and gain `commons-codec`.
- No dependency is new to the build: all three declared artifacts
already resolve into it today
as transitives, at the same versions.
Note on FE UT: `fe-core`'s `testCompile` currently fails on master,
independently of this PR, at
`IvmNormalizeMTMVJoinTest.java:1053` — `IvmRewriteContext`'s constructor
took eight parameters as
of #67646 while the call site added by #67669 passes three. A textual
merge of two green PRs; it
needs its own fix.
- Behavior changed:
- [x] No.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01XYs8EpL1D6bjufhYSt3mC4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
