This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 79bc8db7524 [fix](hudi) Align preloaded Parquet format classes on
branch-4.1 (#68856)
79bc8db7524 is described below
commit 79bc8db7524e9ab8d718e9059a0654bad2626823
Author: Gabriel <[email protected]>
AuthorDate: Sat Oct 10 15:24:01 2026 +0800
[fix](hudi) Align preloaded Parquet format classes on branch-4.1 (#68856)
### What problem does this PR solve?
Hudi JNI scans on branch-4.1 fail with `NoSuchMethodError:
org.apache.parquet.format.Statistics.isSetNan_count()` after the Parquet
1.18.0 upgrade (#68557).
The BE preload assembly unpacks both the explicitly pinned
`parquet-format:2.12.0` and `parquet-hadoop-bundle:1.18.0`. The former
supplies an older `Statistics` class, while the latter supplies a reader
that calls a method absent from that class. The scanner inherits these
incompatible classes from its parent classloader.
Remove the redundant standalone format dependency so the format classes
and readers come from the same Parquet bundle. Keep the existing Parquet
version and classloader behavior.
### Release note
Fix Hudi JNI scan failures caused by incompatible Parquet classes in the
BE preload jar.
### Check List (For Author)
- Test
- [x] Manual test (details below)
- Behavior changed:
- [x] Yes. Hudi JNI scans can convert Parquet statistics without the
missing-method error.
- Does this need documentation?
- [x] No.
### Validation
- Assembled the unmodified preload module and reproduced the
missing-method error by converting INT64 statistics.
- Cleanly reassembled the modified module with `mvn -pl
be-java-extensions/preload-extensions clean
assembly:single@make-assembly-deps` from `fe/`.
- Ran the same statistics conversion against the rebuilt jar, with
Hadoop Common and JTS Core runtime dependencies on the classpath; it
passed. Both `Statistics` and `ParquetMetadataConverter` loaded from the
rebuilt preload jar.
- Compared all 1,795 `org/apache/parquet/**/*.class` entries from the
bundle against the rebuilt assembly; all matched byte for byte.
- Passed `mvn -pl be-java-extensions/preload-extensions -am
checkstyle:check` and checked the reactor dependency tree.
The statistics conversion probe uses:
```java
Statistics stats = new Statistics();
stats.setNull_count(0);
ParquetMetadataConverter.fromParquetStatistics(
"parquet-mr version 1.18.0", stats, PrimitiveTypeName.INT64);
```
Full Hudi integration tests were not run locally. Existing Hudi catalog,
incremental, time-travel, snapshot, and schema-evolution regression
suites cover the affected JNI path and should be rerun in CI.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
fe/be-java-extensions/preload-extensions/pom.xml | 6 +-----
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/fe/be-java-extensions/preload-extensions/pom.xml
b/fe/be-java-extensions/preload-extensions/pom.xml
index 772a98dd653..673069eb2d8 100644
--- a/fe/be-java-extensions/preload-extensions/pom.xml
+++ b/fe/be-java-extensions/preload-extensions/pom.xml
@@ -36,11 +36,7 @@ under the License.
</properties>
<dependencies>
- <dependency>
- <groupId>org.apache.parquet</groupId>
- <artifactId>parquet-format</artifactId>
- <version>2.12.0</version>
- </dependency>
+ <!-- Keep Parquet format classes and readers in the same bundle to
avoid binary incompatibility. -->
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-hadoop-bundle</artifactId>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
