morningman opened a new pull request, #66279:
URL: https://github.com/apache/doris/pull/66279
Backport of #65867 to branch-4.1.
### What problem does this PR solve?
Reading a Paimon table over a metastore-backed or REST catalog (HMS / DLF)
fails on
BE whenever the JNI reader is used — most visibly system tables (e.g.
`$snapshots`,
`$files`), which always go through JNI, and also branch / time-travel /
incremental
reads. Two failures were observed, both caused by the BE rebuilding a
catalog it does
not need:
```
# HMS catalog
[JNI_ERROR] NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
# DLF (REST) catalog
[JNI_ERROR] ClassNotFoundException:
com.aliyun.datalake.metastore.hive2.ProxyMetaStoreClient
```
**Root cause.** A table loaded from a metastore-backed Paimon catalog
carries a Paimon
`CatalogLoader` (e.g. `HiveCatalogLoader`) in its `CatalogEnvironment`. When
FE
serializes that table to BE, `SnapshotManager#latestSnapshotId` resolves the
latest
snapshot through the catalog's `SnapshotLoader`, which on BE reconstructs
the catalog's
metastore client and its whole Hive / DLF-REST stack — even though the BE
only reads
(via FE-resolved splits and the object store) and the snapshots already live
there.
On master this became visible once #65733 replaced `java-udf`'s ~122MB
`hive-catalog-shade` with the slim `hive-udf-shade`; branch-4.1 still ships
the fat
shade jar on BE's shared classpath, so there the HMS variant is partly
masked while the
DLF / REST variant and the `FileIOLoader` gap below are not. Backporting
keeps the two
branches aligned and removes the dependency on that jar staying fat.
**Fix 1 — serialize a catalog-less table to the BE (`PaimonScanNode`).**
- data table: rebuild via `FileStoreTableFactory` with an empty
`CatalogEnvironment`.
A `FileStoreTable` is fully defined by fileIO / location / schema, and its
dynamic
options (time travel, incremental) are already merged into the schema by
`copy(...)`,
so nothing is lost except the catalog loader.
- system table: rebuild it over such a catalog-less data table via
`SystemTableLoader`.
With no catalog loader, `SnapshotManager#latestSnapshotId` lists the
snapshot directory
on the filesystem instead of calling the metastore, so the BE never
reconstructs the
catalog and no longer needs any Hive / metastore classes.
**Fix 2 — co-locate the Paimon FileIO plugins with `paimon-connector`
(packaging).**
Reading the snapshot from the filesystem makes the BE materialize the
object-store
`FileIO` lazily via `FileIO.get()` → `ServiceLoader.load(FileIOLoader.class,
...)`,
which eagerly instantiates every registered provider. The OSS/S3 plugins
(`paimon-s3` → `S3Loader`, `paimon-jindo` → `JindoLoader`) were bundled in
`preload-extensions`, on BE's JVM system (app) classpath, but the
`org.apache.paimon.fs.FileIOLoader` interface they implement ships in
`paimon-common`,
bundled only in paimon-connector's own `JniScannerClassLoader`. That loader
is
parent-first, so the app classloader defines `S3Loader` / `JindoLoader` and
cannot
resolve the child-only `FileIOLoader` → `NoClassDefFoundError:
FileIOLoader`, which
aborts discovery. Moving the two plugins into `paimon-connector` puts the
whole FileIO
SPI (interface + all providers) in one classloader; the Jindo SDK stays on
the app
classpath (`start_be.sh` adds `jindofs` to `DORIS_CLASSPATH`) and is still
reachable
via parent delegation.
**Differences from the master PR.** Content-identical apart from the module
rename:
on branch-4.1 the BE Paimon module is `be-java-extensions/paimon-connector`,
not
`paimon-scanner`, so the pom dependencies, the `build.sh` packaging guard
and the
comments refer to `paimon-connector`. The FE change applies unchanged.
Regression tests to re-run:
`io.trino.tests.product.paimon.TestPaimonSparkCompatibility` (system-table
reads) and
`external_table_p2/paimon/test_paimon_dlf_rest_catalog`.
### Release note
Fix Paimon reads (system tables, branch / time-travel / incremental) failing
on BE over
metastore-backed or REST catalogs (HMS / DLF) with `NoClassDefFoundError`
(`HiveConf` / `FileIOLoader`) or `ClassNotFoundException:
ProxyMetaStoreClient`.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
