pzhdfy opened a new pull request, #65883:
URL: https://github.com/apache/doris/pull/65883
### What problem does this PR solve?
Problem Summary:
Adds a third BE-side reader for Paimon FORMAT_JNI splits, backed by the
`paimon-rust` C bindings (`libpaimon_c.a`, statically linked). Reader
priority is now `rust > cpp > jni`, gated by the session variable
`enable_paimon_rust_reader` (default `false`).
Compared with the existing readers:
- **vs paimon-jni**: no JVM round-trip on the hot path — the split is
deserialized in-process and rows arrive via the Arrow C Data Interface,
which shortens the scanner critical section and removes JNI copy cost.
- **vs paimon-cpp**: reuses paimon's official Rust implementation instead
of a bespoke C++ port. paimon-rust tracks upstream paimon spec directly
(schema, DV, snapshots, predicates), so features/fixes propagate without
a second reimplementation.
**Architecture**
FE (`PaimonScanNode`): when `enable_paimon_rust_reader` is on and the split
is a `DataSplit`, encode it with paimon's native binary serialization
(`PaimonUtil.encodeDataSplitToString`), stamp the range with
`TPaimonReaderType.PAIMON_RUST`, and attach `db_name` / `table_name` /
`paimon_table` so BE can open the table through a catalog.
BE (`FileScanner::_get_next_reader`): dispatches on `reader_type`. On
`PAIMON_RUST` it instantiates `PaimonRustReader`, which:
1. `paimon_catalog_create({warehouse, metastore=filesystem, ...})`
- warehouse is derived from the table root path by stripping
`/<db>.db/<table>` (paimon FS convention).
2. `paimon_identifier_new(db, tbl)` → `paimon_catalog_get_table`
3. `paimon_table_new_read_builder` → projection (case-insensitive) →
optional filter from `PaimonRustPredicateConverter` (translates Doris
push-down conjuncts into paimon-rust predicates, best-effort so any
non-representable conjunct is silently dropped and re-applied by the
engine).
4. `paimon_plan_from_split_bytes(split_bytes)` — one BE scanner reads
exactly its FE-planned split (no re-planning).
5. `paimon_table_read_to_arrow` → streamed arrow record batches → block.
Partition columns are back-filled in `on_after_read_block` from
FE-provided `columns_from_path*`, mirroring `PaimonCppReader`. OSS/S3
hadoop-style credential keys (`fs.oss.*` / `fs.s3a.*`) are remapped to
`AWS_ACCESS_KEY` / `AWS_ENDPOINT` / ... so paimon-rust's FileIO can
authenticate.
**Thirdparty**
- Introduces `paimon-rust-2fb5e49` (built with Rust 1.91.0, feature
`paimon/storage-hdfs`, `--locked`).
- Auto-installs `cbindgen` and regenerates the C header from the Rust
extern "C" surface via a locally-written `cbindgen.toml`
(`language=C, include_guard=PAIMON_C_H, pragma_once=true,
cpp_compat=true`),
so the shipped `paimon.h` always matches the built `.a`.
### Release note
Add `enable_paimon_rust_reader` session variable. When enabled, Paimon
non-native reads on BE use the paimon-rust reader instead of paimon-jni /
paimon-cpp.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
1. `CREATE CATALOG paimon_fs PROPERTIES ("type"="paimon",
"paimon.catalog.type"="filesystem", "warehouse"="hdfs://.../warehouse");`
2. `SET enable_paimon_rust_reader=true;`
3. `SELECT` from a partitioned/DV paimon table; results must match
`enable_paimon_rust_reader=false`.
4. `SELECT count(*)` — should hit table-level row count push-down.
5. `WHERE a=1 AND b>2` — BE INFO log should show `paimon-rust
predicate pushdown: applied`.
- [ ] Regression test
- [ ] No need to test or manual test.
- Behavior changed:
- [x] No. Guarded by `enable_paimon_rust_reader` (default `false`);
when off, all paths are unchanged.
- Does this need documentation?
- [ ] No.
- [x] Yes. <!-- Add doris-website PR link here for the new session
variable -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
