suxiaogang223 opened a new pull request, #70806: URL: https://github.com/apache/doris/pull/70806
### Background Doris reads safe Paimon raw files with its Native ORC/Parquet readers. Rust and JNI handle logical splits that require table-format semantics. ### What problem does this PR solve? Issue Number: #70797 Related PR: None PR Type: Enhancement Problem Summary: Enabling Rust currently diverts compatible raw files from Native and conditionally skips Native historical timestamp checks. Complete Native eligibility first, then consider Rust and JNI. Refresh integer and collect eligibility assumptions against paimon-rust 0.5.0 while retaining independent resource gates. ### Details - **Change map:** 82 changed production lines adjust routing and aggregate compatibility; 225 changed test lines cover precedence, non-raw column merging, compatibility gates and regression profiles. No generated files or dependency pins change. - **Before and after:** Ordinary raw-convertible ORC/Parquet scans remain Native when Rust is enabled. Unsupported Native splits still consider compatible Rust, then JNI. Metadata counts and Native footer aggregates retain their fast paths. - **Configs and defaults:** No new configuration or default change. Rust remains opt-in, forcing JNI retains precedence, and diagnostic split suppression drops ranges without rerouting them. - **Impact on existing users:** Integer sum/product and non-distinct ARRAY collect become recognized by the compatibility helper. Independent gates still exclude unsupported primary-key merge resources, projections and storage configurations. - **Behavior change:** Reader preference becomes Native > compatible/enabled Rust > JNI after metadata COUNT. Native historical time/schema checks always complete before selection. ### Testing - `./build.sh --fe --be-java-extensions -j 2` and `./build.sh --be -j 4`: passed, including Checkstyle; BE used ASAN and the latest official thirdparty prebuilt. - Preset FE runner: 152 tests passed, zero failures/errors/skips. The Native-priority assertion fails on the original implementation. A real two-file column merge with overlapping row IDs selects Rust, then disabled/forced JNI. - Preset local regression passed script-generated output and verification: ordered results, 21 COMPLETE reader profiles, Native ORC/Parquet, Rust column merges, JNI parity, both deprecated scanner flag values, metadata count and footer/full-scan controls. - Independent prebuilt C ABI probes matched Java engine/format controls, integer overflow behavior and non-distinct collect. Distinct collect ordering differs and remains gated. These probes are separate from integrated Doris execution. - Regression framework and changed Groovy syntax checks passed. The two changed external HDFS/MinIO suites have not run. ### Notes for Reviewers The local cluster used an 8 GiB process limit and a 256 MiB memory file cache: Btrfs zero inode counts triggered an existing disk-cache monitor division by zero during initial startup. Disk-cache behavior remains unverified. SDK/C ABI patch-dependent functionality is deferred. Merge budgets, cancellation, recursive read types/Variant, FileIO and incremental integration remain tracked in #70797 and [paimon-rust #1468](https://github.com/apache/paimon-rust/issues/1468). This PR does not complete the full 0.5.0 integration or add runtime JNI retries. ### Release note Enabling the Paimon Rust reader preserves Native execution for compatible raw ORC/Parquet files, including footer aggregates. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes (reader-selection documentation update tracked with #70797) - Pre-submission status: Partially verified: local reader results/profiles passed; changed external HDFS/MinIO suites have not run. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Code, tests, analysis and this description. - Human review: not stated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
