suxiaogang223 opened a new pull request, #67531:
URL: https://github.com/apache/doris/pull/67531
### What problem does this PR solve?
Problem Summary:
Paimon writes currently go through the Java/JNI writer. This PR introduces
the first phase of a Doris-native Paimon writer for eligible append-only tables
while retaining the JNI writer as a compatibility fallback.
Phase 1 supports unpartitioned, primary-key-free, bucket-unaware append-only
tables using Parquet with metadata statistics disabled. Unsupported table
layouts, options, types, overwrite operations, and row-level changelog writes
automatically fall back to JNI.
The implementation:
- Adds a native Paimon BE write backend based on the existing Arrow Parquet
transformer.
- Builds Arrow schemas with Paimon/Parquet field IDs, including nested
collection field IDs.
- Returns structured native data-file metadata to FE.
- Constructs Paimon `DataFileMeta` and commit messages in FE and commits
through the Paimon Java SDK.
- Adds `paimon_insert_mode`, defaulting to `native` as a best-effort
preference; `jni` forces the existing JNI path.
- Keeps `TPaimonWriteBackendType.FFI` reserved for a future paimon-rust
integration.
- Does not include Variant support; that is planned for phase 2.
### Release note
Support native Parquet writes for eligible Paimon append-only tables.
### Validation
- `mvn -pl fe-core -am -DskipTests -Dcheckstyle.skip=true compile`
- `mvn -pl fe-core -am -DskipTests -Dcheckstyle.skip=true test-compile`
- `/home/socrates/ldb_toolchain/bin/clang-format --dry-run -Werror <changed
BE files>`
- `git diff --check`
BE compilation has intentionally not been run yet. The worktree reuses the
main Doris checkout third-party installation and the local LDB toolchain.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test
- [ ] No need to test or manual test.
- Behavior changed:
- [ ] No.
- [x] Yes. Eligible Paimon append-only inserts prefer the native writer
by default and otherwise fall back to JNI. Set `paimon_insert_mode=jni` to
force JNI.
- Does this need documentation?
- [ ] No.
- [x] Yes. The new session variable and supported native-write scope
need documentation before the PR is marked ready.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
