suxiaogang223 opened a new pull request, #65364:
URL: https://github.com/apache/doris/pull/65364
## Summary
This PR implements the v1 of Paimon JNI write support for Doris, based on
the architecture design in `PAIMON_WRITE_DESIGN.md`.
### Architecture
The implementation follows a layered architecture with an abstraction layer
(`IPaimonWriteBackend`) that allows future Rust FFI backend to be plugged in
without changing upper-layer code.
```
VPaimonTableWriter (AsyncResultWriter)
└── IPaimonWriteBackend (abstraction)
├── JniPaimonWriteBackend ← v1 (complete)
└── FfiPaimonWriteBackend ← v2 (stub)
```
### Changes (10 commits, 30+ files, ~4000 lines)
| Layer | Files | Description |
|-------|-------|-------------|
| **Documentation** | 1 | Architecture design document (v4.0, 1661 lines) |
| **Thrift** | 2 | TPaimonTableSink, TPaimonCommitMessage, RPC wire |
| **BE Abstraction** | 3 |
IPaimonWriteBackend/IPaimonWriter/IPaimonCommitter |
| **BE JNI Backend** | 2 | JniPaimonWriteBackend (Arrow IPC bridge) |
| **BE FFI Stub** | 2 | FfiPaimonWriteBackend (reserved for Rust v2) |
| **BE Writer** | 3 | VPaimonTableWriter + Pipeline Operator + Factory |
| **BE RPC** | 2 | RuntimeState + PipelineFragmentContext wiring |
| **FE Planner** | 7 | Nereids Logical/Physical/Unbound + PaimonTableSink |
| **FE Transaction** | 5 | PaimonTransaction + PaimonInsertExecutor + wiring
|
| **Java** | 2 | PaimonJniWriter (Arrow IPC → Paimon SDK) |
### Verified
- ✅ FE Thrift compilation
- ✅ Java paimon-scanner module compilation
- ✅ FE checkstyle: 0 violations across all Paimon modules
- ⬜ BE compilation (pending CI — local environment unavailable)
### Design Highlights
- **Write path**: Doris Block → Arrow IPC → JNI → Paimon BatchTableWrite
- **Commit path**: prepareCommit() → DPCM-framed byte[][] → FE
PaimonTransaction → StreamTableCommit.filterAndCommit()
- **Supported**: HASH_FIXED and BUCKET_UNAWARE bucket modes, all Paimon
types (primitives, DECIMAL, TIMESTAMP, ARRAY, MAP, STRUCT)
- **v1 limitations**: HASH_DYNAMIC / KEY_DYNAMIC / POSTPONE_MODE explicitly
rejected (will be added later)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
