damokelis opened a new issue, #66784:
URL: https://github.com/apache/doris/issues/66784
### Search before asking
I searched existing issues and PRs. #65086 tracks general Paimon write
support, and #65868 implemented the Java/JNI write path for `branch-4.1`. This
issue focuses on bringing that capability to the current `master` architecture
and upgrading the shared runtime to Apache Paimon 2.0.
### Description
Current `master` can read Paimon tables through the new Connector SPI, but
it still uses Paimon 1.3.1 and has no complete external-table write path. The
implementation merged into `branch-4.1` cannot be applied directly because
`master` has moved catalog, transaction, planning, and scan/write contracts
into the new connector modules.
The desired behavior is:
- use one Paimon 2.0 runtime version across FE serialization, BE JNI,
scanner, and writer;
- support `CREATE TABLE`, `DROP TABLE`, `INSERT INTO`, `INSERT OVERWRITE`,
and CTAS against Paimon external catalogs;
- support filesystem, HMS, REST, HDFS, and S3-compatible object storage
catalogs;
- support append-only and primary-key tables, partitioned writes, and Paimon
bucket assignment;
- preserve Doris statement atomicity with FE-coordinated commit, multi-BE
commit aggregation, idempotent retry/reconciliation, and abort cleanup;
- map Doris primitive and complex types, including Paimon 2.0 BLOB plus
Doris VARIANT and VECTOR encodings;
- keep read compatibility for existing Paimon tables after the runtime
upgrade.
### Proposed solution
Port the proven Java SDK/JNI design from #65868 to the current Connector SPI
instead of restoring the old datasource implementation:
```text
ConnectorWritePlanProvider
-> PaimonTableSinkOperator
-> PaimonTableWriter
-> JNI Paimon writer
-> Paimon CommitMessage
-> FE ConnectorTransaction
```
The writer should delegate partition routing, bucket/index assignment,
schema defaults, file creation, and commit semantics to the Paimon 2.0 SDK.
Doris should own query lifecycle, memory limits, commit-message transport,
acknowledgement, retry reconciliation, and cleanup.
BLOB and VECTOR can be represented using Paimon 2.0 physical types plus
stable column directives. VARIANT should use a lossless Arrow struct carrying
value and metadata binaries so nested and typed values survive round trips.
### Validation
The change should include:
- Java unit tests for Arrow conversion, schema mapping, commit codec, JNI
writer, connector write planning, and transaction handling;
- C++ unit tests for VARIANT transport, JNI schema metadata, report budgets,
acknowledgement, and cleanup;
- regression coverage for create/drop, insert, overwrite, CTAS atomicity,
and BLOB/VARIANT/VECTOR;
- live filesystem/HMS/REST/S3-compatible catalog tests against Paimon 2.0
data.
### Are you willing to submit PR?
Yes. A draft implementation for `master` is being validated and will
reference this issue.
### Code of Conduct
I agree to follow the Apache Software Foundation Code of Conduct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
