suxiaogang223 opened a new issue, #520: URL: https://github.com/apache/paimon-rust/issues/520
## Background [Apache Doris](https://github.com/apache/doris) (an MPP OLAP database) is adding Paimon write support. We currently have a working JNI-based write path that bridges to the Paimon Java SDK, but we'd like to add a Rust FFI backend for better performance (zero-copy Arrow, async I/O, no JVM overhead). The existing `bindings/c/` already provides a complete read-side C FFI (catalog, table, scan, plan, predicate, record batch streaming via Arrow C Data Interface). We'd like to contribute write-side C FFI bindings following the same patterns. ## Current state - `bindings/c/` — read-side C FFI ✅ (catalog, table, read builder, scan, plan, predicate, record batch reader) - `bindings/python/` — Python bindings with full write support ✅ (`PyWriteBuilder`, `PyTableWrite`, `PyTableCommit`) - `bindings/c/` — write-side C FFI ❌ (nothing yet) ## Proposed API surface Following the existing naming conventions and opaque handle patterns from `bindings/c/`: ``` // === WriteBuilder === paimon_write_builder *paimon_table_new_write_builder(paimon_table *table); void paimon_write_builder_free(paimon_write_builder *wb); void paimon_write_builder_with_overwrite(paimon_write_builder *wb); // === TableWrite === paimon_table_write *paimon_write_builder_new_write(paimon_write_builder *wb, paimon_error **err); void paimon_table_write_free(paimon_table_write *tw); // write a single Arrow record batch via C Data Interface (zero-copy) paimon_error *paimon_table_write_write_batch( paimon_table_write *tw, FFI_ArrowArray *array, FFI_ArrowSchema *schema); // close file writers and produce commit messages // returns paimon_commit_messages* on success, NULL + err on failure paimon_commit_messages *paimon_table_write_prepare_commit( paimon_table_write *tw, paimon_error **err); // === TableCommit === paimon_table_commit *paimon_write_builder_new_commit(paimon_write_builder *wb); void paimon_table_commit_free(paimon_table_commit *tc); paimon_error *paimon_table_commit_commit( paimon_table_commit *tc, paimon_commit_messages *msgs); paimon_error *paimon_table_commit_overwrite( paimon_table_commit *tc, paimon_commit_messages *msgs, ... static_partitions ...); paimon_error *paimon_table_commit_truncate_table(paimon_table_commit *tc); paimon_error *paimon_table_commit_abort( paimon_table_commit *tc, paimon_commit_messages *msgs); // === CommitMessage serialization === // For cross-process transfer, commit messages need serialization. // Proposal: use protobuf for schema-safe, language-agnostic serialization, // matching Paimon's existing DataFileMeta / CommitMessage struct fields. paimon_bytes paimon_commit_messages_serialize(paimon_commit_messages *msgs); paimon_commit_messages *paimon_commit_messages_deserialize( paimon_bytes data, paimon_error **err); void paimon_commit_messages_free(paimon_commit_messages *msgs); ``` ## CommitMessage serialization — discussion needed This is the key open question. Paimon's `CommitMessage` is a Rust struct with nested `Vec<DataFileMeta>`, `Vec<IndexFileMeta>`, etc. For C FFI consumers (like Doris) that need to transfer messages across a process boundary and eventually to a remote coordinator, serialization is required. Options: | Format | Pros | Cons | |--------|------|------| | **Protobuf** | Schema-safe, cross-language, Paimon already uses protobuf internally | Adds `prost-build` dependency to `bindings/c` | | **JSON** | Simple, no schema needed | Bulky for DataFileMeta, slower | | **FlatBuffers** | Zero-copy deserialization | More complex, less community familiarity | | **Paimon binary format** | Reuses existing `CommitMessageSerializer` logic | Java-specific, needs porting to Rust | We lean toward **Protobuf** — it's the standard approach, Paimon already uses it, and it generates bindings for all languages. But we'd like community input before committing. ## Implementation plan 1. Add `bindings/c/src/write.rs` with the API surface above 2. Add CommitMessage protobuf schema and serialization 3. Add tests under `bindings/c/tests/` (following Python test patterns) 4. Update `bindings/c/src/lib.rs` to register the new module ## Questions for the community 1. Does this API surface look reasonable? Any naming or pattern preferences? 2. **Protobuf vs JSON vs other formats for CommitMessage serialization?** 3. Should `write_arrow_batch` also accept a batch of batches (`&[RecordBatch]`) or keep it single-batch per call? 4. Any concerns about adding a protobuf build dependency to `bindings/c/`? --- We're happy to implement this and contribute back. Looking forward to your feedback! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
