suxiaogang223 opened a new issue, #910: URL: https://github.com/apache/paimon-rust/issues/910
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar. ### Motivation The C write API added by #520 supports writing and committing in one process. `paimon_table_write_prepare_commit` returns an opaque `paimon_commit_messages*` handle backed by Rust objects on the local heap. The handle can only be merged, committed, aborted, or freed in that process. This is insufficient for a distributed write in which worker processes prepare files and a coordinator process performs the final commit. A worker cannot send the pointer to the coordinator; it needs a stable byte representation of the commit messages. Apache Doris is one example of this architecture: BEs write data and prepare commit messages, and the FE collects the messages and commits once. Other native engines using the C FFI have the same requirement. Paimon Java already has a versioned `CommitMessageSerializer`. Using that protocol as the compatibility reference would avoid introducing a separate wire format and would allow Java and Rust components to exchange commit messages. ### Solution Add a versioned serializer for `Vec<CommitMessage>` to the Rust core, then expose it through the C FFI. #### Rust core The serializer should: - use the Java `CommitMessageSerializer` protocol as the compatibility reference; - expose the serializer version instead of requiring FFI consumers to hard-code it; - support the partition row, data files, deleted files, changelog files, and index files required by a commit; - reject malformed input and unsupported versions with a clear error; - define how the current Rust `CommitMessage` maps to Java's data and compaction increments. The Rust model also contains state that may not map directly to the current Java protocol, such as conflict-check and fixed-bucket overwrite state. The implementation should either define a compatible representation or reject unsupported messages explicitly. This mapping needs agreement before implementation. #### C FFI Expose byte export/import APIs equivalent to: ```c paimon_result_bytes paimon_commit_messages_serialize( const paimon_commit_messages *messages); paimon_result_commit_messages paimon_commit_messages_deserialize( const uint8_t *data, size_t data_len, /* commit context, or a handle that supplies it */); void paimon_bytes_free(paimon_bytes bytes); ``` The exact deserialize signature must preserve the context currently stored in the opaque handle: table identity/location, `commit_user`, and overwrite mode. One option is to bind deserialized messages to an existing `paimon_table_commit` handle. Another is to pass an explicit, versioned context structure. The API should not trust table identity from an unvalidated remote payload. Standard and postpone fixed-bucket commit messages should have an explicit supported behavior. They may use separate payload types or entry points if their models differ. The serialized bytes should represent the Paimon commit message payload. An embedding engine can add its own transport envelope, such as a magic value, serializer version, and payload length. #### Compatibility tests Add Java-generated golden fixtures and verify both directions: 1. Java serializes and Rust deserializes equivalent messages. 2. Rust serializes and Java deserializes equivalent messages. 3. A coordinator can deserialize and merge payloads from multiple workers, then commit them once. 4. Truncated data, invalid lengths, and unsupported versions fail without leaks or panics. The fixtures should cover partitioned data, new and deleted files, changelog files, index files, and supported fixed-bucket cases. ### Anything else? Related work found during the issue search: - #520 added the write/commit C FFI. Its discussion explicitly deferred cross-process serialization to a later enhancement. - #414 tracks serializable commit messages for Python/Ray, but does not provide a C FFI API. - #722 added Go postpone fixed-bucket write bindings and states that cross-process commit-message serialization remains the integration's responsibility. Open design questions: 1. Which Java serializer versions should Rust read and write initially? 2. Should the deserializer receive commit context from `paimon_table_commit` or from a separate versioned structure? 3. Should standard and postpone fixed-bucket messages share one wire type? Java compatibility reference: `paimon-core/src/main/java/org/apache/paimon/table/sink/CommitMessageSerializer.java` ### Willingness to contribute - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
