JingsongLi opened a new pull request, #1063: URL: https://github.com/apache/paimon-rust/pull/1063
### Purpose Closes #1062. PyPaimon already supports `with_blob_consumer`, but invoking it currently falls back to the Python writer. Add the missing capability in Rust core so the existing Python API can keep append Blob writes native, using Java's implementation as the reference. ### Brief change log - Add the `BlobConsumer` core trait and `TableWrite::with_blob_consumer`, with thin batch/stream Python wrappers for serialized descriptors and original Python exceptions. - Match Java's scalar and collection callback timing, payload ranges, outer NULL callbacks, skipped NULL collection elements, and record-level flush requests. Primary-key and inline Blob writers do not invoke the append consumer. - Preserve consumer Blob files during automatic cleanup, including callback failures, rolling/group close failures and partial multi-partition prepare failures. Explicit `TableCommit::abort` still discards prepared files when requested by the caller. - Put consumer configuration in the dedicated Blob writer factory. Ordinary format writers keep their existing output path. - Drain local buffered output on requested flushes, including cache invalidation; backend multipart uploads continue to publish on close. Avoid empty segments in non-contiguous output buffers. - Honor `blob.copy-buffer-size` with Java's 4 KiB default and positive int bounds. Keep remote source read-ahead independent of copy chunks to avoid small range requests. ### Tests - `cargo test --locked -p paimon --lib -q`: 3758 passed, 6 existing ignored tests. - Full Python binding test suite: 326 passed. - PyPaimon Blob/collection/managed Blob/write/update regression scope with Native planning, reads, writes, updates and commits enabled: 499 passed; all Native execution counters satisfied. - Existing Python BlobConsumer tests without Native overrides: 6 passed. - `cargo +1.98.0 clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings`: passed. - Formatting, changed Python-file lint and diff checks passed. Regression coverage includes callback/flush failure, no retries, NULL and empty collections, copied descriptor references, external files, batch/stream checkpoints, local payload visibility before prepare, cache invalidation, rolled file ownership, partial prepare failure, explicit abort, and source read-ahead counts. ### API and Format Adds `BlobConsumer` and `TableWrite::with_blob_consumer`, plus matching Python batch/stream write methods. No storage format change or new dependency. ### Documentation Document the callback and ownership contract in Rust API comments and Python stubs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
