JingsongLi opened a new pull request, #1423: URL: https://github.com/apache/paimon-rust/pull/1423
### Purpose Enable Rust and Python clients to write append-only Paimon tables containing packed video-frame columns. Previously, Rust could read `.video` files but rejected writes to tables configured with `video-frame-field`. Also preserve nullable BLOB/VIDEO values when normal-column updates and multiple checkpoints are followed by appends that omit the BLOB column. Normal-file compaction can combine those rows into one logical range while dedicated files physically cover only its populated portions. ### Brief change log - Add a streaming packed-video writer: deduplicate encoded videos by source URI/range, coalesce consecutive frame runs across batches, locate optional MP4 `moov` hints, and write the existing v1 indexes/footer. - Route configured scalar BLOB frame columns to dedicated `.video` files, reusing file rolling, commit metadata, URI readers, and failure cleanup. - Expose `pypaimon_rust.data.VideoFrameDescriptor` and `VideoFormatWriter` for PyPaimon adapters, with serialized-descriptor and Arrow batch inputs, owned output streams, exception preservation, and GC-visible callback references. - Match Java's normal-file anchor semantics: allow physical BLOB coverage gaps within the logical range and NULL-fill them on full scans. Preserve out-of-bounds rejection, non-nullable checks, selected-range clipping, and placeholder fallback. - Add BLOB/VIDEO regressions for ordinary-column update → two stream checkpoints → omitted-column append → normal-file compaction. Check row IDs, retained values, NULLs, and referenced VIDEO payload bytes. ### Tests Passed on the final implementation: - 89 data-evolution reader unit tests. - 43 Rust integration tests across `blob_update_test`, `table_partial_write_test`, `video_write_test`, and `blob_collection_write_test`. - 99 Python tests across `test_blob_partial_checkpoint.py`, `test_video_write.py`, `test_blob_uri_reader.py`, `test_blob_consumer.py`, and `test_update.py`. The optional interoperability case was run against a local PyPaimon checkout. - 4 corresponding Java anchor/gap/clipping contract tests, using existing compiled test classes in a separate Paimon checkout. - `cargo +1.98.0 fmt --all -- --check` - `cargo +1.98.0 clippy --locked -p paimon --lib --tests -- -D warnings` - Python binding Clippy and native-extension build. - Additional review probes: 2,000 randomized `moov` comparisons with PyPaimon; real PyPaimon FileIO wrappers; simultaneous source/destination failures; callback reentry and cross-thread GC. An earlier full core unit run had 4,919 passes, 7 ignored tests, and one failure in `io::cache::disk::shared::tests::process_exit_releases_staging_for_cleanup` (missing staging directory). That test passed on isolated rerun. The final reader fix was validated with the focused suites above; the full core suite was not rerun afterward. ### API and Format - Write the existing Java/PyPaimon packed-video v1 layout: six Delta-Varint indexes and a 29-byte little-endian footer; no new format version. - Inputs are serialized `VideoFrameDescriptor` bytes and NULLs. Packing copies encoded payloads without decoding/re-encoding or requiring FFmpeg/PyAV. - The low-level Python writer owns and closes its output stream; its caller manages incomplete-file removal and table commits. - Scope: scalar frame BLOB columns in append-only data-evolution tables. Video-column updates, video-file compaction, and BlobConsumer callbacks remain unsupported. ### Documentation Update the Python README, Python type stubs, and SQL BLOB documentation with the new interfaces, packed-video semantics, ownership, and limitations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
