JingsongLi opened a new pull request, #1423:
URL: https://github.com/apache/paimon-rust/pull/1423

   ### Purpose
   
   Enable Rust and Python clients to write append-only Paimon tables containing 
packed video-frame columns. Previously, Rust could read `.video` files but 
rejected writes to tables configured with `video-frame-field`.
   
   Also preserve nullable BLOB/VIDEO values when normal-column updates and 
multiple checkpoints are followed by appends that omit the BLOB column. 
Normal-file compaction can combine those rows into one logical range while 
dedicated files physically cover only its populated portions.
   
   ### Brief change log
   
   - Add a streaming packed-video writer: deduplicate encoded videos by source 
URI/range, coalesce consecutive frame runs across batches, locate optional MP4 
`moov` hints, and write the existing v1 indexes/footer.
   - Route configured scalar BLOB frame columns to dedicated `.video` files, 
reusing file rolling, commit metadata, URI readers, and failure cleanup.
   - Expose `pypaimon_rust.data.VideoFrameDescriptor` and `VideoFormatWriter` 
for PyPaimon adapters, with serialized-descriptor and Arrow batch inputs, owned 
output streams, exception preservation, and GC-visible callback references.
   - Match Java's normal-file anchor semantics: allow physical BLOB coverage 
gaps within the logical range and NULL-fill them on full scans. Preserve 
out-of-bounds rejection, non-nullable checks, selected-range clipping, and 
placeholder fallback.
   - Add BLOB/VIDEO regressions for ordinary-column update → two stream 
checkpoints → omitted-column append → normal-file compaction. Check row IDs, 
retained values, NULLs, and referenced VIDEO payload bytes.
   
   ### Tests
   
   Passed on the final implementation:
   
   - 89 data-evolution reader unit tests.
   - 43 Rust integration tests across `blob_update_test`, 
`table_partial_write_test`, `video_write_test`, and 
`blob_collection_write_test`.
   - 99 Python tests across `test_blob_partial_checkpoint.py`, 
`test_video_write.py`, `test_blob_uri_reader.py`, `test_blob_consumer.py`, and 
`test_update.py`. The optional interoperability case was run against a local 
PyPaimon checkout.
   - 4 corresponding Java anchor/gap/clipping contract tests, using existing 
compiled test classes in a separate Paimon checkout.
   - `cargo +1.98.0 fmt --all -- --check`
   - `cargo +1.98.0 clippy --locked -p paimon --lib --tests -- -D warnings`
   - Python binding Clippy and native-extension build.
   - Additional review probes: 2,000 randomized `moov` comparisons with 
PyPaimon; real PyPaimon FileIO wrappers; simultaneous source/destination 
failures; callback reentry and cross-thread GC.
   
   An earlier full core unit run had 4,919 passes, 7 ignored tests, and one 
failure in 
`io::cache::disk::shared::tests::process_exit_releases_staging_for_cleanup` 
(missing staging directory). That test passed on isolated rerun. The final 
reader fix was validated with the focused suites above; the full core suite was 
not rerun afterward.
   
   ### API and Format
   
   - Write the existing Java/PyPaimon packed-video v1 layout: six Delta-Varint 
indexes and a 29-byte little-endian footer; no new format version.
   - Inputs are serialized `VideoFrameDescriptor` bytes and NULLs. Packing 
copies encoded payloads without decoding/re-encoding or requiring FFmpeg/PyAV.
   - The low-level Python writer owns and closes its output stream; its caller 
manages incomplete-file removal and table commits.
   - Scope: scalar frame BLOB columns in append-only data-evolution tables. 
Video-column updates, video-file compaction, and BlobConsumer callbacks remain 
unsupported.
   
   ### Documentation
   
   Update the Python README, Python type stubs, and SQL BLOB documentation with 
the new interfaces, packed-video semantics, ownership, and limitations.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to