JingsongLi opened a new pull request, #1044: URL: https://github.com/apache/paimon-rust/pull/1044
## Purpose Native append writes currently route ARRAY<BLOB> and MAP<X, BLOB> into ordinary Parquet columns instead of Java's dedicated BLOB files. Packed `.video` files are also missing from native reads, including the persisted keyframe indexes introduced in apache/paimon#9831. ## Changes - Select the dedicated BLOB writer for scalar, ARRAY<BLOB> and MAP<X, BLOB> fields. Extract the element writer and share record framing, CRC, streamed descriptor copying and delta-varint indexes. - Match Java's ARRAY and MAP layouts, including NULL/empty parents and values, sliced arrays, duplicate-key rejection, all supported MAP key serializers, and compact/noncompact DECIMAL keys. - Add `VideoFrameDescriptor` v1 and packed video metadata reading. Return lazy frame descriptors carrying the physical video range, frame ordinal and optional keyframe-index range, without decoding or fetching video payloads. - Treat video files like dedicated BLOB files in scans, row-id allocation, data-evolution anchor selection and sequence fallback. Preserve the difference between NULL and placeholder runs. - Apply Java's `video-frame-field` schema rules and reject native video writes explicitly. Reject raw BLOB collection columns in data-evolution partial writes, consistent with Java MERGE INTO restrictions. - Validate non-nullable nested write fields even when imported Arrow schemas match. Trim children to the visible parent/offset ranges and propagate ancestor ROW NULLs, so hidden NULLs, aliases and slices remain valid. Ordinary inputs keep shared payload buffers. ## Verification - Real ARRAY writer output matches the Java fixture's physical records byte for byte, including CRC. - Java video and descriptor v1 fixtures pin the wire layout and frame/keyframe positions. - Dedicated collection integration tests cover raw values, descriptors, NULLs, empty values, rolling and descriptor reads. - Corruption, selection, placeholder fallback and Arrow nullability regressions are covered. - `cargo test --locked -p paimon --lib --test blob_collection_write_test --test dedicated_blob_write_test --features fulltext,vortex`: 3,813 unit tests and 12 integration tests pass (6 ignored). - `cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings` - `cargo fmt --all --check` - The locally built Python binding is exercised by 1,642 passing PyPaimon regression tests with all five native CI switches enabled (3 skipped; 10 Vortex cases deselected because that optional Python package is not installed). Packed video writing and video upserts remain outside this change. No dependencies are added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
