JingsongLi opened a new pull request, #10399: URL: https://github.com/apache/paimon/pull/10399
## Purpose Enable native ARRAY<BLOB>/MAP<X, BLOB> writes and packed video reads using the accompanying paimon-rust change. Preserve Java's lazy video frame references through filters and projections. ## Changes - Remove the append collection-write fallback gate. Keep packed video writes on the Python writer. - Allow dedicated `.video` files in native reads. Preserve video descriptors rather than resolving them as ordinary BLOB payloads, including aliased projections and row filtering. - Allow native normal-column updates and deletion-vector deletes on video tables. Keep video upserts on Python because unmatched rows require packed video writing. - Hand prepared composite files to the committer only after every partition prepares successfully. Later stream checkpoints neither resend nor delete previously handed-off files. - Clean up prepared files when Ray or either Daft worker path fails during close before returning its commit messages. - Add real native/Python cross-write/read tests for BLOB collections, descriptor ranges, rolling, external paths, non-nullable children, aliases, masked parents and slices. Cover late descriptor-copy failures and partial-partition preparation cleanup. - Test packed video frame and keyframe-index references through both Arrow readers, projections, filters, limits, native REST updates, deletion vectors and NULL/placeholder fallback. - Update old fallback and lifecycle assertions to the new native capabilities and Java's file handoff semantics. ## Verification The regression suite runs with all five native CI switches enabled and the locally built Rust binding: 1,642 tests and 130 subtests pass (3 skipped; 10 Vortex cases deselected because that optional Python package is not installed). Tests assert native writers/readers/REST committers are actually used, and reject Python fallback in the video read/update/delete cases. The affected suites include collection/managed BLOB writes, BLOB updates and reads, packed video, nested types, native updates/upserts/REST commits, data evolution, file cleanup, Ray and Daft. Flake8, license headers and Python 3.6 grammar checks for the changed core Python files also pass. ## Dependency Merge https://github.com/apache/paimon-rust/pull/1044 first. The existing native CI job continues building apache/paimon-rust `main`; no old-binding compatibility path or branch pin is added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
