JingsongLi opened a new pull request, #10017: URL: https://github.com/apache/paimon/pull/10017
## Purpose Keep PyPaimon's Rust physical reader available when the scan must be planned in Python. This extends native coverage to Python-planned DataSplit and IndexedSplit inputs, including Data Evolution row ranges and query-authorization wrappers. Depends on apache/paimon-rust#889. `pypaimon-rust` has not been released, so this intentionally targets the paired API without a legacy compatibility shim. Vortex native read remains out of scope. ## Changes - implement a Java-compatible SplitSerializer v1 writer for Python DataSplit and IndexedSplit - serialize the current DataSplit v9 / DataFileMeta v21 layout, preserving stats, deletion vectors, row ranges, external paths, streaming/raw flags, and optional vector scores - bridge Python-planned splits through `pypaimon_rust.datafusion.Split.deserialize` before native reads; scores remain on the Python IndexedSplit while Rust receives the physical row ranges - attach the planned snapshot id to Python DataSplits so stable cross-language serialization is self-contained - carry `_WRITE_COLS_SEQUENCES` / column max sequence numbers through Python manifest metadata - support native reads for homogeneous query-authorization splits with Python-equivalent ordering: authorization filter, shared LIMIT, masking, then public projection - widen native projection for authorization dependencies without exposing those columns to callers - preserve safe Python fallback for unsupported split wrappers, mixed authorization contexts, nested or MAP-key authorization, and authorization filters over BLOB payloads - harden split decoding against truncated, negative-count, malformed-array, inverted-range, oversized modified-UTF, and trailing-byte inputs - add real Python-plan-to-Rust-read integration coverage for ordinary, partitioned, Indexed/Data Evolution, authorization, streaming-batch, projection, parallelism, and fallback paths ## Tests - native read, split serializer, and query authorization: 114 passed - real paired Rust-wheel native integration suite: 35 passed, including Python-planned and Indexed/Data Evolution reads - native Python coverage group: 266 passed, plus 128 parameterized subtests - resolved-schema coverage group: 82 passed - non-Vortex Data Evolution group: 185 passed, 1 skipped, 10 deselected, plus 29 parameterized subtests - changed manifest round-trip test passed - `python -m compileall -q pypaimon` - project flake8 configuration passed on every changed Python file - paired Rust binding tests, clippy, and formatting are recorded in apache/paimon-rust#889 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
