JingsongLi opened a new pull request, #1047: URL: https://github.com/apache/paimon-rust/pull/1047
### Purpose Closes #1046. Implement Java's aligned ROW auxiliary files in the Rust Data Evolution table pipeline. PyPaimon currently disables Native writes and updates when ROW sidecars are enabled, and Native reads when these extra files are present. ### Brief change log - Generate logical `.parquet.row` sidecars alongside normal data files, including partial column updates. Register aligned file names in `extra_files`, roll both outputs together, and preserve ownership until prepare succeeds. - Select ROW sidecars from the original requested row ranges using Java's inclusive count/ratio thresholds. File-index pruning, deletion vectors and LIMIT do not create sidecar eligibility. Resolve aligned external paths and fall back to the primary file only when `scan.ignore-lost-file` permits it. - Keep sidecars disabled for actual dedicated BLOB/vector writers; an inline-only normal schema can still use them. - Encode VECTOR values using Java's ROW layout, including dimension validation and parent NULLs. Verify VECTOR and materialized BLOB bytes against fixtures generated by Java's `RowBlockWriter`. - Materialize inline BLOB references for ROW using `Blob.toData()` semantics, one row at a time. Preserve payload provenance through row-id attachment and column merges, then remove the private Arrow marker before returning batches. Descriptor-mode BLOB reads retain their primary files. - Release resource reservations when ROW blocks flush. Cover producer creation/write/close failures, explicit abort and prepared-file retention. ### Tests - `cargo test --locked -p paimon --all-targets --features fulltext,vortex`: **4245 passed, 6 ignored**. - `cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings`: passed. - `cargo fmt --all -- --check` and `git diff --check`: passed. - 13 table integration cases cover sparse/full selections, thresholds, external paths, missing/corrupt sidecars, column updates, deletion vectors, BLOBs, VECTORs and bounded resource use. - PyPaimon tests with a locally built binding also exercise batch/stream Native write/update and Python/Rust read interoperability. ### API and Format Uses existing Java table properties and the Java ROW format; generation defaults to disabled. No new Python binding API or dependencies. Explicit abort can delete known abandoned files, while close after prepare retains handed-off outputs. ### Documentation Core option accessors and the sidecar module document the existing Java settings and selection/ownership contracts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
