JingsongLi opened a new pull request, #1171: URL: https://github.com/apache/paimon-rust/pull/1171
### Purpose Make ordinary Variant values usable in ROW tables through the native Rust table API. Append reads already supported the v1 Variant codec, but primary-key writes used a thin physical layout while ROW readers expected Java's duplicated key prefix. Raw-convertible PK splits also decoded files against user fields alone. These positional mismatches could report `Malformed Variant metadata version` or return incorrect values. ### Brief change log - Write ROW data and input-changelog files as `[_KEY_*, _SEQUENCE_NUMBER, _VALUE_KIND, user fields...]`. Choose the projection independently for each file's format; keep internal merge, statistics, file-index and blob-reference batches unchanged. - Supply the complete physical schema for raw PK ROW reads, including audit/changelog and positional materialization paths. Match historical key names and types by stable field ID and preserve explicit partial-write schemas. - Cover ordinary and nested Variant round trips, SQL NULL versus Variant null, reordered projections, path extraction, schema evolution, local positions, overlapping PK files and mixed ROW/Parquet changelog formats. ### Tests - `cargo test -p paimon --lib arrow::format::row::tests --locked` - `cargo test -p paimon --lib kv_ --locked` - `cargo test -p paimon --lib table::data_file_reader::row_tests --locked` - `cargo test -p paimon --lib test_primary_key_non_parquet_formats_roundtrip_with_typed_physical_fields --locked` - `cargo test -p paimon --test row_variant_test --locked` - `cargo fmt --all -- --check` - `cargo clippy -p paimon --all-targets --features fulltext,vortex --locked -- -D warnings -A clippy::nonminimal_bool` The focused tests passed (113 passed, one existing ignored codec test). The strict Clippy command without the allowance reports four existing `nonminimal_bool` diagnostics in unchanged `de_vector_read.rs`, `table_merge_into.rs` and `vector_search_common.rs` under Rust 1.95; allowing that lint passes all remaining checks. ### API and Format No public API change. Keep ROW v1 and the existing value/metadata encoding. Correct native PK ROW output to the canonical Java layout; this does not add a compatibility mode for the previously emitted noncanonical thin ROW layout. Parquet output keeps its existing thin layout. Variant path extraction still reads its containing compressed row block; this change does not add ROW shredding or subfield IO. ### Documentation Document ordinary Variant support in Parquet and ROW, the ROW block-read behavior and Parquet-only shredding options. Related Java/Python support: https://github.com/apache/paimon/pull/10650 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
