sundapeng opened a new pull request, #1069: URL: https://github.com/apache/paimon-rust/pull/1069
### Purpose Extracting fields from a shredded VARIANT column is row-wise today. For every row the Parquet reader decodes the dictionary-encoded `metadata` into a `BinaryArray`, so the whole key dictionary is copied per row, and `assemble_variant_extraction_array` then rebuilds the row's shredded object, builds a Variant for each requested path and casts it. When the requested paths are already shredded into typed columns, none of that work is needed: the result is the typed column itself, cast to the requested type. On a wide object (about 5,500 keys, about 280 KB of metadata per row) with 200 DOUBLE fields shredded, reading 1,344 rows through 200 `try_variant_get(..., 'float')` projections took about twice the CPU of the same data unshredded, even though it read 61-76% fewer bytes. A profile showed 31% of the time in `OffsetBuffer::extend_from_dictionary` (the metadata copy) and about 25% in the page faults it causes; most of the rest was the per-row object assembly. ### Brief change log - `dictionary_variant_metadata_schema` also requests `metadata: Dictionary(Int32, Binary)` for shredded Variant structs that are read only for extraction, as #1057 does for plain Variant columns. The metadata child is found by name. - New column-wise path `assemble_typed_shredded_projection`. It is taken when every requested path is a chain of object keys that ends at a numeric typed leaf (INT8 to BIGINT, FLOAT, DOUBLE), every object and field group on the path is non-null in every valid row, and no valid row of the batch carries a residual `value` on the path. The typed leaf is then cast column-wise to FLOAT, DOUBLE or BIGINT, using the same conversions as `cast_variant_to_shredded_value` for those Variant kinds (these casts cannot fail). Any other batch keeps the row-wise path. - The row-wise path (now `assemble_row_wise_variant_extraction`) and the extraction dispatch accept shredded structs with dictionary metadata; `materialize_variant_metadata` turns it back into `Binary` for plain and shredded structs alike. ### Tests - New unit tests: the column-wise path matches the row-wise path value for value (FLOAT, DOUBLE and BIGINT targets, nested object keys, bracket paths, null rows, missing fields), including with dictionary metadata; residual values, non-object rows, missing nested objects, unshredded keys and non-numeric targets fall back and still match the row-wise path; a shredded Parquet file read for extraction returns dictionary metadata and the same projection as the in-memory column. - `cargo test -p paimon --lib` (3,785 passed), `cargo fmt --all -- --check`, `cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings`. - Benchmark, single process, 16 clips x 84 rows per batch through 200 float projections of the object above: CPU per batch 0.49 s before and 0.20 s after with the shredded layout, against 0.27 s for the same data unshredded. Output is identical across layouts. - Benchmark, full PyTorch DataLoader on a 60-vCPU host (8 ranks x 6 workers, batch 16; each batch also does a Steps lookup and image reads): with the pose table shredded, throughput rose from about 405 to about 500 samples/s (median of 13 runs), with the full output hash of every batch unchanged. That run also needed a 256 MB `file-format.metadata-cache.max-size`, because the shredded files have about 400 leaf columns. ### API and Format No. ### Documentation No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
