sundapeng opened a new pull request, #1057: URL: https://github.com/apache/paimon-rust/pull/1057
### Purpose Extracting float fields from a plain (unshredded) VARIANT column copies the column's metadata into every row. Parquet usually stores that metadata dictionary-encoded (a single entry when every row has the same key set), but the reader decodes it into a `BinaryArray`, so the whole key dictionary is copied once per row. `extract_numeric_fields` then compares each row's metadata with the previous one byte by byte, and `VariantFloat32Projection` sorts every row's field offsets when they are not monotonic, which is the usual layout written by the Java and Python builders (values in insertion order, field ids in key order). For wide objects this dominates the read. With about 5,500 keys (about 280 KB of metadata per row), reading 1,344 rows copies about 380 MB into freshly allocated memory, and most of the CPU goes to that copy and the page faults it causes. ### Brief change log - When a plain Variant column is read only for field extraction, request `metadata: Dictionary(Int32, Binary)` from parquet-rs through `ArrowReaderOptions::with_schema`, including in the per-row-group readers. - `extract_numeric_fields` accepts dictionary metadata and rebuilds the projection only when the dictionary key changes; for plain metadata it compares pointers before bytes. - `assemble_plain_variant_projection` accepts dictionary metadata; extraction paths that need plain metadata materialize it back to `Binary` first. - `VariantFloat32Projection` keeps the previous row's offset order while an O(n) check confirms it still sorts the current offsets, instead of sorting every row. Malformed offsets are still rejected. ### Tests - New unit tests: dictionary and plain metadata give identical projections; the cached offset order still rejects a malformed row that follows valid ones; the Parquet reader returns dictionary metadata for an extraction read, and the assembled projection equals the plain path. - `cargo test -p paimon --lib -- variant parquet::tests`, `cargo fmt --all -- --check`, `cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings`. - Benchmark: a PyTorch DataLoader workload on a 60-vCPU host (8 ranks x 4 workers, batch 16; each batch reads 1,344 rows of a ~5,500-key Variant through 200 float projections plus a Steps lookup over a MAP column). CPU per batch fell from 2.48 to 1.54 core-seconds (-38%), throughput rose from 308 to 391 samples/s (+27%), peak memory did not grow, and the full output hash of every batch is unchanged. ### API and Format No. ### Documentation No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
