peterxcli commented on PR #25013: URL: https://github.com/apache/datafusion/pull/25013#issuecomment-5582978854
I benchmarked a downstream implementation of this capability in [peterxcli/datafusion-variant#1](https://github.com/peterxcli/datafusion-variant/pull/1), using DataFusion commit `1180ae33d9903de607cd8dd5cd04c3efbbfa10b4` from this PR. The partially shredded workload shows a substantial benefit even though the encoded root `value` column must still be read. ### Partially shredded Variant results Median execution time with a warm filesystem cache: | Query | Before | PR head, capability disabled | PR head, capability enabled | Before / enabled | |---|---:|---:|---:|---:| | Direct `variant_get` projection | 768.94 ms | 755.45 ms | 179.32 ms | **4.29x** | | Selective `variant_get` filter | 827.26 ms | 818.26 ms | 183.94 ms | **4.50x** | | `SUM(variant_get(...))` | 778.68 ms | 817.34 ms | 782.58 ms | Approximately unchanged | Parquet bytes requested fall from **1,350 MB to 265 MB** for projection and **1,357 MB to 271 MB** for filtering—about **80% fewer bytes**. The capability-disabled control uses the same new dependencies and extraction implementation, isolating the effect of the input-field declaration from the DataFusion/Arrow upgrade. Cold-cache advisory trials gave **4.57x** and **4.20x** speedups for projection and filtering. Linux storage-read counters fell from approximately **1,356 MB to 294 MB / 300 MB**, respectively. These counters include read-ahead; eviction used `POSIX_FADV_DONTNEED` on the benchmark file, without controlling the VM host's storage cache. Warm trials recorded zero storage-read bytes, so their gains reflect reduced processing of cached Parquet data. ### Setup and correctness - One deterministic dataset: **1,048,576 rows**, 16 row groups, Snappy compression, eight unrelated 128-byte typed string fields, and a 256-byte field left encoded in root `value` on every row. The file is approximately 1.36 GB, including 265 MB in that encoded column. - Every tenth `age` exceeds 4 billion and requires encoded fallback from its Int32 shredded field. The filter matches these rows; all row groups contain matches. - Before: downstream `9e1c84698150eaef7b640f50f907dd8e585174f5` with DataFusion 54 / Arrow 58.3.0. After: downstream `fc1ef311d1270f9016dce07a48f893155fb06c8e` with this DataFusion commit / Arrow 59.2.0. Dependency versions match each revision's lockfile. - Optimized release builds, Rust 1.98.0, a 16-vCPU AMD EPYC 7282 VM with 31 GiB RAM, four query workers/target partitions, and affinity to CPUs 0–7. Compilation finished before timing. - Twelve warm trials per case; six cold advisory trials for partially shredded data. Variant execution order was balanced. Times exclude planning and table registration. - **All 405 executions passed result-count/checksum checks**, including warmups and raw/all-object-fields-shredded controls. Enabled filters report **943,718 decoder-pruned rows**, with **104,858 matching rows**; before and capability-disabled runs report zero decoder-pruned rows. ### Remaining limitation `SUM(variant_get(...))` did **not** receive input pruning in the observed plan: the function remains inside `AggregateExec`, and the scan has `projection=[v]`. Direct projection reaches `DataSourceExec` as a function expression and benefits from the declaration. Raw Variant data showed essentially unchanged performance and read volume. These are results for a wide synthetic dataset with a literal path and typed Int64 extraction. They demonstrate input pruning and decoder-filter benefits, rather than a general speedup for every Variant query or row-group statistics pruning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
