raghuvanshraj commented on issue #21306: URL: https://github.com/apache/datafusion/issues/21306#issuecomment-6629829290
A measured data point for the Variant case, from #27007 (closing that as a duplicate of this). One 100k-row Parquet file, two columns with the same 20 fields: `s` as a plain struct, `v` as a Variant shredded on all 20. Reading one small field: | query | scan projection | bytes_scanned | |---|---|---| | `SELECT s['f0'] FROM t` | `get_field(s, f0)` | 11.2 KB | | `SELECT variant_get(v, 'f0', 'BIGINT') FROM t` | `variant_get(v, f0)` | 126.4 MB | The `f0` leaf exists on the Variant side at the same size (1,147 bytes per row group) and is never read. With a `WHERE` on the field, the struct predicate reaches the scan; the Variant one runs in a `FilterExec` after decoding 126 MB. Standalone reproducer and raw `EXPLAIN ANALYZE` output in #27007. For `variant_get` the capability needs two things beyond `get_field`: one argument may require two leaves (the typed leaf and the `value` fallback), and pruning must keep any row group that holds a fallback row. Of the two open PRs, #26282 does both; #25013 does not cover pruning. Happy to test either against shredded and unshredded fixtures. This is the first workstream in #26517. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
