Qiegang Long created SPARK-55817:
------------------------------------
Summary: Enable Parquet row-group skipping for shredded variant
columns
Key: SPARK-55817
URL: https://issues.apache.org/jira/browse/SPARK-55817
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.0.0
Reporter: Qiegang Long
When PushVariantIntoScan rewrites variant_get() calls into struct field
accesses, the rewritten predicates reference logical paths like "v.`0`"
that ParquetFilters cannot resolve to any physical parquet column, so they are
dropped and row-group skipping is disabled for all shredded variant queries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
