[
https://issues.apache.org/jira/browse/SPARK-60766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60766:
-----------------------------------
Labels: pull-request-available (was: )
> Prune nested fields through dynamic array indices and Map keys
> --------------------------------------------------------------
>
> Key: SPARK-60766
> URL: https://issues.apache.org/jira/browse/SPARK-60766
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Chao Sun
> Priority: Major
> Labels: pull-request-available
>
> Nested schema pruning currently stops at GetMapValue/GetArrayItem/ElementAt
> when the key or index cannot be evaluated as a constant. This preserves
> correctness, but makes queries such as m[key_column].id and
> items[index_column].id request the complete value/element Struct, including
> unused payload fields.
> A reproducible Parquet example in spark-shell:
> {code:scala}
> spark.conf.set("spark.sql.optimizer.nestedSchemaPruning.enabled", "true")
> val path = "/tmp/spark-dynamic-selector-pruning"
> spark.sql("""SELECT 'chosen' AS lookup_key,
> map('chosen', named_struct('id', 42, 'payload', 'unused')) AS m""")
> .write.mode("overwrite").parquet(path)
> val input = spark.read.parquet(path)
> input.selectExpr("m['chosen'].id").explain()
> input.selectExpr("m[lookup_key].id").explain()
> {code}
> Both queries return 42. The constant lookup can request only id from the Map
> values, while the dynamic lookup requests both id and payload, plus
> lookup_key. The desired dynamic read schema is
> struct<lookup_key:string,m:map<string,struct<id:int>>>.
> SPARK-42163 introduced the conservative restriction because the selector
> itself may reference a nested field, and pruning must not discard that input.
> The proposed change extends the existing additional-dependency traversal to
> collect every key/index input while SelectedField follows the collection
> input. ProjectionOverSchema also needs to rebind dynamic element_at
> expressions after narrowing.
> Scope: dynamic bracket Map/array access and element_at over scan-rooted
> collections. Preserve selector dependencies even when they refer to the same
> collection, occur beneath null checks, or interact with supported array
> functions. Preserve field metadata, field ordinals, null results and
> ANSI/index error behavior. No new settings, dependencies, or public API.
> Validation should cover Catalyst schema/ordinal behavior and actual Parquet
> read schemas/results across V1/V2 and vectorized/row readers. The existing
> SPARK-42163 regression should require a narrower schema. Include renamed
> field-ID paths and selector dependencies under IsNull/IsNotNull. This is a
> missed optimization, not a reported end-to-end speedup.
> Related: SPARK-42163 (original safety restriction), SPARK-40033 (element_at
> pruning), SPARK-58772 (metadata preservation, already present on master).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
