[
https://issues.apache.org/jira/browse/SPARK-32163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-32163:
----------------------------------
Description:
If the expressions extracting nested fields have cosmetic variations like
qualifier difference, currently nested column pruning cannot work well.
For example, two attributes which are semantically the same, are referred in a
query, but the nested column extractors of them are treated differently when we
deal with nested column pruning.
was:
Note that this is just an optimization issue and not a regression. The newly
introduced optimizer doesn't optimize this corner case.
If the expressions extracting nested fields have cosmetic variations like
qualifier difference, currently nested column pruning cannot work well.
For example, two attributes which are semantically the same, are referred in a
query, but the nested column extractors of them are treated differently when we
deal with nested column pruning.
> Nested pruning should still work for nested column extractors of attributes
> with cosmetic variations
> ----------------------------------------------------------------------------------------------------
>
> Key: SPARK-32163
> URL: https://issues.apache.org/jira/browse/SPARK-32163
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
>
> If the expressions extracting nested fields have cosmetic variations like
> qualifier difference, currently nested column pruning cannot work well.
> For example, two attributes which are semantically the same, are referred in
> a query, but the nested column extractors of them are treated differently
> when we deal with nested column pruning.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
