[
https://issues.apache.org/jira/browse/SPARK-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14649488#comment-14649488
]
Yin Huai commented on SPARK-4366:
---------------------------------
Just a note to myself. Seems we need to project fields and elements of a
complex type used by grouping expressions or aggregate functions first. So,
unsafe convert can be applied to cases like {{group by array[0]}}. Otherwise,
we are taking the entire complex structure as the input of an aggregate
operator and unsafe convert does not understand how to convert it.
For example, in {{lateral_view.q}}, we have a query {{SELECT value, myCol from
(SELECT key, array(value[0]) AS value FROM tmp_pyang_src_rcfile GROUP BY
value[0], key) a
LATERAL VIEW explode(value) myTable AS myCol;}}. Although we only need
{{value[0]}}, the entire value field is the input of the aggregate operator.
> Aggregation Improvement
> -----------------------
>
> Key: SPARK-4366
> URL: https://issues.apache.org/jira/browse/SPARK-4366
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Cheng Hao
> Priority: Critical
> Attachments: aggregatefunction_v1.pdf
>
>
> This improvement actually includes couple of sub tasks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
