[
https://issues.apache.org/jira/browse/SPARK-11619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15021509#comment-15021509
]
Wenchen Fan commented on SPARK-11619:
-------------------------------------
Actually this line:
https://github.com/apache/spark/blob/branch-1.5/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala#L689
When we use `selectExpr`, we pass in `UnresolvedFunction` to `DataFrame.select`
and fall in the last case. A workaround is to do special handling for UDTF like
we did for `explode`(and `json_tuple` in 1.6), wrap it with `MultiAlias`.
Another workaround is using `expr`, for example,
`df.select(expr("explode(a)").as(Nil))`, I think `selectExpr` is no longer
needed after we have the `expr` function....
> cannot use UDTF in DataFrame.selectExpr
> ---------------------------------------
>
> Key: SPARK-11619
> URL: https://issues.apache.org/jira/browse/SPARK-11619
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Wenchen Fan
> Priority: Minor
>
> Currently if use UDTF like `explode`, `json_tuple` in `DataFrame.selectExpr`,
> it will be parsed into `UnresolvedFunction` first, and then alias it with
> `expr.prettyString`. However, UDTF may need MultiAlias so we will get error
> if we run:
> {code}
> val df = Seq((Map("1" -> 1), 1)).toDF("a", "b")
> df.selectExpr("explode(a)").show()
> {code}
> [info] org.apache.spark.sql.AnalysisException: Expect multiple names given
> for org.apache.spark.sql.catalyst.expressions.Explode,
> [info] but only single name ''explode(a)' specified;
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
