GitHub user dilipbiswal reopened a pull request:
https://github.com/apache/spark/pull/9981
[SPARK-11619][SQL] cannot use UDTF in DataFrame.selectExpr
Description of the problem from @cloud-fan
Actually this line:
https://github.com/apache/spark/blob/branch-1.5/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala#L689
When we use `selectExpr`, we pass in `UnresolvedFunction` to
`DataFrame.select` and fall in the last case. A workaround is to do special
handling for UDTF like we did for `explode`(and `json_tuple` in 1.6), wrap it
with `MultiAlias`.
Another workaround is using `expr`, for example,
`df.select(expr("explode(a)").as(Nil))`, I think `selectExpr` is no longer
needed after we have the `expr` function....
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dilipbiswal/spark spark-11619
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9981.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9981
----
commit 96204680d83e47e39b1c9604d1a6b9e91ee89f8c
Author: Dilip Biswal <dbis...@us.ibm.com>
Date: 2015-11-25T19:20:22Z
[SPARK-11619] cannot use UDTF in DataFrame.selectExpr
commit 9bee82b8a46699d35a8972578ec712896f141336
Author: Dilip Biswal <dbis...@us.ibm.com>
Date: 2015-11-26T03:17:48Z
fix test failure
commit af3963c14e947f8a04e51a3690aab66b0027efbe
Author: Dilip Biswal <dbis...@us.ibm.com>
Date: 2015-12-01T07:09:14Z
Incorporate Wenchen's comments.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
