[
https://issues.apache.org/jira/browse/SPARK-22220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16195929#comment-16195929
]
Tejas Patil commented on SPARK-22220:
-------------------------------------
Does this repro with current spark trunk (over CLI) ? I could not repro. Here
is what I tried:
{noformat}
val spark_22220_df = (0 until 1000).map(i => (i.toString,
i.toString.toArray.map(x => x.toString) )).toDF("name", "tags")
sql(" DROP TABLE IF EXISTS records").collect
spark_22220_df.write.format("orc").saveAsTable("records")
sql("""
SELECT name, tag, COUNT(*)
FROM records
LATERAL VIEW OUTER explode(tags) AS tag
GROUP BY name, tag
""").collect
{noformat}
> Spark SQL: LATERAL VIEW OUTER null pointer exception with GROUP BY
> ------------------------------------------------------------------
>
> Key: SPARK-22220
> URL: https://issues.apache.org/jira/browse/SPARK-22220
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.1.1
> Environment: We have Zeppelin using Spark and Livy (error is
> reproducible without Livy) on an Ambari cluster.
> Reporter: Dian Fay
>
> Given a DataFrame having the fields name (a string) and tags (an array of
> strings), the following Spark SQL query fails with a NullPointerException:
> {code}
> SELECT name, tag, COUNT(*)
> FROM records
> LATERAL VIEW OUTER explode(tags) AS tag
> GROUP BY name, tag
> {code}
> {code}
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
> stage 137.0 failed 4 times, most recent failure: Lost task 0.3 in stage 137.0
> (TID 9109, $hostname, executor 1): java.lang.NullPointerException
> {code}
> The query is successful without the "outer", but obviously this excludes rows
> with empty tags arrays. A version with outer but without aggregation also
> succeeds, making it possible to work around this issue with a subquery:
> {code}
> SELECT name, tag
> FROM records
> LATERAL VIEW OUTER explode(tags) AS tag
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
