Dian Fay created SPARK-22220:
--------------------------------
Summary: Spark SQL: LATERAL VIEW OUTER null pointer exception with
GROUP BY
Key: SPARK-22220
URL: https://issues.apache.org/jira/browse/SPARK-22220
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.6.3
Environment: We have Zeppelin using Spark and Livy (error is
reproducible without Livy) on an Ambari cluster.
Reporter: Dian Fay
Given a DataFrame having the fields name (a string) and tags (an array of
strings), the following Spark SQL query fails with a NullPointerException:
{code}
SELECT name, tag, COUNT(*)
FROM records
LATERAL VIEW OUTER explode(tags) AS tag
GROUP BY name, tag
{code}
{code}
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
stage 137.0 failed 4 times, most recent failure: Lost task 0.3 in stage 137.0
(TID 9109, $hostname, executor 1): java.lang.NullPointerException
{code}
The query is successful without the "outer", but obviously this excludes rows
with empty tags arrays. A version with outer but without aggregation also
succeeds, making it possible to work around this issue with a subquery:
{code}
SELECT name, tag
FROM records
LATERAL VIEW OUTER explode(tags) AS tag
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
