[
https://issues.apache.org/jira/browse/SPARK-36379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-36379.
-----------------------------------
Fix Version/s: 3.3.0
3.2.0
Assignee: Hyukjin Kwon
Resolution: Fixed
This is resolved via https://github.com/apache/spark/pull/33608
> Null at root level of a JSON array causes the parsing failure (w/ permissive
> mode)
> ----------------------------------------------------------------------------------
>
> Key: SPARK-36379
> URL: https://issues.apache.org/jira/browse/SPARK-36379
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.2, 3.2.0, 3.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Minor
> Fix For: 3.2.0, 3.3.0
>
>
> {code}
> scala> spark.read.json(Seq("""[{"a": "str"}, null, {"a":
> "str"}]""").toDS).collect()
> {code}
> {code}
> ...
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
> stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0
> (TID 1) (172.30.3.20 executor driver): java.lang.NullPointerException
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)
> {code}
> Since the mode (by default) is permissive, we shouldn't just fail like above.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
