[
https://issues.apache.org/jira/browse/SPARK-39496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-39496:
---------------------------------
Fix Version/s: 3.1.3
> Inline eval path cannot handle null structs
> -------------------------------------------
>
> Key: SPARK-39496
> URL: https://issues.apache.org/jira/browse/SPARK-39496
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.3, 3.2.1, 3.3.0, 3.4.0
> Reporter: Bruce Robbins
> Assignee: Bruce Robbins
> Priority: Major
> Fix For: 3.1.3, 3.2.2, 3.4.0, 3.3.1
>
>
> This issue is somewhat similar to SPARK-39061, but for the eval path rather
> than the codegen path.
> Example:
> {noformat}
> set spark.sql.codegen.wholeStage=false;
> select inline(array(named_struct('a', 1, 'b', 2), null));
> {noformat}
> This results in a NullPointerException:
> {noformat}
> 22/06/16 15:10:06 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
> java.lang.NullPointerException
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
> Source)
> at
> org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$11(GenerateExec.scala:122)
> {noformat}
> The next example doesn't require setting {{spark.sql.codegen.wholeStage}} to
> {{{}false{}}}:
> {noformat}
> val dfWide = (Seq((1))
> .toDF("col0")
> .selectExpr(Seq.tabulate(99)(x => s"$x as col${x + 1}"): _*))
> val df = (dfWide
> .selectExpr("*", "array(named_struct('a', 1, 'b', 2), null) as
> struct_array"))
> df.selectExpr("*", "inline(struct_array)").collect
> {noformat}
> The result is similar:
> {noformat}
> 22/06/16 15:18:55 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)/
> 1]
> java.lang.NullPointerException
> at
> org.apache.spark.sql.catalyst.expressions.JoinedRow.isNullAt(JoinedRow.scala:80)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields_0_8$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
> Source)
> at
> org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$11(GenerateExec.scala:122)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
