[
https://issues.apache.org/jira/browse/SPARK-39061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-39061.
----------------------------------
Fix Version/s: 3.3.0
3.2.2
Resolution: Fixed
Issue resolved by pull request 36883
[https://github.com/apache/spark/pull/36883]
> Incorrect results or NPE when using Inline function against an array of
> dynamically created structs
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-39061
> URL: https://issues.apache.org/jira/browse/SPARK-39061
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.2.1, 3.3.0, 3.4.0
> Reporter: Bruce Robbins
> Assignee: Bruce Robbins
> Priority: Major
> Labels: correctness
> Fix For: 3.3.0, 3.2.2
>
>
> The following query returns incorrect results:
> {noformat}
> spark-sql> select inline(array(named_struct('a', 1, 'b', 2), null));
> 1 2
> -1 -1
> Time taken: 4.053 seconds, Fetched 2 row(s)
> spark-sql>
> {noformat}
> In Hive, the last row is {{NULL, NULL}}:
> {noformat}
> Beeline version 2.3.9 by Apache Hive
> 0: jdbc:hive2://localhost:10000> select inline(array(named_struct('a', 1,
> 'b', 2), null));
> +-------+-------+
> | a | b |
> +-------+-------+
> | 1 | 2 |
> | NULL | NULL |
> +-------+-------+
> 2 rows selected (1.355 seconds)
> 0: jdbc:hive2://localhost:10000>
> {noformat}
> If the struct has string fields, you get a {{NullPointerException}}:
> {noformat}
> spark-sql> select inline(array(named_struct('a', '1', 'b', '2'), null));
> 22/04/28 16:51:54 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 2)
> java.lang.NullPointerException: null
> at
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:110)
> ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.generate_doConsume_0$(Unknown
> Source) ~[?:?]
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
> Source) ~[?:?]
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> ~[spark-sql_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
> {noformat}
> You can work around the issue by casting the null entry of the array:
> {noformat}
> spark-sql> select inline(array(named_struct('a', 1, 'b', 2), cast(null as
> struct<a:int, b:int>)));
> 1 2
> NULL NULL
> Time taken: 0.068 seconds, Fetched 2 row(s)
> spark-sql>
> {noformat}
> As far as I can tell, this issue only happens with arrays of structs where
> the structs are created in an inline table or in a projection.
> The fields of the struct are not getting set to {{nullable = true}} when
> there is no example in the array where the field is set to {{null}}. As a
> result, {{GenerateUnsafeProjection.createCode}} generates bad code: it has no
> code to create a row of null columns, so it just creates a row from variables
> set with default values.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
