[
https://issues.apache.org/jira/browse/SPARK-57729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-57729:
-----------------------------
Description:
h2. Summary
{{CreateNamedStruct.dataType}} builds each field with
{{StructField(name.toString, ...)}}.
When a field name is {{null}}, {{name.toString}} throws a
{{NullPointerException}}. This is
reached eagerly while building a {{RowEncoder}} serializer, so it crashes
before any
analysis/resolution runs.
h2. Affected code
{{org.apache.spark.sql.catalyst.expressions.CreateNamedStruct.dataType}} in
{{sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala}}:
{code:scala}
override lazy val dataType: StructType = {
val fields = names.zip(valExprs).map {
case (name, expr) =>
val metadata = expr match {
case ne: NamedExpression => ne.metadata
case _ => Metadata.empty
}
StructField(name.toString, expr.dataType, expr.nullable, metadata) //
NPE if name == null
}
StructType(fields)
}
{code}
h2. Reproduction
{code:scala}
import org.apache.spark.sql.Row
import org.apache.spark.sql.types._
val schema = StructType(Seq(StructField(null, IntegerType), StructField("b",
IntegerType)))
val df = spark.createDataFrame(spark.sparkContext.parallelize(Seq(Row(1, 2))),
schema)
df.select("b").collect()
{code}
Result:
{code}
java.lang.NullPointerException
at
org.apache.spark.sql.catalyst.expressions.CreateNamedStruct.$anonfun$dataType$9(complexTypeCreator.scala:473)
at
org.apache.spark.sql.catalyst.SerializerBuildHelper$.createSerializerForObject(...)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(...)
{code}
h2. Root cause
A null field name is invalid -- {{CreateNamedStruct.checkInputDataTypes}}
already rejects it
({{names.contains(null)}} -> {{UNEXPECTED_NULL}}) -- but {{dataType}}
dereferences
{{name.toString}} before type checking, and the encoder calls {{dataType}}
directly. The
underlying cause is that {{StructField}} permits a null name (no {{require(name
!= null)}}).
h2. Related
Found while backporting SPARK-57725 (NPE in {{AttributeSeq}} column resolution
with a
null-named attribute). This is an independent, pre-existing NPE on the same
null-field-name
edge case; on master it reproduces via {{createDataFrame}} as shown above.
> Fix NPE in CreateNamedStruct.dataType when a struct field name is null
> ----------------------------------------------------------------------
>
> Key: SPARK-57729
> URL: https://issues.apache.org/jira/browse/SPARK-57729
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Max Gekk
> Priority: Major
>
> h2. Summary
> {{CreateNamedStruct.dataType}} builds each field with
> {{StructField(name.toString, ...)}}.
> When a field name is {{null}}, {{name.toString}} throws a
> {{NullPointerException}}. This is
> reached eagerly while building a {{RowEncoder}} serializer, so it crashes
> before any
> analysis/resolution runs.
> h2. Affected code
> {{org.apache.spark.sql.catalyst.expressions.CreateNamedStruct.dataType}} in
> {{sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala}}:
> {code:scala}
> override lazy val dataType: StructType = {
> val fields = names.zip(valExprs).map {
> case (name, expr) =>
> val metadata = expr match {
> case ne: NamedExpression => ne.metadata
> case _ => Metadata.empty
> }
> StructField(name.toString, expr.dataType, expr.nullable, metadata) //
> NPE if name == null
> }
> StructType(fields)
> }
> {code}
> h2. Reproduction
> {code:scala}
> import org.apache.spark.sql.Row
> import org.apache.spark.sql.types._
> val schema = StructType(Seq(StructField(null, IntegerType), StructField("b",
> IntegerType)))
> val df = spark.createDataFrame(spark.sparkContext.parallelize(Seq(Row(1,
> 2))), schema)
> df.select("b").collect()
> {code}
> Result:
> {code}
> java.lang.NullPointerException
> at
> org.apache.spark.sql.catalyst.expressions.CreateNamedStruct.$anonfun$dataType$9(complexTypeCreator.scala:473)
> at
> org.apache.spark.sql.catalyst.SerializerBuildHelper$.createSerializerForObject(...)
> at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(...)
> {code}
> h2. Root cause
> A null field name is invalid -- {{CreateNamedStruct.checkInputDataTypes}}
> already rejects it
> ({{names.contains(null)}} -> {{UNEXPECTED_NULL}}) -- but {{dataType}}
> dereferences
> {{name.toString}} before type checking, and the encoder calls {{dataType}}
> directly. The
> underlying cause is that {{StructField}} permits a null name (no
> {{require(name != null)}}).
> h2. Related
> Found while backporting SPARK-57725 (NPE in {{AttributeSeq}} column
> resolution with a
> null-named attribute). This is an independent, pre-existing NPE on the same
> null-field-name
> edge case; on master it reproduces via {{createDataFrame}} as shown above.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
