[
https://issues.apache.org/jira/browse/SPARK-57729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-57729:
-----------------------------
Affects Version/s: 4.3.0
(was: 5.0.0)
> Fix NPE in CreateNamedStruct.dataType when a struct field name is null
> ----------------------------------------------------------------------
>
> Key: SPARK-57729
> URL: https://issues.apache.org/jira/browse/SPARK-57729
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Major
>
> h2. Summary
> {{CreateNamedStruct.dataType}} builds each field with
> {{StructField(name.toString, ...)}}.
> When a field name is {{null}}, {{name.toString}} throws a
> {{NullPointerException}}. This is
> reached eagerly while building a {{RowEncoder}} serializer, so it crashes
> before any
> analysis/resolution runs.
> h2. Affected code
> {{org.apache.spark.sql.catalyst.expressions.CreateNamedStruct.dataType}} in
> {{sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala}}:
> {code:scala}
> override lazy val dataType: StructType = {
> val fields = names.zip(valExprs).map {
> case (name, expr) =>
> val metadata = expr match {
> case ne: NamedExpression => ne.metadata
> case _ => Metadata.empty
> }
> StructField(name.toString, expr.dataType, expr.nullable, metadata) //
> NPE if name == null
> }
> StructType(fields)
> }
> {code}
> h2. Reproduction
> {code:scala}
> import org.apache.spark.sql.Row
> import org.apache.spark.sql.types._
> val schema = StructType(Seq(StructField(null, IntegerType), StructField("b",
> IntegerType)))
> val df = spark.createDataFrame(spark.sparkContext.parallelize(Seq(Row(1,
> 2))), schema)
> df.select("b").collect()
> {code}
> Result:
> {code}
> java.lang.NullPointerException
> at
> org.apache.spark.sql.catalyst.expressions.CreateNamedStruct.$anonfun$dataType$9(complexTypeCreator.scala:473)
> at
> org.apache.spark.sql.catalyst.SerializerBuildHelper$.createSerializerForObject(...)
> at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(...)
> {code}
> h2. Root cause
> A null field name is invalid -- {{CreateNamedStruct.checkInputDataTypes}}
> already rejects it
> ({{names.contains(null)}} -> {{UNEXPECTED_NULL}}) -- but {{dataType}}
> dereferences
> {{name.toString}} before type checking, and the encoder calls {{dataType}}
> directly. The
> underlying cause is that {{StructField}} permits a null name (no
> {{require(name != null)}}).
> h2. Related
> Found while backporting SPARK-57725 (NPE in {{AttributeSeq}} column
> resolution with a
> null-named attribute). This is an independent, pre-existing NPE on the same
> null-field-name
> edge case; on master it reproduces via {{createDataFrame}} as shown above.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
