[
https://issues.apache.org/jira/browse/SPARK-57736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk reassigned SPARK-57736:
--------------------------------
Assignee: Max Gekk
> Fix NPE in CreateNamedStruct.dataType when a struct field name is null
> ----------------------------------------------------------------------
>
> Key: SPARK-57736
> URL: https://issues.apache.org/jira/browse/SPARK-57736
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0, 4.1.2, 4.0.3, 4.3.0
> Reporter: Max Gekk
> Assignee: Max Gekk
> Priority: Major
> Labels: pull-request-available
>
> h2. Summary
> {{CreateNamedStruct.dataType}} builds each field with
> {{StructField(name.toString, ...)}}.
> When a field name is {{null}}, {{name.toString}} throws a
> {{NullPointerException}}. This is
> reached eagerly while building a {{RowEncoder}} serializer, so it crashes
> before any
> analysis/resolution runs.
> h2. Affected code
> {{org.apache.spark.sql.catalyst.expressions.CreateNamedStruct.dataType}} in
> {{sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala}}:
> {code:scala}
> override lazy val dataType: StructType = {
> val fields = names.zip(valExprs).map {
> case (name, expr) =>
> val metadata = expr match {
> case ne: NamedExpression => ne.metadata
> case gsf: GetStructField => gsf.metadata
> case _ => Metadata.empty
> }
> StructField(name.toString, expr.dataType, expr.nullable, metadata) //
> NPE if name == null
> }
> StructType(fields)
> }
> {code}
> h2. Reproduction
> {code:scala}
> import org.apache.spark.sql.catalyst.expressions.{CreateNamedStruct, Literal}
> import org.apache.spark.sql.types.{IntegerType, StringType}
> CreateNamedStruct(Seq(Literal.create(null, StringType), Literal(1))).dataType
> {code}
> It also surfaces end-to-end via the encoder:
> {code:scala}
> import org.apache.spark.sql.Row
> import org.apache.spark.sql.types._
> val schema = StructType(Seq(StructField(null, IntegerType), StructField("b",
> IntegerType)))
> spark.createDataFrame(spark.sparkContext.parallelize(Seq(Row(1, 2))), schema)
> // builds RowEncoder -> NPE
> {code}
> Result:
> {code}
> java.lang.NullPointerException
> at
> org.apache.spark.sql.catalyst.expressions.CreateNamedStruct.$anonfun$dataType$9(complexTypeCreator.scala:473)
> at
> org.apache.spark.sql.catalyst.SerializerBuildHelper$.createSerializerForObject(...)
> at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(...)
> {code}
> h2. Root cause
> A null field name is invalid -- {{CreateNamedStruct.checkInputDataTypes}}
> already rejects it
> ({{names.contains(null)}} -> {{UNEXPECTED_NULL}}) -- but {{dataType}}
> dereferences
> {{name.toString}} before type checking, and the encoder calls {{dataType}}
> directly. The
> underlying cause is that {{StructField}} permits a null name (no
> {{require(name != null)}}).
> h2. Proposed fix
> Make {{dataType}} null-safe and preserve the null name (consistent with
> SPARK-57725, which made
> {{AttributeSeq}} tolerate null-named attributes):
> {code:scala}
> StructField(if (name == null) null else name.toString, expr.dataType,
> expr.nullable, metadata)
> {code}
> A regression test in {{ComplexTypeSuite}} asserts {{dataType}} no longer
> throws and preserves the
> null field name.
> h2. Note
> This fixes the specific {{CreateNamedStruct.dataType}} NPE. The full
> {{createDataFrame(schemaWithNullFieldName)}} scenario hits additional,
> independent null-name sites
> further along (e.g. a {{StructField.name.equalsIgnoreCase}} schema comparison
> during resolution),
> which are separate pre-existing issues and out of scope here.
> h2. Related
> Found while backporting SPARK-57725 (NPE in {{AttributeSeq}} column
> resolution with a null-named
> attribute). Independent, pre-existing NPE on the same null-field-name edge
> case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
