This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 6754a42bfeca [SPARK-57725][SQL] Fix NPE in AttributeSeq column
resolution when an attribute has a null name
6754a42bfeca is described below
commit 6754a42bfeca973fb2668eb1958b8d2a83d2cbf1
Author: Maxim Gekk <[email protected]>
AuthorDate: Sat Jun 27 22:50:47 2026 +0200
[SPARK-57725][SQL] Fix NPE in AttributeSeq column resolution when an
attribute has a null name
### What changes were proposed in this pull request?
`AttributeSeq` builds case-insensitive name lookup maps (`direct`,
`qualified`, `qualified3Part`, `qualified4Part`) by grouping the attributes on
`attr.name.toLowerCase(Locale.ROOT)`. The grouping key function dereferences
the name without a null check, so a single attribute whose name is `null` makes
`groupBy(_.name.toLowerCase(...))` throw a `NullPointerException`, aborting
resolution of the whole operator with an `INTERNAL_ERROR` (SQLSTATE XX000)
instead of resolving the other columns.
This PR introduces a `namedAttrs` lazy val that filters out null-named
attributes, and builds the four name maps from it instead of from `attrs`.
Positional and expression-id access (`apply(ordinal)`, `indexOf(exprId)`) still
use the full `attrs`, so they are unaffected.
### Why are the changes needed?
A null-named attribute can arise on the JVM side: `StructField` permits a
null name (no `require(name != null)`), and the name flows unchanged through
`DataTypeUtils.toAttribute` into `AttributeReference`. For example:
```scala
val schema = StructType(Seq(StructField(null, IntegerType),
StructField("b", IntegerType)))
val df = spark.createDataFrame(spark.sparkContext.parallelize(Seq(Row(1,
2))), schema)
df.select("b").collect() // forces resolution -> NPE / INTERNAL_ERROR
```
A null-named attribute is unaddressable by any column reference (a
reference's name parts are never null), so dropping it from the name maps
cannot change resolution of any legitimate reference. It converts the hard
`NullPointerException` into correct resolution of the remaining (named)
attributes, or a normal unresolved-column error if the null-named column is
referenced.
### Does this PR introduce _any_ user-facing change?
No. It only turns an internal `NullPointerException` / `INTERNAL_ERROR`
into normal column-resolution behavior.
### How was this patch tested?
Added a regression test `SPARK-57725: resolution does not fail when an
attribute has a null name` in `AttributeResolutionSuite`, exercising the
`direct`, `qualified`, `qualified3Part`, and `qualified4Part` maps with a
null-named sibling attribute.
```
build/sbt 'catalyst/testOnly *AttributeResolutionSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor
Closes #56831 from MaxGekk/fix-npe.
Authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit 13b3237daf0becda60c23db3f6398547c2c83413)
Signed-off-by: Max Gekk <[email protected]>
---
.../spark/sql/catalyst/expressions/package.scala | 15 ++++++---
.../expressions/AttributeResolutionSuite.scala | 38 ++++++++++++++++++++++
.../org/apache/spark/sql/DataFrameSuite.scala | 11 +++++++
3 files changed, 60 insertions(+), 4 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala
index a3008a949ec0..b3525da97f8d 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala
@@ -100,6 +100,13 @@ package object expressions {
// to an array.
@transient private lazy val attrsArray = attrs.toArray
+ // Attributes with a null name (e.g. produced from a StructField built
with a null name via
+ // the DataFrame API) are unaddressable by any column reference, whose
name parts are never
+ // null. Exclude them from the case-insensitive name maps below so that
grouping by the
+ // lower-cased name does not throw a NullPointerException during
resolution (SPARK-57725).
+ // Filter attrsArray (not attrs) to avoid re-traversing a possibly
linked-list attrs.
+ @transient private lazy val namedAttrs: Seq[Attribute] =
attrsArray.filter(_.name != null).toSeq
+
@transient private lazy val exprIdToOrdinal = {
val arr = attrsArray
val map = Maps.newHashMapWithExpectedSize[ExprId, Int](arr.length)
@@ -131,13 +138,13 @@ package object expressions {
/** Map to use for direct case insensitive attribute lookups. */
@transient private lazy val direct: Map[String, Seq[Attribute]] = {
- unique(attrs.groupBy(_.name.toLowerCase(Locale.ROOT)))
+ unique(namedAttrs.groupBy(_.name.toLowerCase(Locale.ROOT)))
}
/** Map to use for qualified case insensitive attribute lookups with 2
part key */
@transient private lazy val qualified: Map[(String, String),
Seq[Attribute]] = {
// key is 2 part: table/alias and name
- val grouped = attrs.filter(_.qualifier.nonEmpty).groupBy {
+ val grouped = namedAttrs.filter(_.qualifier.nonEmpty).groupBy {
a => (a.qualifier.last.toLowerCase(Locale.ROOT),
a.name.toLowerCase(Locale.ROOT))
}
unique(grouped)
@@ -146,7 +153,7 @@ package object expressions {
/** Map to use for qualified case insensitive attribute lookups with 3
part key */
@transient private lazy val qualified3Part: Map[(String, String, String),
Seq[Attribute]] = {
// key is 3 part: database name, table name and name
- val grouped = attrs.filter(a => a.qualifier.length >= 2 &&
a.qualifier.length <= 3)
+ val grouped = namedAttrs.filter(a => a.qualifier.length >= 2 &&
a.qualifier.length <= 3)
.groupBy { a =>
val qualifier = if (a.qualifier.length == 2) {
a.qualifier
@@ -164,7 +171,7 @@ package object expressions {
@transient
private lazy val qualified4Part: Map[(String, String, String, String),
Seq[Attribute]] = {
// key is 4 part: catalog name, database name, table name and name
- val grouped = attrs.filter(_.qualifier.length == 3).groupBy { a =>
+ val grouped = namedAttrs.filter(_.qualifier.length == 3).groupBy { a =>
a.qualifier match {
case Seq(catalog, db, tbl) =>
(catalog.toLowerCase(Locale.ROOT),
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/AttributeResolutionSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/AttributeResolutionSuite.scala
index 29c5bf3b8d2d..49b041f227c0 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/AttributeResolutionSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/AttributeResolutionSuite.scala
@@ -130,6 +130,44 @@ class AttributeResolutionSuite extends SparkFunSuite {
}
}
+ test("SPARK-57725: resolution does not fail when an attribute has a null
name") {
+ // An AttributeReference can carry a null name (e.g. from a StructField
built with a null name
+ // via the DataFrame API). Such an attribute is unaddressable by any
column reference -- name
+ // parts are never null -- so it must simply be skipped by the
case-insensitive name maps
+ // rather than causing a NullPointerException in
groupBy(_.name.toLowerCase(...)).
+ val a = AttributeReference("a", IntegerType)()
+ val nullNamed = AttributeReference(null, IntegerType)()
+
+ // Plain (no qualifier) -> exercises the `direct` map.
+ val attrs = Seq(a, nullNamed)
+ attrs.resolve(Seq("a"), resolver) match {
+ case Some(attr) => assert(attr.semanticEquals(a))
+ case _ => fail()
+ }
+ assert(attrs.resolve(Seq("b"), resolver).isEmpty)
+
+ // 2-part qualifier -> exercises the `qualified` map.
+ val qa = AttributeReference("a", IntegerType)(qualifier = Seq("ns1", "t1"))
+ val qNullNamed = AttributeReference(null, IntegerType)(qualifier =
Seq("ns1", "t1"))
+ val qattrs = Seq(qa, qNullNamed)
+ qattrs.resolve(Seq("t1", "a"), resolver) match {
+ case Some(attr) => assert(attr.semanticEquals(qa))
+ case _ => fail()
+ }
+
+ // 3-part qualifier -> forces the `qualified3Part` (3-part lookup) and
`qualified4Part`
+ // (4-part lookup) maps, so their population is also exercised with a
null-named sibling.
+ val q3a = AttributeReference("a", IntegerType)(qualifier = Seq("cat",
"db", "tbl"))
+ val q3NullNamed = AttributeReference(null, IntegerType)(qualifier =
Seq("cat", "db", "tbl"))
+ val q3attrs = Seq(q3a, q3NullNamed)
+ Seq(Seq("db", "tbl", "a"), Seq("cat", "db", "tbl", "a")).foreach {
nameParts =>
+ q3attrs.resolve(nameParts, resolver) match {
+ case Some(attr) => assert(attr.semanticEquals(q3a))
+ case _ => fail()
+ }
+ }
+ }
+
test("attribute resolution should try to match the longest qualifier") {
// We have two attributes:
// 1) "a.b" where "a" is the name and "b" is the nested field.
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
index 3d6fd5c1bf5f..577ae025f1df 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
@@ -2785,6 +2785,17 @@ class DataFrameSuite extends SharedSparkSession
val df1 = df.select("a").orderBy("b").orderBy("all")
checkAnswer(df1, Seq(Row(1), Row(4)))
}
+
+ test("SPARK-57725: resolve columns when the input plan has a null-named
attribute") {
+ // A null-named AttributeReference can reach the analyzer (e.g. via a
StructField built with a
+ // null name). Selecting another column must still resolve through the
full analyzer path
+ // instead of failing with an internal NullPointerException from the
case-insensitive name
+ // maps in AttributeSeq.
+ val attrs = Seq(AttributeReference(null, IntegerType)(),
AttributeReference("b", IntegerType)())
+ val relation = LocalRelation.fromExternalRows(attrs, Seq(Row(1, 2)))
+ val df = classic.Dataset.ofRows(spark, relation)
+ checkAnswer(df.select("b"), Row(2))
+ }
}
case class GroupByKey(a: Int, b: Int)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
