This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 11e79a81100e [SPARK-57725][4.1][SQL] Fix NPE in AttributeSeq column
resolution when an attribute has a null name
11e79a81100e is described below
commit 11e79a81100ea500fd038b1f37ab48416b01cdf1
Author: Maxim Gekk <[email protected]>
AuthorDate: Sun Jun 28 11:04:28 2026 +0200
[SPARK-57725][4.1][SQL] Fix NPE in AttributeSeq column resolution when an
attribute has a null name
### What changes were proposed in this pull request?
This is a backport of #56831 to `branch-4.1`.
`AttributeSeq` builds case-insensitive name lookup maps (`direct`,
`qualified`, `qualified3Part`, `qualified4Part`) by grouping the attributes on
`attr.name.toLowerCase(Locale.ROOT)`. The grouping key function dereferences
the name without a null check, so a single attribute whose name is `null` makes
`groupBy(_.name.toLowerCase(...))` throw a `NullPointerException`, aborting
resolution of the whole operator with an `INTERNAL_ERROR` (SQLSTATE XX000)
instead of resolving the other columns.
This PR introduces a `namedAttrs` lazy val that filters out null-named
attributes, and builds the four name maps from it instead of from `attrs`.
Positional and expression-id access (`apply(ordinal)`, `indexOf(exprId)`) still
use the full `attrs`, so they are unaffected.
### Why are the changes needed?
A null-named attribute can arise on the JVM side: `StructField` permits a
null name (no `require(name != null)`), and the name flows unchanged through
`DataTypeUtils.toAttribute` into `AttributeReference`. A null-named attribute
is unaddressable by any column reference (a reference's name parts are never
null), so dropping it from the name maps cannot change resolution of any
legitimate reference. It converts the hard `NullPointerException` into correct
resolution of the remaining (na [...]
### Does this PR introduce _any_ user-facing change?
No. It only turns an internal `NullPointerException` / `INTERNAL_ERROR`
into normal column-resolution behavior.
### How was this patch tested?
Backported regression tests in `AttributeResolutionSuite` (covering the
`direct`, `qualified`, `qualified3Part`, and `qualified4Part` maps) and an
end-to-end test in `DataFrameSuite`. The same tests pass on the `branch-4.2`
backport (#56837); CI runs them here.
```
build/sbt 'catalyst/testOnly *AttributeResolutionSuite'
build/sbt 'sql/testOnly *DataFrameSuite -- -z "SPARK-57725: resolve columns
when the input plan has a null-named attribute"'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor
Closes #56838 from MaxGekk/backport-SPARK-57725-4.1.
Authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
---
.../spark/sql/catalyst/expressions/package.scala | 15 ++++++---
.../expressions/AttributeResolutionSuite.scala | 38 ++++++++++++++++++++++
.../org/apache/spark/sql/DataFrameSuite.scala | 11 +++++++
3 files changed, 60 insertions(+), 4 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala
index 114a43c34c04..008d16064ebc 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala
@@ -100,6 +100,13 @@ package object expressions {
// to an array.
@transient private lazy val attrsArray = attrs.toArray
+ // Attributes with a null name (e.g. produced from a StructField built
with a null name via
+ // the DataFrame API) are unaddressable by any column reference, whose
name parts are never
+ // null. Exclude them from the case-insensitive name maps below so that
grouping by the
+ // lower-cased name does not throw a NullPointerException during
resolution (SPARK-57725).
+ // Filter attrsArray (not attrs) to avoid re-traversing a possibly
linked-list attrs.
+ @transient private lazy val namedAttrs: Seq[Attribute] =
attrsArray.filter(_.name != null).toSeq
+
@transient private lazy val exprIdToOrdinal = {
val arr = attrsArray
val map = Maps.newHashMapWithExpectedSize[ExprId, Int](arr.length)
@@ -131,13 +138,13 @@ package object expressions {
/** Map to use for direct case insensitive attribute lookups. */
@transient private lazy val direct: Map[String, Seq[Attribute]] = {
- unique(attrs.groupBy(_.name.toLowerCase(Locale.ROOT)))
+ unique(namedAttrs.groupBy(_.name.toLowerCase(Locale.ROOT)))
}
/** Map to use for qualified case insensitive attribute lookups with 2
part key */
@transient private lazy val qualified: Map[(String, String),
Seq[Attribute]] = {
// key is 2 part: table/alias and name
- val grouped = attrs.filter(_.qualifier.nonEmpty).groupBy {
+ val grouped = namedAttrs.filter(_.qualifier.nonEmpty).groupBy {
a => (a.qualifier.last.toLowerCase(Locale.ROOT),
a.name.toLowerCase(Locale.ROOT))
}
unique(grouped)
@@ -146,7 +153,7 @@ package object expressions {
/** Map to use for qualified case insensitive attribute lookups with 3
part key */
@transient private lazy val qualified3Part: Map[(String, String, String),
Seq[Attribute]] = {
// key is 3 part: database name, table name and name
- val grouped = attrs.filter(a => a.qualifier.length >= 2 &&
a.qualifier.length <= 3)
+ val grouped = namedAttrs.filter(a => a.qualifier.length >= 2 &&
a.qualifier.length <= 3)
.groupBy { a =>
val qualifier = if (a.qualifier.length == 2) {
a.qualifier
@@ -164,7 +171,7 @@ package object expressions {
@transient
private lazy val qualified4Part: Map[(String, String, String, String),
Seq[Attribute]] = {
// key is 4 part: catalog name, database name, table name and name
- val grouped = attrs.filter(_.qualifier.length == 3).groupBy { a =>
+ val grouped = namedAttrs.filter(_.qualifier.length == 3).groupBy { a =>
a.qualifier match {
case Seq(catalog, db, tbl) =>
(catalog.toLowerCase(Locale.ROOT),
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/AttributeResolutionSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/AttributeResolutionSuite.scala
index 29c5bf3b8d2d..49b041f227c0 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/AttributeResolutionSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/AttributeResolutionSuite.scala
@@ -130,6 +130,44 @@ class AttributeResolutionSuite extends SparkFunSuite {
}
}
+ test("SPARK-57725: resolution does not fail when an attribute has a null
name") {
+ // An AttributeReference can carry a null name (e.g. from a StructField
built with a null name
+ // via the DataFrame API). Such an attribute is unaddressable by any
column reference -- name
+ // parts are never null -- so it must simply be skipped by the
case-insensitive name maps
+ // rather than causing a NullPointerException in
groupBy(_.name.toLowerCase(...)).
+ val a = AttributeReference("a", IntegerType)()
+ val nullNamed = AttributeReference(null, IntegerType)()
+
+ // Plain (no qualifier) -> exercises the `direct` map.
+ val attrs = Seq(a, nullNamed)
+ attrs.resolve(Seq("a"), resolver) match {
+ case Some(attr) => assert(attr.semanticEquals(a))
+ case _ => fail()
+ }
+ assert(attrs.resolve(Seq("b"), resolver).isEmpty)
+
+ // 2-part qualifier -> exercises the `qualified` map.
+ val qa = AttributeReference("a", IntegerType)(qualifier = Seq("ns1", "t1"))
+ val qNullNamed = AttributeReference(null, IntegerType)(qualifier =
Seq("ns1", "t1"))
+ val qattrs = Seq(qa, qNullNamed)
+ qattrs.resolve(Seq("t1", "a"), resolver) match {
+ case Some(attr) => assert(attr.semanticEquals(qa))
+ case _ => fail()
+ }
+
+ // 3-part qualifier -> forces the `qualified3Part` (3-part lookup) and
`qualified4Part`
+ // (4-part lookup) maps, so their population is also exercised with a
null-named sibling.
+ val q3a = AttributeReference("a", IntegerType)(qualifier = Seq("cat",
"db", "tbl"))
+ val q3NullNamed = AttributeReference(null, IntegerType)(qualifier =
Seq("cat", "db", "tbl"))
+ val q3attrs = Seq(q3a, q3NullNamed)
+ Seq(Seq("db", "tbl", "a"), Seq("cat", "db", "tbl", "a")).foreach {
nameParts =>
+ q3attrs.resolve(nameParts, resolver) match {
+ case Some(attr) => assert(attr.semanticEquals(q3a))
+ case _ => fail()
+ }
+ }
+ }
+
test("attribute resolution should try to match the longest qualifier") {
// We have two attributes:
// 1) "a.b" where "a" is the name and "b" is the nested field.
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
index 5b88eeefeca7..d9ce3000a0c1 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
@@ -2785,6 +2785,17 @@ class DataFrameSuite extends QueryTest
val df1 = df.select("a").orderBy("b").orderBy("all")
checkAnswer(df1, Seq(Row(1), Row(4)))
}
+
+ test("SPARK-57725: resolve columns when the input plan has a null-named
attribute") {
+ // A null-named AttributeReference can reach the analyzer (e.g. via a
StructField built with a
+ // null name). Selecting another column must still resolve through the
full analyzer path
+ // instead of failing with an internal NullPointerException from the
case-insensitive name
+ // maps in AttributeSeq.
+ val attrs = Seq(AttributeReference(null, IntegerType)(),
AttributeReference("b", IntegerType)())
+ val relation = LocalRelation.fromExternalRows(attrs, Seq(Row(1, 2)))
+ val df = classic.Dataset.ofRows(spark, relation)
+ checkAnswer(df.select("b"), Row(2))
+ }
}
case class GroupByKey(a: Int, b: Int)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
