Max Gekk created SPARK-57725:
--------------------------------
Summary: NPE in AttributeSeq column resolution when an attribute
has a null name
Key: SPARK-57725
URL: https://issues.apache.org/jira/browse/SPARK-57725
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.3.0
Reporter: Max Gekk
h2. Summary
Column resolution throws an internal {{NullPointerException}} when the input
plan exposes an
{{Attribute}} whose {{name}} is {{null}}. {{AttributeSeq}} builds
case-insensitive name lookup
maps keyed on {{attr.name.toLowerCase(Locale.ROOT)}}, and the grouping key
function dereferences
the name without a null check, so a single null-named attribute aborts
resolution of the whole
operator with an {{INTERNAL_ERROR}} instead of resolving the other columns
(or producing a normal
unresolved-column error).
h2. Affected code
{{org.apache.spark.sql.catalyst.expressions.package.AttributeSeq}} (file
{{sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala}}).
The
{{direct}}, {{qualified}}, {{qualified3Part}} and {{qualified4Part}} lazy
maps all group by
{{_.name.toLowerCase(Locale.ROOT)}}:
{code:scala}
@transient private lazy val direct: Map[String, Seq[Attribute]] = {
unique(attrs.groupBy(_.name.toLowerCase(Locale.ROOT))) // NPE if a.name
== null
}
{code}
This grouping has been present (unchanged) since well before SPARK-50037
reworked
{{AttributeSeq.resolve}}, so the issue is long-standing rather than a recent
regression.
h2. Reproduction (minimal, Catalyst level)
{code:scala}
import org.apache.spark.sql.catalyst.expressions.{Attribute,
AttributeReference}
import org.apache.spark.sql.catalyst.analysis.caseInsensitiveResolution
import org.apache.spark.sql.types.IntegerType
val attrs: Seq[Attribute] = Seq(
AttributeReference("a", IntegerType)(),
AttributeReference(null, IntegerType)()) // an attribute with a null name
// Resolving any real column forces the case-insensitive name map and throws:
attrs.resolve(Seq("a"), caseInsensitiveResolution)
{code}
Result:
{code:none}
java.lang.NullPointerException: Cannot invoke
"String.toLowerCase(java.util.Locale)" because
the return value of
"org.apache.spark.sql.catalyst.expressions.Attribute.name()" is null
at
org.apache.spark.sql.catalyst.expressions.package$AttributeSeq.$anonfun$direct$1(package.scala:...)
at scala.collection.IterableOps.groupBy(...)
at
org.apache.spark.sql.catalyst.expressions.package$AttributeSeq.direct$lzycompute(package.scala:...)
at
org.apache.spark.sql.catalyst.expressions.package$AttributeSeq.matchWithThreeOrLessQualifierParts(...)
at
org.apache.spark.sql.catalyst.expressions.package$AttributeSeq.getCandidatesForResolution(...)
at
org.apache.spark.sql.catalyst.expressions.package$AttributeSeq.resolve(...)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveChildren(LogicalPlan.scala:...)
at
org.apache.spark.sql.catalyst.analysis.ColumnResolutionHelper.resolveExpressionByPlanChildren(...)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences...
{code}
In practice this surfaces during normal analysis (e.g. a {{DataFrame.filter}}
whose child plan
carries an attribute with a null name) as an uncaught {{INTERNAL_ERROR}}
(SQLSTATE {{XX000}}).
h2. Root cause
{{StructField}} permits a null {{name}} (no {{require(name != null)}}), and
the name flows
unchanged through {{DataTypeUtils.toAttribute}} into {{AttributeReference}}.
When such an attribute
reaches {{AttributeSeq}}, the {{groupBy(_.name.toLowerCase(...))}} key
function NPEs. The same
null-unsafe {{_.name.toLowerCase}} pattern exists in all four name maps.
h2. Proposed fix
Exclude null-named attributes when building the case-insensitive name maps. A
null-named attribute
is unaddressable by any column reference — a reference's name parts are never
null — so dropping it
from the name maps cannot change resolution of any legitimate reference. It
converts the hard
{{NullPointerException}} into correct resolution of the remaining (named)
attributes, or a normal
unresolved-column error if the null-named column is referenced:
{code:scala}
// Build the name maps from attributes that actually have a name.
private lazy val namedAttrs: Seq[Attribute] = attrs.filter(_.name != null)
// ... use `namedAttrs` instead of `attrs` in
direct/qualified/qualified3Part/qualified4Part.
{code}
A regression test asserting that {{AttributeSeq.resolve}} no longer throws
when a null-named
attribute is present (covering the unqualified {{direct}} map and the
qualified maps) accompanies
the fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
