rdtr opened a new pull request, #13206:
URL: https://github.com/apache/gluten/pull/13206
## What changes are proposed in this pull request?
I ran the Spark 4.2 test "SPARK-57725: resolve columns when the input plan
has a null-named attribute" in `GlutenDataFrameSuite`, and it failed with a
`NullPointerException`:
```
java.lang.NullPointerException: Cannot invoke
"String.toLowerCase(java.util.Locale)" because "name" is null
at
org.apache.gluten.expression.ConverterUtils$.normalizeColName(ConverterUtils.scala:53)
at
org.apache.gluten.expression.ConverterUtils$.getShortAttributeName(ConverterUtils.scala:57)
at
org.apache.gluten.expression.ConverterUtils$.genColumnNameWithExprId(ConverterUtils.scala:71)
at
org.apache.gluten.expression.ConverterUtils$.collectAttributeNamesWithExprId(ConverterUtils.scala:91)
at
org.apache.gluten.substrait.rel.RelBuilder.makeReadRelForInputIterator(RelBuilder.java:163)
```
A column can have a null name. For example, you can make one with a
`StructField` that has a null name in the DataFrame API. Spark itself also
failed with an NPE on this, and Spark 4.2.0 fixed it (SPARK-57725). The new
Spark test selects column `b` from a `LocalRelation` that also has a column
with a null name.
In Gluten, the plan is `LocalTableScan [null#0, b#1]` ->
`RowToVeloxColumnar` -> `InputIteratorTransformer`. Gluten makes a name like
`b#1` (name + `#` + expression ID) for each input column. `normalizeColName`
makes the name lower case, but it doesn't check for null, so it throws the NPE.
I didn't check why this bug (looks like not new) is surfaced now, likely
because before Spark failed first. Spark 4.2 fixed its side, so now the plan
gets to Gluten and the NPE shows up there.
This PR changes `normalizeColName` to return an empty name for a null name.
The generated name still has the expression ID, so the column becomes `#0`, and
it is still unique. Like in Spark's fix, you can't refer to this column by
name, but the query works.
Part of #13179
## How was this patch tested?
- Added a test to `ConverterUtilsSuite`. It calls `genColumnNameWithExprId`
and `genColumnNameWithoutExprId` with a column that has a null name, and checks
that they return `#<exprId>` and an empty name. It runs on all Spark versions.
- Added a test to `VeloxLocalTableScanSuite`. It does the same thing as the
Spark test: it selects `b` from a `LocalRelation` that has a column with a null
name. It checks the result and that the Project runs in Velox. It runs with
`spark.gluten.sql.columnar.localTableScan` on and off, so it covers both the
native LocalTableScan and the `RowToVeloxColumnar` path. It runs only on Spark
4.2 or later, because older Spark versions fail on this plan by themselves.
Results:
- Without this change: the `VeloxLocalTableScanSuite` test fails on Spark
4.2, and the `ConverterUtilsSuite` test fails on Spark 4.1. Both fail with the
NPE above.
- With this change, Spark 4.2: `ConverterUtilsSuite` (3 tests) and
`VeloxLocalTableScanSuite` (19 tests) pass.
- With this change, Spark 4.1: `ConverterUtilsSuite` (3 tests) and the other
18 tests in `VeloxLocalTableScanSuite` pass.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
