yangshangqing95 opened a new issue, #17632:
URL: https://github.com/apache/iceberg/issues/17632
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
When a floating-point column with an initial default is added to an Iceberg
table, querying older Parquet files with a `notNaN` predicate can throw a
`NullPointerException`.
This can be reproduced in Spark Shell by writing data before adding the
column:
```scala
val table = Spark3Util.loadIcebergTable(spark, tableName)
table
.updateSchema()
.addColumn(
"new_float",
Types.FloatType.get(),
Literal.of(java.lang.Float.valueOf(1.0f)))
.commit()
spark.catalog.refreshTable(tableName)
spark.sql(
s"""
|SELECT id, new_float
|FROM $tableName
|WHERE new_float <> CAST('NaN' AS FLOAT)
|""".stripMargin
).show(false)
```
The query fails with:
```text
java.lang.NullPointerException:
Cannot invoke "java.lang.Boolean.booleanValue()"
because the return value of "java.util.Map.get(Object)" is null
at ParquetDictionaryRowGroupFilter$EvalVisitor.notNaN
```
`mayContainNulls` is built from the physical columns in the Parquet row
group. Because the newly added column is absent from older files, its field ID
has no entry in the map.
The dictionary filter should conservatively retain the row group when null
metadata for the column is unavailable.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
