ArnoldYing opened a new issue, #18173:
URL: https://github.com/apache/iceberg/issues/18173
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
Iceberg's Parquet row-group filter appears to compare raw `TIMESTAMP_MILLIS`
min/max statistics
with Iceberg timestamp literals represented in microseconds.
This can fail in both directions:
- Iceberg may read a row group whose statistics prove it cannot match a
predicate.
- More importantly, Iceberg may prune a row group that contains matching
rows, producing an
incorrect query result.
For a Parquet row group containing only `2020-01-01T00:00:00`, the values
compared are:
```text
Parquet TIMESTAMP_MILLIS statistic for 2020: 1,577,836,800,000
Iceberg timestamp literal for 1990 in micros: 631,152,000,000,000
```
Without converting the Parquet bound from milliseconds to microseconds, the
2020 bound compares
as earlier than the 1990 literal.
### Expected behavior
Iceberg should convert Parquet `TIMESTAMP_MILLIS` statistics to its internal
microsecond
representation before evaluating predicates.
For data containing only 2020 timestamps:
- `event_time < TIMESTAMP '1990-01-01 00:00:00'` should prune the row group.
- `event_time > TIMESTAMP '1990-01-01 00:00:00'` must retain the row group
and return the rows.
### Actual behavior
- The focused row-group test shows that Iceberg retains the row group for `<
1990`.
- The Spark E2E test shows that Iceberg prunes all matching data for `>
1990`; the query returns
zero rows instead of 2,000.
### Minimal reproduction
The reproduction contains two tests plus one Maven POM:
```text
pom.xml
src/test/java/dev/reproducer/TimestampMillisStatsPruningTest.java
src/test/java/dev/reproducer/SparkIcebergIncorrectPruningE2ETest.java
```
Requirements: Java 17 and Maven. The POM pins Iceberg 1.11.0 and Spark 3.5.5.
[pom.xml](https://github.com/user-attachments/files/32425834/pom.xml)
[SparkIcebergIncorrectPruningE2ETest.java](https://github.com/user-attachments/files/32425837/SparkIcebergIncorrectPruningE2ETest.java)
[TimestampMillisStatsPruningTest.java](https://github.com/user-attachments/files/32425838/TimestampMillisStatsPruningTest.java)
Observed with Iceberg 1.11.0:
```text
Direct Parquet read with > 1990: 2000
Unfiltered Iceberg read: 2000
Iceberg read with > 1990: 0
```
The direct Parquet count proves that every source row matches. The
unfiltered Iceberg count proves
that the import retained all rows. Only the filtered Iceberg read loses the
matching rows.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
