AbdelrahmanElhawary opened a new pull request, #17143:
URL: https://github.com/apache/iceberg/pull/17143
### Problem Context
Apache Iceberg supports reading Parquet files with `TIMESTAMP_MILLIS`
annotations by converting the millisecond values to Iceberg's internal
microsecond representation ($\times 1000$). While this scaling logic works
correctly for `PLAIN` encoded pages (via `TimestampMillisReader`), it is
completely bypassed when a column is entirely dictionary-encoded (e.g., columns
with duplicate or low-cardinality values, like a batch extraction timestamp).
When this optimization occurs, the values are displayed as incorrect dates
in the year 1970 because raw millisecond values are treated as microseconds.
Closes #17135
### Root Cause Analysis
In `VectorizedArrowReader#read`, the engine checks whether a column segment
produces a dictionary-encoded vector:
```
boolean dictEncoded =
vectorizedColumnIterator.producesDictionaryEncodedVector();
if (vectorizedColumnIterator.hasNext()) {
if (dictEncoded) {
vectorizedColumnIterator.dictionaryBatchReader().nextBatch(vec, -1,
nullabilityHolder);
} else {
switch (readType) { ... }
}
}
```
If `dictEncoded` is true, the reader completely bypasses the type-specific
switch statement—which normally maps to `ReadType.TIMESTAMP_MILLIS` and uses
the correct `TimestampMillisReader`. Instead, it shortcuts by populating a
generic `IntVector` with raw dictionary IDs and attaches the raw Parquet
`Dictionary` object straight to the `VectorHolder` returned to Spark.
When Spark eventually decodes these IDs via` Dictionary#decodeToLong(id)`,
it receives unscaled milliseconds from the raw Parquet metadata, resulting in
corrupted timestamps.
### Solution
The fix intercepts the raw Parquet `Dictionary` inside
`VectorizedArrowReader#setRowGroupInfo` right after initialization.
If the column's modern `LogicalTypeAnnotation` indicates it is a `TIMESTAMP`
with `TimeUnit.MILLIS` precision, the dictionary is wrapped in a proxy wrapper.
This proxy intercepts calls to` decodeToLong(int id)` and scales the returned
values to microseconds.
This approach resolves the bug gracefully:
It fixes the issue on the optimized dictionary-passthrough path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
