yadavay-amzn opened a new pull request, #57239: URL: https://github.com/apache/spark/pull/57239
### What changes were proposed in this pull request? This is the "option B" follow-up to [SPARK-57551](https://issues.apache.org/jira/browse/SPARK-57551), under the TIME data type epic (SPARK-57550). It re-enables **lazy dictionary decoding for TIME columns** in the vectorized Parquet reader by making the lazy path precision/unit-aware, rather than disabling it. - `ParquetDictionary.decodeToLong` now applies the TIME transform on the lazy path: `micros -> nanos` (when the column is stored as micros) followed by `DateTimeUtils.truncateTimeToPrecision`, producing values byte-for-byte identical to the eager `TimeVectorUpdater`. - `ParquetDictionary` carries the on-disk unit and the requested precision (set at construction in `VectorizedColumnReader` when the column is a `TimeType`). - `TimeTypeParquetOps.supportsLazyDictionaryDecoding` now returns `true`, so TIME columns use the lazy optimization again through the types framework's dispatch. The change is scoped to TIME: non-TIME dictionary reinterpretations and the other INT64 transforms (`TIMESTAMP_MILLIS` upcast, timestamp rebase) remain eager, as before. ### Why are the changes needed? SPARK-57551 added read-time precision truncation for TIME columns and, to guarantee truncation applies, disabled lazy dictionary decoding for `TIME(NANOS)` (option A) so dictionary-encoded TIME columns eager-decoded through the truncating updater. This PR restores the lazy dictionary-decoding optimization (option B) without losing that correctness, by teaching the lazy path the same unit conversion and precision truncation. ### Does this PR introduce _any_ user-facing change? No result change. It is a performance improvement: dictionary-encoded TIME columns can again be read via the lazy dictionary path. ### How was this patch tested? New `ParquetIOSuite` tests exercising the lazy path (dictionary-encoded + vectorized reader + lazy decoding enabled): `TIME(NANOS)` and `TIME(MICROS)` columns read at a lower precision produce truncated values identical to the eager path; a lazy-vs-eager parity test; a nullable multi-value case; and a full-precision (no-truncation) case. `TimeTypeParquetOpsSuite` updated for the re-enabled lazy support. checkstyle + scalastyle clean. ### Was this patch authored or co-authored using generative AI tooling? Authored with assistance by Claude Opus 4.8. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
