sunchao opened a new pull request, #7715: URL: https://github.com/apache/datafusion-comet/pull/7715
## Which issue does this PR close? Partially addresses #5553. This follows #5740 for unfiltered scans. #5739 tracks when to enable checked conversion in scans with data filters. ## Rationale for this change When a Parquet footer's `ARROW:schema` declares a dictionary of millisecond timestamps, arrow-rs reconstructs a `DictionaryArray` on read. This is independent of the Parquet page encoding exercised by the existing overflow tests. [Arrow schema restoration](https://github.com/apache/arrow-rs/blob/59.3.2/parquet/src/arrow/schema/primitive.rs#L95-L105), [primitive reader](https://github.com/apache/arrow-rs/blob/59.3.2/parquet/src/arrow/array_reader/primitive_array.rs#L239-L249) For an unfiltered `SELECT s.ts`, the dictionary wrapper bypasses Comet's checked conversion. A referenced value of `9223372036854776` milliseconds becomes `NULL` in Arrow's safe cast when multiplied by 1,000. Spark raises `ArithmeticException("long overflow")`. At the top level, the same dictionary goes through a SQL cast that assumes the dictionary values already have microsecond precision and can fail even for an in-range value. [Arrow timestamp cast](https://github.com/apache/arrow-rs/blob/59.3.2/arrow-cast/src/cast/mod.rs#L1828-L1848), [Spark Parquet reader](https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java#L800-L833), [Comet SQL cast](https://github.com/apache/datafusion-comet/blob/4a0a01106eb3e7e8ea9bf4090e84630fb71124eb/native/spark-expr/src/utils.rs#L150-L158) ## What changes are included in this PR? Route a top-level dictionary of millisecond timestamps requested as microseconds through the Parquet converter. Look through dictionary types when deciding which nested fields need ancestor validity. When overflow checking is enabled, gather the timestamp values in row order using the dictionary keys, then apply the existing checked multiplication to the visible rows. This excludes unused dictionary entries and values hidden by null parents or slices. Scans with data filters use the safe cast. ## How are these changes tested? A native regression writes Parquet files with the dictionary type in the Arrow schema and scans both a top-level timestamp and a struct child. It checks repeated keys, nulls and the conversion of 7 ms to 7,000 µs, then includes an overflowing value and checks for `ParquetTimestampOverflow`. The existing visibility test now also exercises dictionary timestamps in structs, lists and maps, including slices, both overflow signs, and timestamp types with and without a timezone. Ran `rustfmt --check` on both changed files and `git diff --check` locally. Test execution is delegated to hosted CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
