sunchao opened a new pull request, #7715:
URL: https://github.com/apache/datafusion-comet/pull/7715

   ## Which issue does this PR close?
   
   Partially addresses #5553. This follows #5740 for unfiltered scans. #5739 
tracks when to enable checked conversion in scans with data filters.
   
   ## Rationale for this change
   
   When a Parquet footer's `ARROW:schema` declares a dictionary of millisecond 
timestamps, arrow-rs reconstructs a `DictionaryArray` on read. This is 
independent of the Parquet page encoding exercised by the existing overflow 
tests. [Arrow schema 
restoration](https://github.com/apache/arrow-rs/blob/59.3.2/parquet/src/arrow/schema/primitive.rs#L95-L105),
 [primitive 
reader](https://github.com/apache/arrow-rs/blob/59.3.2/parquet/src/arrow/array_reader/primitive_array.rs#L239-L249)
   
   For an unfiltered `SELECT s.ts`, the dictionary wrapper bypasses Comet's 
checked conversion. A referenced value of `9223372036854776` milliseconds 
becomes `NULL` in Arrow's safe cast when multiplied by 1,000. Spark raises 
`ArithmeticException("long overflow")`. At the top level, the same dictionary 
goes through a SQL cast that assumes the dictionary values already have 
microsecond precision and can fail even for an in-range value. [Arrow timestamp 
cast](https://github.com/apache/arrow-rs/blob/59.3.2/arrow-cast/src/cast/mod.rs#L1828-L1848),
 [Spark Parquet 
reader](https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java#L800-L833),
 [Comet SQL 
cast](https://github.com/apache/datafusion-comet/blob/4a0a01106eb3e7e8ea9bf4090e84630fb71124eb/native/spark-expr/src/utils.rs#L150-L158)
   
   ## What changes are included in this PR?
   
   Route a top-level dictionary of millisecond timestamps requested as 
microseconds through the Parquet converter. Look through dictionary types when 
deciding which nested fields need ancestor validity. When overflow checking is 
enabled, gather the timestamp values in row order using the dictionary keys, 
then apply the existing checked multiplication to the visible rows. This 
excludes unused dictionary entries and values hidden by null parents or slices. 
Scans with data filters use the safe cast.
   
   ## How are these changes tested?
   
   A native regression writes Parquet files with the dictionary type in the 
Arrow schema and scans both a top-level timestamp and a struct child. It checks 
repeated keys, nulls and the conversion of 7 ms to 7,000 µs, then includes an 
overflowing value and checks for `ParquetTimestampOverflow`. The existing 
visibility test now also exercises dictionary timestamps in structs, lists and 
maps, including slices, both overflow signs, and timestamp types with and 
without a timezone.
   
   Ran `rustfmt --check` on both changed files and `git diff --check` locally. 
Test execution is delegated to hosted CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to