ranflarion opened a new issue, #12262:
URL: https://github.com/apache/arrow-rs/issues/12262

   ### Is your feature request related to a problem or challenge?
   
   `ArrowWriter` cannot write a Parquet INT96 column. A Parquet schema that 
declares a timestamp column as INT96, passed with 
`ArrowWriterOptions::with_parquet_schema`, is accepted, and the first `write` 
panics in `write_leaf`:
   
   ```
   internal error: entered unreachable code: Currently unreachable because data 
type not supported
   ```
   
   Spark writes timestamps as INT96 by default 
(`spark.sql.parquet.outputTimestampType=INT96`), so a writer that produces 
Spark-compatible files from Arrow data cannot match it and has to store INT64 
timestamps instead. Reading INT96 already works, into any time unit (#7220), 
and so do its statistics (#11348).
   
   ### Describe the solution you'd like
   
   When the supplied Parquet schema declares INT96 for a leaf whose Arrow type 
is `Timestamp` (any unit), convert each value to the Julian day and the 
nanoseconds of the day, the layout `Int96::to_nanos` reads. Nothing changes for 
callers that do not ask for INT96: the default schema conversion keeps mapping 
timestamps to INT64. A value whose Julian day does not fit in 32 bits, possible 
with second and millisecond units, returns an error.
   
   ### Describe alternatives you've considered
   
   A writer property that makes schema conversion pick INT96 for every 
timestamp, like Arrow C++'s `enable_deprecated_int96_timestamps` (pyarrow's 
`use_deprecated_int96_timestamps`). It could build on the conversion above, but 
it adds API, while a supplied Parquet schema already expresses the choice per 
column.
   
   ### Additional context
   
   INT96 is deprecated in the Parquet format but is still Spark's default 
timestamp type.
   
   ### Willingness to contribute
   
   I plan to contribute to this feature independently
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to