ranflarion opened a new issue, #12262: URL: https://github.com/apache/arrow-rs/issues/12262
### Is your feature request related to a problem or challenge? `ArrowWriter` cannot write a Parquet INT96 column. A Parquet schema that declares a timestamp column as INT96, passed with `ArrowWriterOptions::with_parquet_schema`, is accepted, and the first `write` panics in `write_leaf`: ``` internal error: entered unreachable code: Currently unreachable because data type not supported ``` Spark writes timestamps as INT96 by default (`spark.sql.parquet.outputTimestampType=INT96`), so a writer that produces Spark-compatible files from Arrow data cannot match it and has to store INT64 timestamps instead. Reading INT96 already works, into any time unit (#7220), and so do its statistics (#11348). ### Describe the solution you'd like When the supplied Parquet schema declares INT96 for a leaf whose Arrow type is `Timestamp` (any unit), convert each value to the Julian day and the nanoseconds of the day, the layout `Int96::to_nanos` reads. Nothing changes for callers that do not ask for INT96: the default schema conversion keeps mapping timestamps to INT64. A value whose Julian day does not fit in 32 bits, possible with second and millisecond units, returns an error. ### Describe alternatives you've considered A writer property that makes schema conversion pick INT96 for every timestamp, like Arrow C++'s `enable_deprecated_int96_timestamps` (pyarrow's `use_deprecated_int96_timestamps`). It could build on the conversion above, but it adds API, while a supplied Parquet schema already expresses the choice per column. ### Additional context INT96 is deprecated in the Parquet format but is still Spark's default timestamp type. ### Willingness to contribute I plan to contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
