mbutrovich commented on code in PR #12263:
URL: https://github.com/apache/arrow-rs/pull/12263#discussion_r4592452608


##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -1768,8 +1768,46 @@ fn write_leaf(
                 levels,
             },
         ),
-        ColumnWriter::Int96ColumnWriter(_typed) => {
-            unreachable!("Currently unreachable because data type not 
supported")
+        ColumnWriter::Int96ColumnWriter(typed) => {
+            let (values, nanos_per_unit) = match column.data_type() {
+                ArrowDataType::Timestamp(TimeUnit::Second, _) => (
+                    column.as_primitive::<TimestampSecondType>().values(),
+                    1_000_000_000,
+                ),
+                ArrowDataType::Timestamp(TimeUnit::Millisecond, _) => (
+                    column.as_primitive::<TimestampMillisecondType>().values(),
+                    1_000_000,
+                ),
+                ArrowDataType::Timestamp(TimeUnit::Microsecond, _) => (
+                    column.as_primitive::<TimestampMicrosecondType>().values(),
+                    1_000,
+                ),
+                ArrowDataType::Timestamp(TimeUnit::Nanosecond, _) => {
+                    
(column.as_primitive::<TimestampNanosecondType>().values(), 1)
+                }
+                d => return Err(ParquetError::General(format!("Cannot coerce 
{d} to INT96"))),
+            };
+            let values = indices
+                .iter()
+                .map(|&i| {
+                    Int96::from_epoch_nanos(values[i] as i128 * 
nanos_per_unit).ok_or_else(|| {
+                        ParquetError::General(format!(
+                            "{} value {} is out of range for INT96",
+                            column.data_type(),
+                            values[i]
+                        ))
+                    })
+                })
+                .collect::<Result<Vec<_>>>()?;

Review Comment:
   `values[i] as i128 * nanos_per_unit` makes every value an `i128` division, 
which compiles to a call to `__divti3` per value on both aarch64 and x86_64. 
Could we divide the `i64` value by the unit's count per day with 
`arrow_buffer::DivisorI64` instead? Build it once per batch, match on its 
variant before the loop, and move the variant into the closure, the way 
`divisor_op` in arrow-arith does 
([numeric.rs](https://github.com/apache/arrow-rs/blob/4f1ed42f7a1fb76d090ec1351cfdb6e904f59f0b/arrow-arith/src/numeric.rs#L371-L380)).
 The divisor is always positive, so `MIN / -1` can't happen, and flooring is 
one step: subtract one from the quotient and add the divisor to the remainder 
when the remainder is negative. When I tried this, each value became a high 
multiply and shifts (`smulh` on aarch64, a one-operand `imul` on x86_64), with 
no division and no zero or overflow checks. #11467 measured `DivisorI64` at 
2.4x to 4.9x faster than `i64` `/` and `%` on x86_64 for divisors that aren't p
 owers of two, its smallest gain on any target, and the code here starts from 
an `i128` division.
   
   `collect::<Result<Vec<_>>>()` also calls `GenericShunt::next` out of line 
for each value, so the divisor's fields are reloaded from memory every time. 
The `DivisorI64` docs warn about that 
([divisor.rs](https://github.com/apache/arrow-rs/blob/4f1ed42f7a1fb76d090ec1351cfdb6e904f59f0b/arrow-buffer/src/divisor.rs#L26-L31)).
 Could this be a plain loop that pushes into 
`Vec::with_capacity(indices.len())` and returns on the first error?



##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -1768,8 +1768,46 @@ fn write_leaf(
                 levels,
             },
         ),
-        ColumnWriter::Int96ColumnWriter(_typed) => {
-            unreachable!("Currently unreachable because data type not 
supported")
+        ColumnWriter::Int96ColumnWriter(typed) => {
+            let (values, nanos_per_unit) = match column.data_type() {
+                ArrowDataType::Timestamp(TimeUnit::Second, _) => (
+                    column.as_primitive::<TimestampSecondType>().values(),
+                    1_000_000_000,
+                ),
+                ArrowDataType::Timestamp(TimeUnit::Millisecond, _) => (
+                    column.as_primitive::<TimestampMillisecondType>().values(),
+                    1_000_000,
+                ),
+                ArrowDataType::Timestamp(TimeUnit::Microsecond, _) => (
+                    column.as_primitive::<TimestampMicrosecondType>().values(),
+                    1_000,
+                ),
+                ArrowDataType::Timestamp(TimeUnit::Nanosecond, _) => {
+                    
(column.as_primitive::<TimestampNanosecondType>().values(), 1)
+                }
+                d => return Err(ParquetError::General(format!("Cannot coerce 
{d} to INT96"))),

Review Comment:
   This writes `Timestamp(_, None)` and `Timestamp(_, Some(tz))` as the same 
INT96 values, and INT96 has no `isAdjustedToUTC` to tell them apart. Spark 
reads every INT96 column as `TimestampType`, an instant 
([ParquetSchemaConverter.scala](https://github.com/apache/spark/blob/5ab5ec5fc59d2d4d900f6f87c0e03e92273e2420/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L358-L363)).
 For INT64 it reads `isAdjustedToUTC = false`, which arrow-rs writes for 
`Timestamp(_, None)`, as `TimestampNTZType` 
([ParquetSchemaConverter.scala](https://github.com/apache/spark/blob/5ab5ec5fc59d2d4d900f6f87c0e03e92273e2420/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L328-L333)).
 So a timestamp without a time zone becomes an instant in Spark only when it's 
written as INT96. Spark itself never writes `TimestampNTZType` as INT96 
([ParquetSchemaConverter.scala](https://github.com/apache/spark/blob/5
 
ab5ec5fc59d2d4d900f6f87c0e03e92273e2420/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L742-L760)).
   
   The arrow-rs reader makes the opposite assumption. Without the embedded 
Arrow schema, it reads INT96 as `Timestamp(Nanosecond, None)` 
([primitive.rs](https://github.com/apache/arrow-rs/blob/4f1ed42f7a1fb76d090ec1351cfdb6e904f59f0b/parquet/src/arrow/schema/primitive.rs#L115)).
   
   Should INT96 accept only timestamps with a time zone, and return the "Cannot 
coerce" error for `Timestamp(_, None)`?



##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -1768,8 +1768,46 @@ fn write_leaf(
                 levels,
             },
         ),
-        ColumnWriter::Int96ColumnWriter(_typed) => {
-            unreachable!("Currently unreachable because data type not 
supported")
+        ColumnWriter::Int96ColumnWriter(typed) => {

Review Comment:
   With bloom filters enabled, `ArrowWriter` now writes a bloom filter for an 
INT96 column, but `ArrowSbbf` can't check it. I wrote `Timestamp(Microsecond, 
"UTC")` values to an INT96 column with `set_bloom_filter_enabled(true)`. 
`ArrowSbbf::try_new` with that Arrow type then returned a `Cannot check a bloom 
filter of type INT96 with ...` error. `validate` only accepts the types that 
`arrow_to_parquet_type` produces, and that function never produces INT96 
([bloom_filter.rs](https://github.com/apache/arrow-rs/blob/4f1ed42f7a1fb76d090ec1351cfdb6e904f59f0b/parquet/src/arrow/bloom_filter.rs#L215-L228)).
 The INT96 arm of `check` also still says `ArrowWriter does not write INT96 
columns` 
([bloom_filter.rs](https://github.com/apache/arrow-rs/blob/4f1ed42f7a1fb76d090ec1351cfdb6e904f59f0b/parquet/src/arrow/bloom_filter.rs#L178)).
   
   `ArrowSbbf` exists so that callers can check Arrow values converted the same 
way `ArrowWriter` converts them. Could the conversion move into a `visit_int96` 
that both `write_leaf` and `ArrowSbbf::check` call, the way `visit_i64` works, 
with `validate` accepting `Timestamp` for INT96 columns?



##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -6342,4 +6380,97 @@ mod tests {
         assert_eq!(fixed, (-1, 1), "incorrect Arrow FixedSizeBinaryArray 
stats");
         assert_eq!(binary, (-1, 1), "incorrect Arrow BinaryArray stats");
     }
+
+    #[test]
+    fn arrow_writer_timestamps_to_int96() {
+        let seconds =
+            TimestampSecondArray::from(vec![Some(-1), None, 
Some(1_700_000_000), None, None]);
+        let micros = TimestampMicrosecondArray::from(vec![
+            Some(0),
+            Some(-1),
+            None,
+            Some(-86_400_000_001),
+            Some(1_700_000_000_123_456),
+        ])
+        .with_timezone("UTC");
+        let nanos = TimestampNanosecondArray::from(vec![
+            Some(1),
+            Some(-1),
+            Some(1_700_000_000_123_456_789),
+            None,
+            Some(-86_400_000_000_001),
+        ]);
+        let millis = TimestampMillisecondArray::from(vec![Some(-1), None, 
Some(1_700_000_000_123)]);
+        let list = ListArray::new(
+            Arc::new(Field::new("element", millis.data_type().clone(), true)),
+            OffsetBuffer::from_lengths([2, 0, 1, 0, 0]),
+            Arc::new(millis),
+            None,
+        );
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("s", seconds.data_type().clone(), true),
+            Field::new("us", micros.data_type().clone(), true),
+            Field::new("ns", nanos.data_type().clone(), true),
+            Field::new("list", list.data_type().clone(), false),
+        ]));
+        let batch = RecordBatch::try_new(
+            schema.clone(),
+            vec![
+                Arc::new(seconds),
+                Arc::new(micros),
+                Arc::new(nanos),
+                Arc::new(list),
+            ],
+        )
+        .unwrap();
+        let parquet_schema = crate::schema::parser::parse_message_type(
+            "message schema {
+                OPTIONAL INT96 s;
+                OPTIONAL INT96 us;
+                OPTIONAL INT96 ns;
+                REQUIRED group list (LIST) {
+                    REPEATED group list {
+                        OPTIONAL INT96 element;
+                    }
+                }
+            }",
+        )
+        .unwrap();
+        let options = ArrowWriterOptions::new()
+            
.with_parquet_schema(SchemaDescriptor::new(Arc::new(parquet_schema)));
+        let mut buf = Vec::new();
+        let mut writer = ArrowWriter::try_new_with_options(&mut buf, schema, 
options).unwrap();
+        writer.write(&batch).unwrap();
+        writer.close().unwrap();
+
+        let builder = 
ParquetRecordBatchReaderBuilder::try_new(Bytes::from(buf)).unwrap();
+        let descr = builder.metadata().file_metadata().schema_descr_ptr();
+        assert!(
+            descr
+                .columns()
+                .iter()
+                .all(|c| c.physical_type() == crate::basic::Type::INT96)
+        );
+        let read = builder.build().unwrap().next().unwrap().unwrap();
+        assert_eq!(read, batch);
+    }
+
+    #[test]
+    fn arrow_writer_timestamp_out_of_int96_range() {

Review Comment:
   This test checks one value far past the range. Could we test at the 
boundaries instead? For `Timestamp(Second)`, `185331720383999` and 
`-185753453990400` are the largest and smallest values whose Julian day fits in 
`i32`. They should round trip, and one second past each should return the error.
   
   The conversion skips null slots on purpose, so an out-of-range value under a 
null doesn't fail the write. Could a test cover that too, with a null slot that 
holds `i64::MAX`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to