sunchao commented on a change in pull request #9612:
URL: https://github.com/apache/arrow/pull/9612#discussion_r594048206
##########
File path: rust/parquet/src/arrow/schema.rs
##########
@@ -364,32 +385,51 @@ fn arrow_to_parquet_type(field: &Field) -> Result<Type> {
DataType::Float64 => Type::primitive_type_builder(name,
PhysicalType::DOUBLE)
.with_repetition(repetition)
.build(),
- DataType::Timestamp(time_unit, _) => {
+ DataType::Timestamp(time_unit, zone) => {
Type::primitive_type_builder(name, PhysicalType::INT64)
- .with_converted_type(match time_unit {
- TimeUnit::Second => ConvertedType::TIMESTAMP_MILLIS,
- TimeUnit::Millisecond => ConvertedType::TIMESTAMP_MILLIS,
- TimeUnit::Microsecond => ConvertedType::TIMESTAMP_MICROS,
- TimeUnit::Nanosecond => ConvertedType::TIMESTAMP_MICROS,
- })
+ .with_logical_type(Some(LogicalType::TIMESTAMP(TimestampType {
+ is_adjusted_to_u_t_c: matches!(zone, Some(z) if z.as_str()
== "UTC"),
Review comment:
I think we can perhaps follow Arrow C++ and always set it to true
whenever the timezone info is set, and normalize the timestamp value to UTC
when converting to Parquet. Please see a previous discussion
[here](https://github.com/apache/arrow/pull/4421#issuecomment-498413899) and
the related C++ code
[here](https://github.com/apache/arrow/blob/master/cpp/src/parquet/arrow/schema.cc#L138).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
