nevi-me commented on a change in pull request #9612:
URL: https://github.com/apache/arrow/pull/9612#discussion_r594478176
##########
File path: rust/parquet/src/arrow/schema.rs
##########
@@ -364,32 +385,51 @@ fn arrow_to_parquet_type(field: &Field) -> Result<Type> {
DataType::Float64 => Type::primitive_type_builder(name,
PhysicalType::DOUBLE)
.with_repetition(repetition)
.build(),
- DataType::Timestamp(time_unit, _) => {
+ DataType::Timestamp(time_unit, zone) => {
Type::primitive_type_builder(name, PhysicalType::INT64)
- .with_converted_type(match time_unit {
- TimeUnit::Second => ConvertedType::TIMESTAMP_MILLIS,
- TimeUnit::Millisecond => ConvertedType::TIMESTAMP_MILLIS,
- TimeUnit::Microsecond => ConvertedType::TIMESTAMP_MICROS,
- TimeUnit::Nanosecond => ConvertedType::TIMESTAMP_MICROS,
- })
+ .with_logical_type(Some(LogicalType::TIMESTAMP(TimestampType {
+ is_adjusted_to_u_t_c: matches!(zone, Some(z) if z.as_str()
== "UTC"),
Review comment:
I've read the thread, and my interpretation is this:
- What I was initially doing:
```
no timezone: false
"UTC": true
other timezone: false
```
- What's done in C++
```
no timezone: false
"UTC": true
other timezone: true
```
> and normalize the timestamp value to UTC when converting to Parquet
Arrow timestamps are always in UTC, such that any non-UTC timezone is for
display purposes only (e.g. if we want to print formatted timestamps).
So, we shouldn't need to normalise timezones as they'll always be adjusted
to UTC.
My initial approach was to set `is_adjusted_to_u_t_c = true` whenever
there's a timezone, but I second-guessed myself while working on this code. I
had looked at the C++ implementation, but somehow interpreted the `true` value
to only be set if `timezone = UTC`.
@sunchao are you fine with setting `is_adjusted_to_u_t_c = true` whenever
there's a timezone?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
