liukun4515 commented on PR #5605: URL: https://github.com/apache/arrow-rs/pull/5605#issuecomment-2058139684
> > We need to add the timezone config in the CastOptions which can't be configured by other system like datafusion. > > The conversion should take the date of the timestamp in the timezone configured on the timestamp data type, we should not need to configure this on `CastOptions`, we have very carefully avoided introducing the notion of a "local" timezone. > > So for example if we had `2019-10-12T07:20:50.52 +10:00`, the date returned would be `2019-10-12` even though with reference to the UTC epoch (which is what the timestamp integer encodes) the date would be `2019-10-11`. Hi @tustvold From the definition of the timestamp in the arrow schema: https://github.com/apache/arrow/blob/main/format/Schema.fbs#L283 It's will cause inconvenient usage, let me give a example: When I write a parquet file with timestamp column using the spark engine, the timestamp column is encoded by https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#instant-semantics-timestamps-normalized-to-utc and is normalized to UTC. But when I use arrow-parquet to read the timestamp column, i will just get the column with timestamp(unit, "UTC") data type by this code https://github.com/apache/arrow-rs/blob/0d031cc8aa81296cb1bdfedea7a7cb4ec6aa54ea/parquet/src/arrow/schema/primitive.rs#L86 and https://github.com/apache/arrow-rs/blob/0d031cc8aa81296cb1bdfedea7a7cb4ec6aa54ea/parquet/src/arrow/schema/primitive.rs#L232 It's even possible to use `int96` to store the timestamp in the parquet file, we will get the column with timestamp(nanosecond, None) data type by https://github.com/apache/arrow-rs/blob/0d031cc8aa81296cb1bdfedea7a7cb4ec6aa54ea/parquet/src/arrow/schema/primitive.rs#L99 But i need this is used in the `UTC-7` timezone, how did you resolve this issue? @alamb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
