adamhooper commented on issue #686: URL: https://github.com/apache/arrow-datafusion/issues/686#issuecomment-876420215
@alamb would anybody be horrified/appalled at the following: * `to_timestamp()` always returns `TIMESTAMP WITH TIMEZONE` -- i.e., UTC. * `my_parquet_file_with_isAdjustedToUTC_true` are always `TIMESTAMP WITH TIMEZONE` -- i.e., UTC. (Parquet doesn't store timezones anyway.) * `my_arrow_ipc_file_with_timezone_america_montreal` produces `TIMESTAMP WITH TIMEZONE` -- i.e., UTC. (DataFusion simply drops the metadata, without warning.) * `my_parquet_file_with_isAdjustedToUTC_false` produces an error for now, saying DataFusion doesn't support `TIMESTAMP WITHOUT TIME ZONE`. * `my_arrow_ipc_file_with_timezone_null` gives a deprecation warning and `TIMESTAMP WITH TIMEZONE`; then, some version in the future, DataFusion re-implements it as `TIMESTAMP WITHOUT TIMEZONE`. To me, dropping timezone info is an easy (if rude) decision. DataFusion implements a subset of Postgres, and Postgres has no timezone metadata. There's an argument to be made for passing the timezone through unmodified; but that would be hard/convoluted, probably needing new syntax; and does anybody actually want that? IPC with datetime+timezone tuples is nonstandard and heavyweight; Spark doesn't even have a type for that. Forcing UTC won't stop anybody from accomplishing anything. So the only hairy decision is what to do with Arrow `timezone=null`. Up until today, DataFusion users have been using it as `TIMESTAMP WITH TIMEZONE`, right? If that's the case, then there's great news: nobody who's using DataFusion is using `TIMESTAMP WITHOUT TIMEZONE` :). Sounds like that type would be a new feature ... and I expect it would be a low priority, since nobody asked for it. (Again, Spark has no such type.) (Primer on Postgres, for the uninitiated: `TIMESTAMP WITH TIME ZONE` means UTC; it doesn't store a timezone.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
