MaxGekk commented on issue #24195: [SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp URL: https://github.com/apache/spark/pull/24195#issuecomment-578015476 > From the JIRA REST I get a string that is a timestamp encoded in PST. I guess if timestamp strings in PST, the strings contain timezone info. Looking at https://developer.atlassian.com/cloud/jira/platform/jira-expressions/#date-and-time, they are. - If you parse the strings via JSON datasource, you should set JSON option `timestampFormat` with time zone sub-pattern correctly. - If you want to parse it just as string, use `def to_timestamp(s: Column, fmt: String)`. Spark's TimestampFormatter takes into account time zone info https://github.com/apache/spark/blob/c2734ab1fcc523507174c4f9e7368c1e7a2368e8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala#L59-L60 and shifts local timestamp respectively while converting to UTC timestamps: https://github.com/apache/spark/blob/c2734ab1fcc523507174c4f9e7368c1e7a2368e8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala#L61-L62 If input strings don't contain time zone info, you can either: - append time zone id string manually, or - temporary set SQL config `spark.sql.session.timeZone` while parsing timestamp strings. For me, still not clear why did you decide to even look at the functions.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
