marmbrus edited a comment on issue #24195: [SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp URL: https://github.com/apache/spark/pull/24195#issuecomment-577906090 From the JIRA REST I get a string that is a timestamp encoded in PST. My local clock is set to UTC. I would like to store the data in Delta as a UTC timestamp. This function seems to do what I want. Is there a better way to do this in Spark? I do not want to hard code the offset and do interval subtraction. Regardless of the answer to the above, I think we are being hostile to our users with the current approach. I think we should back this out, and any other similar deprecation that are happening in Spark 3.0 - DO build better ways to work with timestamps safely in Spark. - DO update the documentation to point out the caveats of bad APIs. - DO build mechanisms to let SQL users know about bad patterns. - DO NOT annoy users by breaking code that used to work - DO NOT force them to blindly set flags without giving them some better path forward. - DO NOT both deprecate and flag fail scala code paths. The current error messages: ``` This function is deprecated and will be removed in future versions. ``` or ``` The to_utc_timestamp function has been disabled since Spark 3.0. Set spark.sql.legacy.utcTimestampFunc.enabled to true to enable this function. ``` These messages tell the user nothing, other than the fact that Spark developers would like to make it hard for them to upgrade their cluster. It doesn't even point them to the JIRA discussion (which IMO is still putting too much cognitive burden on them). Do you think any of our users are going to do anything other than blindly set the flag when they see that message (That is what I did)? I worry that this accomplishes nothing other than frustration.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
