gaogaotiantian commented on PR #53161: URL: https://github.com/apache/spark/pull/53161#issuecomment-3577964560
Okay now I rethink about this and I think our `TimestampType` is completely wrong. I don't believe we can patch it without breaking backward compatibility. The key point of `TimestampType` is that it's timezone aware - it's impossible to make it correct, if we support naive timestamp (unless we are able to read session config inside the conversion function). It's not possible to determine which timezone to use, when we get a naive datetime and try to convert it to a timezone aware data type. The only way I think we can go is to enforce timezone for `datetime.datetime` objects if users want to convert it to a `TimestampType`. Raise an error if there is none. If we can get the current session timezone when we try to convert to `TimestampType`, that would work too, but I don't think that's feasible. It also breaks OOP principles. I don't want to fix this so that it's correct in some cases but become wrong in others. We should make a decision about whether we want to fix it (with the risk of breaking users code). If not, that's okay - we can have a timestamp system that sometimes work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
