MaxGekk commented on issue #26134: [SPARK-29486][SQL] CalendarInterval should have 3 fields: months, days and microseconds URL: https://github.com/apache/spark/pull/26134#issuecomment-543246063 > We should convert zoneId to zoneOffset at the beginning of a query, according to the current time of query submission. I cannot agree with you. For example, time zone rules were changed twice in Russia during last 10 years (https://en.wikipedia.org/wiki/Time_in_Russia#Daylight_saving_time). If I have timestamps < 2011, between 2011 and 2014, > 2014 year, processing timestamp before 2014 years give me wrong results. Actually, you are going to force me to split my query to many separate queries per each daylight saving rule otherwise I will get incorrect results where local timestamps are involved. Even if daylight saving rules have been not changed in some region, how should users process their datasets with timestamps belong to winter and summer periods? Let's say an user applies the `hour()` function some winter timestamps but current date belongs to summer period. > This doesn't follow the SQL standard. We should define the timestamp type as timestamp + zoneOffset To be more clear: In Spark, timestamp is instant + session zone id In SQL, timestamp with time zone is local date-time + zone offset per each value. The key thing is per each value. The SQL standard just moved out the scope resolving ZoneId -> ZoneOffsets.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
