GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/16308
[SPARK-18350][SQL][WIP] Support session local timezone.
## What changes were proposed in this pull request?
As of Spark 2.1, Spark SQL assumes the machine timezone for datetime
manipulation, which is bad if users are not in the same timezones as the
machines, or if different users have different timezones.
We should introduce a session local timezone setting that is used for
execution.
An explicit non-goal is locale handling.
### Design of the fix
I introduced an analyzer to pass session local timezone to timezone-aware
expressions and modified DateTimeUtils to take the timezone argument.
## How was this patch tested?
Existing tests and added tests for timezone aware expressions.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ueshin/apache-spark issues/SPARK-18350
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16308.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16308
----
commit c48a70da0f06ebde001091fe765f787382c7409e
Author: Takuya UESHIN <[email protected]>
Date: 2016-12-06T06:41:39Z
Prepare for session local timezone support.
commit 1d21fec497147faa1eabcfc31e524abc102c67c5
Author: Takuya UESHIN <[email protected]>
Date: 2016-12-06T13:43:26Z
Make Cast TimeZoneAwareExpression.
commit 0763c8f67e5e13045d6958b45ffe3d4e0efb0a63
Author: Takuya UESHIN <[email protected]>
Date: 2016-12-06T14:37:47Z
Fix DateTimeUtilsSuite to follow changes.
commit 449d93d416d225a323a18dbffc965dca11c3b55d
Author: Takuya UESHIN <[email protected]>
Date: 2016-12-08T15:20:05Z
Make some datetime expressions TimeZoneAwareExpression.
commit b59d902d80a2d33d7778903b45c334333bd00833
Author: Takuya UESHIN <[email protected]>
Date: 2016-12-08T17:48:20Z
Fix compiler error in sql/core.
commit 3ddfae41af5a3046ba966088cdc552d65574d02b
Author: Takuya UESHIN <[email protected]>
Date: 2016-12-09T07:38:17Z
Add constructors without zoneId to TimeZoneAwareExpressions for
FunctionRepository.
commit f58f00df14a60afccf84c10c94c96d64bf6c4724
Author: Takuya UESHIN <[email protected]>
Date: 2016-12-13T04:01:01Z
Add DateTimeUtils.threadLocalLocalTimeZone to partition-reltated Cast.
commit 8f2040b1be461d71f919d53462e70af86d6e6ada
Author: Takuya UESHIN <[email protected]>
Date: 2016-12-13T07:32:43Z
Fix timezone for Hive timestamp string.
commit 63c103c2e94f45e0a9af74c48c1af57311ef1586
Author: Takuya UESHIN <[email protected]>
Date: 2016-12-13T09:53:30Z
Use defaultTimeZone instead of threadLocalLocalTimeZone.
commit 7066850286fbd5cd845b38d8b3498531420c8a89
Author: Takuya UESHIN <[email protected]>
Date: 2016-12-13T10:08:10Z
Add TimeZone to DateFormats.
commit 1aaca29122d9996da4a2d60a3e9aae01b89e685e
Author: Takuya UESHIN <[email protected]>
Date: 2016-12-14T05:36:33Z
Make `CurrentBatchTimestamp` `TimeZoneAwareExpression`.
commit e5bb246f8b4cbda13d248a0b916c55817d78a926
Author: Takuya UESHIN <[email protected]>
Date: 2016-12-14T10:00:27Z
Add tests for date functions with session local timezone.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
