[
https://issues.apache.org/jira/browse/PIG-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang_intel updated PIG-4606:
----------------------------------
Attachment: PIG-4606.patch
[~mohitsabharwal],[~kexianda],[~praveenr019],[~xuefuz]:
Changes in PIG-4606.patch is:
1.set date time zone according to "pig.datetime.default.tz" in the
conf/pig.properties in spark mode
Before following unit tests fail:
org.apache.pig.test.TestDefaultDateTimeZone.testDST
org.apache.pig.test.TestDefaultDateTimeZone.testLocalExecution
Let's give an example to explain the failure:
edit the "pig.datetime.default.tz" in conf/pig.properties:
pig.datetime.default.tz=America/New_York
TestDefaultDateTimeZone.pig
{code}
a = load './TestDefaultDateTimeZone.txt' as (test:datetime);
b = filter a by test > ToDate('2014-01-01T00:00:00.000');
c = foreach b generate ToString(test, 'Z') as tz;
store c into './TestDefaultDateTimeZone.out';
{code}
cat TestDefaultDateTimeZone.txt
{code}
1970-01-01T00:00:00.000
1970-01-01T00:00:00.000Z
1970-01-03T00:00:00.000
1970-01-03T00:00:00.000Z
1970-01-05T00:00:00.000Z
2014-02-01T00:00:00.000
2014-06-01T00:00:00.000
{code}
in mr, the result is:
{code}
-0500
-0400
{code}
in spark, the result is
{code}
+0800
+0800
{code}
The difference between mr and spark mode is because "pig.datetime.default.tz"
value does not work in spark mode and the DateTimeZone is the current user time
zone, in my env, it is "Asia/Shanghai"(UTC+8). in mr mode, it set
"pig.datetime.default.tz" in
[PigGenericMapBase.java|https://github.com/apache/pig/blob/trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapBase.java#L218].
PIG-4606.patch set the "pig.datetime.default.tz" in SparkLauncher.java.
> Enable "TestDefaultDateTimeZone" unit tests in spark mode
> ---------------------------------------------------------
>
> Key: PIG-4606
> URL: https://issues.apache.org/jira/browse/PIG-4606
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4606.patch
>
>
> In https://builds.apache.org/job/Pig-spark/216/#showFailuresLink, unit tests
> about TestDefaultDateTimeZone fails:
> org.apache.pig.test.TestDefaultDateTimeZone.testDST
> org.apache.pig.test.TestDefaultDateTimeZone.testLocalExecution
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
