[jira] [Commented] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871213#comment-16871213 ] Jiatao Tao commented on SPARK-27546: Hello [~dongjoon] Anyone ? > Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone > - > > Key: SPARK-27546 > URL: https://issues.apache.org/jira/browse/SPARK-27546 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Jiatao Tao >Priority: Minor > Attachments: image-2019-04-23-08-10-00-475.png, > image-2019-04-23-08-10-50-247.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28035) Test JoinSuite."equi-join is hash-join" is incompatible with its title.
[ https://issues.apache.org/jira/browse/SPARK-28035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862928#comment-16862928 ] Jiatao Tao commented on SPARK-28035: this test "multiple-key equi-join is hash-join" also the same situation. > Test JoinSuite."equi-join is hash-join" is incompatible with its title. > --- > > Key: SPARK-28035 > URL: https://issues.apache.org/jira/browse/SPARK-28035 > Project: Spark > Issue Type: Improvement > Components: Tests >Affects Versions: 2.4.3 >Reporter: Jiatao Tao >Priority: Trivial > Attachments: image-2019-06-13-10-32-06-759.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28035) Test JoinSuite."equi-join is hash-join" is incompatible with its title.
[ https://issues.apache.org/jira/browse/SPARK-28035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862927#comment-16862927 ] Jiatao Tao commented on SPARK-28035: The title means hash-join, but when I debug, found it is a merge join. And cannot get the idea that this test do, why only asserting the size? If this is a problem, I would like to modify this test, hope to hear your opinion, thanks. !image-2019-06-13-10-32-06-759.png! > Test JoinSuite."equi-join is hash-join" is incompatible with its title. > --- > > Key: SPARK-28035 > URL: https://issues.apache.org/jira/browse/SPARK-28035 > Project: Spark > Issue Type: Improvement > Components: Tests >Affects Versions: 2.4.3 >Reporter: Jiatao Tao >Priority: Trivial > Attachments: image-2019-06-13-10-32-06-759.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28035) Test JoinSuite."equi-join is hash-join" is incompatible with its title.
[ https://issues.apache.org/jira/browse/SPARK-28035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiatao Tao updated SPARK-28035: --- Attachment: image-2019-06-13-10-32-06-759.png > Test JoinSuite."equi-join is hash-join" is incompatible with its title. > --- > > Key: SPARK-28035 > URL: https://issues.apache.org/jira/browse/SPARK-28035 > Project: Spark > Issue Type: Improvement > Components: Tests >Affects Versions: 2.4.3 >Reporter: Jiatao Tao >Priority: Trivial > Attachments: image-2019-06-13-10-32-06-759.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-28035) Test JoinSuite."equi-join is hash-join" is incompatible with its title.
Jiatao Tao created SPARK-28035: -- Summary: Test JoinSuite."equi-join is hash-join" is incompatible with its title. Key: SPARK-28035 URL: https://issues.apache.org/jira/browse/SPARK-28035 Project: Spark Issue Type: Improvement Components: Tests Affects Versions: 2.4.3 Reporter: Jiatao Tao -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Reopened] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiatao Tao reopened SPARK-27546: Hi, I update the comment, could someone take a look? > Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone > - > > Key: SPARK-27546 > URL: https://issues.apache.org/jira/browse/SPARK-27546 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Jiatao Tao >Priority: Minor > Attachments: image-2019-04-23-08-10-00-475.png, > image-2019-04-23-08-10-50-247.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860491#comment-16860491 ] Jiatao Tao edited comment on SPARK-27546 at 6/11/19 2:46 AM: - Hi [~dongjoon] After reading the code, Although I have set "spark.sql.session.timeZone" to UTC, "DateTimeUtils#defaultTimeZone" will still use "TimeZone.getDefault()". I think the root cause is that When I cast to "TimestampType", it uses UTC(spark.sql.session.timeZone), but then convert to DateType, it uses GMT+8(TimeZone.getDefault). Remember, what I got, is "ts", it should not change with the time zone. was (Author: aron.tao): Hi [~dongjoon] After reading the code, Although I have set "spark.sql.session.timeZone" to UTC, "DateTimeUtils#defaultTimeZone" will still use "TimeZone.getDefault()". > Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone > - > > Key: SPARK-27546 > URL: https://issues.apache.org/jira/browse/SPARK-27546 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Jiatao Tao >Priority: Minor > Attachments: image-2019-04-23-08-10-00-475.png, > image-2019-04-23-08-10-50-247.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860491#comment-16860491 ] Jiatao Tao commented on SPARK-27546: After reading the code, Although I have set "spark.sql.session.timeZone" to UTC, "DateTimeUtils#defaultTimeZone" will still use "TimeZone.getDefault()". > Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone > - > > Key: SPARK-27546 > URL: https://issues.apache.org/jira/browse/SPARK-27546 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Jiatao Tao >Priority: Minor > Attachments: image-2019-04-23-08-10-00-475.png, > image-2019-04-23-08-10-50-247.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860491#comment-16860491 ] Jiatao Tao edited comment on SPARK-27546 at 6/11/19 2:42 AM: - Hi [~dongjoon] After reading the code, Although I have set "spark.sql.session.timeZone" to UTC, "DateTimeUtils#defaultTimeZone" will still use "TimeZone.getDefault()". was (Author: aron.tao): After reading the code, Although I have set "spark.sql.session.timeZone" to UTC, "DateTimeUtils#defaultTimeZone" will still use "TimeZone.getDefault()". > Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone > - > > Key: SPARK-27546 > URL: https://issues.apache.org/jira/browse/SPARK-27546 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Jiatao Tao >Priority: Minor > Attachments: image-2019-04-23-08-10-00-475.png, > image-2019-04-23-08-10-50-247.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
[
https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860489#comment-16860489
]
Jiatao Tao edited comment on SPARK-27546 at 6/11/19 2:40 AM:
-
Hi [~dongjoon]
What I get is "ts", not the "Mon Dec 31 16:00:00 PST 2012".
```
jshell> new Date(135699840L).getTime
$1 ==> 135699840L
jshell> TimeZone.setDefault(TimeZone.getTimeZone("UTC"))
jshell> new Date(135699840L).getTime
$3 ==> 135699840L
```
Whatere the timezone I set, should not influence the ts I get.
was (Author: aron.tao):
Hi [~dongjoon]
What I get is "ts", not the "Mon Dec 31 16:00:00 PST 2012".
jshell> new Date(135699840L).getTime
$1 ==> 135699840L
jshell> TimeZone.setDefault(TimeZone.getTimeZone("UTC"))
jshell> new Date(135699840L).getTime
$3 ==> 135699840L
Whatere the timezone I set, should not influence the ts I get.
> Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
> -
>
> Key: SPARK-27546
> URL: https://issues.apache.org/jira/browse/SPARK-27546
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Jiatao Tao
>Priority: Minor
> Attachments: image-2019-04-23-08-10-00-475.png,
> image-2019-04-23-08-10-50-247.png
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
[
https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860489#comment-16860489
]
Jiatao Tao commented on SPARK-27546:
Hi [~dongjoon]
What I get is "ts", not the "Mon Dec 31 16:00:00 PST 2012".
jshell> new Date(135699840L).getTime
$1 ==> 135699840L
jshell> TimeZone.setDefault(TimeZone.getTimeZone("UTC"))
jshell> new Date(135699840L).getTime
$3 ==> 135699840L
Whatere the timezone I set, should not influence the ts I get.
> Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
> -
>
> Key: SPARK-27546
> URL: https://issues.apache.org/jira/browse/SPARK-27546
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Jiatao Tao
>Priority: Minor
> Attachments: image-2019-04-23-08-10-00-475.png,
> image-2019-04-23-08-10-50-247.png
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823808#comment-16823808 ] Jiatao Tao edited comment on SPARK-27546 at 5/13/19 7:18 AM: - So in my opinion, we should change DateTimeUtils#defaultTimeZone to the timezone that "spark.sql.session.timeZone" configured. And I would like to take this JIRA and give a fix if community think this is indeed a problem. was (Author: aron.tao): So in my opinion, we should change DateTimeUtils#defaultTimeZone to the timezone that "spark.sql.session.timeZone" configured. And I would like to take this JIRA if community think this is indeed a problem. > Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone > - > > Key: SPARK-27546 > URL: https://issues.apache.org/jira/browse/SPARK-27546 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.4.1 >Reporter: Jiatao Tao >Priority: Minor > Attachments: image-2019-04-23-08-10-00-475.png, > image-2019-04-23-08-10-50-247.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
[
https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823801#comment-16823801
]
Jiatao Tao edited comment on SPARK-27546 at 4/23/19 8:17 AM:
-
see an example like this:
{code:java}
// config spark session with "spark.sql.session.timeZone", "UTC"
val df = mockDF
// TimeZone.setDefault(TimeZone.getTimeZone("UTC"))
val date = new Date(135699840L) // 2013-01-01 (UTC)
val ts =
df.select(lit(date).cast(TimestampType).cast(DateType)).head().getDate(0).getTime
{code}
The ts I got is 135696960 (2012-12-31 UTC), not the same with the origin
date 2013-01-01. So I think this is a problem.
And when I set "TimeZone.setDefault(TimeZone.getTimeZone("UTC"))" first, the
result is same with 135699840.
!image-2019-04-23-08-10-00-475.png!
!image-2019-04-23-08-10-50-247.png!
was (Author: aron.tao):
see an example like this:
{code:java}
// config spark session with "spark.sql.session.timeZone", "UTC"
val df = mockDF
// TimeZone.setDefault(TimeZone.getTimeZone("UTC"))
val date = new Date(135699840L) // 2013-01-01 (UTC)
val ts =
df.select(lit(date).cast(TimestampType).cast(DateType)).head().getDate(0).getTime
{code}
The ts I got is: 135696960 (12/31/2012 UTC)
When I set "TimeZone.setDefault(TimeZone.getTimeZone("UTC"))", the ts is same
with 135699840.
!image-2019-04-23-08-10-00-475.png!
!image-2019-04-23-08-10-50-247.png!
> Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
> -
>
> Key: SPARK-27546
> URL: https://issues.apache.org/jira/browse/SPARK-27546
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.4.1
>Reporter: Jiatao Tao
>Priority: Minor
> Attachments: image-2019-04-23-08-10-00-475.png,
> image-2019-04-23-08-10-50-247.png
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823808#comment-16823808 ] Jiatao Tao commented on SPARK-27546: So in my opinion, we should change DateTimeUtils#defaultTimeZone to the timezone that "spark.sql.session.timeZone" configured. And I would like to take this JIRA if community think this is indeed a problem. > Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone > - > > Key: SPARK-27546 > URL: https://issues.apache.org/jira/browse/SPARK-27546 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.4.1 >Reporter: Jiatao Tao >Priority: Minor > Attachments: image-2019-04-23-08-10-00-475.png, > image-2019-04-23-08-10-50-247.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
[
https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823801#comment-16823801
]
Jiatao Tao edited comment on SPARK-27546 at 4/23/19 8:10 AM:
-
see an example like this:
{code:java}
// config spark session with "spark.sql.session.timeZone", "UTC"
val df = mockDF
// TimeZone.setDefault(TimeZone.getTimeZone("UTC"))
val date = new Date(135699840L) // 2013-01-01 (UTC)
val ts =
df.select(lit(date).cast(TimestampType).cast(DateType)).head().getDate(0).getTime
{code}
The ts I got is: 135696960 (12/31/2012 UTC)
When I set "TimeZone.setDefault(TimeZone.getTimeZone("UTC"))", the ts is same
with 135699840.
!image-2019-04-23-08-10-00-475.png!
!image-2019-04-23-08-10-50-247.png!
was (Author: aron.tao):
see an example like this:
{code:java}
// config spark session with "spark.sql.session.timeZone", "UTC"
val df = mockDF
// TimeZone.setDefault(TimeZone.getTimeZone("UTC"))
val date = new Date(135699840L) // 2013-01-01 (UTC)
val ts =
df.select(lit(date).cast(TimestampType).cast(DateType)).head().getDate(0).getTime
{code}
The ts I got is: 135696960 (12/31/2012 UTC)
When I set "TimeZone.setDefault(TimeZone.getTimeZone("UTC"))", the ts is same
with 135699840.
> Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
> -
>
> Key: SPARK-27546
> URL: https://issues.apache.org/jira/browse/SPARK-27546
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.4.1
>Reporter: Jiatao Tao
>Priority: Minor
> Attachments: image-2019-04-23-08-10-00-475.png,
> image-2019-04-23-08-10-50-247.png
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
[
https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823801#comment-16823801
]
Jiatao Tao commented on SPARK-27546:
see an example like this:
{code:java}
// config spark session with "spark.sql.session.timeZone", "UTC"
val df = mockDF
// TimeZone.setDefault(TimeZone.getTimeZone("UTC"))
val date = new Date(135699840L) // 2013-01-01 (UTC)
val ts =
df.select(lit(date).cast(TimestampType).cast(DateType)).head().getDate(0).getTime
{code}
The ts I got is: 135696960 (12/31/2012 UTC)
When I set "TimeZone.setDefault(TimeZone.getTimeZone("UTC"))", the ts is same
with 135699840.
> Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
> -
>
> Key: SPARK-27546
> URL: https://issues.apache.org/jira/browse/SPARK-27546
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.4.1
>Reporter: Jiatao Tao
>Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone
Jiatao Tao created SPARK-27546: -- Summary: Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone Key: SPARK-27546 URL: https://issues.apache.org/jira/browse/SPARK-27546 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 2.4.1 Reporter: Jiatao Tao -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12154) Upgrade to Jersey 2
[ https://issues.apache.org/jira/browse/SPARK-12154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730601#comment-16730601 ] Jiatao Tao commented on SPARK-12154: Hadoop use 1.x, it's a nightmare. > Upgrade to Jersey 2 > --- > > Key: SPARK-12154 > URL: https://issues.apache.org/jira/browse/SPARK-12154 > Project: Spark > Issue Type: Sub-task > Components: Build, Spark Core >Affects Versions: 1.5.2 >Reporter: Matt Cheah >Assignee: Matt Cheah >Priority: Blocker > Fix For: 2.0.0 > > > Fairly self-explanatory, Jersey 1 is a bit old and could use an upgrade. > Library conflicts for Jersey are difficult to workaround - see discussion on > SPARK-11081. It's easier to upgrade Jersey entirely, but we should target > Spark 2.0 since this may be a break for users who were using Jersey 1 in > their Spark jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
