[jira] [Commented] (HIVE-12205) Spark: unify spark statististics aggregation between local and remote spark client
[
https://issues.apache.org/jira/browse/HIVE-12205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15150575#comment-15150575
]
Xuefu Zhang commented on HIVE-12205:
Also cherry-picked to master. Thanks, guys!
> Spark: unify spark statististics aggregation between local and remote spark
> client
> --
>
> Key: HIVE-12205
> URL: https://issues.apache.org/jira/browse/HIVE-12205
> Project: Hive
> Issue Type: Task
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chinna Rao Lalam
> Fix For: spark-branch, 2.1.0
>
> Attachments: HIVE-12205.1.patch, HIVE-12205.2.patch,
> HIVE-12205.3.patch
>
>
> In class {{LocalSparkJobStatus}} and {{RemoteSparkJobStatus}}, spark
> statistics aggregation are done similar but in different code paths. Ideally,
> we should have a unified approach to simply maintenance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12205) Spark: unify spark statististics aggregation between local and remote spark client
[
https://issues.apache.org/jira/browse/HIVE-12205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15150268#comment-15150268
]
Chengxiang Li commented on HIVE-12205:
--
Merged to Spark branch, thanks Chinna for this contribution.
> Spark: unify spark statististics aggregation between local and remote spark
> client
> --
>
> Key: HIVE-12205
> URL: https://issues.apache.org/jira/browse/HIVE-12205
> Project: Hive
> Issue Type: Task
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-12205.1.patch, HIVE-12205.2.patch,
> HIVE-12205.3.patch
>
>
> In class {{LocalSparkJobStatus}} and {{RemoteSparkJobStatus}}, spark
> statistics aggregation are done similar but in different code paths. Ideally,
> we should have a unified approach to simply maintenance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12205) Spark: unify spark statististics aggregation between local and remote spark client
[
https://issues.apache.org/jira/browse/HIVE-12205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149350#comment-15149350
]
Hive QA commented on HIVE-12205:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12788035/HIVE-12205.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9790 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7002/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7002/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7002/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12788035 - PreCommit-HIVE-TRUNK-Build
> Spark: unify spark statististics aggregation between local and remote spark
> client
> --
>
> Key: HIVE-12205
> URL: https://issues.apache.org/jira/browse/HIVE-12205
> Project: Hive
> Issue Type: Task
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-12205.1.patch, HIVE-12205.2.patch,
> HIVE-12205.3.patch
>
>
> In class {{LocalSparkJobStatus}} and {{RemoteSparkJobStatus}}, spark
> statistics aggregation are done similar but in different code paths. Ideally,
> we should have a unified approach to simply maintenance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12205) Spark: unify spark statististics aggregation between local and remote spark client
[
https://issues.apache.org/jira/browse/HIVE-12205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146312#comment-15146312
]
Chengxiang Li commented on HIVE-12205:
--
+1
> Spark: unify spark statististics aggregation between local and remote spark
> client
> --
>
> Key: HIVE-12205
> URL: https://issues.apache.org/jira/browse/HIVE-12205
> Project: Hive
> Issue Type: Task
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-12205.1.patch, HIVE-12205.2.patch
>
>
> In class {{LocalSparkJobStatus}} and {{RemoteSparkJobStatus}}, spark
> statistics aggregation are done similar but in different code paths. Ideally,
> we should have a unified approach to simply maintenance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12205) Spark: unify spark statististics aggregation between local and remote spark client
[
https://issues.apache.org/jira/browse/HIVE-12205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135776#comment-15135776
]
Chengxiang Li commented on HIVE-12205:
--
Thanks, Chinna, i'k
发自我的 iPhone
> Spark: unify spark statististics aggregation between local and remote spark
> client
> --
>
> Key: HIVE-12205
> URL: https://issues.apache.org/jira/browse/HIVE-12205
> Project: Hive
> Issue Type: Task
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-12205.1.patch, HIVE-12205.2.patch
>
>
> In class {{LocalSparkJobStatus}} and {{RemoteSparkJobStatus}}, spark
> statistics aggregation are done similar but in different code paths. Ideally,
> we should have a unified approach to simply maintenance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12205) Spark: unify spark statististics aggregation between local and remote spark client
[
https://issues.apache.org/jira/browse/HIVE-12205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135777#comment-15135777
]
Chengxiang Li commented on HIVE-12205:
--
Thanks, Chinna, I'm on vocation now, I would review this when I'm back a week
later.
>From chengxiang's iPhone
> Spark: unify spark statististics aggregation between local and remote spark
> client
> --
>
> Key: HIVE-12205
> URL: https://issues.apache.org/jira/browse/HIVE-12205
> Project: Hive
> Issue Type: Task
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-12205.1.patch, HIVE-12205.2.patch
>
>
> In class {{LocalSparkJobStatus}} and {{RemoteSparkJobStatus}}, spark
> statistics aggregation are done similar but in different code paths. Ideally,
> we should have a unified approach to simply maintenance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12205) Spark: unify spark statististics aggregation between local and remote spark client
[
https://issues.apache.org/jira/browse/HIVE-12205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088795#comment-15088795
]
Chengxiang Li commented on HIVE-12205:
--
[~chinnalalam], thanks working on this.
In your patch, the statistic aggregation is still computed separately in
different methods(although in same class now) for {{LocalSparkJobStatus}} and
{{RemoteSparkJobStatus}}, i suggest you can add a initialize method in
{{MetrisCollection}} with parameter {{String jobId, Map jobMetrics}}, so that {{LocalSparkJobStatus}} can reuse
{{MetricsCollection}} to aggregate statistics as well. What do you think?
Besides, could you create a ticket on RB for this?
> Spark: unify spark statististics aggregation between local and remote spark
> client
> --
>
> Key: HIVE-12205
> URL: https://issues.apache.org/jira/browse/HIVE-12205
> Project: Hive
> Issue Type: Task
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-12205.1.patch
>
>
> In class {{LocalSparkJobStatus}} and {{RemoteSparkJobStatus}}, spark
> statistics aggregation are done similar but in different code paths. Ideally,
> we should have a unified approach to simply maintenance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12205) Spark: unify spark statististics aggregation between local and remote spark client
[
https://issues.apache.org/jira/browse/HIVE-12205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968752#comment-14968752
]
Chinna Rao Lalam commented on HIVE-12205:
-
I will work on this.
> Spark: unify spark statististics aggregation between local and remote spark
> client
> --
>
> Key: HIVE-12205
> URL: https://issues.apache.org/jira/browse/HIVE-12205
> Project: Hive
> Issue Type: Task
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>
> In class {{LocalSparkJobStatus}} and {{RemoteSparkJobStatus}}, spark
> statistics aggregation are done similar but in different code paths. Ideally,
> we should have a unified approach to simply maintenance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
