[jira] [Commented] (HIVE-18651) Expose additional Spark metrics
[
https://issues.apache.org/jira/browse/HIVE-18651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423883#comment-16423883
]
Hive QA commented on HIVE-18651:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12917259/HIVE-18651.3.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 193 failed/errored test(s), 13290 tests
executed
*Failed tests:*
{noformat}
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=246)
TestExportImport - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=246)
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Commented] (HIVE-18651) Expose additional Spark metrics
[
https://issues.apache.org/jira/browse/HIVE-18651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423822#comment-16423822
]
Hive QA commented on HIVE-18651:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
45s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
6s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
42s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 42s{color}
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 3 new + 0
unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} spark-client: The patch generated 2 new + 18 unchanged
- 0 fixed = 20 total (was 18) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
17s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9972/dev-support/hive-personality.sh
|
| git revision | master / ad9852c |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9972/yetus/patch-mvninstall-itests_hive-unit.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9972/yetus/patch-mvninstall-ql.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9972/yetus/patch-compile-itests_hive-unit.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9972/yetus/patch-compile-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9972/yetus/diff-checkstyle-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9972/yetus/diff-checkstyle-spark-client.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9972/yetus/patch-asflicense-problems.txt
|
| modules | C: itests/hive-unit ql spark-client U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9972/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Expose additional Spark metrics
> ---
>
> Key: HIVE-18651
> URL: https://issues.apache.org/jira/browse/HIVE-18651
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments:
[jira] [Commented] (HIVE-18651) Expose additional Spark metrics
[ https://issues.apache.org/jira/browse/HIVE-18651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423187#comment-16423187 ] Vihang Karajgaonkar commented on HIVE-18651: +1 LGTM. Thanks for making the changes. > Expose additional Spark metrics > --- > > Key: HIVE-18651 > URL: https://issues.apache.org/jira/browse/HIVE-18651 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18651.1.patch, HIVE-18651.2.patch, > HIVE-18651.3.patch > > > There have been multiples additional metrics that get collected via Spark > (such as executor CPU time spent). We should expose them in HoS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18651) Expose additional Spark metrics
[ https://issues.apache.org/jira/browse/HIVE-18651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423182#comment-16423182 ] Sahil Takiar commented on HIVE-18651: - Thanks for taking a look [~vihangk1]. Addressed your comments, updated the RB and attached the updated patch. > Expose additional Spark metrics > --- > > Key: HIVE-18651 > URL: https://issues.apache.org/jira/browse/HIVE-18651 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18651.1.patch, HIVE-18651.2.patch, > HIVE-18651.3.patch > > > There have been multiples additional metrics that get collected via Spark > (such as executor CPU time spent). We should expose them in HoS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18651) Expose additional Spark metrics
[ https://issues.apache.org/jira/browse/HIVE-18651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416106#comment-16416106 ] Sahil Takiar commented on HIVE-18651: - [~vihangk1] can you take a look? This change basically exposes two more additional metrics that Spark already collects - ExecutorCpuTime and ExecutorDeserializeCpuTime. These metrics are collected from Spark and printed in the logs. There are a few other re-factoring changes that were required to add some more unit tests. > Expose additional Spark metrics > --- > > Key: HIVE-18651 > URL: https://issues.apache.org/jira/browse/HIVE-18651 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18651.1.patch, HIVE-18651.2.patch > > > There have been multiples additional metrics that get collected via Spark > (such as executor CPU time spent). We should expose them in HoS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18651) Expose additional Spark metrics
[
https://issues.apache.org/jira/browse/HIVE-18651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415239#comment-16415239
]
Hive QA commented on HIVE-18651:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12916242/HIVE-18651.2.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 70 failed/errored test(s), 13441 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18651) Expose additional Spark metrics
[
https://issues.apache.org/jira/browse/HIVE-18651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415170#comment-16415170
]
Hive QA commented on HIVE-18651:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
34s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
1s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
37s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 37s{color}
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/hive-unit: The patch generated 3 new + 0
unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} spark-client: The patch generated 2 new + 18 unchanged
- 0 fixed = 20 total (was 18) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9862/dev-support/hive-personality.sh
|
| git revision | master / 1e884cc |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9862/yetus/patch-mvninstall-itests_hive-unit.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9862/yetus/patch-mvninstall-ql.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9862/yetus/patch-compile-itests_hive-unit.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9862/yetus/patch-compile-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9862/yetus/diff-checkstyle-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9862/yetus/diff-checkstyle-spark-client.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9862/yetus/patch-asflicense-problems.txt
|
| modules | C: itests/hive-unit ql spark-client U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9862/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Expose additional Spark metrics
> ---
>
> Key: HIVE-18651
> URL: https://issues.apache.org/jira/browse/HIVE-18651
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments:
[jira] [Commented] (HIVE-18651) Expose additional Spark metrics
[
https://issues.apache.org/jira/browse/HIVE-18651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413565#comment-16413565
]
Hive QA commented on HIVE-18651:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12916135/HIVE-18651.1.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 114 failed/errored test(s), 13436 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18651) Expose additional Spark metrics
[
https://issues.apache.org/jira/browse/HIVE-18651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413516#comment-16413516
]
Hive QA commented on HIVE-18651:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
42s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
51s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
36s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 36s{color}
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} itests/hive-unit: The patch generated 3 new + 0
unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} spark-client: The patch generated 2 new + 18 unchanged
- 0 fixed = 20 total (was 18) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9841/dev-support/hive-personality.sh
|
| git revision | master / 5b222db |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9841/yetus/patch-mvninstall-itests_hive-unit.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9841/yetus/patch-mvninstall-ql.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9841/yetus/patch-compile-itests_hive-unit.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9841/yetus/patch-compile-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9841/yetus/diff-checkstyle-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9841/yetus/diff-checkstyle-spark-client.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9841/yetus/patch-asflicense-problems.txt
|
| modules | C: itests/hive-unit ql spark-client U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9841/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Expose additional Spark metrics
> ---
>
> Key: HIVE-18651
> URL: https://issues.apache.org/jira/browse/HIVE-18651
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments:
