[jira] [Commented] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504355#comment-16504355
]
Peter Vary commented on HIVE-19079:
---
Thanks for fixing [~stakiar]! :)
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19079.1.patch, HIVE-19079.2.patch,
> HIVE-19079.3.patch, HIVE-19079.4.patch, HIVE-19079.5.patch, Spark Collapse
> Truncated Query.png, Spark Expanded Truncated Query.png
>
>
> As of HIVE-16601, we place a shortened version of the query into the Spark
> job description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you double click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503675#comment-16503675
]
Sahil Takiar commented on HIVE-19079:
-
Thanks for catching that Peter. Pushed an addendum commit to fix the issue.
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19079.1.patch, HIVE-19079.2.patch,
> HIVE-19079.3.patch, HIVE-19079.4.patch, HIVE-19079.5.patch, Spark Collapse
> Truncated Query.png, Spark Expanded Truncated Query.png
>
>
> As of HIVE-16601, we place a shortened version of the query into the Spark
> job description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you double click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503022#comment-16503022
]
Peter Vary commented on HIVE-19079:
---
As the yetus pointed out, we forget to add ASF license to the following files:
{code:java}
Lines that start with ? in the ASF License report indicate files that do
not have an Apache license header:
!?
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11551/ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveSparkClient.java
!?
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11551/spark-client/src/test/java/org/apache/hive/spark/client/TestSparkLauncherSparkClient.java{code}
Could you please add them?
Thanks,
Peter
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19079.1.patch, HIVE-19079.2.patch,
> HIVE-19079.3.patch, HIVE-19079.4.patch, HIVE-19079.5.patch, Spark Collapse
> Truncated Query.png, Spark Expanded Truncated Query.png
>
>
> As of HIVE-16601, we place a shortened version of the query into the Spark
> job description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you double click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502026#comment-16502026
]
Aihua Xu commented on HIVE-19079:
-
That makes sense as well to keep the consistency across the engines. +1.
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19079.1.patch, HIVE-19079.2.patch,
> HIVE-19079.3.patch, HIVE-19079.4.patch, HIVE-19079.5.patch, Spark Collapse
> Truncated Query.png, Spark Expanded Truncated Query.png
>
>
> As of HIVE-16601, we place a shortened version of the query into the Spark
> job description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you double click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501850#comment-16501850
]
Sahil Takiar commented on HIVE-19079:
-
[~aihuaxu] after looking at this a bit more. I think it makes sense to keep the
current naming. Technically, {{hive.spark.jobname.length}} is still just
controlling the length of the job name (e.g. it controls the size of the value
for {{mapreduce.job.name}}
[here|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L2475]).
Same with {{DagUtils#getQueryName}} - technically it is just getting the "query
name" by reading the config {{hive.query.name}} and if that isn't set, it uses
the {{mapreduce.job.name}} as a fallback - Tez does something similar
[here|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L1470].
The only difference is that {{HiveSparkClient}} interprets the "query name" as
a "query description", but that doesn't change the fact that
{{DagUtils#getQueryName}} is still returning the "query name". I've added some
Javadocs to make this clearer. What do you think?
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19079.1.patch, HIVE-19079.2.patch,
> HIVE-19079.3.patch, HIVE-19079.4.patch, HIVE-19079.5.patch, Spark Collapse
> Truncated Query.png, Spark Expanded Truncated Query.png
>
>
> As of HIVE-16601, we place a shortened version of the query into the Spark
> job description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you double click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500840#comment-16500840
]
Hive QA commented on HIVE-19079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12926089/HIVE-19079.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14460 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11506/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11506/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11506/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12926089 - PreCommit-HIVE-Build
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19079.1.patch, HIVE-19079.2.patch,
> HIVE-19079.3.patch, HIVE-19079.4.patch, Spark Collapse Truncated Query.png,
> Spark Expanded Truncated Query.png
>
>
> As of HIVE-16601, we place a shortened version of the query into the Spark
> job description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you double click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500800#comment-16500800
]
Hive QA commented on HIVE-19079:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2277 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 2 new + 142 unchanged - 0
fixed = 144 total (was 142) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11506/dev-support/hive-personality.sh
|
| git revision | master / d682ca9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11506/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11506/yetus/patch-asflicense-problems.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11506/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19079.1.patch, HIVE-19079.2.patch,
> HIVE-19079.3.patch, HIVE-19079.4.patch, Spark Collapse Truncated Query.png,
> Spark Expanded Truncated Query.png
>
>
> As of HIVE-16601, we place a shortened version of the query into the Spark
> job description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you double click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498598#comment-16498598
]
Aihua Xu commented on HIVE-19079:
-
The patch looks great. Minor comments. Since spark uses description rather than
name, can we have hive.spark.jobdescription.length as the property name? And
also, can we change DagUtils.getQueryName() changes to getQueryDescription()?
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19079.1.patch, HIVE-19079.2.patch,
> HIVE-19079.3.patch, HIVE-19079.4.patch, Spark Collapse Truncated Query.png,
> Spark Expanded Truncated Query.png
>
>
> As of HIVE-16601, we place a shortened version of the query into the Spark
> job description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you double click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498026#comment-16498026
]
Sahil Takiar commented on HIVE-19079:
-
[~aihuaxu] could you take a look?
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19079.1.patch, HIVE-19079.2.patch,
> HIVE-19079.3.patch, HIVE-19079.4.patch, Spark Collapse Truncated Query.png,
> Spark Expanded Truncated Query.png
>
>
> As of HIVE-16601, we place a shortened version of the query into the Spark
> job description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you double click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497792#comment-16497792
]
Hive QA commented on HIVE-19079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12925596/HIVE-19079.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14440 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testDataTypes (batchId=242)
org.apache.hive.spark.client.rpc.TestRpc.testServerPort (batchId=307)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11405/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11405/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11405/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12925596 - PreCommit-HIVE-Build
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19079.1.patch, HIVE-19079.2.patch,
> HIVE-19079.3.patch, Spark Collapse Truncated Query.png, Spark Expanded
> Truncated Query.png
>
>
> As of HIVE-16601, we place a shortened very of the query into the Spark job
> description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497765#comment-16497765
]
Hive QA commented on HIVE-19079:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
28s{color} | {color:blue} ql in master has 2278 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 2 new + 142 unchanged - 0
fixed = 144 total (was 142) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11405/dev-support/hive-personality.sh
|
| git revision | master / 1c970d9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11405/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11405/yetus/patch-asflicense-problems.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11405/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19079.1.patch, HIVE-19079.2.patch,
> HIVE-19079.3.patch, Spark Collapse Truncated Query.png, Spark Expanded
> Truncated Query.png
>
>
> As of HIVE-16601, we place a shortened very of the query into the Spark job
> description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16444810#comment-16444810
]
Hive QA commented on HIVE-19079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12919831/HIVE-19079.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 14279 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_smb] (batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[tez_join_hash]
(batchId=54)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=225)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestAMReporter.testMultipleAM
(batchId=309)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookieNegative
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10351/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10351/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10351/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12919831 - PreCommit-HIVE-Build
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19079.1.patch, HIVE-19079.2.patch, Spark Collapse
> Truncated Query.png, Spark Expanded Truncated Query.png
>
>
> As of HIVE-16601, we place a shortened very of the query into the Spark job
> description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16444719#comment-16444719
]
Hive QA commented on HIVE-19079:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 2 new + 138 unchanged - 0
fixed = 140 total (was 138) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10351/dev-support/hive-personality.sh
|
| git revision | master / 9f15e22 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10351/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10351/yetus/patch-asflicense-problems.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10351/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19079.1.patch, HIVE-19079.2.patch, Spark Collapse
> Truncated Query.png, Spark Expanded Truncated Query.png
>
>
> As of HIVE-16601, we place a shortened very of the query into the Spark job
> description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19079) Add extended query string to Spark job description
[
https://issues.apache.org/jira/browse/HIVE-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438898#comment-16438898
]
Hive QA commented on HIVE-19079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12919029/Expanded-Truncated-Query.png
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10234/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10234/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10234/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-04-16 00:54:30.092
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-10234/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-04-16 00:54:30.094
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 204d715 HIVE-17645 : MM tables patch conflicts with HIVE-17482
(Spark/Acid integration) (Jason Dere via Eugene Koifman)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 204d715 HIVE-17645 : MM tables patch conflicts with HIVE-17482
(Spark/Acid integration) (Jason Dere via Eugene Koifman)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-04-16 00:54:31.312
+ rm -rf ../yetus_PreCommit-HIVE-Build-10234
+ mkdir ../yetus_PreCommit-HIVE-Build-10234
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-10234
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-10234/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: unrecognized input
fatal: unrecognized input
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12919029 - PreCommit-HIVE-Build
> Add extended query string to Spark job description
> --
>
> Key: HIVE-19079
> URL: https://issues.apache.org/jira/browse/HIVE-19079
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: Collapsed-Truncated-Query.png,
> Expanded-Truncated-Query.png, HIVE-19079.1.patch
>
>
> As of HIVE-16601, we place a shortened very of the query into the Spark job
> description. We should look into adding a longer version of the query. It
> seems that the Spark Web UI has a nice feature where long job descriptions
> will be truncated with a {{...}}, but when you click on the {{...}} it
> expands to show the rest of the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
