[jira] [Commented] (HIVE-16005) miscellaneous small fixes to help with llap debuggability
[ https://issues.apache.org/jira/browse/HIVE-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883412#comment-15883412 ] Prasanth Jayachandran commented on HIVE-16005: -- +1 > miscellaneous small fixes to help with llap debuggability > - > > Key: HIVE-16005 > URL: https://issues.apache.org/jira/browse/HIVE-16005 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-16005.01.patch, HIVE-16005.02.patch, > HIVE-16005.03.patch > > > - Include proc_ in cli, beeline, metastore, hs2 process args > - LLAP history logger - log QueryId instead of dagName (dag name is free > flowing text) > - LLAP JXM ExecutorStatus - Log QueryId instead of dagName. Sort by running / > queued > - Include thread name in TaskRunnerCallable so that it shows up in stack > traces (will cause extra output in logs) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16005) miscellaneous small fixes to help with llap debuggability
[
https://issues.apache.org/jira/browse/HIVE-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882388#comment-15882388
]
Hive QA commented on HIVE-16005:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854311/HIVE-16005.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10258 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=140)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3748/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3748/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3748/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12854311 - PreCommit-HIVE-Build
> miscellaneous small fixes to help with llap debuggability
> -
>
> Key: HIVE-16005
> URL: https://issues.apache.org/jira/browse/HIVE-16005
> Project: Hive
> Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-16005.01.patch, HIVE-16005.02.patch,
> HIVE-16005.03.patch
>
>
> - Include proc_ in cli, beeline, metastore, hs2 process args
> - LLAP history logger - log QueryId instead of dagName (dag name is free
> flowing text)
> - LLAP JXM ExecutorStatus - Log QueryId instead of dagName. Sort by running /
> queued
> - Include thread name in TaskRunnerCallable so that it shows up in stack
> traces (will cause extra output in logs)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16005) miscellaneous small fixes to help with llap debuggability
[
https://issues.apache.org/jira/browse/HIVE-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880201#comment-15880201
]
Hive QA commented on HIVE-16005:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854097/HIVE-16005.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10254 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestFirstInFirstOutComparator.testWaitQueueComparatorParallelism
(batchId=278)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparator
(batchId=278)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorAging
(batchId=278)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorParallelism
(batchId=278)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3719/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3719/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3719/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12854097 - PreCommit-HIVE-Build
> miscellaneous small fixes to help with llap debuggability
> -
>
> Key: HIVE-16005
> URL: https://issues.apache.org/jira/browse/HIVE-16005
> Project: Hive
> Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-16005.01.patch, HIVE-16005.02.patch
>
>
> - Include proc_ in cli, beeline, metastore, hs2 process args
> - LLAP history logger - log QueryId instead of dagName (dag name is free
> flowing text)
> - LLAP JXM ExecutorStatus - Log QueryId instead of dagName. Sort by running /
> queued
> - Include thread name in TaskRunnerCallable so that it shows up in stack
> traces (will cause extra output in logs)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16005) miscellaneous small fixes to help with llap debuggability
[ https://issues.apache.org/jira/browse/HIVE-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879455#comment-15879455 ] Prasanth Jayachandran commented on HIVE-16005: -- can we please have queryId+"-"+dagId? just to be consistent with log file and log url. > miscellaneous small fixes to help with llap debuggability > - > > Key: HIVE-16005 > URL: https://issues.apache.org/jira/browse/HIVE-16005 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-16005.01.patch > > > - Include proc_ in cli, beeline, metastore, hs2 process args > - LLAP history logger - log QueryId instead of dagName (dag name is free > flowing text) > - LLAP JXM ExecutorStatus - Log QueryId instead of dagName. Sort by running / > queued > - Include thread name in TaskRunnerCallable so that it shows up in stack > traces (will cause extra output in logs) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16005) miscellaneous small fixes to help with llap debuggability
[
https://issues.apache.org/jira/browse/HIVE-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879399#comment-15879399
]
Sergey Shelukhin commented on HIVE-16005:
-
DagID is generated in TezWork ctor {noformat} public TezWork(String queryId,
Configuration conf) {
this.dagId = queryId + ":" + counter.getAndIncrement(); {noformat} and
retrieved via getDagId. Doesn't really matter to me which one is used (or if
they matched), just a nit.
+1 can be modified on commit if needed.
> miscellaneous small fixes to help with llap debuggability
> -
>
> Key: HIVE-16005
> URL: https://issues.apache.org/jira/browse/HIVE-16005
> Project: Hive
> Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-16005.01.patch
>
>
> - Include proc_ in cli, beeline, metastore, hs2 process args
> - LLAP history logger - log QueryId instead of dagName (dag name is free
> flowing text)
> - LLAP JXM ExecutorStatus - Log QueryId instead of dagName. Sort by running /
> queued
> - Include thread name in TaskRunnerCallable so that it shows up in stack
> traces (will cause extra output in logs)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16005) miscellaneous small fixes to help with llap debuggability
[ https://issues.apache.org/jira/browse/HIVE-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879381#comment-15879381 ] Siddharth Seth commented on HIVE-16005: --- Is there a utility method to construct queryName":"dagIndex ? I can use that, or the one used in the log files. End goal is not to have free flowing text - which is what dagName was. bq. Appending suffix to the thread name, is it primarily to get some context from jstack output? For stacktraces that gets logged will already have these info via NDC. This is to have the thread name in the trace. bq. Also in constructThreadNameSuffix, why does it have to do the dance with all the IDs, aren't all of them appended if you just do attemptId toString? Cutting down the length, since it will be logged on each line. > miscellaneous small fixes to help with llap debuggability > - > > Key: HIVE-16005 > URL: https://issues.apache.org/jira/browse/HIVE-16005 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-16005.01.patch > > > - Include proc_ in cli, beeline, metastore, hs2 process args > - LLAP history logger - log QueryId instead of dagName (dag name is free > flowing text) > - LLAP JXM ExecutorStatus - Log QueryId instead of dagName. Sort by running / > queued > - Include thread name in TaskRunnerCallable so that it shows up in stack > traces (will cause extra output in logs) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16005) miscellaneous small fixes to help with llap debuggability
[ https://issues.apache.org/jira/browse/HIVE-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879067#comment-15879067 ] Sergey Shelukhin commented on HIVE-16005: - I the same method; I think I saw Hive use queryName + ":" + dagIndex elsewhere for this; why not here? Also in constructThreadNameSuffix, why does it have to do the dance with all the IDs, aren't all of them appended if you just do attemptId toString? > miscellaneous small fixes to help with llap debuggability > - > > Key: HIVE-16005 > URL: https://issues.apache.org/jira/browse/HIVE-16005 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-16005.01.patch > > > - Include proc_ in cli, beeline, metastore, hs2 process args > - LLAP history logger - log QueryId instead of dagName (dag name is free > flowing text) > - LLAP JXM ExecutorStatus - Log QueryId instead of dagName. Sort by running / > queued > - Include thread name in TaskRunnerCallable so that it shows up in stack > traces (will cause extra output in logs) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16005) miscellaneous small fixes to help with llap debuggability
[
https://issues.apache.org/jira/browse/HIVE-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878124#comment-15878124
]
Hive QA commented on HIVE-16005:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12853898/HIVE-16005.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10238 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=104)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestFirstInFirstOutComparator.testWaitQueueComparatorParallelism
(batchId=278)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparator
(batchId=278)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorAging
(batchId=278)
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorParallelism
(batchId=278)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3692/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3692/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3692/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12853898 - PreCommit-HIVE-Build
> miscellaneous small fixes to help with llap debuggability
> -
>
> Key: HIVE-16005
> URL: https://issues.apache.org/jira/browse/HIVE-16005
> Project: Hive
> Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-16005.01.patch
>
>
> - Include proc_ in cli, beeline, metastore, hs2 process args
> - LLAP history logger - log QueryId instead of dagName (dag name is free
> flowing text)
> - LLAP JXM ExecutorStatus - Log QueryId instead of dagName. Sort by running /
> queued
> - Include thread name in TaskRunnerCallable so that it shows up in stack
> traces (will cause extra output in logs)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16005) miscellaneous small fixes to help with llap debuggability
[ https://issues.apache.org/jira/browse/HIVE-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877707#comment-15877707 ] Prasanth Jayachandran commented on HIVE-16005: -- for constructUniqueQueryId.. can we use the same format as that of filenames generated by query-routing logger? (queryId-dagId) This way easier to locate the corresponding log file. Appending suffix to the thread name, is it primarily to get some context from jstack output? For stacktraces that gets logged will already have these info via NDC. > miscellaneous small fixes to help with llap debuggability > - > > Key: HIVE-16005 > URL: https://issues.apache.org/jira/browse/HIVE-16005 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-16005.01.patch > > > - Include proc_ in cli, beeline, metastore, hs2 process args > - LLAP history logger - log QueryId instead of dagName (dag name is free > flowing text) > - LLAP JXM ExecutorStatus - Log QueryId instead of dagName. Sort by running / > queued > - Include thread name in TaskRunnerCallable so that it shows up in stack > traces (will cause extra output in logs) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
