[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194597#comment-16194597
]
Barna Zsombor Klara commented on HIVE-17078:
Pushed to master thanks [~Yibing] for the patch and [~pvary] for the review.
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Barna Zsombor Klara
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH, HIVE-17078.5.PATCH,
> HIVE-17078.6.patch, HIVE-17078.7.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193202#comment-16193202
]
Hive QA commented on HIVE-17078:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890534/HIVE-17078.7.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11200 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=232)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=232)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=101)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=240)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23]
(batchId=240)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7141/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7141/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7141/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12890534 - PreCommit-HIVE-Build
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Barna Zsombor Klara
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH, HIVE-17078.5.PATCH,
> HIVE-17078.6.patch, HIVE-17078.7.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191003#comment-16191003
]
Barna Zsombor Klara commented on HIVE-17078:
Failures should not be related.
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH, HIVE-17078.5.PATCH, HIVE-17078.6.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189833#comment-16189833
]
Hive QA commented on HIVE-17078:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890146/HIVE-17078.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 11193 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=232)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=232)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_union_remove_1]
(batchId=83)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=101)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=240)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23]
(batchId=240)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=203)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking1
(batchId=284)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7092/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7092/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7092/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12890146 - PreCommit-HIVE-Build
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH, HIVE-17078.5.PATCH, HIVE-17078.6.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171261#comment-16171261
]
Barna Zsombor Klara commented on HIVE-17078:
Hi [~Yibing],
do you have the free time to do the refactoring suggested by Peter? If not then
I'd be happy to add the final touch. The patch looks good, so it would be a
shame to not commit it.
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH, HIVE-17078.5.PATCH
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121503#comment-16121503
]
Peter Vary commented on HIVE-17078:
---
Hi [~Yibing],
I have found this {{SparkClientImpl.Redirector}} which is called like this:
{code}
redirect("RemoteDriver-stdout-redir-" + threadName, new
Redirector(child.getInputStream()));
{code}
This is essentially the same as the proposed solution here. Do you think it is
worth to move this to hive-common, and use it?
Thanks,
Peter
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH, HIVE-17078.5.PATCH
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089885#comment-16089885
]
Hive QA commented on HIVE-17078:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12877578/HIVE-17078.5.PATCH
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 11074 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=238)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=238)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fp_literal_arithmetic]
(batchId=65)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_2]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_use_op_stats]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_use_ts_stats_for_mapjoin]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=167)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=233)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=233)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=178)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=178)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=178)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6060/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6060/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6060/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12877578 - PreCommit-HIVE-Build
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH, HIVE-17078.5.PATCH
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087586#comment-16087586
]
Sahil Takiar commented on HIVE-17078:
-
Ok sounds good.
Few other comments:
* Can you change the thread name to include something about the child process?
Right now it just includes the current threads id + stderr / stdout which seems
to suggest the logs the stderr / stdout of the specified thread, which isn't
the case
* Where does {{l4j}} print to?
* You can use {{StringUtils.stringifyException}} instead of writing the stack
trace to a {{StringOutputStream}}
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085386#comment-16085386
]
Hive QA commented on HIVE-17078:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12877025/HIVE-17078.4.PATCH
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10889 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=237)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_partition_dynamic]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=145)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=232)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=177)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6003/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6003/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6003/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12877025 - PreCommit-HIVE-Build
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085252#comment-16085252
]
Yibing Shi commented on HIVE-17078:
---
Checked the failed tests.
# org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
fails with something irrelevant to this patch
# org.apache.hive.hcatalog.api.TestHCatClient. The failure also has nothing to
do our patch.
# org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14/23] fails
because the output has changed in order. Nothing serious. We need to somehow
update the .out files, but maybe in a separate JIRA
# The other tests fails because now we have more logs in local tasks. Will
update the .out files.
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085078#comment-16085078
]
Hive QA commented on HIVE-17078:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12876980/HIVE-17078.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10888 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=237)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_without_localtask]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_8]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_convert_join]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=145)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=232)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=232)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=177)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5994/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5994/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5994/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12876980 - PreCommit-HIVE-Build
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084891#comment-16084891
]
Yibing Shi commented on HIVE-17078:
---
I am trying to keep the current behaviour. With Hive CLI, by default Hive logs
are not printed. Some users may rely on the stdout/stderr information. I don't
want to surprise them.
If you still think it is unnecessary to print child stdout/stderr to Hive
stdout/stderr, I can remove the corresponding code.
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084462#comment-16084462
]
Sahil Takiar commented on HIVE-17078:
-
If we are printing the child stdout / stderr to the Hive logs then do we need
to also print them to Hive stdout / stderr too?
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084287#comment-16084287
]
Hive QA commented on HIVE-17078:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12876861/HIVE-17078.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10840 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=237)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=143)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=232)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=177)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConcurrentStatements (batchId=226)
org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5983/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5983/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5983/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12876861 - PreCommit-HIVE-Build
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084016#comment-16084016
]
Hive QA commented on HIVE-17078:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12876842/HIVE-17078.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5981/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5981/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5981/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-07-12 13:54:51.943
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-5981/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-07-12 13:54:51.946
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 26d6de7 HIVE-16730: Vectorization: Schema Evolution for Text
Vectorization / Complex Types
+ git clean -f -d
Removing ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderAdaptor.java
Removing
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java.orig
Removing ql/src/test/queries/clientpositive/llap_acid_fast.q
Removing ql/src/test/results/clientpositive/llap/llap_acid.q.out
Removing ql/src/test/results/clientpositive/llap/llap_acid_fast.q.out
Removing ql/src/test/results/clientpositive/llap_acid_fast.q.out
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 26d6de7 HIVE-16730: Vectorization: Schema Evolution for Text
Vectorization / Complex Types
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-07-12 13:54:57.818
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java:36
error: ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java:
patch does not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12876842 - PreCommit-HIVE-Build
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Minor
> Attachments: HIVE-17078.1.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
