[jira] [Commented] (HIVE-23062) Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue
[
https://issues.apache.org/jira/browse/HIVE-23062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063770#comment-17063770
]
Hive QA commented on HIVE-23062:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
53s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 8 new + 7 unchanged - 0 fixed
= 15 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21204/dev-support/hive-personality.sh
|
| git revision | master / ef99369 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21204/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21204/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue
> --
>
> Key: HIVE-23062
> URL: https://issues.apache.org/jira/browse/HIVE-23062
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-23062.1.patch
>
>
> Currently if custom Tez queue is used, Hive will only check the Http port, so
> it is not handling TLS and Yarn HA mode URL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23061) LLAP crash due to unhandled exception: Cannot invoke unregister on an entity which has not been registered
[
https://issues.apache.org/jira/browse/HIVE-23061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063764#comment-17063764
]
Prasanth Jayachandran commented on HIVE-23061:
--
{quote}I suspect that on that error, that fragment is cleaned up which may
clear the info for the first fragment, and when the first fragment exits it may
hit this. Still need more investigation on this one.
{quote}
This will be a problem always right? The duplicated fragment cannot cleanup the
original/first task fragment..
I am ok with the fix but what said above is in fact happening then that has to
be addressed as it can lead to more issues which will be hard to debug.
Llap ignoring duplicates is correct. But clients has to handle it based on the
reason for rejection. If the reason is duplicate then client should not trigger
taskCleanup.
> LLAP crash due to unhandled exception: Cannot invoke unregister on an entity
> which has not been registered
> --
>
> Key: HIVE-23061
> URL: https://issues.apache.org/jira/browse/HIVE-23061
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-23061.1.patch
>
>
> The following exception goes uncaught and causes the entire LLAP daemon to
> shut down:
> {noformat}
> 2020-03-17T06:49:11,304 ERROR [ExecutionCompletionThread #0 ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread
> Thread[ExecutionCompletionThread #0,5,main] threw an Exception. Shutting down
> now...
> java.lang.IllegalStateException: Cannot invoke unregister on an entity which
> has not been registered
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:508)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo$FinishableStateTracker.unregisterForUpdates(QueryInfo.java:256)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo.unregisterFinishableStateUpdate(QueryInfo.java:209)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryFragmentInfo.unregisterForFinishableStateUpdates(QueryFragmentInfo.java:166)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$TaskWrapper.maybeUnregisterForFinishedStateNotifications(TaskExecutorService.java:1177)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$InternalCompletionListener.onSuccess(TaskExecutorService.java:980)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$InternalCompletionListener.onSuccess(TaskExecutorService.java:944)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1021)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_191]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_191]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063763#comment-17063763
]
Hive QA commented on HIVE-22997:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997267/HIVE-22997.16.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 18128 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testBootstrapFailedDump
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testConcatenatePartitionedTable
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testConstraints
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testIdempotentMoveTaskForInsertFiles
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testIncrementalInsertDropUnpartitionedTable
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testIncrementalInserts
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testIncrementalLoad
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testLoadCmPathMissing
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapFailedDump
(batchId=269)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenatePartitionedTable
(batchId=269)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=269)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIdempotentMoveTaskForInsertFiles
(batchId=269)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalInsertDropUnpartitionedTable
(batchId=269)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalInserts
(batchId=269)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoad
(batchId=269)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testLoadCmPathMissing
(batchId=269)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21203/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21203/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21203/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997267 - PreCommit-HIVE-Build
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch,
> HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch,
> HIVE-22997.15.patch, HIVE-22997.16.patch, HIVE-22997.2.patch,
> HIVE-22997.4.patch, HIVE-22997.5.patch, HIVE-22997.6.patch,
> HIVE-22997.7.patch, HIVE-22997.8.patch, HIVE-22997.9.patch
>
> Time Spent: 9h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063748#comment-17063748
]
Hive QA commented on HIVE-22997:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 6 new + 76 unchanged - 5 fixed
= 82 total (was 81) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
26s{color} | {color:red} itests/hive-unit: The patch generated 11 new + 649
unchanged - 1 fixed = 660 total (was 650) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
56s{color} | {color:red} ql generated 1 new + 1530 unchanged - 1 fixed = 1531
total (was 1531) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | org.apache.hadoop.hive.ql.exec.repl.ReplDumpWork is Serializable;
consider declaring a serialVersionUID At ReplDumpWork.java:a serialVersionUID
At ReplDumpWork.java:[lines 43-173] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21203/dev-support/hive-personality.sh
|
| git revision | master / ef99369 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21203/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21203/yetus/diff-checkstyle-itests_hive-unit.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21203/yetus/new-findbugs-ql.html
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21203/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>
[jira] [Commented] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063747#comment-17063747 ] Ashutosh Chauhan commented on HIVE-23004: - can you please add RB link? > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.10.patch, > HIVE-23004.11.patch, HIVE-23004.12.patch, HIVE-23004.13.patch, > HIVE-23004.2.patch, HIVE-23004.4.patch, HIVE-23004.6.patch, > HIVE-23004.7.patch, HIVE-23004.8.patch, HIVE-23004.9.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23004) Support Decimal64 operations across multiple vertices
[
https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063741#comment-17063741
]
Hive QA commented on HIVE-23004:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997243/HIVE-23004.13.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18121 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21202/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21202/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21202/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997243 - PreCommit-HIVE-Build
> Support Decimal64 operations across multiple vertices
> -
>
> Key: HIVE-23004
> URL: https://issues.apache.org/jira/browse/HIVE-23004
> Project: Hive
> Issue Type: Bug
>Reporter: Ramesh Kumar Thangarajan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-23004.1.patch, HIVE-23004.10.patch,
> HIVE-23004.11.patch, HIVE-23004.12.patch, HIVE-23004.13.patch,
> HIVE-23004.2.patch, HIVE-23004.4.patch, HIVE-23004.6.patch,
> HIVE-23004.7.patch, HIVE-23004.8.patch, HIVE-23004.9.patch
>
>
> Support Decimal64 operations across multiple vertices
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23066) [Subqueries] Throw an error if COALESCE/NVL is used in correlated condition
[
https://issues.apache.org/jira/browse/HIVE-23066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg reassigned HIVE-23066:
--
> [Subqueries] Throw an error if COALESCE/NVL is used in correlated condition
> ---
>
> Key: HIVE-23066
> URL: https://issues.apache.org/jira/browse/HIVE-23066
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> NVL could lead to wrong results
> {code:sql}
> create table TABLEA (id int, lib string);
> insert into TABLEA values (1, 'a'),(2, 'b'),(3, null),(5,'zx');
> create table TABLEB (id int, lib string);
> insert into TABLEB values (1, 'a'),(4, 'c'),(3, null),(5,'zy');
> select *
> from TABLEA a
> where exists (
>select 1
>from TABLEB b
>where nvl(a.lib,0) = nvl(b.lib,0)
> );
> {code}
> ***OUTPUT***
> {noformat}
> +---++
> | a.id | a.lib |
> +---++
> | 1 | a |
> +---++
> {noformat}
> ***EXPECTED***
> {noformat}
> +---++
> | a.id | a.lib |
> +---++
> | 1 | a |
> | 3 | NULL |
> +---++
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=407354=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-407354
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 21/Mar/20 02:29
Start Date: 21/Mar/20 02:29
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on pull request #951: HIVE-22997
: Copy external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r395952745
##
File path:
ql/src/test/org/apache/hadoop/hive/ql/exec/repl/TestReplDumpTask.java
##
@@ -127,13 +128,15 @@ public void
removeDBPropertyToPreventRenameWhenBootstrapDumpOfTableFails() throw
private int tableDumpCount = 0;
@Override
- void dumpTable(String dbName, String tblName, String validTxnList, Path
dbRoot, Path replDataDir,
- long lastReplId, Hive hiveDb, HiveWrapper.Tuple
tuple)
+ List dumpTable(String dbName, String tblName,
String validTxnList, Path dbRoot,
+ Path replDataDir, long
lastReplId, Hive hiveDb,
+ HiveWrapper.Tuple tuple)
throws Exception {
tableDumpCount++;
if (tableDumpCount > 1) {
throw new TestException();
}
+return Collections.EMPTY_LIST;
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 407354)
Time Spent: 8h 20m (was: 8h 10m)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch,
> HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch,
> HIVE-22997.15.patch, HIVE-22997.16.patch, HIVE-22997.2.patch,
> HIVE-22997.4.patch, HIVE-22997.5.patch, HIVE-22997.6.patch,
> HIVE-22997.7.patch, HIVE-22997.8.patch, HIVE-22997.9.patch
>
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=407356=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-407356
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 21/Mar/20 02:29
Start Date: 21/Mar/20 02:29
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on pull request #951: HIVE-22997
: Copy external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r395953009
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -78,18 +81,21 @@
import java.io.IOException;
import java.io.Serializable;
import java.nio.charset.StandardCharsets;
+import java.util.Iterator;
import java.util.Set;
import java.util.HashSet;
import java.util.List;
import java.util.Arrays;
import java.util.Collections;
import java.util.Base64;
-import java.util.ArrayList;
+import java.util.LinkedList;
import java.util.UUID;
+import java.util.ArrayList;
import java.util.concurrent.TimeUnit;
import static org.apache.hadoop.hive.ql.exec.repl.ReplExternalTables.Writer;
public class ReplDumpTask extends Task implements Serializable {
+ private static final long serialVersionUID = 1L;
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 407356)
Time Spent: 8h 40m (was: 8.5h)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch,
> HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch,
> HIVE-22997.15.patch, HIVE-22997.16.patch, HIVE-22997.2.patch,
> HIVE-22997.4.patch, HIVE-22997.5.patch, HIVE-22997.6.patch,
> HIVE-22997.7.patch, HIVE-22997.8.patch, HIVE-22997.9.patch
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=407357=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-407357
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 21/Mar/20 02:29
Start Date: 21/Mar/20 02:29
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on pull request #951: HIVE-22997
: Copy external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r395952774
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplOperationCompleteAckWork.java
##
@@ -18,27 +18,28 @@
package org.apache.hadoop.hive.ql.exec.repl;
+import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.ql.plan.Explain;
import org.apache.hadoop.hive.ql.plan.Explain.Level;
import java.io.Serializable;
/**
- * ReplLoadCompleteAckWork.
- * FS based Acknowledgement for repl load complete
+ * ReplOperationCompleteAckWork.
+ * FS based acknowledgement on repl dump and repl load completion.
*
*/
-@Explain(displayName = "Repl Load Complete Ack", explainLevels = { Level.USER,
Level.DEFAULT, Level.EXTENDED })
-public class ReplLoadCompleteAckWork implements Serializable {
+@Explain(displayName = "Repl Operation Complete Ack", explainLevels = {
Level.USER, Level.DEFAULT, Level.EXTENDED })
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 407357)
Time Spent: 8h 50m (was: 8h 40m)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch,
> HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch,
> HIVE-22997.15.patch, HIVE-22997.16.patch, HIVE-22997.2.patch,
> HIVE-22997.4.patch, HIVE-22997.5.patch, HIVE-22997.6.patch,
> HIVE-22997.7.patch, HIVE-22997.8.patch, HIVE-22997.9.patch
>
> Time Spent: 8h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=407359=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-407359
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 21/Mar/20 02:29
Start Date: 21/Mar/20 02:29
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on pull request #951: HIVE-22997
: Copy external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r395952920
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -122,33 +128,38 @@ public String getName() {
@Override
public int execute() {
try {
- Hive hiveDb = getHive();
- Path dumpRoot = new Path(conf.getVar(HiveConf.ConfVars.REPLDIR),
-
Base64.getEncoder().encodeToString(work.dbNameOrPattern.toLowerCase()
- .getBytes(StandardCharsets.UTF_8.name(;
- Path previousDumpMetaPath = getPreviousDumpMetadataPath(dumpRoot);
- Path previousHiveDumpPath =
- previousDumpMetaPath != null ? new Path(previousDumpMetaPath,
ReplUtils.REPL_HIVE_BASE_DIR) : null;
- //If no previous dump is present or previous dump was loaded, proceed
- if (shouldDump(previousHiveDumpPath)) {
-Path currentDumpPath = new Path(dumpRoot, getNextDumpDir());
-Path hiveDumpRoot = new Path(currentDumpPath,
ReplUtils.REPL_HIVE_BASE_DIR);
-DumpMetaData dmd = new DumpMetaData(hiveDumpRoot, conf);
-// Initialize ReplChangeManager instance since we will require it to
encode file URI.
-ReplChangeManager.getInstance(conf);
-Path cmRoot = new Path(conf.getVar(HiveConf.ConfVars.REPLCMDIR));
-Long lastReplId;
-if (previousHiveDumpPath == null) {
- lastReplId = bootStrapDump(hiveDumpRoot, dmd, cmRoot, hiveDb);
+ if (work.dirCopyIteratorInitialized() ||
work.replPathIteratorInitialized()) {
+intitiateDataCopyTasks();
+ } else {
+Hive hiveDb = getHive();
+Path dumpRoot = new Path(conf.getVar(HiveConf.ConfVars.REPLDIR),
+
Base64.getEncoder().encodeToString(work.dbNameOrPattern.toLowerCase()
+.getBytes(StandardCharsets.UTF_8.name(;
+Path previousDumpMetaPath = getPreviousDumpMetadataPath(dumpRoot);
+Path previousHiveDumpPath =
+previousDumpMetaPath != null ? new Path(previousDumpMetaPath,
ReplUtils.REPL_HIVE_BASE_DIR) : null;
+//If no previous dump is present or previous dump was loaded, proceed
+if (shouldDump(previousHiveDumpPath)) {
+ Path currentDumpPath = new Path(dumpRoot, getNextDumpDir());
+ Path hiveDumpRoot = new Path(currentDumpPath,
ReplUtils.REPL_HIVE_BASE_DIR);
+ DumpMetaData dmd = new DumpMetaData(hiveDumpRoot, conf);
+ // Initialize ReplChangeManager instance since we will require it to
encode file URI.
+ ReplChangeManager.getInstance(conf);
+ Path cmRoot = new Path(conf.getVar(HiveConf.ConfVars.REPLCMDIR));
+ Long lastReplId;
+ if (previousHiveDumpPath == null) {
+lastReplId = bootStrapDump(hiveDumpRoot, dmd, cmRoot, hiveDb);
+ } else {
+
work.setEventFrom(getEventFromPreviousDumpMetadata(previousHiveDumpPath));
+lastReplId = incrementalDump(hiveDumpRoot, dmd, cmRoot, hiveDb);
+ }
+
work.setResultValues(Arrays.asList(currentDumpPath.toUri().toString(),
String.valueOf(lastReplId)));
+ deleteAllPreviousDumpMeta(dumpRoot, currentDumpPath);
+ work.setDumpAckFile(new Path(hiveDumpRoot,
ReplUtils.DUMP_ACKNOWLEDGEMENT));
+ intitiateDataCopyTasks();
} else {
-
work.setEventFrom(getEventFromPreviousDumpMetadata(previousHiveDumpPath));
- lastReplId = incrementalDump(hiveDumpRoot, dmd, cmRoot, hiveDb);
+ LOG.warn("Previous Dump is not yet loaded");
}
-prepareReturnValues(Arrays.asList(currentDumpPath.toUri().toString(),
String.valueOf(lastReplId)));
-writeDumpCompleteAck(hiveDumpRoot);
-deleteAllPreviousDumpMeta(dumpRoot, currentDumpPath);
- } else {
-LOG.warn("Previous Dump is not yet loaded");
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 407359)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
>
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=407360=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-407360
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 21/Mar/20 02:29
Start Date: 21/Mar/20 02:29
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on pull request #951: HIVE-22997
: Copy external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r395953040
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -662,15 +696,37 @@ void dumpTable(String dbName, String tblName, String
validTxnList, Path dbRoot,
replLogger.tableLog(tblName, tableSpec.tableHandle.getTableType());
if (tableSpec.tableHandle.getTableType().equals(TableType.EXTERNAL_TABLE)
|| Utils.shouldDumpMetaDataOnly(conf)) {
- return;
+ return Collections.EMPTY_LIST;
}
-for (ReplPathMapping replPathMapping: replPathMappings) {
- Task copyTask = ReplCopyTask.getLoadCopyTask(
- tuple.replicationSpec, replPathMapping.getSrcPath(),
replPathMapping.getTargetPath(), conf, false);
- this.addDependentTask(copyTask);
- LOG.info("Scheduled a repl copy task from [{}] to [{}]",
- replPathMapping.getSrcPath(), replPathMapping.getTargetPath());
+return replPathMappings;
+ }
+
+ private void intitiateDataCopyTasks() throws SemanticException {
+Iterator extCopyWorkItr =
work.getDirCopyIterator();
+List> childTasks = new ArrayList<>();
+int maxTasks =
conf.getIntVar(HiveConf.ConfVars.REPL_APPROX_MAX_LOAD_TASKS);
+TaskTracker taskTracker = new TaskTracker(maxTasks);
+while (taskTracker.canAddMoreTasks() && hasMoreCopyWork()) {
+ if (work.replPathIteratorInitialized() && extCopyWorkItr.hasNext()) {
+childTasks.addAll(new ExternalTableCopyTaskBuilder(work,
conf).tasks(taskTracker));
+ } else {
+childTasks.addAll(ReplPathMapping.tasks(work, taskTracker, conf));
+ }
+}
+if (!childTasks.isEmpty()) {
+ DAGTraversal.traverse(childTasks, new
AddDependencyToLeaves(TaskFactory.get(work, conf)));
+} else {
+ prepareReturnValues(work.getResultValues());
+ childTasks.add(TaskFactory.get(new
ReplOperationCompleteAckWork(work.getDumpAckFile()), conf));
}
Review comment:
Have refactored a bit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 407360)
Time Spent: 9h (was: 8h 50m)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch,
> HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch,
> HIVE-22997.15.patch, HIVE-22997.16.patch, HIVE-22997.2.patch,
> HIVE-22997.4.patch, HIVE-22997.5.patch, HIVE-22997.6.patch,
> HIVE-22997.7.patch, HIVE-22997.8.patch, HIVE-22997.9.patch
>
> Time Spent: 9h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=407358=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-407358
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 21/Mar/20 02:29
Start Date: 21/Mar/20 02:29
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on pull request #951: HIVE-22997
: Copy external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r395952912
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -122,33 +128,38 @@ public String getName() {
@Override
public int execute() {
try {
- Hive hiveDb = getHive();
- Path dumpRoot = new Path(conf.getVar(HiveConf.ConfVars.REPLDIR),
-
Base64.getEncoder().encodeToString(work.dbNameOrPattern.toLowerCase()
- .getBytes(StandardCharsets.UTF_8.name(;
- Path previousDumpMetaPath = getPreviousDumpMetadataPath(dumpRoot);
- Path previousHiveDumpPath =
- previousDumpMetaPath != null ? new Path(previousDumpMetaPath,
ReplUtils.REPL_HIVE_BASE_DIR) : null;
- //If no previous dump is present or previous dump was loaded, proceed
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 407358)
Time Spent: 8h 50m (was: 8h 40m)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch,
> HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch,
> HIVE-22997.15.patch, HIVE-22997.16.patch, HIVE-22997.2.patch,
> HIVE-22997.4.patch, HIVE-22997.5.patch, HIVE-22997.6.patch,
> HIVE-22997.7.patch, HIVE-22997.8.patch, HIVE-22997.9.patch
>
> Time Spent: 8h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=407355=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-407355 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 21/Mar/20 02:29 Start Date: 21/Mar/20 02:29 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r395952861 ## File path: ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/StageType.java ## @@ -30,7 +30,7 @@ REPL_TXN(15), REPL_INCREMENTAL_LOAD(16), SCHEDULED_QUERY_MAINT(17), - REPL_LOAD_COMPLETE_ACK(18); + REPL_OPERATION_COMPLETE_ACK(18); private final int value; Review comment: Removed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 407355) Time Spent: 8.5h (was: 8h 20m) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch, > HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch, > HIVE-22997.15.patch, HIVE-22997.16.patch, HIVE-22997.2.patch, > HIVE-22997.4.patch, HIVE-22997.5.patch, HIVE-22997.6.patch, > HIVE-22997.7.patch, HIVE-22997.8.patch, HIVE-22997.9.patch > > Time Spent: 8.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23004) Support Decimal64 operations across multiple vertices
[
https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063731#comment-17063731
]
Hive QA commented on HIVE-23004:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} serde: The patch generated 6 new + 289 unchanged - 0
fixed = 295 total (was 289) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
52s{color} | {color:red} ql: The patch generated 32 new + 849 unchanged - 3
fixed = 881 total (was 852) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
50s{color} | {color:red} serde generated 3 new + 194 unchanged - 3 fixed = 197
total (was 197) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
0s{color} | {color:red} ql generated 2 new + 1530 unchanged - 1 fixed = 1532
total (was 1531) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:serde |
| | new
org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead(TypeInfo[],
DataTypePhysicalVariation[], boolean, boolean[], byte[], byte[]) may expose
internal representation by storing an externally mutable object into
BinarySortableDeserializeRead.columnNotNullMarker At
BinarySortableDeserializeRead.java:byte[], byte[]) may expose internal
representation by storing an externally mutable object into
BinarySortableDeserializeRead.columnNotNullMarker At
BinarySortableDeserializeRead.java:[line 145] |
| | new
org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead(TypeInfo[],
DataTypePhysicalVariation[], boolean, boolean[], byte[], byte[]) may expose
internal representation by storing an externally mutable object into
BinarySortableDeserializeRead.columnNullMarker At
BinarySortableDeserializeRead.java:byte[], byte[]) may expose internal
representation by storing an externally mutable object into
BinarySortableDeserializeRead.columnNullMarker At
BinarySortableDeserializeRead.java:[line 144] |
| | new
org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead(TypeInfo[],
DataTypePhysicalVariation[], boolean, boolean[], byte[], byte[]) may expose
internal representation by storing an externally mutable object into
BinarySortableDeserializeRead.columnSortOrderIsDesc At
BinarySortableDeserializeRead.java:byte[],
[jira] [Updated] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-22997: -- Attachment: (was: HIVE-22997.16.patch) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch, > HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch, > HIVE-22997.15.patch, HIVE-22997.16.patch, HIVE-22997.2.patch, > HIVE-22997.4.patch, HIVE-22997.5.patch, HIVE-22997.6.patch, > HIVE-22997.7.patch, HIVE-22997.8.patch, HIVE-22997.9.patch > > Time Spent: 8h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-22997: -- Attachment: HIVE-22997.16.patch > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch, > HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch, > HIVE-22997.15.patch, HIVE-22997.16.patch, HIVE-22997.2.patch, > HIVE-22997.4.patch, HIVE-22997.5.patch, HIVE-22997.6.patch, > HIVE-22997.7.patch, HIVE-22997.8.patch, HIVE-22997.9.patch > > Time Spent: 8h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23062) Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue
[ https://issues.apache.org/jira/browse/HIVE-23062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063726#comment-17063726 ] Jason Dere commented on HIVE-23062: --- Actually good point [~krisden], looks like YarnConfiguration.useHttps() can be used here > Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue > -- > > Key: HIVE-23062 > URL: https://issues.apache.org/jira/browse/HIVE-23062 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-23062.1.patch > > > Currently if custom Tez queue is used, Hive will only check the Http port, so > it is not handling TLS and Yarn HA mode URL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23061) LLAP crash due to unhandled exception: Cannot invoke unregister on an entity which has not been registered
[
https://issues.apache.org/jira/browse/HIVE-23061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063720#comment-17063720
]
Jason Dere commented on HIVE-23061:
---
As for why the error is occurring in the first place, it looks like LLAP is
getting duplicate query fragments submissions from the external client (HWC).
So the duplicate fragment is submitted and it looks like it fails:
{noformat}2020-03-17T06:49:11,239 WARN [IPC Server handler 2 on 15001 ()]
org.apache.hadoop.ipc.Server: IPC Server handler 2 on 15001, call Call#75
Retry#0 org.apache.hadoop.hive.llap.protocol.LlapProtocolBlockingPB.submitWork
from 19.40.252.114:33906
java.lang.IllegalStateException: Only a single registration allowed per entity.
Duplicate for TaskWrapper{task=attempt_1854104024183112753_6052_0_00_000128_1,
inWaitQueue=true, inPreemptionQueue=false, registeredForNotifications=true,
canFinish=true, canFinish(in queue)=true, isGuaranteed=false,
firstAttemptStartTime=1584442003327, dagStartTime=1584442003327,
withinDagPriority=0, vertexParallelism= 2132, selfAndUpstreamParallelism= 2132,
selfAndUpstreamComplete= 0}
at
org.apache.hadoop.hive.llap.daemon.impl.QueryInfo$FinishableStateTracker.registerForUpdates(QueryInfo.java:233)
~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
at
org.apache.hadoop.hive.llap.daemon.impl.QueryInfo.registerForFinishableStateUpdates(QueryInfo.java:205)
~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
at
org.apache.hadoop.hive.llap.daemon.impl.QueryFragmentInfo.registerForFinishableStateUpdates(QueryFragmentInfo.java:160)
~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
at
org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$TaskWrapper.maybeRegisterForFinishedStateNotifications(TaskExecutorService.java:1167)
~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
at
org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.schedule(TaskExecutorService.java:564)
~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
at
org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.schedule(TaskExecutorService.java:93)
~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
at
org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl.submitWork(ContainerRunnerImpl.java:292)
~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
at
org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.submitWork(LlapDaemon.java:610)
~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
at
org.apache.hadoop.hive.llap.daemon.impl.LlapProtocolServerImpl.submitWork(LlapProtocolServerImpl.java:122)
~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
at
org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos$LlapDaemonProtocol$2.callBlockingMethod(LlapDaemonProtocolProtos.java:22695)
~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_191]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
~[hadoop-common-3.1.1.3.1.4.26-3.jar:?]
{noformat}
I suspect that on that error, that fragment is cleaned up which may clear the
info for the first fragment, and when the first fragment exits it may hit this.
Still need more investigation on this one.
But in general, I don't think we should be allowing errors from one fragment
submission cause the entire LLAP daemon to die, which is why I've done this
patch.
> LLAP crash due to unhandled exception: Cannot invoke unregister on an entity
> which has not been registered
> --
>
> Key: HIVE-23061
> URL: https://issues.apache.org/jira/browse/HIVE-23061
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-23061.1.patch
>
>
> The following exception goes uncaught and causes the entire LLAP daemon to
> shut down:
> {noformat}
> 2020-03-17T06:49:11,304 ERROR [ExecutionCompletionThread #0 ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread
>
[jira] [Assigned] (HIVE-23065) Remove Calls to printStackTrace in Module hive-service
[ https://issues.apache.org/jira/browse/HIVE-23065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-23065: - > Remove Calls to printStackTrace in Module hive-service > -- > > Key: HIVE-23065 > URL: https://issues.apache.org/jira/browse/HIVE-23065 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-23065.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23065) Remove Calls to printStackTrace in Module hive-service
[ https://issues.apache.org/jira/browse/HIVE-23065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23065: -- Status: Patch Available (was: Open) > Remove Calls to printStackTrace in Module hive-service > -- > > Key: HIVE-23065 > URL: https://issues.apache.org/jira/browse/HIVE-23065 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-23065.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23065) Remove Calls to printStackTrace in Module hive-service
[ https://issues.apache.org/jira/browse/HIVE-23065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23065: -- Attachment: HIVE-23065.1.patch > Remove Calls to printStackTrace in Module hive-service > -- > > Key: HIVE-23065 > URL: https://issues.apache.org/jira/browse/HIVE-23065 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-23065.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23064) Remove Calls to printStackTrace in Module hive-exec
[ https://issues.apache.org/jira/browse/HIVE-23064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23064: -- Status: Patch Available (was: Open) > Remove Calls to printStackTrace in Module hive-exec > --- > > Key: HIVE-23064 > URL: https://issues.apache.org/jira/browse/HIVE-23064 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23064.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23064) Remove Calls to printStackTrace in Module hive-exec
[ https://issues.apache.org/jira/browse/HIVE-23064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23064: -- Attachment: HIVE-23064.1.patch > Remove Calls to printStackTrace in Module hive-exec > --- > > Key: HIVE-23064 > URL: https://issues.apache.org/jira/browse/HIVE-23064 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23064.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23059) In constraint name uniqueness query use the MTable instead of it's id
[
https://issues.apache.org/jira/browse/HIVE-23059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063712#comment-17063712
]
Hive QA commented on HIVE-23059:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997239/HIVE-23059.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18123 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21201/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21201/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21201/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997239 - PreCommit-HIVE-Build
> In constraint name uniqueness query use the MTable instead of it's id
> -
>
> Key: HIVE-23059
> URL: https://issues.apache.org/jira/browse/HIVE-23059
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23059.01.patch
>
>
> The constraint name uniqueness query uses the MTable instead of it's id. This
> will solve the immediate problem of not being able to create the sys db in
> Oracle, but not solve the same problem about the rest of the queries with
> integer number parameters (Long, Integer, etc).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23064) Remove Calls to printStackTrace in Module hive-exec
[ https://issues.apache.org/jira/browse/HIVE-23064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-23064: - > Remove Calls to printStackTrace in Module hive-exec > --- > > Key: HIVE-23064 > URL: https://issues.apache.org/jira/browse/HIVE-23064 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22484) Remove Calls to printStackTrace
[
https://issues.apache.org/jira/browse/HIVE-22484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22484:
--
Attachment: (was: HIVE-22484.2.patch)
> Remove Calls to printStackTrace
> ---
>
> Key: HIVE-22484
> URL: https://issues.apache.org/jira/browse/HIVE-22484
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> In many cases, the call to {{printStackTrace}} bypasses the logging
> framework, in other cases, the error stack trace is printed and the exception
> is re-thrown (log-and-throw is a bad pattern), and then there are some other
> edge cases.
> Remove this call and replace with calls to the logging framework or remove
> altogether if exception is wrapped and re-thrown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22484) Remove Calls to printStackTrace
[
https://issues.apache.org/jira/browse/HIVE-22484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22484:
--
Attachment: (was: HIVE-22484.1.patch)
> Remove Calls to printStackTrace
> ---
>
> Key: HIVE-22484
> URL: https://issues.apache.org/jira/browse/HIVE-22484
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> In many cases, the call to {{printStackTrace}} bypasses the logging
> framework, in other cases, the error stack trace is printed and the exception
> is re-thrown (log-and-throw is a bad pattern), and then there are some other
> edge cases.
> Remove this call and replace with calls to the logging framework or remove
> altogether if exception is wrapped and re-thrown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[
https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063709#comment-17063709
]
David Mollitor commented on HIVE-23005:
---
{quote}fix the initial fetchsize
{quote}
You'll have to be more specific there. After discussing it, I thought we were
good with the current solution. I will add documentation in a subsequent task
if you can provide me some pointers on where/how to do that.
However, the plan is, once this patch is pushed, in order of precedence, the
fetch size is determined this way :
# Fetch size is set based on what is received from HS2
({{hive.server2.thrift.resultset.default.fetch.size}}) during the client
session open sequence
# Fetch size is set in the JDBC connection string (not well documented:
{{jdbc:hive2://localhost:1;fetchSize=100}})
# Fetch size is set by the application code via JDBC {{setFetchSize}} [1]
Keeping in mind that whatever the client requests for a fetch size will be
overruled on the HS2-side by {{hive.server2.thrift.resultset.max.fetch.size}}.
When a client requests a fetchSize larger than the max, a WARN message is
emitted into the HS2 logs for further investigation and to direct clients to
adjust their expectations (and configurations).
[1]
[https://docs.oracle.com/javase/8/docs/api/java/sql/Statement.html#setFetchSize-int-]
HIVE-22853 allows beeline to set the fetch size directly on the JDBC client
(#3). Therefore:
# If the beeline user does nothing, each query will use the fetch size
received from HS2
# If the beeline user sets the fetchSize in the JDBC connection string, each
query will use the fetch size specified there
# If the user wants to set the fetchSize in the session, they can with the
syntax: {{!set fetchSize xxx}}
## Setting a fetchSize of 0 will direct the driver to use the fetch size
provided from HS2
## Setting a fetchSize greater than 0 will set the driver fetch size to the
specified value
## Setting a fetchSize of -1 directs beeline to use the default JDBC default
behavior: use the connection string fetchSize and, if none is specified,
fallback to the fetch size specified by HS2 (this is the default beeline
fetchSize value)
## Setting a fetchSize of any other negative integer value is an error
> Consider Default JDBC Fetch Size From HS2
> -
>
> Key: HIVE-23005
> URL: https://issues.apache.org/jira/browse/HIVE-23005
> Project: Hive
> Issue Type: Sub-task
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23005.1.patch, HIVE-23005.2.patch,
> HIVE-23005.3.patch, HIVE-23005.4.patch, HIVE-23005.5.patch, HIVE-23005.6.patch
>
>
> # Create two fetch sizes: and "init" fetch size and a "default" fetch size.
> The "init" fetch size comes from the JDBC connection string _fetchSize_ (if
> present) and the "default" fetch size comes from
> _HIVE_SERVER2_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE_ the server response to the
> open session request.
> # When a {{Statement}} is created, its starting fetch size is the "init"
> fetch size (may be 0)
> # Manually setting the fetch size on the {{Statement}} to 0, sets the fetch
> size to be the server default
> Setting to zero defaults to the server's instructed default and adheres to:
> bq. If the value specified is zero, then the hint is ignored. The default
> value is zero.
> That is to say, if the fetch size is 0, the default from the server is used,
> otherwise the user can pass a 'hint' and that will be the number of rows
> fetched instead.
> https://docs.oracle.com/javase/8/docs/api/java/sql/Statement.html#setFetchSize-int-
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23062) Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue
[
https://issues.apache.org/jira/browse/HIVE-23062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063706#comment-17063706
]
Kevin Risden commented on HIVE-23062:
-
In the patch ssl check is on HS2 instead of checking for YARN TLS instead?
{code}
sslForHS2 = conf.getBoolVar(HiveConf.ConfVars.HIVE_SERVER2_USE_SSL);
{code}
This will work if HS2 TLS and YARN TLS are varied together, but that might not
be true?
Usually yarn-site.xml would have "yarn.http.policy=HTTPS_ONLY" which would be
what indicates if YARN is in SSL or not. Reference is here:
https://hadoop.apache.org/docs/current3/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
Since this patch is already checking for other YARN configurations, it probably
makes sense to check for YARN TLS/SSL explicitly and not the HS2 value.
> Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue
> --
>
> Key: HIVE-23062
> URL: https://issues.apache.org/jira/browse/HIVE-23062
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-23062.1.patch
>
>
> Currently if custom Tez queue is used, Hive will only check the Http port, so
> it is not handling TLS and Yarn HA mode URL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[
https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063698#comment-17063698
]
David Mollitor commented on HIVE-23005:
---
[~ngangam] OK. Guava removed from JDBC project. [^HIVE-23005.6.patch]
Thanks for your time on the review.
> Consider Default JDBC Fetch Size From HS2
> -
>
> Key: HIVE-23005
> URL: https://issues.apache.org/jira/browse/HIVE-23005
> Project: Hive
> Issue Type: Sub-task
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23005.1.patch, HIVE-23005.2.patch,
> HIVE-23005.3.patch, HIVE-23005.4.patch, HIVE-23005.5.patch, HIVE-23005.6.patch
>
>
> # Create two fetch sizes: and "init" fetch size and a "default" fetch size.
> The "init" fetch size comes from the JDBC connection string _fetchSize_ (if
> present) and the "default" fetch size comes from
> _HIVE_SERVER2_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE_ the server response to the
> open session request.
> # When a {{Statement}} is created, its starting fetch size is the "init"
> fetch size (may be 0)
> # Manually setting the fetch size on the {{Statement}} to 0, sets the fetch
> size to be the server default
> Setting to zero defaults to the server's instructed default and adheres to:
> bq. If the value specified is zero, then the hint is ignored. The default
> value is zero.
> That is to say, if the fetch size is 0, the default from the server is used,
> otherwise the user can pass a 'hint' and that will be the number of rows
> fetched instead.
> https://docs.oracle.com/javase/8/docs/api/java/sql/Statement.html#setFetchSize-int-
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[
https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23005:
--
Attachment: HIVE-23005.6.patch
> Consider Default JDBC Fetch Size From HS2
> -
>
> Key: HIVE-23005
> URL: https://issues.apache.org/jira/browse/HIVE-23005
> Project: Hive
> Issue Type: Sub-task
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23005.1.patch, HIVE-23005.2.patch,
> HIVE-23005.3.patch, HIVE-23005.4.patch, HIVE-23005.5.patch, HIVE-23005.6.patch
>
>
> # Create two fetch sizes: and "init" fetch size and a "default" fetch size.
> The "init" fetch size comes from the JDBC connection string _fetchSize_ (if
> present) and the "default" fetch size comes from
> _HIVE_SERVER2_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE_ the server response to the
> open session request.
> # When a {{Statement}} is created, its starting fetch size is the "init"
> fetch size (may be 0)
> # Manually setting the fetch size on the {{Statement}} to 0, sets the fetch
> size to be the server default
> Setting to zero defaults to the server's instructed default and adheres to:
> bq. If the value specified is zero, then the hint is ignored. The default
> value is zero.
> That is to say, if the fetch size is 0, the default from the server is used,
> otherwise the user can pass a 'hint' and that will be the number of rows
> fetched instead.
> https://docs.oracle.com/javase/8/docs/api/java/sql/Statement.html#setFetchSize-int-
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23063) Use the same PerfLogger all over Compiler
[ https://issues.apache.org/jira/browse/HIVE-23063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063695#comment-17063695 ] Jesus Camacho Rodriguez commented on HIVE-23063: +1 > Use the same PerfLogger all over Compiler > - > > Key: HIVE-23063 > URL: https://issues.apache.org/jira/browse/HIVE-23063 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23063.01.patch > > > Compiler gets it's PerfLogger again and again from SessionState, relying on > the fact that it is returned from the same ThreadLocal. As this may change in > the future it'd be better to use the same instance, obtained only once. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23063) Use the same PerfLogger all over Compiler
[ https://issues.apache.org/jira/browse/HIVE-23063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063694#comment-17063694 ] David Mollitor commented on HIVE-23063: --- [~mgergely] Please check out my earlier work on the matter. [HIVE-22402] > Use the same PerfLogger all over Compiler > - > > Key: HIVE-23063 > URL: https://issues.apache.org/jira/browse/HIVE-23063 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23063.01.patch > > > Compiler gets it's PerfLogger again and again from SessionState, relying on > the fact that it is returned from the same ThreadLocal. As this may change in > the future it'd be better to use the same instance, obtained only once. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23059) In constraint name uniqueness query use the MTable instead of it's id
[
https://issues.apache.org/jira/browse/HIVE-23059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063692#comment-17063692
]
Hive QA commented on HIVE-23059:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21201/dev-support/hive-personality.sh
|
| git revision | master / ef99369 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21201/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> In constraint name uniqueness query use the MTable instead of it's id
> -
>
> Key: HIVE-23059
> URL: https://issues.apache.org/jira/browse/HIVE-23059
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23059.01.patch
>
>
> The constraint name uniqueness query uses the MTable instead of it's id. This
> will solve the immediate problem of not being able to create the sys db in
> Oracle, but not solve the same problem about the rest of the queries with
> integer number parameters (Long, Integer, etc).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23063) Use the same PerfLogger all over Compiler
[ https://issues.apache.org/jira/browse/HIVE-23063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23063: -- Status: Patch Available (was: Open) > Use the same PerfLogger all over Compiler > - > > Key: HIVE-23063 > URL: https://issues.apache.org/jira/browse/HIVE-23063 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23063.01.patch > > > Compiler gets it's PerfLogger again and again from SessionState, relying on > the fact that it is returned from the same ThreadLocal. As this may change in > the future it'd be better to use the same instance, obtained only once. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23063) Use the same PerfLogger all over Compiler
[ https://issues.apache.org/jira/browse/HIVE-23063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23063: -- Attachment: HIVE-23063.01.patch > Use the same PerfLogger all over Compiler > - > > Key: HIVE-23063 > URL: https://issues.apache.org/jira/browse/HIVE-23063 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23063.01.patch > > > Compiler gets it's PerfLogger again and again from SessionState, relying on > the fact that it is returned from the same ThreadLocal. As this may change in > the future it'd be better to use the same instance, obtained only once. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23061) LLAP crash due to unhandled exception: Cannot invoke unregister on an entity which has not been registered
[
https://issues.apache.org/jira/browse/HIVE-23061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063690#comment-17063690
]
Prasanth Jayachandran commented on HIVE-23061:
--
{quote}Actually all LlapDaemonUncaughtExceptionHandler does is shut down the
LLAP daemon down, if it receives an exception .. is that incorrect logic?
{quote}
This is expected. We cannot recover but we want to log and see why it happened
and how to avoid it. I am wondering if we need a catch in onSuccess and
taskCleanup notifications handlers because the uncaught exceptions are already
logging it. We seem to catch and ignore it but the exception itself should not
have happened in first place? Do we know why there was an exception?
Especially, "java.lang.IllegalStateException: Cannot invoke unregister on an
entity which has not been registered"
is there a race condition or double notification issue here?
> LLAP crash due to unhandled exception: Cannot invoke unregister on an entity
> which has not been registered
> --
>
> Key: HIVE-23061
> URL: https://issues.apache.org/jira/browse/HIVE-23061
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-23061.1.patch
>
>
> The following exception goes uncaught and causes the entire LLAP daemon to
> shut down:
> {noformat}
> 2020-03-17T06:49:11,304 ERROR [ExecutionCompletionThread #0 ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread
> Thread[ExecutionCompletionThread #0,5,main] threw an Exception. Shutting down
> now...
> java.lang.IllegalStateException: Cannot invoke unregister on an entity which
> has not been registered
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:508)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo$FinishableStateTracker.unregisterForUpdates(QueryInfo.java:256)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo.unregisterFinishableStateUpdate(QueryInfo.java:209)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryFragmentInfo.unregisterForFinishableStateUpdates(QueryFragmentInfo.java:166)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$TaskWrapper.maybeUnregisterForFinishedStateNotifications(TaskExecutorService.java:1177)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$InternalCompletionListener.onSuccess(TaskExecutorService.java:980)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$InternalCompletionListener.onSuccess(TaskExecutorService.java:944)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1021)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_191]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_191]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063688#comment-17063688
]
Hive QA commented on HIVE-23052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997238/HIVE-23052.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18123 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21200/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21200/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21200/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997238 - PreCommit-HIVE-Build
> Optimize lock enqueueing in TxnHandler
> --
>
> Key: HIVE-23052
> URL: https://issues.apache.org/jira/browse/HIVE-23052
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23052.1.patch, HIVE-23052.2.patch
>
>
> TxnHandler::enqueueLockWithRetry uses select for update (S4U) to retrieve the
> next lock id from the metastore. This locks the NEXT_LOCK_ID table for
> concurrent lock requests until the transaction has been committed.
> Since this creates contention, we should reduce the scope of this S4U,
> thereby locking the table for only as long as necessary. Currently, within
> the same txn, after the initial S4U, we also insert the transaction
> components (TXN_COMPONENTS table) before committing, even though they are
> independent and do not make use of the lock id. So moving these txn_component
> inserts before the S4U would decrease the time the next_lock_id table is
> locked and thus increase concurrency.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23063) Use the same PerfLogger all over Compiler
[ https://issues.apache.org/jira/browse/HIVE-23063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-23063: - > Use the same PerfLogger all over Compiler > - > > Key: HIVE-23063 > URL: https://issues.apache.org/jira/browse/HIVE-23063 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > > Compiler gets it's PerfLogger again and again from SessionState, relying on > the fact that it is returned from the same ThreadLocal. As this may change in > the future it'd be better to use the same instance, obtained only once. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23061) LLAP crash due to unhandled exception: Cannot invoke unregister on an entity which has not been registered
[
https://issues.apache.org/jira/browse/HIVE-23061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063686#comment-17063686
]
Jason Dere commented on HIVE-23061:
---
Actually, I'm pretty sure we cannot recover properly from
LlapDaemonUncaughtExceptionHandler - at this point we don't know what
thread/executor queue has failed and needs setup/restarting all over again.
> LLAP crash due to unhandled exception: Cannot invoke unregister on an entity
> which has not been registered
> --
>
> Key: HIVE-23061
> URL: https://issues.apache.org/jira/browse/HIVE-23061
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-23061.1.patch
>
>
> The following exception goes uncaught and causes the entire LLAP daemon to
> shut down:
> {noformat}
> 2020-03-17T06:49:11,304 ERROR [ExecutionCompletionThread #0 ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread
> Thread[ExecutionCompletionThread #0,5,main] threw an Exception. Shutting down
> now...
> java.lang.IllegalStateException: Cannot invoke unregister on an entity which
> has not been registered
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:508)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo$FinishableStateTracker.unregisterForUpdates(QueryInfo.java:256)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo.unregisterFinishableStateUpdate(QueryInfo.java:209)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryFragmentInfo.unregisterForFinishableStateUpdates(QueryFragmentInfo.java:166)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$TaskWrapper.maybeUnregisterForFinishedStateNotifications(TaskExecutorService.java:1177)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$InternalCompletionListener.onSuccess(TaskExecutorService.java:980)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$InternalCompletionListener.onSuccess(TaskExecutorService.java:944)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1021)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_191]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_191]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23061) LLAP crash due to unhandled exception: Cannot invoke unregister on an entity which has not been registered
[
https://issues.apache.org/jira/browse/HIVE-23061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063680#comment-17063680
]
Jason Dere commented on HIVE-23061:
---
Hey [~prasanth_j], thanks for reviewing.
{quote}
1) When exception is caught and ignored, use LOG.warn() ? Or explicitly saying
"Ignoring"..
{quote}
Sure.
{quote}
2) The case where we do do nested try.. do we know why it throws exception?
{quote}
I think the last level of nesting is likely unnecessary, but just keeping a
top-level catch just in case we get some other error in the onSuccess/onFailure
callbacks.
{quote}
3) LLAP daemon should already has
Thread.setDefaultUncaughtExceptionHandler(new
LlapDaemonUncaughtExceptionHandler())
This should have caught any unhandled exceptions. Any idea why it wasn't
caught? May be uncaught exception handler is a better way to handle any
uncaught exceptions.
{quote}
Actually all LlapDaemonUncaughtExceptionHandler does is shut down the LLAP
daemon down, if it receives an exception .. is that incorrect logic?
{code}
} else {
LOG.error("Thread {} threw an Exception. Shutting down now...", t, e);
ExitUtil.terminate(-1);
}
{code}
> LLAP crash due to unhandled exception: Cannot invoke unregister on an entity
> which has not been registered
> --
>
> Key: HIVE-23061
> URL: https://issues.apache.org/jira/browse/HIVE-23061
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-23061.1.patch
>
>
> The following exception goes uncaught and causes the entire LLAP daemon to
> shut down:
> {noformat}
> 2020-03-17T06:49:11,304 ERROR [ExecutionCompletionThread #0 ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread
> Thread[ExecutionCompletionThread #0,5,main] threw an Exception. Shutting down
> now...
> java.lang.IllegalStateException: Cannot invoke unregister on an entity which
> has not been registered
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:508)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo$FinishableStateTracker.unregisterForUpdates(QueryInfo.java:256)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo.unregisterFinishableStateUpdate(QueryInfo.java:209)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryFragmentInfo.unregisterForFinishableStateUpdates(QueryFragmentInfo.java:166)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$TaskWrapper.maybeUnregisterForFinishedStateNotifications(TaskExecutorService.java:1177)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$InternalCompletionListener.onSuccess(TaskExecutorService.java:980)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$InternalCompletionListener.onSuccess(TaskExecutorService.java:944)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1021)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_191]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_191]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23061) LLAP crash due to unhandled exception: Cannot invoke unregister on an entity which has not been registered
[
https://issues.apache.org/jira/browse/HIVE-23061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063670#comment-17063670
]
Prasanth Jayachandran commented on HIVE-23061:
--
some minor comments:
1) When exception is caught and ignored, use LOG.warn() ? Or explicitly saying
"Ignoring"..
2) The case where we do do nested try.. do we know why it throws exception?
3) LLAP daemon should already has
Thread.setDefaultUncaughtExceptionHandler(new
LlapDaemonUncaughtExceptionHandler())
This should have caught any unhandled exceptions. Any idea why it wasn't
caught? May be uncaught exception handler is a better way to handle any
uncaught exceptions.
> LLAP crash due to unhandled exception: Cannot invoke unregister on an entity
> which has not been registered
> --
>
> Key: HIVE-23061
> URL: https://issues.apache.org/jira/browse/HIVE-23061
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-23061.1.patch
>
>
> The following exception goes uncaught and causes the entire LLAP daemon to
> shut down:
> {noformat}
> 2020-03-17T06:49:11,304 ERROR [ExecutionCompletionThread #0 ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread
> Thread[ExecutionCompletionThread #0,5,main] threw an Exception. Shutting down
> now...
> java.lang.IllegalStateException: Cannot invoke unregister on an entity which
> has not been registered
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:508)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo$FinishableStateTracker.unregisterForUpdates(QueryInfo.java:256)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo.unregisterFinishableStateUpdate(QueryInfo.java:209)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryFragmentInfo.unregisterForFinishableStateUpdates(QueryFragmentInfo.java:166)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$TaskWrapper.maybeUnregisterForFinishedStateNotifications(TaskExecutorService.java:1177)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$InternalCompletionListener.onSuccess(TaskExecutorService.java:980)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$InternalCompletionListener.onSuccess(TaskExecutorService.java:944)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1021)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_191]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_191]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23018) Provide a bulk API to fire multiple insert events
[ https://issues.apache.org/jira/browse/HIVE-23018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063666#comment-17063666 ] Vihang Karajgaonkar commented on HIVE-23018: Hi [~ngangam] [~ychena] Can you please take a look? > Provide a bulk API to fire multiple insert events > - > > Key: HIVE-23018 > URL: https://issues.apache.org/jira/browse/HIVE-23018 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-23018.01.nothriftpatch, HIVE-23018.01.patch > > > Metastore provides a API to fire a listener event (currently only supports > INSERT event). The problem with that API is that it only takes in one > partition at a time. A typical query may insert data into multiple partitions > at a time. In such a case query engines like HS2 or Impala will have to issue > multiple RPCs to metastore sequentially to fire these events. This can show > up as a slowdown to the user if the query engines do not return the prompt to > the user until all the events are fired (In case of HS2 and Impala). It would > be great if we have bulk API which takes in multiple partitions for a table > so that metastore can generate many such events in one RPC. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063665#comment-17063665
]
Hive QA commented on HIVE-23052:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
11s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 34 new + 547 unchanged - 37 fixed = 581 total (was 584) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21200/dev-support/hive-personality.sh
|
| git revision | master / ef99369 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21200/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21200/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimize lock enqueueing in TxnHandler
> --
>
> Key: HIVE-23052
> URL: https://issues.apache.org/jira/browse/HIVE-23052
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23052.1.patch, HIVE-23052.2.patch
>
>
> TxnHandler::enqueueLockWithRetry uses select for update (S4U) to retrieve the
> next lock id from the metastore. This locks the NEXT_LOCK_ID table for
> concurrent lock requests until the transaction has been committed.
> Since this creates contention, we should reduce the scope of this S4U,
> thereby locking the table for only as long as necessary. Currently, within
> the same txn, after the initial S4U, we also insert the transaction
> components (TXN_COMPONENTS table) before committing, even though they are
> independent and do not make use of the lock id. So moving these txn_component
> inserts before the S4U would decrease the time the next_lock_id table is
> locked and thus increase concurrency.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23062) Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue
[ https://issues.apache.org/jira/browse/HIVE-23062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063660#comment-17063660 ] Jason Dere commented on HIVE-23062: --- +1 pending test resuts > Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue > -- > > Key: HIVE-23062 > URL: https://issues.apache.org/jira/browse/HIVE-23062 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-23062.1.patch > > > Currently if custom Tez queue is used, Hive will only check the Http port, so > it is not handling TLS and Yarn HA mode URL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23037) Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
[
https://issues.apache.org/jira/browse/HIVE-23037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063655#comment-17063655
]
Hive QA commented on HIVE-23037:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997236/HIVE-23037.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18123 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21199/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21199/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21199/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997236 - PreCommit-HIVE-Build
> Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
> -
>
> Key: HIVE-23037
> URL: https://issues.apache.org/jira/browse/HIVE-23037
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23037.1.patch, HIVE-23037.2.patch
>
>
> Was recently trying to troubleshoot an issue and all I got for a log
> statement was:
> {code:none}
> Failed to get files with ID; using regular API: null
> {code}
> No stacktrace, no error message, nothing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23018) Provide a bulk API to fire multiple insert events
[ https://issues.apache.org/jira/browse/HIVE-23018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-23018: --- Attachment: HIVE-23018.01.nothriftpatch > Provide a bulk API to fire multiple insert events > - > > Key: HIVE-23018 > URL: https://issues.apache.org/jira/browse/HIVE-23018 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-23018.01.nothriftpatch, HIVE-23018.01.patch > > > Metastore provides a API to fire a listener event (currently only supports > INSERT event). The problem with that API is that it only takes in one > partition at a time. A typical query may insert data into multiple partitions > at a time. In such a case query engines like HS2 or Impala will have to issue > multiple RPCs to metastore sequentially to fire these events. This can show > up as a slowdown to the user if the query engines do not return the prompt to > the user until all the events are fired (In case of HS2 and Impala). It would > be great if we have bulk API which takes in multiple partitions for a table > so that metastore can generate many such events in one RPC. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23018) Provide a bulk API to fire multiple insert events
[ https://issues.apache.org/jira/browse/HIVE-23018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063654#comment-17063654 ] Vihang Karajgaonkar commented on HIVE-23018: attached the no-thrift patch as well for easier review. > Provide a bulk API to fire multiple insert events > - > > Key: HIVE-23018 > URL: https://issues.apache.org/jira/browse/HIVE-23018 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-23018.01.nothriftpatch, HIVE-23018.01.patch > > > Metastore provides a API to fire a listener event (currently only supports > INSERT event). The problem with that API is that it only takes in one > partition at a time. A typical query may insert data into multiple partitions > at a time. In such a case query engines like HS2 or Impala will have to issue > multiple RPCs to metastore sequentially to fire these events. This can show > up as a slowdown to the user if the query engines do not return the prompt to > the user until all the events are fired (In case of HS2 and Impala). It would > be great if we have bulk API which takes in multiple partitions for a table > so that metastore can generate many such events in one RPC. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23018) Provide a bulk API to fire multiple insert events
[
https://issues.apache.org/jira/browse/HIVE-23018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063650#comment-17063650
]
Vihang Karajgaonkar commented on HIVE-23018:
Attached the first version of the patch. The patch adds new fields to the
{{InsertEventRequestData}} and {{FireEventRequestData}} to optionally give the
ability to clients to send multiple {{InsertEventRequestData}} for individual
partitions in one RPC call.
> Provide a bulk API to fire multiple insert events
> -
>
> Key: HIVE-23018
> URL: https://issues.apache.org/jira/browse/HIVE-23018
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-23018.01.patch
>
>
> Metastore provides a API to fire a listener event (currently only supports
> INSERT event). The problem with that API is that it only takes in one
> partition at a time. A typical query may insert data into multiple partitions
> at a time. In such a case query engines like HS2 or Impala will have to issue
> multiple RPCs to metastore sequentially to fire these events. This can show
> up as a slowdown to the user if the query engines do not return the prompt to
> the user until all the events are fired (In case of HS2 and Impala). It would
> be great if we have bulk API which takes in multiple partitions for a table
> so that metastore can generate many such events in one RPC.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23018) Provide a bulk API to fire multiple insert events
[ https://issues.apache.org/jira/browse/HIVE-23018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-23018: --- Attachment: HIVE-23018.01.patch > Provide a bulk API to fire multiple insert events > - > > Key: HIVE-23018 > URL: https://issues.apache.org/jira/browse/HIVE-23018 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-23018.01.patch > > > Metastore provides a API to fire a listener event (currently only supports > INSERT event). The problem with that API is that it only takes in one > partition at a time. A typical query may insert data into multiple partitions > at a time. In such a case query engines like HS2 or Impala will have to issue > multiple RPCs to metastore sequentially to fire these events. This can show > up as a slowdown to the user if the query engines do not return the prompt to > the user until all the events are fired (In case of HS2 and Impala). It would > be great if we have bulk API which takes in multiple partitions for a table > so that metastore can generate many such events in one RPC. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23018) Provide a bulk API to fire multiple insert events
[ https://issues.apache.org/jira/browse/HIVE-23018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-23018: --- Summary: Provide a bulk API to fire multiple insert events (was: Provide a bulk API to fire multiple listener events) > Provide a bulk API to fire multiple insert events > - > > Key: HIVE-23018 > URL: https://issues.apache.org/jira/browse/HIVE-23018 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > > Metastore provides a API to fire a listener event (currently only supports > INSERT event). The problem with that API is that it only takes in one > partition at a time. A typical query may insert data into multiple partitions > at a time. In such a case query engines like HS2 or Impala will have to issue > multiple RPCs to metastore sequentially to fire these events. This can show > up as a slowdown to the user if the query engines do not return the prompt to > the user until all the events are fired (In case of HS2 and Impala). It would > be great if we have bulk API which takes in multiple partitions for a table > so that metastore can generate many such events in one RPC. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23061) LLAP crash due to unhandled exception: Cannot invoke unregister on an entity which has not been registered
[
https://issues.apache.org/jira/browse/HIVE-23061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-23061:
--
Status: Patch Available (was: Open)
Patch to prevent an exception in
InternalCompletionListener.onSuccess()/onFailure() from unwinding the stack and
failing the entire LLAP application.
[~prasanthj] [~thejas], do you mind reviewing?
> LLAP crash due to unhandled exception: Cannot invoke unregister on an entity
> which has not been registered
> --
>
> Key: HIVE-23061
> URL: https://issues.apache.org/jira/browse/HIVE-23061
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-23061.1.patch
>
>
> The following exception goes uncaught and causes the entire LLAP daemon to
> shut down:
> {noformat}
> 2020-03-17T06:49:11,304 ERROR [ExecutionCompletionThread #0 ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread
> Thread[ExecutionCompletionThread #0,5,main] threw an Exception. Shutting down
> now...
> java.lang.IllegalStateException: Cannot invoke unregister on an entity which
> has not been registered
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:508)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo$FinishableStateTracker.unregisterForUpdates(QueryInfo.java:256)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo.unregisterFinishableStateUpdate(QueryInfo.java:209)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryFragmentInfo.unregisterForFinishableStateUpdates(QueryFragmentInfo.java:166)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$TaskWrapper.maybeUnregisterForFinishedStateNotifications(TaskExecutorService.java:1177)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$InternalCompletionListener.onSuccess(TaskExecutorService.java:980)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$InternalCompletionListener.onSuccess(TaskExecutorService.java:944)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1021)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_191]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_191]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23061) LLAP crash due to unhandled exception: Cannot invoke unregister on an entity which has not been registered
[
https://issues.apache.org/jira/browse/HIVE-23061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-23061:
--
Attachment: HIVE-23061.1.patch
> LLAP crash due to unhandled exception: Cannot invoke unregister on an entity
> which has not been registered
> --
>
> Key: HIVE-23061
> URL: https://issues.apache.org/jira/browse/HIVE-23061
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-23061.1.patch
>
>
> The following exception goes uncaught and causes the entire LLAP daemon to
> shut down:
> {noformat}
> 2020-03-17T06:49:11,304 ERROR [ExecutionCompletionThread #0 ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread
> Thread[ExecutionCompletionThread #0,5,main] threw an Exception. Shutting down
> now...
> java.lang.IllegalStateException: Cannot invoke unregister on an entity which
> has not been registered
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:508)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo$FinishableStateTracker.unregisterForUpdates(QueryInfo.java:256)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo.unregisterFinishableStateUpdate(QueryInfo.java:209)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryFragmentInfo.unregisterForFinishableStateUpdates(QueryFragmentInfo.java:166)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$TaskWrapper.maybeUnregisterForFinishedStateNotifications(TaskExecutorService.java:1177)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$InternalCompletionListener.onSuccess(TaskExecutorService.java:980)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$InternalCompletionListener.onSuccess(TaskExecutorService.java:944)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1021)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_191]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_191]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23037) Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
[
https://issues.apache.org/jira/browse/HIVE-23037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063640#comment-17063640
]
Hive QA commented on HIVE-23037:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
19s{color} | {color:blue} shims/common in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21199/dev-support/hive-personality.sh
|
| git revision | master / ef99369 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: shims/common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21199/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
> -
>
> Key: HIVE-23037
> URL: https://issues.apache.org/jira/browse/HIVE-23037
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23037.1.patch, HIVE-23037.2.patch
>
>
> Was recently trying to troubleshoot an issue and all I got for a log
> statement was:
> {code:none}
> Failed to get files with ID; using regular API: null
> {code}
> No stacktrace, no error message, nothing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23062) Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue
[ https://issues.apache.org/jira/browse/HIVE-23062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-23062: -- Attachment: HIVE-23062.1.patch Status: Patch Available (was: In Progress) > Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue > -- > > Key: HIVE-23062 > URL: https://issues.apache.org/jira/browse/HIVE-23062 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-23062.1.patch > > > Currently if custom Tez queue is used, Hive will only check the Http port, so > it is not handling TLS and Yarn HA mode URL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23062) Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue
[ https://issues.apache.org/jira/browse/HIVE-23062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23062 started by Sam An. - > Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue > -- > > Key: HIVE-23062 > URL: https://issues.apache.org/jira/browse/HIVE-23062 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > > Currently if custom Tez queue is used, Hive will only check the Http port, so > it is not handling TLS and Yarn HA mode URL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23045) Zookeeper SSL/TLS support
[
https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063630#comment-17063630
]
Hive QA commented on HIVE-23045:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997225/HIVE-23045.8.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 18128 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeaderEmptyConfig.testHouseKeepingThreadExistence
(batchId=252)
org.apache.hadoop.hive.ql.TestCreateUdfEntities.testUdfWithDfsResource
(batchId=264)
org.apache.hadoop.hive.ql.TestCreateUdfEntities.testUdfWithLocalResource
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestScheduledReplicationScenarios.testAcidTablesReplLoadBootstrapIncr
(batchId=283)
org.apache.hive.jdbc.TestServiceDiscoveryWithMiniHS2.testConnectionWithConfigsPublished
(batchId=296)
org.apache.hive.jdbc.TestServiceDiscoveryWithMiniHS2.testConnectionWithoutConfigsPublished
(batchId=296)
org.apache.hive.jdbc.TestServiceDiscoveryWithMiniHS2.testGetAllUrlsDirect
(batchId=296)
org.apache.hive.jdbc.TestServiceDiscoveryWithMiniHS2.testGetAllUrlsZk
(batchId=296)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21198/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21198/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21198/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997225 - PreCommit-HIVE-Build
> Zookeeper SSL/TLS support
> -
>
> Key: HIVE-23045
> URL: https://issues.apache.org/jira/browse/HIVE-23045
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, JDBC, Metastore
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-23045.1.patch, HIVE-23045.2.patch,
> HIVE-23045.3.patch, HIVE-23045.4.patch, HIVE-23045.5.patch,
> HIVE-23045.6.patch, HIVE-23045.7.patch, HIVE-23045.8.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Zookeeper 3.5.5 server can operate with SSL/TLS secure connection with its
> clients.
> [https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide]
> The SSL communication should be possible in the different part of HIVE, where
> it communicates with Zookeeper servers. The Zookeeper clients are used in the
> following places:
> * HiveServer2 PrivilegeSynchronizer
> * HiveServer2 register/remove server from Zookeeper
> * HS2ActivePassiveHARegistryClient
> * ZooKeeperHiveLockManager
> * LLapZookeeperRegistryImpl
> * TezAmRegistryImpl
> * WebHCat ZooKeeperStorage
> * JDBC Driver server lookup
> * Metastore - ZookeeperTokenStore
> * Metastore register/remove server from Zookeeper
> The flag to enable SSL communication and the required parameters should be
> provided by different configuration parameters, corresponding the different
> use cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23062) Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue
[ https://issues.apache.org/jira/browse/HIVE-23062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-23062: - > Hive to check Yarn RM URL in TLS and Yarn HA mode for custom Tez queue > -- > > Key: HIVE-23062 > URL: https://issues.apache.org/jira/browse/HIVE-23062 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Sam An >Assignee: Sam An >Priority: Major > > Currently if custom Tez queue is used, Hive will only check the Http port, so > it is not handling TLS and Yarn HA mode URL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23045) Zookeeper SSL/TLS support
[
https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063624#comment-17063624
]
Hive QA commented on HIVE-23045:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
33s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
37s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} llap-client in master has 27 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
14s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} service in master has 50 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} hcatalog/webhcat/svr in master has 96 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} llap-client: The patch generated 0 new + 38
unchanged - 5 fixed = 38 total (was 43) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 36 unchanged - 1 fixed = 36 total (was 37) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} ql: The patch generated 0 new + 7 unchanged - 3
fixed = 7 total (was 10) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} service: The patch generated 0 new + 37 unchanged -
1 fixed = 37 total (was 38) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch jdbc passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} hcatalog/webhcat/svr: The patch generated 0 new + 19
unchanged - 2 fixed = 19 total (was 21) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red}
[jira] [Comment Edited] (HIVE-8123) Support parquet ACID
[ https://issues.apache.org/jira/browse/HIVE-8123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063607#comment-17063607 ] Vipin Vishvkarma edited comment on HIVE-8123 at 3/20/20, 8:06 PM: -- We have picked up the work and progressed a bit on this. Here is a design doc [[https://docs.google.com/document/d/19XQ3W-jyXP2M_94ltAeIc3Gdgum2oxI-CikRsuItcr0/edit#]], feel free to comment on the same. Will upload a WIP patch in the next few days. cc: [~gates] [~gopalv] [~pvary] was (Author: vpnvishv): We have picked up the work and progressed a bit on this. Here is a [design doc | [https://docs.google.com/document/d/19XQ3W-jyXP2M_94ltAeIc3Gdgum2oxI-CikRsuItcr0/edit#]], feel free to comment on the same. Will upload a WIP patch in the next few days. cc: [~gates] [~gopalv] [~pvary] > Support parquet ACID > > > Key: HIVE-8123 > URL: https://issues.apache.org/jira/browse/HIVE-8123 > Project: Hive > Issue Type: Task >Reporter: Brock Noland >Assignee: Ferdinand Xu >Priority: Major > > Hive "ACID" work currently only works with ORC. It should work with Parquet > as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-8123) Support parquet ACID
[ https://issues.apache.org/jira/browse/HIVE-8123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063607#comment-17063607 ] Vipin Vishvkarma edited comment on HIVE-8123 at 3/20/20, 8:05 PM: -- We have picked up the work and progressed a bit on this. Here is a [design doc | [https://docs.google.com/document/d/19XQ3W-jyXP2M_94ltAeIc3Gdgum2oxI-CikRsuItcr0/edit#]], feel free to comment on the same. Will upload a WIP patch in the next few days. cc: [~gates] [~gopalv] [~pvary] was (Author: vpnvishv): We have picked up the work and progressed a bit on this. Here is a [design doc|[https://docs.google.com/document/d/19XQ3W-jyXP2M_94ltAeIc3Gdgum2oxI-CikRsuItcr0/edit#]], feel free to comment on the same. Will upload a WIP patch in the next few days. cc: [~gates] [~gopalv] [~pvary] > Support parquet ACID > > > Key: HIVE-8123 > URL: https://issues.apache.org/jira/browse/HIVE-8123 > Project: Hive > Issue Type: Task >Reporter: Brock Noland >Assignee: Ferdinand Xu >Priority: Major > > Hive "ACID" work currently only works with ORC. It should work with Parquet > as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-8123) Support parquet ACID
[ https://issues.apache.org/jira/browse/HIVE-8123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063607#comment-17063607 ] Vipin Vishvkarma edited comment on HIVE-8123 at 3/20/20, 8:04 PM: -- We have picked up the work and progressed a bit on this. Here is a [design doc|[https://docs.google.com/document/d/19XQ3W-jyXP2M_94ltAeIc3Gdgum2oxI-CikRsuItcr0/edit#]], feel free to comment on the same. Will upload a WIP patch in the next few days. cc: [~gates] [~gopalv] [~pvary] was (Author: vpnvishv): We have picked up the work and progressed a bit on this. Here is a [design doc|[https://docs.google.com/document/d/19XQ3W-jyXP2M_94ltAeIc3Gdgum2oxI-CikRsuItcr0/edit#]], feel free to comment on the same. Will upload a WIP patch in the next few days. cc: [~gates] [~gopalv] [~pvary] > Support parquet ACID > > > Key: HIVE-8123 > URL: https://issues.apache.org/jira/browse/HIVE-8123 > Project: Hive > Issue Type: Task >Reporter: Brock Noland >Assignee: Ferdinand Xu >Priority: Major > > Hive "ACID" work currently only works with ORC. It should work with Parquet > as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-8123) Support parquet ACID
[ https://issues.apache.org/jira/browse/HIVE-8123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063607#comment-17063607 ] Vipin Vishvkarma commented on HIVE-8123: We have picked up the work and progressed a bit on this. Here is a [design doc|[https://docs.google.com/document/d/19XQ3W-jyXP2M_94ltAeIc3Gdgum2oxI-CikRsuItcr0/edit#]], feel free to comment on the same. Will upload a WIP patch in the next few days. cc: [~gates] [~gopalv] [~pvary] > Support parquet ACID > > > Key: HIVE-8123 > URL: https://issues.apache.org/jira/browse/HIVE-8123 > Project: Hive > Issue Type: Task >Reporter: Brock Noland >Assignee: Ferdinand Xu >Priority: Major > > Hive "ACID" work currently only works with ORC. It should work with Parquet > as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23057) ColumnStatsMergerFactory NPE Possible
[
https://issues.apache.org/jira/browse/HIVE-23057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063602#comment-17063602
]
Hive QA commented on HIVE-23057:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997219/HIVE-23057.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18123 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21197/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21197/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21197/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997219 - PreCommit-HIVE-Build
> ColumnStatsMergerFactory NPE Possible
> -
>
> Key: HIVE-23057
> URL: https://issues.apache.org/jira/browse/HIVE-23057
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23057.1.patch
>
>
> {code:java|title=ColumnStatsMergerFactory.java}
> // make sure that they have the same type
> typeNew = typeNew == typeOld ? typeNew : null;
> switch (typeNew) {
> case BOOLEAN_STATS:
> {code}
> This will throw a NPE if 'typeNew' does not equal 'typeOld'. Better to throw
> an Exception with a helpful error message here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063595#comment-17063595 ] Zoltan Haindrich commented on HIVE-22126: - [~euigeun_chung]: I wanted like to attribute the commit to you - but I can't find your email address. Could you share it with me? > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Eugene Chung >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch, > HIVE-22126.03.patch, HIVE-22126.04.patch, HIVE-22126.05.patch, > HIVE-22126.06.patch, HIVE-22126.07.patch, HIVE-22126.08.patch, > HIVE-22126.09.patch, HIVE-22126.09.patch, HIVE-22126.09.patch, > HIVE-22126.09.patch, HIVE-22126.09.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-21806) Renaming a table to another db does not handle Impala stats in partitoned tables
[
https://issues.apache.org/jira/browse/HIVE-21806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csaba Ringhofer resolved HIVE-21806.
Resolution: Fixed
I checked it recent Impala master and the issue no longer comes - the stats are
properly migrated and the drop works well. I am not sure which changed solved
it.
> Renaming a table to another db does not handle Impala stats in partitoned
> tables
>
>
> Key: HIVE-21806
> URL: https://issues.apache.org/jira/browse/HIVE-21806
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Standalone Metastore
>Affects Versions: 3.1.0
>Reporter: Csaba Ringhofer
>Assignee: Thejas Nair
>Priority: Major
>
> The issue seems similar to HIVE-9720, but only happens with partitioned
> tables. I only tested with Hive 3.1 in Impala dev environment.
> The issue can be reproduced by the following statements (it doesn't matter
> whether Hive or Impala runs them with the exception of compute/show stats).
> {code}
> create database db1;
> create database db2;
> create table db1.t(i int) partitioned by (p int);
> insert into db1.t partition (p=1) values (2);
> compute stats db1.t; -- needs Impala
> alter table db1.t rename to db2.t;
> show column stats db2.t; -- needs Impala, null count and ndv will be
> incorrectly -1
> drop table db2.t; -- this will hang
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23055) Issues when Repl dump and Repl load uses Hive Query Scheduler based execution
[ https://issues.apache.org/jira/browse/HIVE-23055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-23055: -- Attachment: HIVE-22997.16.patch > Issues when Repl dump and Repl load uses Hive Query Scheduler based execution > - > > Key: HIVE-23055 > URL: https://issues.apache.org/jira/browse/HIVE-23055 > Project: Hive > Issue Type: Bug >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > > When repl load and repl dump is scheduled to the the same/different scheduler > service, need to investigate and fix if required: > a) Why the time taken for repl dump operation in second test seems to be > higher than it was in the first. > b) The failed cases during scheduling the repl load operation are handled > properly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-22997: -- Attachment: HIVE-22997.16.patch > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch, > HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch, > HIVE-22997.15.patch, HIVE-22997.16.patch, HIVE-22997.2.patch, > HIVE-22997.4.patch, HIVE-22997.5.patch, HIVE-22997.6.patch, > HIVE-22997.7.patch, HIVE-22997.8.patch, HIVE-22997.9.patch > > Time Spent: 8h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23055) Issues when Repl dump and Repl load uses Hive Query Scheduler based execution
[ https://issues.apache.org/jira/browse/HIVE-23055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-23055: -- Attachment: (was: HIVE-22997.16.patch) > Issues when Repl dump and Repl load uses Hive Query Scheduler based execution > - > > Key: HIVE-23055 > URL: https://issues.apache.org/jira/browse/HIVE-23055 > Project: Hive > Issue Type: Bug >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > > When repl load and repl dump is scheduled to the the same/different scheduler > service, need to investigate and fix if required: > a) Why the time taken for repl dump operation in second test seems to be > higher than it was in the first. > b) The failed cases during scheduling the repl load operation are handled > properly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23033) MSSQL metastore schema init script doesn't initialize NOTIFICATION_SEQUENCE
[ https://issues.apache.org/jira/browse/HIVE-23033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063572#comment-17063572 ] David Lavati commented on HIVE-23033: - Thank you for the help & review [~ngangam]! :) > MSSQL metastore schema init script doesn't initialize NOTIFICATION_SEQUENCE > --- > > Key: HIVE-23033 > URL: https://issues.apache.org/jira/browse/HIVE-23033 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 3.1.0, 3.1.1, 3.1.2 >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-23033.1.patch, HIVE-23033.2.branch-3.patch, > HIVE-23033.2.patch > > > * The inital value for this table in the schema scripts was removed in > HIVE-17566: > https://github.com/apache/hive/commit/32b7abac961ca3879d23b074357f211fc7c49131#diff-3d1a4bae0d5d53c8e4ea79951ebf5eceL598 > * This was fixed in a number of scripts in HIVE-18781, but not for mssql: > https://github.com/apache/hive/commit/59483bca262880d3e7ef1b873d3c21176e9294cb#diff-4f43efd5a45cc362cb138287d90dbf82 > * This is as is since then > When using the schematool, the table gets initialized by other means. > This could be backported to all active branches for 3.x as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23057) ColumnStatsMergerFactory NPE Possible
[
https://issues.apache.org/jira/browse/HIVE-23057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063571#comment-17063571
]
Hive QA commented on HIVE-23057:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 3 unchanged - 6 fixed = 3 total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 185 unchanged - 1 fixed = 185 total (was 186) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21197/dev-support/hive-personality.sh
|
| git revision | master / 213ca2e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21197/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ColumnStatsMergerFactory NPE Possible
> -
>
> Key: HIVE-23057
> URL: https://issues.apache.org/jira/browse/HIVE-23057
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23057.1.patch
>
>
> {code:java|title=ColumnStatsMergerFactory.java}
> // make sure that they have the same type
> typeNew = typeNew == typeOld ? typeNew : null;
> switch (typeNew) {
> case BOOLEAN_STATS:
> {code}
> This will throw a NPE if 'typeNew' does not equal 'typeOld'. Better to throw
> an Exception with a helpful error message here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23033) MSSQL metastore schema init script doesn't initialize NOTIFICATION_SEQUENCE
[ https://issues.apache.org/jira/browse/HIVE-23033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-23033: - Resolution: Fixed Status: Resolved (was: Patch Available) Fixes have been merged into branch-3 and master. Thank you for the patch [~dlavati] > MSSQL metastore schema init script doesn't initialize NOTIFICATION_SEQUENCE > --- > > Key: HIVE-23033 > URL: https://issues.apache.org/jira/browse/HIVE-23033 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 3.1.0, 3.1.1, 3.1.2 >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-23033.1.patch, HIVE-23033.2.branch-3.patch, > HIVE-23033.2.patch > > > * The inital value for this table in the schema scripts was removed in > HIVE-17566: > https://github.com/apache/hive/commit/32b7abac961ca3879d23b074357f211fc7c49131#diff-3d1a4bae0d5d53c8e4ea79951ebf5eceL598 > * This was fixed in a number of scripts in HIVE-18781, but not for mssql: > https://github.com/apache/hive/commit/59483bca262880d3e7ef1b873d3c21176e9294cb#diff-4f43efd5a45cc362cb138287d90dbf82 > * This is as is since then > When using the schematool, the table gets initialized by other means. > This could be backported to all active branches for 3.x as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Attachment: HIVE-23004.13.patch Status: Patch Available (was: Open) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.10.patch, > HIVE-23004.11.patch, HIVE-23004.12.patch, HIVE-23004.13.patch, > HIVE-23004.2.patch, HIVE-23004.4.patch, HIVE-23004.6.patch, > HIVE-23004.7.patch, HIVE-23004.8.patch, HIVE-23004.9.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Status: Open (was: Patch Available) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.10.patch, > HIVE-23004.11.patch, HIVE-23004.12.patch, HIVE-23004.13.patch, > HIVE-23004.2.patch, HIVE-23004.4.patch, HIVE-23004.6.patch, > HIVE-23004.7.patch, HIVE-23004.8.patch, HIVE-23004.9.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23053) Clean Up Stats Mergers
[
https://issues.apache.org/jira/browse/HIVE-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063558#comment-17063558
]
Hive QA commented on HIVE-23053:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997217/HIVE-23053.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18123 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21196/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21196/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21196/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997217 - PreCommit-HIVE-Build
> Clean Up Stats Mergers
> --
>
> Key: HIVE-23053
> URL: https://issues.apache.org/jira/browse/HIVE-23053
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23053.1.patch, HIVE-23053.2.patch
>
>
> * DEBUG log each invocation of merge
> * Use the inherited logger properly
> * Use parameterized logging
> * Simplify code
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23061) LLAP crash due to unhandled exception: Cannot invoke unregister on an entity which has not been registered
[
https://issues.apache.org/jira/browse/HIVE-23061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere reassigned HIVE-23061:
-
> LLAP crash due to unhandled exception: Cannot invoke unregister on an entity
> which has not been registered
> --
>
> Key: HIVE-23061
> URL: https://issues.apache.org/jira/browse/HIVE-23061
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
>
> The following exception goes uncaught and causes the entire LLAP daemon to
> shut down:
> {noformat}
> 2020-03-17T06:49:11,304 ERROR [ExecutionCompletionThread #0 ()]
> org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread
> Thread[ExecutionCompletionThread #0,5,main] threw an Exception. Shutting down
> now...
> java.lang.IllegalStateException: Cannot invoke unregister on an entity which
> has not been registered
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:508)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo$FinishableStateTracker.unregisterForUpdates(QueryInfo.java:256)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryInfo.unregisterFinishableStateUpdate(QueryInfo.java:209)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.QueryFragmentInfo.unregisterForFinishableStateUpdates(QueryFragmentInfo.java:166)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$TaskWrapper.maybeUnregisterForFinishedStateNotifications(TaskExecutorService.java:1177)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$InternalCompletionListener.onSuccess(TaskExecutorService.java:980)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$InternalCompletionListener.onSuccess(TaskExecutorService.java:944)
> ~[hive-llap-server-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.26-3]
> at
> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1021)
> ~[hive-exec-3.1.0.3.1.4.26-3.jar:3.1.0.3.1.4.32-1]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_191]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_191]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[
https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063555#comment-17063555
]
Naveen Gangam commented on HIVE-23005:
--
[~belugabehr] I am concerned that we are adding a runtime dependency on the
guava jars for the hive-jdbc jar, with the usage of Preconditions class. I
understand that customers might be using the shaded jdbc jar, but we also ship
this jar for some reason. I am concerned that this will break existing
customers using this jar.
And we could easily replace its current usage with other code. Could you please
eliminate this dependency on the next spin of the patch when you fix the
initial fetchsize when using URL with a fetchsize. Thank you
> Consider Default JDBC Fetch Size From HS2
> -
>
> Key: HIVE-23005
> URL: https://issues.apache.org/jira/browse/HIVE-23005
> Project: Hive
> Issue Type: Sub-task
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23005.1.patch, HIVE-23005.2.patch,
> HIVE-23005.3.patch, HIVE-23005.4.patch, HIVE-23005.5.patch
>
>
> # Create two fetch sizes: and "init" fetch size and a "default" fetch size.
> The "init" fetch size comes from the JDBC connection string _fetchSize_ (if
> present) and the "default" fetch size comes from
> _HIVE_SERVER2_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE_ the server response to the
> open session request.
> # When a {{Statement}} is created, its starting fetch size is the "init"
> fetch size (may be 0)
> # Manually setting the fetch size on the {{Statement}} to 0, sets the fetch
> size to be the server default
> Setting to zero defaults to the server's instructed default and adheres to:
> bq. If the value specified is zero, then the hint is ignored. The default
> value is zero.
> That is to say, if the fetch size is 0, the default from the server is used,
> otherwise the user can pass a 'hint' and that will be the number of rows
> fetched instead.
> https://docs.oracle.com/javase/8/docs/api/java/sql/Statement.html#setFetchSize-int-
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23059) In constraint name uniqueness query use the MTable instead of it's id
[ https://issues.apache.org/jira/browse/HIVE-23059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063536#comment-17063536 ] Jesus Camacho Rodriguez commented on HIVE-23059: +1 (pending tests) > In constraint name uniqueness query use the MTable instead of it's id > - > > Key: HIVE-23059 > URL: https://issues.apache.org/jira/browse/HIVE-23059 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23059.01.patch > > > The constraint name uniqueness query uses the MTable instead of it's id. This > will solve the immediate problem of not being able to create the sys db in > Oracle, but not solve the same problem about the rest of the queries with > integer number parameters (Long, Integer, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23059) In constraint name uniqueness query use the MTable instead of it's id
[ https://issues.apache.org/jira/browse/HIVE-23059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063535#comment-17063535 ] Miklos Gergely commented on HIVE-23059: --- [~kgyrtkirk], yes, that is very likely. > In constraint name uniqueness query use the MTable instead of it's id > - > > Key: HIVE-23059 > URL: https://issues.apache.org/jira/browse/HIVE-23059 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23059.01.patch > > > The constraint name uniqueness query uses the MTable instead of it's id. This > will solve the immediate problem of not being able to create the sys db in > Oracle, but not solve the same problem about the rest of the queries with > integer number parameters (Long, Integer, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23059) In constraint name uniqueness query use the MTable instead of it's id
[ https://issues.apache.org/jira/browse/HIVE-23059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23059: -- Status: Patch Available (was: Open) > In constraint name uniqueness query use the MTable instead of it's id > - > > Key: HIVE-23059 > URL: https://issues.apache.org/jira/browse/HIVE-23059 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23059.01.patch > > > The constraint name uniqueness query uses the MTable instead of it's id. This > will solve the immediate problem of not being able to create the sys db in > Oracle, but not solve the same problem about the rest of the queries with > integer number parameters (Long, Integer, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23059) In constraint name uniqueness query use the MTable instead of it's id
[ https://issues.apache.org/jira/browse/HIVE-23059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23059: -- Attachment: HIVE-23059.01.patch > In constraint name uniqueness query use the MTable instead of it's id > - > > Key: HIVE-23059 > URL: https://issues.apache.org/jira/browse/HIVE-23059 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23059.01.patch > > > The constraint name uniqueness query uses the MTable instead of it's id. This > will solve the immediate problem of not being able to create the sys db in > Oracle, but not solve the same problem about the rest of the queries with > integer number parameters (Long, Integer, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23053) Clean Up Stats Mergers
[
https://issues.apache.org/jira/browse/HIVE-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063532#comment-17063532
]
Hive QA commented on HIVE-23053:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
10s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 27 unchanged - 2 fixed = 28 total (was 29) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21196/dev-support/hive-personality.sh
|
| git revision | master / 213ca2e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21196/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21196/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean Up Stats Mergers
> --
>
> Key: HIVE-23053
> URL: https://issues.apache.org/jira/browse/HIVE-23053
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23053.1.patch, HIVE-23053.2.patch
>
>
> * DEBUG log each invocation of merge
> * Use the inherited logger properly
> * Use parameterized logging
> * Simplify code
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23059) In constraint name uniqueness query use the MTable instead of it's id
[ https://issues.apache.org/jira/browse/HIVE-23059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063522#comment-17063522 ] Zoltan Haindrich commented on HIVE-23059: - this is not the cause of HIVE-23001 ? > In constraint name uniqueness query use the MTable instead of it's id > - > > Key: HIVE-23059 > URL: https://issues.apache.org/jira/browse/HIVE-23059 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > > The constraint name uniqueness query uses the MTable instead of it's id. This > will solve the immediate problem of not being able to create the sys db in > Oracle, but not solve the same problem about the rest of the queries with > integer number parameters (Long, Integer, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23060) Query failing with error "Grouping sets expression is not in GROUP BY key. Error encountered near token"
[
https://issues.apache.org/jira/browse/HIVE-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luis E Martinez-Poblete reassigned HIVE-23060:
--
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
>
>
> Key: HIVE-23060
> URL: https://issues.apache.org/jira/browse/HIVE-23060
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Luis E Martinez-Poblete
>Assignee: mahesh kumar behera
>Priority: Major
>
> Synopsis:
> =
> Query failing with error "Grouping sets expression is not in GROUP BY key.
> Error encountered near token"
> Problem:
>
> A Hive query in a view which fails with the following error:
> Error while compiling statement: FAILED: SemanticException 35:21 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'l0_equities_region_id'
> Reproduction case:
> {noformat}
> create database test;
> create table test.case665558 (c1 string, c2 string);
> -- Working query
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> create view test.viewcase665558
> as
> select
>case
> when GROUPING__ID = 255 then `c1`
>end as `col_1`,
>case
> when GROUPING__ID = 255 then 3
>end as `col_2`,
>`c1`,
>`c2`
> from
>`test`.`case665558`
> group by
>`c1`,
>`c2`
> GROUPING SETS
>(
> (`c1`),
> (`c1`, `c2`)
>);
>
> Select * from test.viewcase665558 ;
> Error: Error while compiling statement: FAILED: SemanticException 17:1 [Error
> 10213]: Grouping sets expression is not in GROUP BY key. Error encountered
> near token 'c1' (state=42000,code=4)
> {noformat}
> The issue is because when the view is created, it adds the name of the table
> to the columns. This seems to be confusing Hive:
> {noformat}
> +-+--+
> | createtab_stmt |
> +-+--+
> | CREATE VIEW `test.viewcase665558` AS select |
> | case|
> | when GROUPING__ID = 255 then `case665558`.`c1` |
> | end as `col_1`, |
> | case|
> | when GROUPING__ID = 255 then 3 |
> | end as `col_2`, |
> | `case665558`.`c1`, |
> | `case665558`.`c2` |
> | from|
> | `test`.`case665558` |
> | group by|
> | `case665558`.`c1`, |
> | `case665558`.`c2` |
> | GROUPING SETS |
> | ( |
> | (c1), |
> | (c1, c2)|
> | ) |
> +-+--+
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23059) In constraint name uniqueness query use the MTable instead of it's id
[ https://issues.apache.org/jira/browse/HIVE-23059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-23059: - > In constraint name uniqueness query use the MTable instead of it's id > - > > Key: HIVE-23059 > URL: https://issues.apache.org/jira/browse/HIVE-23059 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > > The constraint name uniqueness query uses the MTable instead of it's id. This > will solve the immediate problem of not being able to create the sys db in > Oracle, but not solve the same problem about the rest of the queries with > integer number parameters (Long, Integer, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[
https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063521#comment-17063521
]
Hive QA commented on HIVE-22126:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 10m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} ql: The patch generated 0 new + 44 unchanged - 1
fixed = 44 total (was 45) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
4s{color} | {color:green} root: The patch generated 0 new + 50 unchanged - 1
fixed = 50 total (was 51) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch hive-blobstore passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch hive-minikdc passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch qtest-accumulo passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch qtest-kudu passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch qtest-spark passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
8s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
56s{color} | {color:red} ql generated 1 new + 1530 unchanged - 1 fixed = 1531
total (was 1531) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 14m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| |
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveAggregate.deriveRowType(RelDataTypeFactory,
RelDataType, boolean, ImmutableBitSet, List) concatenates
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[
https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063504#comment-17063504
]
Hive QA commented on HIVE-22126:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997215/HIVE-22126.09.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18123 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21195/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21195/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21195/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997215 - PreCommit-HIVE-Build
> hive-exec packaging should shade guava
> --
>
> Key: HIVE-22126
> URL: https://issues.apache.org/jira/browse/HIVE-22126
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Eugene Chung
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch,
> HIVE-22126.03.patch, HIVE-22126.04.patch, HIVE-22126.05.patch,
> HIVE-22126.06.patch, HIVE-22126.07.patch, HIVE-22126.08.patch,
> HIVE-22126.09.patch, HIVE-22126.09.patch, HIVE-22126.09.patch,
> HIVE-22126.09.patch, HIVE-22126.09.patch
>
>
> The ql/pom.xml includes complete guava library into hive-exec.jar
> https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a
> problems for downstream clients of hive which have hive-exec.jar in their
> classpath since they are pinned to the same guava version as that of hive.
> We should shade guava classes so that other components which depend on
> hive-exec can independently use a different version of guava as needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23052: -- Attachment: HIVE-23052.2.patch > Optimize lock enqueueing in TxnHandler > -- > > Key: HIVE-23052 > URL: https://issues.apache.org/jira/browse/HIVE-23052 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23052.1.patch, HIVE-23052.2.patch > > > TxnHandler::enqueueLockWithRetry uses select for update (S4U) to retrieve the > next lock id from the metastore. This locks the NEXT_LOCK_ID table for > concurrent lock requests until the transaction has been committed. > Since this creates contention, we should reduce the scope of this S4U, > thereby locking the table for only as long as necessary. Currently, within > the same txn, after the initial S4U, we also insert the transaction > components (TXN_COMPONENTS table) before committing, even though they are > independent and do not make use of the lock id. So moving these txn_component > inserts before the S4U would decrease the time the next_lock_id table is > locked and thus increase concurrency. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23037) Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
[
https://issues.apache.org/jira/browse/HIVE-23037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23037:
--
Attachment: HIVE-23037.2.patch
> Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
> -
>
> Key: HIVE-23037
> URL: https://issues.apache.org/jira/browse/HIVE-23037
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23037.1.patch, HIVE-23037.2.patch
>
>
> Was recently trying to troubleshoot an issue and all I got for a log
> statement was:
> {code:none}
> Failed to get files with ID; using regular API: null
> {code}
> No stacktrace, no error message, nothing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23037) Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
[
https://issues.apache.org/jira/browse/HIVE-23037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23037:
--
Attachment: (was: HIVE-23037.2.patch)
> Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
> -
>
> Key: HIVE-23037
> URL: https://issues.apache.org/jira/browse/HIVE-23037
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23037.1.patch, HIVE-23037.2.patch
>
>
> Was recently trying to troubleshoot an issue and all I got for a log
> statement was:
> {code:none}
> Failed to get files with ID; using regular API: null
> {code}
> No stacktrace, no error message, nothing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23037) Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
[
https://issues.apache.org/jira/browse/HIVE-23037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063475#comment-17063475
]
David Mollitor commented on HIVE-23037:
---
[~pvary] Check the new patch. Thanks! :)
> Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
> -
>
> Key: HIVE-23037
> URL: https://issues.apache.org/jira/browse/HIVE-23037
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23037.1.patch, HIVE-23037.2.patch
>
>
> Was recently trying to troubleshoot an issue and all I got for a log
> statement was:
> {code:none}
> Failed to get files with ID; using regular API: null
> {code}
> No stacktrace, no error message, nothing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23037) Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
[
https://issues.apache.org/jira/browse/HIVE-23037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23037:
--
Attachment: HIVE-23037.2.patch
> Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
> -
>
> Key: HIVE-23037
> URL: https://issues.apache.org/jira/browse/HIVE-23037
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23037.1.patch, HIVE-23037.2.patch
>
>
> Was recently trying to troubleshoot an issue and all I got for a log
> statement was:
> {code:none}
> Failed to get files with ID; using regular API: null
> {code}
> No stacktrace, no error message, nothing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23007) Do Not Consider Client Session For Default Fetch Size
[
https://issues.apache.org/jira/browse/HIVE-23007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063468#comment-17063468
]
Naveen Gangam commented on HIVE-23007:
--
LGTM. +1 on the patch. Thank you [~belugabehr]
> Do Not Consider Client Session For Default Fetch Size
> -
>
> Key: HIVE-23007
> URL: https://issues.apache.org/jira/browse/HIVE-23007
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23007.1.patch, HIVE-23007.2.patch,
> HIVE-23017.3.patch
>
>
> Right now, when a client makes a new session, it's value override's the
> server's value for {{hive.server2.thrift.resultset.default.fetch.size}}.
> This is a server-side configuration and the client should not dictate what
> the server's default (preferred) size is.
> * Remove superfluous null check
> * Use a Collection's Singleton Map instead of creating a full-sized HashMap
> * Code simplification
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22853) Allow JDBC FetchSize to Be Set in Beeline
[ https://issues.apache.org/jira/browse/HIVE-22853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063467#comment-17063467 ] Naveen Gangam commented on HIVE-22853: -- Patch looks good to me. +1. Thank you [~belugabehr] for taking over. > Allow JDBC FetchSize to Be Set in Beeline > - > > Key: HIVE-22853 > URL: https://issues.apache.org/jira/browse/HIVE-22853 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22853.2.patch, HIVE-22853.3.patch, > HIVE-22853.4.patch, HIVE-22853.5.patch, HIVE-22853.6.patch, > HIVE-22853.7.patch, HIVE-22853.patch > > > Currently beeline uses a hard coded default of 1000 rows for fetchSize. This > default value is different from what the server has set. While the beeline > user can reset the value via set command, its cumbersome to change the > workloads. > Rather it should default to the server-side value and set should be used to > override within the session. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23037) Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
[
https://issues.apache.org/jira/browse/HIVE-23037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063464#comment-17063464
]
David Mollitor commented on HIVE-23037:
---
[~pvary] Hey. Thanks for the context.
I was able to track the error to a NPE in the code. That bug is now fixed, but
I don't want to have to deal with that again.
The log is currently ERROR, so that is going to be scary no matter what,
especially when one is trying to troubleshoot an issue. That is going to catch
the eye.
How about a WARN?
Also, {{Throwable}} is not a normal thing to catch in application code.
{{Throwable}} covers {{Exception}} and {{Error}}. An {{Error}} in Java is a
very serious (fatal) error and should not be caught in the application code
like this. It should bubble up very far up the stack (and probably just cause
a JVM exit).
With this new context, maybe I should make {{UnsupportedOperationException}}
its own block. That is probably the error condition you are talking about?
{code:java}
try {
...
} catch (UnsupportedOperationException uoe) {
LOG.info("Failed to get files with ID; using regular API {}"
uoe.getMessage());
} catch (Exception e) {
LOG.error ("Error", e);
}
{code}
> Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
> -
>
> Key: HIVE-23037
> URL: https://issues.apache.org/jira/browse/HIVE-23037
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23037.1.patch
>
>
> Was recently trying to troubleshoot an issue and all I got for a log
> statement was:
> {code:none}
> Failed to get files with ID; using regular API: null
> {code}
> No stacktrace, no error message, nothing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23037) Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
[
https://issues.apache.org/jira/browse/HIVE-23037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23037:
--
Attachment: (was: HIVE-23037.1.patch)
> Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
> -
>
> Key: HIVE-23037
> URL: https://issues.apache.org/jira/browse/HIVE-23037
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23037.1.patch
>
>
> Was recently trying to troubleshoot an issue and all I got for a log
> statement was:
> {code:none}
> Failed to get files with ID; using regular API: null
> {code}
> No stacktrace, no error message, nothing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23045) Zookeeper SSL/TLS support
[
https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063448#comment-17063448
]
Hive QA commented on HIVE-23045:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997209/HIVE-23045.6.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18128 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21194/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21194/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21194/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997209 - PreCommit-HIVE-Build
> Zookeeper SSL/TLS support
> -
>
> Key: HIVE-23045
> URL: https://issues.apache.org/jira/browse/HIVE-23045
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, JDBC, Metastore
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-23045.1.patch, HIVE-23045.2.patch,
> HIVE-23045.3.patch, HIVE-23045.4.patch, HIVE-23045.5.patch,
> HIVE-23045.6.patch, HIVE-23045.7.patch, HIVE-23045.8.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Zookeeper 3.5.5 server can operate with SSL/TLS secure connection with its
> clients.
> [https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide]
> The SSL communication should be possible in the different part of HIVE, where
> it communicates with Zookeeper servers. The Zookeeper clients are used in the
> following places:
> * HiveServer2 PrivilegeSynchronizer
> * HiveServer2 register/remove server from Zookeeper
> * HS2ActivePassiveHARegistryClient
> * ZooKeeperHiveLockManager
> * LLapZookeeperRegistryImpl
> * TezAmRegistryImpl
> * WebHCat ZooKeeperStorage
> * JDBC Driver server lookup
> * Metastore - ZookeeperTokenStore
> * Metastore register/remove server from Zookeeper
> The flag to enable SSL communication and the required parameters should be
> provided by different configuration parameters, corresponding the different
> use cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23045) Zookeeper SSL/TLS support
[
https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063436#comment-17063436
]
Hive QA commented on HIVE-23045:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
33s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} llap-client in master has 27 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
16s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} service in master has 50 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} hcatalog/webhcat/svr in master has 96 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch llap-client passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 36 unchanged - 1 fixed = 36 total (was 37) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} ql: The patch generated 0 new + 7 unchanged - 3
fixed = 7 total (was 10) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} service: The patch generated 0 new + 37 unchanged -
1 fixed = 37 total (was 38) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch jdbc passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} hcatalog/webhcat/svr: The patch generated 0 new + 19
unchanged - 2 fixed = 19 total (was 21) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} itests/hive-unit: The patch
[jira] [Updated] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-23058: Attachment: HIVE_23058.patch > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
