[jira] [Commented] (HIVE-22492) Amortize lock contention due to LRFU accounting
[
https://issues.apache.org/jira/browse/HIVE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975622#comment-16975622
]
Hive QA commented on HIVE-22492:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
22s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
21s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 21s{color}
| {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} llap-server: The patch generated 1 new + 23 unchanged
- 0 fixed = 24 total (was 23) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19452/dev-support/hive-personality.sh
|
| git revision | master / af0f325 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19452/yetus/patch-mvninstall-llap-server.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19452/yetus/patch-compile-llap-server.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19452/yetus/patch-compile-llap-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19452/yetus/diff-checkstyle-llap-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19452/yetus/patch-findbugs-llap-server.txt
|
| modules | C: common llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19452/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Amortize lock contention due to LRFU accounting
> ---
>
> Key: HIVE-22492
> URL: https://issues.apache.org/jira/browse/HIVE-22492
> Project: Hive
> Issue Type: Improvement
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22492.2.patch, HIVE-22492.patch,
> llap-lock-contention.svg
>
>
> LRFU eviction policy can be a major source of contention under high load.
> This can be see on the following profiles.
> To fix this the
[jira] [Commented] (HIVE-20600) Metastore connection leak
[
https://issues.apache.org/jira/browse/HIVE-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975621#comment-16975621
]
nemonlou commented on HIVE-20600:
-
I think HIVE-21206 has fixed this unintentionally.
In this line :
[https://github.com/apache/hive/commit/8e3c5fcbf16b2688ca435bf7361da3a3d4b1a699#diff-bb53cda370d06a3656d915cea3f7bab5R187]
A simple root cause analyze:
Step 1:
run sql: insert into table t select * from ...;
Handler-thread(compile-thread): create session level metstoreclient and pass it
to background thread
Background-thread(MoveTask, etc which invokes Task.getHive):close previous
metstoreclient and create a new one(not shared session level)
Step 2: run sql again: insert into table t select * from ...;
Handler-thread(compile-thread): create session level metstoreclient and pass it
to background thread ---thus the metastore client created by MoveTask is leaked.
> Metastore connection leak
> -
>
> Key: HIVE-20600
> URL: https://issues.apache.org/jira/browse/HIVE-20600
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.3
>Reporter: Damon Cortesi
>Priority: Major
> Attachments: HIVE-20600.patch, consume_threads.py
>
>
> Within the execute method of HiveServer2, there appears to be a connection
> leak. With fairly straightforward series of INSERT statements, the connection
> count in the logs continues to increase over time. Under certain loads, this
> can also consume all underlying threads of the Hive metastore and result in
> HS2 becoming unresponsive to new connections.
> The log below is the result of some python code executing a single insert
> statement, and then looping through a series of 10 more insert statements. We
> can see there's one dangling connection left open after each execution
> leaving us with 12 open connections (11 from the execute statements + 1 from
> HS2 startup).
> {code}
> 2018-09-19T17:14:32,108 INFO [main([])]: hive.metastore
> (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore,
> current connections: 1
> 2018-09-19T17:14:48,175 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b
> HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore
> (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore,
> current connections: 2
> 2018-09-19T17:15:05,543 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 1
> 2018-09-19T17:15:05,548 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 2
> 2018-09-19T17:15:05,932 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 1
> 2018-09-19T17:15:05,935 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 2
> 2018-09-19T17:15:06,123 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 1
> 2018-09-19T17:15:06,126 INFO [HiveServer2-Background-Pool: Thread-36([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 2
> ...
> 2018-09-19T17:15:20,626 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b
> HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore
> (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore,
> current connections: 12
> 2018-09-19T17:15:21,153 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 11
> 2018-09-19T17:15:21,155 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 12
> 2018-09-19T17:15:21,306 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 11
> 2018-09-19T17:15:21,308 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to
> metastore, current connections: 12
> 2018-09-19T17:15:21,385 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to
> metastore, current connections: 11
> 2018-09-19T17:15:21,387 INFO [HiveServer2-Background-Pool: Thread-162([])]:
> hive.metastore
[jira] [Commented] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[
https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975619#comment-16975619
]
Hive QA commented on HIVE-22476:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985977/HIVE-22476.6.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17707 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=112)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19451/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19451/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19451/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985977 - PreCommit-HIVE-Build
> Hive datediff function provided inconsistent results when
> hive.fetch.task.conversion is set to none
> ---
>
> Key: HIVE-22476
> URL: https://issues.apache.org/jira/browse/HIVE-22476
> Project: Hive
> Issue Type: Bug
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22476.2.patch, HIVE-22476.3.patch,
> HIVE-22476.5.patch, HIVE-22476.6.patch
>
>
> The actual issue stems to the different date parser used by various part of
> the engine.
> Fetch task uses udfdatediff via {code}
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the
> vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}.
> This fix is meant to be not very intrusive and will add more support to the
> GenericUDFToDate by enhancing the parser.
> For the longer term will be better to use one parser for all the operators.
> Thanks [~Rajkumar Singh] for the repro example
> {code}
> create external table testdatediff(datetimecol string) stored as orc;
> insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24');
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> set hive.ferch.task.conversion=none;
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[
https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975607#comment-16975607
]
Hive QA commented on HIVE-22476:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 1544 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19451/dev-support/hive-personality.sh
|
| git revision | master / af0f325 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19451/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive datediff function provided inconsistent results when
> hive.fetch.task.conversion is set to none
> ---
>
> Key: HIVE-22476
> URL: https://issues.apache.org/jira/browse/HIVE-22476
> Project: Hive
> Issue Type: Bug
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22476.2.patch, HIVE-22476.3.patch,
> HIVE-22476.5.patch, HIVE-22476.6.patch
>
>
> The actual issue stems to the different date parser used by various part of
> the engine.
> Fetch task uses udfdatediff via {code}
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the
> vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}.
> This fix is meant to be not very intrusive and will add more support to the
> GenericUDFToDate by enhancing the parser.
> For the longer term will be better to use one parser for all the operators.
> Thanks [~Rajkumar Singh] for the repro example
> {code}
> create external table testdatediff(datetimecol string) stored as orc;
> insert into testdatediff values
[jira] [Commented] (HIVE-22464) Implement support for NULLS FIRST/LAST in TopNKeyOperator
[
https://issues.apache.org/jira/browse/HIVE-22464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975593#comment-16975593
]
Hive QA commented on HIVE-22464:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985976/HIVE-22464.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17706 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.schq.TestScheduledQueryIntegration.testScheduledQueryExecutionImpersonation
(batchId=279)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19450/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19450/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19450/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985976 - PreCommit-HIVE-Build
> Implement support for NULLS FIRST/LAST in TopNKeyOperator
> -
>
> Key: HIVE-22464
> URL: https://issues.apache.org/jira/browse/HIVE-22464
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22464.1.patch, HIVE-22464.2.patch,
> HIVE-22464.3.patch, HIVE-22464.4.patch, HIVE-22464.4.patch,
> HIVE-22464.4.patch, HIVE-22464.4.patch, HIVE-22464.4.patch,
> HIVE-22464.4.patch, HIVE-22464.4.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22464) Implement support for NULLS FIRST/LAST in TopNKeyOperator
[
https://issues.apache.org/jira/browse/HIVE-22464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975577#comment-16975577
]
Hive QA commented on HIVE-22464:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} serde in master has 199 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 1544 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} serde generated 0 new + 198 unchanged - 1 fixed =
198 total (was 199) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} ql generated 0 new + 1541 unchanged - 3 fixed = 1541
total (was 1544) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19450/dev-support/hive-personality.sh
|
| git revision | master / af0f325 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: serde ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19450/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement support for NULLS FIRST/LAST in TopNKeyOperator
> -
>
> Key: HIVE-22464
> URL: https://issues.apache.org/jira/browse/HIVE-22464
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22464.1.patch, HIVE-22464.2.patch,
> HIVE-22464.3.patch, HIVE-22464.4.patch, HIVE-22464.4.patch,
> HIVE-22464.4.patch, HIVE-22464.4.patch, HIVE-22464.4.patch,
> HIVE-22464.4.patch, HIVE-22464.4.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21917) COMPLETED_TXN_COMPONENTS table is never cleaned up unless Compactor runs
[
https://issues.apache.org/jira/browse/HIVE-21917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975574#comment-16975574
]
Hive QA commented on HIVE-21917:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985968/HIVE-21917.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17706 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testManagedPaths
(batchId=276)
org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=284)
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation
(batchId=283)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19449/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19449/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19449/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985968 - PreCommit-HIVE-Build
> COMPLETED_TXN_COMPONENTS table is never cleaned up unless Compactor runs
>
>
> Key: HIVE-21917
> URL: https://issues.apache.org/jira/browse/HIVE-21917
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0, 3.1.1
>Reporter: Craig Condit
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21917.1.patch, HIVE-21917.2.patch
>
>
> The Initiator thread in the metastore repeatedly loops over entries in the
> COMPLETED_TXN_COMPONENTS table to determine which partitions / tables might
> need to be compacted. However, entries are never removed from this table
> except by a completed Compactor run.
> In a cluster where most tables / partitions are write-once read-many, this
> results in stale entries in this table never being cleaned up. In a small
> test cluster, we have observed approximately 45k entries in this table
> (virtually equal to the number of partitions in the cluster) while < 100 of
> these tables have delta files at all. Since most of the tables will never get
> enough writes to trigger a compaction (and in fact have only ever been
> written to once), the initiator thread keeps trying to evaluate them on every
> loop.
> On this test cluster, it takes approximately 10 minutes to loop through all
> the entries and results in severe performance degradation on metastore
> operations. With the default run timing of 5 minutes, the initiator basically
> never stops running.
> On a production cluster with 2M partitions, this would be a non-starter.
> The initiator thread should proactively remove entries from
> COMPLETED_TXN_COMPONENTS when it determines that a compaction is not needed,
> so that they are not evaluated again on the next loop.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Attachment: vectorized_join.q > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: query_error.out, query_vector_explain.out, > vectorized_join.q > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Attachment: vectorized_join.q > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: query_error.out, query_vector_explain.out > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22505: -- Attachment: (was: vectorized_join.q) > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: query_error.out, query_vector_explain.out > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22505) ClassCastException caused by wrong Vectorized operator selection
[ https://issues.apache.org/jira/browse/HIVE-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis reassigned HIVE-22505: - > ClassCastException caused by wrong Vectorized operator selection > > > Key: HIVE-22505 > URL: https://issues.apache.org/jira/browse/HIVE-22505 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Critical > Attachments: query_error.out, query_vector_explain.out > > > VectorMapJoinOuterFilteredOperator does not currently support full outer > joins but using the current Vectorizer logic it can be selected when a there > is a filter involved. This can make queries fail with ClassCastException when > their data and metadata in the VectorMapJoinOuterFilteredOperator do not > match. > The query attached demonstrates the issue and the log attached shows the > java.lang.ClassCastException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22493) Scheduled Query Execution Failure in Tests
[
https://issues.apache.org/jira/browse/HIVE-22493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-22493:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
pushed to master. Thank you Slim for taking a look!
> Scheduled Query Execution Failure in Tests
> --
>
> Key: HIVE-22493
> URL: https://issues.apache.org/jira/browse/HIVE-22493
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: Zoltan Haindrich
>Priority: Critical
> Fix For: 4.0.0
>
> Attachments: HIVE-22493.01.patch
>
>
> {code:none}
> org.apache.hadoop.hive.schq.TestScheduledQueryIntegration.testScheduledQueryExecutionImpersonation
> (batchId=279)
> org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
> org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=284)
> {code}
> {code:none}
> 2019-11-12T18:11:00,181 INFO [pool-20-thread-10] HiveMetaStore.audit:
> ugi=hiveptest ip=127.0.0.1cmd=source:127.0.0.1 scheduled_query_poll
>
> 2019-11-12T18:11:00,182 INFO [pool-20-thread-10] metastore.HiveMetaStore:
> 25: Opening raw store with implementation
> class:org.apache.hadoop.hive.metastore.ObjectStore
> 2019-11-12T18:11:00,183 INFO [pool-20-thread-10]
> metastore.PersistenceManagerProvider: Updating the pmf due to property change
> 2019-11-12T18:11:00,184 ERROR [pool-20-thread-10] metastore.HiveMetaStore:
> Caught exception
> javax.jdo.JDOUserException: Cant close PersistenceManagerFactory while we
> have active transactions.
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.close(JDOPersistenceManagerFactory.java:603)
> ~[datanucleus-api-jdo-4.2.4.jar:?]
> at
> org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(PersistenceManagerProvider.java:199)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:213)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
> ~[hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
> ~[hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:59)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:852)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:820)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:814)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.scheduled_query_poll(HiveMetaStore.java:9660)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_102]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy46.scheduled_query_poll(Unknown Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$scheduled_query_poll.getResult(ThriftHiveMetastore.java:21561)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$scheduled_query_poll.getResult(ThriftHiveMetastore.java:21545)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [libthrift-0.9.3-1.jar:0.9.3-1]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
>
[jira] [Commented] (HIVE-21917) COMPLETED_TXN_COMPONENTS table is never cleaned up unless Compactor runs
[
https://issues.apache.org/jira/browse/HIVE-21917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975556#comment-16975556
]
Hive QA commented on HIVE-21917:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
16s{color} | {color:blue} standalone-metastore/metastore-server in master has
178 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 1544 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
25s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 18 new + 627 unchanged - 1 fixed = 645 total (was 628) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 1 new + 33 unchanged - 0 fixed
= 34 total (was 33) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19449/dev-support/hive-personality.sh
|
| git revision | master / 078182a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19449/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19449/yetus/diff-checkstyle-ql.txt
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19449/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> COMPLETED_TXN_COMPONENTS table is never cleaned up unless Compactor runs
>
>
> Key: HIVE-21917
> URL: https://issues.apache.org/jira/browse/HIVE-21917
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0, 3.1.1
>Reporter: Craig Condit
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21917.1.patch, HIVE-21917.2.patch
>
>
> The Initiator thread in the metastore repeatedly loops over entries in the
> COMPLETED_TXN_COMPONENTS table to determine which
[jira] [Updated] (HIVE-22329) Create DriverContext
[ https://issues.apache.org/jira/browse/HIVE-22329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22329: -- Attachment: HIVE-22329.04.patch > Create DriverContext > > > Key: HIVE-22329 > URL: https://issues.apache.org/jira/browse/HIVE-22329 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22329.01.patch, HIVE-22329.02.patch, > HIVE-22329.03.patch, HIVE-22329.04.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The current DriverContext class is not really a context, rather a queue, thus > it should be renamed to DriverQueue. At the same time Driver has a a lot of > fields, they should be in a real context class called DriverContext, which > would also facilitate the extraction of independent parts of the Driver class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22329) Create DriverContext
[ https://issues.apache.org/jira/browse/HIVE-22329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22329: -- Attachment: (was: HIVE-22329.04.patch) > Create DriverContext > > > Key: HIVE-22329 > URL: https://issues.apache.org/jira/browse/HIVE-22329 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22329.01.patch, HIVE-22329.02.patch, > HIVE-22329.03.patch, HIVE-22329.04.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The current DriverContext class is not really a context, rather a queue, thus > it should be renamed to DriverQueue. At the same time Driver has a a lot of > fields, they should be in a real context class called DriverContext, which > would also facilitate the extraction of independent parts of the Driver class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22488) Break up DDLSemanticAnalyzer - extract Table creation analyzers
[ https://issues.apache.org/jira/browse/HIVE-22488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22488: -- Attachment: HIVE-22488.01.patch > Break up DDLSemanticAnalyzer - extract Table creation analyzers > --- > > Key: HIVE-22488 > URL: https://issues.apache.org/jira/browse/HIVE-22488 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22488.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #9: extract the table creationanalyzers from DDLSemanticAnalyzer, and > move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22488) Break up DDLSemanticAnalyzer - extract Table creation analyzers
[ https://issues.apache.org/jira/browse/HIVE-22488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22488: -- Attachment: (was: HIVE-22488.01.patch) > Break up DDLSemanticAnalyzer - extract Table creation analyzers > --- > > Key: HIVE-22488 > URL: https://issues.apache.org/jira/browse/HIVE-22488 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22488.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #9: extract the table creationanalyzers from DDLSemanticAnalyzer, and > move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22461) NPE Metastore Transformer
[
https://issues.apache.org/jira/browse/HIVE-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975545#comment-16975545
]
Hive QA commented on HIVE-22461:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985965/HIVE-22461.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17706 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19447/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19447/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19447/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985965 - PreCommit-HIVE-Build
> NPE Metastore Transformer
> -
>
> Key: HIVE-22461
> URL: https://issues.apache.org/jira/browse/HIVE-22461
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-22461.1.patch, HIVE-22461.2.patch,
> HIVE-22461.3.patch, HIVE-22461.4.patch, HIVE-22461.5.patch, HIVE-22461.6.patch
>
>
> The stack looks as following:
> {noformat}
> 2019-10-08 18:09:12,198 INFO
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
> [pool-6-thread-328]: Starting translation for processor
> Hiveserver2#3.1.2000.7.0.2.0...@vc0732.halxg.cloudera.com on list 1
> 2019-10-08 18:09:12,198 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-328]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16633)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16617)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-10-08 18:09:12,199 ERROR org.apache.thrift.server.TThreadPoolServer:
> [pool-6-thread-328]: Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
>
[jira] [Updated] (HIVE-22504) Compilation time might get high in case of many parentheses (branch-1)
[ https://issues.apache.org/jira/browse/HIVE-22504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22504: Attachment: HIVE-22504.01.patch > Compilation time might get high in case of many parentheses (branch-1) > -- > > Key: HIVE-22504 > URL: https://issues.apache.org/jira/browse/HIVE-22504 > Project: Hive > Issue Type: Bug > Components: Parser >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22504.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22504) Compilation time might get high in case of many parentheses (branch-1)
[ https://issues.apache.org/jira/browse/HIVE-22504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-22504: --- > Compilation time might get high in case of many parentheses (branch-1) > -- > > Key: HIVE-22504 > URL: https://issues.apache.org/jira/browse/HIVE-22504 > Project: Hive > Issue Type: Bug > Components: Parser >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22461) NPE Metastore Transformer
[
https://issues.apache.org/jira/browse/HIVE-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975518#comment-16975518
]
Hive QA commented on HIVE-22461:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
14s{color} | {color:blue} standalone-metastore/metastore-server in master has
178 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 372 unchanged - 35 fixed = 372 total (was 407) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 177 unchanged - 1 fixed = 177 total (was 178) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19447/dev-support/hive-personality.sh
|
| git revision | master / 078182a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19447/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NPE Metastore Transformer
> -
>
> Key: HIVE-22461
> URL: https://issues.apache.org/jira/browse/HIVE-22461
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-22461.1.patch, HIVE-22461.2.patch,
> HIVE-22461.3.patch, HIVE-22461.4.patch, HIVE-22461.5.patch, HIVE-22461.6.patch
>
>
> The stack looks as following:
> {noformat}
> 2019-10-08 18:09:12,198 INFO
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
> [pool-6-thread-328]: Starting translation for processor
> Hiveserver2#3.1.2000.7.0.2.0...@vc0732.halxg.cloudera.com on list 1
> 2019-10-08 18:09:12,198 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-328]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
>
[jira] [Commented] (HIVE-22501) Stats reported multiple times during MR execution for UNION queries
[
https://issues.apache.org/jira/browse/HIVE-22501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975512#comment-16975512
]
Hive QA commented on HIVE-22501:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985961/HIVE-22501.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17702 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACIDNoAutogather - did not produce a TEST-*.xml
file (likely timed out) (batchId=256)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testManagedPaths
(batchId=276)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19446/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19446/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19446/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985961 - PreCommit-HIVE-Build
> Stats reported multiple times during MR execution for UNION queries
> ---
>
> Key: HIVE-22501
> URL: https://issues.apache.org/jira/browse/HIVE-22501
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Labels: mapreduce
> Attachments: HIVE-22501.1.patch
>
>
> Take the following example:
> {code}
> set hive.execution.engine=mr;
> create table tb(id string) stored as orc;
> insert into tb values('1');
> create table tb2 like tb stored as orc;
> insert into tb2 select * from tb union all select * from tb;
> {code}
> Last insert results in 2 records in the table, but
> {{TOTAL_TABLE_ROWS_WRITTEN}} statistic (and number of affected rows on the
> consolse) is 4.
> We seem to traverse the operator graph multiple times starting from every TS
> operator and increment the counters every time we hit the FS operator.
> UNION-ing the table 3 times results in 9 TOTAL_TABLE_ROWS_WRITTEN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22501) Stats reported multiple times during MR execution for UNION queries
[
https://issues.apache.org/jira/browse/HIVE-22501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975493#comment-16975493
]
Hive QA commented on HIVE-22501:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 1544 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19446/dev-support/hive-personality.sh
|
| git revision | master / 078182a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19446/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Stats reported multiple times during MR execution for UNION queries
> ---
>
> Key: HIVE-22501
> URL: https://issues.apache.org/jira/browse/HIVE-22501
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Labels: mapreduce
> Attachments: HIVE-22501.1.patch
>
>
> Take the following example:
> {code}
> set hive.execution.engine=mr;
> create table tb(id string) stored as orc;
> insert into tb values('1');
> create table tb2 like tb stored as orc;
> insert into tb2 select * from tb union all select * from tb;
> {code}
> Last insert results in 2 records in the table, but
> {{TOTAL_TABLE_ROWS_WRITTEN}} statistic (and number of affected rows on the
> consolse) is 4.
> We seem to traverse the operator graph multiple times starting from every TS
> operator and increment the counters every time we hit the FS operator.
> UNION-ing the table 3 times results in 9 TOTAL_TABLE_ROWS_WRITTEN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22327) Repl: Ignore read-only transactions in notification log
[
https://issues.apache.org/jira/browse/HIVE-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975483#comment-16975483
]
Hive QA commented on HIVE-22327:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985955/HIVE-22327.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17707 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testManagedPaths
(batchId=276)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19445/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19445/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19445/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985955 - PreCommit-HIVE-Build
> Repl: Ignore read-only transactions in notification log
> ---
>
> Key: HIVE-22327
> URL: https://issues.apache.org/jira/browse/HIVE-22327
> Project: Hive
> Issue Type: Improvement
> Components: repl
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22327.1.patch, HIVE-22327.2.patch,
> HIVE-22327.3.patch, HIVE-22327.4.patch, HIVE-22327.5.patch
>
>
> Read txns need not be replicated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22327) Repl: Ignore read-only transactions in notification log
[
https://issues.apache.org/jira/browse/HIVE-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975453#comment-16975453
]
Hive QA commented on HIVE-22327:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
13s{color} | {color:blue} standalone-metastore/metastore-server in master has
178 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
24s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 5 new + 549 unchanged - 3 fixed = 554 total (was 552) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 177 unchanged - 1 fixed = 177 total (was 178) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19445/dev-support/hive-personality.sh
|
| git revision | master / 078182a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19445/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19445/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Repl: Ignore read-only transactions in notification log
> ---
>
> Key: HIVE-22327
> URL: https://issues.apache.org/jira/browse/HIVE-22327
> Project: Hive
> Issue Type: Improvement
> Components: repl
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22327.1.patch, HIVE-22327.2.patch,
> HIVE-22327.3.patch, HIVE-22327.4.patch, HIVE-22327.5.patch
>
>
> Read txns need not be replicated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22488) Break up DDLSemanticAnalyzer - extract Table creation analyzers
[
https://issues.apache.org/jira/browse/HIVE-22488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975434#comment-16975434
]
Hive QA commented on HIVE-22488:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985954/HIVE-22488.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 17707 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.org.apache.hadoop.hive.ql.TestWarehouseExternalDir
(batchId=276)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testExternalDefaultPaths
(batchId=276)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testManagedPaths
(batchId=276)
org.apache.hadoop.hive.schq.TestScheduledQueryIntegration.testScheduledQueryExecutionImpersonation
(batchId=279)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation4 (batchId=288)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19444/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19444/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19444/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985954 - PreCommit-HIVE-Build
> Break up DDLSemanticAnalyzer - extract Table creation analyzers
> ---
>
> Key: HIVE-22488
> URL: https://issues.apache.org/jira/browse/HIVE-22488
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Attachments: HIVE-22488.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is
> to refactor it in order to have everything cut into more handleable classes
> under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each analyzers
> * have a package for each operation, containing an analyzer, a description,
> and an operation, so the amount of classes under a package is more manageable
> Step #9: extract the table creationanalyzers from DDLSemanticAnalyzer, and
> move them under the new package.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22463) Support Decimal64 column multiplication with decimal64 Column/Scalar
[ https://issues.apache.org/jira/browse/HIVE-22463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22463: Status: Open (was: Patch Available) > Support Decimal64 column multiplication with decimal64 Column/Scalar > > > Key: HIVE-22463 > URL: https://issues.apache.org/jira/browse/HIVE-22463 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22463.1.patch, HIVE-22463.2.patch, > HIVE-22463.3.patch, HIVE-22463.5.patch > > > Support Decimal64 column multiplication with decimal64 Column/Scalar -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22463) Support Decimal64 column multiplication with decimal64 Column/Scalar
[ https://issues.apache.org/jira/browse/HIVE-22463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22463: Attachment: HIVE-22463.5.patch Status: Patch Available (was: Open) > Support Decimal64 column multiplication with decimal64 Column/Scalar > > > Key: HIVE-22463 > URL: https://issues.apache.org/jira/browse/HIVE-22463 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22463.1.patch, HIVE-22463.2.patch, > HIVE-22463.3.patch, HIVE-22463.5.patch > > > Support Decimal64 column multiplication with decimal64 Column/Scalar -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22488) Break up DDLSemanticAnalyzer - extract Table creation analyzers
[
https://issues.apache.org/jira/browse/HIVE-22488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975420#comment-16975420
]
Hive QA commented on HIVE-22488:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
17s{color} | {color:blue} ql in master has 1544 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} hcatalog/core in master has 36 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
48s{color} | {color:blue} itests/util in master has 53 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} ql generated 0 new + 1543 unchanged - 1 fixed = 1543
total (was 1544) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} util in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19444/dev-support/hive-personality.sh
|
| git revision | master / 078182a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql hcatalog/core itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19444/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Break up DDLSemanticAnalyzer - extract Table creation analyzers
> ---
>
> Key: HIVE-22488
> URL: https://issues.apache.org/jira/browse/HIVE-22488
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Attachments: HIVE-22488.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is
> to refactor it in order to have everything cut into more handleable classes
> under the
[jira] [Commented] (HIVE-22280) Q tests for partitioned temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975401#comment-16975401
]
Hive QA commented on HIVE-22280:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985940/HIVE-22280.03.patch
{color:green}SUCCESS:{color} +1 due to 67 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17773 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_dynamic_partition_insert]
(batchId=52)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=277)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testManagedPaths
(batchId=278)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19443/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19443/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19443/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985940 - PreCommit-HIVE-Build
> Q tests for partitioned temporary tables
>
>
> Key: HIVE-22280
> URL: https://issues.apache.org/jira/browse/HIVE-22280
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch,
> HIVE-22280.03.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-5208) Provide an easier way to capture DEBUG logging
[ https://issues.apache.org/jira/browse/HIVE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975394#comment-16975394 ] Piyush Patel commented on HIVE-5208: Is there a similar option in Beeline? I searched a bit but didn't find any easy solution for this? At least in Hive CLI, we had --hiveconf hive.root.logger=DEBUG,console option, but it does not work for Beeline. > Provide an easier way to capture DEBUG logging > -- > > Key: HIVE-5208 > URL: https://issues.apache.org/jira/browse/HIVE-5208 > Project: Hive > Issue Type: Improvement > Components: CLI >Affects Versions: 0.11.0 >Reporter: Harsh J >Priority: Minor > > Capturing debug logging for troubleshooting is painful in Hive today: > 1. It doesn't log anywhere by default. > 2. We need to add a long "-hiveconf hive.root.logger=DEBUG,console" to the > Hive CLI just to enable the debug flag, or set an equivalent env-var > appropriately. > I suggest we make this simpler via either one of the below: > 1. Provide a wrapped binary, "hive-debug", so folks can simply re-run > the "hive-debug" command and re-run their query and capture an output. This > could also write to a pre-designated $PWD file. > 2. Provide a simpler switch, such as "-verbose" that automatically > toggles the flag instead, much like what Beeline does today already. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22493) Scheduled Query Execution Failure in Tests
[
https://issues.apache.org/jira/browse/HIVE-22493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975354#comment-16975354
]
Slim Bouguerra commented on HIVE-22493:
---
+1
> Scheduled Query Execution Failure in Tests
> --
>
> Key: HIVE-22493
> URL: https://issues.apache.org/jira/browse/HIVE-22493
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: Zoltan Haindrich
>Priority: Critical
> Attachments: HIVE-22493.01.patch
>
>
> {code:none}
> org.apache.hadoop.hive.schq.TestScheduledQueryIntegration.testScheduledQueryExecutionImpersonation
> (batchId=279)
> org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
> org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=284)
> {code}
> {code:none}
> 2019-11-12T18:11:00,181 INFO [pool-20-thread-10] HiveMetaStore.audit:
> ugi=hiveptest ip=127.0.0.1cmd=source:127.0.0.1 scheduled_query_poll
>
> 2019-11-12T18:11:00,182 INFO [pool-20-thread-10] metastore.HiveMetaStore:
> 25: Opening raw store with implementation
> class:org.apache.hadoop.hive.metastore.ObjectStore
> 2019-11-12T18:11:00,183 INFO [pool-20-thread-10]
> metastore.PersistenceManagerProvider: Updating the pmf due to property change
> 2019-11-12T18:11:00,184 ERROR [pool-20-thread-10] metastore.HiveMetaStore:
> Caught exception
> javax.jdo.JDOUserException: Cant close PersistenceManagerFactory while we
> have active transactions.
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.close(JDOPersistenceManagerFactory.java:603)
> ~[datanucleus-api-jdo-4.2.4.jar:?]
> at
> org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(PersistenceManagerProvider.java:199)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:213)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
> ~[hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
> ~[hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:59)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:852)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:820)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:814)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.scheduled_query_poll(HiveMetaStore.java:9660)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_102]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy46.scheduled_query_poll(Unknown Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$scheduled_query_poll.getResult(ThriftHiveMetastore.java:21561)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$scheduled_query_poll.getResult(ThriftHiveMetastore.java:21545)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [libthrift-0.9.3-1.jar:0.9.3-1]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_102]
> at
[jira] [Commented] (HIVE-22280) Q tests for partitioned temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975346#comment-16975346
]
Hive QA commented on HIVE-22280:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 3m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19443/dev-support/hive-personality.sh
|
| git revision | master / 078182a |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19443/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Q tests for partitioned temporary tables
>
>
> Key: HIVE-22280
> URL: https://issues.apache.org/jira/browse/HIVE-22280
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch,
> HIVE-22280.03.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22066) Upgrade Apache parent POM to version 21
[ https://issues.apache.org/jira/browse/HIVE-22066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22066: -- Attachment: HIVE-22066.2.patch > Upgrade Apache parent POM to version 21 > --- > > Key: HIVE-22066 > URL: https://issues.apache.org/jira/browse/HIVE-22066 > Project: Hive > Issue Type: Improvement >Reporter: Jesus Camacho Rodriguez >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22066.2.patch, HIVE-22066.patch, HIVE-22066.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22066) Upgrade Apache parent POM to version 21
[ https://issues.apache.org/jira/browse/HIVE-22066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-22066: - Assignee: David Mollitor (was: Jesus Camacho Rodriguez) > Upgrade Apache parent POM to version 21 > --- > > Key: HIVE-22066 > URL: https://issues.apache.org/jira/browse/HIVE-22066 > Project: Hive > Issue Type: Improvement >Reporter: Jesus Camacho Rodriguez >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22066.2.patch, HIVE-22066.patch, HIVE-22066.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21146) Enforce TransactionBatch size=1 for blob stores
[
https://issues.apache.org/jira/browse/HIVE-21146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975341#comment-16975341
]
Hive QA commented on HIVE-21146:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985934/HIVE-21146.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17707 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19442/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19442/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19442/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985934 - PreCommit-HIVE-Build
> Enforce TransactionBatch size=1 for blob stores
> ---
>
> Key: HIVE-21146
> URL: https://issues.apache.org/jira/browse/HIVE-21146
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21146.2.patch, HIVE-21146.2.patch,
> HIVE-21146.2.patch, HIVE-21146.2.patch, HIVE-21146.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Streaming Ingest API supports a concept of {{TransactionBatch}} where N
> transactions can be opened at once and the data in all of them will be
> written to the same delta_x_y directory where each transaction in the batch
> can be committed/aborted independently. The implementation relies on
> {{FSDataOutputStream.hflush()}} (called from OrcRecordUpdater}} which is
> available on HDFS but is often implemented as no-op in Blob store backed
> {{FileSystem}} objects.
> Need to add a check to {{HiveStreamingConnection()}} constructor to raise an
> error if {{builder.transactionBatchSize > 1}} and the target table/partitions
> are backed by something that doesn't support {{hflush()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22492) Amortize lock contention due to LRFU accounting
[ https://issues.apache.org/jira/browse/HIVE-22492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22492: -- Attachment: HIVE-22492.2.patch > Amortize lock contention due to LRFU accounting > --- > > Key: HIVE-22492 > URL: https://issues.apache.org/jira/browse/HIVE-22492 > Project: Hive > Issue Type: Improvement >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22492.2.patch, HIVE-22492.patch, > llap-lock-contention.svg > > > LRFU eviction policy can be a major source of contention under high load. > This can be see on the following profiles. > To fix this the idea is to use a batching wrapper to amortize the locking > contention. > The trick i a common way to amortize locking as explained here > http://www.ece.eng.wayne.edu/~sjiang/pubs/papers/ding-09-BP-Wrapper.pdf -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22503) Harmonize JODA Time Version in Module hive-hcatalog-it-unit
[ https://issues.apache.org/jira/browse/HIVE-22503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-22503: - > Harmonize JODA Time Version in Module hive-hcatalog-it-unit > --- > > Key: HIVE-22503 > URL: https://issues.apache.org/jira/browse/HIVE-22503 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > > https://github.com/apache/hive/blob/078182ade4b76e810ca945354f4897dbe36ad5c2/itests/hcatalog-unit/pom.xml#L296 > Currently hard-coded as version 2.2 > See if we can get away with using the same version of Joda as the rest of the > Hive project. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21146) Enforce TransactionBatch size=1 for blob stores
[
https://issues.apache.org/jira/browse/HIVE-21146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975300#comment-16975300
]
Hive QA commented on HIVE-21146:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19442/dev-support/hive-personality.sh
|
| git revision | master / 078182a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: streaming U: streaming |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19442/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enforce TransactionBatch size=1 for blob stores
> ---
>
> Key: HIVE-21146
> URL: https://issues.apache.org/jira/browse/HIVE-21146
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21146.2.patch, HIVE-21146.2.patch,
> HIVE-21146.2.patch, HIVE-21146.2.patch, HIVE-21146.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Streaming Ingest API supports a concept of {{TransactionBatch}} where N
> transactions can be opened at once and the data in all of them will be
> written to the same delta_x_y directory where each transaction in the batch
> can be committed/aborted independently. The implementation relies on
> {{FSDataOutputStream.hflush()}} (called from OrcRecordUpdater}} which is
> available on HDFS but is often implemented as no-op in Blob store backed
> {{FileSystem}} objects.
> Need to add a check to {{HiveStreamingConnection()}} constructor to raise an
> error if {{builder.transactionBatchSize > 1}} and the target table/partitions
> are backed by something that doesn't support {{hflush()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22493) Scheduled Query Execution Failure in Tests
[
https://issues.apache.org/jira/browse/HIVE-22493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975294#comment-16975294
]
Zoltan Haindrich commented on HIVE-22493:
-
[~jcamachorodriguez], [~bslim]: Could you please take a look?
> Scheduled Query Execution Failure in Tests
> --
>
> Key: HIVE-22493
> URL: https://issues.apache.org/jira/browse/HIVE-22493
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: Zoltan Haindrich
>Priority: Critical
> Attachments: HIVE-22493.01.patch
>
>
> {code:none}
> org.apache.hadoop.hive.schq.TestScheduledQueryIntegration.testScheduledQueryExecutionImpersonation
> (batchId=279)
> org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
> org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=284)
> {code}
> {code:none}
> 2019-11-12T18:11:00,181 INFO [pool-20-thread-10] HiveMetaStore.audit:
> ugi=hiveptest ip=127.0.0.1cmd=source:127.0.0.1 scheduled_query_poll
>
> 2019-11-12T18:11:00,182 INFO [pool-20-thread-10] metastore.HiveMetaStore:
> 25: Opening raw store with implementation
> class:org.apache.hadoop.hive.metastore.ObjectStore
> 2019-11-12T18:11:00,183 INFO [pool-20-thread-10]
> metastore.PersistenceManagerProvider: Updating the pmf due to property change
> 2019-11-12T18:11:00,184 ERROR [pool-20-thread-10] metastore.HiveMetaStore:
> Caught exception
> javax.jdo.JDOUserException: Cant close PersistenceManagerFactory while we
> have active transactions.
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.close(JDOPersistenceManagerFactory.java:603)
> ~[datanucleus-api-jdo-4.2.4.jar:?]
> at
> org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(PersistenceManagerProvider.java:199)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:213)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
> ~[hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
> ~[hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:59)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:852)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:820)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:814)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.scheduled_query_poll(HiveMetaStore.java:9660)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_102]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy46.scheduled_query_poll(Unknown Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$scheduled_query_poll.getResult(ThriftHiveMetastore.java:21561)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$scheduled_query_poll.getResult(ThriftHiveMetastore.java:21545)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [libthrift-0.9.3-1.jar:0.9.3-1]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
[jira] [Updated] (HIVE-22502) ConcurrentModificationException in TriggerValidatorRunnable stops trigger processing
[ https://issues.apache.org/jira/browse/HIVE-22502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22502: - Description: An other thread is modifying the list the contains the sessions while TriggerValidatorRunnable is traversing it. This causes the TriggerValidatorRunnable thread to die and triggers are no longer properly monitored. (was: An other thread is modifying the list the contains the sessions while TriggerValidatorRunnable is traversing it. This cause the TriggerValidatorRunnable thread to die and triggers are no longer properly monitored.) > ConcurrentModificationException in TriggerValidatorRunnable stops trigger > processing > > > Key: HIVE-22502 > URL: https://issues.apache.org/jira/browse/HIVE-22502 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > > An other thread is modifying the list the contains the sessions while > TriggerValidatorRunnable is traversing it. This causes the > TriggerValidatorRunnable thread to die and triggers are no longer properly > monitored. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22502) ConcurrentModificationException in TriggerValidatorRunnable stops trigger processing
[ https://issues.apache.org/jira/browse/HIVE-22502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22502: - Description: An other thread is modifying the list the contains the sessions while TriggerValidatorRunnable is traversing it. This cause the TriggerValidatorRunnable thread to die and triggers are no longer properly monitored. (was: An other thread is modifying the list the contains the sessions while ) > ConcurrentModificationException in TriggerValidatorRunnable stops trigger > processing > > > Key: HIVE-22502 > URL: https://issues.apache.org/jira/browse/HIVE-22502 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > > An other thread is modifying the list the contains the sessions while > TriggerValidatorRunnable is traversing it. This cause the > TriggerValidatorRunnable thread to die and triggers are no longer properly > monitored. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22329) Create DriverContext
[
https://issues.apache.org/jira/browse/HIVE-22329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975284#comment-16975284
]
Hive QA commented on HIVE-22329:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985921/HIVE-22329.04.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17706 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19441/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19441/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19441/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985921 - PreCommit-HIVE-Build
> Create DriverContext
>
>
> Key: HIVE-22329
> URL: https://issues.apache.org/jira/browse/HIVE-22329
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22329.01.patch, HIVE-22329.02.patch,
> HIVE-22329.03.patch, HIVE-22329.04.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The current DriverContext class is not really a context, rather a queue, thus
> it should be renamed to DriverQueue. At the same time Driver has a a lot of
> fields, they should be in a real context class called DriverContext, which
> would also facilitate the extraction of independent parts of the Driver class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22502) ConcurrentModificationException in TriggerValidatorRunnable stops trigger processing
[ https://issues.apache.org/jira/browse/HIVE-22502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22502: - Description: An other thread is modifying the list the contains the sessions while > ConcurrentModificationException in TriggerValidatorRunnable stops trigger > processing > > > Key: HIVE-22502 > URL: https://issues.apache.org/jira/browse/HIVE-22502 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > > An other thread is modifying the list the contains the sessions while -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[
https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Slim Bouguerra updated HIVE-22476:
--
Status: Patch Available (was: Open)
> Hive datediff function provided inconsistent results when
> hive.fetch.task.conversion is set to none
> ---
>
> Key: HIVE-22476
> URL: https://issues.apache.org/jira/browse/HIVE-22476
> Project: Hive
> Issue Type: Bug
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22476.2.patch, HIVE-22476.3.patch,
> HIVE-22476.5.patch, HIVE-22476.6.patch
>
>
> The actual issue stems to the different date parser used by various part of
> the engine.
> Fetch task uses udfdatediff via {code}
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the
> vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}.
> This fix is meant to be not very intrusive and will add more support to the
> GenericUDFToDate by enhancing the parser.
> For the longer term will be better to use one parser for all the operators.
> Thanks [~Rajkumar Singh] for the repro example
> {code}
> create external table testdatediff(datetimecol string) stored as orc;
> insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24');
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> set hive.ferch.task.conversion=none;
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[
https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Slim Bouguerra updated HIVE-22476:
--
Attachment: HIVE-22476.6.patch
> Hive datediff function provided inconsistent results when
> hive.fetch.task.conversion is set to none
> ---
>
> Key: HIVE-22476
> URL: https://issues.apache.org/jira/browse/HIVE-22476
> Project: Hive
> Issue Type: Bug
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22476.2.patch, HIVE-22476.3.patch,
> HIVE-22476.5.patch, HIVE-22476.6.patch
>
>
> The actual issue stems to the different date parser used by various part of
> the engine.
> Fetch task uses udfdatediff via {code}
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the
> vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}.
> This fix is meant to be not very intrusive and will add more support to the
> GenericUDFToDate by enhancing the parser.
> For the longer term will be better to use one parser for all the operators.
> Thanks [~Rajkumar Singh] for the repro example
> {code}
> create external table testdatediff(datetimecol string) stored as orc;
> insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24');
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> set hive.ferch.task.conversion=none;
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[
https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Slim Bouguerra updated HIVE-22476:
--
Attachment: (was: HIVE-22476.patch)
> Hive datediff function provided inconsistent results when
> hive.fetch.task.conversion is set to none
> ---
>
> Key: HIVE-22476
> URL: https://issues.apache.org/jira/browse/HIVE-22476
> Project: Hive
> Issue Type: Bug
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22476.2.patch, HIVE-22476.3.patch,
> HIVE-22476.5.patch
>
>
> The actual issue stems to the different date parser used by various part of
> the engine.
> Fetch task uses udfdatediff via {code}
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the
> vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}.
> This fix is meant to be not very intrusive and will add more support to the
> GenericUDFToDate by enhancing the parser.
> For the longer term will be better to use one parser for all the operators.
> Thanks [~Rajkumar Singh] for the repro example
> {code}
> create external table testdatediff(datetimecol string) stored as orc;
> insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24');
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> set hive.ferch.task.conversion=none;
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[
https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Slim Bouguerra updated HIVE-22476:
--
Status: Open (was: Patch Available)
> Hive datediff function provided inconsistent results when
> hive.fetch.task.conversion is set to none
> ---
>
> Key: HIVE-22476
> URL: https://issues.apache.org/jira/browse/HIVE-22476
> Project: Hive
> Issue Type: Bug
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22476.2.patch, HIVE-22476.3.patch,
> HIVE-22476.5.patch
>
>
> The actual issue stems to the different date parser used by various part of
> the engine.
> Fetch task uses udfdatediff via {code}
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the
> vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}.
> This fix is meant to be not very intrusive and will add more support to the
> GenericUDFToDate by enhancing the parser.
> For the longer term will be better to use one parser for all the operators.
> Thanks [~Rajkumar Singh] for the repro example
> {code}
> create external table testdatediff(datetimecol string) stored as orc;
> insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24');
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> set hive.ferch.task.conversion=none;
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22464) Implement support for NULLS FIRST/LAST in TopNKeyOperator
[ https://issues.apache.org/jira/browse/HIVE-22464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22464: -- Status: Patch Available (was: Open) > Implement support for NULLS FIRST/LAST in TopNKeyOperator > - > > Key: HIVE-22464 > URL: https://issues.apache.org/jira/browse/HIVE-22464 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Attachments: HIVE-22464.1.patch, HIVE-22464.2.patch, > HIVE-22464.3.patch, HIVE-22464.4.patch, HIVE-22464.4.patch, > HIVE-22464.4.patch, HIVE-22464.4.patch, HIVE-22464.4.patch, > HIVE-22464.4.patch, HIVE-22464.4.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22464) Implement support for NULLS FIRST/LAST in TopNKeyOperator
[ https://issues.apache.org/jira/browse/HIVE-22464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22464: -- Attachment: HIVE-22464.4.patch > Implement support for NULLS FIRST/LAST in TopNKeyOperator > - > > Key: HIVE-22464 > URL: https://issues.apache.org/jira/browse/HIVE-22464 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Attachments: HIVE-22464.1.patch, HIVE-22464.2.patch, > HIVE-22464.3.patch, HIVE-22464.4.patch, HIVE-22464.4.patch, > HIVE-22464.4.patch, HIVE-22464.4.patch, HIVE-22464.4.patch, > HIVE-22464.4.patch, HIVE-22464.4.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22464) Implement support for NULLS FIRST/LAST in TopNKeyOperator
[ https://issues.apache.org/jira/browse/HIVE-22464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22464: -- Status: Open (was: Patch Available) > Implement support for NULLS FIRST/LAST in TopNKeyOperator > - > > Key: HIVE-22464 > URL: https://issues.apache.org/jira/browse/HIVE-22464 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Attachments: HIVE-22464.1.patch, HIVE-22464.2.patch, > HIVE-22464.3.patch, HIVE-22464.4.patch, HIVE-22464.4.patch, > HIVE-22464.4.patch, HIVE-22464.4.patch, HIVE-22464.4.patch, > HIVE-22464.4.patch, HIVE-22464.4.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22329) Create DriverContext
[
https://issues.apache.org/jira/browse/HIVE-22329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975254#comment-16975254
]
Hive QA commented on HIVE-22329:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 1544 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} hcatalog/core in master has 36 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 10 new + 346 unchanged - 72
fixed = 356 total (was 418) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
17s{color} | {color:red} ql generated 13 new + 1531 unchanged - 13 fixed = 1544
total (was 1544) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
5s{color} | {color:red} ql generated 1 new + 99 unchanged - 1 fixed = 100 total
(was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Naked notify in org.apache.hadoop.hive.ql.TaskQueue.releaseRunnable() At
TaskQueue.java:At TaskQueue.java:[line 98] |
| | Found reliance on default encoding in
org.apache.hadoop.hive.ql.exec.ExplainSQRewriteTask.execute():in
org.apache.hadoop.hive.ql.exec.ExplainSQRewriteTask.execute(): new
java.io.PrintStream(OutputStream) At ExplainSQRewriteTask.java:[line 59] |
| | Dead store to jsonPlan in
org.apache.hadoop.hive.ql.exec.ExplainTask.execute() At
ExplainTask.java:org.apache.hadoop.hive.ql.exec.ExplainTask.execute() At
ExplainTask.java:[line 454] |
| | Found reliance on default encoding in
org.apache.hadoop.hive.ql.exec.ExplainTask.execute():in
org.apache.hadoop.hive.ql.exec.ExplainTask.execute(): new
java.io.PrintStream(OutputStream) At ExplainTask.java:[line 406] |
| | Invocation of toString on
org.apache.hadoop.hive.ql.plan.CopyWork.getFromPaths() in
org.apache.hadoop.hive.ql.exec.ReplCopyTask.execute() At ReplCopyTask.java:in
org.apache.hadoop.hive.ql.exec.ReplCopyTask.execute() At
ReplCopyTask.java:[line 145] |
| | Invocation of toString on
org.apache.hadoop.hive.ql.plan.CopyWork.getToPaths() in
org.apache.hadoop.hive.ql.exec.ReplCopyTask.execute() At ReplCopyTask.java:in
org.apache.hadoop.hive.ql.exec.ReplCopyTask.execute() At
ReplCopyTask.java:[line 145] |
| | Exception is caught when Exception is not thrown in
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute() At MapRedTask.java:is
not thrown in org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute() At
MapRedTask.java:[line 319] |
| |
[jira] [Assigned] (HIVE-22502) ConcurrentModificationException in TriggerValidatorRunnable stops trigger processing
[ https://issues.apache.org/jira/browse/HIVE-22502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar reassigned HIVE-22502: > ConcurrentModificationException in TriggerValidatorRunnable stops trigger > processing > > > Key: HIVE-22502 > URL: https://issues.apache.org/jira/browse/HIVE-22502 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22464) Implement support for NULLS FIRST/LAST in TopNKeyOperator
[
https://issues.apache.org/jira/browse/HIVE-22464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975224#comment-16975224
]
Hive QA commented on HIVE-22464:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985910/HIVE-22464.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17706 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar.org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar
(batchId=248)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19440/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19440/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19440/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985910 - PreCommit-HIVE-Build
> Implement support for NULLS FIRST/LAST in TopNKeyOperator
> -
>
> Key: HIVE-22464
> URL: https://issues.apache.org/jira/browse/HIVE-22464
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22464.1.patch, HIVE-22464.2.patch,
> HIVE-22464.3.patch, HIVE-22464.4.patch, HIVE-22464.4.patch,
> HIVE-22464.4.patch, HIVE-22464.4.patch, HIVE-22464.4.patch, HIVE-22464.4.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21226) Exclude read-only transactions from ValidTxnList
[ https://issues.apache.org/jira/browse/HIVE-21226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975203#comment-16975203 ] Denys Kuzmenko commented on HIVE-21226: --- [~pvary] could you please take a look. > Exclude read-only transactions from ValidTxnList > > > Key: HIVE-21226 > URL: https://issues.apache.org/jira/browse/HIVE-21226 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21226.1.patch, HIVE-21226.2.patch, > HIVE-21226.3.patch, HIVE-21226.4.patch, HIVE-21226.5.patch > > > Once HIVE-21114 is done, we should make sure that ValidTxnList doesn't > contain any read-only txns in the exceptions list since by definition there > is no data tagged with such txnid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22499) LLAP: Add an EncodedReaderOptions to extend ORC impl for options
[ https://issues.apache.org/jira/browse/HIVE-22499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal Vijayaraghavan updated HIVE-22499: Status: Patch Available (was: Open) > LLAP: Add an EncodedReaderOptions to extend ORC impl for options > > > Key: HIVE-22499 > URL: https://issues.apache.org/jira/browse/HIVE-22499 > Project: Hive > Issue Type: Bug > Components: llap, ORC >Reporter: Gopal Vijayaraghavan >Priority: Major > Attachments: HIVE-22499.WIP.patch > > > ORC-570 is an ABI change to the way getFileSystem() by adding an another > exception to the implementation. > To accept and use that change requires waiting for an ORC release, while this > patch serves the same purpose though falls back for a retry for > FileSystem.get() in case the supplier fails at runtime. > Also as a side-note, the FS.get() call is always used in the cases where the > file is not being read from a cache such as EncodedOrcFile (so the upstream > API change might be overkill). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21917) COMPLETED_TXN_COMPONENTS table is never cleaned up unless Compactor runs
[ https://issues.apache.org/jira/browse/HIVE-21917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-21917: -- Attachment: HIVE-21917.2.patch > COMPLETED_TXN_COMPONENTS table is never cleaned up unless Compactor runs > > > Key: HIVE-21917 > URL: https://issues.apache.org/jira/browse/HIVE-21917 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0, 3.1.1 >Reporter: Craig Condit >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21917.1.patch, HIVE-21917.2.patch > > > The Initiator thread in the metastore repeatedly loops over entries in the > COMPLETED_TXN_COMPONENTS table to determine which partitions / tables might > need to be compacted. However, entries are never removed from this table > except by a completed Compactor run. > In a cluster where most tables / partitions are write-once read-many, this > results in stale entries in this table never being cleaned up. In a small > test cluster, we have observed approximately 45k entries in this table > (virtually equal to the number of partitions in the cluster) while < 100 of > these tables have delta files at all. Since most of the tables will never get > enough writes to trigger a compaction (and in fact have only ever been > written to once), the initiator thread keeps trying to evaluate them on every > loop. > On this test cluster, it takes approximately 10 minutes to loop through all > the entries and results in severe performance degradation on metastore > operations. With the default run timing of 5 minutes, the initiator basically > never stops running. > On a production cluster with 2M partitions, this would be a non-starter. > The initiator thread should proactively remove entries from > COMPLETED_TXN_COMPONENTS when it determines that a compaction is not needed, > so that they are not evaluated again on the next loop. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22464) Implement support for NULLS FIRST/LAST in TopNKeyOperator
[
https://issues.apache.org/jira/browse/HIVE-22464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975188#comment-16975188
]
Hive QA commented on HIVE-22464:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} serde in master has 199 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 1544 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} serde generated 0 new + 198 unchanged - 1 fixed =
198 total (was 199) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} ql generated 0 new + 1541 unchanged - 3 fixed = 1541
total (was 1544) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19440/dev-support/hive-personality.sh
|
| git revision | master / 078182a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: serde ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19440/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement support for NULLS FIRST/LAST in TopNKeyOperator
> -
>
> Key: HIVE-22464
> URL: https://issues.apache.org/jira/browse/HIVE-22464
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22464.1.patch, HIVE-22464.2.patch,
> HIVE-22464.3.patch, HIVE-22464.4.patch, HIVE-22464.4.patch,
> HIVE-22464.4.patch, HIVE-22464.4.patch, HIVE-22464.4.patch, HIVE-22464.4.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22461) NPE Metastore Transformer
[
https://issues.apache.org/jira/browse/HIVE-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975176#comment-16975176
]
Yongzhi Chen commented on HIVE-22461:
-
Attached patch 6 to address the review issue.
> NPE Metastore Transformer
> -
>
> Key: HIVE-22461
> URL: https://issues.apache.org/jira/browse/HIVE-22461
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-22461.1.patch, HIVE-22461.2.patch,
> HIVE-22461.3.patch, HIVE-22461.4.patch, HIVE-22461.5.patch, HIVE-22461.6.patch
>
>
> The stack looks as following:
> {noformat}
> 2019-10-08 18:09:12,198 INFO
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
> [pool-6-thread-328]: Starting translation for processor
> Hiveserver2#3.1.2000.7.0.2.0...@vc0732.halxg.cloudera.com on list 1
> 2019-10-08 18:09:12,198 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-328]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16633)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16617)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-10-08 18:09:12,199 ERROR org.apache.thrift.server.TThreadPoolServer:
> [pool-6-thread-328]: Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source) ~[?:?]
> at
>
[jira] [Updated] (HIVE-22461) NPE Metastore Transformer
[
https://issues.apache.org/jira/browse/HIVE-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-22461:
Attachment: (was: HIVE-22461.6.patch)
> NPE Metastore Transformer
> -
>
> Key: HIVE-22461
> URL: https://issues.apache.org/jira/browse/HIVE-22461
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-22461.1.patch, HIVE-22461.2.patch,
> HIVE-22461.3.patch, HIVE-22461.4.patch, HIVE-22461.5.patch, HIVE-22461.6.patch
>
>
> The stack looks as following:
> {noformat}
> 2019-10-08 18:09:12,198 INFO
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
> [pool-6-thread-328]: Starting translation for processor
> Hiveserver2#3.1.2000.7.0.2.0...@vc0732.halxg.cloudera.com on list 1
> 2019-10-08 18:09:12,198 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-328]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16633)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16617)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-10-08 18:09:12,199 ERROR org.apache.thrift.server.TThreadPoolServer:
> [pool-6-thread-328]: Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source) ~[?:?]
> at
>
[jira] [Updated] (HIVE-22461) NPE Metastore Transformer
[
https://issues.apache.org/jira/browse/HIVE-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-22461:
Attachment: HIVE-22461.6.patch
> NPE Metastore Transformer
> -
>
> Key: HIVE-22461
> URL: https://issues.apache.org/jira/browse/HIVE-22461
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-22461.1.patch, HIVE-22461.2.patch,
> HIVE-22461.3.patch, HIVE-22461.4.patch, HIVE-22461.5.patch,
> HIVE-22461.6.patch, HIVE-22461.6.patch
>
>
> The stack looks as following:
> {noformat}
> 2019-10-08 18:09:12,198 INFO
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
> [pool-6-thread-328]: Starting translation for processor
> Hiveserver2#3.1.2000.7.0.2.0...@vc0732.halxg.cloudera.com on list 1
> 2019-10-08 18:09:12,198 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-328]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16633)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16617)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-10-08 18:09:12,199 ERROR org.apache.thrift.server.TThreadPoolServer:
> [pool-6-thread-328]: Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source) ~[?:?]
> at
>
[jira] [Commented] (HIVE-22461) NPE Metastore Transformer
[
https://issues.apache.org/jira/browse/HIVE-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975165#comment-16975165
]
Yongzhi Chen commented on HIVE-22461:
-
Checked the source code, the claim above is not true: in this code, newTable is
a copy of table.
> NPE Metastore Transformer
> -
>
> Key: HIVE-22461
> URL: https://issues.apache.org/jira/browse/HIVE-22461
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-22461.1.patch, HIVE-22461.2.patch,
> HIVE-22461.3.patch, HIVE-22461.4.patch, HIVE-22461.5.patch, HIVE-22461.6.patch
>
>
> The stack looks as following:
> {noformat}
> 2019-10-08 18:09:12,198 INFO
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
> [pool-6-thread-328]: Starting translation for processor
> Hiveserver2#3.1.2000.7.0.2.0...@vc0732.halxg.cloudera.com on list 1
> 2019-10-08 18:09:12,198 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-328]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16633)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16617)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-10-08 18:09:12,199 ERROR org.apache.thrift.server.TThreadPoolServer:
> [pool-6-thread-328]: Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source) ~[?:?]
> at
>
[jira] [Commented] (HIVE-22498) Schema tool enhancements to merge catalogs
[
https://issues.apache.org/jira/browse/HIVE-22498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975158#comment-16975158
]
Hive QA commented on HIVE-22498:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985906/HIVE-22498.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17706 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb_schq]
(batchId=177)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19439/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19439/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19439/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985906 - PreCommit-HIVE-Build
> Schema tool enhancements to merge catalogs
> --
>
> Key: HIVE-22498
> URL: https://issues.apache.org/jira/browse/HIVE-22498
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22498.patch
>
>
> Schema tool currently supports relocation of database from one catalog to
> another, one at a time. While having to do this one at a time is painful, it
> also lacks support for converting them to external tables during migration,
> in lieu of the changes to the translation layer where a MANAGED table is
> strictly ACID-only table.
> Hence we also need to convert them to external tables during relocation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22461) NPE Metastore Transformer
[
https://issues.apache.org/jira/browse/HIVE-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975154#comment-16975154
]
Yongzhi Chen edited comment on HIVE-22461 at 11/15/19 2:57 PM:
---
[~ngangam], I think you said that the newTable is created by hive and sd cannot
be null
was (Author: ychena):
[~ngangam], I think you said that the newTable is created by hive and cannot be
null
> NPE Metastore Transformer
> -
>
> Key: HIVE-22461
> URL: https://issues.apache.org/jira/browse/HIVE-22461
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-22461.1.patch, HIVE-22461.2.patch,
> HIVE-22461.3.patch, HIVE-22461.4.patch, HIVE-22461.5.patch, HIVE-22461.6.patch
>
>
> The stack looks as following:
> {noformat}
> 2019-10-08 18:09:12,198 INFO
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
> [pool-6-thread-328]: Starting translation for processor
> Hiveserver2#3.1.2000.7.0.2.0...@vc0732.halxg.cloudera.com on list 1
> 2019-10-08 18:09:12,198 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-328]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16633)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16617)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-10-08 18:09:12,199 ERROR org.apache.thrift.server.TThreadPoolServer:
> [pool-6-thread-328]: Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
>
[jira] [Commented] (HIVE-22461) NPE Metastore Transformer
[
https://issues.apache.org/jira/browse/HIVE-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975154#comment-16975154
]
Yongzhi Chen commented on HIVE-22461:
-
[~ngangam], I think you said that the newTable is created by hive and cannot be
null
> NPE Metastore Transformer
> -
>
> Key: HIVE-22461
> URL: https://issues.apache.org/jira/browse/HIVE-22461
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-22461.1.patch, HIVE-22461.2.patch,
> HIVE-22461.3.patch, HIVE-22461.4.patch, HIVE-22461.5.patch, HIVE-22461.6.patch
>
>
> The stack looks as following:
> {noformat}
> 2019-10-08 18:09:12,198 INFO
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
> [pool-6-thread-328]: Starting translation for processor
> Hiveserver2#3.1.2000.7.0.2.0...@vc0732.halxg.cloudera.com on list 1
> 2019-10-08 18:09:12,198 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-328]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16633)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16617)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-10-08 18:09:12,199 ERROR org.apache.thrift.server.TThreadPoolServer:
> [pool-6-thread-328]: Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source) ~[?:?]
> at
>
[jira] [Updated] (HIVE-22501) Stats reported multiple times during MR execution for UNION queries
[
https://issues.apache.org/jira/browse/HIVE-22501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Voros updated HIVE-22501:
Status: Patch Available (was: Open)
Attached patch #1 that only let's operators report their stats once.
> Stats reported multiple times during MR execution for UNION queries
> ---
>
> Key: HIVE-22501
> URL: https://issues.apache.org/jira/browse/HIVE-22501
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Labels: mapreduce
> Attachments: HIVE-22501.1.patch
>
>
> Take the following example:
> {code}
> set hive.execution.engine=mr;
> create table tb(id string) stored as orc;
> insert into tb values('1');
> create table tb2 like tb stored as orc;
> insert into tb2 select * from tb union all select * from tb;
> {code}
> Last insert results in 2 records in the table, but
> {{TOTAL_TABLE_ROWS_WRITTEN}} statistic (and number of affected rows on the
> consolse) is 4.
> We seem to traverse the operator graph multiple times starting from every TS
> operator and increment the counters every time we hit the FS operator.
> UNION-ing the table 3 times results in 9 TOTAL_TABLE_ROWS_WRITTEN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22501) Stats reported multiple times during MR execution for UNION queries
[
https://issues.apache.org/jira/browse/HIVE-22501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975148#comment-16975148
]
Daniel Voros edited comment on HIVE-22501 at 11/15/19 2:46 PM:
---
Attached patch #1 that only lets operators report their stats once.
was (Author: dvoros):
Attached patch #1 that only let's operators report their stats once.
> Stats reported multiple times during MR execution for UNION queries
> ---
>
> Key: HIVE-22501
> URL: https://issues.apache.org/jira/browse/HIVE-22501
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Labels: mapreduce
> Attachments: HIVE-22501.1.patch
>
>
> Take the following example:
> {code}
> set hive.execution.engine=mr;
> create table tb(id string) stored as orc;
> insert into tb values('1');
> create table tb2 like tb stored as orc;
> insert into tb2 select * from tb union all select * from tb;
> {code}
> Last insert results in 2 records in the table, but
> {{TOTAL_TABLE_ROWS_WRITTEN}} statistic (and number of affected rows on the
> consolse) is 4.
> We seem to traverse the operator graph multiple times starting from every TS
> operator and increment the counters every time we hit the FS operator.
> UNION-ing the table 3 times results in 9 TOTAL_TABLE_ROWS_WRITTEN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22501) Stats reported multiple times during MR execution for UNION queries
[
https://issues.apache.org/jira/browse/HIVE-22501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Voros updated HIVE-22501:
Attachment: HIVE-22501.1.patch
> Stats reported multiple times during MR execution for UNION queries
> ---
>
> Key: HIVE-22501
> URL: https://issues.apache.org/jira/browse/HIVE-22501
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Labels: mapreduce
> Attachments: HIVE-22501.1.patch
>
>
> Take the following example:
> {code}
> set hive.execution.engine=mr;
> create table tb(id string) stored as orc;
> insert into tb values('1');
> create table tb2 like tb stored as orc;
> insert into tb2 select * from tb union all select * from tb;
> {code}
> Last insert results in 2 records in the table, but
> {{TOTAL_TABLE_ROWS_WRITTEN}} statistic (and number of affected rows on the
> consolse) is 4.
> We seem to traverse the operator graph multiple times starting from every TS
> operator and increment the counters every time we hit the FS operator.
> UNION-ing the table 3 times results in 9 TOTAL_TABLE_ROWS_WRITTEN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22501) Stats reported multiple times during MR execution for UNION queries
[
https://issues.apache.org/jira/browse/HIVE-22501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Voros reassigned HIVE-22501:
---
> Stats reported multiple times during MR execution for UNION queries
> ---
>
> Key: HIVE-22501
> URL: https://issues.apache.org/jira/browse/HIVE-22501
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Labels: mapreduce
>
> Take the following example:
> {code}
> set hive.execution.engine=mr;
> create table tb(id string) stored as orc;
> insert into tb values('1');
> create table tb2 like tb stored as orc;
> insert into tb2 select * from tb union all select * from tb;
> {code}
> Last insert results in 2 records in the table, but
> {{TOTAL_TABLE_ROWS_WRITTEN}} statistic (and number of affected rows on the
> consolse) is 4.
> We seem to traverse the operator graph multiple times starting from every TS
> operator and increment the counters every time we hit the FS operator.
> UNION-ing the table 3 times results in 9 TOTAL_TABLE_ROWS_WRITTEN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-13690) Shade guava in hive-exec fat jar
[ https://issues.apache.org/jira/browse/HIVE-13690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-13690: Resolution: Duplicate Status: Resolved (was: Patch Available) This was recently merged through HIVE-22126, so if you don't have any objections, I'm closing this. > Shade guava in hive-exec fat jar > > > Key: HIVE-13690 > URL: https://issues.apache.org/jira/browse/HIVE-13690 > Project: Hive > Issue Type: Improvement >Reporter: Rohan Agarwal >Assignee: Rohan Agarwal >Priority: Major > Attachments: HIVE-13690.patch > > > The hive-exec fat jar bundles the Guava classes. This makes relying on the > hive-exec fat jar difficult (see HIVE-5733 for more discussion). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21569) Preparation to bump guava version to 27.0-jre
[ https://issues.apache.org/jira/browse/HIVE-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-21569: Summary: Preparation to bump guava version to 27.0-jre (was: Preparation to bump guava version) > Preparation to bump guava version to 27.0-jre > - > > Key: HIVE-21569 > URL: https://issues.apache.org/jira/browse/HIVE-21569 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21569.01.patch, HIVE-21569.02.patch > > > This is a preparation work in hive, before updating guava version to 27.0-jre. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22307) Upgrade Hadoop version to 3.1.3
[ https://issues.apache.org/jira/browse/HIVE-22307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975121#comment-16975121 ] David Lavati commented on HIVE-22307: - Meanwhile hadoop 3.2.1 has also come out, however both versions are now using guava 27.0-jre which blocks this. > Upgrade Hadoop version to 3.1.3 > --- > > Key: HIVE-22307 > URL: https://issues.apache.org/jira/browse/HIVE-22307 > Project: Hive > Issue Type: Improvement >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Attachments: HIVE-22307.patch, HIVE-22307.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HIVE-21569) Preparation to bump guava version
[ https://issues.apache.org/jira/browse/HIVE-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati reopened HIVE-21569: - Assignee: (was: Laszlo Pinter) > Preparation to bump guava version > - > > Key: HIVE-21569 > URL: https://issues.apache.org/jira/browse/HIVE-21569 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21569.01.patch, HIVE-21569.02.patch > > > This is a preparation work in hive, before updating guava version to 27.0-jre. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22327) Repl: Ignore read-only transactions in notification log
[ https://issues.apache.org/jira/browse/HIVE-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-22327: -- Attachment: HIVE-22327.5.patch > Repl: Ignore read-only transactions in notification log > --- > > Key: HIVE-22327 > URL: https://issues.apache.org/jira/browse/HIVE-22327 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Gopal Vijayaraghavan >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22327.1.patch, HIVE-22327.2.patch, > HIVE-22327.3.patch, HIVE-22327.4.patch, HIVE-22327.5.patch > > > Read txns need not be replicated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22498) Schema tool enhancements to merge catalogs
[
https://issues.apache.org/jira/browse/HIVE-22498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975112#comment-16975112
]
Hive QA commented on HIVE-22498:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
13s{color} | {color:blue} standalone-metastore/metastore-server in master has
178 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 8 new + 36 unchanged - 0 fixed = 44 total (was 36) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
24s{color} | {color:red} standalone-metastore/metastore-server generated 4 new
+ 178 unchanged - 0 fixed = 182 total (was 178) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| |
org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolTaskMergeCatalog.execute()
invokes System.exit(...), which shuts down the entire virtual machine At
SchemaToolTaskMergeCatalog.java:down the entire virtual machine At
SchemaToolTaskMergeCatalog.java:[line 85] |
| |
org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolTaskMergeCatalog.execute()
may fail to clean up java.sql.ResultSet Obligation to clean up resource
created at SchemaToolTaskMergeCatalog.java:up java.sql.ResultSet Obligation to
clean up resource created at SchemaToolTaskMergeCatalog.java:[line 74] is not
discharged |
| |
org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolTaskMergeCatalog.execute()
may fail to clean up java.sql.Statement Obligation to clean up resource
created at SchemaToolTaskMergeCatalog.java:up java.sql.Statement Obligation to
clean up resource created at SchemaToolTaskMergeCatalog.java:[line 66] is not
discharged |
| |
org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolTaskMergeCatalog.execute()
may fail to close Statement At SchemaToolTaskMergeCatalog.java:Statement At
SchemaToolTaskMergeCatalog.java:[line 66] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19439/dev-support/hive-personality.sh
|
| git revision | master / 078182a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19439/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19439/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19439/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
|
[jira] [Updated] (HIVE-22488) Break up DDLSemanticAnalyzer - extract Table creation analyzers
[ https://issues.apache.org/jira/browse/HIVE-22488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22488: -- Attachment: HIVE-22488.01.patch > Break up DDLSemanticAnalyzer - extract Table creation analyzers > --- > > Key: HIVE-22488 > URL: https://issues.apache.org/jira/browse/HIVE-22488 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22488.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #9: extract the table creationanalyzers from DDLSemanticAnalyzer, and > move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22488) Break up DDLSemanticAnalyzer - extract Table creation analyzers
[ https://issues.apache.org/jira/browse/HIVE-22488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22488: -- Attachment: (was: HIVE-22488.01.patch) > Break up DDLSemanticAnalyzer - extract Table creation analyzers > --- > > Key: HIVE-22488 > URL: https://issues.apache.org/jira/browse/HIVE-22488 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22488.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #9: extract the table creationanalyzers from DDLSemanticAnalyzer, and > move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22488) Break up DDLSemanticAnalyzer - extract Table creation analyzers
[
https://issues.apache.org/jira/browse/HIVE-22488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975094#comment-16975094
]
Hive QA commented on HIVE-22488:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985895/HIVE-22488.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17702 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACIDNoAutogather - did not produce a TEST-*.xml
file (likely timed out) (batchId=256)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19438/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19438/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19438/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985895 - PreCommit-HIVE-Build
> Break up DDLSemanticAnalyzer - extract Table creation analyzers
> ---
>
> Key: HIVE-22488
> URL: https://issues.apache.org/jira/browse/HIVE-22488
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Attachments: HIVE-22488.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is
> to refactor it in order to have everything cut into more handleable classes
> under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each analyzers
> * have a package for each operation, containing an analyzer, a description,
> and an operation, so the amount of classes under a package is more manageable
> Step #9: extract the table creationanalyzers from DDLSemanticAnalyzer, and
> move them under the new package.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22488) Break up DDLSemanticAnalyzer - extract Table creation analyzers
[
https://issues.apache.org/jira/browse/HIVE-22488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975066#comment-16975066
]
Hive QA commented on HIVE-22488:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
24s{color} | {color:blue} ql in master has 1544 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} hcatalog/core in master has 36 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
50s{color} | {color:blue} itests/util in master has 53 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
28s{color} | {color:green} ql generated 0 new + 1543 unchanged - 1 fixed = 1543
total (was 1544) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} util in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19438/dev-support/hive-personality.sh
|
| git revision | master / 078182a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql hcatalog/core itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19438/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Break up DDLSemanticAnalyzer - extract Table creation analyzers
> ---
>
> Key: HIVE-22488
> URL: https://issues.apache.org/jira/browse/HIVE-22488
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Attachments: HIVE-22488.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is
> to refactor it in order to have everything cut into more handleable classes
> under the
[jira] [Commented] (HIVE-22497) Remove default value for Capabilities from HiveConf
[
https://issues.apache.org/jira/browse/HIVE-22497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975039#comment-16975039
]
Hive QA commented on HIVE-22497:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985878/HIVE-22497.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17706 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19437/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19437/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19437/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985878 - PreCommit-HIVE-Build
> Remove default value for Capabilities from HiveConf
> ---
>
> Key: HIVE-22497
> URL: https://issues.apache.org/jira/browse/HIVE-22497
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22497.patch
>
>
> This class is used and bundled in other jars that 3rd party connectors like
> teradata etc. So it would be good to remove this default value from HiveConf
> but rely on it being set in hive-site.xml instead. The HiveServer2 should
> still set this as part of HS2 initialization or via hiveserver2-site.xml
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22497) Remove default value for Capabilities from HiveConf
[
https://issues.apache.org/jira/browse/HIVE-22497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975012#comment-16975012
]
Hive QA commented on HIVE-22497:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
45s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19437/dev-support/hive-personality.sh
|
| git revision | master / 078182a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common common U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19437/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove default value for Capabilities from HiveConf
> ---
>
> Key: HIVE-22497
> URL: https://issues.apache.org/jira/browse/HIVE-22497
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22497.patch
>
>
> This class is used and bundled in other jars that 3rd party connectors like
> teradata etc. So it would be good to remove this default value from HiveConf
> but rely on it being set in hive-site.xml instead. The HiveServer2 should
> still set this as part of HS2 initialization or via hiveserver2-site.xml
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22495) Parquet count(*) read in all data
[
https://issues.apache.org/jira/browse/HIVE-22495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974994#comment-16974994
]
Hive QA commented on HIVE-22495:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985879/HIVE-22495.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19436/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19436/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19436/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12985879/HIVE-22495.patch was
found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985879 - PreCommit-HIVE-Build
> Parquet count(*) read in all data
> -
>
> Key: HIVE-22495
> URL: https://issues.apache.org/jira/browse/HIVE-22495
> Project: Hive
> Issue Type: Bug
> Components: Reader
>Reporter: Jason Xu
>Assignee: Jason Xu
>Priority: Major
> Attachments: HIVE-22495.patch, HIVE-22495.patch
>
>
> Running a hive query on a Parquet table
> select count ( * ) from test_table
> The query read in all data (all columns) instead of just metadata.
> For comparison, hive 0.13 and Spark read in much less data with my test table.
>
> ||engine||HDFS data read||
> |Hive 2.3.4| 452.9 MB|
> |Hive 0.13| 22.5 KB|
> |Spark| 41.6 KB|
>
> Seems cause is that Parquet read support fall back to file schema if
> indexColumnsWanted is empty, logic still exist in master branch.
> Don't know why this empty list check was added, please suggest if there're
> any other impact.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22496) Update Hadoop Version to 3.1.2
[
https://issues.apache.org/jira/browse/HIVE-22496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974993#comment-16974993
]
Hive QA commented on HIVE-22496:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985876/HIVE-22496.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 647 failed/errored test(s), 17706 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[add_part_multiple]
(batchId=146)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[alter_merge_orc]
(batchId=144)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[alter_merge_stats_orc]
(batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[annotate_stats_join]
(batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join0]
(batchId=154)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join10]
(batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join11]
(batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join12]
(batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join13]
(batchId=151)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join14]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join15]
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join16]
(batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join17]
(batchId=152)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join18]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join18_multi_distinct]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join19]
(batchId=144)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join1]
(batchId=150)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join20]
(batchId=155)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join21]
(batchId=152)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join22]
(batchId=139)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join23]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join24]
(batchId=149)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join26]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join27]
(batchId=155)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join28]
(batchId=147)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join29]
(batchId=139)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join2]
(batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join30]
(batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join31]
(batchId=135)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join32]
(batchId=153)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join3]
(batchId=152)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join4]
(batchId=146)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join5]
(batchId=148)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join6]
(batchId=153)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join7]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join8]
(batchId=153)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join9]
(batchId=149)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_filters]
(batchId=141)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_nulls]
(batchId=145)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_reordering_values]
(batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_stats2]
(batchId=154)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_stats]
(batchId=136)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_without_localtask]
(batchId=114)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_smb_mapjoin_14]
(batchId=141)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_12]
(batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_13]
(batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_14]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_15]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_16]
[jira] [Updated] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22280: - Attachment: HIVE-22280.03.patch > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22496) Update Hadoop Version to 3.1.2
[
https://issues.apache.org/jira/browse/HIVE-22496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974977#comment-16974977
]
Hive QA commented on HIVE-22496:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19435/dev-support/hive-personality.sh
|
| git revision | master / 078182a |
| Default Java | 1.8.0_111 |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19435/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Update Hadoop Version to 3.1.2
> --
>
> Key: HIVE-22496
> URL: https://issues.apache.org/jira/browse/HIVE-22496
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Critical
> Attachments: HIVE-22496.1.patch
>
>
> https://lists.apache.org/thread.html/8313e605c0ed0012f134cce9cc6adca738eea81feccea99c8de87cd9@%3Cgeneral.hadoop.apache.org%3E
> {quote}
> It gives us great pleasure to announce that the Apache Hadoop community has
> voted to release Apache Hadoop 3.1.0.
>- This release is *not* yet ready for production use. Critical issues
>are being ironed out via testing and downstream adoption. Production
> users should wait for a 3.1.1/3.1.2 release.
> {quote}
> Current:
> {code:xml}
> 3.1.0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22500) sort by and timestamp casting filters out too many rows
[
https://issues.apache.org/jira/browse/HIVE-22500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guillaume updated HIVE-22500:
-
Description:
Consider this example.
Preparation:
{code:java}
create temporary table opens as (
select stack(1,
1 , cast('2019-11-13 08:07:28' as timestamp)
) as (id , load_ts )
);
{code}
Queries: This is just about counting the number of rows, with filters always
matching, and possibly sort by. 1 is always expected.
{code:java}
select count(*) from ( select * from opens) t;
select count(*) from ( select * from opens sort by id) t;
select count(*) from ( select * from opens where load_ts >= '2019-11-13
08:07:00' ) t;
select count(*) from ( select * from opens where load_ts >= '2019-11-13
08:07:00' sort by id) t;
select count(*) from ( select * from opens where load_ts <= '2019-11-13
09:07:00') t;
select count(*) from ( select * from opens where load_ts <= '2019-11-13
09:07:00' sort by id) t;
{code}
The latest query (_sort by_ and _<=_ on timestamp) returns 0 rows.
I believe that this is the cause of other issues I have, where I have missing
rows in queries with the timestamp (but not the explicit sort by).
Note that if instead of a temporary table I use a CTE for opens, the issue does
not appear.
I tried workarounds (inverse order of operands, adding _not_ or _not not_ ) to
no avail.
One thing that did work is to explicitly cast the string to a timestamp:
{code:java}
select count(*) from ( select * from opens where load_ts <= cast('2019-11-13
09:07:00' as timestamp) sort by id) t;{code}
It might be good practice indeed, but there still is a discrepancy between how
_>=_ and _<=_ are handled, or how _sort by_ works.
Note: this is on Hive from
[hdp3.1.4|[https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/release-notes/content/patch_hive.html]],
without llap.
was:
Consider this example.
Preparation:
{code:java}
create temporary table opens as (
select stack(1,
1 , cast('2019-11-13 08:07:28' as timestamp)
) as (id , load_ts )
);
{code}
Queries: This is just about counting the number of rows, with filters always
matching, and possibly sort by. 1 is always expected.
{code:java}
select count(*) from ( select * from opens) t;
select count(*) from ( select * from opens sort by id) t;
select count(*) from ( select * from opens where load_ts >= '2019-11-13
08:07:00' ) t;
select count(*) from ( select * from opens where load_ts >= '2019-11-13
08:07:00' sort by id) t;
select count(*) from ( select * from opens where load_ts <= '2019-11-13
09:07:00') t;
select count(*) from ( select * from opens where load_ts <= '2019-11-13
09:07:00' sort by id) t;
{code}
The latest query (_sort by_ and _<=_ on timestamp) returns 0 rows.
I believe that this is the cause of other issues I have, where I have missing
rows in queries with the timestamp (but not the explicit sort by).
Note that if instead of a temporary table I use a CTE for opens, the issue does
not appear.
I tried workarounds (inverse order of operands, adding _not_ or _not not_ ) to
no avail.
One thing that did work is to explicitly cast the string to a timestamp:
{code:java}
select count(*) from ( select * from opens where load_ts <= cast('2019-11-13
09:07:00' as timestamp) sort by id) t;{code}
It might be good practice indeed, but there still is a discrepancy between how
_>=_ and _<=_ are handled, or how _sort by_ works.
Note: this is on Hive from
[hdp3.1.4|[https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/release-notes/content/patch_hive.html]],
without llap.
> sort by and timestamp casting filters out too many rows
> ---
>
> Key: HIVE-22500
> URL: https://issues.apache.org/jira/browse/HIVE-22500
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Guillaume
>Priority: Major
>
> Consider this example.
> Preparation:
> {code:java}
> create temporary table opens as (
> select stack(1,
> 1 , cast('2019-11-13 08:07:28' as timestamp)
>) as (id , load_ts )
> );
> {code}
>
> Queries: This is just about counting the number of rows, with filters always
> matching, and possibly sort by. 1 is always expected.
> {code:java}
> select count(*) from ( select * from opens) t;
> select count(*) from ( select * from opens sort by id) t;
> select count(*) from ( select * from opens where load_ts >= '2019-11-13
> 08:07:00' ) t;
> select count(*) from ( select * from opens where load_ts >= '2019-11-13
> 08:07:00' sort by id) t;
> select count(*) from ( select * from opens where load_ts <= '2019-11-13
> 09:07:00') t;
> select count(*) from ( select * from opens where load_ts <= '2019-11-13
> 09:07:00' sort by id) t;
> {code}
>
> The latest query (_sort by_ and _<=_ on timestamp) returns 0 rows.
> I believe
[jira] [Updated] (HIVE-22500) sort by and timestamp casting filters out too many rows
[
https://issues.apache.org/jira/browse/HIVE-22500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guillaume updated HIVE-22500:
-
Description:
Consider this example.
Preparation:
{code:java}
create temporary table opens as (
select stack(1,
1 , cast('2019-11-13 08:07:28' as timestamp)
) as (id , load_ts )
);
{code}
Queries: This is just about counting the number of rows, with filters always
matching, and possibly sort by. 1 is always expected.
{code:java}
select count(*) from ( select * from opens) t;
select count(*) from ( select * from opens sort by id) t;
select count(*) from ( select * from opens where load_ts >= '2019-11-13
08:07:00' ) t;
select count(*) from ( select * from opens where load_ts >= '2019-11-13
08:07:00' sort by id) t;
select count(*) from ( select * from opens where load_ts <= '2019-11-13
09:07:00') t;
select count(*) from ( select * from opens where load_ts <= '2019-11-13
09:07:00' sort by id) t;
{code}
The last query (_sort by_ and _<=_ on timestamp) returns 0 rows.
I believe that this is the cause of other issues I have, where I have missing
rows in queries with the timestamp (but not the explicit sort by).
Note that if instead of a temporary table I use a CTE for opens, the issue does
not appear.
I tried workarounds (inverse order of operands, adding _not_ or _not not_ ) to
no avail.
One thing that did work is to explicitly cast the string to a timestamp:
{code:java}
select count(*) from ( select * from opens where load_ts <= cast('2019-11-13
09:07:00' as timestamp) sort by id) t;{code}
It might be good practice indeed, but there still is a discrepancy between how
_>=_ and _<=_ are handled, or how _sort by_ works.
Note: this is on Hive from
[hdp3.1.4|[https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/release-notes/content/patch_hive.html]],
without llap.
was:
Consider this example.
Preparation:
{code:java}
create temporary table opens as (
select stack(1,
1 , cast('2019-11-13 08:07:28' as timestamp)
) as (id , load_ts )
);
{code}
Queries: This is just about counting the number of rows, with filters always
matching, and possibly sort by. 1 is always expected.
{code:java}
select count(*) from ( select * from opens) t;
select count(*) from ( select * from opens sort by id) t;
select count(*) from ( select * from opens where load_ts >= '2019-11-13
08:07:00' ) t;
select count(*) from ( select * from opens where load_ts >= '2019-11-13
08:07:00' sort by id) t;
select count(*) from ( select * from opens where load_ts <= '2019-11-13
09:07:00') t;
select count(*) from ( select * from opens where load_ts <= '2019-11-13
09:07:00' sort by id) t;
{code}
The latest query (_sort by_ and _<=_ on timestamp) returns 0 rows.
I believe that this is the cause of other issues I have, where I have missing
rows in queries with the timestamp (but not the explicit sort by).
Note that if instead of a temporary table I use a CTE for opens, the issue does
not appear.
I tried workarounds (inverse order of operands, adding _not_ or _not not_ ) to
no avail.
One thing that did work is to explicitly cast the string to a timestamp:
{code:java}
select count(*) from ( select * from opens where load_ts <= cast('2019-11-13
09:07:00' as timestamp) sort by id) t;{code}
It might be good practice indeed, but there still is a discrepancy between how
_>=_ and _<=_ are handled, or how _sort by_ works.
Note: this is on Hive from
[hdp3.1.4|[https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/release-notes/content/patch_hive.html]],
without llap.
> sort by and timestamp casting filters out too many rows
> ---
>
> Key: HIVE-22500
> URL: https://issues.apache.org/jira/browse/HIVE-22500
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Guillaume
>Priority: Major
>
> Consider this example.
> Preparation:
> {code:java}
> create temporary table opens as (
> select stack(1,
> 1 , cast('2019-11-13 08:07:28' as timestamp)
>) as (id , load_ts )
> );
> {code}
>
> Queries: This is just about counting the number of rows, with filters always
> matching, and possibly sort by. 1 is always expected.
> {code:java}
> select count(*) from ( select * from opens) t;
> select count(*) from ( select * from opens sort by id) t;
> select count(*) from ( select * from opens where load_ts >= '2019-11-13
> 08:07:00' ) t;
> select count(*) from ( select * from opens where load_ts >= '2019-11-13
> 08:07:00' sort by id) t;
> select count(*) from ( select * from opens where load_ts <= '2019-11-13
> 09:07:00') t;
> select count(*) from ( select * from opens where load_ts <= '2019-11-13
> 09:07:00' sort by id) t;
> {code}
> The last query (_sort by_ and _<=_ on timestamp) returns 0 rows.
> I believe that
[jira] [Updated] (HIVE-22500) sort by and timestamp casting filters out too many rows
[
https://issues.apache.org/jira/browse/HIVE-22500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guillaume updated HIVE-22500:
-
Description:
Consider this example.
Preparation:
{code:java}
create temporary table opens as (
select stack(1,
1 , cast('2019-11-13 08:07:28' as timestamp)
) as (id , load_ts )
);
{code}
Queries: This is just about counting the number of rows, with filters always
matching, and possibly sort by. 1 is always expected.
{code:java}
select count(*) from ( select * from opens) t;
select count(*) from ( select * from opens sort by id) t;
select count(*) from ( select * from opens where load_ts >= '2019-11-13
08:07:00' ) t;
select count(*) from ( select * from opens where load_ts >= '2019-11-13
08:07:00' sort by id) t;
select count(*) from ( select * from opens where load_ts <= '2019-11-13
09:07:00') t;
select count(*) from ( select * from opens where load_ts <= '2019-11-13
09:07:00' sort by id) t;
{code}
The latest query (_sort by_ and _<=_ on timestamp) returns 0 rows.
I believe that this is the cause of other issues I have, where I have missing
rows in queries with the timestamp (but not the explicit sort by).
Note that if instead of a temporary table I use a CTE for opens, the issue does
not appear.
I tried workarounds (inverse order of operands, adding _not_ or _not not_ ) to
no avail.
One thing that did work is to explicitly cast the string to a timestamp:
{code:java}
select count(*) from ( select * from opens where load_ts <= cast('2019-11-13
09:07:00' as timestamp) sort by id) t;{code}
It might be good practice indeed, but there still is a discrepancy between how
_>=_ and _<=_ are handled, or how _sort by_ works.
Note: this is on Hive from
[hdp3.1.4|[https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/release-notes/content/patch_hive.html]],
without llap.
was:
Consider this example.
Preparation:
{code:java}
create temporary table opens as (
select stack(1,
1 , cast('2019-11-13 08:07:28' as timestamp)
) as (id , load_ts )
);
{code}
Queries. This is just about counting the number of rows, with filters always
matching, and possibly sort by. 1 is always expected.
{code:java}
select count(*) from ( select * from opens) t;
select count(*) from ( select * from opens sort by id) t;
select count(*) from ( select * from opens where load_ts >= '2019-11-13
08:07:00' ) t;
select count(*) from ( select * from opens where load_ts >= '2019-11-13
08:07:00' sort by id) t;
select count(*) from ( select * from opens where load_ts <= '2019-11-13
09:07:00') t;
select count(*) from ( select * from opens where load_ts <= '2019-11-13
09:07:00' sort by id) t;
{code}
The latest query (_sort by_ and _<=_ on timestamp) returns 0 rows.
I believe that this is the cause of other issues I have, where I have missing
rows in queries with the timestamp (but not the explicit sort by).
Note that if instead of a temporary table I use a CTE for opens, the issue does
not appear.
I tried workarounds (inverse order of operands, adding _not_ or _not not_ ) to
no avail.
One thing that did work is to explicitly cast the string to a timestamp:
{code:java}
select count(*) from ( select * from opens where load_ts <= cast('2019-11-13
09:07:00' as timestamp) sort by id) t;{code}
It might be good practice indeed, but there still is a discrepancy between how
_>=_ and _<=_ are handled, or how _sort by_ works.
Note: this is on Hive from
[hdp3.1.4|[https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/release-notes/content/patch_hive.html]],
without llap.
> sort by and timestamp casting filters out too many rows
> ---
>
> Key: HIVE-22500
> URL: https://issues.apache.org/jira/browse/HIVE-22500
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Guillaume
>Priority: Major
>
> Consider this example.
> Preparation:
> {code:java}
> create temporary table opens as (
> select stack(1,
> 1 , cast('2019-11-13 08:07:28' as timestamp)
>) as (id , load_ts )
> );
> {code}
>
> Queries: This is just about counting the number of rows, with filters always
> matching, and possibly sort by. 1 is always expected.
> {code:java}
> select count(*) from ( select * from opens) t;
> select count(*) from ( select * from opens sort by id) t;
> select count(*) from ( select * from opens where load_ts >= '2019-11-13
> 08:07:00' ) t;
> select count(*) from ( select * from opens where load_ts >= '2019-11-13
> 08:07:00' sort by id) t;
> select count(*) from ( select * from opens where load_ts <= '2019-11-13
> 09:07:00') t;
> select count(*) from ( select * from opens where load_ts <= '2019-11-13
> 09:07:00' sort by id) t;
> {code}
>
> The latest query (_sort by_ and _<=_ on timestamp) returns 0 rows.
> I believe
[jira] [Updated] (HIVE-22500) sort by and timestamp casting filters out too many rows
[
https://issues.apache.org/jira/browse/HIVE-22500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guillaume updated HIVE-22500:
-
Affects Version/s: (was: 3.1.1)
3.1.0
> sort by and timestamp casting filters out too many rows
> ---
>
> Key: HIVE-22500
> URL: https://issues.apache.org/jira/browse/HIVE-22500
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Guillaume
>Priority: Major
>
> Consider this example.
> Preparation:
>
> {code:java}
> create temporary table opens as (
> select stack(1,
> 1 , cast('2019-11-13 08:07:28' as timestamp)
>) as (id , load_ts )
> );
> {code}
> Queries. This is just about counting the number of rows, with filters always
> matching, and possibly sort by. 1 is always expected.
>
> {code:java}
> select count(*) from ( select * from opens) t;
> select count(*) from ( select * from opens sort by id) t;
> select count(*) from ( select * from opens where load_ts >= '2019-11-13
> 08:07:00' ) t;
> select count(*) from ( select * from opens where load_ts >= '2019-11-13
> 08:07:00' sort by id) t;
> select count(*) from ( select * from opens where load_ts <= '2019-11-13
> 09:07:00') t;
> select count(*) from ( select * from opens where load_ts <= '2019-11-13
> 09:07:00' sort by id) t;
> {code}
>
> The latest query (_sort by_ and _<=_ on timestamp) returns 0 rows.
> I believe that this is the cause of other issues I have, where I have missing
> rows in queries with the timestamp (but not the explicit sort by).
> Note that if instead of a temporary table I use a CTE for opens, the issue
> does not appear.
> I tried workarounds (inverse order of operands, adding _not_ or _not not_ )
> to no avail.
> One thing that did work is to explicitly cast the string to a timestamp:
> {code:java}
> select count(*) from ( select * from opens where load_ts <= cast('2019-11-13
> 09:07:00' as timestamp) sort by id) t;{code}
> It might be good practice indeed, but there still is a discrepancy between
> how _>=_ and _<=_ are handled, or how _sort by_ works.
> Note: this is on Hive from
> [hdp3.1.4|[https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/release-notes/content/patch_hive.html]],
> without llap.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21146) Enforce TransactionBatch size=1 for blob stores
[
https://issues.apache.org/jira/browse/HIVE-21146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Lavati updated HIVE-21146:
Attachment: HIVE-21146.2.patch
> Enforce TransactionBatch size=1 for blob stores
> ---
>
> Key: HIVE-21146
> URL: https://issues.apache.org/jira/browse/HIVE-21146
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21146.2.patch, HIVE-21146.2.patch,
> HIVE-21146.2.patch, HIVE-21146.2.patch, HIVE-21146.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Streaming Ingest API supports a concept of {{TransactionBatch}} where N
> transactions can be opened at once and the data in all of them will be
> written to the same delta_x_y directory where each transaction in the batch
> can be committed/aborted independently. The implementation relies on
> {{FSDataOutputStream.hflush()}} (called from OrcRecordUpdater}} which is
> available on HDFS but is often implemented as no-op in Blob store backed
> {{FileSystem}} objects.
> Need to add a check to {{HiveStreamingConnection()}} constructor to raise an
> error if {{builder.transactionBatchSize > 1}} and the target table/partitions
> are backed by something that doesn't support {{hflush()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21146) Enforce TransactionBatch size=1 for blob stores
[
https://issues.apache.org/jira/browse/HIVE-21146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974946#comment-16974946
]
Hive QA commented on HIVE-21146:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985875/HIVE-21146.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 17704 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
org.apache.hive.service.server.TestHS2HttpServer.testApiServletActiveSessions
(batchId=240)
org.apache.hive.service.server.TestHS2HttpServer.testApiServletHistoricalQueries
(batchId=240)
org.apache.hive.service.server.TestHS2HttpServerPam.org.apache.hive.service.server.TestHS2HttpServerPam
(batchId=240)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19434/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19434/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19434/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985875 - PreCommit-HIVE-Build
> Enforce TransactionBatch size=1 for blob stores
> ---
>
> Key: HIVE-21146
> URL: https://issues.apache.org/jira/browse/HIVE-21146
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21146.2.patch, HIVE-21146.2.patch,
> HIVE-21146.2.patch, HIVE-21146.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Streaming Ingest API supports a concept of {{TransactionBatch}} where N
> transactions can be opened at once and the data in all of them will be
> written to the same delta_x_y directory where each transaction in the batch
> can be committed/aborted independently. The implementation relies on
> {{FSDataOutputStream.hflush()}} (called from OrcRecordUpdater}} which is
> available on HDFS but is often implemented as no-op in Blob store backed
> {{FileSystem}} objects.
> Need to add a check to {{HiveStreamingConnection()}} constructor to raise an
> error if {{builder.transactionBatchSize > 1}} and the target table/partitions
> are backed by something that doesn't support {{hflush()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21146) Enforce TransactionBatch size=1 for blob stores
[
https://issues.apache.org/jira/browse/HIVE-21146?focusedWorklogId=344061=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-344061
]
ASF GitHub Bot logged work on HIVE-21146:
-
Author: ASF GitHub Bot
Created on: 15/Nov/19 08:50
Start Date: 15/Nov/19 08:50
Worklog Time Spent: 10m
Work Description: dlavati commented on issue #797: HIVE-21146 Enforce
TransactionBatch size=1 for blob stores
URL: https://github.com/apache/hive/pull/797#issuecomment-554269771
@t3rmin4t0r, @prasanthj is this something you could review please?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 344061)
Time Spent: 20m (was: 10m)
> Enforce TransactionBatch size=1 for blob stores
> ---
>
> Key: HIVE-21146
> URL: https://issues.apache.org/jira/browse/HIVE-21146
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21146.2.patch, HIVE-21146.2.patch,
> HIVE-21146.2.patch, HIVE-21146.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Streaming Ingest API supports a concept of {{TransactionBatch}} where N
> transactions can be opened at once and the data in all of them will be
> written to the same delta_x_y directory where each transaction in the batch
> can be committed/aborted independently. The implementation relies on
> {{FSDataOutputStream.hflush()}} (called from OrcRecordUpdater}} which is
> available on HDFS but is often implemented as no-op in Blob store backed
> {{FileSystem}} objects.
> Need to add a check to {{HiveStreamingConnection()}} constructor to raise an
> error if {{builder.transactionBatchSize > 1}} and the target table/partitions
> are backed by something that doesn't support {{hflush()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
