[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927449#comment-16927449
]
Denys Kuzmenko commented on HIVE-15956:
---
Thank you, [~kgyrtkirk]!
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Denys Kuzmenko
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-15956.2.patch, HIVE-15956.3.patch, HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927435#comment-16927435
]
Zoltan Haindrich commented on HIVE-15956:
-
+1
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-15956.2.patch, HIVE-15956.3.patch, HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926512#comment-16926512
]
Hive QA commented on HIVE-15956:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12979943/HIVE-15956.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16751 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18520/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18520/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18520/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12979943 - PreCommit-HIVE-Build
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-15956.2.patch, HIVE-15956.3.patch, HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926482#comment-16926482
]
Hive QA commented on HIVE-15956:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
18s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 179 unchanged - 2 fixed = 180 total (was 181) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | org.apache.hadoop.hive.metastore.ObjectStore$18.run(List) concatenates
strings using + in a loop At ObjectStore.java:in a loop At
ObjectStore.java:[line 8771] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18520/dev-support/hive-personality.sh
|
| git revision | master / 1dfa2d8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18520/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18520/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-15956.2.patch, HIVE-15956.3.patch, HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
>
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926465#comment-16926465
]
Denys Kuzmenko commented on HIVE-15956:
---
[~kgyrtkirk] could you please review. I have used provided above qtest file for
verification. 10k partitions, batch size 200
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-15956.2.patch, HIVE-15956.3.patch, HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926313#comment-16926313
]
Hive QA commented on HIVE-15956:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12979885/HIVE-15956.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16750 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18515/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18515/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18515/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12979885 - PreCommit-HIVE-Build
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-15956.2.patch, HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926285#comment-16926285
]
Hive QA commented on HIVE-15956:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 412 unchanged - 0 fixed = 413 total (was 412) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
14s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 179 unchanged - 2 fixed = 180 total (was 181) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | org.apache.hadoop.hive.metastore.ObjectStore$18.run(List) concatenates
strings using + in a loop At ObjectStore.java:in a loop At
ObjectStore.java:[line 8771] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18515/dev-support/hive-personality.sh
|
| git revision | master / d244da5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18515/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18515/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18515/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-15956.2.patch, HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409383#comment-16409383
]
Ganesha Shreedhara commented on HIVE-15956:
---
I tested that this patch works fine by dropping the partitions in batches. This
will help when there are thousands of partitions to drop. I tested dropping
~40k partitions in the batches of 500.
It would be great if someone can review this patch.
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
>Priority: Major
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248358#comment-16248358
]
Hive QA commented on HIVE-15956:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856246/HIVE-15956.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7773/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7773/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7773/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-11-11 06:20:33.720
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-7773/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-11-11 06:20:33.723
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 67888cf HIVE-17995 Run checkstyle on standalone-metastore module
with proper configuration (Adam Szita via Alan Gates)
+ git clean -f -d
Removing ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java
Removing ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForOrcMmTable.java
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 67888cf HIVE-17995 Run checkstyle on standalone-metastore module
with proper configuration (Adam Szita via Alan Gates)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-11-11 06:20:36.072
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java: No
such file or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12856246 - PreCommit-HIVE-Build
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
>
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248098#comment-16248098
]
Ashutosh Chauhan commented on HIVE-15956:
-
[~niklaus.xiao] I can repro the issue with test script you provided. But your
patch doesnt seem to fix it. I tried with batch-size 10, 100, 1000
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944220#comment-15944220
]
Zoltan Haindrich commented on HIVE-15956:
-
[~niklaus.xiao] to be on the same page...I think it would be good if I share
what I've used during testing:
I used this script to generate a qfile
{code}
O=ql/src/test/queries/clientpositive/dp.q
cat > $O << EOF
create table test_partition(id int) partitioned by (dt int);
EOF
for i in `seq 1 1`;do
echo "alter table test_partition add partition(dt=$i);"
done >> $O
cat >> $O << EOF
alter table test_partition drop partition(dt<9000);
EOF
{code}
and executed with:
{code}
time mvn install -Dtest=TestCliDriver -DskipSparkTests -Pitests -q
-Dtest.output.overwrite -Dqfile=dp.q -DinitScript=asd -T4
{code}
I think the patch does work; I've checked it via a unit test written against
metastore...but I'm not sure why this qtest doesn't confirm it...
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943339#comment-15943339
]
Zoltan Haindrich commented on HIVE-15956:
-
[~niklaus.xiao] sorry for the late response...yes I've tried it with 100 ; and
it still produced the same issue, I'll take a look again; and peek into it
whenever this could be covered with a unit test or notmy test runs >10
minutes
I've peeked into if a unit test can be created for this case...but it seems to
me that {{dropPartitions}} is currently not covered (or I'm looking at the
wrong files :)
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923852#comment-15923852
]
Niklaus Xiao commented on HIVE-15956:
-
[~kgyrtkirk] This patch works for me, maybe the default batch size is too large
for you, you can change {{hive.metastore.drop.partitions.batch.size}} to 200
and try again.
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906444#comment-15906444
]
Zoltan Haindrich commented on HIVE-15956:
-
[~niklaus.xiao] I've tried your patch...and I've still seen the problem after
applying the fix. Did it work for you? - it might be possible that I've
screwed something up - but after a clean rebuild it still failed
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897176#comment-15897176
]
Niklaus Xiao commented on HIVE-15956:
-
Tests failure not related.
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897130#comment-15897130
]
Hive QA commented on HIVE-15956:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856246/HIVE-15956.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10327 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=229)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table]
(batchId=147)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=224)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=224)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in]
(batchId=119)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3959/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3959/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3959/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12856246 - PreCommit-HIVE-Build
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15956) StackOverflowError when drop lots of partitions
[
https://issues.apache.org/jira/browse/HIVE-15956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896962#comment-15896962
]
Niklaus Xiao commented on HIVE-15956:
-
Add logic to drop partitions in batches, default batch size is 1000.
> StackOverflowError when drop lots of partitions
> ---
>
> Key: HIVE-15956
> URL: https://issues.apache.org/jira/browse/HIVE-15956
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Niklaus Xiao
> Attachments: HIVE-15956.patch
>
>
> Repro steps:
> 1. Create partitioned table and add 1 partitions
> {code}
> create table test_partition(id int) partitioned by (dt int);
> alter table test_partition add partition(dt=1);
> alter table test_partition add partition(dt=3);
> alter table test_partition add partition(dt=4);
> ...
> alter table test_partition add partition(dt=1);
> {code}
> 2. Drop 9000 partitions:
> {code}
> alter table test_partition drop partition(dt<9000);
> {code}
> Step 2 will fail with StackOverflowError:
> {code}
> Exception in thread "pool-7-thread-161" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.ExpressionCompiler.isOperator(ExpressionCompiler.java:819)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:190)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileOrAndExpression(ExpressionCompiler.java:192)
> at
> org.datanucleus.query.expression.ExpressionCompiler.compileExpression(ExpressionCompiler.java:179)
> {code}
> {code}
> Exception in thread "pool-7-thread-198" java.lang.StackOverflowError
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:83)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> at
> org.datanucleus.query.expression.DyadicExpression.bind(DyadicExpression.java:87)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
