[jira] [Commented] (HIVE-16605) Enforce NOT NULL constraints
[ https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355084#comment-16355084 ] Hive QA commented on HIVE-16605: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12909328/HIVE-16605.5.patch {color:green}SUCCESS:{color} +1 due to 10 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 12987 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=49) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=221) org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded] (batchId=206) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280) org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap (batchId=282) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9063/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9063/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9063/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 23 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12909328 - PreCommit-HIVE-Build > Enforce NOT NULL constraints > > > Key: HIVE-16605 > URL: https://issues.apache.org/jira/browse/HIVE-16605 > Project: Hive > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Carter Shanklin >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-16605.1.patch, HIVE-16605.2.patch, > HIVE-16605.3.patch, HIVE-16605.4.patch, HIVE-16605.5.patch > > > Since NOT NULL is so common it would be great to have tables start to enforce > that. > [~ekoifman] described a possible approach in HIVE-16575: > {quote} > One way to enforce not null constraint is to have the optimizer add > enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass > through. > So if 'b' has not null constraint, > Insert into T select a,b,c... would become > Insert into T select a, enforce_not_null(b), c. > This would work for any table type. > {quote} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18464) Fix TestSSL test failures
[ https://issues.apache.org/jira/browse/HIVE-18464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-18464: -- Assignee: Vihang Karajgaonkar > Fix TestSSL test failures > - > > Key: HIVE-18464 > URL: https://issues.apache.org/jira/browse/HIVE-18464 > Project: Hive > Issue Type: Test >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > > TestSSL test has been failing for a while now. Following three tests fail > everytime precommit job runs. > org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232) > org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232) > org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[ https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-18421: --- Attachment: (was: HIVE-18421.06.patch) > Vectorized execution handles overflows in a different manner than > non-vectorized execution > -- > > Key: HIVE-18421 > URL: https://issues.apache.org/jira/browse/HIVE-18421 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch, > HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch, > HIVE-18421.06.patch > > > In vectorized execution arithmetic operations which cause integer overflows > can give wrong results. Issue is reproducible in both Orc and parquet. > Simple test case to reproduce this issue > {noformat} > set hive.vectorized.execution.enabled=true; > create table parquettable (t1 tinyint, t2 tinyint) stored as parquet; > insert into parquettable values (-104, 25), (-112, 24), (54, 9); > select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by > diff desc; > +---+-+---+ > | t1 | t2 | diff | > +---+-+---+ > | -104 | 25 | 127 | > | -112 | 24 | 120 | > | 54| 9 | 45| > +---+-+---+ > {noformat} > When vectorization is turned off the same query produces only one row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[ https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-18421: --- Attachment: HIVE-18421.06.patch > Vectorized execution handles overflows in a different manner than > non-vectorized execution > -- > > Key: HIVE-18421 > URL: https://issues.apache.org/jira/browse/HIVE-18421 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch, > HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch, > HIVE-18421.06.patch > > > In vectorized execution arithmetic operations which cause integer overflows > can give wrong results. Issue is reproducible in both Orc and parquet. > Simple test case to reproduce this issue > {noformat} > set hive.vectorized.execution.enabled=true; > create table parquettable (t1 tinyint, t2 tinyint) stored as parquet; > insert into parquettable values (-104, 25), (-112, 24), (54, 9); > select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by > diff desc; > +---+-+---+ > | t1 | t2 | diff | > +---+-+---+ > | -104 | 25 | 127 | > | -112 | 24 | 120 | > | 54| 9 | 45| > +---+-+---+ > {noformat} > When vectorization is turned off the same query produces only one row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly
[ https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18622: Attachment: (was: HIVE-18622.02.patch) > Vectorization: IF Statements, Comparisons, and more do not handle NULLs > correctly > - > > Key: HIVE-18622 > URL: https://issues.apache.org/jira/browse/HIVE-18622 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-18622.03.patch > > > > Many vector expression classes are missing guards around setting noNulls > among other things. > {code:java} > // Carefully update noNulls... > if (outputColVector.noNulls) { > outputColVector.noNulls = inputColVector.noNulls; > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly
[ https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18622: Attachment: HIVE-18622.03.patch > Vectorization: IF Statements, Comparisons, and more do not handle NULLs > correctly > - > > Key: HIVE-18622 > URL: https://issues.apache.org/jira/browse/HIVE-18622 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-18622.03.patch > > > > Many vector expression classes are missing guards around setting noNulls > among other things. > {code:java} > // Carefully update noNulls... > if (outputColVector.noNulls) { > outputColVector.noNulls = inputColVector.noNulls; > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly
[ https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18622: Status: Patch Available (was: In Progress) Still a Work-In-Progress. > Vectorization: IF Statements, Comparisons, and more do not handle NULLs > correctly > - > > Key: HIVE-18622 > URL: https://issues.apache.org/jira/browse/HIVE-18622 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-18622.03.patch > > > > Many vector expression classes are missing guards around setting noNulls > among other things. > {code:java} > // Carefully update noNulls... > if (outputColVector.noNulls) { > outputColVector.noNulls = inputColVector.noNulls; > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly
[ https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18622: Attachment: (was: HIVE-18622.01.patch) > Vectorization: IF Statements, Comparisons, and more do not handle NULLs > correctly > - > > Key: HIVE-18622 > URL: https://issues.apache.org/jira/browse/HIVE-18622 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-18622.03.patch > > > > Many vector expression classes are missing guards around setting noNulls > among other things. > {code:java} > // Carefully update noNulls... > if (outputColVector.noNulls) { > outputColVector.noNulls = inputColVector.noNulls; > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly
[ https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18622: Status: In Progress (was: Patch Available) > Vectorization: IF Statements, Comparisons, and more do not handle NULLs > correctly > - > > Key: HIVE-18622 > URL: https://issues.apache.org/jira/browse/HIVE-18622 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-18622.01.patch, HIVE-18622.02.patch > > > > Many vector expression classes are missing guards around setting noNulls > among other things. > {code:java} > // Carefully update noNulls... > if (outputColVector.noNulls) { > outputColVector.noNulls = inputColVector.noNulls; > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16605) Enforce NOT NULL constraints
[ https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355042#comment-16355042 ] Hive QA commented on HIVE-16605: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 1s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 37s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 43s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 3s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 47s{color} | {color:red} ql: The patch generated 60 new + 981 unchanged - 271 fixed = 1041 total (was 1252) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 16m 42s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 2422e18 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9063/yetus/diff-checkstyle-ql.txt | | modules | C: common ql itests U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9063/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Enforce NOT NULL constraints > > > Key: HIVE-16605 > URL: https://issues.apache.org/jira/browse/HIVE-16605 > Project: Hive > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Carter Shanklin >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-16605.1.patch, HIVE-16605.2.patch, > HIVE-16605.3.patch, HIVE-16605.4.patch, HIVE-16605.5.patch > > > Since NOT NULL is so common it would be great to have tables start to enforce > that. > [~ekoifman] described a possible approach in HIVE-16575: > {quote} > One way to enforce not null constraint is to have the optimizer add > enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass > through. > So if 'b' has not null constraint, > Insert into T select a,b,c... would become > Insert into T select a, enforce_not_null(b), c. > This would work for any table type. > {quote} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18626) Repl load "with" clause does not pass config to tasks
[ https://issues.apache.org/jira/browse/HIVE-18626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355025#comment-16355025 ] Daniel Dai commented on HIVE-18626: --- Address checkstyle issue. > Repl load "with" clause does not pass config to tasks > - > > Key: HIVE-18626 > URL: https://issues.apache.org/jira/browse/HIVE-18626 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-18626.1.patch, HIVE-18626.2.patch > > > The "with" clause in repl load suppose to pass custom hive config entries to > replication. However, the config is only effective in > BootstrapEventsIterator, but not the generated tasks (such as MoveTask, > DDLTask). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18626) Repl load "with" clause does not pass config to tasks
[ https://issues.apache.org/jira/browse/HIVE-18626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-18626: -- Attachment: HIVE-18626.2.patch > Repl load "with" clause does not pass config to tasks > - > > Key: HIVE-18626 > URL: https://issues.apache.org/jira/browse/HIVE-18626 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-18626.1.patch, HIVE-18626.2.patch > > > The "with" clause in repl load suppose to pass custom hive config entries to > replication. However, the config is only effective in > BootstrapEventsIterator, but not the generated tasks (such as MoveTask, > DDLTask). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18611) Avoid memory allocation of aggregation buffer during stats computation
[ https://issues.apache.org/jira/browse/HIVE-18611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355014#comment-16355014 ] Hive QA commented on HIVE-18611: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12909331/HIVE-18611.2.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 12956 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver (batchId=88) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=49) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=221) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280) org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap (batchId=282) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9062/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9062/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9062/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 23 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12909331 - PreCommit-HIVE-Build > Avoid memory allocation of aggregation buffer during stats computation > --- > > Key: HIVE-18611 > URL: https://issues.apache.org/jira/browse/HIVE-18611 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Statistics >Affects Versions: 3.0.0, 2.3.2 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > Labels: performance > Attachments: HIVE-18611.2.patch, HIVE-18611.patch > > > Bloom filter aggregation buffer may result in allocation of upto ~594MB array > which is unnecessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18626) Repl load "with" clause does not pass config to tasks
[ https://issues.apache.org/jira/browse/HIVE-18626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354985#comment-16354985 ] Thejas M Nair commented on HIVE-18626: -- +1 pending above comments > Repl load "with" clause does not pass config to tasks > - > > Key: HIVE-18626 > URL: https://issues.apache.org/jira/browse/HIVE-18626 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-18626.1.patch > > > The "with" clause in repl load suppose to pass custom hive config entries to > replication. However, the config is only effective in > BootstrapEventsIterator, but not the generated tasks (such as MoveTask, > DDLTask). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18611) Avoid memory allocation of aggregation buffer during stats computation
[ https://issues.apache.org/jira/browse/HIVE-18611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354984#comment-16354984 ] Hive QA commented on HIVE-18611: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 1s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 40s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 54s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 49s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s{color} | {color:red} storage-api: The patch generated 2 new + 7 unchanged - 2 fixed = 9 total (was 9) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 37s{color} | {color:red} ql: The patch generated 1 new + 146 unchanged - 1 fixed = 147 total (was 147) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 56s{color} | {color:red} ql generated 1 new + 99 unchanged - 1 fixed = 100 total (was 100) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 16m 15s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 2422e18 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9062/yetus/diff-checkstyle-storage-api.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9062/yetus/diff-checkstyle-ql.txt | | javadoc | http://104.198.109.242/logs//PreCommit-HIVE-Build-9062/yetus/diff-javadoc-javadoc-ql.txt | | modules | C: storage-api ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9062/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Avoid memory allocation of aggregation buffer during stats computation > --- > > Key: HIVE-18611 > URL: https://issues.apache.org/jira/browse/HIVE-18611 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Statistics >Affects Versions: 3.0.0, 2.3.2 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > Labels: performance > Attachments: HIVE-18611.2.patch, HIVE-18611.patch > > > Bloom filter aggregation buffer may result in allocation of upto ~594MB array > which is unnecessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[ https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354971#comment-16354971 ] Hive QA commented on HIVE-18625: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12909329/HIVE-18625.1.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 12976 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=49) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=221) org.apache.hadoop.hive.metastore.TestAcidTableSetup.testTransactionalValidation (batchId=223) org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded] (batchId=206) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280) org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap (batchId=282) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConnectionSchemaAPIs (batchId=238) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpHeaderSize (batchId=238) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerDAGTotalTasks (batchId=235) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9061/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9061/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9061/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 27 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12909329 - PreCommit-HIVE-Build > SessionState Not Checking For Directory Creation Result > --- > > Key: HIVE-18625 > URL: https://issues.apache.org/jira/browse/HIVE-18625 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0, 2.4.0, 2.3.2 >Reporter: BELUGA BEHR >Assignee: Andrew Sherman >Priority: Minor > Attachments: HIVE-18625.1.patch > > > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773 > {code:java} > private static void createPath(HiveConf conf, Path path, String permission, > boolean isLocal, > boolean isCleanUp) throws IOException { > FsPermission fsPermission = new FsPermission(permission); > FileSystem fs; > if (isLocal) { > fs = FileSystem.getLocal(conf); > } else { > fs = path.getFileSystem(conf); > } > if (!fs.exists(path)) { > fs.mkdirs(path, fsPermission); > String dirType = isLocal ? "local" : "HDFS"; > LOG.info("Created " + dirType + " directory: " + path.toString()); > } > if (isCleanUp) { > fs.deleteOnExit(path); > } > } > {code} > The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value > indicating if the direc
[jira] [Updated] (HIVE-18467) support whole warehouse dump / load + create/drop database events
[ https://issues.apache.org/jira/browse/HIVE-18467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek updated HIVE-18467: --- Resolution: Fixed Status: Resolved (was: Patch Available) > support whole warehouse dump / load + create/drop database events > - > > Key: HIVE-18467 > URL: https://issues.apache.org/jira/browse/HIVE-18467 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18467.0.patch, HIVE-18467.1.patch, > HIVE-18467.2.patch, HIVE-18467.5.patch, HIVE-18467.6.patch > > > A complete hive warehouse might be required to replicate to a DR site for > certain use cases and rather than allowing only a database name in the REPL > DUMP commands, we should allow dumping of all databases using the "*" option > as in > _REPL DUMP *_ > On the repl load side there will not be an option to specify the database > name when loading from a location used to dump multiple databases, hence only > _REPL LOAD FROM [location]_ would be supported when dumping via _REPL DUMP *_ > Additionally, incremental dumps will go through all events across databases > in a warehouse and hence CREATE / DROP Database events have to be serialized > correctly to allow repl load to create them correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18467) support whole warehouse dump / load + create/drop database events
[ https://issues.apache.org/jira/browse/HIVE-18467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354952#comment-16354952 ] anishek commented on HIVE-18467: Thanks [~sankarh] for the review, the tests failures are not relevant to this patch. Patch is committed to master ! > support whole warehouse dump / load + create/drop database events > - > > Key: HIVE-18467 > URL: https://issues.apache.org/jira/browse/HIVE-18467 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18467.0.patch, HIVE-18467.1.patch, > HIVE-18467.2.patch, HIVE-18467.5.patch, HIVE-18467.6.patch > > > A complete hive warehouse might be required to replicate to a DR site for > certain use cases and rather than allowing only a database name in the REPL > DUMP commands, we should allow dumping of all databases using the "*" option > as in > _REPL DUMP *_ > On the repl load side there will not be an option to specify the database > name when loading from a location used to dump multiple databases, hence only > _REPL LOAD FROM [location]_ would be supported when dumping via _REPL DUMP *_ > Additionally, incremental dumps will go through all events across databases > in a warehouse and hence CREATE / DROP Database events have to be serialized > correctly to allow repl load to create them correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354062#comment-16354062 ] Sankar Hariappan edited comment on HIVE-18192 at 2/7/18 4:18 AM: - Added 09.patch with below updates. # Remove entries from TXNS_TO_WRITE_ID table when drop a table/database. Also, split db and table names into 2 columns. # Multi-statement txn support by setting ValidTxnList for open txn and get/set ValidWriteId list for each operation within txn. Pending changes for this feature # Scripts to add Metastore tables for write id management. Add for other non-derby databases too. # CompactionTxnHandler cleanup COMPLETED_TXN_COMPONENTS based on highest write id instead of highest txn id. # Address review comments from Eugene. # Address few more test failures. # Fix check style issues. Out of scope of HIVE-18192: # Cleaner for TXNS_TO_WRITE_ID table entries. Also, need to maintain the LWM for each table write id. # Update classes to use WriteID instead of TxnId as methods and variable names. was (Author: sankarh): Added 09.patch with below updates. # Remove entries from TXNS_TO_WRITE_ID table when drop a table/database. Also, split db and table names into 2 columns. # Multi-statement txn support by setting ValidTxnList for open txn and get/set ValidWriteId list for each operation within txn. Pending changes for this feature # Scripts to add Metastore tables for write id management. Add for other non-derby databases too. # CompactionTxnHandler cleanup COMPLETED_TXN_COMPONENTS based on highest write id instead of highest txn id. # Address few more test failures. # Fix check style issues. Out of scope of HIVE-18192: # Cleaner for TXNS_TO_WRITE_ID table entries. Also, need to maintain the LWM for each table write id. # Update classes to use WriteID instead of TxnId as methods and variable names. > Introduce WriteID per table rather than using global transaction ID > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18192.01.patch, HIVE-18192.02.patch, > HIVE-18192.03.patch, HIVE-18192.04.patch, HIVE-18192.05.patch, > HIVE-18192.06.patch, HIVE-18192.07.patch, HIVE-18192.08.patch, > HIVE-18192.09.patch > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > For each table modified by the given transaction will have a table level > write ID allocated and a persisted map of global txn id -> to table -> write > id for that table has to be maintained to allow Snapshot isolation. > Readers should use the combination of ValidTxnList and > ValidWriteIdList(Table) for snapshot isolation. > > [Hive Replication - ACID > Tables.pdf|https://issues.apache.org/jira/secure/attachment/12903157/Hive%20Replication-%20ACID%20Tables.pdf] > has a section "Per Table Sequences (Write-Id)" with more detials -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18636) fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
[ https://issues.apache.org/jira/browse/HIVE-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18636: -- Status: Patch Available (was: Open) [~sershe] could you review please > fix TestTxnNoBuckets.testCTAS - keeps failing on ptest > -- > > Key: HIVE-18636 > URL: https://issues.apache.org/jira/browse/HIVE-18636 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-18636.01.patch > > > need to update expected result -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18636) fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
[ https://issues.apache.org/jira/browse/HIVE-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18636: -- Attachment: HIVE-18636.01.patch > fix TestTxnNoBuckets.testCTAS - keeps failing on ptest > -- > > Key: HIVE-18636 > URL: https://issues.apache.org/jira/browse/HIVE-18636 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-18636.01.patch > > > need to update expected result -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[ https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354943#comment-16354943 ] Hive QA commented on HIVE-18625: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 16s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 36s{color} | {color:red} ql: The patch generated 1 new + 75 unchanged - 0 fixed = 76 total (was 75) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 14m 11s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 3972bf0 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9061/yetus/diff-checkstyle-ql.txt | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9061/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > SessionState Not Checking For Directory Creation Result > --- > > Key: HIVE-18625 > URL: https://issues.apache.org/jira/browse/HIVE-18625 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0, 2.4.0, 2.3.2 >Reporter: BELUGA BEHR >Assignee: Andrew Sherman >Priority: Minor > Attachments: HIVE-18625.1.patch > > > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773 > {code:java} > private static void createPath(HiveConf conf, Path path, String permission, > boolean isLocal, > boolean isCleanUp) throws IOException { > FsPermission fsPermission = new FsPermission(permission); > FileSystem fs; > if (isLocal) { > fs = FileSystem.getLocal(conf); > } else { > fs = path.getFileSystem(conf); > } > if (!fs.exists(path)) { > fs.mkdirs(path, fsPermission); > String dirType = isLocal ? "local" : "HDFS"; > LOG.info("Created " + dirType + " directory: " + path.toString()); > } > if (isCleanUp) { > fs.deleteOnExit(path); > } > } > {code} > The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value > indicating if the directory creation was successful or not. Hive ignores > this return value and therefore could be acting on a directory that doesn't > exist. > Please capture the result, check it, and throw an Exception if it failed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18467) support whole warehouse dump / load + create/drop database events
[ https://issues.apache.org/jira/browse/HIVE-18467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354925#comment-16354925 ] Sankar Hariappan commented on HIVE-18467: - +1 Patch looks good to me. Also, the test failures seems irrelevant. > support whole warehouse dump / load + create/drop database events > - > > Key: HIVE-18467 > URL: https://issues.apache.org/jira/browse/HIVE-18467 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18467.0.patch, HIVE-18467.1.patch, > HIVE-18467.2.patch, HIVE-18467.5.patch, HIVE-18467.6.patch > > > A complete hive warehouse might be required to replicate to a DR site for > certain use cases and rather than allowing only a database name in the REPL > DUMP commands, we should allow dumping of all databases using the "*" option > as in > _REPL DUMP *_ > On the repl load side there will not be an option to specify the database > name when loading from a location used to dump multiple databases, hence only > _REPL LOAD FROM [location]_ would be supported when dumping via _REPL DUMP *_ > Additionally, incremental dumps will go through all events across databases > in a warehouse and hence CREATE / DROP Database events have to be serialized > correctly to allow repl load to create them correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18553) VectorizedParquetReader fails after adding a new column to table
[ https://issues.apache.org/jira/browse/HIVE-18553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354916#comment-16354916 ] Hive QA commented on HIVE-18553: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12909348/test_result_based_on_HIVE-18553.xlsx {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9060/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9060/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9060/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2018-02-07 03:40:44.316 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-9060/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2018-02-07 03:40:44.320 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 3972bf0 HIVE-18613: Extend JsonSerDe to support BINARY type (Jesus Camacho Rodriguez, reviewed by Prasanth Jayachandran) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 3972bf0 HIVE-18613: Extend JsonSerDe to support BINARY type (Jesus Camacho Rodriguez, reviewed by Prasanth Jayachandran) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2018-02-07 03:40:46.829 + rm -rf ../yetus + mkdir ../yetus + git gc + cp -R . ../yetus + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9060/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch fatal: unrecognized input fatal: unrecognized input fatal: unrecognized input The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12909348 - PreCommit-HIVE-Build > VectorizedParquetReader fails after adding a new column to table > > > Key: HIVE-18553 > URL: https://issues.apache.org/jira/browse/HIVE-18553 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.4.0, 2.3.2 >Reporter: Vihang Karajgaonkar >Assignee: Ferdinand Xu >Priority: Major > Attachments: HIVE-18553.2.patch, HIVE-18553.3.patch, > HIVE-18553.4.patch, HIVE-18553.patch, test_result_based_on_HIVE-18553.xlsx > > > VectorizedParquetReader throws an exception when trying to reading from a > parquet table on which new columns are added. Steps to reproduce below: > {code} > 0: jdbc:hive2://localhost:1/default> desc test_p; > +---++--+ > | col_name | data_type | comment | > +---++--+ > | t1| tinyint| | > | t2| tinyint| | > | i1| int| | > | i2| int| | > +---++--+ > 0: jdbc:hive2://localhost:1/default> set hive.fetch.task.conversion=none; > 0: jdbc:hive2://localhost:1/default> set > hive.vectorized.execution.enabled=true; > 0: jdbc:hive2://localhost:1/default> alter table test_p add columns (ts > timestamp); > 0: jdbc:hive2://localhost:1/default> select * from test_p; > Error: Error while processing statement: FAILED: Execution Error, return code > 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2) > {code} > Following exception is seen in the logs > {code} > Caused by: java.lang.IllegalArgumentException: [ts] BINARY is not in the > store: [[i1] INT32, [i2] INT3
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354914#comment-16354914 ] Hive QA commented on HIVE-18629: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12909347/HIVE-18629.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 12974 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=221) org.apache.hadoop.hive.metastore.client.TestAddAlterDropIndexes.testDropIndexInvalidDB[Embedded] (batchId=206) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280) org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap (batchId=282) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9059/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9059/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9059/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 22 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12909347 - PreCommit-HIVE-Build > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18623) Hive throws an exception "Renames across Mount points not supported" when running in a federated cluster
[ https://issues.apache.org/jira/browse/HIVE-18623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yangfang updated HIVE-18623: Affects Version/s: (was: 2.3.1) (was: 2.3.0) 2.3.3 3.0.0 > Hive throws an exception "Renames across Mount points not supported" when > running in a federated cluster > > > Key: HIVE-18623 > URL: https://issues.apache.org/jira/browse/HIVE-18623 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 2.2.0, 3.0.0, 2.3.2, 2.3.3 > Environment: hadoop 2.7.5, HDFS Federation enabled > hive 3.0.0 >Reporter: yangfang >Assignee: yangfang >Priority: Major > Attachments: HIVE-18623.1.patch > > > > I run a sql query in in a federated cluster and I have two namespaces: > nameservice and nameservice1. I set > hive.exec.stagingdir=/nameservice1/hive_tmp in hive-site.xml and my data > tables are located in the directory of nameservice, then I got the exception > as below: > hive> create external table test_par6(id int,name string) partitioned by(p > int); > OK > Time taken: 1.527 seconds > hive> insert into table test_par6 partition(p = 1) values(1,'Jack'); > Moving data to directory > viewfs://nsX/nameservice1/hive_tmp_hive_2018-02-05_14-09-36_416_3075179128063595297-1/-ext-1 > Loading data to table default.test_par6 partition (p=1) > Failed with exception java.io.IOException: Renames across Mount points not > supported > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Renames across > Mount points not supported > MapReduce Jobs Launched: > Stage-Stage-1: Map: 1 Cumulative CPU: 2.08 sec HDFS Read: 3930 HDFS Write: 7 > SUCCESS > Total MapReduce CPU Time Spent: 2 seconds 80 msec -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15911) Create a view based on another view throws an exception “FAILED: NullPointerException null”
[ https://issues.apache.org/jira/browse/HIVE-15911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yangfang updated HIVE-15911: Affects Version/s: (was: 3.0.0) 2.1.1 > Create a view based on another view throws an exception “FAILED: > NullPointerException null” > --- > > Key: HIVE-15911 > URL: https://issues.apache.org/jira/browse/HIVE-15911 > Project: Hive > Issue Type: Bug > Components: Views >Affects Versions: 2.1.1 > Environment: hive2.1.0 >Reporter: yangfang >Assignee: yangfang >Priority: Major > Attachments: 0001-Hive-15911.patch > > > when I create a new view based on another view I an exception “FAILED: > NullPointerException null”: > hive> create view view2(a,b) as select a, b from view1; //view1 is another > view > FAILED: NullPointerException null > hive> > The hive log show error stack: > 2017-02-15T15:40:25,816 ERROR ql.Driver (SessionState.java:printError(1116)) > - FAILED: NullPointerException null > java.lang.NullPointerException > at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:863) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:552) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1319) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1459) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1239) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1229) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15911) Create a view based on another view throws an exception “FAILED: NullPointerException null”
[ https://issues.apache.org/jira/browse/HIVE-15911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yangfang updated HIVE-15911: Affects Version/s: (was: 2.1.1) (was: 2.1.0) 3.0.0 > Create a view based on another view throws an exception “FAILED: > NullPointerException null” > --- > > Key: HIVE-15911 > URL: https://issues.apache.org/jira/browse/HIVE-15911 > Project: Hive > Issue Type: Bug > Components: Views >Affects Versions: 3.0.0 > Environment: hive2.1.0 >Reporter: yangfang >Assignee: yangfang >Priority: Major > Attachments: 0001-Hive-15911.patch > > > when I create a new view based on another view I an exception “FAILED: > NullPointerException null”: > hive> create view view2(a,b) as select a, b from view1; //view1 is another > view > FAILED: NullPointerException null > hive> > The hive log show error stack: > 2017-02-15T15:40:25,816 ERROR ql.Driver (SessionState.java:printError(1116)) > - FAILED: NullPointerException null > java.lang.NullPointerException > at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:863) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:552) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1319) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1459) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1239) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1229) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354888#comment-16354888 ] Hive QA commented on HIVE-18629: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 38s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 58s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s{color} | {color:red} storage-api: The patch generated 1 new + 4 unchanged - 2 fixed = 5 total (was 6) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s{color} | {color:red} ql: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 16m 10s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 3972bf0 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9059/yetus/diff-checkstyle-storage-api.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9059/yetus/diff-checkstyle-ql.txt | | modules | C: storage-api ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9059/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffe
[jira] [Updated] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18629: Attachment: HIVE-18629.01.patch > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354880#comment-16354880 ] Sergey Shelukhin commented on HIVE-18629: - [~vihangk1] updated the patch. > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15353) Metastore throws NPE if StorageDescriptor.cols is null
[ https://issues.apache.org/jira/browse/HIVE-15353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354874#comment-16354874 ] Hive QA commented on HIVE-15353: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12909352/HIVE-15353.5.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 12974 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=49) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=221) org.apache.hadoop.hive.metastore.TestMarkPartition.testMarkingPartitionSet (batchId=215) org.apache.hadoop.hive.metastore.client.TestTablesGetExists.testGetAllTablesCaseInsensitive[Embedded] (batchId=206) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280) org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap (batchId=282) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout (batchId=238) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9058/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9058/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9058/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 26 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12909352 - PreCommit-HIVE-Build > Metastore throws NPE if StorageDescriptor.cols is null > -- > > Key: HIVE-15353 > URL: https://issues.apache.org/jira/browse/HIVE-15353 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu >Priority: Major > Attachments: HIVE-15353.1.patch, HIVE-15353.2.patch, > HIVE-15353.3.patch, HIVE-15353.4.patch, HIVE-15353.5.patch > > > When using the HiveMetaStoreClient API directly to talk to the metastore, you > get NullPointerExceptions when StorageDescriptor.cols is null in the > Table/Partition object in the following calls: > * create_table > * alter_table > * alter_partition > Calling add_partition with StorageDescriptor.cols set to null causes null to > be stored in the metastore database and subsequent calls to alter_partition > for that partition to fail with an NPE. > Null checks should be added to eliminate the NPEs in the metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18340) Dynamic Min-Max/BloomFilter runtime-filtering in HoS
[ https://issues.apache.org/jira/browse/HIVE-18340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354871#comment-16354871 ] Ke Jia commented on HIVE-18340: --- [~stakiar] : >Have we done any performance analysis for this feature? We have done the benchmark in TPC-DS with this feature in HoT and get improvement(+20%) in query82/88 and (+10%) in query34/90. Now, design doc and HIVE-18340.1.patch is the initial implementation in HoS. Can you help to review the design doc and HIVE-18340.1.patch? Thanks for your help! > Dynamic Min-Max/BloomFilter runtime-filtering in HoS > > > Key: HIVE-18340 > URL: https://issues.apache.org/jira/browse/HIVE-18340 > Project: Hive > Issue Type: New Feature > Components: Spark >Affects Versions: 3.0.0 >Reporter: Ke Jia >Assignee: Ke Jia >Priority: Major > Attachments: HIVE-18340.1.patch > > > Tez implemented Dynamic Min-Max/BloomFilter runtime-filtering in HIVE-15269 > and we should implement the same in HOS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18619) Verification of temporary Micromanaged table atomicity is needed
[ https://issues.apache.org/jira/browse/HIVE-18619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354857#comment-16354857 ] Steve Yeom commented on HIVE-18619: --- Talked with Vineet regarding generating failure in INSERT. Probably using debugger might be the best option. > Verification of temporary Micromanaged table atomicity is needed > - > > Key: HIVE-18619 > URL: https://issues.apache.org/jira/browse/HIVE-18619 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Minor > > Session based temporary table by HIVE-7090 had no consideration of > Micromanaged table > (MM) since there was no insert-only ACID table at its creation tije. > HIVE-18599 addressed the issue of no writes during CTTAS (Create Temporary > Table As Select) > on Micro-Managed table. But atomicity of temporary MM table is not verified. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[ https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-18421: --- Attachment: (was: HIVE-18421.06.patch) > Vectorized execution handles overflows in a different manner than > non-vectorized execution > -- > > Key: HIVE-18421 > URL: https://issues.apache.org/jira/browse/HIVE-18421 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch, > HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch, > HIVE-18421.06.patch > > > In vectorized execution arithmetic operations which cause integer overflows > can give wrong results. Issue is reproducible in both Orc and parquet. > Simple test case to reproduce this issue > {noformat} > set hive.vectorized.execution.enabled=true; > create table parquettable (t1 tinyint, t2 tinyint) stored as parquet; > insert into parquettable values (-104, 25), (-112, 24), (54, 9); > select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by > diff desc; > +---+-+---+ > | t1 | t2 | diff | > +---+-+---+ > | -104 | 25 | 127 | > | -112 | 24 | 120 | > | 54| 9 | 45| > +---+-+---+ > {noformat} > When vectorization is turned off the same query produces only one row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[ https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-18421: --- Attachment: HIVE-18421.06.patch > Vectorized execution handles overflows in a different manner than > non-vectorized execution > -- > > Key: HIVE-18421 > URL: https://issues.apache.org/jira/browse/HIVE-18421 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch, > HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch, > HIVE-18421.06.patch > > > In vectorized execution arithmetic operations which cause integer overflows > can give wrong results. Issue is reproducible in both Orc and parquet. > Simple test case to reproduce this issue > {noformat} > set hive.vectorized.execution.enabled=true; > create table parquettable (t1 tinyint, t2 tinyint) stored as parquet; > insert into parquettable values (-104, 25), (-112, 24), (54, 9); > select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by > diff desc; > +---+-+---+ > | t1 | t2 | diff | > +---+-+---+ > | -104 | 25 | 127 | > | -112 | 24 | 120 | > | 54| 9 | 45| > +---+-+---+ > {noformat} > When vectorization is turned off the same query produces only one row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18643) don't check for archived partitions for ACID ops
[ https://issues.apache.org/jira/browse/HIVE-18643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354840#comment-16354840 ] Eugene Koifman commented on HIVE-18643: --- +1 > don't check for archived partitions for ACID ops > > > Key: HIVE-18643 > URL: https://issues.apache.org/jira/browse/HIVE-18643 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18643.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18643) don't check for archived partitions for ACID ops
[ https://issues.apache.org/jira/browse/HIVE-18643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18643: Attachment: (was: HIVE-18643.patch) > don't check for archived partitions for ACID ops > > > Key: HIVE-18643 > URL: https://issues.apache.org/jira/browse/HIVE-18643 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18643.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18643) don't check for archived partitions for ACID ops
[ https://issues.apache.org/jira/browse/HIVE-18643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18643: Attachment: HIVE-18643.patch > don't check for archived partitions for ACID ops > > > Key: HIVE-18643 > URL: https://issues.apache.org/jira/browse/HIVE-18643 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18643.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15353) Metastore throws NPE if StorageDescriptor.cols is null
[ https://issues.apache.org/jira/browse/HIVE-15353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354829#comment-16354829 ] Hive QA commented on HIVE-15353: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 7s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 11m 26s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 3972bf0 | | Default Java | 1.8.0_111 | | modules | C: standalone-metastore U: standalone-metastore | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9058/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Metastore throws NPE if StorageDescriptor.cols is null > -- > > Key: HIVE-15353 > URL: https://issues.apache.org/jira/browse/HIVE-15353 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu >Priority: Major > Attachments: HIVE-15353.1.patch, HIVE-15353.2.patch, > HIVE-15353.3.patch, HIVE-15353.4.patch, HIVE-15353.5.patch > > > When using the HiveMetaStoreClient API directly to talk to the metastore, you > get NullPointerExceptions when StorageDescriptor.cols is null in the > Table/Partition object in the following calls: > * create_table > * alter_table > * alter_partition > Calling add_partition with StorageDescriptor.cols set to null causes null to > be stored in the metastore database and subsequent calls to alter_partition > for that partition to fail with an NPE. > Null checks should be added to eliminate the NPEs in the metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18643) don't check for archived partitions for ACID ops
[ https://issues.apache.org/jira/browse/HIVE-18643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18643: Status: Patch Available (was: Open) [~ekoifman] can you take a look? tiny patch for some bug > don't check for archived partitions for ACID ops > > > Key: HIVE-18643 > URL: https://issues.apache.org/jira/browse/HIVE-18643 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18643.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18643) don't check for archived partitions for ACID ops
[ https://issues.apache.org/jira/browse/HIVE-18643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18643: Attachment: HIVE-18643.patch > don't check for archived partitions for ACID ops > > > Key: HIVE-18643 > URL: https://issues.apache.org/jira/browse/HIVE-18643 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18643.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18643) don't check for archived partitions for ACID ops
[ https://issues.apache.org/jira/browse/HIVE-18643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18643: --- > don't check for archived partitions for ACID ops > > > Key: HIVE-18643 > URL: https://issues.apache.org/jira/browse/HIVE-18643 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17983) Make the standalone metastore generate tarballs etc.
[ https://issues.apache.org/jira/browse/HIVE-17983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354821#comment-16354821 ] Alan Gates commented on HIVE-17983: --- Patch 4 is identical to patch 3, uploaded again in order to re-run tests. > Make the standalone metastore generate tarballs etc. > > > Key: HIVE-17983 > URL: https://issues.apache.org/jira/browse/HIVE-17983 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Labels: pull-request-available > Attachments: HIVE-17983.2.patch, HIVE-17983.3.patch, > HIVE-17983.4.patch, HIVE-17983.patch > > > In order to be separately installable the standalone metastore needs its own > tarballs, startup scripts, etc. All of the SQL installation and upgrade > scripts also need to move from metastore to standalone-metastore. > I also plan to create Dockerfiles for different database types so that > developers can test the SQL installation and upgrade scripts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17983) Make the standalone metastore generate tarballs etc.
[ https://issues.apache.org/jira/browse/HIVE-17983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-17983: -- Attachment: HIVE-17983.4.patch > Make the standalone metastore generate tarballs etc. > > > Key: HIVE-17983 > URL: https://issues.apache.org/jira/browse/HIVE-17983 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Labels: pull-request-available > Attachments: HIVE-17983.2.patch, HIVE-17983.3.patch, > HIVE-17983.4.patch, HIVE-17983.patch > > > In order to be separately installable the standalone metastore needs its own > tarballs, startup scripts, etc. All of the SQL installation and upgrade > scripts also need to move from metastore to standalone-metastore. > I also plan to create Dockerfiles for different database types so that > developers can test the SQL installation and upgrade scripts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18573) Use proper Calcite operator instead of UDFs
[ https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra reassigned HIVE-18573: - Assignee: Ashutosh Chauhan > Use proper Calcite operator instead of UDFs > --- > > Key: HIVE-18573 > URL: https://issues.apache.org/jira/browse/HIVE-18573 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: slim bouguerra >Assignee: Ashutosh Chauhan >Priority: Major > Attachments: HIVE-18573.2.patch, HIVE-18573.3.patch, > HIVE-18573.4.patch, HIVE-18573.5.patch, HIVE-18573.6.patch, > HIVE-18573.7.patch, HIVE-18573.7.patch, HIVE-18573.patch > > > Currently, Hive is mostly using user-defined black box sql operators during > Query planning. It will be more beneficial to use proper calcite operators, > this is to prepare the ground for pushing complex expressions to > Druid-calcite adapter. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18573) Use proper Calcite operator instead of UDFs
[ https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-18573: -- Attachment: HIVE-18573.7.patch > Use proper Calcite operator instead of UDFs > --- > > Key: HIVE-18573 > URL: https://issues.apache.org/jira/browse/HIVE-18573 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: slim bouguerra >Priority: Major > Attachments: HIVE-18573.2.patch, HIVE-18573.3.patch, > HIVE-18573.4.patch, HIVE-18573.5.patch, HIVE-18573.6.patch, > HIVE-18573.7.patch, HIVE-18573.7.patch, HIVE-18573.patch > > > Currently, Hive is mostly using user-defined black box sql operators during > Query planning. It will be more beneficial to use proper calcite operators, > this is to prepare the ground for pushing complex expressions to > Druid-calcite adapter. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18595) UNIX_TIMESTAMP UDF fails when type is Timestamp with local timezone
[ https://issues.apache.org/jira/browse/HIVE-18595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-18595: -- Attachment: HIVE-18595.patch > UNIX_TIMESTAMP UDF fails when type is Timestamp with local timezone > > > Key: HIVE-18595 > URL: https://issues.apache.org/jira/browse/HIVE-18595 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18595.patch, HIVE-18595.patch > > > {code} > 2018-01-31T12:59:45,464 ERROR [10e97c86-7f90-406b-a8fa-38be5d3529cc main] > ql.Driver: FAILED: SemanticException [Error 10014]: Line 3:456 Wrong > arguments ''-MM-dd HH:mm:ss'': The function UNIX_TIMESTAMP takes only > string/date/timestamp types > org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:456 Wrong arguments > ''-MM-dd HH:mm:ss'': The function UNIX_TIMESTAMP takes only > string/date/timestamp types > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1394) > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89) > at > org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120) > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:235) > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:181) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:11847) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:11780) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genGBLogicalPlan(CalcitePlanner.java:3140) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:4330) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1407) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1354) > at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:118) > at > org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1052) > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:154) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:111) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1159) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1175) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:422) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11393) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:304) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268) > at > org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:163) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:639) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1504) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1632) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1395) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1382) > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:240) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:343) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1331) > at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1305) > at > org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:173) > at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104) > at > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliDriver.java:59) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.re
[jira] [Assigned] (HIVE-18595) UNIX_TIMESTAMP UDF fails when type is Timestamp with local timezone
[ https://issues.apache.org/jira/browse/HIVE-18595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra reassigned HIVE-18595: - Assignee: Jesus Camacho Rodriguez > UNIX_TIMESTAMP UDF fails when type is Timestamp with local timezone > > > Key: HIVE-18595 > URL: https://issues.apache.org/jira/browse/HIVE-18595 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18595.patch, HIVE-18595.patch > > > {code} > 2018-01-31T12:59:45,464 ERROR [10e97c86-7f90-406b-a8fa-38be5d3529cc main] > ql.Driver: FAILED: SemanticException [Error 10014]: Line 3:456 Wrong > arguments ''-MM-dd HH:mm:ss'': The function UNIX_TIMESTAMP takes only > string/date/timestamp types > org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:456 Wrong arguments > ''-MM-dd HH:mm:ss'': The function UNIX_TIMESTAMP takes only > string/date/timestamp types > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1394) > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89) > at > org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120) > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:235) > at > org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:181) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:11847) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:11780) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genGBLogicalPlan(CalcitePlanner.java:3140) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:4330) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1407) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1354) > at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:118) > at > org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1052) > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:154) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:111) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1159) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1175) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:422) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11393) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:304) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268) > at > org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:163) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:639) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1504) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1632) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1395) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1382) > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:240) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:343) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1331) > at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1305) > at > org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:173) > at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104) > at > org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliDriver.java:59) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >
[jira] [Commented] (HIVE-18467) support whole warehouse dump / load + create/drop database events
[ https://issues.apache.org/jira/browse/HIVE-18467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354815#comment-16354815 ] Hive QA commented on HIVE-18467: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12909355/HIVE-18467.6.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 12973 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=49) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat] (batchId=180) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=221) org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded] (batchId=206) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280) org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap (batchId=282) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256) org.apache.hadoop.hive.ql.parse.TestQBCompact.org.apache.hadoop.hive.ql.parse.TestQBCompact (batchId=275) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9057/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9057/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9057/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 25 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12909355 - PreCommit-HIVE-Build > support whole warehouse dump / load + create/drop database events > - > > Key: HIVE-18467 > URL: https://issues.apache.org/jira/browse/HIVE-18467 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18467.0.patch, HIVE-18467.1.patch, > HIVE-18467.2.patch, HIVE-18467.5.patch, HIVE-18467.6.patch > > > A complete hive warehouse might be required to replicate to a DR site for > certain use cases and rather than allowing only a database name in the REPL > DUMP commands, we should allow dumping of all databases using the "*" option > as in > _REPL DUMP *_ > On the repl load side there will not be an option to specify the database > name when loading from a location used to dump multiple databases, hence only > _REPL LOAD FROM [location]_ would be supported when dumping via _REPL DUMP *_ > Additionally, incremental dumps will go through all events across databases > in a warehouse and hence CREATE / DROP Database events have to be serialized > correctly to allow repl load to create them correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17735) ObjectStore.addNotificationEvent is leaking queries
[ https://issues.apache.org/jira/browse/HIVE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-17735: Status: Patch Available (was: In Progress) > ObjectStore.addNotificationEvent is leaking queries > --- > > Key: HIVE-17735 > URL: https://issues.apache.org/jira/browse/HIVE-17735 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Alexander Kolbasov >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-17735.1.patch, HIVE-17735.2.patch > > > In ObjectStore.addNotificationEvent(): > {code} > Query objectQuery = pm.newQuery(MNotificationNextId.class); > Collection ids = (Collection) > objectQuery.execute(); > {code} > The query is never closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17735) ObjectStore.addNotificationEvent is leaking queries
[ https://issues.apache.org/jira/browse/HIVE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-17735: Status: In Progress (was: Patch Available) > ObjectStore.addNotificationEvent is leaking queries > --- > > Key: HIVE-17735 > URL: https://issues.apache.org/jira/browse/HIVE-17735 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Alexander Kolbasov >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-17735.1.patch, HIVE-17735.2.patch > > > In ObjectStore.addNotificationEvent(): > {code} > Query objectQuery = pm.newQuery(MNotificationNextId.class); > Collection ids = (Collection) > objectQuery.execute(); > {code} > The query is never closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18328) Improve schematool validator to report duplicate rows for column statistics
[ https://issues.apache.org/jira/browse/HIVE-18328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354796#comment-16354796 ] Aihua Xu commented on HIVE-18328: - [~ngangam] Sorry that I missed this one. Do you think we can also add similar validation for PART_COL_STATS as well? > Improve schematool validator to report duplicate rows for column statistics > --- > > Key: HIVE-18328 > URL: https://issues.apache.org/jira/browse/HIVE-18328 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.1.1 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-18328.patch > > > By design, in the {{TAB_COL_STATS}} table of the HMS schema, there should be > ONE AND ONLY ONE row, representing its statistics, for each column defined in > hive. A combination of DB_NAME, TABLE_NAME and COLUMN_NAME constitute a > primary key/unique row. > Each time the statistics are computed for a column, this row is updated. > However, if somehow via BDR/replication process, we end up with multiple > rows in this table for a given column, HMS server to recompute the statistics > there after. > So it would be good to detect this data anamoly via the schema validation > tool. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[ https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-18421: --- Attachment: HIVE-18421.06.patch > Vectorized execution handles overflows in a different manner than > non-vectorized execution > -- > > Key: HIVE-18421 > URL: https://issues.apache.org/jira/browse/HIVE-18421 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch, > HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch, > HIVE-18421.06.patch > > > In vectorized execution arithmetic operations which cause integer overflows > can give wrong results. Issue is reproducible in both Orc and parquet. > Simple test case to reproduce this issue > {noformat} > set hive.vectorized.execution.enabled=true; > create table parquettable (t1 tinyint, t2 tinyint) stored as parquet; > insert into parquettable values (-104, 25), (-112, 24), (54, 9); > select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by > diff desc; > +---+-+---+ > | t1 | t2 | diff | > +---+-+---+ > | -104 | 25 | 127 | > | -112 | 24 | 120 | > | 54| 9 | 45| > +---+-+---+ > {noformat} > When vectorization is turned off the same query produces only one row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18640) Update q.out for TestSparkCliDriver.testCliDriver[ppd_join5]
[ https://issues.apache.org/jira/browse/HIVE-18640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354797#comment-16354797 ] Sahil Takiar commented on HIVE-18640: - ummm really not sure what is going on here. Test works locally on OSX. Tried on Ubuntu still works. Tried on an actual ptest-slave and it still works. Tried the entire q-test batch, still works. So not sure why this is failing. JVM version maybe? > Update q.out for TestSparkCliDriver.testCliDriver[ppd_join5] > > > Key: HIVE-18640 > URL: https://issues.apache.org/jira/browse/HIVE-18640 > Project: Hive > Issue Type: Test > Components: Test >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > TestSparkCliDriver.testCliDriver[ppd_join5] has been consistently failing for > 166 builds with a q-file diff. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18626) Repl load "with" clause does not pass config to tasks
[ https://issues.apache.org/jira/browse/HIVE-18626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354787#comment-16354787 ] Thejas M Nair commented on HIVE-18626: -- Changes look good. Can you please take a look at checkstyle warnings (seems like an indentation issue), and also verify that TestMarkPartition.testMarkingPartitionSet is not related to this issue ? (only test failure that has age=1 in jenkins) > Repl load "with" clause does not pass config to tasks > - > > Key: HIVE-18626 > URL: https://issues.apache.org/jira/browse/HIVE-18626 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-18626.1.patch > > > The "with" clause in repl load suppose to pass custom hive config entries to > replication. However, the config is only effective in > BootstrapEventsIterator, but not the generated tasks (such as MoveTask, > DDLTask). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18467) support whole warehouse dump / load + create/drop database events
[ https://issues.apache.org/jira/browse/HIVE-18467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354770#comment-16354770 ] Hive QA commented on HIVE-18467: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 35s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 37s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 35s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 7s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 12s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s{color} | {color:red} standalone-metastore: The patch generated 6 new + 13 unchanged - 4 fixed = 19 total (was 17) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 40s{color} | {color:red} ql: The patch generated 6 new + 515 unchanged - 3 fixed = 521 total (was 518) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s{color} | {color:red} itests/hive-unit: The patch generated 18 new + 19 unchanged - 4 fixed = 37 total (was 23) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 22m 3s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 3972bf0 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9057/yetus/diff-checkstyle-standalone-metastore.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9057/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9057/yetus/diff-checkstyle-itests_hive-unit.txt | | modules | C: standalone-metastore ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9057/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > support whole warehouse dump / load + create/drop database events > - > > Key: HIVE-18467 > URL: https://issues.apache.org/jira/browse/HIVE-18467 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18467.0.patch, HIVE-18467.1.patch, > HIVE-18467.2.patch, HIVE-18467.5.patch, HIVE-18467.6.patch > > > A complete hive warehouse might be required to replicate to a DR site for > certain use cases and rather than allowing only a database name in the RE
[jira] [Commented] (HIVE-17063) insert overwrite partition onto a external table fail when drop partition first
[ https://issues.apache.org/jira/browse/HIVE-17063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354768#comment-16354768 ] Ashutosh Chauhan commented on HIVE-17063: - +1 > insert overwrite partition onto a external table fail when drop partition > first > --- > > Key: HIVE-17063 > URL: https://issues.apache.org/jira/browse/HIVE-17063 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.2, 2.1.1, 2.2.0 >Reporter: Wang Haihua >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-17063.1.patch, HIVE-17063.2.patch, > HIVE-17063.3.patch, HIVE-17063.4.patch > > > The default value of {{hive.exec.stagingdir}} which is a relative path, and > also drop partition on a external table will not clear the real data. As a > result, insert overwrite partition twice will happen to fail because of the > target data to be moved has > already existed. > This happened when we reproduce partition data onto a external table. > I see the target data will not be cleared only when {{immediately generated > data}} is child of {{the target data directory}}, so my proposal is trying > to clear target file already existed finally whe doing rename {{immediately > generated data}} into {{the target data directory}} > Operation reproduced: > {code} > create external table insert_after_drop_partition(key string, val string) > partitioned by (insertdate string); > from src insert overwrite table insert_after_drop_partition partition > (insertdate='2008-01-01') select *; > alter table insert_after_drop_partition drop partition > (insertdate='2008-01-01'); > from src insert overwrite table insert_after_drop_partition partition > (insertdate='2008-01-01') select *; > {code} > Stack trace: > {code} > 2017-07-09T08:32:05,212 ERROR [f3bc51c8-2441-4689-b1c1-d60aef86c3aa main] > exec.Task: Failed with exception java.io.IOException: rename for src path: > pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/.hive-staging_hive_2017-07-09_08-32-03_840_4046825276907030554-1/-ext-1/00_0 > to dest > path:pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/00_0 > returned false > org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: rename > for src path: > pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/.hive-staging_hive_2017-07-09_08-32-03_840_4046825276907030554-1/-ext-1/00_0 > to dest > path:pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/00_0 > returned false > at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2992) > at > org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3248) > at > org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1532) > at > org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1461) > at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:498) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1137) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:) > at > org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:120) > at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_after_drop_partition(TestCliDriver.java:103) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.j
[jira] [Commented] (HIVE-17063) insert overwrite partition onto a external table fail when drop partition first
[ https://issues.apache.org/jira/browse/HIVE-17063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354741#comment-16354741 ] Deepak Jaiswal commented on HIVE-17063: --- [~ashutoshc] can you please review? > insert overwrite partition onto a external table fail when drop partition > first > --- > > Key: HIVE-17063 > URL: https://issues.apache.org/jira/browse/HIVE-17063 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.2, 2.1.1, 2.2.0 >Reporter: Wang Haihua >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-17063.1.patch, HIVE-17063.2.patch, > HIVE-17063.3.patch, HIVE-17063.4.patch > > > The default value of {{hive.exec.stagingdir}} which is a relative path, and > also drop partition on a external table will not clear the real data. As a > result, insert overwrite partition twice will happen to fail because of the > target data to be moved has > already existed. > This happened when we reproduce partition data onto a external table. > I see the target data will not be cleared only when {{immediately generated > data}} is child of {{the target data directory}}, so my proposal is trying > to clear target file already existed finally whe doing rename {{immediately > generated data}} into {{the target data directory}} > Operation reproduced: > {code} > create external table insert_after_drop_partition(key string, val string) > partitioned by (insertdate string); > from src insert overwrite table insert_after_drop_partition partition > (insertdate='2008-01-01') select *; > alter table insert_after_drop_partition drop partition > (insertdate='2008-01-01'); > from src insert overwrite table insert_after_drop_partition partition > (insertdate='2008-01-01') select *; > {code} > Stack trace: > {code} > 2017-07-09T08:32:05,212 ERROR [f3bc51c8-2441-4689-b1c1-d60aef86c3aa main] > exec.Task: Failed with exception java.io.IOException: rename for src path: > pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/.hive-staging_hive_2017-07-09_08-32-03_840_4046825276907030554-1/-ext-1/00_0 > to dest > path:pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/00_0 > returned false > org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: rename > for src path: > pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/.hive-staging_hive_2017-07-09_08-32-03_840_4046825276907030554-1/-ext-1/00_0 > to dest > path:pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/00_0 > returned false > at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2992) > at > org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3248) > at > org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1532) > at > org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1461) > at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:498) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1137) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:) > at > org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:120) > at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_after_drop_partition(TestCliDriver.java:103) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.r
[jira] [Updated] (HIVE-17063) insert overwrite partition onto a external table fail when drop partition first
[ https://issues.apache.org/jira/browse/HIVE-17063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-17063: -- Attachment: HIVE-17063.4.patch > insert overwrite partition onto a external table fail when drop partition > first > --- > > Key: HIVE-17063 > URL: https://issues.apache.org/jira/browse/HIVE-17063 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.2, 2.1.1, 2.2.0 >Reporter: Wang Haihua >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-17063.1.patch, HIVE-17063.2.patch, > HIVE-17063.3.patch, HIVE-17063.4.patch > > > The default value of {{hive.exec.stagingdir}} which is a relative path, and > also drop partition on a external table will not clear the real data. As a > result, insert overwrite partition twice will happen to fail because of the > target data to be moved has > already existed. > This happened when we reproduce partition data onto a external table. > I see the target data will not be cleared only when {{immediately generated > data}} is child of {{the target data directory}}, so my proposal is trying > to clear target file already existed finally whe doing rename {{immediately > generated data}} into {{the target data directory}} > Operation reproduced: > {code} > create external table insert_after_drop_partition(key string, val string) > partitioned by (insertdate string); > from src insert overwrite table insert_after_drop_partition partition > (insertdate='2008-01-01') select *; > alter table insert_after_drop_partition drop partition > (insertdate='2008-01-01'); > from src insert overwrite table insert_after_drop_partition partition > (insertdate='2008-01-01') select *; > {code} > Stack trace: > {code} > 2017-07-09T08:32:05,212 ERROR [f3bc51c8-2441-4689-b1c1-d60aef86c3aa main] > exec.Task: Failed with exception java.io.IOException: rename for src path: > pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/.hive-staging_hive_2017-07-09_08-32-03_840_4046825276907030554-1/-ext-1/00_0 > to dest > path:pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/00_0 > returned false > org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: rename > for src path: > pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/.hive-staging_hive_2017-07-09_08-32-03_840_4046825276907030554-1/-ext-1/00_0 > to dest > path:pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/00_0 > returned false > at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2992) > at > org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3248) > at > org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1532) > at > org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1461) > at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:498) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1137) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:) > at > org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:120) > at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_after_drop_partition(TestCliDriver.java:103) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.j
[jira] [Assigned] (HIVE-17063) insert overwrite partition onto a external table fail when drop partition first
[ https://issues.apache.org/jira/browse/HIVE-17063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal reassigned HIVE-17063: - Assignee: Deepak Jaiswal (was: Wang Haihua) > insert overwrite partition onto a external table fail when drop partition > first > --- > > Key: HIVE-17063 > URL: https://issues.apache.org/jira/browse/HIVE-17063 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.2, 2.1.1, 2.2.0 >Reporter: Wang Haihua >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-17063.1.patch, HIVE-17063.2.patch, > HIVE-17063.3.patch > > > The default value of {{hive.exec.stagingdir}} which is a relative path, and > also drop partition on a external table will not clear the real data. As a > result, insert overwrite partition twice will happen to fail because of the > target data to be moved has > already existed. > This happened when we reproduce partition data onto a external table. > I see the target data will not be cleared only when {{immediately generated > data}} is child of {{the target data directory}}, so my proposal is trying > to clear target file already existed finally whe doing rename {{immediately > generated data}} into {{the target data directory}} > Operation reproduced: > {code} > create external table insert_after_drop_partition(key string, val string) > partitioned by (insertdate string); > from src insert overwrite table insert_after_drop_partition partition > (insertdate='2008-01-01') select *; > alter table insert_after_drop_partition drop partition > (insertdate='2008-01-01'); > from src insert overwrite table insert_after_drop_partition partition > (insertdate='2008-01-01') select *; > {code} > Stack trace: > {code} > 2017-07-09T08:32:05,212 ERROR [f3bc51c8-2441-4689-b1c1-d60aef86c3aa main] > exec.Task: Failed with exception java.io.IOException: rename for src path: > pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/.hive-staging_hive_2017-07-09_08-32-03_840_4046825276907030554-1/-ext-1/00_0 > to dest > path:pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/00_0 > returned false > org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: rename > for src path: > pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/.hive-staging_hive_2017-07-09_08-32-03_840_4046825276907030554-1/-ext-1/00_0 > to dest > path:pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/00_0 > returned false > at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2992) > at > org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3248) > at > org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1532) > at > org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1461) > at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:498) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1137) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:) > at > org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:120) > at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_after_drop_partition(TestCliDriver.java:103) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.
[jira] [Updated] (HIVE-18456) Add some tests for HIVE-18367 to check that the table information contains the query correctly
[ https://issues.apache.org/jira/browse/HIVE-18456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18456: -- Attachment: HIVE-18456.1.patch > Add some tests for HIVE-18367 to check that the table information contains > the query correctly > -- > > Key: HIVE-18456 > URL: https://issues.apache.org/jira/browse/HIVE-18456 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18456.1.patch > > > This cannot be tested with a CliDriver test so add a java test to check the > output of 'describe extended', which is changed by HIVE-18367 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18456) Add some tests for HIVE-18367 to check that the table information contains the query correctly
[ https://issues.apache.org/jira/browse/HIVE-18456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18456: -- Status: Patch Available (was: Open) > Add some tests for HIVE-18367 to check that the table information contains > the query correctly > -- > > Key: HIVE-18456 > URL: https://issues.apache.org/jira/browse/HIVE-18456 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18456.1.patch > > > This cannot be tested with a CliDriver test so add a java test to check the > output of 'describe extended', which is changed by HIVE-18367 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18567) ObjectStore.getPartitionNamesNoTxn doesn't handle max param properly
[ https://issues.apache.org/jira/browse/HIVE-18567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354647#comment-16354647 ] Hive QA commented on HIVE-18567: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12909416/HIVE-18567.1.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 12974 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=49) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=221) org.apache.hadoop.hive.metastore.client.TestFunctions.testGetFunctionNullDatabase[Embedded] (batchId=206) org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded] (batchId=206) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280) org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap (batchId=282) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9055/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9055/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9055/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 25 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12909416 - PreCommit-HIVE-Build > ObjectStore.getPartitionNamesNoTxn doesn't handle max param properly > > > Key: HIVE-18567 > URL: https://issues.apache.org/jira/browse/HIVE-18567 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-18567.0.patch, HIVE-18567.1.patch > > > As per [this HMS API test > case|https://github.com/apache/hive/commit/fa0a8d27d4149cc5cc2dbb49d8eb6b03f46bc279#diff-25c67d898000b53e623a6df9221aad5dR1044] > listing partition names doesn't check tha max param against > MetaStoreConf.LIMIT_PARTITION_REQUEST (as other methods do by > checkLimitNumberOfPartitionsByFilter), and also behaves differently on max=0 > setting compared to other methods. > We should bring this into consistency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354609#comment-16354609 ] Sergey Shelukhin commented on HIVE-18629: - Sorry, yeah, I mixed it up when changing the ifs around. > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354585#comment-16354585 ] Vihang Karajgaonkar commented on HIVE-18629: Hi [~sershe] I took a quick look at the attached patch. Shouldn't the snippet below also guard against NPE by adding a if condition {{if (!(output.isNull[i] = isNull[i]))}} after we do a System.arraycopy {code} } else { System.arraycopy(isNull, 0, output.isNull, 0, size); for (int i = 0; i < size; i++) { output.setVal(i, vector[i], start[i], length[i]); } } {code} > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18449) Add configurable policy for choosing the HMS URI from hive.metastore.uris
[ https://issues.apache.org/jira/browse/HIVE-18449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-18449: --- Attachment: HIVE-18449.3-branch-2.patch > Add configurable policy for choosing the HMS URI from hive.metastore.uris > - > > Key: HIVE-18449 > URL: https://issues.apache.org/jira/browse/HIVE-18449 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Sahil Takiar >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-18449.1.patch, HIVE-18449.2.patch, > HIVE-18449.3-branch-2.patch > > > HIVE-10815 added logic to randomly choose a HMS URI from > {{hive.metastore.uris}}. It would be nice if there was a configurable policy > that determined how a URI is chosen from this list - e.g. one option can be > to randomly pick a URI, another option can be to choose the first URI in the > list (which was the behavior prior to HIVE-10815). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18642) incorrect assertion in TezSessionPool for WM
[ https://issues.apache.org/jira/browse/HIVE-18642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354572#comment-16354572 ] Prasanth Jayachandran commented on HIVE-18642: -- +1 > incorrect assertion in TezSessionPool for WM > > > Key: HIVE-18642 > URL: https://issues.apache.org/jira/browse/HIVE-18642 > Project: Hive > Issue Type: Bug >Reporter: Dileep Kumar Chiguruvada >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18642.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18627) PPD: Handle FLOAT boxing differently for single/double precision constants
[ https://issues.apache.org/jira/browse/HIVE-18627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354570#comment-16354570 ] Prasanth Jayachandran commented on HIVE-18627: -- +1. I wonder why the file system counters are removed for some tests. > PPD: Handle FLOAT boxing differently for single/double precision constants > -- > > Key: HIVE-18627 > URL: https://issues.apache.org/jira/browse/HIVE-18627 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-18627.1.patch, HIVE-18627.2.patch > > > Constants like 0.1 and 0.3 are differently boxed based on intermediate > precision of the compiler codepath. > Disabling CBO produces 0.1BD constants which fail to box correctly to > Double/Float. > Enabling CBO fixes this issue, but cannot be applied all queries in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18567) ObjectStore.getPartitionNamesNoTxn doesn't handle max param properly
[ https://issues.apache.org/jira/browse/HIVE-18567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354559#comment-16354559 ] Hive QA commented on HIVE-18567: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 13s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 18s{color} | {color:red} standalone-metastore: The patch generated 1 new + 474 unchanged - 2 fixed = 475 total (was 476) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 11m 39s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 3972bf0 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9055/yetus/diff-checkstyle-standalone-metastore.txt | | modules | C: standalone-metastore U: standalone-metastore | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9055/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > ObjectStore.getPartitionNamesNoTxn doesn't handle max param properly > > > Key: HIVE-18567 > URL: https://issues.apache.org/jira/browse/HIVE-18567 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-18567.0.patch, HIVE-18567.1.patch > > > As per [this HMS API test > case|https://github.com/apache/hive/commit/fa0a8d27d4149cc5cc2dbb49d8eb6b03f46bc279#diff-25c67d898000b53e623a6df9221aad5dR1044] > listing partition names doesn't check tha max param against > MetaStoreConf.LIMIT_PARTITION_REQUEST (as other methods do by > checkLimitNumberOfPartitionsByFilter), and also behaves differently on max=0 > setting compared to other methods. > We should bring this into consistency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-18614: -- Attachment: HIVE-18614.patch > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18614.patch > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-18614: -- Status: Patch Available (was: Open) > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18614.patch > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-18614: -- Attachment: (was: HIVE-18614.patch) > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-18614: -- Status: Open (was: Patch Available) > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18523) Fix summary row in case there are no inputs
[ https://issues.apache.org/jira/browse/HIVE-18523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354533#comment-16354533 ] Hive QA commented on HIVE-18523: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12909470/HIVE-18523.03.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 12950 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver (batchId=84) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=221) org.apache.hadoop.hive.metastore.TestAcidTableSetup.testTransactionalValidation (batchId=223) org.apache.hadoop.hive.metastore.TestReplChangeManager.testRecycleNonPartTable (batchId=223) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280) org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap (batchId=282) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.authorization.TestJdbcMetadataApiAuth.org.apache.hive.jdbc.authorization.TestJdbcMetadataApiAuth (batchId=239) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9054/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9054/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9054/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 26 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12909470 - PreCommit-HIVE-Build > Fix summary row in case there are no inputs > --- > > Key: HIVE-18523 > URL: https://issues.apache.org/jira/browse/HIVE-18523 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-18523.01.patch, HIVE-18523.03.patch > > > HIVE-17617 added a fix to handle empty groups correctly; however it might be > possible that the summary row is returned multiple times - in case the > mappers are "seem" to be optimized away... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-17063) insert overwrite partition onto a external table fail when drop partition first
[ https://issues.apache.org/jira/browse/HIVE-17063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal reassigned HIVE-17063: - Assignee: Wang Haihua (was: Deepak Jaiswal) > insert overwrite partition onto a external table fail when drop partition > first > --- > > Key: HIVE-17063 > URL: https://issues.apache.org/jira/browse/HIVE-17063 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.2, 2.1.1, 2.2.0 >Reporter: Wang Haihua >Assignee: Wang Haihua >Priority: Major > Attachments: HIVE-17063.1.patch, HIVE-17063.2.patch, > HIVE-17063.3.patch > > > The default value of {{hive.exec.stagingdir}} which is a relative path, and > also drop partition on a external table will not clear the real data. As a > result, insert overwrite partition twice will happen to fail because of the > target data to be moved has > already existed. > This happened when we reproduce partition data onto a external table. > I see the target data will not be cleared only when {{immediately generated > data}} is child of {{the target data directory}}, so my proposal is trying > to clear target file already existed finally whe doing rename {{immediately > generated data}} into {{the target data directory}} > Operation reproduced: > {code} > create external table insert_after_drop_partition(key string, val string) > partitioned by (insertdate string); > from src insert overwrite table insert_after_drop_partition partition > (insertdate='2008-01-01') select *; > alter table insert_after_drop_partition drop partition > (insertdate='2008-01-01'); > from src insert overwrite table insert_after_drop_partition partition > (insertdate='2008-01-01') select *; > {code} > Stack trace: > {code} > 2017-07-09T08:32:05,212 ERROR [f3bc51c8-2441-4689-b1c1-d60aef86c3aa main] > exec.Task: Failed with exception java.io.IOException: rename for src path: > pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/.hive-staging_hive_2017-07-09_08-32-03_840_4046825276907030554-1/-ext-1/00_0 > to dest > path:pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/00_0 > returned false > org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: rename > for src path: > pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/.hive-staging_hive_2017-07-09_08-32-03_840_4046825276907030554-1/-ext-1/00_0 > to dest > path:pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/00_0 > returned false > at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2992) > at > org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3248) > at > org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1532) > at > org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1461) > at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:498) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1137) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:) > at > org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:120) > at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_after_drop_partition(TestCliDriver.java:103) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.jun
[jira] [Commented] (HIVE-18340) Dynamic Min-Max/BloomFilter runtime-filtering in HoS
[ https://issues.apache.org/jira/browse/HIVE-18340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354529#comment-16354529 ] Sahil Takiar commented on HIVE-18340: - Have we done any performance analysis for this feature? > Dynamic Min-Max/BloomFilter runtime-filtering in HoS > > > Key: HIVE-18340 > URL: https://issues.apache.org/jira/browse/HIVE-18340 > Project: Hive > Issue Type: New Feature > Components: Spark >Affects Versions: 3.0.0 >Reporter: Ke Jia >Assignee: Ke Jia >Priority: Major > Attachments: HIVE-18340.1.patch > > > Tez implemented Dynamic Min-Max/BloomFilter runtime-filtering in HIVE-15269 > and we should implement the same in HOS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18641) Remove MCreationMetadata from MTable class
[ https://issues.apache.org/jira/browse/HIVE-18641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354523#comment-16354523 ] Ashutosh Chauhan commented on HIVE-18641: - +1 > Remove MCreationMetadata from MTable class > -- > > Key: HIVE-18641 > URL: https://issues.apache.org/jira/browse/HIVE-18641 > Project: Hive > Issue Type: Bug > Components: Materialized views, Metastore >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18641.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[ https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354516#comment-16354516 ] Vihang Karajgaonkar commented on HIVE-18421: latest patch fixes TestCliDriver.testCliDriver[vectorization_numeric_overflows] and some Yetus checkstyle changes. > Vectorized execution handles overflows in a different manner than > non-vectorized execution > -- > > Key: HIVE-18421 > URL: https://issues.apache.org/jira/browse/HIVE-18421 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch, > HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch > > > In vectorized execution arithmetic operations which cause integer overflows > can give wrong results. Issue is reproducible in both Orc and parquet. > Simple test case to reproduce this issue > {noformat} > set hive.vectorized.execution.enabled=true; > create table parquettable (t1 tinyint, t2 tinyint) stored as parquet; > insert into parquettable values (-104, 25), (-112, 24), (54, 9); > select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by > diff desc; > +---+-+---+ > | t1 | t2 | diff | > +---+-+---+ > | -104 | 25 | 127 | > | -112 | 24 | 120 | > | 54| 9 | 45| > +---+-+---+ > {noformat} > When vectorization is turned off the same query produces only one row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[ https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-18421: --- Attachment: HIVE-18421.05.patch > Vectorized execution handles overflows in a different manner than > non-vectorized execution > -- > > Key: HIVE-18421 > URL: https://issues.apache.org/jira/browse/HIVE-18421 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch, > HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch > > > In vectorized execution arithmetic operations which cause integer overflows > can give wrong results. Issue is reproducible in both Orc and parquet. > Simple test case to reproduce this issue > {noformat} > set hive.vectorized.execution.enabled=true; > create table parquettable (t1 tinyint, t2 tinyint) stored as parquet; > insert into parquettable values (-104, 25), (-112, 24), (54, 9); > select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by > diff desc; > +---+-+---+ > | t1 | t2 | diff | > +---+-+---+ > | -104 | 25 | 127 | > | -112 | 24 | 120 | > | 54| 9 | 45| > +---+-+---+ > {noformat} > When vectorization is turned off the same query produces only one row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18588) Add 'checkin' profile that runs slower tests in standalone-metastore
[ https://issues.apache.org/jira/browse/HIVE-18588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354491#comment-16354491 ] Alan Gates commented on HIVE-18588: --- +1 to running the precommit tests with groups set to "". Do you want to make changes to ptest as part of this patch or do you want to do that separately? > Add 'checkin' profile that runs slower tests in standalone-metastore > > > Key: HIVE-18588 > URL: https://issues.apache.org/jira/browse/HIVE-18588 > Project: Hive > Issue Type: Test > Components: Standalone Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Attachments: HIVE-18588.2.patch, HIVE-18588.patch > > > Runtime for unit tests in standalone-metastore are now exceeding 25 minutes. > Ideally unit tests should finish within 2-3 minutes so users will run them > frequently. To solve this I propose to carve off many of the slower tests to > run in a new 'checkin' profile. This profile should be run before checkin > and by the ptest infrastructure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18642) incorrect assertion in TezSessionPool for WM
[ https://issues.apache.org/jira/browse/HIVE-18642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18642: Status: Patch Available (was: Open) > incorrect assertion in TezSessionPool for WM > > > Key: HIVE-18642 > URL: https://issues.apache.org/jira/browse/HIVE-18642 > Project: Hive > Issue Type: Bug >Reporter: Dileep Kumar Chiguruvada >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18642.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18642) incorrect assertion in TezSessionPool for WM
[ https://issues.apache.org/jira/browse/HIVE-18642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354487#comment-16354487 ] Sergey Shelukhin commented on HIVE-18642: - [~prasanth_j] can you take a look? a tiny patch > incorrect assertion in TezSessionPool for WM > > > Key: HIVE-18642 > URL: https://issues.apache.org/jira/browse/HIVE-18642 > Project: Hive > Issue Type: Bug >Reporter: Dileep Kumar Chiguruvada >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18642.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18642) incorrect assertion in TezSessionPool for WM
[ https://issues.apache.org/jira/browse/HIVE-18642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18642: --- > incorrect assertion in TezSessionPool for WM > > > Key: HIVE-18642 > URL: https://issues.apache.org/jira/browse/HIVE-18642 > Project: Hive > Issue Type: Bug >Reporter: Dileep Kumar Chiguruvada >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18642.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18642) incorrect assertion in TezSessionPool for WM
[ https://issues.apache.org/jira/browse/HIVE-18642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18642: Attachment: HIVE-18642.patch > incorrect assertion in TezSessionPool for WM > > > Key: HIVE-18642 > URL: https://issues.apache.org/jira/browse/HIVE-18642 > Project: Hive > Issue Type: Bug >Reporter: Dileep Kumar Chiguruvada >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18642.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18523) Fix summary row in case there are no inputs
[ https://issues.apache.org/jira/browse/HIVE-18523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354484#comment-16354484 ] Hive QA commented on HIVE-18523: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 4s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 41s{color} | {color:red} ql: The patch generated 1 new + 270 unchanged - 24 fixed = 271 total (was 294) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 13m 58s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 3972bf0 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9054/yetus/diff-checkstyle-ql.txt | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9054/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Fix summary row in case there are no inputs > --- > > Key: HIVE-18523 > URL: https://issues.apache.org/jira/browse/HIVE-18523 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-18523.01.patch, HIVE-18523.03.patch > > > HIVE-17617 added a fix to handle empty groups correctly; however it might be > possible that the summary row is returned multiple times - in case the > mappers are "seem" to be optimized away... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18627) PPD: Handle FLOAT boxing differently for single/double precision constants
[ https://issues.apache.org/jira/browse/HIVE-18627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-18627: --- Attachment: HIVE-18627.2.patch > PPD: Handle FLOAT boxing differently for single/double precision constants > -- > > Key: HIVE-18627 > URL: https://issues.apache.org/jira/browse/HIVE-18627 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-18627.1.patch, HIVE-18627.2.patch > > > Constants like 0.1 and 0.3 are differently boxed based on intermediate > precision of the compiler codepath. > Disabling CBO produces 0.1BD constants which fail to box correctly to > Double/Float. > Enabling CBO fixes this issue, but cannot be applied all queries in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18628) Make tez dag status check interval configurable
[ https://issues.apache.org/jira/browse/HIVE-18628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354474#comment-16354474 ] Prasanth Jayachandran commented on HIVE-18628: -- [~sershe] still discussing with tez team about aggregated counters when tasks are running. I will create a hive or tez ticket based on the response (not sure if there is a way in tez to get aggregated live counters when task is running, if there is a way it will be hive follow up else tez ticket) > Make tez dag status check interval configurable > --- > > Key: HIVE-18628 > URL: https://issues.apache.org/jira/browse/HIVE-18628 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18628.1.patch > > > DAG status check interval in TezJobMonitor is not configurable. This can > cause cached data to be returned and fail some trigger validation tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18641) Remove MCreationMetadata from MTable class
[ https://issues.apache.org/jira/browse/HIVE-18641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354472#comment-16354472 ] Jesus Camacho Rodriguez commented on HIVE-18641: [~ashutoshc], could you take a look? This should have been pushed within HIVE-18546 since the {{CreationMetadataID}} column was removed from {{TBLS}} and thus should not be contained in {{MTable}}. Thanks > Remove MCreationMetadata from MTable class > -- > > Key: HIVE-18641 > URL: https://issues.apache.org/jira/browse/HIVE-18641 > Project: Hive > Issue Type: Bug > Components: Materialized views, Metastore >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18641.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18628) Make tez dag status check interval configurable
[ https://issues.apache.org/jira/browse/HIVE-18628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354469#comment-16354469 ] Sergey Shelukhin commented on HIVE-18628: - +1. Should there be a Tez jira/follow up jira for the commented out test? That seems like a problem. > Make tez dag status check interval configurable > --- > > Key: HIVE-18628 > URL: https://issues.apache.org/jira/browse/HIVE-18628 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18628.1.patch > > > DAG status check interval in TezJobMonitor is not configurable. This can > cause cached data to be returned and fail some trigger validation tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[ https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354459#comment-16354459 ] Hive QA commented on HIVE-18238: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12909455/HIVE-18238.07.patch {color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 12975 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=49) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=221) org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded] (batchId=206) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280) org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap (batchId=282) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9053/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9053/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9053/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 23 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12909455 - PreCommit-HIVE-Build > Driver execution may not have configuration changing sideeffects > - > > Key: HIVE-18238 > URL: https://issues.apache.org/jira/browse/HIVE-18238 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-18238.01wip01.patch, HIVE-18238.02.patch, > HIVE-18238.03.patch, HIVE-18238.04.patch, HIVE-18238.04wip01.patch, > HIVE-18238.07.patch > > > {{Driver}} executes sql statements which use "hiveconf" settings; > but the {{Driver}} itself may *not* change the configuration... > I've found an example; which shows how hazardous this is... > {code} > set hive.mapred.mode=strict; > select "${hiveconf:hive.mapred.mode}"; > create table t (a int); > analyze table t compute statistics; > select "${hiveconf:hive.mapred.mode}"; > {code} > currently; the last select returns {{nonstrict}} because of > [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1696] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18641) Remove MCreationMetadata from MTable class
[ https://issues.apache.org/jira/browse/HIVE-18641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18641: --- Attachment: HIVE-18641.patch > Remove MCreationMetadata from MTable class > -- > > Key: HIVE-18641 > URL: https://issues.apache.org/jira/browse/HIVE-18641 > Project: Hive > Issue Type: Bug > Components: Materialized views, Metastore >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18641.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-18641) Remove MCreationMetadata from MTable class
[ https://issues.apache.org/jira/browse/HIVE-18641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-18641 started by Jesus Camacho Rodriguez. -- > Remove MCreationMetadata from MTable class > -- > > Key: HIVE-18641 > URL: https://issues.apache.org/jira/browse/HIVE-18641 > Project: Hive > Issue Type: Bug > Components: Materialized views, Metastore >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18641) Remove MCreationMetadata from MTable class
[ https://issues.apache.org/jira/browse/HIVE-18641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18641: --- Status: Patch Available (was: In Progress) > Remove MCreationMetadata from MTable class > -- > > Key: HIVE-18641 > URL: https://issues.apache.org/jira/browse/HIVE-18641 > Project: Hive > Issue Type: Bug > Components: Materialized views, Metastore >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18641) Remove MCreationMetadata from MTable class
[ https://issues.apache.org/jira/browse/HIVE-18641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-18641: -- > Remove MCreationMetadata from MTable class > -- > > Key: HIVE-18641 > URL: https://issues.apache.org/jira/browse/HIVE-18641 > Project: Hive > Issue Type: Bug > Components: Materialized views, Metastore >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17735) ObjectStore.addNotificationEvent is leaking queries
[ https://issues.apache.org/jira/browse/HIVE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354427#comment-16354427 ] Aihua Xu commented on HIVE-17735: - Patch-2: fix one test failure. The others don't seem related. > ObjectStore.addNotificationEvent is leaking queries > --- > > Key: HIVE-17735 > URL: https://issues.apache.org/jira/browse/HIVE-17735 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Alexander Kolbasov >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-17735.1.patch, HIVE-17735.2.patch > > > In ObjectStore.addNotificationEvent(): > {code} > Query objectQuery = pm.newQuery(MNotificationNextId.class); > Collection ids = (Collection) > objectQuery.execute(); > {code} > The query is never closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17735) ObjectStore.addNotificationEvent is leaking queries
[ https://issues.apache.org/jira/browse/HIVE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-17735: Attachment: HIVE-17735.2.patch > ObjectStore.addNotificationEvent is leaking queries > --- > > Key: HIVE-17735 > URL: https://issues.apache.org/jira/browse/HIVE-17735 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Alexander Kolbasov >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-17735.1.patch, HIVE-17735.2.patch > > > In ObjectStore.addNotificationEvent(): > {code} > Query objectQuery = pm.newQuery(MNotificationNextId.class); > Collection ids = (Collection) > objectQuery.execute(); > {code} > The query is never closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-17063) insert overwrite partition onto a external table fail when drop partition first
[ https://issues.apache.org/jira/browse/HIVE-17063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal reassigned HIVE-17063: - Assignee: Deepak Jaiswal (was: Wang Haihua) > insert overwrite partition onto a external table fail when drop partition > first > --- > > Key: HIVE-17063 > URL: https://issues.apache.org/jira/browse/HIVE-17063 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.2, 2.1.1, 2.2.0 >Reporter: Wang Haihua >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-17063.1.patch, HIVE-17063.2.patch, > HIVE-17063.3.patch > > > The default value of {{hive.exec.stagingdir}} which is a relative path, and > also drop partition on a external table will not clear the real data. As a > result, insert overwrite partition twice will happen to fail because of the > target data to be moved has > already existed. > This happened when we reproduce partition data onto a external table. > I see the target data will not be cleared only when {{immediately generated > data}} is child of {{the target data directory}}, so my proposal is trying > to clear target file already existed finally whe doing rename {{immediately > generated data}} into {{the target data directory}} > Operation reproduced: > {code} > create external table insert_after_drop_partition(key string, val string) > partitioned by (insertdate string); > from src insert overwrite table insert_after_drop_partition partition > (insertdate='2008-01-01') select *; > alter table insert_after_drop_partition drop partition > (insertdate='2008-01-01'); > from src insert overwrite table insert_after_drop_partition partition > (insertdate='2008-01-01') select *; > {code} > Stack trace: > {code} > 2017-07-09T08:32:05,212 ERROR [f3bc51c8-2441-4689-b1c1-d60aef86c3aa main] > exec.Task: Failed with exception java.io.IOException: rename for src path: > pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/.hive-staging_hive_2017-07-09_08-32-03_840_4046825276907030554-1/-ext-1/00_0 > to dest > path:pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/00_0 > returned false > org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: rename > for src path: > pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/.hive-staging_hive_2017-07-09_08-32-03_840_4046825276907030554-1/-ext-1/00_0 > to dest > path:pfile:/data/haihua/official/hive/itests/qtest/target/warehouse/insert_after_drop_partition/insertdate=2008-01-01/00_0 > returned false > at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2992) > at > org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3248) > at > org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1532) > at > org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1461) > at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:498) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1137) > at > org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:) > at > org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:120) > at > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_after_drop_partition(TestCliDriver.java:103) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.
[jira] [Commented] (HIVE-18627) PPD: Handle FLOAT boxing differently for single/double precision constants
[ https://issues.apache.org/jira/browse/HIVE-18627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354414#comment-16354414 ] Gopal V commented on HIVE-18627: This is the test that fails if I fix it by Boxing from Decimal -> Double. If I do Decimal -> Float -> Double, it passes. This test fails if I fixed it without the new Enum to box those two differently. I'll add a 1 row Double test, to cover the broken case. > PPD: Handle FLOAT boxing differently for single/double precision constants > -- > > Key: HIVE-18627 > URL: https://issues.apache.org/jira/browse/HIVE-18627 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-18627.1.patch > > > Constants like 0.1 and 0.3 are differently boxed based on intermediate > precision of the compiler codepath. > Disabling CBO produces 0.1BD constants which fail to box correctly to > Double/Float. > Enabling CBO fixes this issue, but cannot be applied all queries in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18627) PPD: Handle FLOAT boxing differently for single/double precision constants
[ https://issues.apache.org/jira/browse/HIVE-18627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354395#comment-16354395 ] Prasanth Jayachandran commented on HIVE-18627: -- [~gopalv] how come the tests weren't failing then? can we add a test for the delete with filter 0.1 which was originally causing this issue? > PPD: Handle FLOAT boxing differently for single/double precision constants > -- > > Key: HIVE-18627 > URL: https://issues.apache.org/jira/browse/HIVE-18627 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-18627.1.patch > > > Constants like 0.1 and 0.3 are differently boxed based on intermediate > precision of the compiler codepath. > Disabling CBO produces 0.1BD constants which fail to box correctly to > Double/Float. > Enabling CBO fixes this issue, but cannot be applied all queries in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)