[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022660#comment-17022660 ] Eugene Chung commented on HIVE-22126: - Because of calcite dependency on guava, I think it should be included in hive-exec.jar too. > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Eugene Chung >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22767) beeline doesn't parse semicolons in comments properly
[ https://issues.apache.org/jira/browse/HIVE-22767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022713#comment-17022713 ] Hive QA commented on HIVE-22767: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991675/HIVE-22767.01.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 17946 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20312/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20312/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20312/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12991675 - PreCommit-HIVE-Build > beeline doesn't parse semicolons in comments properly > - > > Key: HIVE-22767 > URL: https://issues.apache.org/jira/browse/HIVE-22767 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Zoltan Matyus >Assignee: Zoltan Matyus >Priority: Major > Attachments: HIVE-22767.01.patch > > > HIVE-12646 fixed the handling of semicolons in quoted strings, but leaves the > problem of semicolons in comments. E.g. with beeline connected to any > database... > this works: {code:sql}select 1; select /* */ 2; select /* */ 3;{code} > this doesn't work: {code:sql}select 1; select /* ; */ 2; select /* ; */ > 3;{code} > This has been fixed and reintroduced before (possibly multiple times). > Ideally, there should be a single utility method somewhere to separate > comments, strings and commands -- with the proper testing in place (q files). > However, I'm trying to make this fix back-portable, so a light touch is > needed. I'm focusing on beeline for now, and only writing (very thorough) > unit tests, as I cannot exclude any new q files from TestCliDriver (which > would break, since it's using a different parsing method). > > P.S. excerpt of the error message: > {noformat} > 0: jdbc:hive2://...> select 1; select /* ; */ 2; select /* ; */ 3; > INFO : Compiling command(queryId=...): select 1 > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, > type:int, comment:null)], properties:null) > INFO : Completed compiling command(queryId=...); Time taken: 0.38 seconds > INFO : Executing command(queryId=...): select 1 > INFO : Completed executing command(queryId=...); Time taken: 0.004 seconds > INFO : OK > +--+ > | _c0 | > +--+ > | 1| > +--+ > 1 row selected (2.007 seconds) > INFO : Compiling command(queryId=...): select /* > ERROR : FAILED: ParseException line 1:9 cannot recognize input near '' > '' '' in select clause > org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize > input near '' '' '' in select clause > at > org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:233) > at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:79) > at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:72) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:598) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1505) > at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1452) > at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1447) > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126) > at ... > {noformat} > > Similarly, the following query also fails: > {code:sql}select /* ' */ 1; select /* ' */ 2;{code} > I suspect line comments are also not handled properly but I cannot reproduce > this in interactive beeline... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22746) Make TypeCheckProcFactory generic
[ https://issues.apache.org/jira/browse/HIVE-22746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022725#comment-17022725 ] Hive QA commented on HIVE-22746: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 31s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 48s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 51s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 51s{color} | {color:red} ql: The patch generated 142 new + 1331 unchanged - 15 fixed = 1473 total (was 1346) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 1s{color} | {color:red} ql generated 1 new + 1530 unchanged - 2 fixed = 1531 total (was 1532) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 57s{color} | {color:red} ql generated 1 new + 99 unchanged - 1 fixed = 100 total (was 100) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 24m 15s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Nullcheck of procCtx at line 1499 of value previously dereferenced in org.apache.hadoop.hive.ql.parse.type.TypeCheckProcFactory.processGByExpr(Node, Object) At TypeCheckProcFactory.java:1499 of value previously dereferenced in org.apache.hadoop.hive.ql.parse.type.TypeCheckProcFactory.processGByExpr(Node, Object) At TypeCheckProcFactory.java:[line 1492] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-20313/dev-support/hive-personality.sh | | git revision | master / 4193b54 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-20313/yetus/diff-checkstyle-ql.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-20313/yetus/new-findbugs-ql.html | | javadoc | http://104.198.109.242/logs//PreCommit-HIVE-Build-20313/yetus/diff-javadoc-javadoc-ql.txt | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-20313/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Make TypeCheckProcFactory generic > - > > Key: HIVE-22746 > URL: https://issues.apache.org/jira/browse/HIVE-22746 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22746.01.patch, HIVE-22746.01.patch, > HIVE-22746.patch > > Time Spent: 10m > Remaining Estimate: 0h > > {{TypeCheckProcFactory}} is responsible for processing AST nodes and > generating ExprNode objects from them. When we generate the expressions for > Calcite planning, we go through a {{AST node -> ExprNode -> RexNode}} > transformation. We would like to avoid
[jira] [Commented] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022640#comment-17022640 ] Hive QA commented on HIVE-22726: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991666/HIVE-22726.3.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20309/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20309/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20309/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2020-01-24 01:40:18.344 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-20309/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2020-01-24 01:40:18.346 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 4193b54 HIVE-22751: Move locking in HiveServer2::isDeregisteredWithZooKeeper to ZooKeeperHiveHelper (Rajesh Balamohan, reviewed by Anishek Agarwal) + git clean -f -d Removing ${project.basedir}/ Removing itests/${project.basedir}/ Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 4193b54 HIVE-22751: Move locking in HiveServer2::isDeregisteredWithZooKeeper to ZooKeeperHiveHelper (Rajesh Balamohan, reviewed by Anishek Agarwal) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2020-01-24 01:40:19.523 + rm -rf ../yetus_PreCommit-HIVE-Build-20309 + mkdir ../yetus_PreCommit-HIVE-Build-20309 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-20309 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20309/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch Trying to apply the patch with -p0 error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyFilter.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorTopNKeyOperator.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/wrapper/VectorHashKeyWrapperBatch.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/wrapper/VectorHashKeyWrapperGeneralComparator.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java: does not exist in index Trying to apply the patch with -p1 error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java:38 Falling back to three-way merge... Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java' with conflicts. error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java:58 Falling back to three-way merge... Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java' with conflicts. Going to apply patch with: git apply -p1 error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java:38 Falling back to three-way merge... Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java' with conflicts. error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java:58 Falling
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022639#comment-17022639 ] Hive QA commented on HIVE-20801: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991665/HIVE-20801.4.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 17946 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20308/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20308/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20308/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12991665 - PreCommit-HIVE-Build > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal Vijayaraghavan >Assignee: Gopal Vijayaraghavan >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch, HIVE-20801.4.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22768) Remove the usages of the Utilities.makeList method call
[ https://issues.apache.org/jira/browse/HIVE-22768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022680#comment-17022680 ] Hive QA commented on HIVE-22768: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991673/HIVE-22768.1.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 86 failed/errored test(s), 17945 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_1] (batchId=25) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_2] (batchId=96) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join0] (batchId=101) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join12] (batchId=28) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join13] (batchId=92) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join20] (batchId=102) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join21] (batchId=93) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join22] (batchId=64) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=82) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join28] (batchId=81) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join29] (batchId=62) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join2] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join31] (batchId=51) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join3] (batchId=93) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join7] (batchId=30) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_stats2] (batchId=100) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_stats] (batchId=55) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_without_localtask] (batchId=1) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark4] (batchId=1) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_cross_product_check_2] (batchId=23) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer5] (batchId=80) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer7] (batchId=25) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cross_product_check_2] (batchId=102) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_join2] (batchId=45) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[empty_join] (batchId=91) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_rearrange] (batchId=14) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join26] (batchId=22) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join28] (batchId=96) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join29] (batchId=50) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=103) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join32] (batchId=21) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join33] (batchId=18) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_on_varchar] (batchId=52) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_star] (batchId=33) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[manyViewJoin] (batchId=41) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin47] (batchId=68) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_decimal_vectorized] (batchId=44) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_distinct] (batchId=64) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=14) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_mapjoin] (batchId=57) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_subquery2] (batchId=6) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_subquery] (batchId=58) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[multi_join_union] (batchId=2) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonblock_op_deduplicate] (batchId=26) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reduce_deduplicate_exclude_join] (batchId=87) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[runtime_skewjoin_mapjoin_spark] (batchId=62) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin] (batchId=26) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin10] (batchId=38) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin11] (batchId=59) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin1] (batchId=98) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin2] (batchId=31) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin3] (batchId=72)
[jira] [Commented] (HIVE-10362) Support Type check/conversion in dynamic partition column
[ https://issues.apache.org/jira/browse/HIVE-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022659#comment-17022659 ] Hive QA commented on HIVE-10362: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991672/HIVE-10362.01.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 99 failed/errored test(s), 17945 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1] (batchId=303) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory] (batchId=306) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_part] (batchId=306) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[parquet_format_part] (batchId=306) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_format_part] (batchId=306) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_6] (batchId=75) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_merge] (batchId=42) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid2] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_num_buckets] (batchId=69) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input13] (batchId=87) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input45] (batchId=61) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[inputwherefalse] (batchId=97) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_overwrite_directory2] (batchId=76) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_overwrite_directory] (batchId=31) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_overwrite_local_directory_1] (batchId=32) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_static_ptn_into_bucketed_table] (batchId=23) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[merge_empty] (batchId=44) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[multi_insert_move_tasks_share_dependencies] (batchId=63) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[null_column] (batchId=29) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nullformatdir] (batchId=6) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge10] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge11] (batchId=45) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge1] (batchId=23) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge2] (batchId=103) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge_diff_fs] (batchId=1) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_multi_insert] (batchId=86) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_transform] (batchId=87) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_join_partition_key] (batchId=15) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subq] (batchId=50) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_alias] (batchId=28) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union] (batchId=5) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge10] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge1] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge2] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge_diff_fs] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_merge2] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schemeAuthority2] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schemeAuthority] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_16] (batchId=180) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_vectorization] (batchId=178) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization] (batchId=180) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid] (batchId=177) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_dir_distcp] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_insert] (batchId=174) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge11] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dml] (batchId=173) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_insert_overwrite_local_directory_1] (batchId=166)
[jira] [Updated] (HIVE-22769) Incorrect query results and query failure during split generation for compressed text files
[ https://issues.apache.org/jira/browse/HIVE-22769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chiran Ravani updated HIVE-22769: - Summary: Incorrect query results and query failure during split generation for compressed text files (was: Incorrect query results and query failure during Split generation for compressed text files) > Incorrect query results and query failure during split generation for > compressed text files > --- > > Key: HIVE-22769 > URL: https://issues.apache.org/jira/browse/HIVE-22769 > Project: Hive > Issue Type: Bug > Components: File Formats >Affects Versions: 3.0.0, 3.1.0 >Reporter: Chiran Ravani >Priority: Critical > Attachments: testcase1.csv.bz2, testcase2.csv.bz2 > > > Hive Query produces incorrect results when data is in text format and > compressed and for certain data the query fails during split generation. > This behavior is seen when skip.header.line.count and skip.footer.line.count > are set for table. > Case 1: Select count/aggregate query produces Incorrect row counts/displays > all rows (when hive.fetch.task.conversion=none) > Steps to reproduce: > 1. Create table as below > {code} > CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase1' > TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1"); > {code} > 2. Upload attached testcase1.csv.bz2 file to /user/hive/testcase1 > 3. Run count(*) on table. > {code} > > select * from testcase1; > INFO : Compiling > command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f): > select * from testcase1 > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: > Schema(fieldSchemas:[FieldSchema(name:testcase1.id, type:string, > comment:null), FieldSchema(name:testcase1.name, type:string, comment:null)], > properties:null) > INFO : Completed compiling > command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); > Time taken: 0.07 seconds > INFO : Executing > command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f): > select * from testcase1 > INFO : Completed executing > command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); > Time taken: 0.007 seconds > INFO : OK > +---+-+ > | testcase1.id | testcase1.name | > +---+-+ > | 2 | 2019-12-31 | > +---+-+ > 1 row selected (0.111 seconds) > > select count(*) from testcase1 > INFO : Compiling > command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7): > select count(*) from testcase1 > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, > type:bigint, comment:null)], properties:null) > INFO : Completed compiling > command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); > Time taken: 0.073 seconds > INFO : Executing > command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7): > select count(*) from testcase1 > INFO : Query ID = hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7 > INFO : Total jobs = 1 > INFO : Launching Job 1 out of 1 > INFO : Starting task [Stage-1:MAPRED] in serial mode > INFO : Subscribed to counters: [] for queryId: > hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7 > INFO : Session is already open > INFO : Dag name: select count(*) from testcase1 (Stage-1) > INFO : Status: Running (Executing on YARN cluster with App id > application_1579811438512_0046) > . > . > . > INFO : Completed executing > command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); > Time taken: 4.228 seconds > INFO : OK > +--+ > | _c0 | > +--+ > | 3| > +--+ > 1 row selected (4.335 seconds) > {code} > Case 2: Select count/aggregate query fails with java.lang.ClassCastException: > java.io.PushbackInputStream cannot be cast to org.apache.hadoop.fs.Seekable > The issue is only seen when there is a space in a field (eg:- "3,2019-12-31 > 01" second column has a space) > Steps to reproduce: > 1. Create table as below > {code} > CREATE EXTERNAL TABLE `testcase2`(id int, name string) ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase2' > TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1"); > {code} > 2. Upload attached testcase2.csv.bz2 file to /user/hive/testcase2 > 3. Run count(*) on table. > {code} > 0: > select * from testcase2; > INFO : Compiling > command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134): > select * from
[jira] [Updated] (HIVE-22769) Incorrect query results and query failure during Split generation for compressed text files
[ https://issues.apache.org/jira/browse/HIVE-22769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chiran Ravani updated HIVE-22769: - Description: Hive Query produces incorrect results when data is in text format and compressed and for certain data the query fails during split generation. This behavior is seen when skip.header.line.count and skip.footer.line.count are set for table. Case 1: Select count/aggregate query produces Incorrect row counts/displays all rows (when hive.fetch.task.conversion=none) Steps to reproduce: 1. Create table as below {code} CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase1' TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1"); {code} 2. Upload attached testcase1.csv.bz2 file to /user/hive/testcase1 3. Run count(*) on table. {code} > select * from testcase1; INFO : Compiling command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f): select * from testcase1 INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:testcase1.id, type:string, comment:null), FieldSchema(name:testcase1.name, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); Time taken: 0.07 seconds INFO : Executing command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f): select * from testcase1 INFO : Completed executing command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); Time taken: 0.007 seconds INFO : OK +---+-+ | testcase1.id | testcase1.name | +---+-+ | 2 | 2019-12-31 | +---+-+ 1 row selected (0.111 seconds) > select count(*) from testcase1 INFO : Compiling command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7): select count(*) from testcase1 INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); Time taken: 0.073 seconds INFO : Executing command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7): select count(*) from testcase1 INFO : Query ID = hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7 INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : Subscribed to counters: [] for queryId: hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7 INFO : Session is already open INFO : Dag name: select count(*) from testcase1 (Stage-1) INFO : Status: Running (Executing on YARN cluster with App id application_1579811438512_0046) . . . INFO : Completed executing command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); Time taken: 4.228 seconds INFO : OK +--+ | _c0 | +--+ | 3| +--+ 1 row selected (4.335 seconds) {code} Case 2: Select count/aggregate query fails with java.lang.ClassCastException: java.io.PushbackInputStream cannot be cast to org.apache.hadoop.fs.Seekable The issue is only seen when there is a space in a field (eg:- "3,2019-12-31 01" second column has a space) Steps to reproduce: 1. Create table as below {code} CREATE EXTERNAL TABLE `testcase2`(id int, name string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase2' TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1"); {code} 2. Upload attached testcase2.csv.bz2 file to /user/hive/testcase2 3. Run count(*) on table. {code} 0: > select * from testcase2; INFO : Compiling command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134): select * from testcase2 INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:testcase2.id, type:string, comment:null), FieldSchema(name:testcase2.name, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134); Time taken: 0.075 seconds INFO : Executing command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134): select * from testcase2 INFO : Completed executing command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134); Time taken: 0.01 seconds INFO : OK +---+-+ | testcase2.id | testcase2.name | +---+-+ | 2 | 2019-12-31 01 | +---+-+ 1 row selected (0.119 seconds) {code} > select count(*) from testcase2; INFO : Compiling
[jira] [Updated] (HIVE-22769) Incorrect query results and query failure during Split generation for compressed text files
[ https://issues.apache.org/jira/browse/HIVE-22769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chiran Ravani updated HIVE-22769: - Description: Hive Query produces incorrect results when data is in text format and compressed and for certain data the query fails during split generation. This behavior is seen when skip.header.line.count and skip.footer.line.count are set for table. Case 1: Select count/aggregate query produces Incorrect row counts/displays all rows (when hive.fetch.task.conversion=none) Steps to reproduce: 1. Create table as below {code} CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase1' TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1"); {code} 2. Upload attached testcase1.csv.bz2 file to /user/hive/testcase1 3. Run count(**) on table. {code} > select * from testcase1; INFO : Compiling command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f): select * from testcase1 INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:testcase1.id, type:string, comment:null), FieldSchema(name:testcase1.name, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); Time taken: 0.07 seconds INFO : Executing command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f): select * from testcase1 INFO : Completed executing command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); Time taken: 0.007 seconds INFO : OK +---+-+ | testcase1.id | testcase1.name | +---+-+ | 2 | 2019-12-31 | +---+-+ 1 row selected (0.111 seconds) > select count(*) from testcase1 INFO : Compiling command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7): select count(*) from testcase1 INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); Time taken: 0.073 seconds INFO : Executing command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7): select count(*) from testcase1 INFO : Query ID = hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7 INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : Subscribed to counters: [] for queryId: hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7 INFO : Session is already open INFO : Dag name: select count(*) from testcase1 (Stage-1) INFO : Status: Running (Executing on YARN cluster with App id application_1579811438512_0046) . . . INFO : Completed executing command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); Time taken: 4.228 seconds INFO : OK +--+ | _c0 | +--+ | 3| +--+ 1 row selected (4.335 seconds) {code} Case 2: Select count/aggregate query fails with java.lang.ClassCastException: java.io.PushbackInputStream cannot be cast to org.apache.hadoop.fs.Seekable The issue is only seen when there is a space in a field (eg:- "3,2019-12-31 01" second column has a space) Steps to reproduce: 1. Create table as below {code} CREATE EXTERNAL TABLE `testcase2`(id int, name string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase2' TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1"); {code} 2. Upload attached testcase2.csv.bz2 file to /user/hive/testcase2 3. Run count(*) on table. {code} 0: > select * from testcase2; INFO : Compiling command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134): select * from testcase2 INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:testcase2.id, type:string, comment:null), FieldSchema(name:testcase2.name, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134); Time taken: 0.075 seconds INFO : Executing command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134): select * from testcase2 INFO : Completed executing command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134); Time taken: 0.01 seconds INFO : OK +---+-+ | testcase2.id | testcase2.name | +---+-+ | 2 | 2019-12-31 01 | +---+-+ 1 row selected (0.119 seconds) {code} > select count(*) from testcase2; INFO : Compiling
[jira] [Commented] (HIVE-22746) Make TypeCheckProcFactory generic
[ https://issues.apache.org/jira/browse/HIVE-22746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022734#comment-17022734 ] Hive QA commented on HIVE-22746: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991685/HIVE-22746.01.patch {color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 17945 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20313/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20313/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20313/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12991685 - PreCommit-HIVE-Build > Make TypeCheckProcFactory generic > - > > Key: HIVE-22746 > URL: https://issues.apache.org/jira/browse/HIVE-22746 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22746.01.patch, HIVE-22746.01.patch, > HIVE-22746.patch > > Time Spent: 10m > Remaining Estimate: 0h > > {{TypeCheckProcFactory}} is responsible for processing AST nodes and > generating ExprNode objects from them. When we generate the expressions for > Calcite planning, we go through a {{AST node -> ExprNode -> RexNode}} > transformation. We would like to avoid the overhead of going through the > ExprNode, and thus generate directly the RexNode from the AST. > To do that, the first step is to make {{TypeCheckProcFactory}} generic, so it > can receive an expression factory and create expressions in different realms. > For the time being, the only factory implementation is the ExprNode factory. > Thus, this patch focuses mainly on refactoring {{TypeCheckProcFactory}} > without breaking anything that is already working. > In a follow-up patch, we will create a {{RexNode}} factory and use it when we > parse the query in CalcitePlanner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-10362) Support Type check/conversion in dynamic partition column
[ https://issues.apache.org/jira/browse/HIVE-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022649#comment-17022649 ] Hive QA commented on HIVE-10362: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 21s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 6s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 43s{color} | {color:red} ql: The patch generated 1 new + 353 unchanged - 1 fixed = 354 total (was 354) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 25m 37s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-20310/dev-support/hive-personality.sh | | git revision | master / 4193b54 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-20310/yetus/diff-checkstyle-ql.txt | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-20310/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Support Type check/conversion in dynamic partition column > - > > Key: HIVE-10362 > URL: https://issues.apache.org/jira/browse/HIVE-10362 > Project: Hive > Issue Type: Improvement > Components: Query Processor, Types >Affects Versions: 1.0.0, 3.0.0, 4.0.0 >Reporter: Chaoyu Tang >Assignee: Karen Coppage >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-10362.01.patch > > > There are quite a lot of issues associated with the non-noramlized or > type-mismatched values for partition column. Hive has many ways to introduce > such problematic data. HIVE-10307 mainly provides the support to type > check/convert/normalize the partition column value in static partition > specification. This JIRA tries to deal with the partition column type in > dynamic partition insert. Currently any data can be inserted as a partition > column value as long as it is quoted as a string. For example, > create table dynparttypechecknum (key int, value string) partitioned by (part > int); > insert into dynparttypechecknum partition (part) select key, value, '1' > from src limit 1; > show partitions dynparttypechecknum; > -- > part=1 > The partition column value is non-normalized int 1. It causes some > unnecessary problems such as integer partition column JDO filter pushdown > (see HIVE-6052) and others like HIVE-10210. -- This message was sent by Atlassian Jira
[jira] [Commented] (HIVE-22767) beeline doesn't parse semicolons in comments properly
[ https://issues.apache.org/jira/browse/HIVE-22767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022689#comment-17022689 ] Hive QA commented on HIVE-22767: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 44s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 28s{color} | {color:blue} beeline in master has 48 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} beeline: The patch generated 28 new + 36 unchanged - 0 fixed = 64 total (was 36) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 12m 9s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-20312/dev-support/hive-personality.sh | | git revision | master / 4193b54 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-20312/yetus/diff-checkstyle-beeline.txt | | modules | C: beeline U: beeline | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-20312/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > beeline doesn't parse semicolons in comments properly > - > > Key: HIVE-22767 > URL: https://issues.apache.org/jira/browse/HIVE-22767 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Zoltan Matyus >Assignee: Zoltan Matyus >Priority: Major > Attachments: HIVE-22767.01.patch > > > HIVE-12646 fixed the handling of semicolons in quoted strings, but leaves the > problem of semicolons in comments. E.g. with beeline connected to any > database... > this works: {code:sql}select 1; select /* */ 2; select /* */ 3;{code} > this doesn't work: {code:sql}select 1; select /* ; */ 2; select /* ; */ > 3;{code} > This has been fixed and reintroduced before (possibly multiple times). > Ideally, there should be a single utility method somewhere to separate > comments, strings and commands -- with the proper testing in place (q files). > However, I'm trying to make this fix back-portable, so a light touch is > needed. I'm focusing on beeline for now, and only writing (very thorough) > unit tests, as I cannot exclude any new q files from TestCliDriver (which > would break, since it's using a different parsing method). > > P.S. excerpt of the error message: > {noformat} > 0: jdbc:hive2://...> select 1; select /* ; */ 2; select /* ; */ 3; > INFO : Compiling command(queryId=...): select 1 > INFO : Semantic Analysis Completed (retrial = false) > INFO :
[jira] [Updated] (HIVE-22769) Incorrect query results and query failure during Split generation for compressed text files
[ https://issues.apache.org/jira/browse/HIVE-22769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chiran Ravani updated HIVE-22769: - Description: Hive Query produces incorrect results when data is in text format and compressed and for certain data the query fails during split generation. This behavior is seen when skip.header.line.count and skip.footer.line.count are set for table. Case 1: Select count/aggregate query produces Incorrect row counts/displays all rows (when hive.fetch.task.conversion=none) Steps to reproduce: 1. Create table as below {code} CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase1' TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1"); {code} 2. Upload attached testcase1.csv.bz2 file to /user/hive/testcase1 3. Run count(*) on table. {code} > select * from testcase1; INFO : Compiling command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f): select * from testcase1 INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:testcase1.id, type:string, comment:null), FieldSchema(name:testcase1.name, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); Time taken: 0.07 seconds INFO : Executing command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f): select * from testcase1 INFO : Completed executing command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); Time taken: 0.007 seconds INFO : OK +---+-+ | testcase1.id | testcase1.name | +---+-+ | 2 | 2019-12-31 | +---+-+ 1 row selected (0.111 seconds) > select count(*) from testcase1 INFO : Compiling command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7): select count(*) from testcase1 INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); Time taken: 0.073 seconds INFO : Executing command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7): select count(*) from testcase1 INFO : Query ID = hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7 INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : Subscribed to counters: [] for queryId: hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7 INFO : Session is already open INFO : Dag name: select count(*) from testcase1 (Stage-1) INFO : Status: Running (Executing on YARN cluster with App id application_1579811438512_0046) . . . INFO : Completed executing command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); Time taken: 4.228 seconds INFO : OK +--+ | _c0 | +--+ | 3| +--+ 1 row selected (4.335 seconds) {code} Case 2: Select count/aggregate query fails with java.lang.ClassCastException: java.io.PushbackInputStream cannot be cast to org.apache.hadoop.fs.Seekable The issue is only seen when there is a space in a field (eg:- "3,2019-12-31 01" second column has a space) Steps to reproduce: 1. Create table as below {code} CREATE EXTERNAL TABLE `testcase2`(id int, name string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase2' TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1"); {code} 2. Upload attached testcase2.csv.bz2 file to /user/hive/testcase2 3. Run count(*) on table. {code} 0: > select * from testcase2; INFO : Compiling command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134): select * from testcase2 INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:testcase2.id, type:string, comment:null), FieldSchema(name:testcase2.name, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134); Time taken: 0.075 seconds INFO : Executing command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134): select * from testcase2 INFO : Completed executing command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134); Time taken: 0.01 seconds INFO : OK +---+-+ | testcase2.id | testcase2.name | +---+-+ | 2 | 2019-12-31 01 | +---+-+ 1 row selected (0.119 seconds) > select count(*) from testcase2; INFO : Compiling
[jira] [Updated] (HIVE-22769) Incorrect query results and query failure during Split generation for compressed text files
[ https://issues.apache.org/jira/browse/HIVE-22769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chiran Ravani updated HIVE-22769: - Description: Hive Query produces incorrect results when data is in text format and compressed and for certain data the query fails during split generation. This behavior is seen when skip.header.line.count and skip.footer.line.count are set for table. Case 1: Select count/aggregate query produces Incorrect row counts/displays all rows (when hive.fetch.task.conversion=none) Steps to reproduce: 1. Create table as below {code} CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase1' TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1"); {code} 2. Upload attached testcase1.csv.bz2 file to /user/hive/testcase1 3. Run count(*) on table. {code} > select * from testcase1; INFO : Compiling command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f): select * from testcase1 INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:testcase1.id, type:string, comment:null), FieldSchema(name:testcase1.name, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); Time taken: 0.07 seconds INFO : Executing command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f): select * from testcase1 INFO : Completed executing command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); Time taken: 0.007 seconds INFO : OK +---+-+ | testcase1.id | testcase1.name | +---+-+ | 2 | 2019-12-31 | +---+-+ 1 row selected (0.111 seconds) > select count(*) from testcase1 INFO : Compiling command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7): select count(*) from testcase1 INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); Time taken: 0.073 seconds INFO : Executing command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7): select count(*) from testcase1 INFO : Query ID = hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7 INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : Subscribed to counters: [] for queryId: hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7 INFO : Session is already open INFO : Dag name: select count(*) from testcase1 (Stage-1) INFO : Status: Running (Executing on YARN cluster with App id application_1579811438512_0046) . . . INFO : Completed executing command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); Time taken: 4.228 seconds INFO : OK +--+ | _c0 | +--+ | 3| +--+ 1 row selected (4.335 seconds) {code} Case 2: Select count/aggregate query fails with java.lang.ClassCastException: java.io.PushbackInputStream cannot be cast to org.apache.hadoop.fs.Seekable The issue is only seen when there is a space in a field (eg:- "3,2019-12-31 01" second column has a space) Steps to reproduce: 1. Create table as below {code} CREATE EXTERNAL TABLE `testcase2`(id int, name string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase2' TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1"); {code} 2. Upload attached testcase2.csv.bz2 file to /user/hive/testcase2 3. Run count(*) on table. {code} 0: > select * from testcase2; INFO : Compiling command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134): select * from testcase2 INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:testcase2.id, type:string, comment:null), FieldSchema(name:testcase2.name, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134); Time taken: 0.075 seconds INFO : Executing command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134): select * from testcase2 INFO : Completed executing command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134); Time taken: 0.01 seconds INFO : OK +---+-+ | testcase2.id | testcase2.name | +---+-+ | 2 | 2019-12-31 01 | +---+-+ 1 row selected (0.119 seconds) > select count(*) from testcase2; INFO : Compiling
[jira] [Updated] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22538: -- Status: Patch Available (was: Open) > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch, > HIVE-22538.6.patch, HIVE-22538.patch > > Time Spent: 20m > Remaining Estimate: 0h > > For transactional tables, that property might be overriden to 1, which can > lead to merging final aggregation into a single stage (hence leading to > performance degradation). For instance, when autogather column stats is > enabled, this can happen for the following query: > {code} > set hive.support.concurrency=true; > set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > EXPLAIN > CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS > SELECT * FROM SRC x CLUSTER BY x.key; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22538: -- Status: Open (was: Patch Available) > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch, > HIVE-22538.6.patch, HIVE-22538.patch > > Time Spent: 20m > Remaining Estimate: 0h > > For transactional tables, that property might be overriden to 1, which can > lead to merging final aggregation into a single stage (hence leading to > performance degradation). For instance, when autogather column stats is > enabled, this can happen for the following query: > {code} > set hive.support.concurrency=true; > set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > EXPLAIN > CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS > SELECT * FROM SRC x CLUSTER BY x.key; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22538: -- Attachment: HIVE-22538.6.patch > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch, > HIVE-22538.6.patch, HIVE-22538.patch > > Time Spent: 20m > Remaining Estimate: 0h > > For transactional tables, that property might be overriden to 1, which can > lead to merging final aggregation into a single stage (hence leading to > performance degradation). For instance, when autogather column stats is > enabled, this can happen for the following query: > {code} > set hive.support.concurrency=true; > set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > EXPLAIN > CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS > SELECT * FROM SRC x CLUSTER BY x.key; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22768) Remove the usages of the Utilities.makeList method call
[ https://issues.apache.org/jira/browse/HIVE-22768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022667#comment-17022667 ] Hive QA commented on HIVE-22768: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 8s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 56s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 39s{color} | {color:red} ql: The patch generated 10 new + 181 unchanged - 1 fixed = 191 total (was 182) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch 15 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 24m 43s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-20311/dev-support/hive-personality.sh | | git revision | master / 4193b54 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-20311/yetus/diff-checkstyle-ql.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-20311/yetus/whitespace-eol.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-20311/yetus/whitespace-tabs.txt | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-20311/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Remove the usages of the Utilities.makeList method call > --- > > Key: HIVE-22768 > URL: https://issues.apache.org/jira/browse/HIVE-22768 > Project: Hive > Issue Type: Improvement >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-22768.1.patch > > > The aforementioned > org.apache.hadoop.hive.ql.exec.Utilities.makeList(Object...) method is non > generic and most of its usages can be replaced by a simple > Collections.singletonList(Object) call. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-22768) Remove the usages of the Utilities.makeList method call
[ https://issues.apache.org/jira/browse/HIVE-22768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller resolved HIVE-22768. Release Note: The resulted list should be mutable, so the proposed solution won't work Resolution: Won't Fix > Remove the usages of the Utilities.makeList method call > --- > > Key: HIVE-22768 > URL: https://issues.apache.org/jira/browse/HIVE-22768 > Project: Hive > Issue Type: Improvement >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-22768.1.patch > > > The aforementioned > org.apache.hadoop.hive.ql.exec.Utilities.makeList(Object...) method is non > generic and most of its usages can be replaced by a simple > Collections.singletonList(Object) call. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22768) Remove the usages of the Utilities.makeList method call
[ https://issues.apache.org/jira/browse/HIVE-22768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-22768: --- Status: Open (was: Patch Available) That was a misunderstanding. > Remove the usages of the Utilities.makeList method call > --- > > Key: HIVE-22768 > URL: https://issues.apache.org/jira/browse/HIVE-22768 > Project: Hive > Issue Type: Improvement >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-22768.1.patch > > > The aforementioned > org.apache.hadoop.hive.ql.exec.Utilities.makeList(Object...) method is non > generic and most of its usages can be replaced by a simple > Collections.singletonList(Object) call. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22764) Create new command for "optimize" compaction and have basic implementation.
[ https://issues.apache.org/jira/browse/HIVE-22764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Shah reassigned HIVE-22764: -- > Create new command for "optimize" compaction and have basic implementation. > --- > > Key: HIVE-22764 > URL: https://issues.apache.org/jira/browse/HIVE-22764 > Project: Hive > Issue Type: Sub-task >Reporter: Aditya Shah >Assignee: Aditya Shah >Priority: Major > > Created new blocking compaction (added compaction type "optimize") by adding > a lock request on the compaction's transaction. It works mostly like > mmMajorCompaction and writes files w/o row_IDs. I have added an additional > table property to provide optimize columns that is used by the compactor to > cluster the data by. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Attachment: HIVE-22736.patch Status: Patch Available (was: In Progress) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Status: In Progress (was: Patch Available) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: HIVE-22729.08.patch > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch, > HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch, > HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22538: -- Attachment: HIVE-22538.5.patch > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.patch > > Time Spent: 20m > Remaining Estimate: 0h > > For transactional tables, that property might be overriden to 1, which can > lead to merging final aggregation into a single stage (hence leading to > performance degradation). For instance, when autogather column stats is > enabled, this can happen for the following query: > {code} > set hive.support.concurrency=true; > set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > EXPLAIN > CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS > SELECT * FROM SRC x CLUSTER BY x.key; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22538: -- Status: Open (was: Patch Available) > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.patch > > Time Spent: 20m > Remaining Estimate: 0h > > For transactional tables, that property might be overriden to 1, which can > lead to merging final aggregation into a single stage (hence leading to > performance degradation). For instance, when autogather column stats is > enabled, this can happen for the following query: > {code} > set hive.support.concurrency=true; > set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > EXPLAIN > CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS > SELECT * FROM SRC x CLUSTER BY x.key; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[ https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-21487: - Attachment: HIVE-21847.02.patch > COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes > > > Key: HIVE-21487 > URL: https://issues.apache.org/jira/browse/HIVE-21487 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Todd Lipcon >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-21847.01.patch, HIVE-21847.02.patch > > > Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant > stream of queries of the form: > {code} > select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE = > 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION = > 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc; > {code} > but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting > in a full table scan over 115k rows, which takes around 100ms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Attachment: HIVE-22736.patch Status: Patch Available (was: In Progress) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22538: -- Status: Patch Available (was: Open) > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.patch > > Time Spent: 20m > Remaining Estimate: 0h > > For transactional tables, that property might be overriden to 1, which can > lead to merging final aggregation into a single stage (hence leading to > performance degradation). For instance, when autogather column stats is > enabled, this can happen for the following query: > {code} > set hive.support.concurrency=true; > set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > EXPLAIN > CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS > SELECT * FROM SRC x CLUSTER BY x.key; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22764) Create new command for "optimize" compaction and have basic implementation.
[ https://issues.apache.org/jira/browse/HIVE-22764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021905#comment-17021905 ] Laszlo Pinter commented on HIVE-22764: -- [~aditya-shah] Please provide a review board/pull request link to make the review easier. > Create new command for "optimize" compaction and have basic implementation. > --- > > Key: HIVE-22764 > URL: https://issues.apache.org/jira/browse/HIVE-22764 > Project: Hive > Issue Type: Sub-task >Reporter: Aditya Shah >Assignee: Aditya Shah >Priority: Major > Attachments: HIVE-22764.patch > > > Created new blocking compaction (added compaction type "optimize") by adding > a lock request on the compaction's transaction. It works mostly like > mmMajorCompaction and writes files w/o row_IDs. I have added an additional > table property to provide optimize columns that is used by the compactor to > cluster the data by. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[ https://issues.apache.org/jira/browse/HIVE-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021971#comment-17021971 ] Hive QA commented on HIVE-22366: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991597/HIVE-22366.6.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 17945 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20294/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20294/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20294/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12991597 - PreCommit-HIVE-Build > Multiple metastore calls for same table and constraints during planning > --- > > Key: HIVE-22366 > URL: https://issues.apache.org/jira/browse/HIVE-22366 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch, > HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch, HIVE-22366.6.patch > > > e.g. for a simple query like > {code:sql} > explain select count(i) from test2 group by j limit 3; > {code} > planner makes the following calls: > {code} > 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table : > tbl=hive.default.test2 > 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_primary_keys : tbl=hive.default.test2 > 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default > foreigntbl=test2 > 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_unique_constraints : tbl=hive.default.test2 > 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_not_null_constraints : tbl=hive.default.test2 > 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_partitions : tbl=hive.default.test2 > 2019-10-17T22:44:49,997 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_aggr_stats_for: table=hive.default.test2 > 2019-10-17T22:44:50,014 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table : > tbl=hive.default.test2 > 2019-10-17T22:44:50,023 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_primary_keys : tbl=hive.default.test2 > 2019-10-17T22:44:50,024 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default > foreigntbl=test2 > 2019-10-17T22:44:50,025 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_unique_constraints : tbl=hive.default.test2 > 2019-10-17T22:44:50,026 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main] > HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr > cmd=get_not_null_constraints : tbl=hive.default.test2 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Status: In Progress (was: Patch Available) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021993#comment-17021993 ] Hive QA commented on HIVE-22538: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 21s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 19s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 38s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 58s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 48s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 42s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 41s{color} | {color:red} ql: The patch generated 6 new + 500 unchanged - 3 fixed = 506 total (was 503) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 170 unchanged - 2 fixed = 172 total (was 172) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 29m 7s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-20298/dev-support/hive-personality.sh | | git revision | master / 6357dbc | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-20298/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-20298/yetus/diff-checkstyle-itests_hive-unit.txt | | modules | C: ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-20298/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.patch > > Time Spent: 20m > Remaining Estimate: 0h > > For transactional tables, that property
[jira] [Commented] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022052#comment-17022052 ] Hive QA commented on HIVE-22538: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991623/HIVE-22538.5.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17946 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[clusterctas] (batchId=55) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz_2] (batchId=91) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20298/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20298/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20298/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12991623 - PreCommit-HIVE-Build > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.patch > > Time Spent: 20m > Remaining Estimate: 0h > > For transactional tables, that property might be overriden to 1, which can > lead to merging final aggregation into a single stage (hence leading to > performance degradation). For instance, when autogather column stats is > enabled, this can happen for the following query: > {code} > set hive.support.concurrency=true; > set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > EXPLAIN > CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS > SELECT * FROM SRC x CLUSTER BY x.key; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22761) Scheduled query executor fails to report query state as errored if session initialization fails
[ https://issues.apache.org/jira/browse/HIVE-22761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021900#comment-17021900 ] Hive QA commented on HIVE-22761: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991584/HIVE-22761.01.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 17945 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20293/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20293/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20293/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12991584 - PreCommit-HIVE-Build > Scheduled query executor fails to report query state as errored if session > initialization fails > --- > > Key: HIVE-22761 > URL: https://issues.apache.org/jira/browse/HIVE-22761 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22761.01.patch > > > right now the info object is only initialized after the sessionstate is > inited - which might get into trouble... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22761) Scheduled query executor fails to report query state as errored if session initialization fails
[ https://issues.apache.org/jira/browse/HIVE-22761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021930#comment-17021930 ] Miklos Gergely commented on HIVE-22761: --- +1 > Scheduled query executor fails to report query state as errored if session > initialization fails > --- > > Key: HIVE-22761 > URL: https://issues.apache.org/jira/browse/HIVE-22761 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22761.01.patch > > > right now the info object is only initialized after the sessionstate is > inited - which might get into trouble... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?focusedWorklogId=376191=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376191 ] ASF GitHub Bot logged work on HIVE-22736: - Author: ASF GitHub Bot Created on: 23/Jan/20 11:11 Start Date: 23/Jan/20 11:11 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #883: HIVE-22736 Support multiple encryption zones in replication URL: https://github.com/apache/hive/pull/883#discussion_r370053902 ## File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java ## @@ -2835,17 +2834,25 @@ private boolean checkTableDataShouldBeDeleted(Table tbl, boolean deleteData) { * @param tablePath * @param ifPurge completely purge the table (skipping trash) while removing *data from warehouse - * @param db database the table belongs to + * @param shouldEnableCm If cm should be enabled */ -private void deleteTableData(Path tablePath, boolean ifPurge, Database db) { +private void deleteTableData(Path tablePath, boolean ifPurge, boolean shouldEnableCm) throws MetaException { + if (tablePath != null) { +wh.deleteDir(tablePath, true, ifPurge, shouldEnableCm); + } +} +/** + * Deletes the data in a table's location, if it fails logs an error. + * + * @param tablePath + * @param ifPurge completely purge the table (skipping trash) while removing + *data from warehouse + * @param db Database + */ +private void deleteTableData(Path tablePath, boolean ifPurge, Database db) throws MetaException { if (tablePath != null) { -try { - wh.deleteDir(tablePath, true, ifPurge, db); -} catch (Exception e) { - LOG.error("Failed to delete table directory: " + tablePath + - " " + e.getMessage()); -} +wh.deleteDir(tablePath, true, ifPurge, db); Review comment: for now ..just log the error ..ignore the exception while deleting. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 376191) Time Spent: 40m (was: 0.5h) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?focusedWorklogId=376189=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376189 ] ASF GitHub Bot logged work on HIVE-22736: - Author: ASF GitHub Bot Created on: 23/Jan/20 11:11 Start Date: 23/Jan/20 11:11 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #883: HIVE-22736 Support multiple encryption zones in replication URL: https://github.com/apache/hive/pull/883#discussion_r370049810 ## File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java ## @@ -493,4 +524,29 @@ public static String joinWithSeparator(Iterable strings) { public static String[] getListFromSeparatedString(String commaSeparatedString) { return commaSeparatedString.split("\\s*" + TXN_WRITE_EVENT_FILE_SEPARATOR + "\\s*"); } + + private static Path getCmRoot(Path path) throws IOException { +Path cmroot = null; +HdfsEncryptionShim pathEncryptionShim = hadoopShims.createHdfsEncryptionShim(path.getFileSystem(conf), conf); +if (!pathEncryptionShim.isPathEncrypted(path)) { + cmroot = cmRootMapping.get(NO_ENCRYPTION); +} else { + EncryptionZone encryptionZone = pathEncryptionShim.getEncryptionZoneForPath(path); + cmroot = cmRootMapping.get(encryptionZone.getPath()); Review comment: will it work fine if the path is not encrypted ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 376189) Time Spent: 40m (was: 0.5h) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?focusedWorklogId=376194=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376194 ] ASF GitHub Bot logged work on HIVE-22736: - Author: ASF GitHub Bot Created on: 23/Jan/20 11:11 Start Date: 23/Jan/20 11:11 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #883: HIVE-22736 Support multiple encryption zones in replication URL: https://github.com/apache/hive/pull/883#discussion_r370051549 ## File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java ## @@ -915,6 +915,8 @@ public static ConfVars getMetaConf(String name) { "This class is used to store and retrieval of raw metadata objects such as table, database"), REPLCMDIR("metastore.repl.cmrootdir", "hive.repl.cmrootdir", "/user/${system:user.name}/cmroot/", "Root dir for ChangeManager, used for deleted files."), +REPLCMENCRYPTEDDIR("metastore.encrypted.repl.cmrootdir", "hive.encrypted.repl.cmrootdir", "/cmroot/", +"Root dir for ChangeManager if encryption zones are enabled, used for deleted files."), Review comment: Same as above ..the name should start with metastore.repl.cm. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 376194) Time Spent: 50m (was: 40m) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?focusedWorklogId=376193=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376193 ] ASF GitHub Bot logged work on HIVE-22736: - Author: ASF GitHub Bot Created on: 23/Jan/20 11:11 Start Date: 23/Jan/20 11:11 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #883: HIVE-22736 Support multiple encryption zones in replication URL: https://github.com/apache/hive/pull/883#discussion_r370057676 ## File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java ## @@ -493,4 +524,29 @@ public static String joinWithSeparator(Iterable strings) { public static String[] getListFromSeparatedString(String commaSeparatedString) { return commaSeparatedString.split("\\s*" + TXN_WRITE_EVENT_FILE_SEPARATOR + "\\s*"); } + + private static Path getCmRoot(Path path) throws IOException { Review comment: Need some synchronization to avoid multiple path being added for same EZ..does file system gives that gurantee ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 376193) Time Spent: 50m (was: 40m) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?focusedWorklogId=376192=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376192 ] ASF GitHub Bot logged work on HIVE-22736: - Author: ASF GitHub Bot Created on: 23/Jan/20 11:11 Start Date: 23/Jan/20 11:11 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #883: HIVE-22736 Support multiple encryption zones in replication URL: https://github.com/apache/hive/pull/883#discussion_r370051185 ## File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java ## @@ -493,4 +524,29 @@ public static String joinWithSeparator(Iterable strings) { public static String[] getListFromSeparatedString(String commaSeparatedString) { return commaSeparatedString.split("\\s*" + TXN_WRITE_EVENT_FILE_SEPARATOR + "\\s*"); } + + private static Path getCmRoot(Path path) throws IOException { +Path cmroot = null; +HdfsEncryptionShim pathEncryptionShim = hadoopShims.createHdfsEncryptionShim(path.getFileSystem(conf), conf); +if (!pathEncryptionShim.isPathEncrypted(path)) { + cmroot = cmRootMapping.get(NO_ENCRYPTION); +} else { + EncryptionZone encryptionZone = pathEncryptionShim.getEncryptionZoneForPath(path); + cmroot = cmRootMapping.get(encryptionZone.getPath()); + if (cmroot == null) { +synchronized (instance) { + cmroot = new Path(path.getFileSystem(conf).getUri() + encryptionZone.getPath() + + MetastoreConf.getVar(conf, ConfVars.REPLCMENCRYPTEDDIR)); + FileSystem cmFs = cmroot.getFileSystem(conf); + // Create cmroot with permission 700 if not exist + if (!cmFs.exists(cmroot)) { +cmFs.mkdirs(cmroot); +cmFs.setPermission(cmroot, new FsPermission("700")); + } + cmRootMapping.put(encryptionZone.getPath(), cmroot); Review comment: Do we need a map..or a set is fine ..as the mapping from EZ root to CM root is constant ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 376192) Time Spent: 40m (was: 0.5h) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?focusedWorklogId=376190=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376190 ] ASF GitHub Bot logged work on HIVE-22736: - Author: ASF GitHub Bot Created on: 23/Jan/20 11:11 Start Date: 23/Jan/20 11:11 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #883: HIVE-22736 Support multiple encryption zones in replication URL: https://github.com/apache/hive/pull/883#discussion_r370046276 ## File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java ## @@ -213,9 +227,18 @@ public int recycle(Path path, RecycleType type, boolean ifPurge) throws IOExcept switch (type) { case MOVE: { LOG.info("Moving {} to {}", path.toString(), cmPath.toString()); - // Rename fails if the file with same name already exist. - success = fs.rename(path, cmPath); + Retry retriable = new Retry(IOException.class) { +@Override +public Boolean execute() throws IOException { + return fs.rename(path, cmPath); +} + }; + try { +success = retriable.run(); + } catch (Exception e) { +throw new MetaException(org.apache.hadoop.util.StringUtils.stringifyException(e)); Review comment: It can be a IOException This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 376190) Time Spent: 40m (was: 0.5h) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?focusedWorklogId=376186=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376186 ] ASF GitHub Bot logged work on HIVE-22736: - Author: ASF GitHub Bot Created on: 23/Jan/20 11:11 Start Date: 23/Jan/20 11:11 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #883: HIVE-22736 Support multiple encryption zones in replication URL: https://github.com/apache/hive/pull/883#discussion_r370040731 ## File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ## @@ -452,6 +452,8 @@ private static void populateLlapDaemonVarsSet(Set llapDaemonVarsSetLocal REPLCMRETIAN("hive.repl.cm.retain","24h", new TimeValidator(TimeUnit.HOURS), "Time to retain removed files in cmrootdir."), +REPLCMENCRYPTEDDIR("hive.encrypted.repl.cmrootdir", "/cmroot/", Review comment: it can be .cmroot This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 376186) Time Spent: 20m (was: 10m) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?focusedWorklogId=376188=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376188 ] ASF GitHub Bot logged work on HIVE-22736: - Author: ASF GitHub Bot Created on: 23/Jan/20 11:11 Start Date: 23/Jan/20 11:11 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #883: HIVE-22736 Support multiple encryption zones in replication URL: https://github.com/apache/hive/pull/883#discussion_r370045759 ## File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java ## @@ -138,15 +149,18 @@ private ReplChangeManager(Configuration conf) throws MetaException { if (!inited) { if (MetastoreConf.getBoolVar(conf, ConfVars.REPLCMENABLED)) { ReplChangeManager.enabled = true; - ReplChangeManager.cmroot = new Path(MetastoreConf.getVar(conf, ConfVars.REPLCMDIR)); ReplChangeManager.conf = conf; + //Create default cm root + Path cmroot = new Path(MetastoreConf.getVar(conf, ConfVars.REPLCMDIR)); FileSystem cmFs = cmroot.getFileSystem(conf); // Create cmroot with permission 700 if not exist if (!cmFs.exists(cmroot)) { cmFs.mkdirs(cmroot); cmFs.setPermission(cmroot, new FsPermission("700")); } + cmRootMapping.put(NO_ENCRYPTION, cmroot); Review comment: should we check if default path is not encrypted ..then only store it against NO_ENCRYPTION? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 376188) Time Spent: 0.5h (was: 20m) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?focusedWorklogId=376195=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376195 ] ASF GitHub Bot logged work on HIVE-22736: - Author: ASF GitHub Bot Created on: 23/Jan/20 11:11 Start Date: 23/Jan/20 11:11 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #883: HIVE-22736 Support multiple encryption zones in replication URL: https://github.com/apache/hive/pull/883#discussion_r370047831 ## File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java ## @@ -461,12 +485,19 @@ static void scheduleCMClearer(Configuration conf) { .namingPattern(CM_THREAD_NAME_PREFIX + "%d") .daemon(true) .build()); - executor.scheduleAtFixedRate(new CMClearer(MetastoreConf.getVar(conf, ConfVars.REPLCMDIR), - MetastoreConf.getTimeVar(conf, ConfVars.REPLCMRETIAN, TimeUnit.SECONDS), conf), - 0, MetastoreConf.getTimeVar(conf, ConfVars.REPLCMINTERVAL, TimeUnit.SECONDS), TimeUnit.SECONDS); + for (Path cmroot : cmRootMapping.values()) { Review comment: The map is populated at run time..so at init time there may be no entry added to the map. I think we should send the map to the scheduler and let it iterate to cleanup the cm root for each EZ in the map. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 376195) Time Spent: 1h (was: 50m) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?focusedWorklogId=376187=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376187 ] ASF GitHub Bot logged work on HIVE-22736: - Author: ASF GitHub Bot Created on: 23/Jan/20 11:11 Start Date: 23/Jan/20 11:11 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #883: HIVE-22736 Support multiple encryption zones in replication URL: https://github.com/apache/hive/pull/883#discussion_r370041263 ## File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ## @@ -452,6 +452,8 @@ private static void populateLlapDaemonVarsSet(Set llapDaemonVarsSetLocal REPLCMRETIAN("hive.repl.cm.retain","24h", new TimeValidator(TimeUnit.HOURS), "Time to retain removed files in cmrootdir."), +REPLCMENCRYPTEDDIR("hive.encrypted.repl.cmrootdir", "/cmroot/", Review comment: The name can hive.repl.cm.encryptionzone.rootdir This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 376187) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021975#comment-17021975 ] Hive QA commented on HIVE-22736: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991629/HIVE-22736.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20295/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20295/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20295/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2020-01-23 11:17:48.507 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-20295/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2020-01-23 11:17:48.510 + cd apache-github-source-source + git fetch origin >From https://github.com/apache/hive 05cabc8..6357dbc master -> origin/master + git reset --hard HEAD HEAD is now at 05cabc8 HIVE-22666: Introduce TopNKey operator for PTF Reduce Sink (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez) + git clean -f -d Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded. (use "git pull" to update your local branch) + git reset --hard origin/master HEAD is now at 6357dbc HIVE-22627: Add schema changes introduced in HIVE-21443 to the schema upgrade scripts (Zoltan Chovan via Peter Vary) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2020-01-23 11:17:50.077 + rm -rf ../yetus_PreCommit-HIVE-Build-20295 + mkdir ../yetus_PreCommit-HIVE-Build-20295 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-20295 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20295/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch Trying to apply the patch with -p0 error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableArchiveOperation.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableArchiveUtils.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/AlterTableUnarchiveOperation.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java: does not exist in index error: a/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java: does not exist in index error: a/shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java: does not exist in index error: a/standalone-metastore/metastore-common/pom.xml: does not exist in index error: a/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java: does not exist in index error: a/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java: does not exist in index error: a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java: does not exist in index error: a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java: does not exist in index error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not exist in
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021976#comment-17021976 ] Hive QA commented on HIVE-22729: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991615/HIVE-22729.07.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20296/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20296/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20296/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2020-01-23 11:19:03.943 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-20296/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2020-01-23 11:19:03.945 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 6357dbc HIVE-22627: Add schema changes introduced in HIVE-21443 to the schema upgrade scripts (Zoltan Chovan via Peter Vary) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 6357dbc HIVE-22627: Add schema changes introduced in HIVE-21443 to the schema upgrade scripts (Zoltan Chovan via Peter Vary) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2020-01-23 11:19:04.664 + rm -rf ../yetus_PreCommit-HIVE-Build-20296 + mkdir ../yetus_PreCommit-HIVE-Build-20296 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-20296 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20296/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch Trying to apply the patch with -p0 error: a/ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsDesc.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java: does not exist in index error: a/ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java: does not exist in index error: a/ql/src/test/results/clientpositive/dbtxnmgr_showlocks.q.out: does not exist in index error: a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java: does not exist in index error: a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponseElement.java: does not exist in index error: a/standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php: does not exist in index error: a/standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py: does not exist in index error: a/standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb: does not exist in index error: a/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift: does not exist in index error: a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java: does not exist in index error: a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java: does not exist in index error:
[jira] [Assigned] (HIVE-22765) Quote table names in CompactionTxnHandler
[ https://issues.apache.org/jira/browse/HIVE-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-22765: > Quote table names in CompactionTxnHandler > - > > Key: HIVE-22765 > URL: https://issues.apache.org/jira/browse/HIVE-22765 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Assignee: Zoltan Chovan >Priority: Major > > When I run a metastore backed up by a postgres db, I get the following > exception: > {code:java} > MetaException(message:Unable to connect to transaction database > org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does > not existMetaException(message:Unable to connect to transaction database > org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does > not exist Position: 83 at > org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2468) > at > org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2211) > at > org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:309) > at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446) at > org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370) at > org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:311) at > org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:297) at > org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:274) at > org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:225) at > org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108) > at > org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java) > at > org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:157) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.find_next_compact(HiveMetaStore.java:8061) > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) > at com.sun.proxy.$Proxy28.find_next_compact(Unknown Source) at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.findNextCompact(HiveMetaStoreClient.java:4046) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208) > at com.sun.proxy.$Proxy29.findNextCompact(Unknown Source) at > org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:102){code} > This is due to missing quotes around table names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22518) SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests
[ https://issues.apache.org/jira/browse/HIVE-22518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022033#comment-17022033 ] Zoltan Haindrich commented on HIVE-22518: - we were talking about this with [~mgergely], and he suggested to not set user.name in the qoption; but try to also migrate the other existing tests to use this option to set up authorization...for the latter; I'll open a followup - as that will be "only" a "bulk" change of q files... > SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests > > > Key: HIVE-22518 > URL: https://issues.apache.org/jira/browse/HIVE-22518 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22518.01.patch, HIVE-22518.01.patch, > HIVE-22518.02.patch, HIVE-22518.02.patch, HIVE-22518.03.patch, > HIVE-22518.03.patch, HIVE-22518.04.patch, HIVE-22518.05.patch, > HIVE-22518.05.patch, HIVE-22518.06.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22518) SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests
[ https://issues.apache.org/jira/browse/HIVE-22518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22518: Attachment: HIVE-22518.06.patch > SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests > > > Key: HIVE-22518 > URL: https://issues.apache.org/jira/browse/HIVE-22518 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22518.01.patch, HIVE-22518.01.patch, > HIVE-22518.02.patch, HIVE-22518.02.patch, HIVE-22518.03.patch, > HIVE-22518.03.patch, HIVE-22518.04.patch, HIVE-22518.05.patch, > HIVE-22518.05.patch, HIVE-22518.06.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021977#comment-17021977 ] Hive QA commented on HIVE-22736: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991629/HIVE-22736.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20297/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20297/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20297/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Tests exited with: Exception: Patch URL https://issues.apache.org/jira/secure/attachment/12991629/HIVE-22736.patch was found in seen patch url's cache and a test was probably run already on it. Aborting... {noformat} This message is automatically generated. ATTACHMENT ID: 12991629 - PreCommit-HIVE-Build > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22764) Create new command for "optimize" compaction and have basic implementation.
[ https://issues.apache.org/jira/browse/HIVE-22764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021893#comment-17021893 ] Aditya Shah edited comment on HIVE-22764 at 1/23/20 9:46 AM: - I have submitted a basic implementation for the new compaction. [~pvary] [~lpinter] can you please review the patch as well as the [[https://docs.google.com/document/d/10zWk7FR6I0CMy57Uykbkcox4HZTMQv2sgLoZrHVeLYU/edit?usp=sharing]] Thanks, Aditya was (Author: aditya-shah): I have submitted a basic implementation for the new compaction. [~pvary] [~lpinter] can you please review the patch as well as the [doc|[http://example.com|https://docs.google.com/document/d/10zWk7FR6I0CMy57Uykbkcox4HZTMQv2sgLoZrHVeLYU/edit?usp=sharing]] Thanks, Aditya > Create new command for "optimize" compaction and have basic implementation. > --- > > Key: HIVE-22764 > URL: https://issues.apache.org/jira/browse/HIVE-22764 > Project: Hive > Issue Type: Sub-task >Reporter: Aditya Shah >Assignee: Aditya Shah >Priority: Major > Attachments: HIVE-22764.patch > > > Created new blocking compaction (added compaction type "optimize") by adding > a lock request on the compaction's transaction. It works mostly like > mmMajorCompaction and writes files w/o row_IDs. I have added an additional > table property to provide optimize columns that is used by the compactor to > cluster the data by. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22627) Add schema changes introduced in HIVE-21443 to the schema upgrade scripts
[ https://issues.apache.org/jira/browse/HIVE-22627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021895#comment-17021895 ] Zoltan Chovan commented on HIVE-22627: -- [~pvary] rebase is done > Add schema changes introduced in HIVE-21443 to the schema upgrade scripts > - > > Key: HIVE-22627 > URL: https://issues.apache.org/jira/browse/HIVE-22627 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22627.2.patch, HIVE-22627.3.patch, > HIVE-22627.4.patch, HIVE-22627.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22764) Create new command for "optimize" compaction and have basic implementation.
[ https://issues.apache.org/jira/browse/HIVE-22764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021893#comment-17021893 ] Aditya Shah edited comment on HIVE-22764 at 1/23/20 9:46 AM: - I have submitted a basic implementation for the new compaction. [~pvary] [~lpinter] can you please review the patch as well as the doc.[[https://docs.google.com/document/d/10zWk7FR6I0CMy57Uykbkcox4HZTMQv2sgLoZrHVeLYU/edit?usp=sharing]] Thanks, Aditya was (Author: aditya-shah): I have submitted a basic implementation for the new compaction. [~pvary] [~lpinter] can you please review the patch as well as the [[https://docs.google.com/document/d/10zWk7FR6I0CMy57Uykbkcox4HZTMQv2sgLoZrHVeLYU/edit?usp=sharing]] Thanks, Aditya > Create new command for "optimize" compaction and have basic implementation. > --- > > Key: HIVE-22764 > URL: https://issues.apache.org/jira/browse/HIVE-22764 > Project: Hive > Issue Type: Sub-task >Reporter: Aditya Shah >Assignee: Aditya Shah >Priority: Major > Attachments: HIVE-22764.patch > > > Created new blocking compaction (added compaction type "optimize") by adding > a lock request on the compaction's transaction. It works mostly like > mmMajorCompaction and writes files w/o row_IDs. I have added an additional > table property to provide optimize columns that is used by the compactor to > cluster the data by. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021897#comment-17021897 ] Krisztian Kasa commented on HIVE-22538: --- 1. TestAcidOnTez.testBucketedAcidInsertWithRemoveUnion failed because the HiveConf (confForTez) was not passed to the runStatementOnDriver() method and this version of the method does create a new Driver instance. Calling the version which creates a new driver for each call fix this. 2. materialized_view_create_rewrite_4 failed at 4th execution of the command {code} ALTER MATERIALIZED VIEW cmv_mat_view_n5 REBUILD; {code} The prior call of this command creates base version of buckets when saving the mv's data. One of these buckets are empty and not in an acid format (it's schema doesn't contains the rowid) however the mv is defined to be acid. Adding the proper schema to the empty bucket when saving fix this. > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.patch > > Time Spent: 20m > Remaining Estimate: 0h > > For transactional tables, that property might be overriden to 1, which can > lead to merging final aggregation into a single stage (hence leading to > performance degradation). For instance, when autogather column stats is > enabled, this can happen for the following query: > {code} > set hive.support.concurrency=true; > set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > EXPLAIN > CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS > SELECT * FROM SRC x CLUSTER BY x.key; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22764) Create new command for "optimize" compaction and have basic implementation.
[ https://issues.apache.org/jira/browse/HIVE-22764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Shah updated HIVE-22764: --- Attachment: HIVE-22764.patch Status: Patch Available (was: Open) I have submitted a basic implementation for the new compaction. [~pvary] [~lpinter] can you please review the patch as well as the [doc|[http://example.com|https://docs.google.com/document/d/10zWk7FR6I0CMy57Uykbkcox4HZTMQv2sgLoZrHVeLYU/edit?usp=sharing]] Thanks, Aditya > Create new command for "optimize" compaction and have basic implementation. > --- > > Key: HIVE-22764 > URL: https://issues.apache.org/jira/browse/HIVE-22764 > Project: Hive > Issue Type: Sub-task >Reporter: Aditya Shah >Assignee: Aditya Shah >Priority: Major > Attachments: HIVE-22764.patch > > > Created new blocking compaction (added compaction type "optimize") by adding > a lock request on the compaction's transaction. It works mostly like > mmMajorCompaction and writes files w/o row_IDs. I have added an additional > table property to provide optimize columns that is used by the compactor to > cluster the data by. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[ https://issues.apache.org/jira/browse/HIVE-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021939#comment-17021939 ] Hive QA commented on HIVE-22366: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 36s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 15s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 53s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 22s{color} | {color:blue} standalone-metastore/metastore-common in master has 37 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 51s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 51s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 42s{color} | {color:red} ql: The patch generated 3 new + 570 unchanged - 37 fixed = 573 total (was 607) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 58s{color} | {color:red} ql generated 6 new + 1532 unchanged - 0 fixed = 1538 total (was 1532) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 33m 11s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | The field org.apache.hadoop.hive.ql.metadata.Table.cc is transient but isn't set by deserialization In Table.java:but isn't set by deserialization In Table.java | | | The field org.apache.hadoop.hive.ql.metadata.Table.dc is transient but isn't set by deserialization In Table.java:but isn't set by deserialization In Table.java | | | The field org.apache.hadoop.hive.ql.metadata.Table.fki is transient but isn't set by deserialization In Table.java:but isn't set by deserialization In Table.java | | | The field org.apache.hadoop.hive.ql.metadata.Table.nnc is transient but isn't set by deserialization In Table.java:but isn't set by deserialization In Table.java | | | The field org.apache.hadoop.hive.ql.metadata.Table.pki is transient but isn't set by deserialization In Table.java:but isn't set by deserialization In Table.java | | | The field org.apache.hadoop.hive.ql.metadata.Table.uki is transient but isn't set by deserialization In Table.java:but isn't set by deserialization In Table.java | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-20294/dev-support/hive-personality.sh | | git revision | master / 05cabc8 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-20294/yetus/diff-checkstyle-ql.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-20294/yetus/new-findbugs-ql.html | | modules | C: standalone-metastore/metastore-common ql U: .
[jira] [Commented] (HIVE-22706) Jdbc storage handler incorrectly interprets boolean column value in derby
[ https://issues.apache.org/jira/browse/HIVE-22706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021942#comment-17021942 ] Miklos Gergely commented on HIVE-22706: --- +1 > Jdbc storage handler incorrectly interprets boolean column value in derby > - > > Key: HIVE-22706 > URL: https://issues.apache.org/jira/browse/HIVE-22706 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22706.01.patch, HIVE-22706.01.patch, > HIVE-22706.01.patch, HIVE-22706.01.patch, HIVE-22706.02.patch, > HIVE-22706.02.patch, HIVE-22706.02.patch, HIVE-22706.02.patch > > > in case the column value is false ; the storage handler interprets it as true -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22627) Add schema changes introduced in HIVE-21443 to the schema upgrade scripts
[ https://issues.apache.org/jira/browse/HIVE-22627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-22627: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the patch [~zchovan]! > Add schema changes introduced in HIVE-21443 to the schema upgrade scripts > - > > Key: HIVE-22627 > URL: https://issues.apache.org/jira/browse/HIVE-22627 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22627.2.patch, HIVE-22627.3.patch, > HIVE-22627.4.patch, HIVE-22627.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021964#comment-17021964 ] Eugene Chung edited comment on HIVE-22126 at 1/23/20 11:02 AM: --- * [^HIVE-22126.02.patch] * Now the packages of guava are properly shaded. * As a result, calcite APIs with guava, which are deprecated, cannot be used anymore. was (Author: euigeun_chung): * Now the packages of guava are properly shaded. * As a result, calcite APIs with guava, which are deprecated, cannot be used anymore. > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Eugene Chung >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Chung updated HIVE-22126: Attachment: HIVE-22126.02.patch Status: Patch Available (was: Reopened) * Now the packages of guava are properly shaded. * As a result, calcite APIs with guava, which are deprecated, cannot be used anymore. > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Eugene Chung >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22761) Scheduled query executor fails to report query state as errored if session initialization fails
[ https://issues.apache.org/jira/browse/HIVE-22761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021863#comment-17021863 ] Hive QA commented on HIVE-22761: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 31s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 49s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 23m 45s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-20293/dev-support/hive-personality.sh | | git revision | master / 05cabc8 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-20293/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Scheduled query executor fails to report query state as errored if session > initialization fails > --- > > Key: HIVE-22761 > URL: https://issues.apache.org/jira/browse/HIVE-22761 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22761.01.patch > > > right now the info object is only initialized after the sessionstate is > inited - which might get into trouble... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Attachment: HIVE-22736.patch Status: Patch Available (was: In Progress) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Status: In Progress (was: Patch Available) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021849#comment-17021849 ] Hive QA commented on HIVE-22726: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991579/HIVE-22726.3.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20292/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20292/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20292/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2020-01-23 08:18:56.259 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-20292/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2020-01-23 08:18:56.262 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 05cabc8 HIVE-22666: Introduce TopNKey operator for PTF Reduce Sink (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez) + git clean -f -d Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 05cabc8 HIVE-22666: Introduce TopNKey operator for PTF Reduce Sink (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2020-01-23 08:18:57.371 + rm -rf ../yetus_PreCommit-HIVE-Build-20292 + mkdir ../yetus_PreCommit-HIVE-Build-20292 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-20292 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20292/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch Trying to apply the patch with -p0 error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyFilter.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorTopNKeyOperator.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/wrapper/VectorHashKeyWrapperBatch.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/wrapper/VectorHashKeyWrapperGeneralComparator.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java: does not exist in index Trying to apply the patch with -p1 error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java:38 Falling back to three-way merge... Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java' with conflicts. error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java:58 Falling back to three-way merge... Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java' with conflicts. Going to apply patch with: git apply -p1 error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java:38 Falling back to three-way merge... Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java' with conflicts. error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java:58 Falling back to three-way merge... Applied patch to
[jira] [Commented] (HIVE-22437) LLAP Metadata cache NPE on locking metadata.
[ https://issues.apache.org/jira/browse/HIVE-22437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021845#comment-17021845 ] Hive QA commented on HIVE-22437: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991554/HIVE-22437.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 17934 tests executed *Failed tests:* {noformat} TestJdbcWithMiniLlapArrow - did not produce a TEST-*.xml file (likely timed out) (batchId=288) org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query20] (batchId=303) org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryEnabledBothTableTypes (batchId=227) org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoverySkipInvalidPath (batchId=227) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosIncrementalLoadAcidTables.org.apache.hadoop.hive.ql.parse.TestReplicationScenariosIncrementalLoadAcidTables (batchId=276) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20291/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20291/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20291/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12991554 - PreCommit-HIVE-Build > LLAP Metadata cache NPE on locking metadata. > > > Key: HIVE-22437 > URL: https://issues.apache.org/jira/browse/HIVE-22437 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22437.patch > > > {code} > java.lang.NullPointerException > at > org.apache.hadoop.hive.llap.io.metadata.MetadataCache.unlockSingleBuffer(MetadataCache.java:464) > at > org.apache.hadoop.hive.llap.io.metadata.MetadataCache.lockBuffer(MetadataCache.java:409) > at > org.apache.hadoop.hive.llap.io.metadata.MetadataCache.lockOldVal(MetadataCache.java:314) > at > org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putInternal(MetadataCache.java:287) > at > org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:199) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22762) Leap day is incorrectly parsed during cast in Hive
[ https://issues.apache.org/jira/browse/HIVE-22762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-22762: > Leap day is incorrectly parsed during cast in Hive > -- > > Key: HIVE-22762 > URL: https://issues.apache.org/jira/browse/HIVE-22762 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Minor > Fix For: 4.0.0 > > > While casting a string to a date with a custom date format having day token > before year and moth tokens, the date is parsed incorrectly for leap days. > h3. How to reproduce > Execute {code}select cast("29 02 0" as date format "dd mm rr"){code} with > Hive. The query results in *2020-02-28*, incorrectly. > > Executing the another cast with a slightly modified representation of the > date (day is preceded by year and moth) is however correctly parsed: > {code}select cast("0 02 29" as date format "rr mm dd"){code} > It returns *2020-02-29*. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22763) 0 is accepted in 12-hour format during timestamp cast
[ https://issues.apache.org/jira/browse/HIVE-22763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-22763: > 0 is accepted in 12-hour format during timestamp cast > - > > Key: HIVE-22763 > URL: https://issues.apache.org/jira/browse/HIVE-22763 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Minor > Fix For: 4.0.0 > > > Having a timestamp string in 12-hour format can be parsed if the hour is 0, > however, based on the [design > document|https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit], > it should be rejected. > h3. How to reproduce > Run {code}select cast("2020-01-01 0 am 00" as timestamp format "-mm-dd > hh12 p.m. ss"){code} > It shouldn' t be parsed, as the hour component is 0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: HIVE-22729.07.patch > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch, > HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch, > HIVE-22729.06.patch, HIVE-22729.07.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22726: - Attachment: (was: HIVE-22726.3.patch) > TopN Key optimizer should use array instead of priority queue > - > > Key: HIVE-22726 > URL: https://issues.apache.org/jira/browse/HIVE-22726 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22726.1.patch, HIVE-22726.2.patch, > HIVE-22726.3.patch > > > The TopN key optimizer currently uses a priority queue for keeping track of > the largest/smallest rows. Its max size is the same as the user specified > limit. This should be replaced a more cache line friendly array with a small > (128) maximum size and see how much performance is gained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22764) Create new command for "optimize" compaction and have basic implementation.
[ https://issues.apache.org/jira/browse/HIVE-22764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022140#comment-17022140 ] Hive QA commented on HIVE-22764: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991626/HIVE-22764.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17946 tests executed *Failed tests:* {noformat} org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions (batchId=291) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll (batchId=291) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes (batchId=291) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20299/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20299/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20299/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12991626 - PreCommit-HIVE-Build > Create new command for "optimize" compaction and have basic implementation. > --- > > Key: HIVE-22764 > URL: https://issues.apache.org/jira/browse/HIVE-22764 > Project: Hive > Issue Type: Sub-task >Reporter: Aditya Shah >Assignee: Aditya Shah >Priority: Major > Attachments: HIVE-22764.patch > > > Created new blocking compaction (added compaction type "optimize") by adding > a lock request on the compaction's transaction. It works mostly like > mmMajorCompaction and writes files w/o row_IDs. I have added an additional > table property to provide optimize columns that is used by the compactor to > cluster the data by. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22766) Upgrade to Kryo4
[ https://issues.apache.org/jira/browse/HIVE-22766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022142#comment-17022142 ] László Bodor commented on HIVE-22766: - Sorry, I've just realized that upstream it's already done by HIVE-21440, closing this as invalid. > Upgrade to Kryo4 > > > Key: HIVE-22766 > URL: https://issues.apache.org/jira/browse/HIVE-22766 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > > Kryo 4 FieldSerializer is more flexible in a way that allows us to change the > serialization/deserialization logic to be changed more easily. From > performance point of view, I'm not sure, but I don't think we need serious > benefit there. > Kryo 4.0.0 release notes: > https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.0 > https://github.com/EsotericSoftware/kryo#kryo-versioning-and-upgrading -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-22766) Upgrade to Kryo4
[ https://issues.apache.org/jira/browse/HIVE-22766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor resolved HIVE-22766. - Resolution: Won't Fix > Upgrade to Kryo4 > > > Key: HIVE-22766 > URL: https://issues.apache.org/jira/browse/HIVE-22766 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Priority: Major > > Kryo 4 FieldSerializer is more flexible in a way that allows us to change the > serialization/deserialization logic to be changed more easily. From > performance point of view, I'm not sure, but I don't think we need serious > benefit there. > Kryo 4.0.0 release notes: > https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.0 > https://github.com/EsotericSoftware/kryo#kryo-versioning-and-upgrading -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22766) Upgrade to Kryo4
[ https://issues.apache.org/jira/browse/HIVE-22766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor reassigned HIVE-22766: --- Assignee: (was: László Bodor) > Upgrade to Kryo4 > > > Key: HIVE-22766 > URL: https://issues.apache.org/jira/browse/HIVE-22766 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Priority: Major > > Kryo 4 FieldSerializer is more flexible in a way that allows us to change the > serialization/deserialization logic to be changed more easily. From > performance point of view, I'm not sure, but I don't think we need serious > benefit there. > Kryo 4.0.0 release notes: > https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.0 > https://github.com/EsotericSoftware/kryo#kryo-versioning-and-upgrading -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?focusedWorklogId=376258=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376258 ] ASF GitHub Bot logged work on HIVE-22736: - Author: ASF GitHub Bot Created on: 23/Jan/20 14:32 Start Date: 23/Jan/20 14:32 Worklog Time Spent: 10m Work Description: aasha commented on pull request #883: HIVE-22736 Support multiple encryption zones in replication URL: https://github.com/apache/hive/pull/883#discussion_r370151392 ## File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java ## @@ -138,15 +149,18 @@ private ReplChangeManager(Configuration conf) throws MetaException { if (!inited) { if (MetastoreConf.getBoolVar(conf, ConfVars.REPLCMENABLED)) { ReplChangeManager.enabled = true; - ReplChangeManager.cmroot = new Path(MetastoreConf.getVar(conf, ConfVars.REPLCMDIR)); ReplChangeManager.conf = conf; + //Create default cm root + Path cmroot = new Path(MetastoreConf.getVar(conf, ConfVars.REPLCMDIR)); FileSystem cmFs = cmroot.getFileSystem(conf); // Create cmroot with permission 700 if not exist if (!cmFs.exists(cmroot)) { cmFs.mkdirs(cmroot); cmFs.setPermission(cmroot, new FsPermission("700")); } + cmRootMapping.put(NO_ENCRYPTION, cmroot); Review comment: This is for REPLCMDIR. This will always be unencrypted. Encryption cm directory we have separated. If there is any unencrypted path for which recycle is called, we will use this. Else we will create a cm for that zone This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 376258) Time Spent: 1h 10m (was: 1h) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22679) Replace Base64 in metastore-common Package
[ https://issues.apache.org/jira/browse/HIVE-22679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022189#comment-17022189 ] David Mollitor commented on HIVE-22679: --- [~ngangam] Can you please take a look at this one? > Replace Base64 in metastore-common Package > -- > > Key: HIVE-22679 > URL: https://issues.apache.org/jira/browse/HIVE-22679 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22679.1.patch, HIVE-22679.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22538: -- Status: Open (was: Patch Available) > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch, HIVE-22538.patch > > Time Spent: 20m > Remaining Estimate: 0h > > For transactional tables, that property might be overriden to 1, which can > lead to merging final aggregation into a single stage (hence leading to > performance degradation). For instance, when autogather column stats is > enabled, this can happen for the following query: > {code} > set hive.support.concurrency=true; > set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > EXPLAIN > CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS > SELECT * FROM SRC x CLUSTER BY x.key; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22538: -- Attachment: HIVE-22538.6.patch > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch, HIVE-22538.patch > > Time Spent: 20m > Remaining Estimate: 0h > > For transactional tables, that property might be overriden to 1, which can > lead to merging final aggregation into a single stage (hence leading to > performance degradation). For instance, when autogather column stats is > enabled, this can happen for the following query: > {code} > set hive.support.concurrency=true; > set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > EXPLAIN > CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS > SELECT * FROM SRC x CLUSTER BY x.key; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[ https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22538: -- Status: Patch Available (was: Open) > RS deduplication does not always enforce > hive.optimize.reducededuplication.min.reducer > -- > > Key: HIVE-22538 > URL: https://issues.apache.org/jira/browse/HIVE-22538 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch, > HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch, HIVE-22538.patch > > Time Spent: 20m > Remaining Estimate: 0h > > For transactional tables, that property might be overriden to 1, which can > lead to merging final aggregation into a single stage (hence leading to > performance degradation). For instance, when autogather column stats is > enabled, this can happen for the following query: > {code} > set hive.support.concurrency=true; > set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > EXPLAIN > CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS > SELECT * FROM SRC x CLUSTER BY x.key; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22764) Create new command for "optimize" compaction and have basic implementation.
[ https://issues.apache.org/jira/browse/HIVE-22764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022100#comment-17022100 ] Hive QA commented on HIVE-22764: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 33s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 20s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 33s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 33s{color} | {color:blue} standalone-metastore/metastore-common in master has 37 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 13s{color} | {color:blue} standalone-metastore/metastore-server in master has 181 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 59s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 45s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 49s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 47s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 25s{color} | {color:red} standalone-metastore/metastore-server: The patch generated 10 new + 705 unchanged - 0 fixed = 715 total (was 705) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 42s{color} | {color:red} ql: The patch generated 7 new + 46 unchanged - 2 fixed = 53 total (was 48) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s{color} | {color:red} itests/hive-unit: The patch generated 4 new + 11 unchanged - 0 fixed = 15 total (was 11) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 48s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 45m 54s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-20299/dev-support/hive-personality.sh | | git revision | master / 6357dbc | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-20299/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-20299/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-20299/yetus/diff-checkstyle-itests_hive-unit.txt | | modules | C: standalone-metastore/metastore-common standalone-metastore/metastore-server ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-20299/yetus.txt | | Powered by | Apache Yetus
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022130#comment-17022130 ] Denys Kuzmenko commented on HIVE-20801: --- rebased > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal Vijayaraghavan >Assignee: Gopal Vijayaraghavan >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch, HIVE-20801.4.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko reassigned HIVE-20801: - Assignee: Gopal Vijayaraghavan (was: Denys Kuzmenko) > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal Vijayaraghavan >Assignee: Gopal Vijayaraghavan >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch, HIVE-20801.4.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-20801: -- Attachment: HIVE-20801.4.patch > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal Vijayaraghavan >Assignee: Denys Kuzmenko >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch, HIVE-20801.4.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko reassigned HIVE-20801: - Assignee: Denys Kuzmenko (was: Gopal Vijayaraghavan) > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal Vijayaraghavan >Assignee: Denys Kuzmenko >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch, HIVE-20801.4.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22766) Upgrade to Kryo4
[ https://issues.apache.org/jira/browse/HIVE-22766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22766: Description: Kryo 4 FieldSerializer is more flexible in a way that it allows us to change the serialization/deserialization logic to be changed more easily. From performance point of view, I'm not sure, but I don't think we need serious benefit there. > Upgrade to Kryo4 > > > Key: HIVE-22766 > URL: https://issues.apache.org/jira/browse/HIVE-22766 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > > Kryo 4 FieldSerializer is more flexible in a way that it allows us to change > the serialization/deserialization logic to be changed more easily. From > performance point of view, I'm not sure, but I don't think we need serious > benefit there. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22726: - Attachment: HIVE-22726.3.patch > TopN Key optimizer should use array instead of priority queue > - > > Key: HIVE-22726 > URL: https://issues.apache.org/jira/browse/HIVE-22726 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22726.1.patch, HIVE-22726.2.patch, > HIVE-22726.3.patch > > > The TopN key optimizer currently uses a priority queue for keeping track of > the largest/smallest rows. Its max size is the same as the user specified > limit. This should be replaced a more cache line friendly array with a small > (128) maximum size and see how much performance is gained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22726: - Status: Patch Available (was: Open) > TopN Key optimizer should use array instead of priority queue > - > > Key: HIVE-22726 > URL: https://issues.apache.org/jira/browse/HIVE-22726 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22726.1.patch, HIVE-22726.2.patch, > HIVE-22726.3.patch > > > The TopN key optimizer currently uses a priority queue for keeping track of > the largest/smallest rows. Its max size is the same as the user specified > limit. This should be replaced a more cache line friendly array with a small > (128) maximum size and see how much performance is gained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22726: - Status: Open (was: Patch Available) > TopN Key optimizer should use array instead of priority queue > - > > Key: HIVE-22726 > URL: https://issues.apache.org/jira/browse/HIVE-22726 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22726.1.patch, HIVE-22726.2.patch, > HIVE-22726.3.patch > > > The TopN key optimizer currently uses a priority queue for keeping track of > the largest/smallest rows. Its max size is the same as the user specified > limit. This should be replaced a more cache line friendly array with a small > (128) maximum size and see how much performance is gained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?focusedWorklogId=376259=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376259 ] ASF GitHub Bot logged work on HIVE-22736: - Author: ASF GitHub Bot Created on: 23/Jan/20 14:34 Start Date: 23/Jan/20 14:34 Worklog Time Spent: 10m Work Description: aasha commented on pull request #883: HIVE-22736 Support multiple encryption zones in replication URL: https://github.com/apache/hive/pull/883#discussion_r370152190 ## File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java ## @@ -213,9 +227,18 @@ public int recycle(Path path, RecycleType type, boolean ifPurge) throws IOExcept switch (type) { case MOVE: { LOG.info("Moving {} to {}", path.toString(), cmPath.toString()); - // Rename fails if the file with same name already exist. - success = fs.rename(path, cmPath); + Retry retriable = new Retry(IOException.class) { +@Override +public Boolean execute() throws IOException { + return fs.rename(path, cmPath); +} + }; + try { +success = retriable.run(); + } catch (Exception e) { +throw new MetaException(org.apache.hadoop.util.StringUtils.stringifyException(e)); Review comment: Yes in this case it is just an IOException. But wanted to keep the Retry interface generic. So added Exception. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 376259) Time Spent: 1h 20m (was: 1h 10m) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022182#comment-17022182 ] Hive QA commented on HIVE-22736: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 29s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 50s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 14s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 2s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 17s{color} | {color:blue} shims/common in master has 6 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 21s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 23s{color} | {color:blue} standalone-metastore/metastore-common in master has 37 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 33s{color} | {color:blue} common in master has 63 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 8s{color} | {color:blue} standalone-metastore/metastore-server in master has 181 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 56s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 10s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 17s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 9s{color} | {color:red} shims/common: The patch generated 1 new + 93 unchanged - 1 fixed = 94 total (was 94) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} standalone-metastore/metastore-common: The patch generated 3 new + 98 unchanged - 0 fixed = 101 total (was 98) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 16s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 18s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 51m 40s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile xml | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality |
[jira] [Updated] (HIVE-22679) Replace Base64 in metastore-common Package
[ https://issues.apache.org/jira/browse/HIVE-22679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22679: -- Attachment: (was: HIVE-22679.1.patch) > Replace Base64 in metastore-common Package > -- > > Key: HIVE-22679 > URL: https://issues.apache.org/jira/browse/HIVE-22679 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22679.1.patch, HIVE-22679.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22679) Replace Base64 in metastore-common Package
[ https://issues.apache.org/jira/browse/HIVE-22679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22679: -- Attachment: (was: HIVE-22679.1.patch) > Replace Base64 in metastore-common Package > -- > > Key: HIVE-22679 > URL: https://issues.apache.org/jira/browse/HIVE-22679 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22679.1.patch, HIVE-22679.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22766) Upgrade to Kryo4
[ https://issues.apache.org/jira/browse/HIVE-22766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor reassigned HIVE-22766: --- Assignee: László Bodor > Upgrade to Kryo4 > > > Key: HIVE-22766 > URL: https://issues.apache.org/jira/browse/HIVE-22766 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22766) Upgrade to Kryo4
[ https://issues.apache.org/jira/browse/HIVE-22766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22766: Description: Kryo 4 FieldSerializer is more flexible in a way that allows us to change the serialization/deserialization logic to be changed more easily. From performance point of view, I'm not sure, but I don't think we need serious benefit there. Kryo 4.0.0 release notes: https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.0 https://github.com/EsotericSoftware/kryo#kryo-versioning-and-upgrading was: Kryo 4 FieldSerializer is more flexible in a way that it allows us to change the serialization/deserialization logic to be changed more easily. From performance point of view, I'm not sure, but I don't think we need serious benefit there. > Upgrade to Kryo4 > > > Key: HIVE-22766 > URL: https://issues.apache.org/jira/browse/HIVE-22766 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > > Kryo 4 FieldSerializer is more flexible in a way that allows us to change the > serialization/deserialization logic to be changed more easily. From > performance point of view, I'm not sure, but I don't think we need serious > benefit there. > Kryo 4.0.0 release notes: > https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.0 > https://github.com/EsotericSoftware/kryo#kryo-versioning-and-upgrading -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22767) beeline doesn't parse semicolons in comments properly
[ https://issues.apache.org/jira/browse/HIVE-22767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Matyus reassigned HIVE-22767: Assignee: Zoltan Matyus > beeline doesn't parse semicolons in comments properly > - > > Key: HIVE-22767 > URL: https://issues.apache.org/jira/browse/HIVE-22767 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Zoltan Matyus >Assignee: Zoltan Matyus >Priority: Major > > HIVE-12646 fixed the handling of semicolons in quoted strings, but leaves the > problem of semicolons in comments. E.g. with beeline connected to any > database... > this works: {code:sql}select 1; select /* */ 2; select /* */ 3;{code} > this doesn't work: {code:sql}select 1; select /* ; */ 2; select /* ; */ > 3;{code} > This has been fixed and reintroduced before (possibly multiple times). > Ideally, there should be a single utility method somewhere to separate > comments, strings and commands -- with the proper testing in place (q files). > However, I'm trying to make this fix back-portable, so a light touch is > needed. I'm focusing on beeline for now, and only writing (very thorough) > unit tests, as I cannot exclude any new q files from TestCliDriver (which > would break, since it's using a different parsing method). > > P.S. excerpt of the error message: > {noformat} > 0: jdbc:hive2://...> select 1; select /* ; */ 2; select /* ; */ 3; > INFO : Compiling command(queryId=...): select 1 > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, > type:int, comment:null)], properties:null) > INFO : Completed compiling command(queryId=...); Time taken: 0.38 seconds > INFO : Executing command(queryId=...): select 1 > INFO : Completed executing command(queryId=...); Time taken: 0.004 seconds > INFO : OK > +--+ > | _c0 | > +--+ > | 1| > +--+ > 1 row selected (2.007 seconds) > INFO : Compiling command(queryId=...): select /* > ERROR : FAILED: ParseException line 1:9 cannot recognize input near '' > '' '' in select clause > org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize > input near '' '' '' in select clause > at > org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:233) > at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:79) > at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:72) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:598) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1505) > at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1452) > at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1447) > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126) > at ... > {noformat} > > Similarly, the following query also fails: > {code:sql}select /* ' */ 1; select /* ' */ 2;{code} > I suspect line comments are also not handled properly but I cannot reproduce > this in interactive beeline... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22746) Make TypeCheckProcFactory generic
[ https://issues.apache.org/jira/browse/HIVE-22746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022306#comment-17022306 ] Jesus Camacho Rodriguez commented on HIVE-22746: [~vgarg], [~mgergely], can any of you review this patch? https://github.com/apache/hive/pull/887 Cc [~scarlin] > Make TypeCheckProcFactory generic > - > > Key: HIVE-22746 > URL: https://issues.apache.org/jira/browse/HIVE-22746 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22746.01.patch, HIVE-22746.01.patch, > HIVE-22746.patch > > Time Spent: 10m > Remaining Estimate: 0h > > {{TypeCheckProcFactory}} is responsible for processing AST nodes and > generating ExprNode objects from them. When we generate the expressions for > Calcite planning, we go through a {{AST node -> ExprNode -> RexNode}} > transformation. We would like to avoid the overhead of going through the > ExprNode, and thus generate directly the RexNode from the AST. > To do that, the first step is to make {{TypeCheckProcFactory}} generic, so it > can receive an expression factory and create expressions in different realms. > For the time being, the only factory implementation is the ExprNode factory. > Thus, this patch focuses mainly on refactoring {{TypeCheckProcFactory}} > without breaking anything that is already working. > In a follow-up patch, we will create a {{RexNode}} factory and use it when we > parse the query in CalcitePlanner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022200#comment-17022200 ] Hive QA commented on HIVE-22736: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12991649/HIVE-22736.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 17958 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_ddl1] (batchId=92) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2] (batchId=175) org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testPartitionOpsForUpdateUsingEvents (batchId=249) org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testTableColumnStatistics (batchId=249) org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testTableColumnStatisticsTxnTable (batchId=249) org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testTableOpsForUpdateUsingEvents (batchId=249) org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testDumpAcidTableWithTableDirMissing (batchId=273) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testDumpAcidTableWithTableDirMissing (batchId=275) org.apache.hadoop.hive.ql.txn.compactor.TestCleanerWithReplication.cleanupAfterMajorPartitionCompaction (batchId=279) org.apache.hadoop.hive.ql.txn.compactor.TestCleanerWithReplication.cleanupAfterMajorTableCompaction (batchId=279) org.apache.hadoop.hive.ql.txn.compactor.TestCleanerWithReplication.cleanupAfterMinorPartitionCompaction (batchId=279) org.apache.hadoop.hive.ql.txn.compactor.TestCleanerWithReplication.cleanupAfterMinorTableCompaction (batchId=279) org.apache.hive.hcatalog.listener.TestDbNotificationListener.insertPartition (batchId=297) org.apache.hive.hcatalog.listener.TestDbNotificationListener.insertTable (batchId=297) org.apache.hive.hcatalog.listener.TestDbNotificationListener.sqlCTAS (batchId=297) org.apache.hive.hcatalog.listener.TestDbNotificationListener.sqlInsertPartition (batchId=297) org.apache.hive.hcatalog.listener.TestDbNotificationListener.sqlInsertTable (batchId=297) org.apache.hive.jdbc.TestJdbcDriver2.testGetQueryLogForReplCommands (batchId=289) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testReplDumpResultSet (batchId=291) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20300/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20300/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20300/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 19 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12991649 - PreCommit-HIVE-Build > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-10362) Support Type check/conversion in dynamic partition column
[ https://issues.apache.org/jira/browse/HIVE-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-10362: - Attachment: HIVE-10362.01.patch Fix Version/s: 4.0.0 Affects Version/s: 4.0.0 3.0.0 Status: Patch Available (was: Open) > Support Type check/conversion in dynamic partition column > - > > Key: HIVE-10362 > URL: https://issues.apache.org/jira/browse/HIVE-10362 > Project: Hive > Issue Type: Improvement > Components: Query Processor, Types >Affects Versions: 3.0.0, 1.0.0, 4.0.0 >Reporter: Chaoyu Tang >Assignee: Karen Coppage >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-10362.01.patch > > > There are quite a lot of issues associated with the non-noramlized or > type-mismatched values for partition column. Hive has many ways to introduce > such problematic data. HIVE-10307 mainly provides the support to type > check/convert/normalize the partition column value in static partition > specification. This JIRA tries to deal with the partition column type in > dynamic partition insert. Currently any data can be inserted as a partition > column value as long as it is quoted as a string. For example, > create table dynparttypechecknum (key int, value string) partitioned by (part > int); > insert into dynparttypechecknum partition (part) select key, value, '1' > from src limit 1; > show partitions dynparttypechecknum; > -- > part=1 > The partition column value is non-normalized int 1. It causes some > unnecessary problems such as integer partition column JDO filter pushdown > (see HIVE-6052) and others like HIVE-10210. -- This message was sent by Atlassian Jira (v8.3.4#803005)