[jira] [Commented] (HIVE-21746) ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled
[ https://issues.apache.org/jira/browse/HIVE-21746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871659#comment-16871659 ] Jesus Camacho Rodriguez commented on HIVE-21746: +1 > ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with > CBO disabled > -- > > Key: HIVE-21746 > URL: https://issues.apache.org/jira/browse/HIVE-21746 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-21746.hive2.patch, HIVE-21746.patch, > HIVE-21746.patch, HIVE-21746.patch > > > ArrayIndexOutOfBounds exception during query execution with dynamically > partitioned hash join. > Found on Hive 2.x. Seems to occur with CBO disabled/failed. > Disabling constant propagation seems to allow the query to succeed. > {noformat} > java.lang.ArrayIndexOutOfBoundsException: 203 > at > org.apache.hadoop.hive.serde2.io.TimestampWritable.getTotalLength(TimestampWritable.java:217) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:205) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getFieldsAsList(LazyBinaryStruct.java:281) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.unpack(MapJoinBytesTableContainer.java:744) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:730) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:605) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:70) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:34) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:819) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:924) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:456) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:359) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:290) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:189) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:377) > ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3] > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) > ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3] > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) > ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3] > at java.security.AccessController.doPrivileged(Native Method) > ~[?:1.8.0_112] > at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112] > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) > ~[hadoop-common-2.7.3.2.6.4.119-3.jar:?] > at >
[jira] [Commented] (HIVE-21746) ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled
[ https://issues.apache.org/jira/browse/HIVE-21746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871654#comment-16871654 ] Jason Dere commented on HIVE-21746: --- [~jcamachorodriguez] I was unable to get a qfile test working for this one, but would you still be able to +1 this patch as this does fix error? > ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with > CBO disabled > -- > > Key: HIVE-21746 > URL: https://issues.apache.org/jira/browse/HIVE-21746 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-21746.hive2.patch, HIVE-21746.patch, > HIVE-21746.patch, HIVE-21746.patch > > > ArrayIndexOutOfBounds exception during query execution with dynamically > partitioned hash join. > Found on Hive 2.x. Seems to occur with CBO disabled/failed. > Disabling constant propagation seems to allow the query to succeed. > {noformat} > java.lang.ArrayIndexOutOfBoundsException: 203 > at > org.apache.hadoop.hive.serde2.io.TimestampWritable.getTotalLength(TimestampWritable.java:217) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:205) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getFieldsAsList(LazyBinaryStruct.java:281) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.unpack(MapJoinBytesTableContainer.java:744) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:730) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:605) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:70) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:34) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:819) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:924) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:456) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:359) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:290) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:189) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:377) > ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3] > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) > ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3] > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) > ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3] > at java.security.AccessController.doPrivileged(Native Method) > ~[?:1.8.0_112] > at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112] > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) >
[jira] [Commented] (HIVE-21746) ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled
[ https://issues.apache.org/jira/browse/HIVE-21746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846317#comment-16846317 ] Hive QA commented on HIVE-21746: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12969438/HIVE-21746.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 16057 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17283/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17283/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17283/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12969438 - PreCommit-HIVE-Build > ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with > CBO disabled > -- > > Key: HIVE-21746 > URL: https://issues.apache.org/jira/browse/HIVE-21746 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-21746.hive2.patch, HIVE-21746.patch, > HIVE-21746.patch, HIVE-21746.patch > > > ArrayIndexOutOfBounds exception during query execution with dynamically > partitioned hash join. > Found on Hive 2.x. Seems to occur with CBO disabled/failed. > Disabling constant propagation seems to allow the query to succeed. > {noformat} > java.lang.ArrayIndexOutOfBoundsException: 203 > at > org.apache.hadoop.hive.serde2.io.TimestampWritable.getTotalLength(TimestampWritable.java:217) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:205) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getFieldsAsList(LazyBinaryStruct.java:281) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.unpack(MapJoinBytesTableContainer.java:744) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:730) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:605) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:70) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:34) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:819) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:924) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:456) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:359) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:290) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:189) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at >
[jira] [Commented] (HIVE-21746) ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled
[ https://issues.apache.org/jira/browse/HIVE-21746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846292#comment-16846292 ] Hive QA commented on HIVE-21746: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 38s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 27s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 47s{color} | {color:green} ql: The patch generated 0 new + 221 unchanged - 1 fixed = 221 total (was 222) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 27m 18s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17283/dev-support/hive-personality.sh | | git revision | master / 9cdb553 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17283/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with > CBO disabled > -- > > Key: HIVE-21746 > URL: https://issues.apache.org/jira/browse/HIVE-21746 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-21746.hive2.patch, HIVE-21746.patch, > HIVE-21746.patch, HIVE-21746.patch > > > ArrayIndexOutOfBounds exception during query execution with dynamically > partitioned hash join. > Found on Hive 2.x. Seems to occur with CBO disabled/failed. > Disabling constant propagation seems to allow the query to succeed. > {noformat} > java.lang.ArrayIndexOutOfBoundsException: 203 > at > org.apache.hadoop.hive.serde2.io.TimestampWritable.getTotalLength(TimestampWritable.java:217) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:205) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getFieldsAsList(LazyBinaryStruct.java:281) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at >
[jira] [Commented] (HIVE-21746) ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled
[ https://issues.apache.org/jira/browse/HIVE-21746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846257#comment-16846257 ] Hive QA commented on HIVE-21746: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12969423/HIVE-21746.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16057 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.ql.io.TestHiveBinarySearchRecordReader.testLessThanOrEqualOpClass (batchId=311) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17282/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17282/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17282/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12969423 - PreCommit-HIVE-Build > ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with > CBO disabled > -- > > Key: HIVE-21746 > URL: https://issues.apache.org/jira/browse/HIVE-21746 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-21746.hive2.patch, HIVE-21746.patch, > HIVE-21746.patch > > > ArrayIndexOutOfBounds exception during query execution with dynamically > partitioned hash join. > Found on Hive 2.x. Seems to occur with CBO disabled/failed. > Disabling constant propagation seems to allow the query to succeed. > {noformat} > java.lang.ArrayIndexOutOfBoundsException: 203 > at > org.apache.hadoop.hive.serde2.io.TimestampWritable.getTotalLength(TimestampWritable.java:217) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:205) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getFieldsAsList(LazyBinaryStruct.java:281) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.unpack(MapJoinBytesTableContainer.java:744) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:730) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:605) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:70) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:34) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:819) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:924) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:456) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:359) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:290) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:189) >
[jira] [Commented] (HIVE-21746) ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled
[ https://issues.apache.org/jira/browse/HIVE-21746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846234#comment-16846234 ] Hive QA commented on HIVE-21746: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 18s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 48s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 32s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 48s{color} | {color:green} ql: The patch generated 0 new + 221 unchanged - 1 fixed = 221 total (was 222) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 28m 19s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17282/dev-support/hive-personality.sh | | git revision | master / 9cdb553 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17282/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with > CBO disabled > -- > > Key: HIVE-21746 > URL: https://issues.apache.org/jira/browse/HIVE-21746 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-21746.hive2.patch, HIVE-21746.patch, > HIVE-21746.patch > > > ArrayIndexOutOfBounds exception during query execution with dynamically > partitioned hash join. > Found on Hive 2.x. Seems to occur with CBO disabled/failed. > Disabling constant propagation seems to allow the query to succeed. > {noformat} > java.lang.ArrayIndexOutOfBoundsException: 203 > at > org.apache.hadoop.hive.serde2.io.TimestampWritable.getTotalLength(TimestampWritable.java:217) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:205) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getFieldsAsList(LazyBinaryStruct.java:281) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at >
[jira] [Commented] (HIVE-21746) ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled
[ https://issues.apache.org/jira/browse/HIVE-21746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846166#comment-16846166 ] Hive QA commented on HIVE-21746: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12969416/HIVE-21746.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 16057 tests executed *Failed tests:* {noformat} org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testCancelRenewTokenFlow (batchId=287) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testConnection (batchId=287) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValid (batchId=287) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValidNeg (batchId=287) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeProxyAuth (batchId=287) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeTokenAuth (batchId=287) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testProxyAuth (batchId=287) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testRenewDelegationToken (batchId=287) org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=287) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17280/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17280/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17280/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12969416 - PreCommit-HIVE-Build > ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with > CBO disabled > -- > > Key: HIVE-21746 > URL: https://issues.apache.org/jira/browse/HIVE-21746 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-21746.hive2.patch, HIVE-21746.patch > > > ArrayIndexOutOfBounds exception during query execution with dynamically > partitioned hash join. > Found on Hive 2.x. Seems to occur with CBO disabled/failed. > Disabling constant propagation seems to allow the query to succeed. > {noformat} > java.lang.ArrayIndexOutOfBoundsException: 203 > at > org.apache.hadoop.hive.serde2.io.TimestampWritable.getTotalLength(TimestampWritable.java:217) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:205) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getFieldsAsList(LazyBinaryStruct.java:281) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.unpack(MapJoinBytesTableContainer.java:744) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:730) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:605) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:70) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:34) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:819) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:924) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:456) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at >
[jira] [Commented] (HIVE-21746) ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled
[ https://issues.apache.org/jira/browse/HIVE-21746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846121#comment-16846121 ] Hive QA commented on HIVE-21746: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 53s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 47s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 29s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s{color} | {color:green} ql: The patch generated 0 new + 221 unchanged - 1 fixed = 221 total (was 222) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 27m 48s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17280/dev-support/hive-personality.sh | | git revision | master / 97d0f8c | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17280/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with > CBO disabled > -- > > Key: HIVE-21746 > URL: https://issues.apache.org/jira/browse/HIVE-21746 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-21746.hive2.patch, HIVE-21746.patch > > > ArrayIndexOutOfBounds exception during query execution with dynamically > partitioned hash join. > Found on Hive 2.x. Seems to occur with CBO disabled/failed. > Disabling constant propagation seems to allow the query to succeed. > {noformat} > java.lang.ArrayIndexOutOfBoundsException: 203 > at > org.apache.hadoop.hive.serde2.io.TimestampWritable.getTotalLength(TimestampWritable.java:217) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:205) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getFieldsAsList(LazyBinaryStruct.java:281) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at >
[jira] [Commented] (HIVE-21746) ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled
[ https://issues.apache.org/jira/browse/HIVE-21746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846055#comment-16846055 ] Jesus Camacho Rodriguez commented on HIVE-21746: [~jdere], I have rebased the patch so it applies to master branch. I think the approach makes sense, let's see if we get any ptest failures. > ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with > CBO disabled > -- > > Key: HIVE-21746 > URL: https://issues.apache.org/jira/browse/HIVE-21746 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-21746.hive2.patch, HIVE-21746.patch > > > ArrayIndexOutOfBounds exception during query execution with dynamically > partitioned hash join. > Found on Hive 2.x. Seems to occur with CBO disabled/failed. > Disabling constant propagation seems to allow the query to succeed. > {noformat} > java.lang.ArrayIndexOutOfBoundsException: 203 > at > org.apache.hadoop.hive.serde2.io.TimestampWritable.getTotalLength(TimestampWritable.java:217) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:205) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getFieldsAsList(LazyBinaryStruct.java:281) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.unpack(MapJoinBytesTableContainer.java:744) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:730) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:605) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:70) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:34) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:819) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:924) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:456) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:359) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:290) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:189) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:377) > ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3] > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) > ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3] > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) > ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3] > at java.security.AccessController.doPrivileged(Native Method) > ~[?:1.8.0_112] > at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112] > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) >
[jira] [Commented] (HIVE-21746) ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled
[ https://issues.apache.org/jira/browse/HIVE-21746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841815#comment-16841815 ] Jason Dere commented on HIVE-21746: --- Initial patch on hive 2.x. This adds foldedFromTab to ExprNodeConstantDesc and makes use of this information during ExprNodeDescUtils.resolveJoinKeysAsRSColumns(). So far I've been unable to create a locally failing qfile test. I'll update with a patch for master branch when I do. > ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with > CBO disabled > -- > > Key: HIVE-21746 > URL: https://issues.apache.org/jira/browse/HIVE-21746 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-21746.hive2.patch > > > ArrayIndexOutOfBounds exception during query execution with dynamically > partitioned hash join. > Found on Hive 2.x. Seems to occur with CBO disabled/failed. > Disabling constant propagation seems to allow the query to succeed. > {noformat} > java.lang.ArrayIndexOutOfBoundsException: 203 > at > org.apache.hadoop.hive.serde2.io.TimestampWritable.getTotalLength(TimestampWritable.java:217) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:205) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getFieldsAsList(LazyBinaryStruct.java:281) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.unpack(MapJoinBytesTableContainer.java:744) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:730) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:605) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:70) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.UnwrapRowContainer.next(UnwrapRowContainer.java:34) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:819) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:924) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:456) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:359) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:290) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:189) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:377) > ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3] > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) > ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3] > at > org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) > ~[tez-runtime-internals-0.8.4.2.6.4.119-3.jar:0.8.4.2.6.4.119-3] > at java.security.AccessController.doPrivileged(Native Method) > ~[?:1.8.0_112] > at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112] > at >
[jira] [Commented] (HIVE-21746) ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled
[ https://issues.apache.org/jira/browse/HIVE-21746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841809#comment-16841809 ] Jason Dere commented on HIVE-21746: --- I believe the dynamically partitioned hash join has issues when the join keys are constant folded. Looking at the ReduceSink output that feeds into the dynamically partitioned hash join: {noformat} Reduce Output Operator key expressions: _col20 (type: string), 'HR3' (type: string) null sort order: aa sort order: ++ Map-reduce partition columns: _col20 (type: string), 'HR3' (type: string) Statistics: Num rows: 380 Data size: 1288485344 Basic stats: COMPLETE Column stats: PARTIAL tag: 0 value expressions: _col2 (type: timestamp), _col3 (type: timestamp), _col51 (type: timestamp), _col124 (type: timestamp) {noformat} So the value expressions in the ReduceSink consists of 4 timestamp columns. And it appears that the data written out and sent to the Join also matches that. However, the input schema to the MapJoin operator shows 5 columns rather than 4: {noformat} *** valCols[0] for JOIN JOIN_13: [Column[VALUE._col2], Column[VALUE._col3], Column[KEY.reducesinkkey1], Column[VALUE._col49], Column[VALUE._col122]] {noformat} With types (timestamp, timestamp, string, timestamp, timestamp) Note that the third column in this list is KEY.reducesinkkey1. Key columns should have been filtered out from the values columns in MapJoinProcessor.getMapJoinDesc(), during the section that populates valueTableDescs. But the keyExprMap generated by ExprNodeDescUtils.resolveJoinKeysAsRSColumns(), which is only done for dynamically partitioned hash join, does not properly match the KEY.reducesinkkey1 column from the ReduceSinkOperator, when filtering the key columns from the value columns. The column reference generated from the constant folded column, in keyExprMap: {noformat} 1 = {ExprNodeColumnDesc@9714} "Column[KEY.reducesinkkey1]" column = "KEY.reducesinkkey1" tabAlias = "" isPartitionColOrVirtualCol = false isSkewedCol = false typeInfo = {PrimitiveTypeInfo@9719} "string" {noformat} What should have been the corresponding key in the ReduceSinkOperator: {noformat} expr = {ExprNodeColumnDesc@8704} "Column[KEY.reducesinkkey1]" column = "KEY.reducesinkkey1" tabAlias = "t2" isPartitionColOrVirtualCol = true isSkewedCol = false typeInfo = {PrimitiveTypeInfo@9719} "string" {noformat} The difference is the ReduceSinkOperator key has tabAlias = "t2". The one generated by ExprNodeDescUtils.resolveJoinKeysAsRSColumns() currently has a tabAlias hardcoded to "". One solution is for ExprNodeConstantDesc to keep a foldedFromTab for the table alias, in addition to foldedFromCol which it already has. That way ExprNodeDescUtils.resolveJoinKeysAsRSColumns() can generate a column reference with the same matching tableAlias as its parent ReduceSinkOperator. > ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with > CBO disabled > -- > > Key: HIVE-21746 > URL: https://issues.apache.org/jira/browse/HIVE-21746 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > ArrayIndexOutOfBounds exception during query execution with dynamically > partitioned hash join. > Found on Hive 2.x. Seems to occur with CBO disabled/failed. > Disabling constant propagation seems to allow the query to succeed. > {noformat} > java.lang.ArrayIndexOutOfBoundsException: 203 > at > org.apache.hadoop.hive.serde2.io.TimestampWritable.getTotalLength(TimestampWritable.java:217) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.checkObjectByteInfo(LazyBinaryUtils.java:205) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.parse(LazyBinaryStruct.java:142) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getFieldsAsList(LazyBinaryStruct.java:281) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.unpack(MapJoinBytesTableContainer.java:744) > ~[hive-exec-2.1.0.2.6.4.119-3.jar:2.1.0.2.6.4.119-3] > at > org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$ReusableRowContainer.next(MapJoinBytesTableContainer.java:730) >