[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555811#comment-16555811 ] Matt McCline commented on HIVE-20207: - Committed to master. [~teddy.choi] thank you for your review! > Vectorization: Fix NULL / Wrong Results issues in Filter / Compare > -- > > Key: HIVE-20207 > URL: https://issues.apache.org/jira/browse/HIVE-20207 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 4.0.0 > > Attachments: HIVE-20207.01.patch, HIVE-20207.02.patch, > HIVE-20207.03.patch, HIVE-20207.04.patch, HIVE-20207.05.patch, > HIVE-20207.06.patch, HIVE-20207.07.patch, HIVE-20207.08.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized filter and compare. > BUGS: > 1) LongColLessLongColumn SIMD optimization do not work for very large > integers: > -7272907770454997143 < 8976171455044006767 > outputVector[i] = (vector1[i] - vector2[i]) >>> 63; > Produces 0 instead of 1... > Also, add DECIMAL_64 testing. Add missing DECIMAL/DECIMAL_64 Comparison and > IF vectorized expression classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1676#comment-1676 ] Hive QA commented on HIVE-20207: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12933009/HIVE-20207.08.patch {color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 14792 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12850/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12850/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12850/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12933009 - PreCommit-HIVE-Build > Vectorization: Fix NULL / Wrong Results issues in Filter / Compare > -- > > Key: HIVE-20207 > URL: https://issues.apache.org/jira/browse/HIVE-20207 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20207.01.patch, HIVE-20207.02.patch, > HIVE-20207.03.patch, HIVE-20207.04.patch, HIVE-20207.05.patch, > HIVE-20207.06.patch, HIVE-20207.07.patch, HIVE-20207.08.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized filter and compare. > BUGS: > 1) LongColLessLongColumn SIMD optimization do not work for very large > integers: > -7272907770454997143 < 8976171455044006767 > outputVector[i] = (vector1[i] - vector2[i]) >>> 63; > Produces 0 instead of 1... > Also, add DECIMAL_64 testing. Add missing DECIMAL/DECIMAL_64 Comparison and > IF vectorized expression classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1657#comment-1657 ] Hive QA commented on HIVE-20207: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 29s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 21s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 23s{color} | {color:blue} storage-api in master has 48 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 59s{color} | {color:red} branch/itests/hive-jmh cannot run convertXmlToText from findbugs {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 4s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 20s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 28s{color} | {color:red} ql in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 19s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 19s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s{color} | {color:red} itests/hive-jmh: The patch generated 1 new + 20 unchanged - 0 fixed = 21 total (was 20) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 50s{color} | {color:red} ql: The patch generated 86 new + 1036 unchanged - 85 fixed = 1122 total (was 1121) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} vector-code-gen: The patch generated 3 new + 319 unchanged - 0 fixed = 322 total (was 319) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch 2 line(s) with tabs. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 15s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 21s{color} | {color:red} ql generated 17 new + 2280 unchanged - 0 fixed = 2297 total (was 2280) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 33m 23s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Redundant nullcheck of undecoratedTypeName, which is known to be non-null in org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getDecimal64VectorExpressionForUdf(GenericUDF, Class, List, int, VectorExpressionDescriptor$Mode, TypeInfo) Redundant null check at VectorizationContext.java:is known to be non-null in org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getDecimal64VectorExpressionForUdf(GenericUDF, Class, List, int, VectorExpressionDescriptor$Mode, TypeInfo) Redundant null check at VectorizationContext.java:[line 1588] | | | Class org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColEqualDecimalScalar defines non-transient non-serializable instance field value In DecimalColEqualDecimalScalar.java:instance field value In DecimalColEqualDecimalScalar.java | | | Class org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColGreaterDecimalScalar defines
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555221#comment-16555221 ] Hive QA commented on HIVE-20207: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12932951/HIVE-20207.07.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12842/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12842/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12842/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Tests exited with: Exception: Patch URL https://issues.apache.org/jira/secure/attachment/12932951/HIVE-20207.07.patch was found in seen patch url's cache and a test was probably run already on it. Aborting... {noformat} This message is automatically generated. ATTACHMENT ID: 12932951 - PreCommit-HIVE-Build > Vectorization: Fix NULL / Wrong Results issues in Filter / Compare > -- > > Key: HIVE-20207 > URL: https://issues.apache.org/jira/browse/HIVE-20207 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20207.01.patch, HIVE-20207.02.patch, > HIVE-20207.03.patch, HIVE-20207.04.patch, HIVE-20207.05.patch, > HIVE-20207.06.patch, HIVE-20207.07.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized filter and compare. > BUGS: > 1) LongColLessLongColumn SIMD optimization do not work for very large > integers: > -7272907770454997143 < 8976171455044006767 > outputVector[i] = (vector1[i] - vector2[i]) >>> 63; > Produces 0 instead of 1... > Also, add DECIMAL_64 testing. Add missing DECIMAL/DECIMAL_64 Comparison and > IF vectorized expression classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555218#comment-16555218 ] Hive QA commented on HIVE-20207: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12932951/HIVE-20207.07.patch {color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14792 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.ql.exec.tez.TestDynamicPartitionPruner.testSingleSourceMultipleFiltersOrdering1 (batchId=305) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12840/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12840/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12840/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12932951 - PreCommit-HIVE-Build > Vectorization: Fix NULL / Wrong Results issues in Filter / Compare > -- > > Key: HIVE-20207 > URL: https://issues.apache.org/jira/browse/HIVE-20207 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20207.01.patch, HIVE-20207.02.patch, > HIVE-20207.03.patch, HIVE-20207.04.patch, HIVE-20207.05.patch, > HIVE-20207.06.patch, HIVE-20207.07.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized filter and compare. > BUGS: > 1) LongColLessLongColumn SIMD optimization do not work for very large > integers: > -7272907770454997143 < 8976171455044006767 > outputVector[i] = (vector1[i] - vector2[i]) >>> 63; > Produces 0 instead of 1... > Also, add DECIMAL_64 testing. Add missing DECIMAL/DECIMAL_64 Comparison and > IF vectorized expression classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555195#comment-16555195 ] Hive QA commented on HIVE-20207: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 46s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 57s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 51s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 26s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 27s{color} | {color:blue} storage-api in master has 48 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 35s{color} | {color:red} branch/itests/hive-jmh cannot run convertXmlToText from findbugs {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 18s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 19s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 29s{color} | {color:red} ql in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 20s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 20s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s{color} | {color:red} itests/hive-jmh: The patch generated 1 new + 20 unchanged - 0 fixed = 21 total (was 20) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 50s{color} | {color:red} ql: The patch generated 86 new + 1036 unchanged - 85 fixed = 1122 total (was 1121) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} vector-code-gen: The patch generated 3 new + 319 unchanged - 0 fixed = 322 total (was 319) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s{color} | {color:red} The patch 2 line(s) with tabs. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 15s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 22s{color} | {color:red} ql generated 17 new + 2280 unchanged - 0 fixed = 2297 total (was 2280) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 35m 32s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Redundant nullcheck of undecoratedTypeName, which is known to be non-null in org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getDecimal64VectorExpressionForUdf(GenericUDF, Class, List, int, VectorExpressionDescriptor$Mode, TypeInfo) Redundant null check at VectorizationContext.java:is known to be non-null in org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getDecimal64VectorExpressionForUdf(GenericUDF, Class, List, int, VectorExpressionDescriptor$Mode, TypeInfo) Redundant null check at VectorizationContext.java:[line 1588] | | | Class org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColEqualDecimalScalar defines non-transient non-serializable instance field value In DecimalColEqualDecimalScalar.java:instance field value In DecimalColEqualDecimalScalar.java | | | Class org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColGreaterDecimalScalar defines
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554512#comment-16554512 ] Hive QA commented on HIVE-20207: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12932822/HIVE-20207.06.patch {color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14789 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz] (batchId=193) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_joins] (batchId=193) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_masking] (batchId=193) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12825/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12825/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12825/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12932822 - PreCommit-HIVE-Build > Vectorization: Fix NULL / Wrong Results issues in Filter / Compare > -- > > Key: HIVE-20207 > URL: https://issues.apache.org/jira/browse/HIVE-20207 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20207.01.patch, HIVE-20207.02.patch, > HIVE-20207.03.patch, HIVE-20207.04.patch, HIVE-20207.05.patch, > HIVE-20207.06.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized filter and compare. > BUGS: > 1) LongColLessLongColumn SIMD optimization do not work for very large > integers: > -7272907770454997143 < 8976171455044006767 > outputVector[i] = (vector1[i] - vector2[i]) >>> 63; > Produces 0 instead of 1... > Also, add DECIMAL_64 testing. Add missing DECIMAL/DECIMAL_64 Comparison and > IF vectorized expression classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554489#comment-16554489 ] Hive QA commented on HIVE-20207: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 42s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 43s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 20s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 25s{color} | {color:blue} storage-api in master has 48 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 15m 25s{color} | {color:red} branch/itests/hive-jmh cannot run convertXmlToText from findbugs {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 13s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 18s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 27s{color} | {color:red} ql in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 18s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 18s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 9s{color} | {color:red} itests/hive-jmh: The patch generated 1 new + 20 unchanged - 0 fixed = 21 total (was 20) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 46s{color} | {color:red} ql: The patch generated 86 new + 1036 unchanged - 85 fixed = 1122 total (was 1121) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} vector-code-gen: The patch generated 3 new + 319 unchanged - 0 fixed = 322 total (was 319) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 16s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 24s{color} | {color:red} ql generated 17 new + 2280 unchanged - 0 fixed = 2297 total (was 2280) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 45m 27s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Redundant nullcheck of undecoratedTypeName, which is known to be non-null in org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getDecimal64VectorExpressionForUdf(GenericUDF, Class, List, int, VectorExpressionDescriptor$Mode, TypeInfo) Redundant null check at VectorizationContext.java:is known to be non-null in org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getDecimal64VectorExpressionForUdf(GenericUDF, Class, List, int, VectorExpressionDescriptor$Mode, TypeInfo) Redundant null check at VectorizationContext.java:[line 1595] | | | Class org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColEqualDecimalScalar defines non-transient non-serializable instance field value In DecimalColEqualDecimalScalar.java:instance field value In DecimalColEqualDecimalScalar.java | | | Class org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColGreaterDecimalScalar
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553665#comment-16553665 ] Teddy Choi commented on HIVE-20207: --- LGTM +1. Thanks for fixing my mistakes. :D > Vectorization: Fix NULL / Wrong Results issues in Filter / Compare > -- > > Key: HIVE-20207 > URL: https://issues.apache.org/jira/browse/HIVE-20207 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20207.01.patch, HIVE-20207.02.patch, > HIVE-20207.03.patch, HIVE-20207.04.patch, HIVE-20207.05.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized filter and compare. > BUGS: > 1) LongColLessLongColumn SIMD optimization do not work for very large > integers: > -7272907770454997143 < 8976171455044006767 > outputVector[i] = (vector1[i] - vector2[i]) >>> 63; > Produces 0 instead of 1... > Also, add DECIMAL_64 testing. Add missing DECIMAL/DECIMAL_64 Comparison and > IF vectorized expression classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553619#comment-16553619 ] Hive QA commented on HIVE-20207: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12932764/HIVE-20207.05.patch {color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 14788 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_dynamic_partition] (batchId=192) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions] (batchId=192) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1] (batchId=192) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter] (batchId=192) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert] (batchId=192) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12805/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12805/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12805/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12932764 - PreCommit-HIVE-Build > Vectorization: Fix NULL / Wrong Results issues in Filter / Compare > -- > > Key: HIVE-20207 > URL: https://issues.apache.org/jira/browse/HIVE-20207 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20207.01.patch, HIVE-20207.02.patch, > HIVE-20207.03.patch, HIVE-20207.04.patch, HIVE-20207.05.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized filter and compare. > BUGS: > 1) LongColLessLongColumn SIMD optimization do not work for very large > integers: > -7272907770454997143 < 8976171455044006767 > outputVector[i] = (vector1[i] - vector2[i]) >>> 63; > Produces 0 instead of 1... > Also, add DECIMAL_64 testing. Add missing DECIMAL/DECIMAL_64 Comparison and > IF vectorized expression classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553599#comment-16553599 ] Hive QA commented on HIVE-20207: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 37s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 16s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 44s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 21s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 22s{color} | {color:blue} storage-api in master has 48 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 14m 12s{color} | {color:red} branch/itests/hive-jmh cannot run convertXmlToText from findbugs {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 56s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 19s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 27s{color} | {color:red} ql in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 19s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 19s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s{color} | {color:red} itests/hive-jmh: The patch generated 1 new + 20 unchanged - 0 fixed = 21 total (was 20) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 46s{color} | {color:red} ql: The patch generated 86 new + 1036 unchanged - 85 fixed = 1122 total (was 1121) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} vector-code-gen: The patch generated 3 new + 319 unchanged - 0 fixed = 322 total (was 319) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 16s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 15s{color} | {color:red} ql generated 17 new + 2280 unchanged - 0 fixed = 2297 total (was 2280) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 43m 10s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Redundant nullcheck of undecoratedTypeName, which is known to be non-null in org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getDecimal64VectorExpressionForUdf(GenericUDF, Class, List, int, VectorExpressionDescriptor$Mode, TypeInfo) Redundant null check at VectorizationContext.java:is known to be non-null in org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getDecimal64VectorExpressionForUdf(GenericUDF, Class, List, int, VectorExpressionDescriptor$Mode, TypeInfo) Redundant null check at VectorizationContext.java:[line 1595] | | | Class org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColEqualDecimalScalar defines non-transient non-serializable instance field value In DecimalColEqualDecimalScalar.java:instance field value In DecimalColEqualDecimalScalar.java | | | Class org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColGreaterDecimalScalar
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552204#comment-16552204 ] Matt McCline commented on HIVE-20207: - Successful test run. > Vectorization: Fix NULL / Wrong Results issues in Filter / Compare > -- > > Key: HIVE-20207 > URL: https://issues.apache.org/jira/browse/HIVE-20207 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20207.01.patch, HIVE-20207.02.patch, > HIVE-20207.03.patch, HIVE-20207.04.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized filter and compare. > BUGS: > 1) LongColLessLongColumn SIMD optimization do not work for very large > integers: > -7272907770454997143 < 8976171455044006767 > outputVector[i] = (vector1[i] - vector2[i]) >>> 63; > Produces 0 instead of 1... > Also, add DECIMAL_64 testing. Add missing DECIMAL/DECIMAL_64 Comparison and > IF vectorized expression classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552186#comment-16552186 ] Hive QA commented on HIVE-20207: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12932606/HIVE-20207.04.patch {color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 14782 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12783/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12783/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12783/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12932606 - PreCommit-HIVE-Build > Vectorization: Fix NULL / Wrong Results issues in Filter / Compare > -- > > Key: HIVE-20207 > URL: https://issues.apache.org/jira/browse/HIVE-20207 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20207.01.patch, HIVE-20207.02.patch, > HIVE-20207.03.patch, HIVE-20207.04.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized filter and compare. > BUGS: > 1) LongColLessLongColumn SIMD optimization do not work for very large > integers: > -7272907770454997143 < 8976171455044006767 > outputVector[i] = (vector1[i] - vector2[i]) >>> 63; > Produces 0 instead of 1... > Also, add DECIMAL_64 testing. Add missing DECIMAL/DECIMAL_64 Comparison and > IF vectorized expression classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552183#comment-16552183 ] Hive QA commented on HIVE-20207: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 40s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 11s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 18s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 24s{color} | {color:blue} storage-api in master has 48 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 0s{color} | {color:red} branch/itests/hive-jmh cannot run convertXmlToText from findbugs {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 54s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 19s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 27s{color} | {color:red} ql in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 19s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 19s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 9s{color} | {color:red} itests/hive-jmh: The patch generated 1 new + 20 unchanged - 0 fixed = 21 total (was 20) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 50s{color} | {color:red} ql: The patch generated 88 new + 1036 unchanged - 85 fixed = 1124 total (was 1121) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} vector-code-gen: The patch generated 3 new + 319 unchanged - 0 fixed = 322 total (was 319) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 15s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 8s{color} | {color:red} ql generated 17 new + 2280 unchanged - 0 fixed = 2297 total (was 2280) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 32m 38s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Redundant nullcheck of undecoratedTypeName, which is known to be non-null in org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getDecimal64VectorExpressionForUdf(GenericUDF, Class, List, int, VectorExpressionDescriptor$Mode, TypeInfo) Redundant null check at VectorizationContext.java:is known to be non-null in org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getDecimal64VectorExpressionForUdf(GenericUDF, Class, List, int, VectorExpressionDescriptor$Mode, TypeInfo) Redundant null check at VectorizationContext.java:[line 1595] | | | Class org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColEqualDecimalScalar defines non-transient non-serializable instance field value In DecimalColEqualDecimalScalar.java:instance field value In DecimalColEqualDecimalScalar.java | | | Class org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColGreaterDecimalScalar
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552018#comment-16552018 ] Hive QA commented on HIVE-20207: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12932591/HIVE-20207.03.patch {color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 14782 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_casts] (batchId=87) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_timestamp_funcs] (batchId=31) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_timestamp_ints_casts] (batchId=52) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_udf_adaptor_1] (batchId=179) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_casts] (batchId=178) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_timestamp_funcs] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_timestamp_ints_casts] (batchId=169) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_timestamp_funcs] (batchId=122) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12778/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12778/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12778/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12932591 - PreCommit-HIVE-Build > Vectorization: Fix NULL / Wrong Results issues in Filter / Compare > -- > > Key: HIVE-20207 > URL: https://issues.apache.org/jira/browse/HIVE-20207 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20207.01.patch, HIVE-20207.02.patch, > HIVE-20207.03.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized filter and compare. > BUGS: > 1) LongColLessLongColumn SIMD optimization do not work for very large > integers: > -7272907770454997143 < 8976171455044006767 > outputVector[i] = (vector1[i] - vector2[i]) >>> 63; > Produces 0 instead of 1... > Also, add DECIMAL_64 testing. Add missing DECIMAL/DECIMAL_64 Comparison and > IF vectorized expression classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551994#comment-16551994 ] Hive QA commented on HIVE-20207: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 46s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 40s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 18s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 22s{color} | {color:blue} storage-api in master has 48 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 9m 45s{color} | {color:red} branch/itests/hive-jmh cannot run convertXmlToText from findbugs {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 56s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 19s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 26s{color} | {color:red} ql in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 20s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 20s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s{color} | {color:red} itests/hive-jmh: The patch generated 1 new + 20 unchanged - 0 fixed = 21 total (was 20) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 46s{color} | {color:red} ql: The patch generated 88 new + 1036 unchanged - 85 fixed = 1124 total (was 1121) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} vector-code-gen: The patch generated 3 new + 319 unchanged - 0 fixed = 322 total (was 319) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 14s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 8s{color} | {color:red} ql generated 17 new + 2280 unchanged - 0 fixed = 2297 total (was 2280) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 38m 14s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Redundant nullcheck of undecoratedTypeName, which is known to be non-null in org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getDecimal64VectorExpressionForUdf(GenericUDF, Class, List, int, VectorExpressionDescriptor$Mode, TypeInfo) Redundant null check at VectorizationContext.java:is known to be non-null in org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getDecimal64VectorExpressionForUdf(GenericUDF, Class, List, int, VectorExpressionDescriptor$Mode, TypeInfo) Redundant null check at VectorizationContext.java:[line 1595] | | | Class org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColEqualDecimalScalar defines non-transient non-serializable instance field value In DecimalColEqualDecimalScalar.java:instance field value In DecimalColEqualDecimalScalar.java | | | Class org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColGreaterDecimalScalar
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551974#comment-16551974 ] Hive QA commented on HIVE-20207: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12932580/HIVE-20207.02.patch {color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 14782 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[convert_decimal64_to_decimal] (batchId=52) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_case_when_1] (batchId=89) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_mapjoin] (batchId=59) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_case] (batchId=61) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_casts] (batchId=87) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_timestamp_funcs] (batchId=31) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_timestamp_ints_casts] (batchId=52) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[convert_decimal64_to_decimal] (batchId=169) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_case_when_1] (batchId=178) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_mapjoin] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_outer_reference_windowed] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_udf_adaptor_1] (batchId=179) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_case] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_casts] (batchId=178) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_semijoin_reduction2] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_mapjoin3] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_timestamp_funcs] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_timestamp_ints_casts] (batchId=169) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_reducers_power_two] (batchId=186) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_decimal_mapjoin] (batchId=134) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_case] (batchId=134) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_timestamp_funcs] (batchId=122) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12777/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12777/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12777/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 22 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12932580 - PreCommit-HIVE-Build > Vectorization: Fix NULL / Wrong Results issues in Filter / Compare > -- > > Key: HIVE-20207 > URL: https://issues.apache.org/jira/browse/HIVE-20207 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20207.01.patch, HIVE-20207.02.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized filter and compare. > BUGS: > 1) LongColLessLongColumn SIMD optimization do not work for very large > integers: > -7272907770454997143 < 8976171455044006767 > outputVector[i] = (vector1[i] - vector2[i]) >>> 63; > Produces 0 instead of 1... > Also, add DECIMAL_64 testing. Add missing DECIMAL/DECIMAL_64 Comparison and > IF vectorized expression classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551962#comment-16551962 ] Hive QA commented on HIVE-20207: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 36s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 18s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 17s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 26s{color} | {color:blue} storage-api in master has 48 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 41s{color} | {color:red} branch/itests/hive-jmh cannot run convertXmlToText from findbugs {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 48s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 19s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 25s{color} | {color:red} ql in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 17s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 17s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 9s{color} | {color:red} itests/hive-jmh: The patch generated 1 new + 20 unchanged - 0 fixed = 21 total (was 20) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 46s{color} | {color:red} ql: The patch generated 88 new + 1036 unchanged - 85 fixed = 1124 total (was 1121) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s{color} | {color:red} vector-code-gen: The patch generated 3 new + 319 unchanged - 0 fixed = 322 total (was 319) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 14s{color} | {color:red} hive-jmh in the patch failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 4s{color} | {color:red} ql generated 17 new + 2280 unchanged - 0 fixed = 2297 total (was 2280) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 31m 38s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Redundant nullcheck of undecoratedTypeName, which is known to be non-null in org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getDecimal64VectorExpressionForUdf(GenericUDF, Class, List, int, VectorExpressionDescriptor$Mode, TypeInfo) Redundant null check at VectorizationContext.java:is known to be non-null in org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.getDecimal64VectorExpressionForUdf(GenericUDF, Class, List, int, VectorExpressionDescriptor$Mode, TypeInfo) Redundant null check at VectorizationContext.java:[line 1595] | | | Class org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColEqualDecimalScalar defines non-transient non-serializable instance field value In DecimalColEqualDecimalScalar.java:instance field value In DecimalColEqualDecimalScalar.java | | | Class org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DecimalColGreaterDecimalScalar
[jira] [Commented] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551631#comment-16551631 ] Hive QA commented on HIVE-20207: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12932492/HIVE-20207.01.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12761/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12761/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12761/ Messages: {noformat} This message was trimmed, see log for full details [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/fs/FileStatus.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-server/1.19/jersey-server-1.19.jar(com/sun/jersey/api/wadl/config/WadlGeneratorConfig.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-server/1.19/jersey-server-1.19.jar(com/sun/jersey/api/wadl/config/WadlGeneratorDescription.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-server/1.19/jersey-server-1.19.jar(com/sun/jersey/server/wadl/generators/resourcedoc/WadlGeneratorResourceDocSupport.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/BufferedReader.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/InputStream.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/InputStreamReader.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/PrintWriter.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/Map$Entry.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/Semaphore.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/commons/commons-exec/1.1/commons-exec-1.1.jar(org/apache/commons/exec/CommandLine.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/commons/commons-exec/1.1/commons-exec-1.1.jar(org/apache/commons/exec/DefaultExecutor.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/commons/commons-exec/1.1/commons-exec-1.1.jar(org/apache/commons/exec/ExecuteWatchdog.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/commons/commons-exec/1.1/commons-exec-1.1.jar(org/apache/commons/exec/PumpStreamHandler.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/util/Shell.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Thread.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Runnable.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/ws/rs/javax.ws.rs-api/2.0.1/javax.ws.rs-api-2.0.1.jar(javax/ws/rs/ext/ExceptionMapper.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/ws/rs/javax.ws.rs-api/2.0.1/javax.ws.rs-api-2.0.1.jar(javax/ws/rs/ext/Provider.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-server/1.19/jersey-server-1.19.jar(com/sun/jersey/api/NotFoundException.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapred/JobID.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/security/Groups.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/HashSet.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/Set.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/ConcurrentHashMap.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/common/target/hive-common-4.0.0-SNAPSHOT.jar(org/apache/hive/common/util/HiveVersionInfo.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/classification/target/hive-classification-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceStability$Evolving.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/DataInput.class)]] [loading