[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365817#comment-16365817
]
Vihang Karajgaonkar commented on HIVE-18421:
Patch merged in the master branch. Thanks for the review [~stakiar]
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch, HIVE-18421.08.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364647#comment-16364647
]
Vihang Karajgaonkar commented on HIVE-18421:
I would like to commit this patch today. I have a +1 from [~stakiar] and I have
added a test case in benchmark suite as suggested by [~gopalv]. Since
[~mmccline] is busy with HIVE-18622, I will create a separate JIRA if he raises
any serious concerns with this patch in the future. The config defaults to
false as suggested by him above.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch, HIVE-18421.08.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362777#comment-16362777
]
Hive QA commented on HIVE-18421:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12910346/HIVE-18421.08.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 13853 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=241)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_filter]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_opt_shuffle_serde]
(batchId=180)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query1]
(batchId=251)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=222)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=235)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=235)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9193/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9193/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9193/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12910346 - PreCommit-HIVE-Build
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch, HIVE-18421.08.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362701#comment-16362701
]
Hive QA commented on HIVE-18421:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
16s{color} | {color:red} hive-jmh in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
27s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
14s{color} | {color:red} hive-jmh in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 14s{color}
| {color:red} hive-jmh in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} itests/hive-jmh: The patch generated 2 new + 9
unchanged - 0 fixed = 11 total (was 9) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 9 new + 505 unchanged - 0
fixed = 514 total (was 505) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} vector-code-gen: The patch generated 29 new + 345
unchanged - 3 fixed = 374 total (was 348) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red} itests_hive-jmh generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 9a02aa8 |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9193/yetus/patch-mvninstall-itests_hive-jmh.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9193/yetus/patch-mvninstall-ql.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9193/yetus/patch-compile-itests_hive-jmh.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9193/yetus/patch-compile-itests_hive-jmh.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9193/yetus/diff-checkstyle-itests_hive-jmh.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9193/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9193/yetus/diff-checkstyle-vector-code-gen.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9193/yetus/diff-javadoc-javadoc-itests_hive-jmh.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9193/yetus/patch-asflicense-problems.txt
|
| modules | C: common itests/hive-jmh ql vector-code-gen U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9193/yetus.txt |
| Powered by | Apache Yetus
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361929#comment-16361929
]
Vihang Karajgaonkar commented on HIVE-18421:
Hi [~gopalv] I added a couple of tests to the VectorizedArithmeticBench in the
latest patch. Can you please take a look?
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch, HIVE-18421.08.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358701#comment-16358701
]
Aihua Xu commented on HIVE-18421:
-
Originally I thought non-vectorization version was handling the overflow but
since both are returning the incorrect results, I'm fine with current patch. I
will leave to [~mmccline] and [~gopalv] for a second review since you are more
familiar on this part.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357849#comment-16357849
]
Hive QA commented on HIVE-18421:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909678/HIVE-18421.07.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 13674 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_bmj_schema_evolution]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=180)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_opt_shuffle_serde]
(batchId=180)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query1]
(batchId=250)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9105/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9105/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9105/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909678 - PreCommit-HIVE-Build
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357816#comment-16357816
]
Hive QA commented on HIVE-18421:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} vector-code-gen: The patch generated 29 new + 345
unchanged - 3 fixed = 374 total (was 348) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 9 new + 505 unchanged - 0
fixed = 514 total (was 505) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 3464df4 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9105/yetus/diff-checkstyle-vector-code-gen.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9105/yetus/diff-checkstyle-ql.txt
|
| modules | C: common vector-code-gen ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9105/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
>
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357722#comment-16357722
]
Gopal V commented on HIVE-18421:
The performance question was primarily about inserting the check in the inner
loop.
What you have done with the Util doing a second traversal is likely to be an
overhead only where the checked impl is needed for correctness.
If you add a small test to VectorizedBench in hive-jmh, I can go over the
assembly and make sure we're JIT'ing the bigint operators exactly the same way
and also check the L1 cache rate (the 2nd iteration in the overflow util should
get a 100% hit-rate, hopefully).
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357718#comment-16357718
]
Vihang Karajgaonkar commented on HIVE-18421:
I can enable this config by default but that would trigger an q.out update of
over 300 q files. Also, there were concerns raised by [~mmccline] and [~gopalv]
above regarding performance. I didn't investigate the performance overhead of
the fix. The issue is there is no well-defined policy within Hive overall on
how to handle overflows. So it is arguable if users would want to enable this
config by default or not since Hive itself doesn't handle overflows well
overall. The config does handle overflows in the "right" way. It only makes
vectorized execution overflow handling similar to non-vectorized handling. That
was the reason I disabled the config by default. I can investigate the overhead
and turn it on by default if there is no significant overhead. Any thoughts
[~gopalv] [~aihuaxu] on this?
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357392#comment-16357392
]
Aihua Xu commented on HIVE-18421:
-
Talked to [~vihangk1] offline and got better understanding. The
non-vectorization version actually is also not handling the overflow properly
and that is a larger problem to address overall.
[~vihangk1] Do you think we should enable this property to true by default or
just have the CHECKED version?
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356455#comment-16356455
]
Aihua Xu commented on HIVE-18421:
-
[~vihangk1] Sorry for the late reply. I left comment in RB. Basically I don't
follow why we need both CHECKED and UNCHECKED implementations. Seems we should
only have CHECKED one if UNCHECKED one would generate incorrect result. The
user would get incorrect result without notice, right?
Of course, even we want to support UNCHECKED implementation, we should error
out/fail the query if there is overflow so the user knows to set the flag to
true. BTW: how much performance impact for this and why (don't exactly follow
previous discussion)?
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356213#comment-16356213
]
Vihang Karajgaonkar commented on HIVE-18421:
[~mmccline] would you be able to take a look at this? This patch introduces new
checked vector expressions and uses them when we set the newly introduced
config {{hive.vectorized.use.checked.expressions}}. I introduced checked
expressions for the arithmetic operators and some others where I could see the
issue could cause different results after overflow based on my analysis.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356202#comment-16356202
]
Vihang Karajgaonkar commented on HIVE-18421:
Okay, I finally figured out the problem with vectorization_numeric_overflows.q.
Turns out on CentOS the ordering of rows is different for a query result when
compared to Ubuntu/MacOS. Both the results are valid since the key on which the
order by clause was issued have same value on 2 rows. So the order of these 2
rows with the same value of key is non-deterministic. Fixed the query so that
it orders by a different column which is always deterministic.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356118#comment-16356118
]
Vihang Karajgaonkar commented on HIVE-18421:
vectorization_numeric_overflows.q failure is very weird. I tested it locally
(macOS) and also on a ubuntu VM (Linux with OpenJDK). I also tested it with the
exact same batch as being run by the precommit job. It always works for me
without any changes.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355686#comment-16355686
]
Hive QA commented on HIVE-18421:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909579/HIVE-18421.06.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 13657 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_numeric_overflows]
(batchId=69)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=180)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9071/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9071/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9071/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909579 - PreCommit-HIVE-Build
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355690#comment-16355690
]
Sahil Takiar commented on HIVE-18421:
-
+1 LGTM
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355622#comment-16355622
]
Hive QA commented on HIVE-18421:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} vector-code-gen: The patch generated 29 new + 345
unchanged - 3 fixed = 374 total (was 348) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 9 new + 505 unchanged - 0
fixed = 514 total (was 505) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / acc62e3 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9071/yetus/diff-checkstyle-vector-code-gen.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9071/yetus/diff-checkstyle-ql.txt
|
| modules | C: common vector-code-gen ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9071/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354516#comment-16354516
]
Vihang Karajgaonkar commented on HIVE-18421:
latest patch fixes TestCliDriver.testCliDriver[vectorization_numeric_overflows]
and some Yetus checkstyle changes.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354168#comment-16354168
]
Hive QA commented on HIVE-18421:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909388/HIVE-18421.04.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 13653 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_numeric_overflows]
(batchId=69)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=180)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9051/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9051/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9051/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909388 - PreCommit-HIVE-Build
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354092#comment-16354092
]
Hive QA commented on HIVE-18421:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} vector-code-gen: The patch generated 29 new + 345
unchanged - 3 fixed = 374 total (was 348) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 8 new + 505 unchanged - 0
fixed = 513 total (was 505) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 443b10b |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9051/yetus/diff-checkstyle-vector-code-gen.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9051/yetus/diff-checkstyle-ql.txt
|
| modules | C: common vector-code-gen ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9051/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353538#comment-16353538
]
Vihang Karajgaonkar commented on HIVE-18421:
Fixed TestVectorMathFunctions
TestCliDriver.testCliDriver[vectorization_numeric_overflows] is working for my
on my laptop. I am not sure why it fails on the precommit job. Will investigate
further. While I fix that it would be great if you can please take a look
[~mmccline] [~aihuaxu].
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353327#comment-16353327
]
Hive QA commented on HIVE-18421:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909277/HIVE-18421.03.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 13648 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avrocountemptytbl]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_numeric_overflows]
(batchId=69)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorPosMod
(batchId=284)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9033/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9033/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9033/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909277 - PreCommit-HIVE-Build
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353296#comment-16353296
]
Hive QA commented on HIVE-18421:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} vector-code-gen: The patch generated 29 new + 345
unchanged - 3 fixed = 374 total (was 348) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 8 new + 501 unchanged - 0
fixed = 509 total (was 501) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 1bcc88f |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9033/yetus/diff-checkstyle-vector-code-gen.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9033/yetus/diff-checkstyle-ql.txt
|
| modules | C: common vector-code-gen ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9033/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352918#comment-16352918
]
Vihang Karajgaonkar commented on HIVE-18421:
latest patch should fix most of the tests. Also, made changes to keep yetus
happy.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352725#comment-16352725
]
Vihang Karajgaonkar commented on HIVE-18421:
Copy pasting the test result from precommit job since its not posting it to the
Jira for some reason.
[Test
Result|https://builds.apache.org/job/PreCommit-HIVE-Build/9014/testReport/]
(312 failures / +285)
Most of these test failures are due to explain plan differences since the
{{hive.vectorized.use.checked.expressions}} is set to true by default in the
patch. We can either update all these q.out files or change the default value
of this config to false.
The tests TestColumnScalarOperationVectorExpressionEvaluation are failing
because they do not set the OutputTypeInfo in the test case. This will also get
resolved if we set the outputTypeInfo appropriately in the test template. I
think I will change the patch so that
{{OverflowUtils.accountForOverflow[Double|Long]}} will not throw an exception
if the outputTypeInfo is set to null since based on the tests, it doesn't look
like its always going to be true for all the expression. [~mmccline] do you
know if this understanding is correct?
I will update the patch which should fix most of these test failures.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352019#comment-16352019
]
Hive QA commented on HIVE-18421:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} vector-code-gen: The patch generated 53 new + 306
unchanged - 2 fixed = 359 total (was 308) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 17 new + 501 unchanged - 0
fixed = 518 total (was 501) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0a328f0 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9014/yetus/diff-checkstyle-vector-code-gen.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9014/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9014/yetus/patch-asflicense-problems.txt
|
| modules | C: common vector-code-gen ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9014/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2,
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351345#comment-16351345
]
Hive QA commented on HIVE-18421:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} vector-code-gen: The patch generated 53 new + 306
unchanged - 2 fixed = 359 total (was 308) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 17 new + 501 unchanged - 0
fixed = 518 total (was 501) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 4a33ec8 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8999/yetus/diff-checkstyle-vector-code-gen.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8999/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8999/yetus/patch-asflicense-problems.txt
|
| modules | C: common vector-code-gen ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8999/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351216#comment-16351216
]
Vihang Karajgaonkar commented on HIVE-18421:
adding reviewboard link
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351212#comment-16351212
]
Vihang Karajgaonkar commented on HIVE-18421:
Updated the Jira description appropriately. This patch doesn't add code to
handle overflows in a "right" manner. It only makes vectorized code execution
handle overflow in a "consistent" manner when compared to non-vectorized
execution.
I feel that handling overflows according to a configurable policy (wraparound,
warn, error, set to null) would be a good addition and it can be taken up as a
separate effort.
Attaching the first version of this patch. I reviewed almost all the vector
expressions (phew ..;)) and found a subset of expressions which do not handle
overflows in a consistent manner. As discussed above, it introduces a new
config which enables usage of checked expressions when available.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
