[jira] [Commented] (HIVE-22074) Slow compilation due to IN to OR transformation
[
https://issues.apache.org/jira/browse/HIVE-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910892#comment-16910892
]
Hive QA commented on HIVE-22074:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12978008/HIVE-22074.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16743 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=360)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18368/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18368/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18368/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12978008 - PreCommit-HIVE-Build
> Slow compilation due to IN to OR transformation
> ---
>
> Key: HIVE-22074
> URL: https://issues.apache.org/jira/browse/HIVE-22074
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22074.1.patch, HIVE-22074.2.patch,
> HIVE-22074.3.patch, HIVE-22074.4.patch, HIVE-22074.5.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Currently Hive transform IN expressions to OR to apply various CBO rules.
> This incur significant performance hit if IN consist of large number of
> expressions.
> It is better to not transform IN expressions to OR in such cases because
> overall benefit of various optimizations/transformations is unrealized due to
> the compilation overhead
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22074) Slow compilation due to IN to OR transformation
[
https://issues.apache.org/jira/browse/HIVE-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910869#comment-16910869
]
Hive QA commented on HIVE-22074:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} common in master has 61 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 2251 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 2 new + 259 unchanged - 2
fixed = 261 total (was 261) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18368/dev-support/hive-personality.sh
|
| git revision | master / 99de134 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18368/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18368/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Slow compilation due to IN to OR transformation
> ---
>
> Key: HIVE-22074
> URL: https://issues.apache.org/jira/browse/HIVE-22074
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22074.1.patch, HIVE-22074.2.patch,
> HIVE-22074.3.patch, HIVE-22074.4.patch, HIVE-22074.5.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Currently Hive transform IN expressions to OR to apply various CBO rules.
> This incur significant performance hit if IN consist of large number of
> expressions.
> It is better to not transform IN expressions to OR in such cases because
> overall benefit of various optimizations/transformations is unrealized due to
> the compilation overhead
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22107) Correlated subquery producing wrong schema
[
https://issues.apache.org/jira/browse/HIVE-22107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910855#comment-16910855
]
Hive QA commented on HIVE-22107:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12978001/HIVE-22107.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16743 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18367/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18367/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18367/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12978001 - PreCommit-HIVE-Build
> Correlated subquery producing wrong schema
> --
>
> Key: HIVE-22107
> URL: https://issues.apache.org/jira/browse/HIVE-22107
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22107.1.patch, HIVE-22107.2.patch,
> HIVE-22107.3.patch, HIVE-22107.4.patch, HIVE-22107.5.patch
>
>
> *Repro*
> {code:sql}
> create table test(id int, name string,dept string);
> insert into test values(1,'a','it'),(2,'b','eee'),(NULL, 'c', 'cse');
> select distinct 'empno' as eid, a.id from test a where NOT EXISTS (select
> c.id from test c where a.id=c.id);
> {code}
> {code}
> +---++
> | eid | a.id |
> +---++
> | NULL | empno |
> +---++
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22107) Correlated subquery producing wrong schema
[
https://issues.apache.org/jira/browse/HIVE-22107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910822#comment-16910822
]
Hive QA commented on HIVE-22107:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2251 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 2 new + 21 unchanged - 0 fixed
= 23 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18367/dev-support/hive-personality.sh
|
| git revision | master / 99de134 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18367/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18367/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Correlated subquery producing wrong schema
> --
>
> Key: HIVE-22107
> URL: https://issues.apache.org/jira/browse/HIVE-22107
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22107.1.patch, HIVE-22107.2.patch,
> HIVE-22107.3.patch, HIVE-22107.4.patch, HIVE-22107.5.patch
>
>
> *Repro*
> {code:sql}
> create table test(id int, name string,dept string);
> insert into test values(1,'a','it'),(2,'b','eee'),(NULL, 'c', 'cse');
> select distinct 'empno' as eid, a.id from test a where NOT EXISTS (select
> c.id from test c where a.id=c.id);
> {code}
> {code}
> +---++
> | eid | a.id |
> +---++
> | NULL | empno |
> +---++
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22074) Slow compilation due to IN to OR transformation
[ https://issues.apache.org/jira/browse/HIVE-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22074: --- Status: Patch Available (was: Open) > Slow compilation due to IN to OR transformation > --- > > Key: HIVE-22074 > URL: https://issues.apache.org/jira/browse/HIVE-22074 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22074.1.patch, HIVE-22074.2.patch, > HIVE-22074.3.patch, HIVE-22074.4.patch, HIVE-22074.5.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently Hive transform IN expressions to OR to apply various CBO rules. > This incur significant performance hit if IN consist of large number of > expressions. > It is better to not transform IN expressions to OR in such cases because > overall benefit of various optimizations/transformations is unrealized due to > the compilation overhead -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22074) Slow compilation due to IN to OR transformation
[ https://issues.apache.org/jira/browse/HIVE-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22074: --- Status: Open (was: Patch Available) > Slow compilation due to IN to OR transformation > --- > > Key: HIVE-22074 > URL: https://issues.apache.org/jira/browse/HIVE-22074 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22074.1.patch, HIVE-22074.2.patch, > HIVE-22074.3.patch, HIVE-22074.4.patch, HIVE-22074.5.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently Hive transform IN expressions to OR to apply various CBO rules. > This incur significant performance hit if IN consist of large number of > expressions. > It is better to not transform IN expressions to OR in such cases because > overall benefit of various optimizations/transformations is unrealized due to > the compilation overhead -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (HIVE-22074) Slow compilation due to IN to OR transformation
[
https://issues.apache.org/jira/browse/HIVE-22074?focusedWorklogId=297464=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-297464
]
ASF GitHub Bot logged work on HIVE-22074:
-
Author: ASF GitHub Bot
Created on: 19/Aug/19 22:09
Start Date: 19/Aug/19 22:09
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #746:
HIVE-22074: Slow compilation due to IN to OR transformation
URL: https://github.com/apache/hive/pull/746#discussion_r315433550
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
##
@@ -1220,16 +1220,26 @@ protected ExprNodeDesc
getXpathOrFuncExprNodeDesc(ASTNode expr,
}
outputOpList.add(nullConst);
}
+
if (!ctx.isCBOExecuted()) {
-ArrayList orOperands =
TypeCheckProcFactoryUtils.rewriteInToOR(children);
-if (orOperands != null) {
- if (orOperands.size() == 1) {
-orOperands.add(new
ExprNodeConstantDesc(TypeInfoFactory.booleanTypeInfo, false));
+
+HiveConf conf;
+try {
+ conf = Hive.get().getConf();
Review comment:
Ok. I'll create a separate jira and tackle this in a follow-up.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 297464)
Time Spent: 1h 10m (was: 1h)
> Slow compilation due to IN to OR transformation
> ---
>
> Key: HIVE-22074
> URL: https://issues.apache.org/jira/browse/HIVE-22074
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22074.1.patch, HIVE-22074.2.patch,
> HIVE-22074.3.patch, HIVE-22074.4.patch, HIVE-22074.5.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Currently Hive transform IN expressions to OR to apply various CBO rules.
> This incur significant performance hit if IN consist of large number of
> expressions.
> It is better to not transform IN expressions to OR in such cases because
> overall benefit of various optimizations/transformations is unrealized due to
> the compilation overhead
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22074) Slow compilation due to IN to OR transformation
[ https://issues.apache.org/jira/browse/HIVE-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22074: --- Attachment: HIVE-22074.5.patch > Slow compilation due to IN to OR transformation > --- > > Key: HIVE-22074 > URL: https://issues.apache.org/jira/browse/HIVE-22074 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22074.1.patch, HIVE-22074.2.patch, > HIVE-22074.3.patch, HIVE-22074.4.patch, HIVE-22074.5.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently Hive transform IN expressions to OR to apply various CBO rules. > This incur significant performance hit if IN consist of large number of > expressions. > It is better to not transform IN expressions to OR in such cases because > overall benefit of various optimizations/transformations is unrealized due to > the compilation overhead -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-20057) For ALTER TABLE t SET TBLPROPERTIES ('EXTERNAL'='TRUE'); `TBL_TYPE` attribute change not reflecting for non-CAPS
[
https://issues.apache.org/jira/browse/HIVE-20057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-20057:
--
Fix Version/s: (was: 3.1.0)
3.1.2
> For ALTER TABLE t SET TBLPROPERTIES ('EXTERNAL'='TRUE'); `TBL_TYPE` attribute
> change not reflecting for non-CAPS
>
>
> Key: HIVE-20057
> URL: https://issues.apache.org/jira/browse/HIVE-20057
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: All Versions
>Reporter: Anirudh
>Assignee: Anirudh
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.1.2
>
> Attachments: hive20057.patch
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Hive EXTERNAL table shown as MANAGED after conversion using
> {code} ALTER TABLE t SET TBLPROPERTIES ('EXTERNAL'='True')
> {code}
>
> The DESCRIBE FORMATTED shows:
> {code}
> Table Type: MANAGED_TABLE
> Table Parameters:
> EXTERNAL True
> {code}
>
> This is actually a External table but is shown wrongly, as 'True' was used in
> place of 'TRUE' in the ALTER statement.
> Issue explained here:
> [StakOverflow - Hive Table is MANAGED or
> EXTERNAL|https://stackoverflow.com/questions/51103317/hive-table-is-managed-or-external/51142873#51142873]
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-22126: -- > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (HIVE-22105) Update ORC to 1.5.6.
[ https://issues.apache.org/jira/browse/HIVE-22105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates resolved HIVE-22105. --- Fix Version/s: 3.1.2 Resolution: Fixed Tests pass when run locally. Committed to branch-3.1 > Update ORC to 1.5.6. > > > Key: HIVE-22105 > URL: https://issues.apache.org/jira/browse/HIVE-22105 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Major > Labels: pull-request-available > Fix For: 3.1.2 > > Time Spent: 10m > Remaining Estimate: 0h > > ORC has had some important fixes in the 1.5 branch and they should be picked > up by Hive. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22107) Correlated subquery producing wrong schema
[
https://issues.apache.org/jira/browse/HIVE-22107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22107:
---
Status: Open (was: Patch Available)
> Correlated subquery producing wrong schema
> --
>
> Key: HIVE-22107
> URL: https://issues.apache.org/jira/browse/HIVE-22107
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22107.1.patch, HIVE-22107.2.patch,
> HIVE-22107.3.patch, HIVE-22107.4.patch, HIVE-22107.5.patch
>
>
> *Repro*
> {code:sql}
> create table test(id int, name string,dept string);
> insert into test values(1,'a','it'),(2,'b','eee'),(NULL, 'c', 'cse');
> select distinct 'empno' as eid, a.id from test a where NOT EXISTS (select
> c.id from test c where a.id=c.id);
> {code}
> {code}
> +---++
> | eid | a.id |
> +---++
> | NULL | empno |
> +---++
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22107) Correlated subquery producing wrong schema
[
https://issues.apache.org/jira/browse/HIVE-22107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22107:
---
Status: Patch Available (was: Open)
> Correlated subquery producing wrong schema
> --
>
> Key: HIVE-22107
> URL: https://issues.apache.org/jira/browse/HIVE-22107
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22107.1.patch, HIVE-22107.2.patch,
> HIVE-22107.3.patch, HIVE-22107.4.patch, HIVE-22107.5.patch
>
>
> *Repro*
> {code:sql}
> create table test(id int, name string,dept string);
> insert into test values(1,'a','it'),(2,'b','eee'),(NULL, 'c', 'cse');
> select distinct 'empno' as eid, a.id from test a where NOT EXISTS (select
> c.id from test c where a.id=c.id);
> {code}
> {code}
> +---++
> | eid | a.id |
> +---++
> | NULL | empno |
> +---++
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22107) Correlated subquery producing wrong schema
[
https://issues.apache.org/jira/browse/HIVE-22107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22107:
---
Attachment: HIVE-22107.5.patch
> Correlated subquery producing wrong schema
> --
>
> Key: HIVE-22107
> URL: https://issues.apache.org/jira/browse/HIVE-22107
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22107.1.patch, HIVE-22107.2.patch,
> HIVE-22107.3.patch, HIVE-22107.4.patch, HIVE-22107.5.patch
>
>
> *Repro*
> {code:sql}
> create table test(id int, name string,dept string);
> insert into test values(1,'a','it'),(2,'b','eee'),(NULL, 'c', 'cse');
> select distinct 'empno' as eid, a.id from test a where NOT EXISTS (select
> c.id from test c where a.id=c.id);
> {code}
> {code}
> +---++
> | eid | a.id |
> +---++
> | NULL | empno |
> +---++
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22120) Fix wrong results/ArrayOutOfBound exception in left outer map joins on specific boundary conditions
[ https://issues.apache.org/jira/browse/HIVE-22120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-22120: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master > Fix wrong results/ArrayOutOfBound exception in left outer map joins on > specific boundary conditions > --- > > Key: HIVE-22120 > URL: https://issues.apache.org/jira/browse/HIVE-22120 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22120.1.patch, HIVE-22120.2.patch, > HIVE-22120.3.patch > > > Vectorized version of left outer map join produces wrong results or > encounters ArrayOutOfBound exception. > The boundary conditions are: > * The complete batch of the big table should have the join key repeated for > all the join columns. > * The complete batch of the big table should have not have a matched key > value in the small table > * The repeated value should not be a null value > * Some rows should be filtered out as part of the on clause filter. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22120) Fix wrong results/ArrayOutOfBound exception in left outer map joins on specific boundary conditions
[ https://issues.apache.org/jira/browse/HIVE-22120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910690#comment-16910690 ] Jason Dere commented on HIVE-22120: --- +1 > Fix wrong results/ArrayOutOfBound exception in left outer map joins on > specific boundary conditions > --- > > Key: HIVE-22120 > URL: https://issues.apache.org/jira/browse/HIVE-22120 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22120.1.patch, HIVE-22120.2.patch, > HIVE-22120.3.patch > > > Vectorized version of left outer map join produces wrong results or > encounters ArrayOutOfBound exception. > The boundary conditions are: > * The complete batch of the big table should have the join key repeated for > all the join columns. > * The complete batch of the big table should have not have a matched key > value in the small table > * The repeated value should not be a null value > * Some rows should be filtered out as part of the on clause filter. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22124) Fix wrong results/ArrayOutOfBound exception in left outer map joins on specific boundary conditions with limit
[ https://issues.apache.org/jira/browse/HIVE-22124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910659#comment-16910659 ] Ramesh Kumar Thangarajan commented on HIVE-22124: - https://issues.apache.org/jira/browse/HIVE-22120 will take care of fixing this issue. Since the two issues cannot be separated into a two different patches as they rely on each other. > Fix wrong results/ArrayOutOfBound exception in left outer map joins on > specific boundary conditions with limit > -- > > Key: HIVE-22124 > URL: https://issues.apache.org/jira/browse/HIVE-22124 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > > This is the extension of the bug HIVE-22120. All the boundary conditions > mentioned in HIVE-22120 also apply here plus the query needs to have a limit. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22120) Fix wrong results/ArrayOutOfBound exception in left outer map joins on specific boundary conditions
[ https://issues.apache.org/jira/browse/HIVE-22120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910658#comment-16910658 ] Ramesh Kumar Thangarajan commented on HIVE-22120: - Hi [~jdere] I have addressed your offline comments about the Limit operator resizing the selected array and modified the test files to read data from a csv file. Can you please review my pull request [https://github.com/apache/hive/pull/753] and let me know your thoughts? Thanks, Ramesh > Fix wrong results/ArrayOutOfBound exception in left outer map joins on > specific boundary conditions > --- > > Key: HIVE-22120 > URL: https://issues.apache.org/jira/browse/HIVE-22120 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22120.1.patch, HIVE-22120.2.patch, > HIVE-22120.3.patch > > > Vectorized version of left outer map join produces wrong results or > encounters ArrayOutOfBound exception. > The boundary conditions are: > * The complete batch of the big table should have the join key repeated for > all the join columns. > * The complete batch of the big table should have not have a matched key > value in the small table > * The repeated value should not be a null value > * Some rows should be filtered out as part of the on clause filter. -- This message was sent by Atlassian Jira (v8.3.2#803003)
