[jira] [Commented] (HIVE-13582) E061-07 and E061-12: Quantified Comparison Predicates
[
https://issues.apache.org/jira/browse/HIVE-13582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833179#comment-16833179
]
Hive QA commented on HIVE-13582:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967841/HIVE-13582.5.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15951 tests
executed
*Failed tests:*
{noformat}
TestContribCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=281)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_ALL]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_ANY]
(batchId=176)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17119/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17119/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17119/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12967841 - PreCommit-HIVE-Build
> E061-07 and E061-12: Quantified Comparison Predicates
> -
>
> Key: HIVE-13582
> URL: https://issues.apache.org/jira/browse/HIVE-13582
> Project: Hive
> Issue Type: Sub-task
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-13582.1.patch, HIVE-13582.2.patch,
> HIVE-13582.3.patch, HIVE-13582.4.patch, HIVE-13582.5.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> This is a part of the SQL:2011 Analytics Complete Umbrella JIRA HIVE-13554.
> Quantified comparison predicates (ANY/SOME/ALL) are mandatory in the SQL
> standard. Hive should support the predicates (E061-07) and you should be able
> to use these with subqueries (E061-12)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-13582) E061-07 and E061-12: Quantified Comparison Predicates
[
https://issues.apache.org/jira/browse/HIVE-13582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833176#comment-16833176
]
Hive QA commented on HIVE-13582:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 15 new + 539 unchanged - 38
fixed = 554 total (was 577) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 10 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 8 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
26s{color} | {color:red} ql generated 14 new + 2244 unchanged - 9 fixed = 2258
total (was 2253) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Switch statement found in
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveSubQueryRemoveRule.rewriteInExists(RexSubQuery,
Set, RelOptUtil$Logic, HiveSubQRemoveRelBuilder, int, boolean) where one case
falls through to the next case At HiveSubQueryRemoveRule.java:Set,
RelOptUtil$Logic, HiveSubQRemoveRelBuilder, int, boolean) where one case falls
through to the next case At HiveSubQueryRemoveRule.java:[lines 460-463] |
| | Switch statement found in
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveSubQueryRemoveRule.rewriteInExists(RexSubQuery,
Set, RelOptUtil$Logic, HiveSubQRemoveRelBuilder, int, boolean) where default
case is missing At HiveSubQueryRemoveRule.java:Set, RelOptUtil$Logic,
HiveSubQRemoveRelBuilder, int, boolean) where default case is missing At
HiveSubQueryRemoveRule.java:[lines 356-378] |
| | Dead store to stream_retval in
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceSimilarExpressionQuantifierPredicate(CommonTree)
At
HiveParser_IdentifiersParser.java:org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceSimilarExpressionQuantifierPredicate(CommonTree)
At HiveParser_IdentifiersParser.java:[line 9822] |
| | Dead store to stream_retval in
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.quantifierType()
At
HiveParser_IdentifiersParser.java:org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.quantifierType()
At HiveParser_IdentifiersParser.java:[line 9940] |
| | Redundant nullcheck of nonReserved311, which is known to be non-null in
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier()
Redundant null check at HiveParser_IdentifiersParser.java:is known to be
[jira] [Commented] (HIVE-21690) Support outer joins with HiveAggregateJoinTransposeRule and turn it on by default
[
https://issues.apache.org/jira/browse/HIVE-21690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833173#comment-16833173
]
Hive QA commented on HIVE-21690:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967839/HIVE-21690.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 15972 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constant_prop_3]
(batchId=48)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer2]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_smb_ptf]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mapjoin_hint]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin_reddedup]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_select]
(batchId=171)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_select]
(batchId=130)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query64]
(batchId=285)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query64]
(batchId=285)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query30]
(batchId=285)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query54]
(batchId=285)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query64]
(batchId=285)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query81]
(batchId=285)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query30]
(batchId=285)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query54]
(batchId=285)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query64]
(batchId=285)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query81]
(batchId=285)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17118/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17118/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17118/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12967839 - PreCommit-HIVE-Build
> Support outer joins with HiveAggregateJoinTransposeRule and turn it on by
> default
> -
>
> Key: HIVE-21690
> URL: https://issues.apache.org/jira/browse/HIVE-21690
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21690.1.patch
>
>
> 1) This optimization is off by default. We would like to turn on this
> optimization wherein group by is pushed down to join, in some cases top
> aggregate is removed but in most of the cases this optimization adds extra
> aggregate nodes. To measure if those extra aggregates are beneficial or not
> (they might add extra overhead without reducing rows) cost is computed and
> compared b/w previous plan and new plan.
> Since Hive's cost model only consider JOIN's cost and discard cost of rest of
> the nodes, this comparison always favor new plan (since adding aggregate
> beneath join reduces the total number of rows processed by the join and
> therefore reduces the join cost). Therefore turning on this optimization with
> existing cost model is not a good idea.
> One approach to fix this is to localize the cost computation to the rule
> itself, i.e compute the non-cumulative cost of existing aggregate and join
> and compare it with new cost of new aggregates, join and top aggregate.
> Better approach in my opinion would be to fix the cost model and take
> aggregate cost into account (along with the join). This could affect other
> queries and can cause performance regression but those will most likely be
> issues with the planning and should be investigated and fixed.
> 2) This optimization currently only support INNER JOIN. This can be extended
> to
[jira] [Commented] (HIVE-21690) Support outer joins with HiveAggregateJoinTransposeRule and turn it on by default
[
https://issues.apache.org/jira/browse/HIVE-21690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833166#comment-16833166
]
Hive QA commented on HIVE-21690:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
14s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 1 new + 7 unchanged - 0 fixed
= 8 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17118/dev-support/hive-personality.sh
|
| git revision | master / 341fc33 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17118/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17118/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support outer joins with HiveAggregateJoinTransposeRule and turn it on by
> default
> -
>
> Key: HIVE-21690
> URL: https://issues.apache.org/jira/browse/HIVE-21690
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21690.1.patch
>
>
> 1) This optimization is off by default. We would like to turn on this
> optimization wherein group by is pushed down to join, in some cases top
> aggregate is removed but in most of the cases this optimization adds extra
> aggregate nodes. To measure if those extra aggregates are beneficial or not
> (they might add extra overhead without reducing rows) cost is computed and
> compared b/w previous plan and new plan.
> Since Hive's cost model only consider JOIN's cost and discard cost of rest of
> the nodes, this comparison always favor new plan (since adding aggregate
> beneath join reduces the total number of rows processed by the join and
> therefore reduces the join cost). Therefore turning on this optimization with
> existing cost model is not a good idea.
> One approach to fix this is to localize the cost computation to the rule
> itself, i.e compute the non-cumulative cost of existing aggregate and join
> and compare it with new cost of new aggregates, join and top aggregate.
> Better approach in my opinion would
[jira] [Updated] (HIVE-13582) E061-07 and E061-12: Quantified Comparison Predicates
[ https://issues.apache.org/jira/browse/HIVE-13582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-13582: --- Status: Open (was: Patch Available) > E061-07 and E061-12: Quantified Comparison Predicates > - > > Key: HIVE-13582 > URL: https://issues.apache.org/jira/browse/HIVE-13582 > Project: Hive > Issue Type: Sub-task >Reporter: Carter Shanklin >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-13582.1.patch, HIVE-13582.2.patch, > HIVE-13582.3.patch, HIVE-13582.4.patch, HIVE-13582.5.patch > > Time Spent: 1.5h > Remaining Estimate: 0h > > This is a part of the SQL:2011 Analytics Complete Umbrella JIRA HIVE-13554. > Quantified comparison predicates (ANY/SOME/ALL) are mandatory in the SQL > standard. Hive should support the predicates (E061-07) and you should be able > to use these with subqueries (E061-12) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-13582) E061-07 and E061-12: Quantified Comparison Predicates
[ https://issues.apache.org/jira/browse/HIVE-13582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-13582: --- Status: Patch Available (was: Open) > E061-07 and E061-12: Quantified Comparison Predicates > - > > Key: HIVE-13582 > URL: https://issues.apache.org/jira/browse/HIVE-13582 > Project: Hive > Issue Type: Sub-task >Reporter: Carter Shanklin >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-13582.1.patch, HIVE-13582.2.patch, > HIVE-13582.3.patch, HIVE-13582.4.patch, HIVE-13582.5.patch > > Time Spent: 1.5h > Remaining Estimate: 0h > > This is a part of the SQL:2011 Analytics Complete Umbrella JIRA HIVE-13554. > Quantified comparison predicates (ANY/SOME/ALL) are mandatory in the SQL > standard. Hive should support the predicates (E061-07) and you should be able > to use these with subqueries (E061-12) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-13582) E061-07 and E061-12: Quantified Comparison Predicates
[ https://issues.apache.org/jira/browse/HIVE-13582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-13582: --- Attachment: HIVE-13582.5.patch > E061-07 and E061-12: Quantified Comparison Predicates > - > > Key: HIVE-13582 > URL: https://issues.apache.org/jira/browse/HIVE-13582 > Project: Hive > Issue Type: Sub-task >Reporter: Carter Shanklin >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-13582.1.patch, HIVE-13582.2.patch, > HIVE-13582.3.patch, HIVE-13582.4.patch, HIVE-13582.5.patch > > Time Spent: 1.5h > Remaining Estimate: 0h > > This is a part of the SQL:2011 Analytics Complete Umbrella JIRA HIVE-13554. > Quantified comparison predicates (ANY/SOME/ALL) are mandatory in the SQL > standard. Hive should support the predicates (E061-07) and you should be able > to use these with subqueries (E061-12) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-13582) E061-07 and E061-12: Quantified Comparison Predicates
[
https://issues.apache.org/jira/browse/HIVE-13582?focusedWorklogId=237253=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-237253
]
ASF GitHub Bot logged work on HIVE-13582:
-
Author: ASF GitHub Bot
Created on: 04/May/19 21:36
Start Date: 04/May/19 21:36
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #612:
HIVE-13582: Support for Quantified predicates for non-correlated queries
URL: https://github.com/apache/hive/pull/612#discussion_r280996152
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java
##
@@ -101,6 +104,67 @@ static ASTNode isNull(ASTNode expr) {
return node;
}
+ static public void subqueryRestrictionCheck(QB qb, ASTNode subqueryExprNode,
RelNode srcRel,
+ boolean forHavingClause, Set corrScalarQueries, Context ctx,
+ LinkedHashMap relToHiveRR)
+ throws SemanticException {
+
+assert(subqueryExprNode.getType() == HiveParser.TOK_SUBQUERY_EXPR);
+
+/*
+ * Restriction : Subquery is not allowed in LHS
+ */
+if (subqueryExprNode.getChildren().size() == 3
+&& subqueryExprNode.getChild(2).getType() ==
HiveParser.TOK_SUBQUERY_EXPR) {
+ throw new
CalciteSubquerySemanticException(ErrorMsg.UNSUPPORTED_SUBQUERY_EXPRESSION
+ .getMsg(subqueryExprNode.getChild(2), "SubQuery on left hand side is
not supported."));
+}
+
+// avoid subquery restrictions for SOME/ALL for now
+if(subqueryExprNode.getChild(0).getChildCount() > 1
+&& (subqueryExprNode.getChild(0).getChild(1).getType() ==
HiveParser.KW_SOME
+ || subqueryExprNode.getChild(0).getChild(1).getType() ==
HiveParser.KW_ALL)) {
+ return;
+}
+
+// TOK_SUBQUERY_EXPR
+// 0. TOK_SUBQUERY_OP
+// 0. TYPE: IN/SOME/EXISTS
+// 1. Comparion op: >, < etc
+// 1. TOK_QUERY: Subquery
+// 2. LHS expr
+//final ASTNode clonedSubExprAST = (ASTNode)
SubQueryUtils.adaptor.dupTree(subqueryExprNode);
+ //we do not care about the transformation or rewriting of AST
Review comment:
Will fix it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 237253)
Time Spent: 1.5h (was: 1h 20m)
> E061-07 and E061-12: Quantified Comparison Predicates
> -
>
> Key: HIVE-13582
> URL: https://issues.apache.org/jira/browse/HIVE-13582
> Project: Hive
> Issue Type: Sub-task
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-13582.1.patch, HIVE-13582.2.patch,
> HIVE-13582.3.patch, HIVE-13582.4.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> This is a part of the SQL:2011 Analytics Complete Umbrella JIRA HIVE-13554.
> Quantified comparison predicates (ANY/SOME/ALL) are mandatory in the SQL
> standard. Hive should support the predicates (E061-07) and you should be able
> to use these with subqueries (E061-12)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-13582) E061-07 and E061-12: Quantified Comparison Predicates
[ https://issues.apache.org/jira/browse/HIVE-13582?focusedWorklogId=237252=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-237252 ] ASF GitHub Bot logged work on HIVE-13582: - Author: ASF GitHub Bot Created on: 04/May/19 21:36 Start Date: 04/May/19 21:36 Worklog Time Spent: 10m Work Description: vineetgarg02 commented on pull request #612: HIVE-13582: Support for Quantified predicates for non-correlated queries URL: https://github.com/apache/hive/pull/612#discussion_r280996147 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g ## @@ -45,6 +45,7 @@ KW_TRUE : 'TRUE'; KW_FALSE : 'FALSE'; KW_UNKNOWN : 'UNKNOWN'; KW_ALL : 'ALL'; +KW_SOME : 'SOME'; Review comment: I'll add them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 237252) Time Spent: 1h 20m (was: 1h 10m) > E061-07 and E061-12: Quantified Comparison Predicates > - > > Key: HIVE-13582 > URL: https://issues.apache.org/jira/browse/HIVE-13582 > Project: Hive > Issue Type: Sub-task >Reporter: Carter Shanklin >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-13582.1.patch, HIVE-13582.2.patch, > HIVE-13582.3.patch, HIVE-13582.4.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > This is a part of the SQL:2011 Analytics Complete Umbrella JIRA HIVE-13554. > Quantified comparison predicates (ANY/SOME/ALL) are mandatory in the SQL > standard. Hive should support the predicates (E061-07) and you should be able > to use these with subqueries (E061-12) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-13582) E061-07 and E061-12: Quantified Comparison Predicates
[
https://issues.apache.org/jira/browse/HIVE-13582?focusedWorklogId=237251=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-237251
]
ASF GitHub Bot logged work on HIVE-13582:
-
Author: ASF GitHub Bot
Created on: 04/May/19 21:35
Start Date: 04/May/19 21:35
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #612:
HIVE-13582: Support for Quantified predicates for non-correlated queries
URL: https://github.com/apache/hive/pull/612#discussion_r280996133
##
File path: ql/src/test/results/clientpositive/llap/subquery_ALL.q.out
##
@@ -0,0 +1,542 @@
+PREHOOK: query: create table tempty(i int, j int)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@tempty
+POSTHOOK: query: create table tempty(i int, j int)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@tempty
+PREHOOK: query: CREATE TABLE part_null_n0 as select * from part
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@part
+PREHOOK: Output: database:default
+PREHOOK: Output: default@part_null_n0
+POSTHOOK: query: CREATE TABLE part_null_n0 as select * from part
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@part
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@part_null_n0
+POSTHOOK: Lineage: part_null_n0.p_brand SIMPLE
[(part)part.FieldSchema(name:p_brand, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_comment SIMPLE
[(part)part.FieldSchema(name:p_comment, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_container SIMPLE
[(part)part.FieldSchema(name:p_container, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_mfgr SIMPLE
[(part)part.FieldSchema(name:p_mfgr, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_name SIMPLE
[(part)part.FieldSchema(name:p_name, type:string, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_partkey SIMPLE
[(part)part.FieldSchema(name:p_partkey, type:int, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_retailprice SIMPLE
[(part)part.FieldSchema(name:p_retailprice, type:double, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_size SIMPLE
[(part)part.FieldSchema(name:p_size, type:int, comment:null), ]
+POSTHOOK: Lineage: part_null_n0.p_type SIMPLE
[(part)part.FieldSchema(name:p_type, type:string, comment:null), ]
+PREHOOK: query: insert into part_null_n0 values(NULL,NULL,NULL,NULL,NULL,
NULL, NULL,NULL,NULL)
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@part_null_n0
+POSTHOOK: query: insert into part_null_n0 values(NULL,NULL,NULL,NULL,NULL,
NULL, NULL,NULL,NULL)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@part_null_n0
+POSTHOOK: Lineage: part_null_n0.p_brand EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_comment EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_container EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_mfgr EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_name EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_partkey EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_retailprice EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_size EXPRESSION []
+POSTHOOK: Lineage: part_null_n0.p_type EXPRESSION []
+Warning: Shuffle Join MERGEJOIN[37][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in
Stage 'Reducer 3' is a cross product
+PREHOOK: query: explain cbo select count(*) from part where p_partkey <> ALL
(select p_partkey from part)
+PREHOOK: type: QUERY
+PREHOOK: Input: default@part
+ A masked pattern was here
+POSTHOOK: query: explain cbo select count(*) from part where p_partkey <> ALL
(select p_partkey from part)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@part
+ A masked pattern was here
+CBO PLAN:
+HiveAggregate(group=[{}], agg#0=[count()])
+ HiveFilter(condition=[AND(OR(IS NULL($4), =($1, 0)), OR(IS NOT NULL($0),
=($1, 0), IS NOT NULL($4)), OR(>=($2, $1), =($1, 0), IS NOT NULL($4), IS
NULL($0)))])
Review comment:
Hmm that's a good point and I am not sure why not. Is there a rule which
does this? Or the join itself needs to be created with the condition?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 237251)
Time Spent: 1h 10m (was: 1h)
> E061-07 and E061-12: Quantified Comparison Predicates
> -
>
> Key: HIVE-13582
> URL: https://issues.apache.org/jira/browse/HIVE-13582
>
[jira] [Work logged] (HIVE-13582) E061-07 and E061-12: Quantified Comparison Predicates
[ https://issues.apache.org/jira/browse/HIVE-13582?focusedWorklogId=237246=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-237246 ] ASF GitHub Bot logged work on HIVE-13582: - Author: ASF GitHub Bot Created on: 04/May/19 21:26 Start Date: 04/May/19 21:26 Worklog Time Spent: 10m Work Description: vineetgarg02 commented on pull request #612: HIVE-13582: Support for Quantified predicates for non-correlated queries URL: https://github.com/apache/hive/pull/612#discussion_r280995806 ## File path: ql/src/test/queries/clientnegative/subquery_all_equal.q ## @@ -0,0 +1,3 @@ +--! qt:dataset:part +-- =ALL is not allowed +explain select * from part where p_type = ALL(select max(p_type) from part); Review comment: Not yet, but I'll create the jiras This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 237246) Time Spent: 1h (was: 50m) > E061-07 and E061-12: Quantified Comparison Predicates > - > > Key: HIVE-13582 > URL: https://issues.apache.org/jira/browse/HIVE-13582 > Project: Hive > Issue Type: Sub-task >Reporter: Carter Shanklin >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-13582.1.patch, HIVE-13582.2.patch, > HIVE-13582.3.patch, HIVE-13582.4.patch > > Time Spent: 1h > Remaining Estimate: 0h > > This is a part of the SQL:2011 Analytics Complete Umbrella JIRA HIVE-13554. > Quantified comparison predicates (ANY/SOME/ALL) are mandatory in the SQL > standard. Hive should support the predicates (E061-07) and you should be able > to use these with subqueries (E061-12) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21690) Support outer joins with HiveAggregateJoinTransposeRule and turn it on by default
[ https://issues.apache.org/jira/browse/HIVE-21690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21690: --- Attachment: HIVE-21690.1.patch > Support outer joins with HiveAggregateJoinTransposeRule and turn it on by > default > - > > Key: HIVE-21690 > URL: https://issues.apache.org/jira/browse/HIVE-21690 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21690.1.patch > > > 1) This optimization is off by default. We would like to turn on this > optimization wherein group by is pushed down to join, in some cases top > aggregate is removed but in most of the cases this optimization adds extra > aggregate nodes. To measure if those extra aggregates are beneficial or not > (they might add extra overhead without reducing rows) cost is computed and > compared b/w previous plan and new plan. > Since Hive's cost model only consider JOIN's cost and discard cost of rest of > the nodes, this comparison always favor new plan (since adding aggregate > beneath join reduces the total number of rows processed by the join and > therefore reduces the join cost). Therefore turning on this optimization with > existing cost model is not a good idea. > One approach to fix this is to localize the cost computation to the rule > itself, i.e compute the non-cumulative cost of existing aggregate and join > and compare it with new cost of new aggregates, join and top aggregate. > Better approach in my opinion would be to fix the cost model and take > aggregate cost into account (along with the join). This could affect other > queries and can cause performance regression but those will most likely be > issues with the planning and should be investigated and fixed. > 2) This optimization currently only support INNER JOIN. This can be extended > to support OUTER joins. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21690) Support outer joins with HiveAggregateJoinTransposeRule and turn it on by default
[ https://issues.apache.org/jira/browse/HIVE-21690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833160#comment-16833160 ] Vineet Garg commented on HIVE-21690: First patch includes change to cost model and a minor change in the rule itself. > Support outer joins with HiveAggregateJoinTransposeRule and turn it on by > default > - > > Key: HIVE-21690 > URL: https://issues.apache.org/jira/browse/HIVE-21690 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21690.1.patch > > > 1) This optimization is off by default. We would like to turn on this > optimization wherein group by is pushed down to join, in some cases top > aggregate is removed but in most of the cases this optimization adds extra > aggregate nodes. To measure if those extra aggregates are beneficial or not > (they might add extra overhead without reducing rows) cost is computed and > compared b/w previous plan and new plan. > Since Hive's cost model only consider JOIN's cost and discard cost of rest of > the nodes, this comparison always favor new plan (since adding aggregate > beneath join reduces the total number of rows processed by the join and > therefore reduces the join cost). Therefore turning on this optimization with > existing cost model is not a good idea. > One approach to fix this is to localize the cost computation to the rule > itself, i.e compute the non-cumulative cost of existing aggregate and join > and compare it with new cost of new aggregates, join and top aggregate. > Better approach in my opinion would be to fix the cost model and take > aggregate cost into account (along with the join). This could affect other > queries and can cause performance regression but those will most likely be > issues with the planning and should be investigated and fixed. > 2) This optimization currently only support INNER JOIN. This can be extended > to support OUTER joins. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21690) Support outer joins with HiveAggregateJoinTransposeRule and turn it on by default
[ https://issues.apache.org/jira/browse/HIVE-21690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21690: --- Status: Patch Available (was: Open) > Support outer joins with HiveAggregateJoinTransposeRule and turn it on by > default > - > > Key: HIVE-21690 > URL: https://issues.apache.org/jira/browse/HIVE-21690 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21690.1.patch > > > 1) This optimization is off by default. We would like to turn on this > optimization wherein group by is pushed down to join, in some cases top > aggregate is removed but in most of the cases this optimization adds extra > aggregate nodes. To measure if those extra aggregates are beneficial or not > (they might add extra overhead without reducing rows) cost is computed and > compared b/w previous plan and new plan. > Since Hive's cost model only consider JOIN's cost and discard cost of rest of > the nodes, this comparison always favor new plan (since adding aggregate > beneath join reduces the total number of rows processed by the join and > therefore reduces the join cost). Therefore turning on this optimization with > existing cost model is not a good idea. > One approach to fix this is to localize the cost computation to the rule > itself, i.e compute the non-cumulative cost of existing aggregate and join > and compare it with new cost of new aggregates, join and top aggregate. > Better approach in my opinion would be to fix the cost model and take > aggregate cost into account (along with the join). This could affect other > queries and can cause performance regression but those will most likely be > issues with the planning and should be investigated and fixed. > 2) This optimization currently only support INNER JOIN. This can be extended > to support OUTER joins. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21690) Support outer joins with HiveAggregateJoinTransposeRule and turn it on by default
[ https://issues.apache.org/jira/browse/HIVE-21690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-21690: -- > Support outer joins with HiveAggregateJoinTransposeRule and turn it on by > default > - > > Key: HIVE-21690 > URL: https://issues.apache.org/jira/browse/HIVE-21690 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > > 1) This optimization is off by default. We would like to turn on this > optimization wherein group by is pushed down to join, in some cases top > aggregate is removed but in most of the cases this optimization adds extra > aggregate nodes. To measure if those extra aggregates are beneficial or not > (they might add extra overhead without reducing rows) cost is computed and > compared b/w previous plan and new plan. > Since Hive's cost model only consider JOIN's cost and discard cost of rest of > the nodes, this comparison always favor new plan (since adding aggregate > beneath join reduces the total number of rows processed by the join and > therefore reduces the join cost). Therefore turning on this optimization with > existing cost model is not a good idea. > One approach to fix this is to localize the cost computation to the rule > itself, i.e compute the non-cumulative cost of existing aggregate and join > and compare it with new cost of new aggregates, join and top aggregate. > Better approach in my opinion would be to fix the cost model and take > aggregate cost into account (along with the join). This could affect other > queries and can cause performance regression but those will most likely be > issues with the planning and should be investigated and fixed. > 2) This optimization currently only support INNER JOIN. This can be extended > to support OUTER joins. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14669) Have the actual error reported when a q test fails instead of having to go through the logs
[ https://issues.apache.org/jira/browse/HIVE-14669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-14669: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) pushed to master. Thank you [~abstractdog]! > Have the actual error reported when a q test fails instead of having to go > through the logs > --- > > Key: HIVE-14669 > URL: https://issues.apache.org/jira/browse/HIVE-14669 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: 01_mvn_out.png, 02_hive_log.png, HIVE-14469.01.patch, > HIVE-14469.02.patch, HIVE-14469.03.patch, Screen Shot 2019-04-11 at 4.54.41 > PM.png > > > QTest runs end up invoking CliDriver.processLine. This, in most cases, > reports a numeric exit code - 0 for success. Non-zero for various different > error types (which are defined everywhere in the code). > Internally CliDriver does have more information via CommandResult. A lot of > this is not exposed though. That's alright for the end user cli - (Command > line tool translating the error to a code and message). However, it makes > debugging very difficult for QTests - since the log needs to be looked at > each time. > Errors generated by the actual backend execution are mostly available to the > client, and information about these should show up as well. (If it doesn't - > we have a usability issues to fix). > cc [~ekoifman] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21670) Replacing mockito-all with mockito-core dependency
[ https://issues.apache.org/jira/browse/HIVE-21670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21670: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) pushed to master. Thank you [~isuller]! > Replacing mockito-all with mockito-core dependency > -- > > Key: HIVE-21670 > URL: https://issues.apache.org/jira/browse/HIVE-21670 > Project: Hive > Issue Type: Bug >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21670.1.patch, HIVE-21670.1.patch > > > The mockito-all dependency contains an old version of Hamcrest core which can > collide with other Hamcrest dependencies. Replacint it with mockito-core > should be straightforward. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21635) Break up DDLTask - extract Workload Management related operations
[ https://issues.apache.org/jira/browse/HIVE-21635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21635: Resolution: Fixed Status: Resolved (was: Patch Available) pushed to master. Thank you [~mgergely]! > Break up DDLTask - extract Workload Management related operations > - > > Key: HIVE-21635 > URL: https://issues.apache.org/jira/browse/HIVE-21635 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21635.01.patch, HIVE-21635.02.patch, > HIVE-21635.03.patch, HIVE-21635.04.patch, HIVE-21635.05.patch, > HIVE-21635.06.patch, HIVE-21635.07.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #6: extract all the workload management related operations from the old > DDLTask, and move them under the new package. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21685) Wrong simplification in query with multiple IN clauses
[
https://issues.apache.org/jira/browse/HIVE-21685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833122#comment-16833122
]
Zoltan Haindrich commented on HIVE-21685:
-
yes; I think it's redundant - but it can be removed later as well
+1
> Wrong simplification in query with multiple IN clauses
> --
>
> Key: HIVE-21685
> URL: https://issues.apache.org/jira/browse/HIVE-21685
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Oliver Draese
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21685.patch, HIVE-21685.patch, HIVE-21685.patch
>
>
> Simple test to reproduce:
> {code}
> select * from table1 where name IN(‘g’,‘r’) AND name IN(‘a’,‘b’);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
