> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/lib/SubQueryWalker.java, line 27 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582079#file1582079line27> > > > > Better name: Expression walker?
Updated class name to ExpressionWalker > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/lib/SubQueryWalker.java, line 40 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582079#file1582079line40> > > > > When is later? > > SubQueryExprProcessor already handles TOK_SUBQUERY_OP so why do we need > > to bypass subquery processing? Later is when we generate plan for subquery. (Shouldn't have said later since we have already processed and created logical plan for subquery at this point). SubQueryWalker bypasses subquery since we are seperately generating logical plan for it. SubQueryExprProcessor then uses this generated pland and creates appropriate ExprNodeSubQueryDesc. I'll update comments to make it more clear. > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java, > > line 82 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582083#file1582083line82> > > > > Each HiveFilter will be visited by RelShuttle. Why do we need to > > traverse the tree here? RelShuttle is used by decorrelation logic but SubqueryRemoveRule uses getVariableSet function which needs to traverse the filter to get set of all corr vars. > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java, > > line 283 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line283> > > > > can be called before Decorrelator? This particular rule is defined in HiveRelDecorrelator and is called just before decorrelation. Edit: BTW we plan to use Calcite's RelDecorrelator soon and will have all these rules in decorrelation anyway. Not sure if it is worth updating it now. > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java, > > line 419 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line419> > > > > This can be removed since we alreafy have overloaded version of this > > with HiveAggregate. We can not remove this since at this point we have a plan with mix of LogicalAggregate and HiveAggregate. Decorrelator uses reflection to dynamically figure out the type and call appropriate method. > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java, > > lines 1053-1054 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line1053> > > > > Can this be removed? If so, what above comment above? Yes removed. > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java, > > line 860 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line860> > > > > Can be removed. Can't remove (same as above) > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java, > > line 1144 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line1144> > > > > Can be removed. Can not (same as above) > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java, > > line 1355 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line1355> > > > > can be removed. Cannot (reason is same as above) > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java, > > line 77 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582087#file1582087line77> > > > > This assert can fail if there is no subquery, no? this rule is called only if there is subquery > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java, > > line 513 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582089#file1582089line513> > > > > this should be RexCorrelVariable, not RexFieldAccess? No? No it consists of RexCorrelVariable (corExpr here) > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeSubQueryDesc.java, line > > 29 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582095#file1582095line29> > > > > I am not sure about ExprNodeSubQueryDesc class since it encapsulates > > both calcite as well as Hive data structure. Till Now, ExprNodeDesc is > > exclusively Hive concept. > > Is it necessary to mix these two? I am not sure how else could I encapsulate all subquery related info which is required later to convert to RexSubQuery. > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/test/queries/clientpositive/semijoin5.q, line 1 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582102#file1582102line1> > > > > Any reason for this? No I added it to make plan more readable and forgot to remove it. I'll get rid of it. > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/test/results/clientpositive/constant_prop_3.q.out, line 161 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582110#file1582110line161> > > > > Plan change because of semi join flag? No this has a NOT IN subquery > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/test/results/clientpositive/constprog_partitioner.q.out, lines 84-86 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582111#file1582111line84> > > > > Semijoin flag? No this has an IN subquery > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/test/results/clientpositive/masking_3.q.out, lines 19-21 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582125#file1582125line19> > > > > Need to set semijoin flag in q file. As we looked at it the other day this test has a subquery mask, that's why you see this plan change. > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java, > > line 1931 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line1931> > > > > This should be Aggregate and Project instead. It doesn't work with Aggregate and Project because rule's onMatch expects LogicalAggregate and on receiving HiveAggregate it throws an exception of incomptabile types. These rules will need to be re-written to support Hive node. I think re-writing decorrelate to instead work with Aggregate, Project etc should resolve this issue. > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java, > > line 2036 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line2036> > > > > instanceof Filter, instead. same as above > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveReplicatedRelBuilder.java, > > lines 141-156 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582082#file1582082line141> > > > > These should be Hive Rel Factories, not default. Tried replacing these with HiveRel factories but bunch of tests started failing while creating Project. I'll look into this later. > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java, > > lines 214-218 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line214> > > > > Do we need to fire all these rules? Some of the rules are already used > > during optimization of calcite tree. Shall we just call those rules before > > calling Decorrelator? Will avoid calling rules twice. AdjustProjectForCountAggregateRule is implemented in HiveRelDecorrelator and all of these rules are called before decorrelation. After decorrelation the plan looks very different and it might be helpful calling Filter rules again. I am not yet sure if we need all of these rules before decorrelation. I replicated this code from Calcite. I'll look into this. > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java, > > lines 233-234 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line233> > > > > Similarly these rules can be called after decorrelator on whole tree > > from Calcite planner. These rules are called after decorrelation. Let me see if these rules are already called from Calcite planner, if so we can get rid of these rules > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java, > > line 64 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582083#file1582083line64> > > > > Wont this assert fail (correctly) for non-correlated variables? You are right. Looking into this > On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java, line 2082 > > <https://reviews.apache.org/r/54517/diff/3/?file=1582090#file1582090line2082> > > > > should generate a new id for new QB. How do I generate a new id? I copied this code from previous subquery code. - Vineet ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54517/#review158889 ----------------------------------------------------------- On Dec. 12, 2016, 6:17 p.m., Vineet Garg wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/54517/ > ----------------------------------------------------------- > > (Updated Dec. 12, 2016, 6:17 p.m.) > > > Review request for hive and Ashutosh Chauhan. > > > Bugs: HIVE-15192 > https://issues.apache.org/jira/browse/HIVE-15192 > > > Repository: hive-git > > > Description > ------- > > This patch is 1st phase of getting rid of Hive's subquery transformation and > de-corelation logic and leverage Calcite's functionality to plan sub-queries. > > Known issues with this patch > * Few return path tests are failing and are disabled with this patch. > * Semi-join optimization (HIVE-15227) is disabled currently as it doesn't > work with this patch. > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9064e49 > itests/src/test/resources/testconfiguration.properties aa3d72d > ql/src/java/org/apache/hadoop/hive/ql/lib/SubQueryWalker.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelShuttle.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelShuttleImpl.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveReplicatedRelBuilder.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java > 0410c91 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveJoin.java > ba9483e > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java > 3e0a9a6 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java > d494c9f > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java > f8fb475 > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java f1f3bf9 > ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java 3458fb6 > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 79e55b2 > ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckCtx.java 02896ff > ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java > ace3eaf > ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeSubQueryDesc.java > PRE-CREATION > ql/src/test/queries/clientnegative/subquery_nested_subquery.q > ql/src/test/queries/clientnegative/subquery_restrictions.q PRE-CREATION > ql/src/test/queries/clientnegative/subquery_shared_alias.q > ql/src/test/queries/clientpositive/cbo_rp_auto_join1.q cbfb5d5 > ql/src/test/queries/clientpositive/join31.q c79105f > ql/src/test/queries/clientpositive/multiMapJoin2.q cf5dbb0 > ql/src/test/queries/clientpositive/semijoin5.q 3e7c20a > ql/src/test/queries/clientpositive/subquery_in.q c01ae70 > ql/src/test/queries/clientpositive/subquery_notin.q 3f4fb7f > ql/src/test/queries/clientpositive/subquery_notin_having.q 05148df > ql/src/test/results/clientnegative/subquery_in_groupby.q.out 809bb0a > ql/src/test/results/clientnegative/subquery_in_select.q.out 3d74132 > ql/src/test/results/clientnegative/subquery_nested_subquery.q.out 140b093 > ql/src/test/results/clientnegative/subquery_restrictions.q.out PRE-CREATION > ql/src/test/results/clientpositive/constant_prop_3.q.out 58f1065 > ql/src/test/results/clientpositive/constprog_partitioner.q.out d1016ad > ql/src/test/results/clientpositive/llap/cbo_rp_subq_in.q.out f6bfad2 > ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out > f993cf0 > ql/src/test/results/clientpositive/llap/explainuser_1.q.out 70ec02f > ql/src/test/results/clientpositive/llap/lineage3.q.out 1a532da > ql/src/test/results/clientpositive/llap/subquery_exists.q.out 1a006d8 > ql/src/test/results/clientpositive/llap/subquery_in.q.out 321f1cc > ql/src/test/results/clientpositive/llap/subquery_nested_subquery.q.out > PRE-CREATION > ql/src/test/results/clientpositive/llap/subquery_notin.q.out 3da1acb > ql/src/test/results/clientpositive/llap/subquery_shared_alias.q.out > PRE-CREATION > ql/src/test/results/clientpositive/llap/subquery_views.q.out 35e80ae > ql/src/test/results/clientpositive/llap/vector_groupby_mapjoin.q.out > ff658d7 > ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out a075662 > > ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out > 76c8404 > ql/src/test/results/clientpositive/masking_3.q.out 1925dce > ql/src/test/results/clientpositive/masking_4.q.out 7e923e8 > ql/src/test/results/clientpositive/perf/query45.q.out 7bc137c > ql/src/test/results/clientpositive/perf/query70.q.out 611af74 > ql/src/test/results/clientpositive/semijoin4.q.out 3c065a9 > ql/src/test/results/clientpositive/semijoin5.q.out 63b477c > ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out > 567c6d3 > ql/src/test/results/clientpositive/spark/subquery_exists.q.out b58fcbe > ql/src/test/results/clientpositive/spark/subquery_in.q.out 21a48ec > ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out > 012c3eb > ql/src/test/results/clientpositive/subq_where_serialization.q.out f689651 > ql/src/test/results/clientpositive/subquery_exists.q.out 86f9089 > ql/src/test/results/clientpositive/subquery_exists_having.q.out 8861c82 > ql/src/test/results/clientpositive/subquery_in_having.q.out 854aa36 > ql/src/test/results/clientpositive/subquery_notexists.q.out ede7855 > ql/src/test/results/clientpositive/subquery_notexists_having.q.out 9349f2d > ql/src/test/results/clientpositive/subquery_notin_having.q.out 804f411 > ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out c7e1f02 > ql/src/test/results/clientpositive/vector_groupby_mapjoin.q.out 3468657 > ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out 160b088 > > Diff: https://reviews.apache.org/r/54517/diff/ > > > Testing > ------- > > * Added new tests. > * Pre-commit testing on-going > > > Thanks, > > Vineet Garg > >
