Re: Review Request 57518: HIVE-16091 Support subqueries in select

2017-03-14 Thread Vineet Garg 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57518/
---

(Updated March 14, 2017, 10:17 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Fixes fail ptest


Bugs: HIVE-16091
https://issues.apache.org/jira/browse/HIVE-16091


Repository: hive-git


Description
---

Support for subqueries in select statement (with limitations)
Wiki: https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties e445d3b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
 b7b16b8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
 c1768f4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java bf0a11b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java f0165dd 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java 6c10704 
  ql/src/test/queries/clientnegative/subquery_in_select.q 1365389 
  ql/src/test/queries/clientnegative/subquery_select_aggregate.q PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_select_complex_expr.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_select_distinct.q PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_select_distinct2.q PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_select_udf.q PRE-CREATION 
  ql/src/test/queries/clientpositive/subquery_select.q PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_in_select.q.out 5c69690 
  ql/src/test/results/clientnegative/subquery_select_aggregate.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_select_distinct.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_select_distinct2.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_select_udf.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/subquery_select.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/57518/diff/4/

Changes: https://reviews.apache.org/r/57518/diff/3-4/


Testing
---

Added new tests
Done pre-commit testing
Pending: More tests


Thanks,

Vineet Garg

Re: Review Request 57518: HIVE-16091 Support subqueries in select

2017-03-14 Thread Vineet Garg 


> On March 10, 2017, 7:46 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
> > Lines 2815 (patched)
> > 
> >
> > Is this needed? Having this check in genSelectLogicalPlan() should be 
> > sufficient?

This is needed since genGBLogicalPlan is called first and I remember for 
subqueries with distinct there was some issue and I had to add this check. Let 
me check and confirm this.


> On March 10, 2017, 7:46 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/subquery_select.q.out
> > Lines 19-24 (patched)
> > 
> >
> > This plan is doing null counting for inner query for IN clause. I think 
> > this is unnecessary.

Since we care if IN returns true, false or NULL we need to know presence of 
NULL so doing count() is necessary.


- Vineet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57518/#review168654
---


On March 14, 2017, 5:05 a.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57518/
> ---
> 
> (Updated March 14, 2017, 5:05 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16091
> https://issues.apache.org/jira/browse/HIVE-16091
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Support for subqueries in select statement (with limitations)
> Wiki: https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 2a7627a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
>  b7b16b8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
>  c1768f4 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java bf0a11b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java f0165dd 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java 6c10704 
>   ql/src/test/queries/clientnegative/subquery_in_select.q 1365389 
>   ql/src/test/queries/clientnegative/subquery_select_aggregate.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/subquery_select_complex_expr.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/subquery_select_distinct.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/subquery_select_distinct2.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/subquery_select_udf.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/subquery_select.q PRE-CREATION 
>   ql/src/test/results/clientnegative/subquery_in_select.q.out 5c69690 
>   ql/src/test/results/clientnegative/subquery_select_aggregate.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/subquery_select_distinct.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/subquery_select_distinct2.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/subquery_select_udf.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/subquery_select.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/57518/diff/3/
> 
> 
> Testing
> ---
> 
> Added new tests
> Done pre-commit testing
> Pending: More tests
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

Re: Review Request 57518: HIVE-16091 Support subqueries in select

2017-03-13 Thread Vineet Garg 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57518/
---

(Updated March 14, 2017, 5:05 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16091
https://issues.apache.org/jira/browse/HIVE-16091


Repository: hive-git


Description
---

Support for subqueries in select statement (with limitations)
Wiki: https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 2a7627a 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
 b7b16b8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
 c1768f4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java bf0a11b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java f0165dd 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java 6c10704 
  ql/src/test/queries/clientnegative/subquery_in_select.q 1365389 
  ql/src/test/queries/clientnegative/subquery_select_aggregate.q PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_select_complex_expr.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_select_distinct.q PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_select_distinct2.q PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_select_udf.q PRE-CREATION 
  ql/src/test/queries/clientpositive/subquery_select.q PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_in_select.q.out 5c69690 
  ql/src/test/results/clientnegative/subquery_select_aggregate.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_select_distinct.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_select_distinct2.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_select_udf.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/subquery_select.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/57518/diff/3/

Changes: https://reviews.apache.org/r/57518/diff/2-3/


Testing
---

Added new tests
Done pre-commit testing
Pending: More tests


Thanks,

Vineet Garg

Re: Review Request 57518: HIVE-16091 Support subqueries in select

2017-03-12 Thread Vineet Garg 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57518/
---

(Updated March 13, 2017, 1:44 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16091
https://issues.apache.org/jira/browse/HIVE-16091


Repository: hive-git


Description
---

Support for subqueries in select statement (with limitations)
Wiki: https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 2a7627a 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
 b7b16b8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
 c1768f4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java bf0a11b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java f0165dd 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java 6c10704 
  ql/src/test/queries/clientnegative/subquery_in_select.q 1365389 
  ql/src/test/queries/clientnegative/subquery_select_aggregate.q PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_select_complex_expr.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_select_distinct.q PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_select_distinct2.q PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_select_udf.q PRE-CREATION 
  ql/src/test/queries/clientpositive/subquery_select.q PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_in_select.q.out 5c69690 
  ql/src/test/results/clientnegative/subquery_select_aggregate.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_select_distinct.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_select_distinct2.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_select_udf.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/subquery_select.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/57518/diff/2/

Changes: https://reviews.apache.org/r/57518/diff/1-2/


Testing
---

Added new tests
Done pre-commit testing
Pending: More tests


Thanks,

Vineet Garg

Re: Review Request 57518: HIVE-16091 Support subqueries in select

2017-03-10 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57518/#review168654
---




ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Lines 2815 (patched)


Is this needed? Having this check in genSelectLogicalPlan() should be 
sufficient?



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Lines 3656 (patched)


This will never be ExprNodeColumnDesc, so should always be false.



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Lines 3710-3711 (patched)


commented code can be removed.



ql/src/test/queries/clientpositive/subquery_select.q
Lines 101 (patched)


Can you also add tests for not in and not exists?



ql/src/test/results/clientpositive/llap/subquery_select.q.out
Lines 19-24 (patched)


This plan is doing null counting for inner query for IN clause. I think 
this is unnecessary.


- Ashutosh Chauhan


On March 10, 2017, 6:32 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57518/
> ---
> 
> (Updated March 10, 2017, 6:32 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16091
> https://issues.apache.org/jira/browse/HIVE-16091
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Support for subqueries in select statement (with limitations)
> Wiki: https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties b01ebd8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
>  b7b16b8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
>  c1768f4 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java db8d46e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java f0165dd 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java 6c10704 
>   ql/src/test/queries/clientnegative/subquery_select_aggregate.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/subquery_select_complex_expr.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/subquery_select_distinct.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/subquery_select_distinct2.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/subquery_select_udf.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/subquery_select.q PRE-CREATION 
>   ql/src/test/results/clientnegative/subquery_select_aggregate.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/subquery_select_complex_expr.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/subquery_select_distinct.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/subquery_select_distinct2.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/subquery_select_udf.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/subquery_select.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/57518/diff/1/
> 
> 
> Testing
> ---
> 
> Added new tests
> Done pre-commit testing
> Pending: More tests
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

Review Request 57518: HIVE-16091 Support subqueries in select

2017-03-10 Thread Vineet Garg 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57518/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16091
https://issues.apache.org/jira/browse/HIVE-16091


Repository: hive-git


Description
---

Support for subqueries in select statement (with limitations)
Wiki: https://cwiki.apache.org/confluence/display/Hive/Subqueries+in+SELECT


Diffs
-

  itests/src/test/resources/testconfiguration.properties b01ebd8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
 b7b16b8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
 c1768f4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java db8d46e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java f0165dd 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java 6c10704 
  ql/src/test/queries/clientnegative/subquery_select_aggregate.q PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_select_complex_expr.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_select_distinct.q PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_select_distinct2.q PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_select_udf.q PRE-CREATION 
  ql/src/test/queries/clientpositive/subquery_select.q PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_select_aggregate.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_select_complex_expr.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_select_distinct.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_select_distinct2.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_select_udf.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/subquery_select.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/57518/diff/1/


Testing
---

Added new tests
Done pre-commit testing
Pending: More tests


Thanks,

Vineet Garg