Re: Review Request 68868: HIVE-17043: Remove non unique columns from group by keys if not referenced later

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68868/
---

(Updated Oct. 7, 2018, 4:14 a.m.)


Review request for hive and Jesús Camacho Rodríguez.


Changes
---

Addressed review comments


Bugs: HIVE-17043
https://issues.apache.org/jira/browse/HIVE-17043


Repository: hive-git


Description
---

This patch implements/test the following optimizations
* Removal of group by on primary keys
* Reduction of group by keys on primary keys
* is NOT NULL filter removal if NOT NULL constraint is defined


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 9984ce5eed 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 66280b2da1 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveDefaultRelMetadataProvider.java
 635d27e723 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
42e60de6a8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java
 f43ef01293 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java
 5857f730a8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/EstimateUniqueKeys.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdColumnUniqueness.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
 1ca1937ed9 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdUniqueKeys.java
 3bf62c535c 
  
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
 0a499b1a1b 
  ql/src/test/queries/clientpositive/constraints_optimization.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_1.q.out 
735a4db2ee 


Diff: https://reviews.apache.org/r/68868/diff/6/

Changes: https://reviews.apache.org/r/68868/diff/5-6/


Testing
---


Thanks,

Vineet Garg

[jira] [Created] (HIVE-20706) external_jdbc_table2.q failing intermittently

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-20706:
--

 Summary: external_jdbc_table2.q failing intermittently
 Key: HIVE-20706
 URL: https://issues.apache.org/jira/browse/HIVE-20706
 Project: Hive
  Issue Type: Bug
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Collision with external_jdbc_table.q tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68868: HIVE-17043: Remove non unique columns from group by keys if not referenced later

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68868/#review209289
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java
Line 102 (original), 100 (patched)


I had not realized this field was final. It makes sense so object is 
ummutable and field cannot be changed after instantiation. Hence the pair 
option for generateKeys makes more sense (Calcite has a pair class, it should 
be quite straightforward).



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/EstimateUniqueKeys.java
Lines 190 (patched)


Can we change this as we mentioned (first pick up best, otherwise pick up 
original?)



ql/src/test/results/clientpositive/llap/constraints_optimization.q.out
Lines 813 (patched)


Can we add a query with filter on the key:

explain select key1 from dest_g21 where key1 > 1 group by key1, value1;

Query should be optimized since column remains unique and not null.


- Jesús Camacho Rodríguez


On Oct. 6, 2018, 10:18 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68868/
> ---
> 
> (Updated Oct. 6, 2018, 10:18 p.m.)
> 
> 
> Review request for hive and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-17043
> https://issues.apache.org/jira/browse/HIVE-17043
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch implements/test the following optimizations
> * Removal of group by on primary keys
> * Reduction of group by keys on primary keys
> * is NOT NULL filter removal if NOT NULL constraint is defined
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 9984ce5eed 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveDefaultRelMetadataProvider.java
>  635d27e723 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 42e60de6a8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java
>  f43ef01293 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java
>  5857f730a8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/EstimateUniqueKeys.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdColumnUniqueness.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
>  1ca1937ed9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdUniqueKeys.java
>  3bf62c535c 
>   ql/src/test/queries/clientpositive/constraints_optimization.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/materialized_view_rewrite_1.q.out 
> 735a4db2ee 
> 
> 
> Diff: https://reviews.apache.org/r/68868/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

Re: Review Request 68868: HIVE-17043: Remove non unique columns from group by keys if not referenced later

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68868/
---

(Updated Oct. 6, 2018, 10:18 p.m.)


Review request for hive and Jesús Camacho Rodríguez.


Changes
---

Codestyle violation fixes


Bugs: HIVE-17043
https://issues.apache.org/jira/browse/HIVE-17043


Repository: hive-git


Description
---

This patch implements/test the following optimizations
* Removal of group by on primary keys
* Reduction of group by keys on primary keys
* is NOT NULL filter removal if NOT NULL constraint is defined


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 9984ce5eed 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveDefaultRelMetadataProvider.java
 635d27e723 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
42e60de6a8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java
 f43ef01293 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java
 5857f730a8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/EstimateUniqueKeys.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdColumnUniqueness.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
 1ca1937ed9 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdUniqueKeys.java
 3bf62c535c 
  ql/src/test/queries/clientpositive/constraints_optimization.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/materialized_view_rewrite_1.q.out 
735a4db2ee 


Diff: https://reviews.apache.org/r/68868/diff/5/

Changes: https://reviews.apache.org/r/68868/diff/4-5/


Testing
---


Thanks,

Vineet Garg

[GitHub] hive pull request #441: HIVE-20646: Partition filter condition is not pushed...

[sankarh]

Github user sankarh closed the pull request at:

https://github.com/apache/hive/pull/441


---