subject:"Review Request 61165\: HIVE\-16811 Estimate statistics in absence of stats"

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-09-01 Thread Vineet Garg


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/
---

(Updated Sept. 1, 2017, 10:21 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Q files update + review comments


Bugs: HIVE-16811
https://issues.apache.org/jira/browse/HIVE-16811


Repository: hive-git


Description
---

This patch introduces estimation of statistics if stats doesn't already exist.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
  itests/src/test/resources/testconfiguration.properties f452341d1c 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
22790de209 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 ad29d65abb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
  ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
156be41502 
  ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
23f5fcfc76 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
a7c9b3fc41 
  ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
  ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
  ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
  ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
f77c75c232 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
f04519f8c5 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
f15d536945 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
1ec5dc415c 
  ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
  ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 97f8d6b4cc 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 8bc280cedd 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
4b646dcba5 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
ab94c37fed 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
7e3fe4c862 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
054cf1af7e 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
9c37848896 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
21f1b3faff 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
03ff5a6659 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
42e4cb7a5c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
64c10ce953 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
96854a9665 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
b77a879b78 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
9e6053675b 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
13d3accf4c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
7fc90fd19f 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
278c55b3e4 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 649afb502a 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out bac4ae086c 
  ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
  ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out bda8070ba5 
  ql/src/test/results/clientpositive/llap/bucketmapjoin3.q.out ae17c1b707 
  ql/src/test/results/clientpositive/llap/bucketmapjoin4.q.out aab43431e6 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 3a2cbb2b14 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 922f23a29e 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_2.q.out 
e4fed1124b

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-09-01 Thread Vineet Garg


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/
---

(Updated Sept. 1, 2017, 7 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Addressed review comments


Bugs: HIVE-16811
https://issues.apache.org/jira/browse/HIVE-16811


Repository: hive-git


Description
---

This patch introduces estimation of statistics if stats doesn't already exist.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
  itests/src/test/resources/testconfiguration.properties f452341d1c 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
22790de209 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 ad29d65abb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
  ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
156be41502 
  ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
23f5fcfc76 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
a7c9b3fc41 
  ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
  ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
  ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
  ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
f77c75c232 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
f04519f8c5 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
f15d536945 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
1ec5dc415c 
  ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
  ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 97f8d6b4cc 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 8bc280cedd 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
4b646dcba5 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
ab94c37fed 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
7e3fe4c862 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
054cf1af7e 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
9c37848896 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
21f1b3faff 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
03ff5a6659 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
42e4cb7a5c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
64c10ce953 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
96854a9665 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
b77a879b78 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
9e6053675b 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
13d3accf4c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
7fc90fd19f 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
278c55b3e4 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 649afb502a 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out bac4ae086c 
  ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
  ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out bda8070ba5 
  ql/src/test/results/clientpositive/llap/bucketmapjoin3.q.out ae17c1b707 
  ql/src/test/results/clientpositive/llap/bucketmapjoin4.q.out aab43431e6 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 3a2cbb2b14 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 922f23a29e 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_2.q.out 
e4fed1124b 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_6.q.out

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-09-01 Thread Ashutosh Chauhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/#review184359
---




ql/src/test/queries/clientpositive/join_reordering_no_stats.q
Lines 52 (patched)


Need test for estimateStats=false.



ql/src/test/results/clientpositive/annotate_stats_table.q.out
Line 45 (original), 45 (patched)


This test is specifically for stats state. This change looks incorrect.



ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out
Line 325 (original), 325 (patched)


This doesn't look correct. Estimated stats with state: COMPLETE.


- Ashutosh Chauhan


On Sept. 1, 2017, 7:49 a.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61165/
> ---
> 
> (Updated Sept. 1, 2017, 7:49 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16811
> https://issues.apache.org/jira/browse/HIVE-16811
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch introduces estimation of statistics if stats doesn't already exist.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
>   itests/src/test/resources/testconfiguration.properties f452341d1c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 22790de209 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  ad29d65abb 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
>   ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
>   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
>   ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
>   ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
> 156be41502 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
>   ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
> 23f5fcfc76 
>   
> ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
> a7c9b3fc41 
>   ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
>   ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
>   ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
>   ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
> f77c75c232 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
> f04519f8c5 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
> f15d536945 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
> 1ec5dc415c 
>   ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
>   ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
>   ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 
> 97f8d6b4cc 
>   ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 
> 8bc280cedd 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
> 4b646dcba5 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
> ab94c37fed 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
> 7e3fe4c862 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
> 054cf1af7e 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
> 9c37848896 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
> 21f1b3faff 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
> 03ff5a6659 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
> 42e4cb7a5c 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
> 64c10ce953 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
> 96854a9665 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
> b77a879b78 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
> 9e6053675b 
>

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-09-01 Thread Ashutosh Chauhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/#review184356
---




ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Lines 203 (patched)


This function returns 0 even when stats are absent, making rest of logic to 
not work correctly.



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Lines 222 (patched)


This function also returns 0 when stats are absent, making rest of logic to 
not work correctly.



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Lines 325 (patched)


Its also possible that stats for few cols are present and absent for 
others. Seems like that case is not handled.



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Lines 1006 (patched)


This wont be correct for tiny and small int.



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Lines 1009 (patched)


Long.max and Long.min



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Lines 1034 (patched)


nulls are trouble some. Lets arbitartily pick start of 1970 (unix epoch) 
and end of unix epoch (2038). Can set same for ts.



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Lines 1036 (patched)


todo needed


- Ashutosh Chauhan


On Sept. 1, 2017, 7:49 a.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61165/
> ---
> 
> (Updated Sept. 1, 2017, 7:49 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16811
> https://issues.apache.org/jira/browse/HIVE-16811
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch introduces estimation of statistics if stats doesn't already exist.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
>   itests/src/test/resources/testconfiguration.properties f452341d1c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 22790de209 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  ad29d65abb 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
>   ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
>   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
>   ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
>   ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
> 156be41502 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
>   ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
> 23f5fcfc76 
>   
> ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
> a7c9b3fc41 
>   ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
>   ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
>   ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
>   ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
> f77c75c232 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
> f04519f8c5 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
> f15d536945 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
> 1ec5dc415c 
>   ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
>   ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
>   ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 
> 97f8d6b4cc 
>   ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 
> 8bc280cedd 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
> 4b646dcba5 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
> ab94c37fed 
>

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-09-01 Thread Vineet Garg


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/
---

(Updated Sept. 1, 2017, 7:49 a.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Q files update


Bugs: HIVE-16811
https://issues.apache.org/jira/browse/HIVE-16811


Repository: hive-git


Description
---

This patch introduces estimation of statistics if stats doesn't already exist.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
  itests/src/test/resources/testconfiguration.properties f452341d1c 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
22790de209 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 ad29d65abb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
  ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
156be41502 
  ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
23f5fcfc76 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
a7c9b3fc41 
  ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
  ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
  ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
  ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
f77c75c232 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
f04519f8c5 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
f15d536945 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
1ec5dc415c 
  ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
  ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 97f8d6b4cc 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 8bc280cedd 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
4b646dcba5 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
ab94c37fed 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
7e3fe4c862 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
054cf1af7e 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
9c37848896 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
21f1b3faff 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
03ff5a6659 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
42e4cb7a5c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
64c10ce953 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
96854a9665 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
b77a879b78 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
9e6053675b 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
13d3accf4c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
7fc90fd19f 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
278c55b3e4 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 649afb502a 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out bac4ae086c 
  ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
  ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out bda8070ba5 
  ql/src/test/results/clientpositive/llap/bucketmapjoin3.q.out ae17c1b707 
  ql/src/test/results/clientpositive/llap/bucketmapjoin4.q.out aab43431e6 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 3a2cbb2b14 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 922f23a29e 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_2.q.out 
e4fed1124b 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_6.q.out 
67e925ad99

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-08-31 Thread Vineet Garg


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/
---

(Updated Sept. 1, 2017, 5:20 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16811
https://issues.apache.org/jira/browse/HIVE-16811


Repository: hive-git


Description
---

This patch introduces estimation of statistics if stats doesn't already exist.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
  itests/src/test/resources/testconfiguration.properties f452341d1c 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
22790de209 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 ad29d65abb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
  ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
156be41502 
  ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
23f5fcfc76 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
a7c9b3fc41 
  ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
  ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
  ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
  ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
f77c75c232 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
f04519f8c5 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
f15d536945 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
1ec5dc415c 
  ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
  ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 97f8d6b4cc 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 8bc280cedd 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
4b646dcba5 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
ab94c37fed 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
7e3fe4c862 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
054cf1af7e 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
9c37848896 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
21f1b3faff 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
03ff5a6659 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
42e4cb7a5c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
64c10ce953 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
96854a9665 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
b77a879b78 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
9e6053675b 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
13d3accf4c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
7fc90fd19f 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
278c55b3e4 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 649afb502a 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out bac4ae086c 
  ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
  ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out bda8070ba5 
  ql/src/test/results/clientpositive/llap/bucketmapjoin3.q.out ae17c1b707 
  ql/src/test/results/clientpositive/llap/bucketmapjoin4.q.out aab43431e6 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 3a2cbb2b14 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 922f23a29e 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_2.q.out 
e4fed1124b 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_6.q.out 
67e925ad99

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-08-31 Thread Vineet Garg



> On Aug. 29, 2017, 9:26 p.m., Ashutosh Chauhan wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 1672 (patched)
> > 
> >
> > varchar has length in its type. We shall use that.

Can you elobrate please? Where is varchar type? I don't see length field in 
VarcharTypeInfo.


> On Aug. 29, 2017, 9:26 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
> > Lines 199 (patched)
> > 
> >
> > Need to pass shouldEstimateStats to this function so that it respects 
> > it.

Updated the code to return early if shouldEstimateStats is false.


> On Aug. 29, 2017, 9:26 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
> > Lines 231 (patched)
> > 
> >
> > if config is false, it will return 1 as numrows, is that what we want?
> > It seems like caller has no way to figure out that stats are not 
> > estimated so it shall turn off stats based optimization.

Updated the code to return whatever we retrieve from metastore if 
shouldEstimateStat is off.


> On Aug. 29, 2017, 9:26 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/insert1.q.out
> > Line 46 (original), 46 (patched)
> > 
> >
> > Is change to COMPLETE expected?

Yes Basic stat's state is changed to COMPLETE since now we have both data size 
and number of rows.


> On Aug. 29, 2017, 9:26 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/jdbc_handler.q.out
> > Line 129 (original), 129 (patched)
> > 
> >
> > Expected?

Yes


> On Aug. 29, 2017, 9:26 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out
> > Line 231 (original), 231 (patched)
> > 
> >
> > Overflow in data size calculation?

Yeh this was an existing BUG in GroupBy's stats computation for aggregation 
columns. I have fixed it. Thanks for pointing it out.


- Vineet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/#review184071
---


On Sept. 1, 2017, 12:04 a.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61165/
> ---
> 
> (Updated Sept. 1, 2017, 12:04 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16811
> https://issues.apache.org/jira/browse/HIVE-16811
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch introduces estimation of statistics if stats doesn't already exist.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
>   itests/src/test/resources/testconfiguration.properties fa6a2aaea0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 22790de209 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  ad29d65abb 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
>   ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
>   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
>   ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
>   ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
> 156be41502 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
>   ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
> 23f5fcfc76 
>   
> ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
> a7c9b3fc41 
>   ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
>   ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
>   ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
>   ql/src/test/results/clientpositive/join43.q.out 24168ca4fe

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-08-31 Thread Vineet Garg


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/
---

(Updated Sept. 1, 2017, 12:04 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16811
https://issues.apache.org/jira/browse/HIVE-16811


Repository: hive-git


Description
---

This patch introduces estimation of statistics if stats doesn't already exist.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
  itests/src/test/resources/testconfiguration.properties fa6a2aaea0 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
22790de209 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 ad29d65abb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
  ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
156be41502 
  ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
23f5fcfc76 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
a7c9b3fc41 
  ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
  ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
  ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
  ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
f77c75c232 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
f04519f8c5 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
f15d536945 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
1ec5dc415c 
  ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
  ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 97f8d6b4cc 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 8bc280cedd 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
4b646dcba5 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
ab94c37fed 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
7e3fe4c862 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
054cf1af7e 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
9c37848896 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
21f1b3faff 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
03ff5a6659 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
42e4cb7a5c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
64c10ce953 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
96854a9665 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
b77a879b78 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
9e6053675b 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
13d3accf4c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
7fc90fd19f 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
278c55b3e4 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 649afb502a 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out bac4ae086c 
  ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
  ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out bda8070ba5 
  ql/src/test/results/clientpositive/llap/bucketmapjoin3.q.out ae17c1b707 
  ql/src/test/results/clientpositive/llap/bucketmapjoin4.q.out aab43431e6 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 3a2cbb2b14 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 922f23a29e 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_2.q.out 
e4fed1124b 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_6.q.out 
67e925ad99

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-08-31 Thread Ashutosh Chauhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/#review184318
---



Latest patch doesn't address all review comments. Shall I wait for new patch 
before reviewing this?

- Ashutosh Chauhan


On Aug. 31, 2017, 10:02 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61165/
> ---
> 
> (Updated Aug. 31, 2017, 10:02 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16811
> https://issues.apache.org/jira/browse/HIVE-16811
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch introduces estimation of statistics if stats doesn't already exist.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
>   itests/src/test/resources/testconfiguration.properties fa6a2aaea0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 22790de209 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  ad29d65abb 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
>   ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
>   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
>   ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
>   ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
> 156be41502 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
>   ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
> 23f5fcfc76 
>   
> ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
> a7c9b3fc41 
>   ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
>   ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
>   ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
>   ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
> f77c75c232 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
> f04519f8c5 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
> f15d536945 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
> 1ec5dc415c 
>   ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
>   ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
>   ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 
> 97f8d6b4cc 
>   ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 
> 8bc280cedd 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
> 4b646dcba5 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
> ab94c37fed 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
> 7e3fe4c862 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
> 054cf1af7e 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
> 9c37848896 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
> 21f1b3faff 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
> 03ff5a6659 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
> 42e4cb7a5c 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
> 64c10ce953 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
> 96854a9665 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
> b77a879b78 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
> 9e6053675b 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
> 13d3accf4c 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
> 7fc90fd19f 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
> 278c55b3e4 
>   ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 
> 649afb502a 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out 
> bac4ae086c 
>

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-08-31 Thread Vineet Garg


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/
---

(Updated Aug. 31, 2017, 10:02 p.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16811
https://issues.apache.org/jira/browse/HIVE-16811


Repository: hive-git


Description
---

This patch introduces estimation of statistics if stats doesn't already exist.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
  itests/src/test/resources/testconfiguration.properties fa6a2aaea0 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
22790de209 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 ad29d65abb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
  ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
156be41502 
  ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
23f5fcfc76 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
a7c9b3fc41 
  ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
  ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
  ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
  ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
f77c75c232 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
f04519f8c5 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
f15d536945 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
1ec5dc415c 
  ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
  ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 97f8d6b4cc 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 8bc280cedd 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
4b646dcba5 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
ab94c37fed 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
7e3fe4c862 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
054cf1af7e 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
9c37848896 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
21f1b3faff 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
03ff5a6659 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
42e4cb7a5c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
64c10ce953 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
96854a9665 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
b77a879b78 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
9e6053675b 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
13d3accf4c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
7fc90fd19f 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
278c55b3e4 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 649afb502a 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out bac4ae086c 
  ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
  ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out bda8070ba5 
  ql/src/test/results/clientpositive/llap/bucketmapjoin3.q.out ae17c1b707 
  ql/src/test/results/clientpositive/llap/bucketmapjoin4.q.out aab43431e6 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 3a2cbb2b14 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 922f23a29e 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_2.q.out 
e4fed1124b 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_6.q.out 
67e925ad99

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-08-31 Thread Vineet Garg


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/
---

(Updated Aug. 31, 2017, 9:58 p.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16811
https://issues.apache.org/jira/browse/HIVE-16811


Repository: hive-git


Description
---

This patch introduces estimation of statistics if stats doesn't already exist.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
  itests/src/test/resources/testconfiguration.properties fa6a2aaea0 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
22790de209 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 ad29d65abb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
  ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
156be41502 
  ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
23f5fcfc76 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
a7c9b3fc41 
  ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
  ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
  ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
  ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
f77c75c232 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
f04519f8c5 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
f15d536945 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
1ec5dc415c 
  ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
  ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 97f8d6b4cc 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 8bc280cedd 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
4b646dcba5 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
ab94c37fed 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
7e3fe4c862 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
054cf1af7e 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
9c37848896 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
21f1b3faff 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
03ff5a6659 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
42e4cb7a5c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
64c10ce953 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
96854a9665 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
b77a879b78 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
9e6053675b 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
13d3accf4c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
7fc90fd19f 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
278c55b3e4 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 649afb502a 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out bac4ae086c 
  ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
  ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out bda8070ba5 
  ql/src/test/results/clientpositive/llap/bucketmapjoin3.q.out ae17c1b707 
  ql/src/test/results/clientpositive/llap/bucketmapjoin4.q.out aab43431e6 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 3a2cbb2b14 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 922f23a29e 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_2.q.out 
e4fed1124b 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_6.q.out 
67e925ad99

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-08-29 Thread Ashutosh Chauhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/#review184071
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 1672 (patched)


varchar has length in its type. We shall use that.



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Lines 199 (patched)


Need to pass shouldEstimateStats to this function so that it respects it.



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Lines 231 (patched)


if config is false, it will return 1 as numrows, is that what we want?
It seems like caller has no way to figure out that stats are not estimated 
so it shall turn off stats based optimization.



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Line 1156 (original), 1316 (patched)


Can you leave a TODO here?



ql/src/test/results/clientpositive/llap/explainuser_1.q.out
Line 138 (original), 138 (patched)


This doesn't look correct. Is this overflow?



ql/src/test/results/clientpositive/llap/insert1.q.out
Line 46 (original), 46 (patched)


Is change to COMPLETE expected?



ql/src/test/results/clientpositive/llap/jdbc_handler.q.out
Line 129 (original), 129 (patched)


Expected?



ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out
Line 231 (original), 231 (patched)


Overflow in data size calculation?



ql/src/test/results/clientpositive/llap/orc_predicate_pushdown.q.out
Line 145 (original), 145 (patched)


Seems to happen only with Gby operator.



ql/src/test/results/clientpositive/llap/sqlmerge.q.out
Line 55 (original), 55 (patched)


Expected change?



ql/src/test/results/clientpositive/llap/subquery_select.q.out
Line 2490 (original), 2490 (patched)


stats change.. expected?



ql/src/test/results/clientpositive/llap/vector_leftsemi_mapjoin.q.out
Line  (original),  (patched)


Expected?



ql/src/test/results/clientpositive/llap/vector_number_compare_projection.q.out
Line 200 (original), 200 (patched)


Overflow.



ql/src/test/results/clientpositive/llap/vectorization_short_regress.q.out
Line 3567 (original), 3567 (patched)


State change expected?


- Ashutosh Chauhan


On Aug. 27, 2017, 11:11 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61165/
> ---
> 
> (Updated Aug. 27, 2017, 11:11 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16811
> https://issues.apache.org/jira/browse/HIVE-16811
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch introduces estimation of statistics if stats doesn't already exist.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0d8d7ae030 
>   itests/src/test/resources/testconfiguration.properties fa6a2aaea0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 22790de209 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  ad29d65abb 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
>   ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
>   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
>   ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
>   ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
> 156be41502 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
>

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-08-27 Thread Vineet Garg


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/
---

(Updated Aug. 27, 2017, 11:11 p.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16811
https://issues.apache.org/jira/browse/HIVE-16811


Repository: hive-git


Description
---

This patch introduces estimation of statistics if stats doesn't already exist.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0d8d7ae030 
  itests/src/test/resources/testconfiguration.properties fa6a2aaea0 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
22790de209 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 ad29d65abb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
  ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
156be41502 
  ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
23f5fcfc76 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
a7c9b3fc41 
  ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
  ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
  ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
  ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
f77c75c232 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
f04519f8c5 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
f15d536945 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
1ec5dc415c 
  ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
  ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 97f8d6b4cc 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 8bc280cedd 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
4b646dcba5 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
ab94c37fed 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
7e3fe4c862 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
054cf1af7e 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
9c37848896 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
21f1b3faff 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
03ff5a6659 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
42e4cb7a5c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
64c10ce953 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
96854a9665 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
b77a879b78 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
9e6053675b 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
13d3accf4c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
7fc90fd19f 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
278c55b3e4 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 649afb502a 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out bac4ae086c 
  ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
  ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out bda8070ba5 
  ql/src/test/results/clientpositive/llap/bucketmapjoin3.q.out ae17c1b707 
  ql/src/test/results/clientpositive/llap/bucketmapjoin4.q.out aab43431e6 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 3a2cbb2b14 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 922f23a29e 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_2.q.out 
e4fed1124b 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_6.q.out 
67e925ad99

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-08-25 Thread Vineet Garg


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/
---

(Updated Aug. 26, 2017, 12:49 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16811
https://issues.apache.org/jira/browse/HIVE-16811


Repository: hive-git


Description
---

This patch introduces estimation of statistics if stats doesn't already exist.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2fb004c0ec 
  itests/src/test/resources/testconfiguration.properties 772113acda 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
22790de209 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 ad29d65abb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java eb02a91953 
  ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out 67d134ba4a 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
156be41502 
  ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
23f5fcfc76 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
a7c9b3fc41 
  ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
  ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
  ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
  ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
f77c75c232 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
f04519f8c5 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
f15d536945 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
1ec5dc415c 
  ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
  ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 97f8d6b4cc 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 8bc280cedd 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
4b646dcba5 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
ab94c37fed 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
7e3fe4c862 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
054cf1af7e 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
9c37848896 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
21f1b3faff 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
03ff5a6659 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
42e4cb7a5c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
64c10ce953 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
96854a9665 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
b77a879b78 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
9e6053675b 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
13d3accf4c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
7fc90fd19f 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
278c55b3e4 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 649afb502a 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out bac4ae086c 
  ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
  ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out bda8070ba5 
  ql/src/test/results/clientpositive/llap/bucketmapjoin3.q.out ae17c1b707 
  ql/src/test/results/clientpositive/llap/bucketmapjoin4.q.out aab43431e6 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 3a2cbb2b14 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 922f23a29e 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_2.q.out 
e4fed1124b 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_6.q.out 
67e925ad99

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-08-23 Thread Vineet Garg

> On Aug. 16, 2017, 10:58 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java
> > Lines 304 (patched)
> > 
> >
> > Can this ever be null ?

I think it could be null for columns which have complex data type

> On Aug. 16, 2017, 10:58 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
> > Lines 227 (patched)
> > 
> >
> > I think we shall intro a config variable here, which can be configured 
> > to
> > a) Throw unchecked exception which is not caught above and fails query: 
> > mostly for testing.
> > b) warning: should be default. Shall print warning about missing stats. 
> > Logic which is there in RelOptHiveTable.
> > c) Throw checked exception: which is caught above and turns off cbo. 
> > This will be useful for cloud setting, where any FS interaction is slow and 
> > not worth slowing down compilation.

Regarding c) Problem with adding this functionality is that this code is shared 
among logical planning and physical planning. Throwing checked exception will 
work for logical planning only but not physical planning and I don't think we 
want to throw an exception during physical planning when there is nothing to 
fall back on. Regarding a) I am not sure what are the scenarios where we will 
ever need this for testing. I am concerned about making this code more 
complicated by introducing too many configs/code paths. Regarding b) I agree we 
should have a way to warn users about missing statistics. I'll add code to show 
warnings about missing stats.

> On Aug. 16, 2017, 10:58 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
> > Lines 295-296 (original), 371-374 (patched)
> > 
> >
> > Can be merged.

Sorry not sure what you mean by that. Can you explain what could be merged?

> On Aug. 16, 2017, 10:58 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
> > Lines 866 (patched)
> > 
> >
> > Same config as described above can be used here too.

See above response.

> On Aug. 16, 2017, 10:58 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
> > Lines 868 (patched)
> > 
> >
> > Also, colstate should be set to NONE in this case.

ColStatistics do not keep any state, Statistics keep the state which is dervied 
based on if there is ColStatistics object or not. 
So we can't mark the state to NONE here. So we will to go through methods which 
call this function and set the state there. I'll update the code

> On Aug. 16, 2017, 10:58 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
> > Lines 914 (patched)
> > 
> >
> > its an estimate anyway, may use hive.stats.max.variable.length to 
> > return this.

The estimation is not done assuming that hive doesn't support computing column 
statistics for complex data type. Let me add estimation for complex data types 
in another patch because I don't know what are the repercussions of adding 
estimation for complex data types. I don't think it is trivial either and I 
would like to do it properly.

> On Aug. 16, 2017, 10:58 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
> > Lines 1220 (patched)
> > 
> >
> > Better is to traverse inside this complex type and do recursive call of 
> > this method. 
> > Or, leave a TODO and use hive.stats.max.variable.length, 
> > hive.stats.list.num.entries , hive.stats.map.num.entries configs to make an 
> > estimate.

Since we just get the name of datatype it is not possible to traverse within 
types here. I'll see if we can add an estimate for complex data types

- Vineet

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/#review183073
---

On Aug. 8, 2017, 8:10 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61165/
> ---
> 
> (Updated Aug. 8, 2017, 8:10 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16811
> https://issues.apache.org/jira/browse/HIVE-16811
> 
>

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-08-16 Thread Ashutosh Chauhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/#review183080
---



- Ashutosh Chauhan


On Aug. 8, 2017, 8:10 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61165/
> ---
> 
> (Updated Aug. 8, 2017, 8:10 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16811
> https://issues.apache.org/jira/browse/HIVE-16811
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch introduces estimation of statistics if stats doesn't already exist.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c16880ed2b 
>   itests/src/test/resources/testconfiguration.properties 772113acda 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 1d49568c8c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  6cb0559ea0 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 3b9ab41bed 
>   ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
>   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
>   ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 67d134ba4a 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
>   ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
> 156be41502 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
>   ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
> 23f5fcfc76 
>   
> ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
> a7c9b3fc41 
>   ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
>   ql/src/test/results/clientpositive/filter_join_breaktask.q.out 8f9b6363f4 
>   ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
>   ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
>   ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
> f77c75c232 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
> f04519f8c5 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
> f15d536945 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
> 1ec5dc415c 
>   ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
>   ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
>   ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 
> 97f8d6b4cc 
>   ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 
> 8bc280cedd 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
> 4b646dcba5 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
> ab94c37fed 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
> 7e3fe4c862 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
> 054cf1af7e 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
> 9c37848896 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
> 21f1b3faff 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
> 03ff5a6659 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
> 42e4cb7a5c 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
> 64c10ce953 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
> 96854a9665 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
> b77a879b78 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
> 9e6053675b 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
> 13d3accf4c 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
> 7fc90fd19f 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
> 278c55b3e4 
>   ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 
> 649afb502a 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out 
> bac4ae086c 
>   ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
>   ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out bda8070ba5 
>

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-08-16 Thread Ashutosh Chauhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/#review183073
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 1653-1655 (patched)


These should be float to allow 0.5 values which people may want to use.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java
Lines 219-221 (original), 212-214 (patched)


We will never hit this now. We use this to generate warnings. We shall move 
this logic to StatsUtils now.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java
Lines 304 (patched)


Can this ever be null ?



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Lines 227 (patched)


I think we shall intro a config variable here, which can be configured to
a) Throw unchecked exception which is not caught above and fails query: 
mostly for testing.
b) warning: should be default. Shall print warning about missing stats. 
Logic which is there in RelOptHiveTable.
c) Throw checked exception: which is caught above and turns off cbo. This 
will be useful for cloud setting, where any FS interaction is slow and not 
worth slowing down compilation.



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Lines 295-296 (original), 371-374 (patched)


Can be merged.



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Lines 866 (patched)


Same config as described above can be used here too.



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Lines 868 (patched)


Also, colstate should be set to NONE in this case.



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Lines 870 (patched)


May use config variable hive.stats.max.variable.length



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Lines 914 (patched)


its an estimate anyway, may use hive.stats.max.variable.length to return 
this.



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
Lines 1220 (patched)


Better is to traverse inside this complex type and do recursive call of 
this method. 
Or, leave a TODO and use hive.stats.max.variable.length, 
hive.stats.list.num.entries , hive.stats.map.num.entries configs to make an 
estimate.


- Ashutosh Chauhan


On Aug. 8, 2017, 8:10 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61165/
> ---
> 
> (Updated Aug. 8, 2017, 8:10 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16811
> https://issues.apache.org/jira/browse/HIVE-16811
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch introduces estimation of statistics if stats doesn't already exist.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c16880ed2b 
>   itests/src/test/resources/testconfiguration.properties 772113acda 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 1d49568c8c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  6cb0559ea0 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 3b9ab41bed 
>   ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
>   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
>   ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 67d134ba4a 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
>   ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
> 156be41502 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
>   ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
> 23f5fcfc76 
>   
> ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
> a7c9b3fc41 
>

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-08-08 Thread Vineet Garg


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/
---

(Updated Aug. 8, 2017, 8:10 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Various fixes


Bugs: HIVE-16811
https://issues.apache.org/jira/browse/HIVE-16811


Repository: hive-git


Description
---

This patch introduces estimation of statistics if stats doesn't already exist.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c16880ed2b 
  itests/src/test/resources/testconfiguration.properties 772113acda 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
1d49568c8c 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 6cb0559ea0 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 3b9ab41bed 
  ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out 67d134ba4a 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
156be41502 
  ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
23f5fcfc76 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
a7c9b3fc41 
  ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
  ql/src/test/results/clientpositive/filter_join_breaktask.q.out 8f9b6363f4 
  ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
  ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
  ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
f77c75c232 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
f04519f8c5 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
f15d536945 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
1ec5dc415c 
  ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
  ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 97f8d6b4cc 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 8bc280cedd 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
4b646dcba5 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
ab94c37fed 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
7e3fe4c862 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
054cf1af7e 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
9c37848896 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
21f1b3faff 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
03ff5a6659 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
42e4cb7a5c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
64c10ce953 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
96854a9665 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
b77a879b78 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
9e6053675b 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
13d3accf4c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
7fc90fd19f 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
278c55b3e4 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 649afb502a 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out bac4ae086c 
  ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
  ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out bda8070ba5 
  ql/src/test/results/clientpositive/llap/bucketmapjoin3.q.out ae17c1b707 
  ql/src/test/results/clientpositive/llap/bucketmapjoin4.q.out aab43431e6 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 3a2cbb2b14 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 922f23a29e 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_2.q.out 
e4fed1124b 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_6.q.out 
67e925ad99

Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

2017-07-26 Thread Vineet Garg


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16811
https://issues.apache.org/jira/browse/HIVE-16811


Repository: hive-git


Description
---

This patch introduces estimation of statistics if stats doesn't already exist.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 05f6cc9592 
  itests/src/test/resources/testconfiguration.properties f66e19be3e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
1d49568c8c 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 3b9ab41bed 
  ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
156be41502 
  ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
  ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
  ql/src/test/results/clientpositive/filter_join_breaktask.q.out 8f9b6363f4 
  ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
  ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
  ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
f77c75c232 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
f04519f8c5 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
f15d536945 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
1ec5dc415c 
  ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
  ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 8bc280cedd 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
7e3fe4c862 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
278c55b3e4 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 649afb502a 
  ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 922f23a29e 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_7.q.out 
8691d0d34e 
  
ql/src/test/results/clientpositive/llap/columnStatsUpdateForStatsOptimizer_1.q.out
 6d941fd4f2 
  ql/src/test/results/clientpositive/llap/column_access_stats.q.out c56c818617 
  ql/src/test/results/clientpositive/llap/column_table_stats.q.out c7726fec30 
  ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 
3de40ce871 
  ql/src/test/results/clientpositive/llap/dynamic_partition_pruning_2.q.out 
e78429fc7f 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out 
1dc9ed5a8c 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization2.q.out 
90756c6776 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 
788854aa04 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out b36fe2a83f 
  ql/src/test/results/clientpositive/llap/filter_join_breaktask.q.out 
c5cdb77afc 
  ql/src/test/results/clientpositive/llap/join_reordering_no_stats.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/llap_partitioned.q.out d35501e82b 
  ql/src/test/results/clientpositive/llap/mergejoin.q.out 9df5e42df2 
  ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out 
594e7e7fd6 
  ql/src/test/results/clientpositive/llap/partition_shared_scan.q.out 
bd3d247d48 
  ql/src/test/results/clientpositive/llap/ppd_union_view.q.out 543d17630a 
  ql/src/test/results/clientpositive/llap/reduce_deduplicate.q.out d74e4c7c78 
  ql/src/test/results/clientpositive/llap/sample10.q.out a3da78754f 
  ql/src/test/results/clientpositive/llap/schema_evol_orc_nonvec_part.q.out 
0ebb378d08 
  
ql/src/test/results/clientpositive/llap/schema_evol_orc_nonvec_part_all_complex.q.out
 22c7745345 
  
ql/src/test/results/clientpositive/llap/schema_evol_orc_nonvec_part_all_primitive.q.out
 4e94322705 
  ql/src/test/results/clientpositive/llap/schema_evol_orc_vec_part.q.out 
c6fc7b5dc6 
  
ql/src/test/results/clientpositive/llap/schema_evol_orc_vec_part_all_complex.q.out
 f57481dcc1 
  
ql/src/test/results/clientpositive/llap/schema_evol_orc_vec_part_all_primitive.q.out
 3a5232ad2e 
  ql/src/test/results/clientpositive/llap/schema_evol_text_nonvec_part.q.out 
8daee2883f 
  
ql/src/test/results/clientpositive/llap/schema_evol_text_nonvec_part_all_complex.q.out
 12c7db4af8 
  
ql/src/test/results/clientpositive/llap/schema_evol_text_nonvec_part_all_primitive.q.out
 757ea3a6c0

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

19 matches

Site Navigation

Mail list logo

Footer information