Re: Review Request 56810: Compute table stats when user computes column stats

2017-03-07 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56810/
---

(Updated March 8, 2017, 5:52 a.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-15903


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 9babc77 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 3e749eb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 7f5fdff 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java c13a404 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ee9101b 
  ql/src/test/queries/clientpositive/column_table_stats.q PRE-CREATION 
  ql/src/test/queries/clientpositive/column_table_stats_orc.q PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/alter_table_invalidate_column_stats.q.out
 c1c5f62 
  
ql/src/test/results/clientpositive/llap/columnStatsUpdateForStatsOptimizer_1.q.out
 80ccddd 
  ql/src/test/results/clientpositive/llap/column_table_stats.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/column_table_stats_orc.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/columnstats_part_coltype.q.out 
3e28e58 
  ql/src/test/results/clientpositive/llap/deleteAnalyze.q.out ba14835 
  ql/src/test/results/clientpositive/llap/drop_partition_with_stats.q.out 
c6ab40d 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out e898111 
  
ql/src/test/results/clientpositive/llap/extrapolate_part_stats_partial_ndv.q.out
 c9b8cfd 
  ql/src/test/results/clientpositive/llap/llap_stats.q.out f6921f1 
  ql/src/test/results/clientpositive/llap/llapdecider.q.out d514f42 
  ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out c8190bd 
  
ql/src/test/results/clientpositive/llap/metadata_only_queries_with_filters.q.out
 6dea3e0 
  ql/src/test/results/clientpositive/llap/schema_evol_stats.q.out 906e5e8 
  ql/src/test/results/clientpositive/llap/schema_evol_text_vec_table.q.out 
5bfe159 
  ql/src/test/results/clientpositive/llap/special_character_in_tabnames_1.q.out 
d01b373 
  ql/src/test/results/clientpositive/llap/stats_only_null.q.out c905ceb 
  ql/src/test/results/clientpositive/llap/union_remove_26.q.out 18b9aa5 
  ql/src/test/results/clientpositive/llap/vector_outer_join1.q.out 072399e 
  ql/src/test/results/clientpositive/llap/vector_outer_join2.q.out d79c71b 
  ql/src/test/results/clientpositive/llap/vector_outer_join3.q.out dbbfd34 
  ql/src/test/results/clientpositive/llap/vector_outer_join4.q.out ffce9e6 
  ql/src/test/results/clientpositive/llap/vector_outer_join5.q.out 4f25253 
  
ql/src/test/results/clientpositive/llap/vectorized_dynamic_semijoin_reduction.q.out
 eb0f405 
  
ql/src/test/results/clientpositive/llap/vectorized_dynamic_semijoin_reduction2.q.out
 b369e7c 
  ql/src/test/results/clientpositive/perf/query14.q.out 9821180 
  ql/src/test/results/clientpositive/tez/explainanalyze_3.q.out 20c330a 
  ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out ee9affb 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out 74e4693 


Diff: https://reviews.apache.org/r/56810/diff/8/

Changes: https://reviews.apache.org/r/56810/diff/7-8/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 56810: Compute table stats when user computes column stats

2017-03-07 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56810/
---

(Updated March 8, 2017, 5:51 a.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

with golden file updates


Repository: hive-git


Description
---

HIVE-15903


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 9babc77 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 3e749eb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 7f5fdff 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java c13a404 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ee9101b 
  ql/src/test/queries/clientpositive/column_table_stats.q PRE-CREATION 
  ql/src/test/queries/clientpositive/column_table_stats_orc.q PRE-CREATION 
  
ql/src/test/results/clientpositive/llap/alter_table_invalidate_column_stats.q.out
 c1c5f62 
  
ql/src/test/results/clientpositive/llap/columnStatsUpdateForStatsOptimizer_1.q.out
 80ccddd 
  ql/src/test/results/clientpositive/llap/column_table_stats.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/column_table_stats_orc.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/columnstats_part_coltype.q.out 
3e28e58 
  ql/src/test/results/clientpositive/llap/deleteAnalyze.q.out ba14835 
  ql/src/test/results/clientpositive/llap/drop_partition_with_stats.q.out 
c6ab40d 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out e898111 
  
ql/src/test/results/clientpositive/llap/extrapolate_part_stats_partial_ndv.q.out
 c9b8cfd 
  ql/src/test/results/clientpositive/llap/llap_stats.q.out f6921f1 
  ql/src/test/results/clientpositive/llap/llapdecider.q.out d514f42 
  ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out c8190bd 
  
ql/src/test/results/clientpositive/llap/metadata_only_queries_with_filters.q.out
 6dea3e0 
  ql/src/test/results/clientpositive/llap/schema_evol_stats.q.out 906e5e8 
  ql/src/test/results/clientpositive/llap/schema_evol_text_vec_table.q.out 
5bfe159 
  ql/src/test/results/clientpositive/llap/special_character_in_tabnames_1.q.out 
d01b373 
  ql/src/test/results/clientpositive/llap/stats_only_null.q.out c905ceb 
  ql/src/test/results/clientpositive/llap/union_remove_26.q.out 18b9aa5 
  ql/src/test/results/clientpositive/llap/vector_outer_join1.q.out 072399e 
  ql/src/test/results/clientpositive/llap/vector_outer_join2.q.out d79c71b 
  ql/src/test/results/clientpositive/llap/vector_outer_join3.q.out dbbfd34 
  ql/src/test/results/clientpositive/llap/vector_outer_join4.q.out ffce9e6 
  ql/src/test/results/clientpositive/llap/vector_outer_join5.q.out 4f25253 
  
ql/src/test/results/clientpositive/llap/vectorized_dynamic_semijoin_reduction.q.out
 eb0f405 
  
ql/src/test/results/clientpositive/llap/vectorized_dynamic_semijoin_reduction2.q.out
 b369e7c 
  ql/src/test/results/clientpositive/perf/query14.q.out 9821180 
  ql/src/test/results/clientpositive/tez/explainanalyze_3.q.out 20c330a 
  ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out ee9affb 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out 74e4693 


Diff: https://reviews.apache.org/r/56810/diff/7/

Changes: https://reviews.apache.org/r/56810/diff/6-7/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 56810: Compute table stats when user computes column stats

2017-03-04 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56810/
---

(Updated March 4, 2017, 7:41 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

address ashutosh's comments.


Repository: hive-git


Description
---

HIVE-15903


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties b01ebd8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 3e749eb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 7f5fdff 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java c13a404 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 0872e53 
  ql/src/test/queries/clientpositive/column_table_stats.q PRE-CREATION 
  ql/src/test/queries/clientpositive/column_table_stats_orc.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/column_table_stats.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/column_table_stats_orc.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query14.q.out 9821180 
  ql/src/test/results/clientpositive/tez/explainanalyze_3.q.out 20c330a 
  ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out ee9affb 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out 74e4693 


Diff: https://reviews.apache.org/r/56810/diff/6/

Changes: https://reviews.apache.org/r/56810/diff/5-6/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 56810: Compute table stats when user computes column stats

2017-03-03 Thread pengcheng xiong 


> On March 3, 2017, 11:54 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java
> > Lines 170 (patched)
> > 
> >
> > tezWork.getAllWork().get(0).setGatheringStats(true);

This will throw NPE. Note that ProcessAnalyzeTable will be fired when it 
matches TS while GenTezUtils will be fined when it matches RS. 
ProcessAnalyzeTable will be fired before any map work is generated. Thus, it 
will not have any work at this place.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56810/#review167896
---


On March 2, 2017, 10:37 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56810/
> ---
> 
> (Updated March 2, 2017, 10:37 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15903
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties b01ebd8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
> 3e749eb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 7f5fdff 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
> c13a404 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 0872e53 
>   ql/src/test/queries/clientpositive/column_table_stats.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/column_table_stats_orc.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/column_table_stats.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/column_table_stats_orc.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query14.q.out 9821180 
>   ql/src/test/results/clientpositive/tez/explainanalyze_3.q.out 20c330a 
>   ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out ee9affb 
>   ql/src/test/results/clientpositive/tez/explainuser_3.q.out 74e4693 
> 
> 
> Diff: https://reviews.apache.org/r/56810/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 56810: Compute table stats when user computes column stats

2017-03-03 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56810/#review167896
---



Can you also add golden file updates in RB, wanna go through those changes as 
well.


ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java
Lines 202-205 (patched)


This should be done in ProcessAnalyzeTable.



ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java
Lines 170 (patched)


tezWork.getAllWork().get(0).setGatheringStats(true);



ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java
Lines 189 (patched)


this is not needed.. since at this point partitions has already been read.



ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java
Lines 190 (patched)


LOG.debug()



ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java
Lines 198 (patched)


Better comment: For ORC, there is no Tez Job.



ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java
Lines 212 (patched)


Remove this. Both incorrect as well as misplaced.


- Ashutosh Chauhan


On March 2, 2017, 10:37 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56810/
> ---
> 
> (Updated March 2, 2017, 10:37 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15903
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties b01ebd8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
> 3e749eb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 7f5fdff 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
> c13a404 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 0872e53 
>   ql/src/test/queries/clientpositive/column_table_stats.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/column_table_stats_orc.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/column_table_stats.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/column_table_stats_orc.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query14.q.out 9821180 
>   ql/src/test/results/clientpositive/tez/explainanalyze_3.q.out 20c330a 
>   ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out ee9affb 
>   ql/src/test/results/clientpositive/tez/explainuser_3.q.out 74e4693 
> 
> 
> Diff: https://reviews.apache.org/r/56810/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 56810: Compute table stats when user computes column stats

2017-03-02 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56810/
---

(Updated March 2, 2017, 10:37 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-15903


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties b01ebd8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 3e749eb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 7f5fdff 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java c13a404 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 0872e53 
  ql/src/test/queries/clientpositive/column_table_stats.q PRE-CREATION 
  ql/src/test/queries/clientpositive/column_table_stats_orc.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/column_table_stats.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/column_table_stats_orc.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query14.q.out 9821180 
  ql/src/test/results/clientpositive/tez/explainanalyze_3.q.out 20c330a 
  ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out ee9affb 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out 74e4693 


Diff: https://reviews.apache.org/r/56810/diff/5/

Changes: https://reviews.apache.org/r/56810/diff/4-5/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 56810: Compute table stats when user computes column stats

2017-03-01 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56810/
---

(Updated March 2, 2017, 6:26 a.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-15903


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties b01ebd8 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 0e67ea6 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 3e749eb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
93b8183 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java c13a404 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 0872e53 
  ql/src/test/queries/clientpositive/column_table_stats.q PRE-CREATION 
  ql/src/test/queries/clientpositive/exec_parallel_column_stats.q ceacc24 
  ql/src/test/results/clientpositive/alter_table_update_status.q.out a3c4f1a 
  ql/src/test/results/clientpositive/analyze_tbl_date.q.out a0cdbca 
  ql/src/test/results/clientpositive/analyze_tbl_part.q.out ed90b6f 
  ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 32644dc 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out a8e4854 
  ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out 31c4ed1 
  ql/src/test/results/clientpositive/annotate_stats_join.q.out 5d4fe6c 
  ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out b4d46d2 
  ql/src/test/results/clientpositive/annotate_stats_limit.q.out ea181cb 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1ab 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f 
  ql/src/test/results/clientpositive/annotate_stats_union.q.out 059f261 
  ql/src/test/results/clientpositive/autoColumnStats_3.q.out dca158b 
  ql/src/test/results/clientpositive/avro_decimal.q.out 64e65ca 
  ql/src/test/results/clientpositive/avro_decimal_native.q.out cebc342 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
f260f03 
  ql/src/test/results/clientpositive/cbo_rp_auto_join1.q.out fe4bc4f 
  ql/src/test/results/clientpositive/colstats_all_nulls.q.out f67f81b 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
af21343 
  ql/src/test/results/clientpositive/columnstats_partlvl.q.out 5c9f901 
  ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out 08fb42e 
  ql/src/test/results/clientpositive/columnstats_quoting.q.out 52e3538 
  ql/src/test/results/clientpositive/columnstats_tbllvl.q.out 60652e0 
  ql/src/test/results/clientpositive/compustat_avro.q.out 50c03bd 
  ql/src/test/results/clientpositive/compute_stats_date.q.out 73f08ad 
  ql/src/test/results/clientpositive/constGby.q.out fd8ecc2 
  ql/src/test/results/clientpositive/constant_prop_3.q.out ecf6e57 
  ql/src/test/results/clientpositive/decimal_stats.q.out cb02f76 
  ql/src/test/results/clientpositive/deleteAnalyze.q.out 4382522 
  ql/src/test/results/clientpositive/display_colstats_tbllvl.q.out e10edb4 
  ql/src/test/results/clientpositive/distinct_stats.q.out 73b4add 
  ql/src/test/results/clientpositive/drop_table_with_stats.q.out 52aa10a 
  ql/src/test/results/clientpositive/exec_parallel_column_stats.q.out f256ec1 
  ql/src/test/results/clientpositive/extrapolate_part_stats_date.q.out b12d3a1 
  
ql/src/test/results/clientpositive/llap/alter_table_invalidate_column_stats.q.out
 85d7dc4 
  
ql/src/test/results/clientpositive/llap/columnStatsUpdateForStatsOptimizer_1.q.out
 80ccddd 
  ql/src/test/results/clientpositive/llap/column_table_stats.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/columnstats_part_coltype.q.out 
3e28e58 
  ql/src/test/results/clientpositive/llap/deleteAnalyze.q.out ba14835 
  ql/src/test/results/clientpositive/llap/drop_partition_with_stats.q.out 
c6ab40d 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out e898111 
  ql/src/test/results/clientpositive/llap/llap_stats.q.out f6921f1 
  ql/src/test/results/clientpositive/llap/llapdecider.q.out d514f42 
  ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out c8190bd 
  ql/src/test/results/clientpositive/llap/schema_evol_stats.q.out be8ca4a 
  ql/src/test/results/clientpositive/llap/special_character_in_tabnames_1.q.out 
d01b373 
  ql/src/test/results/clientpositive/llap/stats_only_null.q.out c905ceb 
  ql/src/test/results/clientpositive/llap/union_remove_26.q.out 18b9aa5 
  ql/src/test/results/clientpositive/llap/vector_outer_join1.q.out 072399e 
  ql/src/test/results/clientpositive/llap/vector_outer_join2.q.out d79c71b 
  ql/src/test/results/clientpositive/llap/vector_outer_join3.q.out dbbfd34 
  ql/src/test/results/clientpositive/llap/vector_outer_join4.q.out ffce9e6

Re: Review Request 56810: Compute table stats when user computes column stats

2017-02-24 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56810/#review166791
---




ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (line 10250)


In collect column stats case, isAnalyzeCommand is false and analyzerewrite 
is not null



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (line 10257)


It is null at this point.



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (line 10262)


ok



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (line 10320)


ok



ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java (lines 307 - 343)


So you mean you only want this new feature to work on Tez, not other 
engines like MR or Spark?


- pengcheng xiong


On Feb. 23, 2017, 6:52 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56810/
> ---
> 
> (Updated Feb. 23, 2017, 6:52 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15903
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
> 3e749eb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
> 93b8183 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 2430811 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 5f9ccc8 
>   ql/src/test/queries/clientpositive/column_table_stats.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/exec_parallel_column_stats.q ceacc24 
>   ql/src/test/results/clientpositive/alter_table_update_status.q.out a3c4f1a 
>   ql/src/test/results/clientpositive/analyze_tbl_date.q.out a0cdbca 
>   ql/src/test/results/clientpositive/analyze_tbl_part.q.out ed90b6f 
>   ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
> 32644dc 
>   ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef 
>   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out a8e4854 
>   ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out 31c4ed1 
>   ql/src/test/results/clientpositive/annotate_stats_join.q.out 5d4fe6c 
>   ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out b4d46d2 
>   ql/src/test/results/clientpositive/annotate_stats_limit.q.out ea181cb 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1ab 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f 
>   ql/src/test/results/clientpositive/annotate_stats_union.q.out 059f261 
>   ql/src/test/results/clientpositive/autoColumnStats_3.q.out dca158b 
>   ql/src/test/results/clientpositive/avro_decimal.q.out 64e65ca 
>   ql/src/test/results/clientpositive/avro_decimal_native.q.out cebc342 
>   ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
> f260f03 
>   ql/src/test/results/clientpositive/cbo_rp_auto_join1.q.out fe4bc4f 
>   ql/src/test/results/clientpositive/colstats_all_nulls.q.out f67f81b 
>   
> ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
> af21343 
>   ql/src/test/results/clientpositive/column_table_stats.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/columnstats_partlvl.q.out 5c9f901 
>   ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out 08fb42e 
>   ql/src/test/results/clientpositive/columnstats_quoting.q.out 52e3538 
>   ql/src/test/results/clientpositive/columnstats_tbllvl.q.out 60652e0 
>   ql/src/test/results/clientpositive/compustat_avro.q.out 50c03bd 
>   ql/src/test/results/clientpositive/compute_stats_date.q.out 73f08ad 
>   ql/src/test/results/clientpositive/constGby.q.out fd8ecc2 
>   ql/src/test/results/clientpositive/constant_prop_3.q.out ecf6e57 
>   ql/src/test/results/clientpositive/decimal_stats.q.out cb02f76 
>   ql/src/test/results/clientpositive/deleteAnalyze.q.out 4382522 
>   ql/src/test/results/clientpositive/display_colstats_tbllvl.q.out e10edb4 
>   ql/src/test/results/clientpositive/distinct_stats.q.out 73b4add 
>   ql/src/test/results/clientpositive/drop_table_with_stats.q.out 52aa10a 
>   ql/src/test/results/clientpositive/exec_parallel_column_stats.q.out f256ec1 
>   ql/src/test/results/clientpositive/extrapolate_part_stats_date.q.out 
> b12d3a1 
>   
> ql/src/test/results/clientpositive/llap/alter_table_invalidate_column_stats.q.out
>  85d7dc4 
>   
>

Re: Review Request 56810: Compute table stats when user computes column stats

2017-02-24 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56810/#review166776
---




ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
(line 72)


Why is that. Lets say you have 
Table t1 has 2 partitions p1=1 and p1=2 and user runs
analyze table t1 partition (p1=2) for columns

In this case you can still collect basic stats (numrows etc.) for p1=2. 
Basic stats wont be collected for p1=1 but that should be fine.



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (line 10250)


Won't analyzeRewrite always be null when isAnyalzeCommand() is false. If 
so, this change is not needed.



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (line 10257)


tblSpec cant be null at this point. It will already be constructed with all 
the info needed during getMetaData(). Can remove this reconstruction of tblSpec.



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (line 10262)


LOG.debug()



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (line 10320)


LOG.debug ("Adding partition: {}", partn);



ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java (lines 307 - 343)


This logic of generating tasks should be migrated to 
TezCompiler::ProcessAnalyzeTable() 
Currently its breaking the abstraction.


- Ashutosh Chauhan


On Feb. 23, 2017, 6:52 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56810/
> ---
> 
> (Updated Feb. 23, 2017, 6:52 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15903
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
> 3e749eb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
> 93b8183 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 2430811 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 5f9ccc8 
>   ql/src/test/queries/clientpositive/column_table_stats.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/exec_parallel_column_stats.q ceacc24 
>   ql/src/test/results/clientpositive/alter_table_update_status.q.out a3c4f1a 
>   ql/src/test/results/clientpositive/analyze_tbl_date.q.out a0cdbca 
>   ql/src/test/results/clientpositive/analyze_tbl_part.q.out ed90b6f 
>   ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
> 32644dc 
>   ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef 
>   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out a8e4854 
>   ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out 31c4ed1 
>   ql/src/test/results/clientpositive/annotate_stats_join.q.out 5d4fe6c 
>   ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out b4d46d2 
>   ql/src/test/results/clientpositive/annotate_stats_limit.q.out ea181cb 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1ab 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f 
>   ql/src/test/results/clientpositive/annotate_stats_union.q.out 059f261 
>   ql/src/test/results/clientpositive/autoColumnStats_3.q.out dca158b 
>   ql/src/test/results/clientpositive/avro_decimal.q.out 64e65ca 
>   ql/src/test/results/clientpositive/avro_decimal_native.q.out cebc342 
>   ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
> f260f03 
>   ql/src/test/results/clientpositive/cbo_rp_auto_join1.q.out fe4bc4f 
>   ql/src/test/results/clientpositive/colstats_all_nulls.q.out f67f81b 
>   
> ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
> af21343 
>   ql/src/test/results/clientpositive/column_table_stats.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/columnstats_partlvl.q.out 5c9f901 
>   ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out 08fb42e 
>   ql/src/test/results/clientpositive/columnstats_quoting.q.out 52e3538 
>   ql/src/test/results/clientpositive/columnstats_tbllvl.q.out 60652e0 
>   ql/src/test/results/clientpositive/compustat_avro.q.out 50c03bd 
>   ql/src/test/results/clientpositive/compute_stats_date.q.out 73f08ad 
>   ql/src/test/results/clientpositive/constGby.q.out fd8ecc2 
>   ql/src/test/results/clientpositive/constant_prop_3.q.out ecf6e57 
>

Re: Review Request 56810: Compute table stats when user computes column stats

2017-02-23 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56810/
---

(Updated Feb. 23, 2017, 6:52 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

update golden files


Repository: hive-git


Description
---

HIVE-15903


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 3e749eb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
93b8183 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 2430811 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 5f9ccc8 
  ql/src/test/queries/clientpositive/column_table_stats.q PRE-CREATION 
  ql/src/test/queries/clientpositive/exec_parallel_column_stats.q ceacc24 
  ql/src/test/results/clientpositive/alter_table_update_status.q.out a3c4f1a 
  ql/src/test/results/clientpositive/analyze_tbl_date.q.out a0cdbca 
  ql/src/test/results/clientpositive/analyze_tbl_part.q.out ed90b6f 
  ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 32644dc 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out a8e4854 
  ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out 31c4ed1 
  ql/src/test/results/clientpositive/annotate_stats_join.q.out 5d4fe6c 
  ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out b4d46d2 
  ql/src/test/results/clientpositive/annotate_stats_limit.q.out ea181cb 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1ab 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f 
  ql/src/test/results/clientpositive/annotate_stats_union.q.out 059f261 
  ql/src/test/results/clientpositive/autoColumnStats_3.q.out dca158b 
  ql/src/test/results/clientpositive/avro_decimal.q.out 64e65ca 
  ql/src/test/results/clientpositive/avro_decimal_native.q.out cebc342 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
f260f03 
  ql/src/test/results/clientpositive/cbo_rp_auto_join1.q.out fe4bc4f 
  ql/src/test/results/clientpositive/colstats_all_nulls.q.out f67f81b 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
af21343 
  ql/src/test/results/clientpositive/column_table_stats.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/columnstats_partlvl.q.out 5c9f901 
  ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out 08fb42e 
  ql/src/test/results/clientpositive/columnstats_quoting.q.out 52e3538 
  ql/src/test/results/clientpositive/columnstats_tbllvl.q.out 60652e0 
  ql/src/test/results/clientpositive/compustat_avro.q.out 50c03bd 
  ql/src/test/results/clientpositive/compute_stats_date.q.out 73f08ad 
  ql/src/test/results/clientpositive/constGby.q.out fd8ecc2 
  ql/src/test/results/clientpositive/constant_prop_3.q.out ecf6e57 
  ql/src/test/results/clientpositive/decimal_stats.q.out cb02f76 
  ql/src/test/results/clientpositive/deleteAnalyze.q.out 4382522 
  ql/src/test/results/clientpositive/display_colstats_tbllvl.q.out e10edb4 
  ql/src/test/results/clientpositive/distinct_stats.q.out 73b4add 
  ql/src/test/results/clientpositive/drop_table_with_stats.q.out 52aa10a 
  ql/src/test/results/clientpositive/exec_parallel_column_stats.q.out f256ec1 
  ql/src/test/results/clientpositive/extrapolate_part_stats_date.q.out b12d3a1 
  
ql/src/test/results/clientpositive/llap/alter_table_invalidate_column_stats.q.out
 85d7dc4 
  
ql/src/test/results/clientpositive/llap/columnStatsUpdateForStatsOptimizer_1.q.out
 80ccddd 
  ql/src/test/results/clientpositive/llap/columnstats_part_coltype.q.out 
3e28e58 
  ql/src/test/results/clientpositive/llap/deleteAnalyze.q.out ba14835 
  ql/src/test/results/clientpositive/llap/drop_partition_with_stats.q.out 
c6ab40d 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 67f960a 
  ql/src/test/results/clientpositive/llap/llap_stats.q.out f6921f1 
  ql/src/test/results/clientpositive/llap/llapdecider.q.out d514f42 
  ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out c8190bd 
  ql/src/test/results/clientpositive/llap/schema_evol_stats.q.out be8ca4a 
  ql/src/test/results/clientpositive/llap/special_character_in_tabnames_1.q.out 
d01b373 
  ql/src/test/results/clientpositive/llap/stats_only_null.q.out c905ceb 
  ql/src/test/results/clientpositive/llap/union_remove_26.q.out 18b9aa5 
  ql/src/test/results/clientpositive/llap/vector_outer_join1.q.out 2ea8c8f 
  ql/src/test/results/clientpositive/llap/vector_outer_join2.q.out 2077dce 
  ql/src/test/results/clientpositive/llap/vector_outer_join3.q.out dbbfd34 
  ql/src/test/results/clientpositive/llap/vector_outer_join4.q.out ffce9e6 
  ql/src/test/results/clientpositive/llap/vector_outer_join5.q.out 4f25253

Re: Review Request 56810: Compute table stats when user computes column stats

2017-02-22 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56810/
---

(Updated Feb. 23, 2017, 2 a.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-15903


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 3e749eb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
93b8183 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 2430811 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 5f9ccc8 
  ql/src/test/queries/clientpositive/column_table_stats.q PRE-CREATION 
  ql/src/test/queries/clientpositive/exec_parallel_column_stats.q ceacc24 
  ql/src/test/results/clientpositive/alter_table_update_status.q.out a3c4f1a 
  ql/src/test/results/clientpositive/analyze_tbl_date.q.out a0cdbca 
  ql/src/test/results/clientpositive/analyze_tbl_part.q.out ed90b6f 
  ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 32644dc 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out a8e4854 
  ql/src/test/results/clientpositive/annotate_stats_groupby2.q.out 31c4ed1 
  ql/src/test/results/clientpositive/annotate_stats_join.q.out 5d4fe6c 
  ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out b4d46d2 
  ql/src/test/results/clientpositive/annotate_stats_limit.q.out ea181cb 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1ab 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f 
  ql/src/test/results/clientpositive/annotate_stats_union.q.out 059f261 
  ql/src/test/results/clientpositive/autoColumnStats_3.q.out dca158b 
  ql/src/test/results/clientpositive/autoColumnStats_4.q.out c7b9b4f 
  ql/src/test/results/clientpositive/avro_decimal.q.out 64e65ca 
  ql/src/test/results/clientpositive/avro_decimal_native.q.out cebc342 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
f260f03 
  ql/src/test/results/clientpositive/cbo_rp_auto_join1.q.out fe4bc4f 
  ql/src/test/results/clientpositive/colstats_all_nulls.q.out f67f81b 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
af21343 
  ql/src/test/results/clientpositive/column_table_stats.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/columnstats_partlvl.q.out 5c9f901 
  ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out 08fb42e 
  ql/src/test/results/clientpositive/columnstats_quoting.q.out 52e3538 
  ql/src/test/results/clientpositive/columnstats_tbllvl.q.out 60652e0 
  ql/src/test/results/clientpositive/compustat_avro.q.out 50c03bd 
  ql/src/test/results/clientpositive/compute_stats_date.q.out 73f08ad 
  ql/src/test/results/clientpositive/constGby.q.out fd8ecc2 
  ql/src/test/results/clientpositive/constant_prop_3.q.out ecf6e57 
  ql/src/test/results/clientpositive/decimal_stats.q.out cb02f76 
  ql/src/test/results/clientpositive/deleteAnalyze.q.out 4382522 
  ql/src/test/results/clientpositive/display_colstats_tbllvl.q.out e10edb4 
  ql/src/test/results/clientpositive/distinct_stats.q.out 73b4add 
  ql/src/test/results/clientpositive/drop_table_with_stats.q.out 52aa10a 
  ql/src/test/results/clientpositive/exec_parallel_column_stats.q.out f256ec1 
  ql/src/test/results/clientpositive/extrapolate_part_stats_date.q.out b12d3a1 
  
ql/src/test/results/clientpositive/llap/alter_table_invalidate_column_stats.q.out
 85d7dc4 
  
ql/src/test/results/clientpositive/llap/columnStatsUpdateForStatsOptimizer_1.q.out
 80ccddd 
  ql/src/test/results/clientpositive/llap/columnstats_part_coltype.q.out 
3e28e58 
  ql/src/test/results/clientpositive/llap/deleteAnalyze.q.out ba14835 
  ql/src/test/results/clientpositive/llap/drop_partition_with_stats.q.out 
c6ab40d 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out 67f960a 
  ql/src/test/results/clientpositive/llap/llap_stats.q.out f6921f1 
  ql/src/test/results/clientpositive/llap/llapdecider.q.out d514f42 
  ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out c8190bd 
  ql/src/test/results/clientpositive/llap/schema_evol_stats.q.out be8ca4a 
  ql/src/test/results/clientpositive/llap/special_character_in_tabnames_1.q.out 
d01b373 
  ql/src/test/results/clientpositive/llap/stats_only_null.q.out c905ceb 
  ql/src/test/results/clientpositive/llap/union_remove_26.q.out 18b9aa5 
  ql/src/test/results/clientpositive/llap/vector_outer_join1.q.out 2ea8c8f 
  ql/src/test/results/clientpositive/llap/vector_outer_join2.q.out 2077dce 
  ql/src/test/results/clientpositive/llap/vector_outer_join3.q.out dbbfd34 
  ql/src/test/results/clientpositive/llap/vector_outer_join4.q.out ffce9e6 
  ql/src/test/results/clientpositive/llap/vector_outer_join5.q.out 4f25253

Review Request 56810: Compute table stats when user computes column stats

2017-02-17 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56810/
---

Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-15903


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 3e749eb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
ff07b42 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f053093 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 5f9ccc8 
  ql/src/test/queries/clientpositive/column_table_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/column_table_stats.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/56810/diff/


Testing
---


Thanks,

pengcheng xiong