Re: Review Request 42960: HIVE-11752

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42960/#review117538
---




ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 513)
<https://reviews.apache.org/r/42960/#comment178749>

I think there is a materialized table lifecycle problem.

1. What if there is already a table by that name (temp/regular)
2. CTE Temp table should be cleaned up at the end of the container query 
(successful/non successful execution)


- John Pullokkaran


On Feb. 1, 2016, 4:36 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42960/
> ---
> 
> (Updated Feb. 1, 2016, 4:36 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, John Pullokkaran, and Navis Ryu.
> 
> 
> Bugs: HIVE-11752
> https://issues.apache.org/jira/browse/HIVE-11752
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Pre-materializing complex CTE queries
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> ff376a853b7d89262157de67e355b22b941aeebe 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 
> 746456bf39d1a5d700b42d3e0f2c0dcd4001e1ad 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
> 4c89812403f65c7d4beae37952dafc42f44c1519 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java 
> 9132a212f8bd3600467c4337a0562b1e64bed908 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> 2e4591313f38d9bd1bfe1b54968c835fdcd53717 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 
> 40c89cb9681567a18181514cbffc497e0b2f937e 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 
> a5217eb2781dba7f1f35a8a1f87e6784c7ea59ef 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 46bf04c738917e2c3d9f186319fea6f6f2811d3f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
> af1ee20a8636523a77b45bf596daddb1330b09d5 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> e2d404b0bd1d3916e498bc4c4076cdf168ef2272 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java 
> e1e3eb23f13591114153ec5810a0d6c28245085d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java 
> ec5ef0eb2be9bde851c70cddbb167a7dc30c78c3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 607c2f32e00d3d62636fd99c8864ed5bcf24b53b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 
> 89897d7e5805e05903181f9c202852a9bfdba552 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 
> 2c6bb217ec880bcc1d4a1cf3c6c891cf779cff52 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java 
> cc462bea07e371629a94e809dba910625cc68508 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 
> ea506fc13f6a69600e850b4cdcb7a62f613af8b7 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestGenTezWork.java 
> be51edc49581f881a4a92b464e0323bf7661b17f 
>   ql/src/test/queries/clientpositive/cte_3.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/cte_4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/cte_mat_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/cte_mat_2.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/cte_mat_3.q PRE-CREATION 
>   ql/src/test/results/clientnegative/analyze1.q.out 
> 589a6ee1bfe9429e5250f5b78202d7aa586b6973 
>   ql/src/test/results/clientnegative/dyn_part1.q.out 
> 62d01fbef8af967474127e2d5f00bd6d2f4858d7 
>   ql/src/test/results/clientpositive/alter_view_as_select.q.out 
> c89c0dc8602716ad4247208cfaefbab1eabec8e6 
>   ql/src/test/results/clientpositive/cte_3.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cte_4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cte_mat_1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cte_mat_2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cte_mat_3.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/tez_union.q.out 
> f5cd30fdaf5794d9320a0ff3a7474840e2dc3a92 
>   ql/src/test/results/clientpositive/tez/tez_union.q.out 
> 8baa5cf6feae021bbf02d60dc5d5c80692160e53 
> 
> Diff: https://reviews.apache.org/r/42960/diff/
> 
> 
> Testing
> ---
> 
> Existing plus additional q tests.
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 43115: HIVE-12924 CBO: Calcite Operator To Hive Operator (Calcite Return Path): TestCliDriver groupby_ppr_multi_distinct.q failure

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43115/#review117552
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveGBOpConvUtil.java
 (line 225)
<https://reviews.apache.org/r/43115/#comment178783>

Instead of walking the expression you could keep a map of inputref to 
exprnode.


- John Pullokkaran


On Feb. 2, 2016, 10:16 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43115/
> ---
> 
> (Updated Feb. 2, 2016, 10:16 p.m.)
> 
> 
> Review request for hive, Jesús Camacho Rodríguez and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> CBO: Calcite Operator To Hive Operator (Calcite Return Path): TestCliDriver 
> groupby_ppr_multi_distinct.q failure
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveGBOpConvUtil.java
>  7fbf8cd 
> 
> Diff: https://reviews.apache.org/r/43115/diff/
> 
> 
> Testing
> ---
> 
> Precommit runs
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 43115: HIVE-12924 CBO: Calcite Operator To Hive Operator (Calcite Return Path): TestCliDriver groupby_ppr_multi_distinct.q failure

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43115/#review117553
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveGBOpConvUtil.java
 (line 730)
<https://reviews.apache.org/r/43115/#comment178784>

Why is this being changed on MapSide RS.
I thought the issue was with distincts on ReduceSide GB when map side is 
turned off.


- John Pullokkaran


On Feb. 2, 2016, 10:16 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43115/
> ---
> 
> (Updated Feb. 2, 2016, 10:16 p.m.)
> 
> 
> Review request for hive, Jesús Camacho Rodríguez and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> CBO: Calcite Operator To Hive Operator (Calcite Return Path): TestCliDriver 
> groupby_ppr_multi_distinct.q failure
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveGBOpConvUtil.java
>  7fbf8cd 
> 
> Diff: https://reviews.apache.org/r/43115/diff/
> 
> 
> Testing
> ---
> 
> Precommit runs
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 41807: further improve user level explain

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41807/#review116704
---


Ship it!




Ship It!

- John Pullokkaran


On Jan. 21, 2016, 7:46 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41807/
> ---
> 
> (Updated Jan. 21, 2016, 7:46 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12709
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Attr.java 
> 5d355d2 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Op.java 
> d0c1037 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Stage.java 
> 455d59f 
>   
> common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParser.java
>  b193fef 
>   
> common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParserUtils.java
>  PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Vertex.java 
> be01b8b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java f48db6a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 
> bc67e5a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java ad34b98 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 3f38f74 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableLikeDesc.java f051712 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java 40a8477 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d04cb78 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 0f2855e 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 7c8eee2 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java eb83fd6 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java 12f01e5 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java 427aac1 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 4b93e7c 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 5bea6fb 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 2f69b7f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceWork.java 1c31962 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/SelectDesc.java e7bbab4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 4e52bac 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 2fdb0a1 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 098aa89 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFQueryInputDef.java 
> ae55741 
>   
> ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PartitionedTableFunctionDef.java
>  46f106c 
>   ql/src/test/queries/clientpositive/explainuser_4.q PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query12.q.out 648d64d 
>   ql/src/test/results/clientpositive/perf/query13.q.out 6113d18 
>   ql/src/test/results/clientpositive/perf/query15.q.out c237828 
>   ql/src/test/results/clientpositive/perf/query17.q.out 0e42d05 
>   ql/src/test/results/clientpositive/perf/query18.q.out 83f21dc 
>   ql/src/test/results/clientpositive/perf/query19.q.out d565839 
>   ql/src/test/results/clientpositive/perf/query20.q.out 40c9a17 
>   ql/src/test/results/clientpositive/perf/query21.q.out 2634d3d 
>   ql/src/test/results/clientpositive/perf/query22.q.out 72935ed 
>   ql/src/test/results/clientpositive/perf/query25.q.out dabc1c5 
>   ql/src/test/results/clientpositive/perf/query26.q.out 84cc238 
>   ql/src/test/results/clientpositive/perf/query27.q.out ac38683 
>   ql/src/test/results/clientpositive/perf/query28.q.out cac9ac3 
>   ql/src/test/results/clientpositive/perf/query29.q.out 8991f35 
>   ql/src/test/results/clientpositive/perf/query3.q.out 871ad74 
>   ql/src/test/results/clientpositive/perf/query31.q.out 909d64c 
>   ql/src/test/results/clientpositive/perf/query32.q.out da2ac8f 
>   ql/src/test/results/clientpositive/perf/query34.q.out 1cb7c28 
>   ql/src/test/results/clientpositive/perf/query39.q.out 9f3e650 
>   ql/src/test/results/clientpositive/perf/query40.q.out b2d6262 
>   ql/src/test/results/clientpositive/perf/query42.q.out 9ede45d 
>   ql/src/test/results/clientpositive/perf/query43.q.out 7fa4f53 
>   ql/src/test/results/clientpositive/perf/query45.q.out f55f20c 
>   ql/src/test/results/clientpositive/perf/query46.q.out 0ded912 
>   ql/src/test/results/clientpositive/perf/query48.q.out 40b76

Re: Review Request 42487: Use bit vector to track NDV

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42487/#review116706
---


Ship it!




Ship It!

- John Pullokkaran


On Jan. 26, 2016, 8:10 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42487/
> ---
> 
> (Updated Jan. 26, 2016, 8:10 a.m.)
> 
> 
> Review request for hive, Alan Gates and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12763
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/HiveStatsUtils.java 9193f80 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/hbase/TestHBaseSchemaTool.java
>  9fbbf90 
>   itests/src/test/resources/testconfiguration.properties 8318c3a 
>   metastore/if/hive_metastore.thrift 81837e6 
>   metastore/pom.xml a8e84a1 
>   
> metastore/src/gen/protobuf/gen-java/org/apache/hadoop/hive/metastore/hbase/HbaseMetastoreProto.java
>  39a7278 
>   metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h ce1d7da 
>   metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp 0203b06 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BinaryColumnStatsData.java
>  84e393c 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BooleanColumnStatsData.java
>  6aa4668 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DateColumnStatsData.java
>  2ebb811 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DecimalColumnStatsData.java
>  720176a 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DoubleColumnStatsData.java
>  5d48b5d 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LongColumnStatsData.java
>  2f41c5a 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StringColumnStatsData.java
>  bd8a922 
>   metastore/src/gen/thrift/gen-php/metastore/Types.php 380e6d0 
>   metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py 409c247 
>   metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb a473611 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/NumDistinctValueEstimator.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseUtils.java 
> f4df2e2 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java 
> 5ec60be 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BinaryColumnStatsAggregator.java
>  bbd2c7b 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BooleanColumnStatsAggregator.java
>  9047f68 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregator.java
>  217b654 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregatorFactory.java
>  a8dbc1f 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java
>  ec25b31 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java
>  71af0ac 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java
>  15b8cf7 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnStatsAggregator.java
>  fe1a04c 
>   
> metastore/src/protobuf/org/apache/hadoop/hive/metastore/hbase/hbase_metastore_proto.proto
>  0d0ef89 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestHBaseStoreBitVector.java
>  PRE-CREATION 
>   ql/pom.xml 358cd2a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java 7914471 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
> 1f30cbd 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java b4cf58f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java
>  0e96f89 
>   ql/src/test/queries/clientpositive/tez_aggr_part_stats.q PRE-CREATION 
>   ql/src/test/results/clientpositive/char_udf1.q.java1.7.out bfed116 
>   ql/src/test/results/clientpositive/columnstats_partlvl.q.out b7c9075 
>   ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out 9685202 
>   ql/src/test/results/clientpositive/columnstats_tbllvl.q.out 0aadae3 
>   ql/src/test/results/clientpositive/compute_stats_date.q.out b57a862 
>  

Re: Review Request 41807: further improve user level explain

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41807/#review116548
---




ql/src/test/results/clientpositive/perf/query12.q.out (line 47)
<https://reviews.apache.org/r/41807/#comment177571>

Could we write keys as expression
i.e instead of keys
RS_9:_col0=RS_11:d_date_sk


- John Pullokkaran


On Jan. 21, 2016, 7:46 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41807/
> ---
> 
> (Updated Jan. 21, 2016, 7:46 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12709
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Attr.java 
> 5d355d2 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Op.java 
> d0c1037 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Stage.java 
> 455d59f 
>   
> common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParser.java
>  b193fef 
>   
> common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParserUtils.java
>  PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Vertex.java 
> be01b8b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java f48db6a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 
> bc67e5a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java ad34b98 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 3f38f74 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableLikeDesc.java f051712 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java 40a8477 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d04cb78 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 0f2855e 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 7c8eee2 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java eb83fd6 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java 12f01e5 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java 427aac1 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 4b93e7c 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 5bea6fb 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 2f69b7f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceWork.java 1c31962 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/SelectDesc.java e7bbab4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 4e52bac 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 2fdb0a1 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 098aa89 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFQueryInputDef.java 
> ae55741 
>   
> ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PartitionedTableFunctionDef.java
>  46f106c 
>   ql/src/test/queries/clientpositive/explainuser_4.q PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query12.q.out 648d64d 
>   ql/src/test/results/clientpositive/perf/query13.q.out 6113d18 
>   ql/src/test/results/clientpositive/perf/query15.q.out c237828 
>   ql/src/test/results/clientpositive/perf/query17.q.out 0e42d05 
>   ql/src/test/results/clientpositive/perf/query18.q.out 83f21dc 
>   ql/src/test/results/clientpositive/perf/query19.q.out d565839 
>   ql/src/test/results/clientpositive/perf/query20.q.out 40c9a17 
>   ql/src/test/results/clientpositive/perf/query21.q.out 2634d3d 
>   ql/src/test/results/clientpositive/perf/query22.q.out 72935ed 
>   ql/src/test/results/clientpositive/perf/query25.q.out dabc1c5 
>   ql/src/test/results/clientpositive/perf/query26.q.out 84cc238 
>   ql/src/test/results/clientpositive/perf/query27.q.out ac38683 
>   ql/src/test/results/clientpositive/perf/query28.q.out cac9ac3 
>   ql/src/test/results/clientpositive/perf/query29.q.out 8991f35 
>   ql/src/test/results/clientpositive/perf/query3.q.out 871ad74 
>   ql/src/test/results/clientpositive/perf/query31.q.out 909d64c 
>   ql/src/test/results/clientpositive/perf/query32.q.out da2ac8f 
>   ql/src/test/results/clientpositive/perf/query34.q.out 1cb7c28 
>   ql/src/test/results/clientpositive/perf/query39.q.out 9f3e650 
>   ql/src/test/results/clientpositive/perf/query40.q.out b2d6262 
>   ql/src/test/results/clientpositive/perf/query42.q.out 9ede45d 
>   ql/src/test/results/clientpositive/perf/query43.q.out 7fa4f53 
>   

Re: Review Request 41807: further improve user level explain

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41807/#review116546
---




ql/src/test/results/clientpositive/perf/query90.q.out (line 100)
<https://reviews.apache.org/r/41807/#comment177570>

We are missing info on join type: inner vs outer


- John Pullokkaran


On Jan. 21, 2016, 7:46 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41807/
> ---
> 
> (Updated Jan. 21, 2016, 7:46 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12709
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Attr.java 
> 5d355d2 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Op.java 
> d0c1037 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Stage.java 
> 455d59f 
>   
> common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParser.java
>  b193fef 
>   
> common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParserUtils.java
>  PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Vertex.java 
> be01b8b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java f48db6a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java 
> bc67e5a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java ad34b98 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 3f38f74 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableLikeDesc.java f051712 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java 40a8477 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d04cb78 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 0f2855e 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 7c8eee2 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java eb83fd6 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java 12f01e5 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java 427aac1 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 4b93e7c 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 5bea6fb 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 2f69b7f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceWork.java 1c31962 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/SelectDesc.java e7bbab4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 4e52bac 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 2fdb0a1 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 098aa89 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFQueryInputDef.java 
> ae55741 
>   
> ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PartitionedTableFunctionDef.java
>  46f106c 
>   ql/src/test/queries/clientpositive/explainuser_4.q PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query12.q.out 648d64d 
>   ql/src/test/results/clientpositive/perf/query13.q.out 6113d18 
>   ql/src/test/results/clientpositive/perf/query15.q.out c237828 
>   ql/src/test/results/clientpositive/perf/query17.q.out 0e42d05 
>   ql/src/test/results/clientpositive/perf/query18.q.out 83f21dc 
>   ql/src/test/results/clientpositive/perf/query19.q.out d565839 
>   ql/src/test/results/clientpositive/perf/query20.q.out 40c9a17 
>   ql/src/test/results/clientpositive/perf/query21.q.out 2634d3d 
>   ql/src/test/results/clientpositive/perf/query22.q.out 72935ed 
>   ql/src/test/results/clientpositive/perf/query25.q.out dabc1c5 
>   ql/src/test/results/clientpositive/perf/query26.q.out 84cc238 
>   ql/src/test/results/clientpositive/perf/query27.q.out ac38683 
>   ql/src/test/results/clientpositive/perf/query28.q.out cac9ac3 
>   ql/src/test/results/clientpositive/perf/query29.q.out 8991f35 
>   ql/src/test/results/clientpositive/perf/query3.q.out 871ad74 
>   ql/src/test/results/clientpositive/perf/query31.q.out 909d64c 
>   ql/src/test/results/clientpositive/perf/query32.q.out da2ac8f 
>   ql/src/test/results/clientpositive/perf/query34.q.out 1cb7c28 
>   ql/src/test/results/clientpositive/perf/query39.q.out 9f3e650 
>   ql/src/test/results/clientpositive/perf/query40.q.out b2d6262 
>   ql/src/test/results/clientpositive/perf/query42.q.out 9ede45d 
>   ql/src/test/results/clientpositive/perf/query43.q.out 7fa4f53 
>   ql/src/test/results/clientpositive/pe

Re: Review Request 42190: HIVE-12478

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42190/#review116291
---


Ship it!




Ship It!

- John Pullokkaran


On Jan. 25, 2016, 7:20 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42190/
> ---
> 
> (Updated Jan. 25, 2016, 7:20 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-12478
> https://issues.apache.org/jira/browse/HIVE-12478
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Improve Hive/Calcite Trasitive Predicate inference
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> 4825a617876374085b6fac1192ba1531ec916bce 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveHepPlannerContext.java
>  ad79aeec2fbc0454ab1ccc608944752d01324dca 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveVolcanoPlannerContext.java
>  8859fc268666cef1be283a9179aa0beb7ef1bdeb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveCostModel.java
>  d15d885d2348d666df069228a93d6c5f914c79df 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveVolcanoPlanner.java
>  8610edc5ddc00d523610fb29f5e504c3e876a542 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveJoin.java
>  27b1e76a104dc961cb4bce554602d90b3aa867e0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSemiJoin.java
>  35586768c2c2b81e4213495632e4457dd3d70443 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java
>  8b57b3504c407b8a1e73d48ea240c4ec7558b327 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  de880ce26f1e172288f700c8566fbe71f42af115 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinPushTransitivePredicatesRule.java
>  703c8c6dbdfa281443cbcf7b08de2266697da8a9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePreFilteringRule.java
>  d37fc0e08d5e41b29539a990e6638385c1135eec 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRulesRegistry.java
>  18a065e87e1ec266bf28b4ccfe10a1f863f847c2 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 3fefbd710c4bb81d5f746cd91889b532b0a6029f 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/TestCBORuleFiredOnlyOnce.java
>  f1d8d1de00e9de7fa9ffea7d3aa2400e5073ac9c 
>   ql/src/test/results/clientpositive/annotate_stats_join.q.out 
> 7fc754d5712d5f05efc943b66d3c829d47312d19 
>   ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out 
> f13643e8db57cc0a85b2626c37437fd030f72029 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 8ef3664764d04f53f3685d8f66dc4a353776a488 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> c1da6d2968697d304311044d358f1af267dc6e60 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> d40b1655e27fa70efc8dbf0475c688f6e2b3608f 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 
> cfb95be72b32a354faeddc79dc52bc29c7593a2b 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_6.q.out 
> a7ad04c7208f8e8486e79c1e749184e5b532a1fc 
>   ql/src/test/results/clientpositive/cbo_const.q.out 
> adc5232a67b2243dd5f09acaf1f7c49baea5daad 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out 
> 1b2a2ab1af5992753c37d053942ecb2ebf775921 
>   ql/src/test/results/clientpositive/constprog_partitioner.q.out 
> 08c0aebafe1644a5adbec6a56879b39617bfbf69 
>   ql/src/test/results/clientpositive/correlationoptimizer1.q.out 
> 38764d6b127bd1aba1efa5800b16c0c1e387a310 
>   ql/src/test/results/clientpositive/correlationoptimizer10.q.out 
> 6bac698155221be4b0cb5b0bb3b2cdbfec5236ea 
>   ql/src/test/results/clientpositive/correlationoptimizer13.q.out 
> 61b7bcb675a80dc3267365c65545e107ecbe12d2 
>   ql/src/test/results/clientpositive/correlationoptimizer8.q.out 
> 368a1147d404ae9aab18a64618abf8ed7b3f7189 
>   ql/src/test/results/clientpositive/correlationoptimizer9.q.out 
> 104a97afbb6657255a027ff6d51101f6963c2358 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 
> 743865e073e1768f7f6234fde35ad2e56d8babe9 
>   ql/src/test/results/clientpositive/filter_cond_pushdown.q.out 
> 5e0edbc9efb442467c4998021ac1ecfb9d07579d 
>   ql/src/test/resu

Re: Review Request 42487: Use bit vector to track NDV

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42487/#review115948
---




metastore/src/java/org/apache/hadoop/hive/metastore/NumDistinctValueEstimator.java
 (line 46)
<https://reviews.apache.org/r/42487/#comment176992>

We should meassure, computational/space complexity of Bitwise opeartions & 
FastBitSet.

It would be good to get a meassure for 1000, 1, 50 partitions



ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java (line 1563)
<https://reviews.apache.org/r/42487/#comment176980>

Why is this needed?
Seems like its not used.


- John Pullokkaran


On Jan. 22, 2016, 7:09 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42487/
> ---
> 
> (Updated Jan. 22, 2016, 7:09 p.m.)
> 
> 
> Review request for hive, Alan Gates and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12763
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/HiveStatsUtils.java 9193f80 
>   metastore/if/hive_metastore.thrift 81837e6 
>   metastore/pom.xml a8e84a1 
>   
> metastore/src/gen/protobuf/gen-java/org/apache/hadoop/hive/metastore/hbase/HbaseMetastoreProto.java
>  39a7278 
>   metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h ce1d7da 
>   metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp 0203b06 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BinaryColumnStatsData.java
>  84e393c 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BooleanColumnStatsData.java
>  6aa4668 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DateColumnStatsData.java
>  2ebb811 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DecimalColumnStatsData.java
>  720176a 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DoubleColumnStatsData.java
>  5d48b5d 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LongColumnStatsData.java
>  2f41c5a 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StringColumnStatsData.java
>  bd8a922 
>   metastore/src/gen/thrift/gen-php/metastore/Types.php 380e6d0 
>   metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py 409c247 
>   metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb a473611 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/NumDistinctValueEstimator.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseUtils.java 
> f4df2e2 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java 
> 5ec60be 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BinaryColumnStatsAggregator.java
>  bbd2c7b 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BooleanColumnStatsAggregator.java
>  9047f68 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregator.java
>  217b654 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregatorFactory.java
>  a8dbc1f 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java
>  ec25b31 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java
>  71af0ac 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java
>  15b8cf7 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnStatsAggregator.java
>  fe1a04c 
>   
> metastore/src/protobuf/org/apache/hadoop/hive/metastore/hbase/hbase_metastore_proto.proto
>  0d0ef89 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestHBaseStoreBitVector.java
>  PRE-CREATION 
>   ql/pom.xml 358cd2a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java 7914471 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
> 1f30cbd 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java b4cf58f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java
>  0e96f89 
>   ql/src/test/queries/clientpositive/tez_aggr_part_stats.q PRE-CREATION 
>   ql/src/test/results/clientpositive/char_udf1.q.java1.7.out bfed116 
>   ql/

Re: Review Request 42190: HIVE-12478

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42190/#review115960
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
(line 636)
<https://reviews.apache.org/r/42190/#comment176997>

NitPick: Doc seems unfinished



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveJoin.java
 (line 112)
<https://reviews.apache.org/r/42190/#comment177001>

What about other operators?
HiveFilter, HiveProject .



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRulesRegistry.java
 (line 34)
<https://reviews.apache.org/r/42190/#comment176996>

Does this needs to be multimap?


- John Pullokkaran


On Jan. 21, 2016, 10:45 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42190/
> ---
> 
> (Updated Jan. 21, 2016, 10:45 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-12478
> https://issues.apache.org/jira/browse/HIVE-12478
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Improve Hive/Calcite Trasitive Predicate inference
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> 4825a617876374085b6fac1192ba1531ec916bce 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveHepPlannerContext.java
>  ad79aeec2fbc0454ab1ccc608944752d01324dca 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveVolcanoPlannerContext.java
>  8859fc268666cef1be283a9179aa0beb7ef1bdeb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveCostModel.java
>  d15d885d2348d666df069228a93d6c5f914c79df 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveVolcanoPlanner.java
>  8610edc5ddc00d523610fb29f5e504c3e876a542 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveJoin.java
>  27b1e76a104dc961cb4bce554602d90b3aa867e0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSemiJoin.java
>  35586768c2c2b81e4213495632e4457dd3d70443 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java
>  8b57b3504c407b8a1e73d48ea240c4ec7558b327 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  de880ce26f1e172288f700c8566fbe71f42af115 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinPushTransitivePredicatesRule.java
>  703c8c6dbdfa281443cbcf7b08de2266697da8a9 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePreFilteringRule.java
>  d37fc0e08d5e41b29539a990e6638385c1135eec 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRulesRegistry.java
>  18a065e87e1ec266bf28b4ccfe10a1f863f847c2 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 3fefbd710c4bb81d5f746cd91889b532b0a6029f 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/TestCBORuleFiredOnlyOnce.java
>  f1d8d1de00e9de7fa9ffea7d3aa2400e5073ac9c 
>   ql/src/test/results/clientpositive/annotate_stats_join.q.out 
> 7fc754d5712d5f05efc943b66d3c829d47312d19 
>   ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out 
> f13643e8db57cc0a85b2626c37437fd030f72029 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 
> b158d8567f6cc02d990d175e93996239aba0c5ed 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 8ef3664764d04f53f3685d8f66dc4a353776a488 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> c1da6d2968697d304311044d358f1af267dc6e60 
>   ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
> 59aa738c779d50a760e1b0d36e4ce83295b0d70f 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> d40b1655e27fa70efc8dbf0475c688f6e2b3608f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
> cb87f761be7c58ecc6435bb0a9b0e96c46a36828 
>   ql/src/test/results/clientpositive/bucket_map_join_spark4.q.out 
> 4abdab53b5562dc129a2d9a73c63cf44d066c05e 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 
> cfb95be72b32a354faeddc79dc52bc29c7593a2b 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_6.q.out 
> a7ad04c7208f8e8486e79c1e749184e5b532a1fc 
>   ql/src/test/results/clientpositive/cast1.q.out 
> 48a0c14031ef38dd5b4df7ef

Re: Review Request 42521: HIVE-12867 : Semantic Exception Error Msg should be with in the range of "10000 to 19999"

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42521/#review115755
---

Ship it!


Ship It!

- John Pullokkaran


On Jan. 21, 2016, 11:08 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42521/
> ---
> 
> (Updated Jan. 21, 2016, 11:08 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Semantic Exception Error Msg should be with in the range of "1 to 1"
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 6a62592 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
> 8a9411a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 141c2e6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java 179f9c2 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java 
> ff385d0 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
> 88c4b95 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java a8f9f50 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 3f38f74 
> 
> Diff: https://reviews.apache.org/r/42521/diff/
> 
> 
> Testing
> ---
> 
> Precommit build runs
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 42521: HIVE-12867 : Semantic Exception Error Msg should be with in the range of "10000 to 19999"

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42521/#review115235
---



ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java (line 445)
<https://reviews.apache.org/r/42521/#comment176140>

May be change this to "Semantic error"
Also why do we need a generic error


- John Pullokkaran


On Jan. 19, 2016, 9:03 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42521/
> ---
> 
> (Updated Jan. 19, 2016, 9:03 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Semantic Exception Error Msg should be with in the range of "1 to 1"
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 6a62592 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
> 8a9411a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 5e6b606 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java 179f9c2 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java 
> ff385d0 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
> 88c4b95 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java a8f9f50 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 3f38f74 
> 
> Diff: https://reviews.apache.org/r/42521/diff/
> 
> 
> Testing
> ---
> 
> Precommit build runs
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 42461: HIVE-12879

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42461/#review115093
---

Ship it!


Ship It!

- John Pullokkaran


On Jan. 18, 2016, 2:40 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42461/
> ---
> 
> (Updated Jan. 18, 2016, 2:40 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-12879
> https://issues.apache.org/jira/browse/HIVE-12879
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> RowResolver of Semijoin not updated in CalcitePlanner
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> f50f4d323220b1f90e33d5cf7daef9162998f5ec 
>   ql/src/test/queries/clientpositive/semijoin4.q PRE-CREATION 
>   ql/src/test/results/clientpositive/semijoin4.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/42461/diff/
> 
> 
> Testing
> ---
> 
> Added new test that reproduces the problem.
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 42281: Upgrade Hive to Calcite 1.6

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42281/#review114781
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
(line 903)
<https://reviews.apache.org/r/42281/#comment175622>

This should return false & not null.
I agree we don't use this today.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
(line 155)
<https://reviews.apache.org/r/42281/#comment175623>

Add a TODO: to handle CorrelationId


- John Pullokkaran


On Jan. 15, 2016, 7:10 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42281/
> ---
> 
> (Updated Jan. 15, 2016, 7:10 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12839
> 
> 
> Diffs
> -
> 
>   pom.xml 848432c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> 4825a61 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
> eeec44e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveAlgorithmsUtil.java
>  6840418 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveDefaultCostModel.java
>  6669d32 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveOnTezCostModel.java
>  61a3a64 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveRelMdCost.java
>  90f6a7e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java
>  aae57e6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
>  2f8cc7d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveJoin.java
>  27b1e76 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  c5376bf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSemiJoin.java
>  3558676 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveTableScan.java
>  5788805 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java
>  070c7ea 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExpandDistinctAggregatesRule.java
>  7d7631b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinPushTransitivePredicatesRule.java
>  703c8c6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  50e139b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
>  0af60e8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java
>  618c717 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
>  0ec8bf1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/FilterSelectivityEstimator.java
>  c04060f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdCollation.java
>  84fa518 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdDistinctRowCount.java
>  1220401 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdDistribution.java
>  b83f240 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdMemory.java
>  bea5943 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdParallelism.java
>  2f51d3b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdPredicates.java
>  b7244fd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
>  caf8978 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java
>  a0eb83d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSize.java
>  3224039 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdUniqueKeys.java
>  7c22c33 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
>  00f1acb 
>   ql/src/test/results/clientpositive/auto_join12.q.out 8ef3664 
>   ql/src/test/results/clientpositive/auto_join13.q.out fa03d2c 
>   ql/src/test/resu

Re: Review Request 42281: Upgrade Hive to Calcite 1.6

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42281/#review114798
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdDistribution.java
 
<https://reviews.apache.org/r/42281/#comment175650>

Why is this removed?


- John Pullokkaran


On Jan. 15, 2016, 7:10 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42281/
> ---
> 
> (Updated Jan. 15, 2016, 7:10 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12839
> 
> 
> Diffs
> -
> 
>   pom.xml 848432c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> 4825a61 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
> eeec44e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveAlgorithmsUtil.java
>  6840418 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveDefaultCostModel.java
>  6669d32 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveOnTezCostModel.java
>  61a3a64 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/cost/HiveRelMdCost.java
>  90f6a7e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java
>  aae57e6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
>  2f8cc7d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveJoin.java
>  27b1e76 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  c5376bf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSemiJoin.java
>  3558676 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveTableScan.java
>  5788805 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java
>  070c7ea 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExpandDistinctAggregatesRule.java
>  7d7631b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinPushTransitivePredicatesRule.java
>  703c8c6 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  50e139b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
>  0af60e8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java
>  618c717 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
>  0ec8bf1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/FilterSelectivityEstimator.java
>  c04060f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdCollation.java
>  84fa518 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdDistinctRowCount.java
>  1220401 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdDistribution.java
>  b83f240 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdMemory.java
>  bea5943 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdParallelism.java
>  2f51d3b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdPredicates.java
>  b7244fd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
>  caf8978 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java
>  a0eb83d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSize.java
>  3224039 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdUniqueKeys.java
>  7c22c33 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
>  00f1acb 
>   ql/src/test/results/clientpositive/auto_join12.q.out 8ef3664 
>   ql/src/test/results/clientpositive/auto_join13.q.out fa03d2c 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> d40b165 
>   ql/src/test/results/clientpositive/cbo_const.q.out adc5232 
>   ql/src/test/results/clientpositive/correlationoptimizer10.q.out 6bac698 
>   ql/src/test/results/client

[jira] [Created] (HIVE-12867) Semantic Exception Error Msg should be with in the range of "10000 to 19999"

[Laljo John Pullokkaran (JIRA)]

Laljo John Pullokkaran created HIVE-12867:
-

 Summary: Semantic Exception Error Msg should be with in the range 
of "1 to 1"
 Key: HIVE-12867
 URL: https://issues.apache.org/jira/browse/HIVE-12867
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.2.1
Reporter: Laljo John Pullokkaran
Assignee: Hari Sankar Sivarama Subramaniyan


At many places errors encountered during semantic exception is translated as 
generic error(GENERIC_ERROR, 4) msg as opposed to semantic error msg.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41297: HIVE-12590

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41297/#review113337
---

Ship it!


Ship It!

- John Pullokkaran


On Jan. 7, 2016, 9:17 p.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41297/
> ---
> 
> (Updated Jan. 7, 2016, 9:17 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-12590
> https://issues.apache.org/jira/browse/HIVE-12590
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Repeated UDAFs with literals can produce incorrect result
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java b96e2eb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 891b1f7 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ab9271f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 9d8b352 
>   ql/src/test/queries/clientpositive/groupby_duplicate_key.q 7f38efe 
>   ql/src/test/results/clientpositive/case_sensitivity.q.out a5b14e8 
>   ql/src/test/results/clientpositive/groupby_duplicate_key.q.out fc95f41 
>   
> ql/src/test/results/clientpositive/tez/schema_evol_orc_acid_mapwork_table.q.out
>  0317a99 
>   
> ql/src/test/results/clientpositive/tez/schema_evol_orc_acidvec_mapwork_table.q.out
>  3edaff0 
> 
> Diff: https://reviews.apache.org/r/41297/diff/
> 
> 
> Testing
> ---
> 
> Added new test
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>

Re: Review Request 41297: HIVE-12590

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41297/#review113329
---



ql/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java (line 245)
<https://reviews.apache.org/r/41297/#comment173867>

May be use a local var to store the str value; this way we could avoid 
duplicating logic at 2 places ?


- John Pullokkaran


On Jan. 7, 2016, 9:17 p.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41297/
> ---
> 
> (Updated Jan. 7, 2016, 9:17 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-12590
> https://issues.apache.org/jira/browse/HIVE-12590
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Repeated UDAFs with literals can produce incorrect result
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java b96e2eb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 891b1f7 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ab9271f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 9d8b352 
>   ql/src/test/queries/clientpositive/groupby_duplicate_key.q 7f38efe 
>   ql/src/test/results/clientpositive/case_sensitivity.q.out a5b14e8 
>   ql/src/test/results/clientpositive/groupby_duplicate_key.q.out fc95f41 
>   
> ql/src/test/results/clientpositive/tez/schema_evol_orc_acid_mapwork_table.q.out
>  0317a99 
>   
> ql/src/test/results/clientpositive/tez/schema_evol_orc_acidvec_mapwork_table.q.out
>  3edaff0 
> 
> Diff: https://reviews.apache.org/r/41297/diff/
> 
> 
> Testing
> ---
> 
> Added new test
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>

Re: Review Request 41297: HIVE-12590

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41297/#review111650
---



ql/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java (line 267)
<https://reviews.apache.org/r/41297/#comment171861>

It may be better to remove toStringTree and replace it with 
toLowerStringTree.

Since in Hive all col names, udf/udaf are case insensitive, we might be ok 
if we make only literals case sensitive.


- John Pullokkaran


On Dec. 17, 2015, 6:36 p.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41297/
> ---
> 
> (Updated Dec. 17, 2015, 6:36 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-12590
> https://issues.apache.org/jira/browse/HIVE-12590
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Repeated UDAFs with literals can produce incorrect result
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java b96e2eb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 891b1f7 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 27549dc 
>   ql/src/test/queries/clientpositive/groupby_duplicate_key.q 7f38efe 
>   ql/src/test/results/clientpositive/groupby_duplicate_key.q.out fc95f41 
> 
> Diff: https://reviews.apache.org/r/41297/diff/
> 
> 
> Testing
> ---
> 
> Added new test
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review111537
---

Ship it!


Ship It!

- John Pullokkaran


On Dec. 21, 2015, 5:44 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 21, 2015, 5:44 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 87b18b7 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 13d8a50 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out 79f76bf 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out d9143c8 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out 4e2b93e 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 3153c7e 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out 52cbdf4 
>   ql/src/test/results/clientpositive/perf/query39.q.out a18cdaf 
>   ql/src/test/results/clientpositive/perf/query42.q.out 3954829 
>   ql/src/test/results/clientpositive/perf/query52.q.out 63f0bf6 
>   ql/src/test/results/clientpositive/perf/query64.q.out a63 
>   ql/src/test/results/clientpositive/perf/query66.q.out 22eaf61 
>   ql/src/test/results/clientpositive/perf/query75.q.out d54000b 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
>   ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 
> 0656cd5 
>   ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out 
> 2d6eb38 
>   ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out 9043fb1 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 239e803 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 
> 8370bbe 
>   ql/src/test/results/clientpositive/spark/union_remove_25.q.out b771fe9 
>   ql/src/test/results/clientpositive/spark/union_view.q.out cce7710 
>   ql/src/test/results/clientpositive/subquery_notin.q.out ed86079 
>   ql/src/test/results/clientpositive/subquery_notin_having.q.java1.7.out 

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review111536
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
 (line 320)
<https://reviews.apache.org/r/40865/#comment171729>

Remove this code; seems like not used.


- John Pullokkaran


On Dec. 21, 2015, 5:44 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 21, 2015, 5:44 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 87b18b7 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 13d8a50 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out 79f76bf 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out d9143c8 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out 4e2b93e 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 3153c7e 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out 52cbdf4 
>   ql/src/test/results/clientpositive/perf/query39.q.out a18cdaf 
>   ql/src/test/results/clientpositive/perf/query42.q.out 3954829 
>   ql/src/test/results/clientpositive/perf/query52.q.out 63f0bf6 
>   ql/src/test/results/clientpositive/perf/query64.q.out a63 
>   ql/src/test/results/clientpositive/perf/query66.q.out 22eaf61 
>   ql/src/test/results/clientpositive/perf/query75.q.out d54000b 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
>   ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 
> 0656cd5 
>   ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out 
> 2d6eb38 
>   ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out 9043fb1 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 239e803 
>   ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 
> 8370bbe 
>   ql/src/test/results/clientpositive/spark/union_remove_25.q.out b771fe9 
>   ql/src/test/results/clientpo

Re: Review Request 41457: HIVE-11865

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/#review111254
---



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 491)
<https://reviews.apache.org/r/41457/#comment171355>

This is fragile; i.e reusing the flag whether to run CBO or not for whether 
CBO was actually run successfully.

We already keep state in parsecontext about this.


- John Pullokkaran


On Dec. 18, 2015, 5:28 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41457/
> ---
> 
> (Updated Dec. 18, 2015, 5:28 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Bugs: HIVE-11865
> https://issues.apache.org/jira/browse/HIVE-11865
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Disable Hive PPD optimizer when CBO has optimized the plan
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> 3907bc946679c0ce2492692dcb7e2cf252c6e188 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
> 8f48e7df4b9c081528b82545d445f75f67f93bfc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  142812ca703d5e667bd532a711d29594123bde50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
>  8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
>  7e484b90deed2dfb120faa2699eb9b9b7a4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  c8de1d89a1df083cd125ae8c11bf854b09915680 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  efde6652ca270d8e8d8f682fc0ff260702f46d5a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  feec3c25929cac0bb3b6f50d074557a3c2745889 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 7c5a43ff4060c9a4d9c4c5523d851ef959da3932 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 27549dceac8b5913b1fb1604bde213ae26b243bf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> ccc4bb454e61e3c39b1f90f29de5e754f8d886bf 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
> 1702628447d38b09c9b7d0dcf38218a9037a2c64 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> accfb3b9f5631f25220c88b759dba6c5299b4044 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 6f08aa8f043113f0a99acd10c848f60a122a3f0b 
>   ql/src/test/results/clientpositive/auto_join13.q.out 
> d29818cf438449a6a1413edd7d474c4ab35f2148 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> 38738ede2b4cd49aa24490eafb81dec1de78bea5 
>   ql/src/test/results/clientpositive/auto_join2.q.out 
> ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
>   ql/src/test/results/clientpositive/auto_join33.q.out 
> 8b13bd0cb7fd869c58c9d092b2e8710391c96503 
>   ql/src/test/results/clientpositive/auto_join_filters.q.out 
> e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
>   ql/src/test/results/clientpositive/auto_join_nulls.q.out 
> 954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 
> feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 
> e0d0146dd6b86e9f69a467dcde0c020c31af499d 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
>   ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
> 096f5d445bc7e61381775270bef04e911eac5bb6 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
> 30853c42a52e1def5a913352580e0b5391ac2050 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
> 3e8dbcfd5bcc66ccd83b01c08c7745b02f05328f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 
> e6c2832003a1bb804f99125866bff6117e4dab73 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out 
> 326e4d67f00158929f4aaa0a2ec53cedabe3e92c 
>   ql/src/test/results/clien

Re: Review Request 41457: HIVE-11865

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/#review111247
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java (line 110)
<https://reviews.apache.org/r/41457/#comment171344>

In the CBO Success path the additional const prop seems unnecessary. All we 
have done between two const prop is is synthetic join predicates (not null 
predicates) && Simple predicate push down.


- John Pullokkaran


On Dec. 18, 2015, 5:28 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41457/
> ---
> 
> (Updated Dec. 18, 2015, 5:28 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Bugs: HIVE-11865
> https://issues.apache.org/jira/browse/HIVE-11865
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Disable Hive PPD optimizer when CBO has optimized the plan
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> 3907bc946679c0ce2492692dcb7e2cf252c6e188 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
> 8f48e7df4b9c081528b82545d445f75f67f93bfc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  142812ca703d5e667bd532a711d29594123bde50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
>  8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
>  7e484b90deed2dfb120faa2699eb9b9b7a4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  c8de1d89a1df083cd125ae8c11bf854b09915680 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  efde6652ca270d8e8d8f682fc0ff260702f46d5a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  feec3c25929cac0bb3b6f50d074557a3c2745889 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 7c5a43ff4060c9a4d9c4c5523d851ef959da3932 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 27549dceac8b5913b1fb1604bde213ae26b243bf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> ccc4bb454e61e3c39b1f90f29de5e754f8d886bf 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
> 1702628447d38b09c9b7d0dcf38218a9037a2c64 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> accfb3b9f5631f25220c88b759dba6c5299b4044 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 6f08aa8f043113f0a99acd10c848f60a122a3f0b 
>   ql/src/test/results/clientpositive/auto_join13.q.out 
> d29818cf438449a6a1413edd7d474c4ab35f2148 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> 38738ede2b4cd49aa24490eafb81dec1de78bea5 
>   ql/src/test/results/clientpositive/auto_join2.q.out 
> ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
>   ql/src/test/results/clientpositive/auto_join33.q.out 
> 8b13bd0cb7fd869c58c9d092b2e8710391c96503 
>   ql/src/test/results/clientpositive/auto_join_filters.q.out 
> e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
>   ql/src/test/results/clientpositive/auto_join_nulls.q.out 
> 954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 
> feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 
> e0d0146dd6b86e9f69a467dcde0c020c31af499d 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
>   ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
> 096f5d445bc7e61381775270bef04e911eac5bb6 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
> 30853c42a52e1def5a913352580e0b5391ac2050 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
> 3e8dbcfd5bcc66ccd83b01c08c7745b02f05328f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 
> e6c2832003a1bb804f99125866bff6117e4dab73 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out 
> 326e4d67f00158929f4aaa

Re: Review Request 41457: HIVE-11865

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/#review111255
---



ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java (line 167)
<https://reviews.apache.org/r/41457/#comment171356>

This is fragile.
State will have to be maintained; i.e unset if a filter merge happened.


- John Pullokkaran


On Dec. 18, 2015, 5:28 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41457/
> ---
> 
> (Updated Dec. 18, 2015, 5:28 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Bugs: HIVE-11865
> https://issues.apache.org/jira/browse/HIVE-11865
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Disable Hive PPD optimizer when CBO has optimized the plan
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> 3907bc946679c0ce2492692dcb7e2cf252c6e188 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
> 8f48e7df4b9c081528b82545d445f75f67f93bfc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  142812ca703d5e667bd532a711d29594123bde50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
>  8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
>  7e484b90deed2dfb120faa2699eb9b9b7a4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  c8de1d89a1df083cd125ae8c11bf854b09915680 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  efde6652ca270d8e8d8f682fc0ff260702f46d5a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  feec3c25929cac0bb3b6f50d074557a3c2745889 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 7c5a43ff4060c9a4d9c4c5523d851ef959da3932 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 27549dceac8b5913b1fb1604bde213ae26b243bf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> ccc4bb454e61e3c39b1f90f29de5e754f8d886bf 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
> 1702628447d38b09c9b7d0dcf38218a9037a2c64 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> accfb3b9f5631f25220c88b759dba6c5299b4044 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 6f08aa8f043113f0a99acd10c848f60a122a3f0b 
>   ql/src/test/results/clientpositive/auto_join13.q.out 
> d29818cf438449a6a1413edd7d474c4ab35f2148 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> 38738ede2b4cd49aa24490eafb81dec1de78bea5 
>   ql/src/test/results/clientpositive/auto_join2.q.out 
> ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
>   ql/src/test/results/clientpositive/auto_join33.q.out 
> 8b13bd0cb7fd869c58c9d092b2e8710391c96503 
>   ql/src/test/results/clientpositive/auto_join_filters.q.out 
> e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
>   ql/src/test/results/clientpositive/auto_join_nulls.q.out 
> 954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 
> feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 
> e0d0146dd6b86e9f69a467dcde0c020c31af499d 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
>   ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
> 096f5d445bc7e61381775270bef04e911eac5bb6 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
> 30853c42a52e1def5a913352580e0b5391ac2050 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
> 3e8dbcfd5bcc66ccd83b01c08c7745b02f05328f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 
> e6c2832003a1bb804f99125866bff6117e4dab73 
>   ql/src/test/results/clientpositive/bucketsortoptimize_insert_7.q.out 
> 326e4d67f00158929f4aaa0a2ec53cedabe3e92c 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b1e43

Re: Review Request 41457: HIVE-11865

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/#review111248
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
 (line 1)
<https://reviews.apache.org/r/41457/#comment171347>

These seems cosmetic, do we need these?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
 (line 10)
<https://reviews.apache.org/r/41457/#comment171348>

These seems cosmetic, do we need these?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
 (line 1)
<https://reviews.apache.org/r/41457/#comment171349>

These seems cosmetic, do we need these?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
 (line 10)
<https://reviews.apache.org/r/41457/#comment171350>

These seems cosmetic, do we need these?


- John Pullokkaran


On Dec. 18, 2015, 5:28 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41457/
> ---
> 
> (Updated Dec. 18, 2015, 5:28 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Bugs: HIVE-11865
> https://issues.apache.org/jira/browse/HIVE-11865
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Disable Hive PPD optimizer when CBO has optimized the plan
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> 3907bc946679c0ce2492692dcb7e2cf252c6e188 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
> 8f48e7df4b9c081528b82545d445f75f67f93bfc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  142812ca703d5e667bd532a711d29594123bde50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
>  8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
>  7e484b90deed2dfb120faa2699eb9b9b7a4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  c8de1d89a1df083cd125ae8c11bf854b09915680 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  efde6652ca270d8e8d8f682fc0ff260702f46d5a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  feec3c25929cac0bb3b6f50d074557a3c2745889 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 7c5a43ff4060c9a4d9c4c5523d851ef959da3932 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 27549dceac8b5913b1fb1604bde213ae26b243bf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
> ccc4bb454e61e3c39b1f90f29de5e754f8d886bf 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 
> 1702628447d38b09c9b7d0dcf38218a9037a2c64 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
> accfb3b9f5631f25220c88b759dba6c5299b4044 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 6f08aa8f043113f0a99acd10c848f60a122a3f0b 
>   ql/src/test/results/clientpositive/auto_join13.q.out 
> d29818cf438449a6a1413edd7d474c4ab35f2148 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> 38738ede2b4cd49aa24490eafb81dec1de78bea5 
>   ql/src/test/results/clientpositive/auto_join2.q.out 
> ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
>   ql/src/test/results/clientpositive/auto_join33.q.out 
> 8b13bd0cb7fd869c58c9d092b2e8710391c96503 
>   ql/src/test/results/clientpositive/auto_join_filters.q.out 
> e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
>   ql/src/test/results/clientpositive/auto_join_nulls.q.out 
> 954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 
> feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 
> e0d0146dd6b86e9f69a467dcde0c020c31af499d 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
>   ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
> 096f5d445bc7e61381775270bef04e911eac5bb6 

Re: Review Request 40589: Implement limit push down through union all in CBO

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review111311
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
 (line 84)
<https://reviews.apache.org/r/40589/#comment171432>

The real value of pushing limit past union is to push it further down (by 
pushing it past sel, join).

If the sortrel ended up just below union & if sortrel also had sort in it 
(i.e if union's input is synthetic limit) then i think we should remove it.

May be we should enhance HiveSortRemoveRule to look for pattern 
SortRel-union-SortRel and remove synthetic SortRel below union if it has sort 
in it.

Otherwise we end up paying double sorting cost.


- John Pullokkaran


On Dec. 17, 2015, 10:47 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> ---
> 
> (Updated Dec. 17, 2015, 10:47 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11775
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf 
>   ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7 
>   ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40589/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review111348
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
 (line 334)
<https://reviews.apache.org/r/40865/#comment171500>

Hive's null literal has void type.
This is handled down in void switch.

So we might want to switch calcite null literal to void.


- John Pullokkaran


On Dec. 19, 2015, 12:28 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 19, 2015, 12:28 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  3f2267d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
>  2825f77 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7c5a43f 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out b14caa8 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out e7d6900 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out df438c9 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out b85d387 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out c479d91 
>   ql/src/test/results/clientpositive/perf/query39.q.out 0ad62cc 
>   ql/src/test/results/clientpositive/perf/query42.q.out a3264c7 
>   ql/src/test/results/clientpositive/perf/query52.q.out ac0c1e6 
>   ql/src/test/results/clientpositive/perf/query64.q.out 9735b45 
>   ql/src/test/results/clientpositive/perf/query66.q.out 42bcb83 
>   ql/src/test/results/clientpositive/perf/query75.q.out 70a1649 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
>   ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 
> 5b03dcf 
>   ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out 
> 93c502d 
>   ql/src/test/results/clientpositive/s

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review111351
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
 (line 320)
<https://reviews.apache.org/r/40865/#comment171507>

Rename class?


- John Pullokkaran


On Dec. 19, 2015, 12:28 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 19, 2015, 12:28 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  3f2267d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
>  2825f77 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7c5a43f 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out b14caa8 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out e7d6900 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out df438c9 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out b85d387 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out c479d91 
>   ql/src/test/results/clientpositive/perf/query39.q.out 0ad62cc 
>   ql/src/test/results/clientpositive/perf/query42.q.out a3264c7 
>   ql/src/test/results/clientpositive/perf/query52.q.out ac0c1e6 
>   ql/src/test/results/clientpositive/perf/query64.q.out 9735b45 
>   ql/src/test/results/clientpositive/perf/query66.q.out 42bcb83 
>   ql/src/test/results/clientpositive/perf/query75.q.out 70a1649 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
>   ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 
> 5b03dcf 
>   ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out 
> 93c502d 
>   ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out b597ebd 
>   ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 2

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review111345
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
 (line 183)
<https://reviews.apache.org/r/40865/#comment171492>

Encoding binary as string literal wouldn't this result in lossy translation?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
 (line 249)
<https://reviews.apache.org/r/40865/#comment171494>

Why add milli seconds?


- John Pullokkaran


On Dec. 19, 2015, 12:28 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 19, 2015, 12:28 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  3f2267d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
>  2825f77 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7c5a43f 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out b14caa8 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out e7d6900 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out df438c9 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out b85d387 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out c479d91 
>   ql/src/test/results/clientpositive/perf/query39.q.out 0ad62cc 
>   ql/src/test/results/clientpositive/perf/query42.q.out a3264c7 
>   ql/src/test/results/clientpositive/perf/query52.q.out ac0c1e6 
>   ql/src/test/results/clientpositive/perf/query64.q.out 9735b45 
>   ql/src/test/results/clientpositive/perf/query66.q.out 42bcb83 
>   ql/src/test/results/clientpositive/perf/query75.q.out 70a1649 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
>   ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 
> 5b03dcf 
>   ql/sr

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review111344
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
 (line 325)
<https://reviews.apache.org/r/40865/#comment171491>

This is a change in interface.
Instead of translating a type now we are translating a RexCall.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
 (line 348)
<https://reviews.apache.org/r/40865/#comment171490>

Why are we looking at operands arguments?


- John Pullokkaran


On Dec. 19, 2015, 12:28 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 19, 2015, 12:28 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  3f2267d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
>  2825f77 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7c5a43f 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out b14caa8 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out e7d6900 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out df438c9 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out b85d387 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out c479d91 
>   ql/src/test/results/clientpositive/perf/query39.q.out 0ad62cc 
>   ql/src/test/results/clientpositive/perf/query42.q.out a3264c7 
>   ql/src/test/results/clientpositive/perf/query52.q.out ac0c1e6 
>   ql/src/test/results/clientpositive/perf/query64.q.out 9735b45 
>   ql/src/test/results/clientpositive/perf/query66.q.out 42bcb83 
>   ql/src/test/results/clientpositive/perf/query75.q.out 70a1649 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
>   ql/src/test/results/clientpositive/spark/cross_product_check_1

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review111349
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
 (line 396)
<https://reviews.apache.org/r/40865/#comment171501>

Not sure about isInfinite.



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 1210)
<https://reviews.apache.org/r/40865/#comment171506>

Could we add executor to previous hepPlan and check for null



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 1267)
<https://reviews.apache.org/r/40865/#comment171505>

Could we add executor to previous hepPlan and check for null


- John Pullokkaran


On Dec. 19, 2015, 12:28 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 19, 2015, 12:28 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  3f2267d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
>  2825f77 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7c5a43f 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out b14caa8 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out e7d6900 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out df438c9 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out b85d387 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out c479d91 
>   ql/src/test/results/clientpositive/perf/query39.q.out 0ad62cc 
>   ql/src/test/results/clientpositive/perf/query42.q.out a3264c7 
>   ql/src/test/results/clientpositive/perf/query52.q.out ac0c1e6 
>   ql/src/test/results/clientpositive/perf/query64.q.out 9735b45 
>   ql/src/test/results/clientpositive/perf/query66.q.out 42bcb83 
>   ql/src/test/results/clientpositive/perf/query75.q.out 70a1649 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/

Re: Review Request 40589: Implement limit push down through union all in CBO

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review111019
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
 (line 63)
<https://reviews.apache.org/r/40589/#comment171015>

Same as above



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
 (line 103)
<https://reviews.apache.org/r/40589/#comment171017>

May be rename the flag that signifies its use



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
 (line 106)
<https://reviews.apache.org/r/40589/#comment171019>

Shouldn't this be < instead of >=.


- John Pullokkaran


On Dec. 16, 2015, 6:46 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> ---
> 
> (Updated Dec. 16, 2015, 6:46 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11775
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf 
>   ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7 
>   ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40589/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 40589: Implement limit push down through union all in CBO

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review111018
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
 (line 57)
<https://reviews.apache.org/r/40589/#comment171014>

This is not needed for our Parallel execution env as there is no guarantee 
that set ops will preserve order.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
 (line 94)
<https://reviews.apache.org/r/40589/#comment171013>

In case of pure oder by fetch will be null.
Wouldn't "RexLiteral.intValue(sort.fetch)" result in an NPE?


- John Pullokkaran


On Dec. 16, 2015, 6:46 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> ---
> 
> (Updated Dec. 16, 2015, 6:46 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11775
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf 
>   ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7 
>   ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40589/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 40589: Implement limit push down through union all in CBO

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review111079
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
 (line 56)
<https://reviews.apache.org/r/40589/#comment171053>

Nit Pick: extra space



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
 (line 81)
<https://reviews.apache.org/r/40589/#comment171054>

This should really be 
if ((RexLiteral.intValue(sort.offset) + RexLiteral.intValue(sort.fetch)) < 
RelMetadataQuery.getRowCount(input)) {
}


- John Pullokkaran


On Dec. 17, 2015, 10:09 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> ---
> 
> (Updated Dec. 17, 2015, 10:09 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11775
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf 
>   ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7 
>   ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40589/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 40589: Implement limit push down through union all in CBO

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review111083
---

Ship it!


Ship It!

- John Pullokkaran


On Dec. 17, 2015, 10:47 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> ---
> 
> (Updated Dec. 17, 2015, 10:47 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11775
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf 
>   ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7 
>   ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40589/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 41387: HIVE-12675: PerfLogger should log performance metrics at debug level

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41387/#review110748
---

Ship it!


Ship It!

- John Pullokkaran


On Dec. 15, 2015, 2:26 a.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41387/
> ---
> 
> (Updated Dec. 15, 2015, 2:26 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> PerfLogger should log performance metrics at debug level. Please see the jira.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java f6eb64b 
> 
> Diff: https://reviews.apache.org/r/41387/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 41457: HIVE-11865

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41457/#review110839
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java (line 104)
<https://reviews.apache.org/r/41457/#comment170852>

What about const propagation?


- John Pullokkaran


On Dec. 16, 2015, 7:14 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41457/
> ---
> 
> (Updated Dec. 16, 2015, 7:14 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Bugs: HIVE-11865
> https://issues.apache.org/jira/browse/HIVE-11865
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Disable Hive PPD optimizer when CBO has optimized the plan
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_pushdown.q.out 
> be96eecc8642f85a493ae6c72964bc4e30c4db57 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out 
> 3907bc946679c0ce2492692dcb7e2cf252c6e188 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
> 8f48e7df4b9c081528b82545d445f75f67f93bfc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  db625c1c3d47b5ee2744950db2fb71804945cbec 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTSTransposeRule.java
>  8321504774c5ec5c7c9de3aecdf78dcd8408c4cf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterProjectTransposeRule.java
>  7e484b90deed2dfb120faa2699eb9b9b7a4e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
>  2a415d5abd9c054956f46cd26f9df2257d67ecf4 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  efde6652ca270d8e8d8f682fc0ff260702f46d5a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  feec3c25929cac0bb3b6f50d074557a3c2745889 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 36a12bf6b2c3ef7233eda7c949725e02c880d7f7 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 27549dceac8b5913b1fb1604bde213ae26b243bf 
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/SimplePredicatePushDown.java 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/auto_join12.q.out 
> 6f08aa8f043113f0a99acd10c848f60a122a3f0b 
>   ql/src/test/results/clientpositive/auto_join13.q.out 
> d29818cf438449a6a1413edd7d474c4ab35f2148 
>   ql/src/test/results/clientpositive/auto_join16.q.out 
> 38738ede2b4cd49aa24490eafb81dec1de78bea5 
>   ql/src/test/results/clientpositive/auto_join2.q.out 
> ce40ff30cb9cc2ab734ee37c0f3dc0ae85bbd202 
>   ql/src/test/results/clientpositive/auto_join33.q.out 
> 8b13bd0cb7fd869c58c9d092b2e8710391c96503 
>   ql/src/test/results/clientpositive/auto_join8.q.out 
> 80dd575c58add873ae34967b1d255ee5b63a9b07 
>   ql/src/test/results/clientpositive/auto_join_filters.q.out 
> e0ed3737f9b25d45bdb365b13aeed3a40c50135f 
>   ql/src/test/results/clientpositive/auto_join_nulls.q.out 
> 954bf069760c6f0d7e5420c7e9a3b9756d5aa4df 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out 
> feb818679df7d9f63f2d2d7a7c9afb2d0f5874d8 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 
> e0d0146dd6b86e9f69a467dcde0c020c31af499d 
>   ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 
> c63016d0a3610eaa6cd82a4adc5d1bf6f0925cce 
>   ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out 
> 096f5d445bc7e61381775270bef04e911eac5bb6 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
> 30853c42a52e1def5a913352580e0b5391ac2050 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 
> 3e8dbcfd5bcc66ccd83b01c08c7745b02f05328f 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 
> e6c2832003a1bb804f99125866bff6117e4dab73 
>   ql/src/test/results/clientpositive/cbo_rp_auto_join1.q.out 
> 5c09fe0ad2d30c6c597abdad27b309fd8d00f678 
>   ql/src/test/results/clientpositive/cbo_rp_join0.q.out 
> 3c6bb7319fd7605a62f082b989f723d5aab6c3fe 
>   ql/src/test/results/clientpositive/cbo_rp_join1.q.out 
> e77002866174f85168859c4145e7016c9d829bb6 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out 
> b14caa80b3346036f9ec835fe4

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review110836
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
 (line 43)
<https://reviews.apache.org/r/40865/#comment170846>

Nit Pick: trailing space



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
 (line 63)
<https://reviews.apache.org/r/40865/#comment170848>

Why are we printing stack trace?
Log it instead



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
 (line 235)
<https://reviews.apache.org/r/40865/#comment170849>

Nit Pick: space


- John Pullokkaran


On Dec. 17, 2015, 1:26 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 17, 2015, 1:26 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  3f2267d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
>  2825f77 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out b14caa8 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out e7d6900 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out df438c9 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out b85d387 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out c479d91 
>   ql/src/test/results/clientpositive/perf/query39.q.out 0ad62cc 
>   ql/src/test/results/clientpositive/perf/query42.q.out a3264c7 
>   ql/src/test/results/clientpositive/perf/query52.q.out ac0c1e6 
>   ql/src/test/results/clientpositive/perf/query64.q.out 9735b45 
>   ql/src/test/results/clientpositive/perf/query66.q.out 42bcb83 
>   ql/src/test/results/clientpositive/perf/query75.q.out 70a1649 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review110835
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
(line 41)
<https://reviews.apache.org/r/40865/#comment170845>

Remove trailing space.


- John Pullokkaran


On Dec. 17, 2015, 1:26 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 17, 2015, 1:26 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  3f2267d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
>  2825f77 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out b14caa8 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out e7d6900 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out df438c9 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out b85d387 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out c479d91 
>   ql/src/test/results/clientpositive/perf/query39.q.out 0ad62cc 
>   ql/src/test/results/clientpositive/perf/query42.q.out a3264c7 
>   ql/src/test/results/clientpositive/perf/query52.q.out ac0c1e6 
>   ql/src/test/results/clientpositive/perf/query64.q.out 9735b45 
>   ql/src/test/results/clientpositive/perf/query66.q.out 42bcb83 
>   ql/src/test/results/clientpositive/perf/query75.q.out 70a1649 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
>   ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 
> 5b03dcf 
>   ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out 
> 93c502d 
>   ql/src/test/results/clientpositive/spark/dynamic_rdd

Re: Review Request 40865: Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40865/#review110834
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
(line 49)
<https://reviews.apache.org/r/40865/#comment170843>

What about rest of the factories?


- John Pullokkaran


On Dec. 17, 2015, 1:26 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40865/
> ---
> 
> (Updated Dec. 17, 2015, 1:26 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11927
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  336745b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexExecutorImpl.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  425514d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  3f2267d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  631a4ca 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
>  2825f77 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf 
>   ql/src/test/queries/clientpositive/cbo_const.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/constantfolding.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out c4d59c8 
>   ql/src/test/results/clientpositive/bucketizedhiveinputformat.q.out 277b0f7 
>   ql/src/test/results/clientpositive/cast1.q.out 0bdecba 
>   ql/src/test/results/clientpositive/cbo_const.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> 62b611b 
>   ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out b14caa8 
>   ql/src/test/results/clientpositive/constantfolding.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cross_product_check_1.q.out e7d6900 
>   ql/src/test/results/clientpositive/cross_product_check_2.q.out df438c9 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out b85d387 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 24ac550 
>   ql/src/test/results/clientpositive/groupby_ppd.q.out 6164a26 
>   ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 7333677 
>   ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out e19d1de 
>   ql/src/test/results/clientpositive/input_part1.q.out d6f4d3e 
>   ql/src/test/results/clientpositive/input_part5.q.out f2d7335 
>   ql/src/test/results/clientpositive/input_part6.q.out fa51cdf 
>   ql/src/test/results/clientpositive/lineage2.q.out ec8b76b 
>   ql/src/test/results/clientpositive/lineage3.q.out 747dc9a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> be77ba8 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 79348f3 
>   ql/src/test/results/clientpositive/orc_predicate_pushdown.q.out f6d8388 
>   ql/src/test/results/clientpositive/parquet_predicate_pushdown.q.out b322ef1 
>   ql/src/test/results/clientpositive/partition_multilevels.q.out c1c8778 
>   ql/src/test/results/clientpositive/perf/query31.q.out c479d91 
>   ql/src/test/results/clientpositive/perf/query39.q.out 0ad62cc 
>   ql/src/test/results/clientpositive/perf/query42.q.out a3264c7 
>   ql/src/test/results/clientpositive/perf/query52.q.out ac0c1e6 
>   ql/src/test/results/clientpositive/perf/query64.q.out 9735b45 
>   ql/src/test/results/clientpositive/perf/query66.q.out 42bcb83 
>   ql/src/test/results/clientpositive/perf/query75.q.out 70a1649 
>   ql/src/test/results/clientpositive/pointlookup2.q.out a442425 
>   ql/src/test/results/clientpositive/quotedid_basic.q.out 519f647 
>   ql/src/test/results/clientpositive/quotedid_partition.q.out d34a005 
>   ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out b5e7846 
>   ql/src/test/results/clientpositive/spark/cross_product_check_1.q.out 
> 5b03dcf 
>   ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out 
> 93c502d 
>   ql/src/test/results/clientpositive/spark/dyn

Re: Review Request 40589: Implement limit push down through union all in CBO

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review110361
---



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 1071)
<https://reviews.apache.org/r/40589/#comment170171>

This should be added to previous hepplan invocation so that limit can be 
pushed as low as possible.

Ex: Limit-Sel-Union-Join-Proj-TS1
  TS2


- John Pullokkaran


On Dec. 14, 2015, 6:59 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> ---
> 
> (Updated Dec. 14, 2015, 6:59 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11775
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 56a39df 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java a0d14dc 
>   ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6 
>   ql/src/test/queries/clientpositive/parquet_ppd_boolean.q 059da68 
>   ql/src/test/queries/clientpositive/parquet_ppd_char.q eaddcb4 
>   ql/src/test/queries/clientpositive/parquet_ppd_date.q 41d0d64 
>   ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486 
>   ql/src/test/queries/clientpositive/parquet_ppd_timestamp.q 1b9f6ff 
>   ql/src/test/queries/clientpositive/parquet_ppd_varchar.q 6449c6d 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7 
>   ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/constprog_dpp.q.out a1addb7 
>   ql/src/test/results/clientpositive/parquet_ppd_boolean.q.out 6e62ee4 
>   ql/src/test/results/clientpositive/parquet_ppd_char.q.out c1565f8 
>   ql/src/test/results/clientpositive/parquet_ppd_date.q.out 55231e9 
>   ql/src/test/results/clientpositive/parquet_ppd_decimal.q.out a30820e 
>   ql/src/test/results/clientpositive/parquet_ppd_timestamp.q.out 3b3e5b7 
>   ql/src/test/results/clientpositive/parquet_ppd_varchar.q.out 5a62e80 
>   ql/src/test/results/clientpositive/spark/temp_table.q.out 119d26d 
> 
> Diff: https://reviews.apache.org/r/40589/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 41051: HIVE-12570 : Incorrect error message Expression not in GROUP BY key thrown instead of Invalid function

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41051/#review110038
---



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (line 537)
<https://reviews.apache.org/r/41051/#comment169791>

What about Window Functions that is not valid?
Check gen plan if it handles it correctly.
If not then that should also be covered.

ex: select average(x) over() from t1;


- John Pullokkaran


On Dec. 7, 2015, 9:11 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41051/
> ---
> 
> (Updated Dec. 7, 2015, 9:11 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Incorrect error message Expression not in GROUP BY key thrown instead of 
> Invalid function
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java dca3081 
>   ql/src/test/queries/clientnegative/invalid_select_fn.q PRE-CREATION 
>   ql/src/test/results/clientnegative/invalid_select_fn.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/41051/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 41051: HIVE-12570 : Incorrect error message Expression not in GROUP BY key thrown instead of Invalid function

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41051/#review110051
---



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (line 539)
<https://reviews.apache.org/r/41051/#comment169827>

Why don't we use functionName instead of expressionTree.getChild(0) ?


- John Pullokkaran


On Dec. 7, 2015, 9:11 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41051/
> ---
> 
> (Updated Dec. 7, 2015, 9:11 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Incorrect error message Expression not in GROUP BY key thrown instead of 
> Invalid function
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java dca3081 
>   ql/src/test/queries/clientnegative/invalid_select_fn.q PRE-CREATION 
>   ql/src/test/results/clientnegative/invalid_select_fn.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/41051/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 41051: HIVE-12570 : Incorrect error message Expression not in GROUP BY key thrown instead of Invalid function

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41051/#review110061
---

Ship it!


Ship It!

- John Pullokkaran


On Dec. 12, 2015, 12:53 a.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41051/
> ---
> 
> (Updated Dec. 12, 2015, 12:53 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Incorrect error message Expression not in GROUP BY key thrown instead of 
> Invalid function
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 945a6d7 
>   ql/src/test/queries/clientnegative/invalid_select_fn.q PRE-CREATION 
>   ql/src/test/results/clientnegative/invalid_select_fn.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/41051/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

[jira] [Created] (HIVE-12590) Repeated UDAFs with literals can produce incorrect result

[Laljo John Pullokkaran (JIRA)]

Laljo John Pullokkaran created HIVE-12590:
-

 Summary: Repeated UDAFs with literals can produce incorrect result
 Key: HIVE-12590
 URL: https://issues.apache.org/jira/browse/HIVE-12590
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Affects Versions: 1.2.1, 1.1.1, 1.0.1, 2.0.0
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran
Priority: Critical


Repeated UDAF with literals could produce wrong result.
This is not a common use case, nevertheless a bug.

hive> select max('pants'), max('pANTS') from t1 group by key;
 Total MapReduce CPU Time Spent: 0 msec
OK
pANTS   pANTS
pANTS   pANTS
pANTS   pANTS
pANTS   pANTS
pANTS   pANTS
Time taken: 296.252 seconds, Fetched: 5 row(s)






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40589: Implement limit push down through union all in CBO

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review108692
---



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 1090)
<https://reviews.apache.org/r/40589/#comment168147>

This should be conditionally turned off.
Limit push through Join is protected by "hive.optimize.limitjointranspose". 
Rename this to "hive.optimize.limitpushdown" or something similar.

You should then add this rule to where limit push through join is.


- John Pullokkaran


On Nov. 23, 2015, 5:59 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> ---
> 
> (Updated Nov. 23, 2015, 5:59 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11775
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java c005b1a 
>   ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_ppd_boolean.q 059da68 
>   ql/src/test/queries/clientpositive/parquet_ppd_char.q eaddcb4 
>   ql/src/test/queries/clientpositive/parquet_ppd_date.q 41d0d64 
>   ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486 
>   ql/src/test/queries/clientpositive/parquet_ppd_timestamp.q 1b9f6ff 
>   ql/src/test/queries/clientpositive/parquet_ppd_varchar.q 6449c6d 
>   ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/merge4.q.out 031376f 
>   ql/src/test/results/clientpositive/parquet_ppd_boolean.q.out 6e62ee4 
>   ql/src/test/results/clientpositive/parquet_ppd_char.q.out defaa9d 
>   ql/src/test/results/clientpositive/parquet_ppd_date.q.out 55231e9 
>   ql/src/test/results/clientpositive/parquet_ppd_decimal.q.out a30820e 
>   ql/src/test/results/clientpositive/parquet_ppd_timestamp.q.out 3b3e5b7 
>   ql/src/test/results/clientpositive/parquet_ppd_varchar.q.out 5a62e80 
>   ql/src/test/results/clientpositive/spark/temp_table.q.out 718a8a4 
>   ql/src/test/results/clientpositive/temp_table.q.out a9f2bae 
>   ql/src/test/results/clientpositive/tez/temp_table.q.out 200ccdd 
> 
> Diff: https://reviews.apache.org/r/40589/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 40589: Implement limit push down through union all in CBO

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review108698
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
 (line 110)
<https://reviews.apache.org/r/40589/#comment168155>

what if the previous limit got pushed past project/join due to 
limitjointranspose rules.

Then this will fire in a loop.

Limit-Union-Proj-Limit


- John Pullokkaran


On Nov. 23, 2015, 5:59 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> ---
> 
> (Updated Nov. 23, 2015, 5:59 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-11775
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java c005b1a 
>   ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/parquet_ppd_boolean.q 059da68 
>   ql/src/test/queries/clientpositive/parquet_ppd_char.q eaddcb4 
>   ql/src/test/queries/clientpositive/parquet_ppd_date.q 41d0d64 
>   ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486 
>   ql/src/test/queries/clientpositive/parquet_ppd_timestamp.q 1b9f6ff 
>   ql/src/test/queries/clientpositive/parquet_ppd_varchar.q 6449c6d 
>   ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/merge4.q.out 031376f 
>   ql/src/test/results/clientpositive/parquet_ppd_boolean.q.out 6e62ee4 
>   ql/src/test/results/clientpositive/parquet_ppd_char.q.out defaa9d 
>   ql/src/test/results/clientpositive/parquet_ppd_date.q.out 55231e9 
>   ql/src/test/results/clientpositive/parquet_ppd_decimal.q.out a30820e 
>   ql/src/test/results/clientpositive/parquet_ppd_timestamp.q.out 3b3e5b7 
>   ql/src/test/results/clientpositive/parquet_ppd_varchar.q.out 5a62e80 
>   ql/src/test/results/clientpositive/spark/temp_table.q.out 718a8a4 
>   ql/src/test/results/clientpositive/temp_table.q.out a9f2bae 
>   ql/src/test/results/clientpositive/tez/temp_table.q.out 200ccdd 
> 
> Diff: https://reviews.apache.org/r/40589/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

[jira] [Created] (HIVE-12559) Vectorization on MR produces different results

[Laljo John Pullokkaran (JIRA)]

Laljo John Pullokkaran created HIVE-12559:
-

 Summary: Vectorization on MR produces different results
 Key: HIVE-12559
 URL: https://issues.apache.org/jira/browse/HIVE-12559
 Project: Hive
  Issue Type: Bug
Reporter: Laljo John Pullokkaran
Assignee: Matt McCline


Vectorization on MR produces different results for semantically equivalent 
queries.

SET hive.vectorized.execution.enabled=true;
SET hive.auto.convert.join=true;
SET hive.auto.convert.join.noconditionaltask=true;
SET hive.auto.convert.join.noconditionaltask.size=10;
SET hive.cbo.enable=false;

select sum(v1.cdouble) from alltypesorc v3 join alltypesorc v1 on 
v1.csmallint=v3.csmallint join alltypesorc v2 on v1.ctinyint=v2.ctinyint;
-- Produces 6.065190932488167E11

select sum(v1.cdouble) from alltypesorc v1 join alltypesorc v2 on 
v1.ctinyint=v2.ctinyint join alltypesorc v3 on v1.csmallint=v3.csmallint;
-- Produces 6.065190932486892E11





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] New PMC Member : John Pullokkaran

[John Pullokkaran]

Thanks for all the wishes.
Glad to be part of Hive team.

Thanks
John

On 11/24/15, 10:47 PM, "Vaibhav Gumashta" <vgumas...@hortonworks.com>
wrote:

>Congrats John!
>
>
>
>
>
>On Tue, Nov 24, 2015 at 8:55 PM -0800, "Thejas Nair"
><thejas.n...@gmail.com<mailto:thejas.n...@gmail.com>> wrote:
>
>Congrats John!
>
>
>On Tue, Nov 24, 2015 at 4:41 PM, Pengcheng Xiong <pxi...@apache.org>
>wrote:
>> Congrats! John!
>>
>> On Wed, Nov 25, 2015 at 6:59 AM, Ashutosh Chauhan <hashut...@apache.org>
>> wrote:
>>
>>> On behalf of the Hive PMC I am delighted to announce John Pullokkaran
>>>is
>>> joining Hive PMC.
>>> John is a long time contributor in Hive and is focusing on compiler and
>>> optimizer areas these days.
>>> Please give John a warm welcome to the project!
>>>
>>> Ashutosh
>>>
>

[jira] [Created] (HIVE-12524) SemiJoinProjectTranspose Rule in CBO is not firing

[Laljo John Pullokkaran (JIRA)]

Laljo John Pullokkaran created HIVE-12524:
-

 Summary: SemiJoinProjectTranspose Rule in CBO is not firing
 Key: HIVE-12524
 URL: https://issues.apache.org/jira/browse/HIVE-12524
 Project: Hive
  Issue Type: Bug
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran


SemiJoinProjectTransposeRule uses LogicalProject and hence doesn't fire for CBO.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38211: HIVE-11684

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review107412
---

Ship it!


Ship It!

- John Pullokkaran


On Nov. 18, 2015, 10:40 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> ---
> 
> (Updated Nov. 18, 2015, 10:40 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Implement limit pushdown through outer join in CBO
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 953e52c4de49bbc083647033b370a1ca80f5cb97 
>   itests/src/test/resources/testconfiguration.properties 
> 402914cdfb4d72998a9c219e812d561e67612897 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java
>  82fdc0e04011aa585912ce5aae5e3245e5e5bb01 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
>  728c5aa91f8d6eab6bcc2696c059ae1bdbeea689 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java
>  715f24fc1872a54637a47bf80bb29e4a2e053779 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> de67b54a2c6cfd9bc4413ebf7f715e54c61b966f 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
>   ql/src/test/results/clientpositive/join32_lessSize.q.out 
> 5ea40241d329617220bbe775dcba91a9d7fb2f89 
>   ql/src/test/results/clientpositive/join_merging.q.out 
> b42bdc5ee92e9d725b7c0b8323c7763a55f3d059 
>   ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/spark/join32_lessSize.q.out 
> 937e8fcb0c7b89c13548a01e182176102ebcb681 
>   ql/src/test/results/clientpositive/spark/join_merging.q.out 
> 6c5089b4868e839a2e72546a39a7a5d4741464ed 
>   ql/src/test/results/clientpositive/subquery_notin.q.out 
> 5563794f9761a8269a56c7e716545c5edb1e4076 
>   ql/src/test/results/clientpositive/subquery_notin_having.q.java1.7.out 
> 9689ae37dfedc8c3becd6915e1e4cd5a5da29f9b 
>   ql/src/test/results/clientpositive/tez/explainuser_1.q.out 
> fa29dfec8fe38fcbec2cfa22859821898e07dbaa 
>   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38211/diff/
> 
> 
> Testing
> ---
> 
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q 
> (only Tez)
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

[jira] [Created] (HIVE-12478) Improve Hive/Calcite Trasitive Predicate inference

[Laljo John Pullokkaran (JIRA)]

Laljo John Pullokkaran created HIVE-12478:
-

 Summary: Improve Hive/Calcite Trasitive Predicate inference
 Key: HIVE-12478
 URL: https://issues.apache.org/jira/browse/HIVE-12478
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.2.1
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran


HiveJoinPushTransitivePredicatesRule does not pull up predicates for transitive 
inference if they contain more than one column.

EXPLAIN select * from srcpart join (select ds as ds, ds as `date` from srcpart 
where  (ds = '2008-04-08' and value=1)) s on (srcpart.ds = s.ds);




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40359: HIVE-11110 Cost Based Optimizer improvements

[John Pullokkaran]

> On Nov. 17, 2015, 12:08 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out, 
> > line 1951
> > <https://reviews.apache.org/r/40359/diff/1/?file=1127046#file1127046line1951>
> >
> > Partition pruning off?

With additional null predicate Transitive inference is not kicking in.
HIVE-12478 is the improvement request to track it.


> On Nov. 17, 2015, 12:08 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out, 
> > line 4371
> > <https://reviews.apache.org/r/40359/diff/1/?file=1127046#file1127046line4371>
> >
> > Partitioning pruning off?

With additional null predicate Transitive inference is not kicking in.
HIVE-12478 is the improvement request to track it.


- John


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40359/#review106759
-------


On Nov. 16, 2015, 6:54 p.m., John Pullokkaran wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40359/
> ---
> 
> (Updated Nov. 16, 2015, 6:54 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Hari Sankar Sivarama Subramaniyan, 
> and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-0
> https://issues.apache.org/jira/browse/HIVE-0
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-0 Introduces not null filters, improves filter selectivity 
> estimation, Streamlines pre-join order optimizations
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out d044c7e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> e1b60b0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> cce3588 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/TraitsUtil.java 
> be28828 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  4b7887a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePreFilteringRule.java
>  82d9600 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/FilterSelectivityEstimator.java
>  b52779c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
>  a17fb94 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  a8ff158 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java de67b54 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java e291a48 
>   ql/src/test/queries/clientpositive/special_character_in_tabnames_1.q 
> 7867ae1 
>   ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 623c2e8 
>   ql/src/test/results/clientnegative/sortmerge_mapjoin_mismatch_1.q.out 
> b2a7d89 
>   ql/src/test/results/clientpositive/allcolref_in_udf.q.out 216b037 
>   ql/src/test/results/clientpositive/ambiguous_col.q.out 7f04e89 
>   ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
> 20ccda5 
>   ql/src/test/results/clientpositive/annotate_stats_join.q.out ee05e6e 
>   ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out aa380b2 
>   ql/src/test/results/clientpositive/auto_join1.q.out 48ad641 
>   ql/src/test/results/clientpositive/auto_join10.q.out fa6f62d 
>   ql/src/test/results/clientpositive/auto_join12.q.out 7d8db0a 
>   ql/src/test/results/clientpositive/auto_join13.q.out 952dbf8 
>   ql/src/test/results/clientpositive/auto_join15.q.out 8e4b24c 
>   ql/src/test/results/clientpositive/auto_join16.q.out 1bad0f9 
>   ql/src/test/results/clientpositive/auto_join17.q.out e85cae8 
>   ql/src/test/results/clientpositive/auto_join19.q.out 8a57cb0 
>   ql/src/test/results/clientpositive/auto_join2.q.out abfc611 
>   ql/src/test/results/clientpositive/auto_join22.q.out bdee886 
>   ql/src/test/results/clientpositive/auto_join24.q.out 5b57303 
>   ql/src/test/results/clientpositive/auto_join26.q.out 94ab76f 
>   ql/src/test/results/clientpositive/auto_join3.q.out d015449 
>   ql/src/test/results/clientpositive/auto_join30.q.out 5437b7f 
>   ql/src/test/results/clientpositive/auto_join33.q.out 0dcd91d 
>   ql/src/test/results/clientpositive/auto_join4.q.out dbbee56 
>   ql/src/test/results/clientpositive/auto_join5.q.out 3209d07 
>   ql/src/test/results/clientpositive/auto_join8.q.out 2ca26aa 
>   ql/src/test/results/clientpositive

Re: Review Request 40359: HIVE-11110 Cost Based Optimizer improvements

[John Pullokkaran]

> On Nov. 17, 2015, 12:08 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/rand_partitionpruner3.q.out, lines 
> > 280-282
> > <https://reviews.apache.org/r/40359/diff/1/?file=1127111#file1127111line280>
> >
> > Incorrect result.

The test case is testing if Partition pruning logic takes out non-deterministic 
functions from experessions.
If you disable CBO & disable PPD you will get a different result that with 
CBO=false, PPD=true.

This is because random(1)<0.1 is non deterministic.
With this patch, CBO first performs partition pruning and then applies col 
pruning.
Col Pruning introduces a select on top of TS below filter. Hence the difference 
in result.

This is not a real issue


- John


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40359/#review106759
---


On Nov. 16, 2015, 6:54 p.m., John Pullokkaran wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40359/
> ---
> 
> (Updated Nov. 16, 2015, 6:54 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Hari Sankar Sivarama Subramaniyan, 
> and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-0
> https://issues.apache.org/jira/browse/HIVE-0
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-0 Introduces not null filters, improves filter selectivity 
> estimation, Streamlines pre-join order optimizations
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out d044c7e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> e1b60b0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> cce3588 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/TraitsUtil.java 
> be28828 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  4b7887a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePreFilteringRule.java
>  82d9600 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/FilterSelectivityEstimator.java
>  b52779c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
>  a17fb94 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  a8ff158 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java de67b54 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java e291a48 
>   ql/src/test/queries/clientpositive/special_character_in_tabnames_1.q 
> 7867ae1 
>   ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 623c2e8 
>   ql/src/test/results/clientnegative/sortmerge_mapjoin_mismatch_1.q.out 
> b2a7d89 
>   ql/src/test/results/clientpositive/allcolref_in_udf.q.out 216b037 
>   ql/src/test/results/clientpositive/ambiguous_col.q.out 7f04e89 
>   ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
> 20ccda5 
>   ql/src/test/results/clientpositive/annotate_stats_join.q.out ee05e6e 
>   ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out aa380b2 
>   ql/src/test/results/clientpositive/auto_join1.q.out 48ad641 
>   ql/src/test/results/clientpositive/auto_join10.q.out fa6f62d 
>   ql/src/test/results/clientpositive/auto_join12.q.out 7d8db0a 
>   ql/src/test/results/clientpositive/auto_join13.q.out 952dbf8 
>   ql/src/test/results/clientpositive/auto_join15.q.out 8e4b24c 
>   ql/src/test/results/clientpositive/auto_join16.q.out 1bad0f9 
>   ql/src/test/results/clientpositive/auto_join17.q.out e85cae8 
>   ql/src/test/results/clientpositive/auto_join19.q.out 8a57cb0 
>   ql/src/test/results/clientpositive/auto_join2.q.out abfc611 
>   ql/src/test/results/clientpositive/auto_join22.q.out bdee886 
>   ql/src/test/results/clientpositive/auto_join24.q.out 5b57303 
>   ql/src/test/results/clientpositive/auto_join26.q.out 94ab76f 
>   ql/src/test/results/clientpositive/auto_join3.q.out d015449 
>   ql/src/test/results/clientpositive/auto_join30.q.out 5437b7f 
>   ql/src/test/results/clientpositive/auto_join33.q.out 0dcd91d 
>   ql/src/test/results/clientpositive/auto_join4.q.out dbbee56 
>   ql/src/test/results/clientpositive/auto_join5.q.out 3209d07 
>   ql/src/test/results/clientpositive/auto_join8.q.out 2ca26aa 
>   ql/src/test/results/clientpositive/auto_join9.q.out 13dd5de 
>   ql/src/test/results/clientpositive/auto_join_reordering_va

Review Request 40359: HIVE-11110 Cost Based Optimizer improvements

[John Pullokkaran]

/vectorized_shufflejoin.q.out f32cc38 

Diff: https://reviews.apache.org/r/40359/diff/


Testing
---

HIVE QA


Thanks,

John Pullokkaran

[jira] [Created] (HIVE-12423) Remove Union Operator in Physical Compiler

[Laljo John Pullokkaran (JIRA)]

Laljo John Pullokkaran created HIVE-12423:
-

 Summary: Remove Union Operator in Physical Compiler
 Key: HIVE-12423
 URL: https://issues.apache.org/jira/browse/HIVE-12423
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 1.2.0
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40359: HIVE-11110 Cost Based Optimizer improvements

[John Pullokkaran]

> On Nov. 16, 2015, 9:43 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePreFilteringRule.java,
> >  line 138
> > <https://reviews.apache.org/r/40359/diff/1/?file=1126854#file1126854line138>
> >
> > Add returns a boolean, which can be used to indicate if expr was 
> > already present or not. Using that will avoid doing contains() always.

Good catch


> On Nov. 16, 2015, 9:43 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java,
> >  line 251
> > <https://reviews.apache.org/r/40359/diff/1/?file=1126851#file1126851line251>
> >
> > Seems like this will make ExprNodeColumnDesc generated in 
> > ExprNodeConverter to loose info about it being partition or virtual col or 
> > not. Is that intended?

It shouldn't cause any issues since expression is just used for Partition 
Pruning. But that is making assumptions about partition pruning internals. I 
have fixed this in latest patch.


- John


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40359/#review106736
---


On Nov. 16, 2015, 6:54 p.m., John Pullokkaran wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40359/
> ---
> 
> (Updated Nov. 16, 2015, 6:54 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Hari Sankar Sivarama Subramaniyan, 
> and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-0
> https://issues.apache.org/jira/browse/HIVE-0
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-0 Introduces not null filters, improves filter selectivity 
> estimation, Streamlines pre-join order optimizations
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out d044c7e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> e1b60b0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> cce3588 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/TraitsUtil.java 
> be28828 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  4b7887a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePreFilteringRule.java
>  82d9600 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/FilterSelectivityEstimator.java
>  b52779c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
>  a17fb94 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  a8ff158 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java de67b54 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java e291a48 
>   ql/src/test/queries/clientpositive/special_character_in_tabnames_1.q 
> 7867ae1 
>   ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 623c2e8 
>   ql/src/test/results/clientnegative/sortmerge_mapjoin_mismatch_1.q.out 
> b2a7d89 
>   ql/src/test/results/clientpositive/allcolref_in_udf.q.out 216b037 
>   ql/src/test/results/clientpositive/ambiguous_col.q.out 7f04e89 
>   ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
> 20ccda5 
>   ql/src/test/results/clientpositive/annotate_stats_join.q.out ee05e6e 
>   ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out aa380b2 
>   ql/src/test/results/clientpositive/auto_join1.q.out 48ad641 
>   ql/src/test/results/clientpositive/auto_join10.q.out fa6f62d 
>   ql/src/test/results/clientpositive/auto_join12.q.out 7d8db0a 
>   ql/src/test/results/clientpositive/auto_join13.q.out 952dbf8 
>   ql/src/test/results/clientpositive/auto_join15.q.out 8e4b24c 
>   ql/src/test/results/clientpositive/auto_join16.q.out 1bad0f9 
>   ql/src/test/results/clientpositive/auto_join17.q.out e85cae8 
>   ql/src/test/results/clientpositive/auto_join19.q.out 8a57cb0 
>   ql/src/test/results/clientpositive/auto_join2.q.out abfc611 
>   ql/src/test/results/clientpositive/auto_join22.q.out bdee886 
>   ql/src/test/results/clientpositive/auto_join24.q.out 5b57303 
>   ql/src/test/results/clientpositive/auto_join26.q.out 94ab76f 
>   ql/src/test/results/clientpositive/auto_join3.q.out d015449 
>   ql/src/test/results/clientpositive/auto_join30.q.out 5437b7f 
>   ql/src/test/results/clientpositive/auto_join33.q.out 0dcd91d 
&

Re: Review Request 40359: HIVE-11110 Cost Based Optimizer improvements

[John Pullokkaran]

> On Nov. 16, 2015, 9:43 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/FilterSelectivityEstimator.java,
> >  line 97
> > <https://reviews.apache.org/r/40359/diff/1/?file=1126855#file1126855line97>
> >
> > You want to change that to assert ? Throwing RTE in production for this 
> > recoverable change might not be ideal.

This is not a recoverable change for the given query.
It implies we have bad stats. Cost based optimization can not proceed with bad 
stats.


- John


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40359/#review106736
---


On Nov. 16, 2015, 6:54 p.m., John Pullokkaran wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40359/
> ---
> 
> (Updated Nov. 16, 2015, 6:54 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Hari Sankar Sivarama Subramaniyan, 
> and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-0
> https://issues.apache.org/jira/browse/HIVE-0
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-0 Introduces not null filters, improves filter selectivity 
> estimation, Streamlines pre-join order optimizations
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_queries.q.out d044c7e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> e1b60b0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> cce3588 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/TraitsUtil.java 
> be28828 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
>  4b7887a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePreFilteringRule.java
>  82d9600 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/FilterSelectivityEstimator.java
>  b52779c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
>  a17fb94 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  a8ff158 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java de67b54 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java e291a48 
>   ql/src/test/queries/clientpositive/special_character_in_tabnames_1.q 
> 7867ae1 
>   ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 623c2e8 
>   ql/src/test/results/clientnegative/sortmerge_mapjoin_mismatch_1.q.out 
> b2a7d89 
>   ql/src/test/results/clientpositive/allcolref_in_udf.q.out 216b037 
>   ql/src/test/results/clientpositive/ambiguous_col.q.out 7f04e89 
>   ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
> 20ccda5 
>   ql/src/test/results/clientpositive/annotate_stats_join.q.out ee05e6e 
>   ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out aa380b2 
>   ql/src/test/results/clientpositive/auto_join1.q.out 48ad641 
>   ql/src/test/results/clientpositive/auto_join10.q.out fa6f62d 
>   ql/src/test/results/clientpositive/auto_join12.q.out 7d8db0a 
>   ql/src/test/results/clientpositive/auto_join13.q.out 952dbf8 
>   ql/src/test/results/clientpositive/auto_join15.q.out 8e4b24c 
>   ql/src/test/results/clientpositive/auto_join16.q.out 1bad0f9 
>   ql/src/test/results/clientpositive/auto_join17.q.out e85cae8 
>   ql/src/test/results/clientpositive/auto_join19.q.out 8a57cb0 
>   ql/src/test/results/clientpositive/auto_join2.q.out abfc611 
>   ql/src/test/results/clientpositive/auto_join22.q.out bdee886 
>   ql/src/test/results/clientpositive/auto_join24.q.out 5b57303 
>   ql/src/test/results/clientpositive/auto_join26.q.out 94ab76f 
>   ql/src/test/results/clientpositive/auto_join3.q.out d015449 
>   ql/src/test/results/clientpositive/auto_join30.q.out 5437b7f 
>   ql/src/test/results/clientpositive/auto_join33.q.out 0dcd91d 
>   ql/src/test/results/clientpositive/auto_join4.q.out dbbee56 
>   ql/src/test/results/clientpositive/auto_join5.q.out 3209d07 
>   ql/src/test/results/clientpositive/auto_join8.q.out 2ca26aa 
>   ql/src/test/results/clientpositive/auto_join9.q.out 13dd5de 
>   ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
> f42b45e 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out d75d6c4 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out a0aefa3 
>   ql/src/test/r

Re: Review Request 38211: HIVE-11684

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review106333
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
 (line 60)
<https://reviews.apache.org/r/38211/#comment165065>

You don't have to add collation to traitdef; HiveSortLimit does this for 
you automatically.

Also the traits of sortnode from above should not be used; theoretically we 
should use the trait from project input. Practically Once sort is introduced 
only ordering trait can survive.


- John Pullokkaran


On Nov. 12, 2015, 10:47 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> ---
> 
> (Updated Nov. 12, 2015, 10:47 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Implement limit pushdown through outer join in CBO
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 7a8517b166185f5c501aceacc780dfab21de1c0a 
>   itests/src/test/resources/testconfiguration.properties 
> 70f96da40c3c5b74e3ad2dd640321a2967cfcb86 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> 90c2067e8279808fc0040e16d99a801b5d167eec 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java
>  82fdc0e04011aa585912ce5aae5e3245e5e5bb01 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
>  728c5aa91f8d6eab6bcc2696c059ae1bdbeea689 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java
>  715f24fc1872a54637a47bf80bb29e4a2e053779 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> de67b54a2c6cfd9bc4413ebf7f715e54c61b966f 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
>   ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38211/diff/
> 
> 
> Testing
> ---
> 
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q 
> (only Tez)
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 38211: HIVE-11684

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review106332
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
 (line 56)
<https://reviews.apache.org/r/38211/#comment165064>

Don't you need to merge the collation from top?
Otherwise you will top OB.


- John Pullokkaran


On Nov. 12, 2015, 10:47 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> ---
> 
> (Updated Nov. 12, 2015, 10:47 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Implement limit pushdown through outer join in CBO
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 7a8517b166185f5c501aceacc780dfab21de1c0a 
>   itests/src/test/resources/testconfiguration.properties 
> 70f96da40c3c5b74e3ad2dd640321a2967cfcb86 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> 90c2067e8279808fc0040e16d99a801b5d167eec 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java
>  82fdc0e04011aa585912ce5aae5e3245e5e5bb01 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
>  728c5aa91f8d6eab6bcc2696c059ae1bdbeea689 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java
>  715f24fc1872a54637a47bf80bb29e4a2e053779 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> de67b54a2c6cfd9bc4413ebf7f715e54c61b966f 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
>   ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38211/diff/
> 
> 
> Testing
> ---
> 
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q 
> (only Tez)
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

[jira] [Created] (HIVE-12384) Union Operator may produce incorrect result on TEZ

[Laljo John Pullokkaran (JIRA)]

Laljo John Pullokkaran created HIVE-12384:
-

 Summary: Union Operator may produce incorrect result on TEZ
 Key: HIVE-12384
 URL: https://issues.apache.org/jira/browse/HIVE-12384
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 0.14.0
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran


Union queries may produce incorrect result on TEZ.
TEZ removes union op, thus might loose the implicit cast in union op.

Reproduction test case:
set hive.cbo.enable=false;
set hive.execution.engine=tez;
select (x/sum(x) over())  as y from(select cast(1 as decimal(10,0))  as x from 
(select * from src limit 2)s1 union all select cast(1 as decimal(10,0)) x from 
(select * from src limit 2) s2 union all select '1' x from (select * 
from src limit 2) s3)u order by y;

select (x/sum(x) over()) as y from(select cast(1 as decimal(10,0))  as x from 
(select * from src limit 2)s1 union all select cast(1 as decimal(10,0)) x from 
(select * from src limit 2) s2 union all select cast (null as string) x from 
(select * from src limit 2) s3)u order by y;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 39425: Invalid column ref error in order by when using column alias in select clause and using having

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39425/#review106005
---

Ship it!


Ship It!

- John Pullokkaran


On Oct. 18, 2015, 9:47 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39425/
> ---
> 
> (Updated Oct. 18, 2015, 9:47 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive CLI session:
> {noformat}
> hive> create table abc(foo int, bar string);
> OK
> Time taken: 0.633 seconds
> hive> select foo as c0, count(*) as c1 from abc group by foo, bar having bar 
> like '%abc%' order by foo;
> FAILED: SemanticException [Error 10004]: Line 1:93 Invalid table alias or 
> column reference 'foo': (possible column names are: c0, c1)
> {noformat}
> Without having clause, the query runs fine, example:
> {code}
> select foo as c0, count(*) as c1 from abc group by foo, bar order by foo;
> {code}
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e13356c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 14a7e9c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 3262887 
>   ql/src/test/queries/clientnegative/alias_groupby_orderby.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/lineage3.q 70d4e57 
>   ql/src/test/results/clientnegative/alias_groupby_orderby.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/lineage3.q.out ad965c8 
> 
> Diff: https://reviews.apache.org/r/39425/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

[jira] [Created] (HIVE-12361) Hive 0.14: IndexOutOfBoundsException thrown when accessing a union all subquery and filtering on a column which does not exist in all underlying tables

[Laljo John Pullokkaran (JIRA)]

Laljo John Pullokkaran created HIVE-12361:
-

 Summary: Hive 0.14: IndexOutOfBoundsException thrown when 
accessing a union all subquery and filtering on a column which does not exist 
in all underlying tables
 Key: HIVE-12361
 URL: https://issues.apache.org/jira/browse/HIVE-12361
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 0.14.0
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran


This is a clone of HIVE-11603.
Creating a different bug to track fix for Hive 0.14.

Hive0.14 has simpler patch than 1.20..



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12356) Capture if a rule mutated the plan.

[Laljo John Pullokkaran (JIRA)]

Laljo John Pullokkaran created HIVE-12356:
-

 Summary: Capture if a rule mutated the plan.
 Key: HIVE-12356
 URL: https://issues.apache.org/jira/browse/HIVE-12356
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran


Currently Hive Optimizer doesn't capture if a rule mutated the plan.
This info could be useful in:
1. determining if a subsequent optimization rule needs to be run or not
   (Ex if Constant propagation didn't mutate plan, then don't run subsequent 
PPD)
2. Explain can contain info about which optimizations are applied effectively 
on the query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12355) Keep Obj Inspectors in Sync with RowSchema

[Laljo John Pullokkaran (JIRA)]

Laljo John Pullokkaran created HIVE-12355:
-

 Summary: Keep Obj Inspectors in Sync with RowSchema
 Key: HIVE-12355
 URL: https://issues.apache.org/jira/browse/HIVE-12355
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 1.2.1, 1.1.0, 1.0.0
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran


Currently Not all operators match their Output Obj inspectors to Row schema.
Many times OutputObjectInspectors may be more than needed.

This causes problems especially with union.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 39872: expanding view doesn't quote reserved keyword

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39872/#review105120
---

Ship it!


Ship It!

- John Pullokkaran


On Nov. 2, 2015, 7:25 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39872/
> ---
> 
> (Updated Nov. 2, 2015, 7:25 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> hive> create table testreserved (data struct<`end`:string, id: string>);
> OK
> Time taken: 0.274 seconds
> hive> create view testreservedview as select data.`end` as data_end, data.id 
> as data_id from testreserved;
> OK
> Time taken: 0.769 seconds
> hive> select data.`end` from testreserved;
> OK
> Time taken: 1.852 seconds
> hive> select data_id from testreservedview;
> NoViableAltException(98@[])
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:10858)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6438)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2907)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1373)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
>   at 
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
>   at 
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
> ...
> FAILED: SemanticException line 1:29 cannot recognize input near 'end' 'as' 
> 'data_end' in expression specification in definition of VIEW testreservedview 
> [
> select `testreserved`.`data`.end as `data_end`, `testreserved`.`data`.id as 
> `data_id` from `test`.`testreserved`
> ] used as testreservedview at Line 1:20
> 
> When view is expanded, field should be quote with backquote.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d2c3a7c 
> 
> Diff: https://reviews.apache.org/r/39872/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 37778: HIVE-11634

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/#review104741
---

Ship it!


- John Pullokkaran


On Oct. 29, 2015, 11:32 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37778/
> ---
> 
> (Updated Oct. 29, 2015, 11:32 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
> Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 59b66cd 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 7ee5081 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
> 4799b4d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
> 7cdc730 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
> 7262164 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
>   ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
>   ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
>   ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
> eca29df 
>   ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 
> 4320f01 
>   
> ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
>  e30465d 
>   ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
>   ql/src/test/results/clientpositive/tez/dynamic_partition_pruning.q.out 
> 7e28c77 
>   
> ql/src/test/results/clientpositive/tez/vectorized_dynamic_partition_pruning.q.out
>  1103e80 
> 
> Diff: https://reviews.apache.org/r/37778/diff/
> 
> 
> Testing
> ---
> 
> Local testing done. More unit tests coming in the next patch.
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 37778: HIVE-11634

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/#review104740
---

Ship it!


Ship It!

- John Pullokkaran


On Oct. 29, 2015, 11:32 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37778/
> ---
> 
> (Updated Oct. 29, 2015, 11:32 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
> Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 59b66cd 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 7ee5081 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
> 4799b4d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
> 7cdc730 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
> 7262164 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
>   ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
>   ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
>   ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
> eca29df 
>   ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 
> 4320f01 
>   
> ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
>  e30465d 
>   ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
>   ql/src/test/results/clientpositive/tez/dynamic_partition_pruning.q.out 
> 7e28c77 
>   
> ql/src/test/results/clientpositive/tez/vectorized_dynamic_partition_pruning.q.out
>  1103e80 
> 
> Diff: https://reviews.apache.org/r/37778/diff/
> 
> 
> Testing
> ---
> 
> Local testing done. More unit tests coming in the next patch.
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 37778: HIVE-11634

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/#review104502
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java 
(line 338)
<https://reviews.apache.org/r/37778/#comment162753>

If all expr nodes are partition cols/constants and if they come from same 
table then we should bail out.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java 
(line 470)
<https://reviews.apache.org/r/37778/#comment162754>

The col names in struct should be unique.
GetName would retutn class name and could result in data corruption as 
there could be duplicate names in struct typeinfo.

We should instead use ExprNodeGenericFuncDesc.newInsatnce which will take 
care of generating appropriate struct col names..


- John Pullokkaran


On Oct. 29, 2015, 11:32 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37778/
> ---
> 
> (Updated Oct. 29, 2015, 11:32 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
> Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 59b66cd 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 7ee5081 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
> 4799b4d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
> 7cdc730 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
> 7262164 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
>   ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
>   ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
>   ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
> eca29df 
>   ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 
> 4320f01 
>   
> ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
>  e30465d 
>   ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
>   ql/src/test/results/clientpositive/tez/dynamic_partition_pruning.q.out 
> 7e28c77 
>   
> ql/src/test/results/clientpositive/tez/vectorized_dynamic_partition_pruning.q.out
>  1103e80 
> 
> Diff: https://reviews.apache.org/r/37778/diff/
> 
> 
> Testing
> ---
> 
> Local testing done. More unit tests coming in the next patch.
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 39425: Invalid column ref error in order by when using column alias in select clause and using having

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39425/#review104096
---



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
<https://reviews.apache.org/r/39425/#comment162320>

The aliases should be added only in select that succeds the groupby. 
Otherwise other operators like having/winodwing could misuse the col alias.
Ex:
explain select key as c0, count(*) as c1, sum(c1) over() from t1 group by 
key, value having c1 like '%abc%' order by c0;

With this patch it will let this query through which is obviously wrong.


- John Pullokkaran


On Oct. 18, 2015, 9:47 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39425/
> ---
> 
> (Updated Oct. 18, 2015, 9:47 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Hive CLI session:
> {noformat}
> hive> create table abc(foo int, bar string);
> OK
> Time taken: 0.633 seconds
> hive> select foo as c0, count(*) as c1 from abc group by foo, bar having bar 
> like '%abc%' order by foo;
> FAILED: SemanticException [Error 10004]: Line 1:93 Invalid table alias or 
> column reference 'foo': (possible column names are: c0, c1)
> {noformat}
> Without having clause, the query runs fine, example:
> {code}
> select foo as c0, count(*) as c1 from abc group by foo, bar order by foo;
> {code}
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e13356c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 14a7e9c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 3262887 
>   ql/src/test/queries/clientnegative/alias_groupby_orderby.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/lineage3.q 70d4e57 
>   ql/src/test/results/clientnegative/alias_groupby_orderby.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/lineage3.q.out ad965c8 
> 
> Diff: https://reviews.apache.org/r/39425/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 39199: HIVE-12084 : Hive queries with ORDER BY and large LIMIT fails with OutOfMemoryError Java heap space

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39199/#review103649
---

Ship it!


Ship It!

- John Pullokkaran


On Oct. 22, 2015, 12:41 a.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39199/
> ---
> 
> (Updated Oct. 22, 2015, 12:41 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Please look at https://issues.apache.org/jira/browse/HIVE-12084
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TopNHash.java 484006a 
>   ql/src/test/queries/clientpositive/topn.q PRE-CREATION 
>   ql/src/test/results/clientpositive/topn.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/39199/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 38211: HIVE-11684

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review103667
---



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 997)
<https://reviews.apache.org/r/38211/#comment161733>

Redundant Limit removal should happen bottom up.


- John Pullokkaran


On Oct. 11, 2015, 12:57 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> ---
> 
> (Updated Oct. 11, 2015, 12:57 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Implement limit pushdown through outer join in CBO
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> bf48f69dd8914020dcd4eeba8dd448f4ee8d5b24 
>   itests/src/test/resources/testconfiguration.properties 
> ad47fac2e3e5716f2cdf4b9c25ee2d6d14eab5ec 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java
>  82fdc0e04011aa585912ce5aae5e3245e5e5bb01 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
>  6948fb16c6a3c9ce456a900536bee4c0d9b1614e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 61ee2bdf21000aa20d9fff02a3d926a1031410bc 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
>   ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38211/diff/
> 
> 
> Testing
> ---
> 
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q 
> (only Tez)
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 38211: HIVE-11684

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review103668
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
 (line 39)
<https://reviews.apache.org/r/38211/#comment161734>

We should enhance Join cardinality estimation for outer joins.

Outer Join cardinality should be: max(outer side cardinality, inner join 
cardinality)


- John Pullokkaran


On Oct. 11, 2015, 12:57 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> ---
> 
> (Updated Oct. 11, 2015, 12:57 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Implement limit pushdown through outer join in CBO
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> bf48f69dd8914020dcd4eeba8dd448f4ee8d5b24 
>   itests/src/test/resources/testconfiguration.properties 
> ad47fac2e3e5716f2cdf4b9c25ee2d6d14eab5ec 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java
>  82fdc0e04011aa585912ce5aae5e3245e5e5bb01 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
>  6948fb16c6a3c9ce456a900536bee4c0d9b1614e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 61ee2bdf21000aa20d9fff02a3d926a1031410bc 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
>   ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38211/diff/
> 
> 
> Testing
> ---
> 
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q 
> (only Tez)
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 38211: HIVE-11684

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review103714
---



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 1307)
<https://reviews.apache.org/r/38211/#comment161785>

May be all the config should be changed to LimitPushDown instead of using 
join; since limit can be pushed down below outer joins, projects, union, Map 
Side GB...


- John Pullokkaran


On Oct. 23, 2015, 3:21 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> ---
> 
> (Updated Oct. 23, 2015, 3:21 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Implement limit pushdown through outer join in CBO
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 23ae0dcca405bfe5521b1638f5c7f4e720c1a493 
>   itests/src/test/resources/testconfiguration.properties 
> fa1d89d2169023b3842829e762e02b896909b4b0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java
>  82fdc0e04011aa585912ce5aae5e3245e5e5bb01 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
>  6948fb16c6a3c9ce456a900536bee4c0d9b1614e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java
>  715f24fc1872a54637a47bf80bb29e4a2e053779 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> e13356ce16626bda1d9b6553ac5f191a65243394 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
>   ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38211/diff/
> 
> 
> Testing
> ---
> 
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q 
> (only Tez)
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 38211: HIVE-11684

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review103715
---



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 962)
<https://reviews.apache.org/r/38211/#comment161786>

Pushing just limit without pushing orderby is not sufficent.

Limit almost always will have preceeding order by.
Order by pushdown also nneds to be handled.


- John Pullokkaran


On Oct. 23, 2015, 3:21 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> ---
> 
> (Updated Oct. 23, 2015, 3:21 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Implement limit pushdown through outer join in CBO
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 23ae0dcca405bfe5521b1638f5c7f4e720c1a493 
>   itests/src/test/resources/testconfiguration.properties 
> fa1d89d2169023b3842829e762e02b896909b4b0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java
>  82fdc0e04011aa585912ce5aae5e3245e5e5bb01 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
>  6948fb16c6a3c9ce456a900536bee4c0d9b1614e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java
>  715f24fc1872a54637a47bf80bb29e4a2e053779 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> e13356ce16626bda1d9b6553ac5f191a65243394 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
>   ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38211/diff/
> 
> 
> Testing
> ---
> 
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q 
> (only Tez)
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 39199: HIVE-12084 : Hive queries with ORDER BY and large LIMIT fails with OutOfMemoryError Java heap space

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39199/#review102712
---



ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java (line 525)
<https://reviews.apache.org/r/39199/#comment160464>

This should be in ReduceSinkDesc since thats the one that has knowledge of 
reducer function


- John Pullokkaran


On Oct. 14, 2015, 6:10 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39199/
> ---
> 
> (Updated Oct. 14, 2015, 6:10 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Please look at https://issues.apache.org/jira/browse/HIVE-12084
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/PTFTopNHash.java f93b420 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java e33c1d4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TopNHash.java 484006a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java
>  49706b1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/VerifyTopNMemoryUsage.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java a60527b 
>   ql/src/test/queries/clientpositive/topn.q PRE-CREATION 
>   ql/src/test/results/clientpositive/topn.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/39199/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 39199: HIVE-12084 : Hive queries with ORDER BY and large LIMIT fails with OutOfMemoryError Java heap space

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39199/#review102714
---



ql/src/java/org/apache/hadoop/hive/ql/exec/PTFTopNHash.java (line 180)
<https://reviews.apache.org/r/39199/#comment160465>

This seems not right:
"partitionHeaps" seems to be allocating memory for each key TopNHash. So we 
would need to know how many distinct keys are present; i.e we need to know the 
NDV.
For each Key we going to store TopN.

If col stats is available then we should use it. Also we need to handle 
composite keys.

If stats are not available then may be a configurable heuristics may be an 
option; i.e lets say NDV is 10% of cardinality.


- John Pullokkaran


On Oct. 14, 2015, 6:10 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39199/
> ---
> 
> (Updated Oct. 14, 2015, 6:10 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Please look at https://issues.apache.org/jira/browse/HIVE-12084
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/PTFTopNHash.java f93b420 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java e33c1d4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TopNHash.java 484006a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java
>  49706b1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/VerifyTopNMemoryUsage.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java a60527b 
>   ql/src/test/queries/clientpositive/topn.q PRE-CREATION 
>   ql/src/test/results/clientpositive/topn.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/39199/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 39199: HIVE-12084 : Hive queries with ORDER BY and large LIMIT fails with OutOfMemoryError Java heap space

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39199/#review102716
---



ql/src/java/org/apache/hadoop/hive/ql/exec/TopNHash.java 
<https://reviews.apache.org/r/39199/#comment160467>

We may be better of by flushing the hash table if we can not fit in memory; 
this logic can be very similar to map side GB.


- John Pullokkaran


On Oct. 14, 2015, 6:10 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39199/
> ---
> 
> (Updated Oct. 14, 2015, 6:10 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Please look at https://issues.apache.org/jira/browse/HIVE-12084
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/PTFTopNHash.java f93b420 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java e33c1d4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TopNHash.java 484006a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java
>  49706b1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/VerifyTopNMemoryUsage.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java a60527b 
>   ql/src/test/queries/clientpositive/topn.q PRE-CREATION 
>   ql/src/test/results/clientpositive/topn.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/39199/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 39199: HIVE-12084 : Hive queries with ORDER BY and large LIMIT fails with OutOfMemoryError Java heap space

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39199/#review102527
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/VerifyTopNMemoryUsage.java
 (line 51)
<https://reviews.apache.org/r/39199/#comment160219>

1. Memory requirements for hashreducer should be in TopNHash & overridden 
by subclasses if needed.
2. We should use avgtuple size & remove dependency on key size
3. In runtime checks in TopNHash use memory required from #1


- John Pullokkaran


On Oct. 13, 2015, 7:37 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39199/
> ---
> 
> (Updated Oct. 13, 2015, 7:37 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Please look at https://issues.apache.org/jira/browse/HIVE-12084
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java
>  49706b1 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/VerifyTopNMemoryUsage.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java a60527b 
>   ql/src/test/queries/clientpositive/topn.q PRE-CREATION 
>   ql/src/test/results/clientpositive/topn.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/39199/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 38211: HIVE-11684

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review101952
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
 (line 4)
<https://reviews.apache.org/r/38211/#comment159463>

As we discussed:
we should seperate Limit push down from constraining condition. 

I.e first push limit as low as possible then pull it up in the subsequent 
phase if it doesn't meet reduction criteria.

Also the reduction criteria probably needs to specify the minimun no of 
tuples to reduce for limit to be cost effective (this should go away once we 
have execution cost model).


- John Pullokkaran


On Sept. 28, 2015, 11:40 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> ---
> 
> (Updated Sept. 28, 2015, 11:40 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Implement limit pushdown through outer join in CBO
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> dffdb5c14114d9c2d1e4c44a36f6b60966dcfc86 
>   itests/src/test/resources/testconfiguration.properties 
> 4f7b25f2563683933b9d677626f0447705927966 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 9c731b8c5e58570570812b525b8bf0edb0f4c0a9 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
>   ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38211/diff/
> 
> 
> Testing
> ---
> 
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q 
> (only Tez)
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 37778: HIVE-11634

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/#review101810
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
(line 384)
<https://reviews.apache.org/r/37778/#comment159305>

WALKERSTATE should be UNKNOWN.


- John Pullokkaran


On Oct. 7, 2015, 7:47 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37778/
> ---
> 
> (Updated Oct. 7, 2015, 7:47 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
> Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54a529e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 439f616 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
> d83636d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
> 825938a 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
> 7262164 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
>   ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
>   ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
>   ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
> eca29df 
>   ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
>   ql/src/test/results/clientpositive/pointlookup2.q.out 55edd90 
>   ql/src/test/results/clientpositive/pointlookup3.q.out 4cfb97e 
> 
> Diff: https://reviews.apache.org/r/37778/diff/
> 
> 
> Testing
> ---
> 
> Local testing done. More unit tests coming in the next patch.
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 37778: HIVE-11634

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/#review101814
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
(line 385)
<https://reviews.apache.org/r/37778/#comment159310>

Add unit test cases for:
1. DNF(CNF)
expr1 or expr2 (where expr2 is conjuctine tree)
2. In clause involving part/non-part col that are combined by disjunction


- John Pullokkaran


On Oct. 7, 2015, 7:47 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37778/
> ---
> 
> (Updated Oct. 7, 2015, 7:47 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
> Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54a529e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 439f616 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
> d83636d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
> 825938a 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
> 7262164 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
>   ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
>   ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
>   ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
> eca29df 
>   ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
>   ql/src/test/results/clientpositive/pointlookup2.q.out 55edd90 
>   ql/src/test/results/clientpositive/pointlookup3.q.out 4cfb97e 
> 
> Diff: https://reviews.apache.org/r/37778/diff/
> 
> 
> Testing
> ---
> 
> Local testing done. More unit tests coming in the next patch.
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 37778: HIVE-11634

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/#review101840
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
(line 392)
<https://reviews.apache.org/r/37778/#comment159360>

This too should be unknown right?


- John Pullokkaran


On Oct. 7, 2015, 10:57 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37778/
> ---
> 
> (Updated Oct. 7, 2015, 10:57 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
> Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java bf48f69 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 439f616 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
> d83636d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
> 825938a 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
> 7262164 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
>   ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
>   ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
>   ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
> eca29df 
>   ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
>   ql/src/test/results/clientpositive/pointlookup2.q.out 55edd90 
>   ql/src/test/results/clientpositive/pointlookup3.q.out 4cfb97e 
> 
> Diff: https://reviews.apache.org/r/37778/diff/
> 
> 
> Testing
> ---
> 
> Local testing done. More unit tests coming in the next patch.
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 38211: HIVE-11684

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review101851
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
 (line 80)
<https://reviews.apache.org/r/38211/#comment159364>

"reduction" as it is set is not a %.
Shouldn't you multiply by ((100-reduction)/100)

guarding for reduction to be <= 100.


- John Pullokkaran


On Sept. 28, 2015, 11:40 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> ---
> 
> (Updated Sept. 28, 2015, 11:40 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Implement limit pushdown through outer join in CBO
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> dffdb5c14114d9c2d1e4c44a36f6b60966dcfc86 
>   itests/src/test/resources/testconfiguration.properties 
> 4f7b25f2563683933b9d677626f0447705927966 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 9c731b8c5e58570570812b525b8bf0edb0f4c0a9 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
>   ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38211/diff/
> 
> 
> Testing
> ---
> 
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q 
> (only Tez)
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 37778: HIVE-11634

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/#review101804
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
(line 377)
<https://reviews.apache.org/r/37778/#comment159293>

1. We shouldn't remove non deterministic UDFs.
2. Also make sure that partitionPruner isn't considering non deterministic 
UDFs
3. Please add testcase that involves non deterministic udf
4. Add a test case involving conditional expressions (case, If..)
Ex: select key from t1 where (case when key=1 then 10 else 20 end) in 
(10,11);


- John Pullokkaran


On Oct. 1, 2015, 11:42 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37778/
> ---
> 
> (Updated Oct. 1, 2015, 11:42 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
> Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java dffdb5c 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 439f616 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
> d83636d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
> 825938a 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
> 7262164 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
>   ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
>   ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
>   ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
> eca29df 
>   ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
>   ql/src/test/results/clientpositive/pointlookup2.q.out 55edd90 
>   ql/src/test/results/clientpositive/pointlookup3.q.out 4cfb97e 
> 
> Diff: https://reviews.apache.org/r/37778/diff/
> 
> 
> Testing
> ---
> 
> Local testing done. More unit tests coming in the next patch.
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Review Request 39004: HIVE-11919: Union type during Union OP inits may choose type different than decided during plan gen.

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39004/
---

Review request for hive and Jesús Camacho Rodríguez.


Bugs: HIVE-11919
https://issues.apache.org/jira/browse/HIVE-11919


Repository: hive-git


Description
---

HIVE-11919: Union type during Union OP inits may choose type different than 
decided during plan gen.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnInfo.java 12bb1d7 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 9c731b8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java dbc6d8f 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUtils.java 
222e0e0 
  ql/src/test/queries/clientpositive/union36.q PRE-CREATION 
  ql/src/test/results/clientpositive/union36.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/39004/diff/


Testing
---

QA Run
Added unit tests


Thanks,

John Pullokkaran

Re: Review Request 38940: unable to create temporary table using CTAS if regular table with that name already exists

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38940/#review101559
---

Ship it!


Ship It!

- John Pullokkaran


On Oct. 1, 2015, 10:50 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38940/
> ---
> 
> (Updated Oct. 1, 2015, 10:50 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> CTAS temporary table query fails if regular table with the same name already 
> exists. 
> Steps to reproduce the issue:
> {noformat}
> hive> use dbtemptable;
> OK
> Time taken: 0.273 seconds
> hive> create table a(i int);
> OK
> Time taken: 0.297 seconds
> hive> create temporary table a(i int);
> OK
> Time taken: 0.165 seconds
> hive> create table b(i int);
> OK
> Time taken: 0.212 seconds
> hive> create temporary table b as select * from a;
> FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: 
> Table already exists: dbtemptable.b
> hive> create table c(i int);
> OK
> Time taken: 0.264 seconds
> hive> create temporary table b as select * from c;
> FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: 
> Table already exists: dbtemptable.b
> {noformat}
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  51ff262 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 4bec228 
>   ql/src/test/queries/clientpositive/temp_table.q e587f3f 
>   ql/src/test/results/clientpositive/spark/temp_table.q.out 65e256d 
>   ql/src/test/results/clientpositive/temp_table.q.out e2987fe 
>   ql/src/test/results/clientpositive/tez/temp_table.q.out 49f57c2 
> 
> Diff: https://reviews.apache.org/r/38940/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 37778: HIVE-11634

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/#review101292
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
(line 372)
<https://reviews.apache.org/r/37778/#comment158671>

1. The check "children.size() < 2" is either unnecessary or we have a bug 
some where else.
2. We should also remove single col in clause where col is partition col


- John Pullokkaran


On Oct. 1, 2015, 11:42 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37778/
> ---
> 
> (Updated Oct. 1, 2015, 11:42 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
> Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java dffdb5c 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 439f616 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
> d83636d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
> 825938a 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
> 7262164 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
>   ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
>   ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
>   ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
> eca29df 
>   ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
>   ql/src/test/results/clientpositive/pointlookup2.q.out 55edd90 
>   ql/src/test/results/clientpositive/pointlookup3.q.out 4cfb97e 
> 
> Diff: https://reviews.apache.org/r/37778/diff/
> 
> 
> Testing
> ---
> 
> Local testing done. More unit tests coming in the next patch.
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 38503: Support special characters in quoted table names

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38503/#review101162
---

Ship it!


Ship It!

- John Pullokkaran


On Sept. 30, 2015, 9:57 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38503/
> ---
> 
> (Updated Sept. 30, 2015, 9:57 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Right now table names can only be "[a-zA-z_0-9]+". This patch tries to 
> investigate how much change there should be if we would like to support 
> special characters, e.g., "/" in table names.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f29da2 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
> ee20430 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> a80f686 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> f88f4dd 
>   metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java bc0f6e3 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 43159c6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 210736b 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java be5a593 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fadd074 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c78e8f4 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 52ed4a3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java
>  4966d89 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
> 8b7a2e8 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 1e2feaa 
>   ql/src/test/queries/clientnegative/special_character_in_tabnames_1.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/special_character_in_tabnames_1.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/special_character_in_tabnames_2.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/special_character_in_tabnames_3.q 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/special_character_in_tabnames_1.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/special_character_in_tabnames_1.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/special_character_in_tabnames_2.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/special_character_in_tabnames_3.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38503/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 38503: Support special characters in quoted table names

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38503/#review101045
---



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 2177)
<https://reviews.apache.org/r/38503/#comment158352>

We should list what special chars are supported; may be by accessing 
MetaStoreUtils field.


- John Pullokkaran


On Sept. 22, 2015, 6:12 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38503/
> ---
> 
> (Updated Sept. 22, 2015, 6:12 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Right now table names can only be "[a-zA-z_0-9]+". This patch tries to 
> investigate how much change there should be if we would like to support 
> special characters, e.g., "/" in table names.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f29da2 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
> ee20430 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> a80f686 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> f88f4dd 
>   metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java bc0f6e3 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 43159c6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 210736b 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java be5a593 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fadd074 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c78e8f4 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 52ed4a3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java
>  4966d89 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
> 8b7a2e8 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 1e2feaa 
>   ql/src/test/queries/clientnegative/special_character_in_tabnames_1.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/special_character_in_tabnames_1.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/special_character_in_tabnames_2.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/special_character_in_tabnames_3.q 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/special_character_in_tabnames_1.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/special_character_in_tabnames_1.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/special_character_in_tabnames_2.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/special_character_in_tabnames_3.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38503/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 38503: Support special characters in quoted table names

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38503/#review101046
---



metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java (line 
1737)
<https://reviews.apache.org/r/38503/#comment158353>

Make a note that unicode is not supported in table names. And we have 
explicit checks for it.


- John Pullokkaran


On Sept. 22, 2015, 6:12 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38503/
> ---
> 
> (Updated Sept. 22, 2015, 6:12 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Right now table names can only be "[a-zA-z_0-9]+". This patch tries to 
> investigate how much change there should be if we would like to support 
> special characters, e.g., "/" in table names.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f29da2 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
> ee20430 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> a80f686 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> f88f4dd 
>   metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java bc0f6e3 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 43159c6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 210736b 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java be5a593 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fadd074 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c78e8f4 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 52ed4a3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java
>  4966d89 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
> 8b7a2e8 
>   ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 1e2feaa 
>   ql/src/test/queries/clientnegative/special_character_in_tabnames_1.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/special_character_in_tabnames_1.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/special_character_in_tabnames_2.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/special_character_in_tabnames_3.q 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/special_character_in_tabnames_1.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/special_character_in_tabnames_1.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/special_character_in_tabnames_2.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/special_character_in_tabnames_3.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38503/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 38768: HIVE-11964: RelOptHiveTable.hiveColStatsMap might contain mismatched column stats

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38768/#review101012
---

Ship it!


Ship It!

- John Pullokkaran


On Sept. 25, 2015, 6:37 p.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38768/
> ---
> 
> (Updated Sept. 25, 2015, 6:37 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11964
> https://issues.apache.org/jira/browse/HIVE-11964
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The stats in hiveColStats should be ordered in that of the columns in 
> nonPartColIndxsThatRqrStats/nonPartColNamesThatRqrStats before being put in 
> the hiveColStatsMap
> ===
> for (int i = 0; i < hiveColStats.size(); i++) {
>hiveColStatsMap.put(nonPartColIndxsThatRqrStats.get(i), 
> hiveColStats.get(i));
> }
> 
> Since hiveColStats = StatsUtils.getTableColumnStats(hiveTblMetadata, 
> hiveNonPartitionCols, nonPartColNamesThatRqrStats); might not return the 
> stats in same order as those in nonPartColNamesThatRqrStats.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 6c0bd25 
> 
> Diff: https://reviews.apache.org/r/38768/diff/
> 
> 
> Testing
> ---
> 
> Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Re: Review Request 38768: HIVE-11964: RelOptHiveTable.hiveColStatsMap might contain mismatched column stats

[John Pullokkaran]

> On Sept. 25, 2015, 6:52 p.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java,
> >  line 315
> > <https://reviews.apache.org/r/38768/diff/1/?file=1084857#file1084857line315>
> >
> > Shouldn't we fix the API to provide the stats in the same order as it 
> > was given?
> > 
> > In this case cols are given in the same order as it appears in the 
> > table.
> 
> Chaoyu Tang wrote:
> You meant to fix the HMS getTableColumnStatistics API to return the stats 
> in the order of the requested columns? If so, what if some requested columns 
> do not have stats and miss in the returned list? In this case, the ordering 
> seems not much necessary or helpful. In addition, not all 
> getTableColumnStatistics callers are interested in the order of the returned 
> stats, so I think it probably more reasonable to leave this ordering burden 
> to its caller instead of API itself. Does it makes sense? Please let me know.

Thats fair.


- John


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38768/#review100654
---


On Sept. 25, 2015, 6:37 p.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38768/
> ---
> 
> (Updated Sept. 25, 2015, 6:37 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11964
> https://issues.apache.org/jira/browse/HIVE-11964
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The stats in hiveColStats should be ordered in that of the columns in 
> nonPartColIndxsThatRqrStats/nonPartColNamesThatRqrStats before being put in 
> the hiveColStatsMap
> ===
> for (int i = 0; i < hiveColStats.size(); i++) {
>hiveColStatsMap.put(nonPartColIndxsThatRqrStats.get(i), 
> hiveColStats.get(i));
> }
> 
> Since hiveColStats = StatsUtils.getTableColumnStats(hiveTblMetadata, 
> hiveNonPartitionCols, nonPartColNamesThatRqrStats); might not return the 
> stats in same order as those in nonPartColNamesThatRqrStats.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 6c0bd25 
> 
> Diff: https://reviews.apache.org/r/38768/diff/
> 
> 
> Testing
> ---
> 
> Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Re: Review Request 38768: HIVE-11964: RelOptHiveTable.hiveColStatsMap might contain mismatched column stats

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38768/#review100685
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
(line 315)
<https://reviews.apache.org/r/38768/#comment157948>

yes.


- John Pullokkaran


On Sept. 25, 2015, 6:37 p.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38768/
> ---
> 
> (Updated Sept. 25, 2015, 6:37 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11964
> https://issues.apache.org/jira/browse/HIVE-11964
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The stats in hiveColStats should be ordered in that of the columns in 
> nonPartColIndxsThatRqrStats/nonPartColNamesThatRqrStats before being put in 
> the hiveColStatsMap
> ===
> for (int i = 0; i < hiveColStats.size(); i++) {
>hiveColStatsMap.put(nonPartColIndxsThatRqrStats.get(i), 
> hiveColStats.get(i));
> }
> 
> Since hiveColStats = StatsUtils.getTableColumnStats(hiveTblMetadata, 
> hiveNonPartitionCols, nonPartColNamesThatRqrStats); might not return the 
> stats in same order as those in nonPartColNamesThatRqrStats.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 6c0bd25 
> 
> Diff: https://reviews.apache.org/r/38768/diff/
> 
> 
> Testing
> ---
> 
> Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Re: Review Request 38768: HIVE-11964: RelOptHiveTable.hiveColStatsMap might contain mismatched column stats

[John Pullokkaran]

> On Sept. 25, 2015, 6:55 p.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java,
> >  line 316
> > <https://reviews.apache.org/r/38768/diff/1/?file=1084857#file1084857line316>
> >
> > Also why is this only at the else condition?
> > Are we assuming that if the requested cols & returned cols size matches 
> > then order is preserved.
> > 
> > This seems rather shaky assumptions.
> > Its better to fix stats API to guarantee the order.
> 
> Chaoyu Tang wrote:
> No, I did not assume that. The reasons I put the logic in this else 
> conditiion are because:
> 1. only when the requested cols & returned cols size matches, the stats 
> is mapped to an index and put into the hiveColStatsMap, see following code in 
> updateColStats (line 363 -367)
> ==
>   if (hiveColStats != null && hiveColStats.size() == 
> nonPartColNamesThatRqrStats.size()) {
> for (int i = 0; i < hiveColStats.size(); i++) {
>   // the columns in 
> nonPartColIndxsThatRqrStats/nonPartColNamesThatRqrStats/hiveColStats
>   // are in same order
>   hiveColStatsMap.put(nonPartColIndxsThatRqrStats.get(i), 
> hiveColStats.get(i));
> }
>   }
> ==
> 
> 2. For partitioned table, the hiveColStats is already built in the order 
> of nonPartColNamesThatRqrStats. There is not necessary to reorder it.
> ==
>   for (String c : nonPartColNamesThatRqrStats) {
>   ColStatistics cs = stats.getColumnStatisticsFromColName(c);
>   if (cs != null) {
> hiveColStats.add(cs);
>   } else {
> colNamesFailedStats.add(c);
>   }
>}
> ==
> 
> 3. HMS stats API might not be a proper place to enforce the order as I 
> explained in last comment.

ok


- John


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38768/#review100655
---


On Sept. 25, 2015, 6:37 p.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38768/
> ---
> 
> (Updated Sept. 25, 2015, 6:37 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11964
> https://issues.apache.org/jira/browse/HIVE-11964
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The stats in hiveColStats should be ordered in that of the columns in 
> nonPartColIndxsThatRqrStats/nonPartColNamesThatRqrStats before being put in 
> the hiveColStatsMap
> ===
> for (int i = 0; i < hiveColStats.size(); i++) {
>hiveColStatsMap.put(nonPartColIndxsThatRqrStats.get(i), 
> hiveColStats.get(i));
> }
> 
> Since hiveColStats = StatsUtils.getTableColumnStats(hiveTblMetadata, 
> hiveNonPartitionCols, nonPartColNamesThatRqrStats); might not return the 
> stats in same order as those in nonPartColNamesThatRqrStats.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 6c0bd25 
> 
> Diff: https://reviews.apache.org/r/38768/diff/
> 
> 
> Testing
> ---
> 
> Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Re: Review Request 38768: HIVE-11964: RelOptHiveTable.hiveColStatsMap might contain mismatched column stats

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38768/#review100655
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
(line 316)
<https://reviews.apache.org/r/38768/#comment157885>

Also why is this only at the else condition?
Are we assuming that if the requested cols & returned cols size matches 
then order is preserved.

This seems rather shaky assumptions.
Its better to fix stats API to guarantee the order.


- John Pullokkaran


On Sept. 25, 2015, 6:37 p.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38768/
> ---
> 
> (Updated Sept. 25, 2015, 6:37 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11964
> https://issues.apache.org/jira/browse/HIVE-11964
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The stats in hiveColStats should be ordered in that of the columns in 
> nonPartColIndxsThatRqrStats/nonPartColNamesThatRqrStats before being put in 
> the hiveColStatsMap
> ===
> for (int i = 0; i < hiveColStats.size(); i++) {
>hiveColStatsMap.put(nonPartColIndxsThatRqrStats.get(i), 
> hiveColStats.get(i));
> }
> 
> Since hiveColStats = StatsUtils.getTableColumnStats(hiveTblMetadata, 
> hiveNonPartitionCols, nonPartColNamesThatRqrStats); might not return the 
> stats in same order as those in nonPartColNamesThatRqrStats.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 6c0bd25 
> 
> Diff: https://reviews.apache.org/r/38768/diff/
> 
> 
> Testing
> ---
> 
> Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Re: Review Request 38768: HIVE-11964: RelOptHiveTable.hiveColStatsMap might contain mismatched column stats

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38768/#review100654
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
(line 315)
<https://reviews.apache.org/r/38768/#comment157883>

Shouldn't we fix the API to provide the stats in the same order as it was 
given?

In this case cols are given in the same order as it appears in the table.


- John Pullokkaran


On Sept. 25, 2015, 6:37 p.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38768/
> ---
> 
> (Updated Sept. 25, 2015, 6:37 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11964
> https://issues.apache.org/jira/browse/HIVE-11964
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The stats in hiveColStats should be ordered in that of the columns in 
> nonPartColIndxsThatRqrStats/nonPartColNamesThatRqrStats before being put in 
> the hiveColStatsMap
> ===
> for (int i = 0; i < hiveColStats.size(); i++) {
>hiveColStatsMap.put(nonPartColIndxsThatRqrStats.get(i), 
> hiveColStats.get(i));
> }
> 
> Since hiveColStats = StatsUtils.getTableColumnStats(hiveTblMetadata, 
> hiveNonPartitionCols, nonPartColNamesThatRqrStats); might not return the 
> stats in same order as those in nonPartColNamesThatRqrStats.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 6c0bd25 
> 
> Diff: https://reviews.apache.org/r/38768/diff/
> 
> 
> Testing
> ---
> 
> Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Re: Review Request 38211: HIVE-11684

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review100456
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
 (line 172)
<https://reviews.apache.org/r/38211/#comment157655>

We should seperate Limit push through Project from Limit Push through Join. 
Looking for Limit/Proj/Join is a hard coupling.


- John Pullokkaran


On Sept. 17, 2015, 1:55 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> ---
> 
> (Updated Sept. 17, 2015, 1:55 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Implement limit pushdown through outer join in CBO
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 7f29da29f861c0e6c11065a95092d9e7e7b08aba 
>   itests/src/test/resources/testconfiguration.properties 
> 0d3e1cc17df496e8b4dd8c3622017cd15384348a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 0a7ce3a5f81f3175cddc59319753be3bb0b2af04 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
>   ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38211/diff/
> 
> 
> Testing
> ---
> 
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q 
> (only Tez)
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 38211: HIVE-11684

[John Pullokkaran]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review100455
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
 (line 73)
<https://reviews.apache.org/r/38211/#comment157654>

May be reuse HiveCalciteUtil.orderRelNode


- John Pullokkaran


On Sept. 17, 2015, 1:55 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> ---
> 
> (Updated Sept. 17, 2015, 1:55 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Implement limit pushdown through outer join in CBO
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> 7f29da29f861c0e6c11065a95092d9e7e7b08aba 
>   itests/src/test/resources/testconfiguration.properties 
> 0d3e1cc17df496e8b4dd8c3622017cd15384348a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 0a7ce3a5f81f3175cddc59319753be3bb0b2af04 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
>   ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38211/diff/
> 
> 
> Testing
> ---
> 
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q 
> (only Tez)
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

[jira] [Created] (HIVE-11919) Hive Union Type Mismatch

[Laljo John Pullokkaran (JIRA)]

Laljo John Pullokkaran created HIVE-11919:
-

 Summary: Hive Union Type Mismatch
 Key: HIVE-11919
 URL: https://issues.apache.org/jira/browse/HIVE-11919
 Project: Hive
  Issue Type: Bug
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran


In Hive for union right most type wins out for most primitive types during plan 
gen. However when union op gets initialized the type gets switched.

This could result in bad data & type exceptions.

This happens only in non cbo mode.
In CBO mode, Hive would add explicit type casts that would prevent such type 
issues.

Sample Query: 
select cd/sum(cd) over() from(select cd from u1 union all select cd from u2 
union all select cd from u3)u4;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)