Re: Review Request 59984: Improve plans for subqueries with non-equi co-related predicates

2017-06-19 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59984/#review178248
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java
Lines 137-139 (patched)


Seems like this if branch is unnecessary.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java
Lines 47-49 (patched)


Seems like this if branch is unnecessary.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java
Lines 46-48 (patched)


Seems like this if branch is unnecessary.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
Lines 102 (patched)


We use org.slf4j.Logger;



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
Line 1236 (original), 1257 (patched)


Lets use RexSimplify.simplify() here.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
Line 1384 (original), 1419 (patched)


Add assert corDef.getPredicateKind() == SqlKind.EQUALS



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
Lines 1868 (patched)


Seems like value generator shouldn't have generated those predicates at 
all. Lets leave a TODO to remove this logic once value generator is fixed to 
not generate unnecessary predicates.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
Lines 3088 (patched)


Might be better to initialize this variable to true. Since, value being 
false is known and limited set.


- Ashutosh Chauhan


On June 11, 2017, 7:17 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59984/
> ---
> 
> (Updated June 11, 2017, 7:17 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16838
> https://issues.apache.org/jira/browse/HIVE-16838
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch improves plans for subqueries which have not equal corelated 
> predicates. 
> Currently to retrieve all possible correlated predicates inner table is 
> joined with outer query. This is un-necessary in most of the cases (exception 
> is if subquery has an aggregate).
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java
>  63bbdaccfb 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java
>  19e1e026f4 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java
>  7cfb007a9d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
>  4c99932759 
>   ql/src/test/queries/clientpositive/subquery_in.q 4ba170a706 
>   ql/src/test/results/clientpositive/constprog_partitioner.q.out 8c7f9d3f29 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 8b04bc9261 
>   ql/src/test/results/clientpositive/llap/subquery_exists.q.out 3004e36c9d 
>   ql/src/test/results/clientpositive/llap/subquery_in.q.out 1f9c9e4474 
>   ql/src/test/results/clientpositive/llap/subquery_multi.q.out 29516eff82 
>   ql/src/test/results/clientpositive/llap/subquery_notin.q.out b4af91579b 
>   ql/src/test/results/clientpositive/llap/subquery_scalar.q.out b78df8b9f5 
>   ql/src/test/results/clientpositive/llap/subquery_select.q.out 202980e975 
>   ql/src/test/results/clientpositive/llap/subquery_views.q.out 1a21a02a30 
>   ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out 
> d3586e0db2 
>   ql/src/test/results/clientpositive/perf/query16.q.out a7f93f9ec2 
>   ql/src/test/results/clientpositive/perf/query94.q.out c5fc9e7f00 
>   ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out 
> 3467215d63 
>   ql/src/test/results/clientpositive/spark/subquery_exists.q.out 8768b45166 
>   ql/src/test/results/clientpositive/spark/subquery_in.q.out 80a350656d 
>   ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out 
> 2f2609f03e 
>   ql/src/test/results/clientpositive/subquery_exists.q.out cfc76520ce 
>   ql/src/test/results/clientpositive/subquery_exists_having.q.out 2c41ff6c33 
>   ql/src/test/results/clientpositive/subquery_in_having.q.out c4569ba035 
>

Re: Review Request 59808: Enhance HiveFilterSetOpTransposeRule to remove union branches

2017-06-19 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59808/
---

(Updated June 19, 2017, 6:11 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-16797


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 1f6939bc91 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSetOpTransposeRule.java
 3ee29e0482 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveUnionMergeRule.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 348331e052 
  ql/src/test/queries/clientpositive/filter_union.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query11.q PRE-CREATION 
  ql/src/test/results/clientpositive/filter_aggr.q.out db7dcaed3f 
  ql/src/test/results/clientpositive/filter_union.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 8b04bc9261 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out e3f70b097f 
  ql/src/test/results/clientpositive/llap/filter_union.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 5382c42412 
  ql/src/test/results/clientpositive/llap/tez_union_multiinsert.q.out 
14e8e4389f 
  ql/src/test/results/clientpositive/perf/query11.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query14.q.out 048a17f92f 
  ql/src/test/results/clientpositive/perf/query23.q.out 1fd8cb4f25 
  ql/src/test/results/clientpositive/perf/query33.q.out c1a5fa28ed 
  ql/src/test/results/clientpositive/perf/query4.q.out 1b2048649a 
  ql/src/test/results/clientpositive/perf/query5.q.out a3f2d58fec 
  ql/src/test/results/clientpositive/perf/query56.q.out 4ec7201fa7 
  ql/src/test/results/clientpositive/perf/query60.q.out 12d8cdd9b4 
  ql/src/test/results/clientpositive/perf/query71.q.out 44658081b5 
  ql/src/test/results/clientpositive/perf/query74.q.out bb4a71e6ce 
  ql/src/test/results/clientpositive/perf/query76.q.out dcd5004166 
  ql/src/test/results/clientpositive/perf/query77.q.out d46ba6b13c 
  ql/src/test/results/clientpositive/perf/query80.q.out 3cf41f3fed 
  ql/src/test/results/clientpositive/spark/union30.q.out 12eda1d3b6 
  ql/src/test/results/clientpositive/tez/explainanalyze_2.q.out f6844c4a38 
  ql/src/test/results/clientpositive/union24.q.out d6b1a79b20 
  ql/src/test/results/clientpositive/union30.q.out 26a27c8e15 
  ql/src/test/results/clientpositive/union34.q.out 9d593315af 
  ql/src/test/results/clientpositive/unionall_unbalancedppd.q.out b3e128a3d6 


Diff: https://reviews.apache.org/r/59808/diff/3/

Changes: https://reviews.apache.org/r/59808/diff/2-3/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 59808: Enhance HiveFilterSetOpTransposeRule to remove union branches

2017-06-19 Thread pengcheng xiong 


> On June 18, 2017, 8:30 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/filter_union.q.out
> > Lines 39-43 (patched)
> > 
> >
> > Metadataonly optimizer should have kicked in and turned this to null 
> > scan. Seems like it didn't. Can you confirm with explain extended?

Yes, it is kicked in. I have updated the q file to show explain extended. As 
you will see, for 2 branches out of 4 branches, nullscan is kicking in.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59808/#review178162
---


On June 10, 2017, 9:57 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59808/
> ---
> 
> (Updated June 10, 2017, 9:57 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16797
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSetOpTransposeRule.java
>  3ee29e0482 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveUnionMergeRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 348331e052 
>   ql/src/test/queries/clientpositive/filter_union.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query11.q PRE-CREATION 
>   ql/src/test/results/clientpositive/filter_aggr.q.out db7dcaed3f 
>   ql/src/test/results/clientpositive/filter_union.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 8b04bc9261 
>   ql/src/test/results/clientpositive/llap/explainuser_2.q.out e3f70b097f 
>   ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 5382c42412 
>   ql/src/test/results/clientpositive/llap/tez_union_multiinsert.q.out 
> 14e8e4389f 
>   ql/src/test/results/clientpositive/perf/query11.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query14.q.out 048a17f92f 
>   ql/src/test/results/clientpositive/perf/query23.q.out 1fd8cb4f25 
>   ql/src/test/results/clientpositive/perf/query33.q.out c1a5fa28ed 
>   ql/src/test/results/clientpositive/perf/query4.q.out 1b2048649a 
>   ql/src/test/results/clientpositive/perf/query5.q.out a3f2d58fec 
>   ql/src/test/results/clientpositive/perf/query56.q.out 4ec7201fa7 
>   ql/src/test/results/clientpositive/perf/query60.q.out 12d8cdd9b4 
>   ql/src/test/results/clientpositive/perf/query71.q.out 44658081b5 
>   ql/src/test/results/clientpositive/perf/query74.q.out bb4a71e6ce 
>   ql/src/test/results/clientpositive/perf/query76.q.out dcd5004166 
>   ql/src/test/results/clientpositive/perf/query77.q.out d46ba6b13c 
>   ql/src/test/results/clientpositive/perf/query80.q.out 3cf41f3fed 
>   ql/src/test/results/clientpositive/spark/union30.q.out 12eda1d3b6 
>   ql/src/test/results/clientpositive/tez/explainanalyze_2.q.out f6844c4a38 
>   ql/src/test/results/clientpositive/union24.q.out d6b1a79b20 
>   ql/src/test/results/clientpositive/union30.q.out 26a27c8e15 
>   ql/src/test/results/clientpositive/union34.q.out 9d593315af 
>   ql/src/test/results/clientpositive/unionall_unbalancedppd.q.out b3e128a3d6 
> 
> 
> Diff: https://reviews.apache.org/r/59808/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

[jira] [Created] (HIVE-16918) Skip ReplCopyTask distcp for _metadata copying. Also enable -pb for distcp

2017-06-19 Thread Sushanth Sowmyan (JIRA) 
Sushanth Sowmyan created HIVE-16918:
---

 Summary: Skip ReplCopyTask distcp for _metadata copying. Also 
enable -pb for distcp
 Key: HIVE-16918
 URL: https://issues.apache.org/jira/browse/HIVE-16918
 Project: Hive
  Issue Type: Bug
  Components: repl
Affects Versions: 3.0.0
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan


With HIVE-16686, we switched ReplCopyTask to always use a privileged DistCp. 
This, however, is incorrect for copying _metadata generated from a temporary 
scratch directory to hdfs. We need to change that so that routes to using a 
regular CopyTask.

Also, in the spirit of following up HIVE-16686, we missed adding "-pb" as a 
default for invocations of distcp from hive. Adding that in. This would not be 
necessary if HADOOP-8143 had made it in, but till it doesn't go in, we need it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-16917) HiveServer2 guard rails - Limit concurrent connections from user

2017-06-19 Thread Thejas M Nair (JIRA) 
Thejas M Nair created HIVE-16917:


 Summary: HiveServer2 guard rails - Limit concurrent connections 
from user
 Key: HIVE-16917
 URL: https://issues.apache.org/jira/browse/HIVE-16917
 Project: Hive
  Issue Type: New Feature
  Components: HiveServer2
Reporter: Thejas M Nair


Rogue applications can make HS2 unusable for others by making too many 
connections at a time.

HS2 should start rejecting the number of connections from a user, after it has 
reached a configurable threshold.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60116: HIVE-16885

2017-06-19 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60116/#review178259
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 1092 (patched)


Sounds like a plan, I'll update the patch.


- Jesús Camacho Rodríguez


On June 15, 2017, 10:49 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60116/
> ---
> 
> (Updated June 15, 2017, 10:49 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16885
> https://issues.apache.org/jira/browse/HIVE-16885
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16885
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> fce8db3df1026de8b6ee8c59567e55db40696217 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 
> 6651900e79a5c3d4ad8329afbe3894544ce9f46e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 
> 07fd653dedc9a98d89b492ae6b49da70984569f7 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 737aad1b764ee6487b420f2b9ea651c42e08e9bf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  fc6adafa0ebd0bd49d59cd0f4a82f70e9646ca6d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 9e84a29470c481d932d4f2d12e2898e05a925e5b 
>   ql/src/test/queries/clientpositive/join47.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/mapjoin47.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/smb_mapjoin_47.q PRE-CREATION 
>   ql/src/test/results/clientpositive/join47.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/mapjoin47.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/smb_mapjoin_47.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/60116/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 60116: HIVE-16885

2017-06-19 Thread Ashutosh Chauhan 


> On June 19, 2017, 3:21 p.m., Jesús Camacho Rodríguez wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 1092 (patched)
> > 
> >
> > The problem is not cross-joins, but all kind of inner joins.
> > 
> > For instance, consider _JOIN ON (a=b and c>10)_. With this property set 
> > to true, _a_ and _b_ are keys and _c>10_ is the residual. Thus, the plan 
> > (in fact, the work containing the JOIN) will not be vectorized, however if 
> > optimization is disabled, this would not happen.
> > 
> > My idea was to create a follow-up to close the gap for vectorization 
> > and then enable it by default. Another option would be to push the residual 
> > within the join only for cross joins and lift the restriction when the 
> > vectorization support for residual predicates is there. What do you think?

That still depends on data distribution. Non-vectorized path may still be 
faster.
I think what we shall do is to a) turn this on for tests via 
data/conf/llap/hive-site.xml and data/conf/tez/hive-site.xml so that we get 
coverage in tests from compiler side and b) Create a follow-up jira for 
vectorization work.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60116/#review178249
---


On June 15, 2017, 10:49 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60116/
> ---
> 
> (Updated June 15, 2017, 10:49 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16885
> https://issues.apache.org/jira/browse/HIVE-16885
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16885
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> fce8db3df1026de8b6ee8c59567e55db40696217 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 
> 6651900e79a5c3d4ad8329afbe3894544ce9f46e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 
> 07fd653dedc9a98d89b492ae6b49da70984569f7 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 737aad1b764ee6487b420f2b9ea651c42e08e9bf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  fc6adafa0ebd0bd49d59cd0f4a82f70e9646ca6d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 9e84a29470c481d932d4f2d12e2898e05a925e5b 
>   ql/src/test/queries/clientpositive/join47.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/mapjoin47.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/smb_mapjoin_47.q PRE-CREATION 
>   ql/src/test/results/clientpositive/join47.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/mapjoin47.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/smb_mapjoin_47.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/60116/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 60116: HIVE-16885

2017-06-19 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60116/#review178249
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 1092 (patched)


The problem is not cross-joins, but all kind of inner joins.

For instance, consider _JOIN ON (a=b and c>10)_. With this property set to 
true, _a_ and _b_ are keys and _c>10_ is the residual. Thus, the plan (in fact, 
the work containing the JOIN) will not be vectorized, however if optimization 
is disabled, this would not happen.

My idea was to create a follow-up to close the gap for vectorization and 
then enable it by default. Another option would be to push the residual within 
the join only for cross joins and lift the restriction when the vectorization 
support for residual predicates is there. What do you think?


- Jesús Camacho Rodríguez


On June 15, 2017, 10:49 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60116/
> ---
> 
> (Updated June 15, 2017, 10:49 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16885
> https://issues.apache.org/jira/browse/HIVE-16885
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16885
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> fce8db3df1026de8b6ee8c59567e55db40696217 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 
> 6651900e79a5c3d4ad8329afbe3894544ce9f46e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 
> 07fd653dedc9a98d89b492ae6b49da70984569f7 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 737aad1b764ee6487b420f2b9ea651c42e08e9bf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  fc6adafa0ebd0bd49d59cd0f4a82f70e9646ca6d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 9e84a29470c481d932d4f2d12e2898e05a925e5b 
>   ql/src/test/queries/clientpositive/join47.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/mapjoin47.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/smb_mapjoin_47.q PRE-CREATION 
>   ql/src/test/results/clientpositive/join47.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/mapjoin47.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/smb_mapjoin_47.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/60116/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 60116: HIVE-16885

2017-06-19 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60116/#review178246
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 1092 (patched)


This should be true by default. Compiler should always generate an optimal 
plan. 
Further, its not clear if vectorized cross join followed by filter will 
actually be faster than non-vectorized join which doesn't generate unnecessary 
rows.


- Ashutosh Chauhan


On June 15, 2017, 10:49 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60116/
> ---
> 
> (Updated June 15, 2017, 10:49 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16885
> https://issues.apache.org/jira/browse/HIVE-16885
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16885
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> fce8db3df1026de8b6ee8c59567e55db40696217 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 
> 6651900e79a5c3d4ad8329afbe3894544ce9f46e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 
> 07fd653dedc9a98d89b492ae6b49da70984569f7 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 737aad1b764ee6487b420f2b9ea651c42e08e9bf 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  fc6adafa0ebd0bd49d59cd0f4a82f70e9646ca6d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 9e84a29470c481d932d4f2d12e2898e05a925e5b 
>   ql/src/test/queries/clientpositive/join47.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/mapjoin47.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/smb_mapjoin_47.q PRE-CREATION 
>   ql/src/test/results/clientpositive/join47.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/mapjoin47.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/smb_mapjoin_47.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/60116/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 59446: HIVE-16559: Parquet schema evolution for partitioned tables may break if table and partition serdes differ

2017-06-19 Thread Barna Zsombor Klara 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59446/
---

(Updated June 19, 2017, 9:52 a.m.)


Review request for hive and Sergio Pena.


Bugs: HIVE-16559
https://issues.apache.org/jira/browse/HIVE-16559


Repository: hive-git


Description
---

HIVE-16559: Parquet schema evolution for partitioned tables may break if table 
and partition serdes differ


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 
6651900e79a5c3d4ad8329afbe3894544ce9f46e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
87928ee930b5ee974d5e4144a584773a243f8d6f 
  ql/src/test/queries/clientnegative/parquet_alter_part_table_drop_columns.q 
PRE-CREATION 
  
ql/src/test/results/clientnegative/parquet_alter_part_table_drop_columns.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/59446/diff/2/

Changes: https://reviews.apache.org/r/59446/diff/1-2/


Testing (updated)
---

Added a negative qtest. Manually tested that no regression is caused for avro 
and textfile SerDes when columns are added or replaced in a partitioned table.


Thanks,

Barna Zsombor Klara