Re: Review Request 47155: compute stats in certain cases fails with NPE

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47155/#review132474
---


Ship it!




Ship It!

- Vikram Dixit Kumaraswamy


On May 10, 2016, 4:22 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47155/
> ---
> 
> (Updated May 10, 2016, 4:22 a.m.)
> 
> 
> Review request for hive, Gunther Hagleitner and Vikram Dixit Kumaraswamy.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13621
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 5aedd10 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseUtils.java 
> e0b449b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java f330564 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  320dc10 
>   ql/src/test/queries/clientpositive/deleteAnalyze.q PRE-CREATION 
>   ql/src/test/results/clientpositive/deleteAnalyze.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/deleteAnalyze.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47155/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: [ANNOUNCE] Apache Hive 2.0.0 Released

[Vikram Dixit Kumaraswamy]

Great news indeed! Thanks Sergey!

On 2/16/16, 11:54 AM, "Prasanth Jayachandran"
 wrote:

>Great news! Thanks Sergey for the effort.
>
>Thanks
>Prasanth
>
>> On Feb 16, 2016, at 1:44 PM, Sergey Shelukhin  wrote:
>> 
>> The Apache Hive team is proud to announce the the release of Apache Hive
>> version 2.0.0.
>> 
>> The Apache Hive (TM) data warehouse software facilitates querying and
>> managing large datasets residing in distributed storage. Built on top of
>> Apache Hadoop (TM), it provides:
>> 
>> * Tools to enable easy data extract/transform/load (ETL)
>> 
>> * A mechanism to impose structure on a variety of data formats
>> 
>> * Access to files stored either directly in Apache HDFS (TM) or in other
>> data storage systems such as Apache HBase (TM)
>> 
>> * Query execution via Apache Hadoop MapReduce and Apache Tez frameworks.
>> 
>> For Hive release details and downloads, please visit:
>> https://hive.apache.org/downloads.html
>> 
>> Hive 2.0.0 Release Notes are available here:
>> 
>>https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12332641
>>ro
>> jectId=12310843
>> 
>> We would like to thank the many contributors who made this release
>> possible.
>> 
>> Regards,
>> 
>> The Apache Hive Team
>> 
>> 
>> 
>
>

Re: Review Request 41431: HIVE-12674 HS2 Tez session lifetime

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41431/#review111805
---



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java (line 
120)
<https://reviews.apache.org/r/41431/#comment172061>

Need a condition for the restart thread as well.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java (line 
455)
<https://reviews.apache.org/r/41431/#comment172083>

All of this looks a bit complicated. I still did not understand why we need 
to re-check the queue below. I think this can be simplified but now may not be 
the time for it.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java (line 
459)
<https://reviews.apache.org/r/41431/#comment172079>

Don't we need to refresh this session? It looks like the number of sessions 
in the queue can go down this way.


- Vikram Dixit Kumaraswamy


On Dec. 23, 2015, 1:30 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41431/
> ---
> 
> (Updated Dec. 23, 2015, 1:30 a.m.)
> 
> 
> Review request for hive, Siddharth Seth and Vikram Dixit Kumaraswamy.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d2dd9c6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 971dac9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java f6bc19c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 0d84340 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java e5df2ec 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java c066c7a 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
> d55c9fe 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> 11c0325 
> 
> Diff: https://reviews.apache.org/r/41431/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 41431: HIVE-12674 HS2 Tez session lifetime

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41431/#review110852
---



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java (line 
71)
<https://reviews.apache.org/r/41431/#comment170868>

Nit: Whitespace at the end of the declaration needs to be removed.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java (line 
127)
<https://reviews.apache.org/r/41431/#comment170870>

expiration queue length is fixed at 11? Shouldn't it be a configurable 
value?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java (line 
235)
<https://reviews.apache.org/r/41431/#comment170878>

Given the assertion below, you should rename this method to 
returnSessionToPool or something like that if it is guaranteed to be a pool 
session or change the assertion below.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java (line 
247)
<https://reviews.apache.org/r/41431/#comment170879>

Change this assertion or change the method name.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java (line 
247)
<https://reviews.apache.org/r/41431/#comment170881>

Change this assertion or the method name because it does not look like this 
was the intention from the previous code.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java (line 
449)
<https://reviews.apache.org/r/41431/#comment170912>

Can you raise a jira for follow up on this?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java (line 
459)
<https://reviews.apache.org/r/41431/#comment170940>

Can you add more comments here explaining the states and how the 
interaction with the session pool occurs.


- Vikram Dixit Kumaraswamy


On Dec. 16, 2015, 1:57 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41431/
> ---
> 
> (Updated Dec. 16, 2015, 1:57 a.m.)
> 
> 
> Review request for hive, Siddharth Seth and Vikram Dixit Kumaraswamy.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 971dac9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java f6bc19c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> 0d84340 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java e1a8041 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java c066c7a 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java 
> d55c9fe 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> 11c0325 
> 
> Diff: https://reviews.apache.org/r/41431/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 34059: HIVE-10673 Dynamically partitioned hash join for Tez

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34059/#review92321
---



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java (line 
235)
https://reviews.apache.org/r/34059/#comment146398

This will likely fail when we have a vectorized shuffle join. Can you add a 
note about needing to fix this in case we add a vectorized shuffle join.


- Vikram Dixit Kumaraswamy


On July 10, 2015, 10:17 p.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34059/
 ---
 
 (Updated July 10, 2015, 10:17 p.m.)
 
 
 Review request for hive, Matt McCline and Vikram Dixit Kumaraswamy.
 
 
 Bugs: HIVE-10673
 https://issues.apache.org/jira/browse/HIVE-10673
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Reduce-side hash join (using MapJoinOperator), where the Tez inputs to the 
 reducer are unsorted.
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 39477d6 
   itests/src/test/resources/testconfiguration.properties 97715fc 
   ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java 7b57550 
   ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 15cafdd 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java d7f1b42 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesAdapter.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValue.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValues.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
 545d7c6 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java 
 7d79e87 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 
 e9bd44a 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java
  4c8c4b1 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
 5a87bd6 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 
 4d84f0f 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
 bca91dd 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java f474eae 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 93ad145 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java 6b3e19d 
   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java fa697ef 
   ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java f9c34cb 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java fb3c4a3 
   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cee9100 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceWork.java a78a92e 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_1.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_2.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_vector_dynpart_hashjoin_1.q 
 PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_vector_dynpart_hashjoin_2.q 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_2.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_vector_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_vector_dynpart_hashjoin_2.q.out 
 PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/34059/diff/
 
 
 Testing
 ---
 
 q-file tests added
 
 
 Thanks,
 
 Jason Dere
 

Re: Review Request 36486: HIVE-11262 Skip MapJoin processing if the join hash table is empty

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36486/#review91926
---

Ship it!


Ship It!

- Vikram Dixit Kumaraswamy


On July 16, 2015, 12:45 a.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36486/
 ---
 
 (Updated July 16, 2015, 12:45 a.m.)
 
 
 Review request for hive, Matt McCline, Vikram Dixit Kumaraswamy, and Wei 
 Zheng.
 
 
 Bugs: HIVE-11262
 https://issues.apache.org/jira/browse/HIVE-11262
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 - Added size() method to HashTableContainer interface/implementations.
 - After loading hashTable, check if size == 0 and if join is all inner joins. 
 If so, set done on the MapJoinOperator.
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 15cafdd 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HybridHashTableContainer.java
  e338a31 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
  83a1521 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java
  9d8cbcb 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTable.java
  fbe6b4c 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastTableContainer.java
  4b1d6f6 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/hashtable/VectorMapJoinHashTable.java
  7e219ec 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedHashTable.java
  a2d4e4c 
 
 Diff: https://reviews.apache.org/r/36486/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Jason Dere
 

Re: Review Request 34059: HIVE-10673 Dynamically partitioned hash join for Tez

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34059/#review83846
---



ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java (line 672)
https://reviews.apache.org/r/34059/#comment134936

Move this to ExprNodeUtils?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
(line 210)
https://reviews.apache.org/r/34059/#comment134932

Minreducers for conservative estimation



ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java (line 669)
https://reviews.apache.org/r/34059/#comment141831

Nit: typo initializion - initialization



ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java (line 691)
https://reviews.apache.org/r/34059/#comment141832

May want to remove this comment in case we don't need it.



ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java (line 695)
https://reviews.apache.org/r/34059/#comment141833

Rename function to flattenExprList makes code easier to read.



ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java (line 705)
https://reviews.apache.org/r/34059/#comment141834

Incomplete comment needs fixing.



ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java (line 721)
https://reviews.apache.org/r/34059/#comment141835

Can you add comments as to why we need to do that?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValue.java (line 
26)
https://reviews.apache.org/r/34059/#comment141840

Can you add comments for describing the need/use of this class?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValues.java 
(line 24)
https://reviews.apache.org/r/34059/#comment141841

Comments please.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java (line 
186)
https://reviews.apache.org/r/34059/#comment141870

Not sure if this is correct? Shouldn't this be true only if this is 
dynamically partitioned hash join?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java (line 
652)
https://reviews.apache.org/r/34059/#comment141884

Is there  in the code here?


- Vikram Dixit Kumaraswamy


On June 22, 2015, 7:30 p.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34059/
 ---
 
 (Updated June 22, 2015, 7:30 p.m.)
 
 
 Review request for hive, Matt McCline and Vikram Dixit Kumaraswamy.
 
 
 Bugs: HIVE-10673
 https://issues.apache.org/jira/browse/HIVE-10673
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Reduce-side hash join (using MapJoinOperator), where the Tez inputs to the 
 reducer are unsorted.
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 27f68df 
   itests/src/test/resources/testconfiguration.properties 7b7559a 
   ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 15cafdd 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java d7f1b42 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesAdapter.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValue.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValues.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
 545d7c6 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java 
 cdabe3a 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 
 e9bd44a 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java
  4c8c4b1 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
 5a87bd6 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 
 4d84f0f 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
 bca91dd 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java adc31ae 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 11c1df6 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java 6db8220 
   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java a342738 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java fb3c4a3 
   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cee9100 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_1.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_2.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_vector_dynpart_hashjoin_1.q 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_2.q.out 
 PRE-CREATION 
   ql/src/test/results

Re: Review Request 30786: HIVE-6069 Improve error message in GenericUDFRound

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30786/#review72015
---

Ship it!


Ship It!

- Vikram Dixit Kumaraswamy


On Feb. 9, 2015, 6:32 a.m., Alexander Pivovarov wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/30786/
 ---
 
 (Updated Feb. 9, 2015, 6:32 a.m.)
 
 
 Review request for hive, Jason Dere and Xuefu Zhang.
 
 
 Bugs: HIVE-6069
 https://issues.apache.org/jira/browse/HIVE-6069
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-6069 Improve error message in GenericUDFRound
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRound.java 
 387de5e3103e2399a3fdb9b2376ac790501eab8e 
 
 Diff: https://reviews.apache.org/r/30786/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Alexander Pivovarov
 

Re: Review Request 30788: HIVE-9617 UDF from_utc_timestamp throws NPE if the second argument is null

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30788/#review72023
---



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFromUtcTimestamp.java
https://reviews.apache.org/r/30788/#comment117957

Don't we need the null check still for the arguments array?

arguments[1] != null


- Vikram Dixit Kumaraswamy


On Feb. 11, 2015, 8:42 p.m., Alexander Pivovarov wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/30788/
 ---
 
 (Updated Feb. 11, 2015, 8:42 p.m.)
 
 
 Review request for hive, Jason Dere, Siying Dong, and Vikram Dixit 
 Kumaraswamy.
 
 
 Bugs: HIVE-9617
 https://issues.apache.org/jira/browse/HIVE-9617
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-9617 UDF from_utc_timestamp throws NPE if the second argument is null
 
 
 Diffs
 -
 
   
 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFromUtcTimestamp.java
  f76fc104774cf77597d8467c9dcf3fe8d05cddce 
 
 Diff: https://reviews.apache.org/r/30788/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Alexander Pivovarov
 

Re: Review Request 24627: HIVE-7704: Create tez task for fast file merging

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24627/#review53087
---

Ship it!


Ship It!

- Vikram Dixit Kumaraswamy


On Sept. 9, 2014, 7:32 a.m., Prasanth_J wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/24627/
 ---
 
 (Updated Sept. 9, 2014, 7:32 a.m.)
 
 
 Review request for hive and Gunther Hagleitner.
 
 
 Bugs: HIVE-7704
 https://issues.apache.org/jira/browse/HIVE-7704
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Currently tez falls back to MR task for merge file task. It will beneficial 
 to convert the merge file tasks to tez task to make use of the performance 
 gains from tez.
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 31aeba9 
   itests/src/test/resources/testconfiguration.properties 99049ca 
   
 ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java
  6f23575 
   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e076683 
   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199 
   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 8946221 
   ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/RCFileMergeOperator.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 3d74459 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5bbf3f6 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 4ff568d1 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileTezProcessor.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java 994721f 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 831e6a5 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileInputFormat.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileMapper.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileTask.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeInputFormat.java 
 4651920 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeMapper.java 6c691b1 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeOutputFormat.java 
 a3ce699 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeTask.java c30476b 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeWork.java 9efee3c 
   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileMergeMapper.java 
 13ec642 
   
 ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileStripeMergeInputFormat.java
  a6c92fb 
   ql/src/java/org/apache/hadoop/hive/ql/io/orc/Writer.java c391b0e 
   ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 195d60e 
   
 ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeInputFormat.java
  6809c79 
   
 ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java 
 dee6b1c 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7129ed8 
   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 11a9419 
   ql/src/java/org/apache/hadoop/hive/ql/plan/FileMergeDesc.java PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/plan/OrcFileMergeDesc.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/plan/RCFileMergeDesc.java 
 PRE-CREATION 
   ql/src/test/queries/clientpositive/list_bucket_dml_8.q 9e81b8d 
   ql/src/test/queries/clientpositive/orc_merge1.q ee65b98 
   ql/src/test/queries/clientpositive/orc_merge5.q PRE-CREATION 
   ql/src/test/queries/clientpositive/orc_merge6.q PRE-CREATION 
   ql/src/test/queries/clientpositive/orc_merge7.q PRE-CREATION 
   ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out 11c7578 
   ql/src/test/results/clientpositive/list_bucket_dml_10.q.out 8de452f 
   ql/src/test/results/clientpositive/list_bucket_dml_4.q.out b1c060e 
   ql/src/test/results/clientpositive/list_bucket_dml_6.q.out 3450d63 
   ql/src/test/results/clientpositive/list_bucket_dml_7.q.out f6a4cb5 
   ql/src/test/results/clientpositive/list_bucket_dml_9.q.out 796c7af 
   ql/src/test/results/clientpositive/merge_dynamic_partition4.q.out 0899648 
   ql/src/test/results/clientpositive/merge_dynamic_partition5.q.out 0653469 
   ql/src/test/results/clientpositive/orc_createas1.q.out 993c853 
   ql/src/test/results

Re: Review Request 24627: HIVE-7704: Create tez task for fast file merging

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24627/#review52632
---



ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java
https://reviews.apache.org/r/24627/#comment91490

inputDepth or tmpPath?



ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java
https://reviews.apache.org/r/24627/#comment91489

inputDepth perhaps?



ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java
https://reviews.apache.org/r/24627/#comment91491

nit updated.



ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java
https://reviews.apache.org/r/24627/#comment91502

nit: has slight difference w.r.t



ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
https://reviews.apache.org/r/24627/#comment91505

Usually done in the reverse way. The java imports come before the 
hadoop/hive imports.



ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
https://reviews.apache.org/r/24627/#comment91507

Don't really need it here. Can be moved down to line 582.



ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
https://reviews.apache.org/r/24627/#comment91506

HashMap only on the LHS. Don't use LinkedHashMap.



ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java
https://reviews.apache.org/r/24627/#comment91508

Nice work! Although this section usually comes above the hadoop imports 
though.



ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java
https://reviews.apache.org/r/24627/#comment91509

nit: comma after merge makes it easier to read.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
https://reviews.apache.org/r/24627/#comment91515

Nice work!



ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java
https://reviews.apache.org/r/24627/#comment91517

What purpose does this class serve? Doesn't look like it does anything. 
Comments please.


- Vikram Dixit Kumaraswamy


On Sept. 6, 2014, 2:03 a.m., Prasanth_J wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/24627/
 ---
 
 (Updated Sept. 6, 2014, 2:03 a.m.)
 
 
 Review request for hive and Gunther Hagleitner.
 
 
 Bugs: HIVE-7704
 https://issues.apache.org/jira/browse/HIVE-7704
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Currently tez falls back to MR task for merge file task. It will beneficial 
 to convert the merge file tasks to tez task to make use of the performance 
 gains from tez.
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54e2b18 
   itests/src/test/resources/testconfiguration.properties 99049ca 
   
 ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java
  6f23575 
   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e076683 
   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199 
   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 8946221 
   ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/RCFileMergeOperator.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 3d74459 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 2d9b9c3 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 4ff568d1 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileTezProcessor.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java 994721f 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 831e6a5 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileInputFormat.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileMapper.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileTask.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeInputFormat.java 
 4651920 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeMapper.java 6c691b1 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeOutputFormat.java 
 a3ce699 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeTask.java c30476b 
   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeWork.java 9efee3c 
   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileMergeMapper.java

Re: Review Request 23967: Enable auto conversion of SMBjoin in presence of constant propagate optimization

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23967/#review48907
---

Ship it!


Ship It!

- Vikram Dixit Kumaraswamy


On July 27, 2014, 5:18 p.m., Ashutosh Chauhan wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/23967/
 ---
 
 (Updated July 27, 2014, 5:18 p.m.)
 
 
 Review request for hive and Ted Xu.
 
 
 Bugs: HIVE-7524
 https://issues.apache.org/jira/browse/HIVE-7524
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Enable auto conversion of SMBjoin in presence of constant propagate 
 optimization
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractBucketJoinProc.java 
 6042470 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagate.java 
 3c8940f 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java
  c1cc9f4 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedMergeJoinProc.java 
 5f7682e 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java 
 2420971 
   ql/src/test/results/clientpositive/join_nullsafe.q.out 37b6978 
   ql/src/test/results/clientpositive/smb_mapjoin_25.q.out bd289c3 
 
 Diff: https://reviews.apache.org/r/23967/diff/
 
 
 Testing
 ---
 
 smb_mapjoin_25.q used to fail to convert joins to SMBJoins, now it does.
 
 
 Thanks,
 
 Ashutosh Chauhan
 

Re: Review Request 19789: HIVE-6739 Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19789/#review39521
---



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
https://reviews.apache.org/r/19789/#comment71934

Document all the input params/just rename the params to reflect their use. 
hdfsPathStr is the destination directory right?



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
https://reviews.apache.org/r/19789/#comment71932

Actually looks like this method localizes an array of files.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java
https://reviews.apache.org/r/19789/#comment71933

Definitely needs method(s) renaming.



ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java
https://reviews.apache.org/r/19789/#comment71936

Open a jira for this.



ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java
https://reviews.apache.org/r/19789/#comment71938

Aren't you walking the work graph here to populate the job conf?


- Vikram Dixit Kumaraswamy


On April 3, 2014, 2:26 a.m., Sergey Shelukhin wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/19789/
 ---
 
 (Updated April 3, 2014, 2:26 a.m.)
 
 
 Review request for hive, Gunther Hagleitner and Vikram Dixit Kumaraswamy.
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 See jira
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 78e965b 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
 720b8d5 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 5f0f353 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 385e22e 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java fdbd996 
   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 38c4c11 
   ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java e1cc3f4 
   ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java 996268e 
   ql/src/java/org/apache/hadoop/hive/ql/plan/UnionWork.java 60781e6 
   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 93eccf8 
   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
 d2c332c 
   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java 
 5ad4250 
   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java 4810fd0 
 
 Diff: https://reviews.apache.org/r/19789/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Sergey Shelukhin
 

Re: Review Request 17471: HIVE-6325: Enable using multiple concurrent sessions in tez

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17471/
---

(Updated March 3, 2014, 10:39 p.m.)


Review request for hive.


Changes
---

Addressed comments.


Bugs: HIVE-6325
https://issues.apache.org/jira/browse/HIVE-6325


Repository: hive-git


Description
---

Enable using multiple concurrent sessions in tez.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 388a604 
  conf/hive-default.xml.template 3f01e0b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java aef6e68 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java d30ec8e 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 16e9166 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 8700c5b 

Diff: https://reviews.apache.org/r/17471/diff/


Testing
---

Added multi-threaded junit tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 17471: HIVE-6325: Enable using multiple concurrent sessions in tez

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17471/
---

(Updated Feb. 28, 2014, 2:25 a.m.)


Review request for hive.


Changes
---

Adding ASF headers to new files. No changes to code from previous iteration.


Bugs: HIVE-6325
https://issues.apache.org/jira/browse/HIVE-6325


Repository: hive-git


Description
---

Enable using multiple concurrent sessions in tez.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6802b4d 
  conf/hive-default.xml.template 0b86b9c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b8552a3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java bef5ba3 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 16e9166 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 8700c5b 

Diff: https://reviews.apache.org/r/17471/diff/


Testing
---

Added multi-threaded junit tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 17471: HIVE-6325: Enable using multiple concurrent sessions in tez

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17471/
---

(Updated Feb. 26, 2014, 3:56 a.m.)


Review request for hive.


Changes
---

Small nit fix.


Bugs: HIVE-6325
https://issues.apache.org/jira/browse/HIVE-6325


Repository: hive-git


Description
---

Enable using multiple concurrent sessions in tez.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 46f9f42 
  conf/hive-default.xml.template 0b86b9c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b8552a3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java bef5ba3 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 300a415 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 8700c5b 

Diff: https://reviews.apache.org/r/17471/diff/


Testing
---

Added multi-threaded junit tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 17471: HIVE-6325: Enable using multiple concurrent sessions in tez

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17471/
---

(Updated Feb. 16, 2014, 12:06 a.m.)


Review request for hive.


Bugs: HIVE-6325
https://issues.apache.org/jira/browse/HIVE-6325


Repository: hive-git


Description
---

Enable using multiple concurrent sessions in tez.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a182cd7 
  conf/hive-default.xml.template 0d08aa2 
  hcatalog/core/.gitignore 0a7a9c5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b8552a3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java bef5ba3 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 99b6d77 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 8700c5b 

Diff: https://reviews.apache.org/r/17471/diff/


Testing
---

Added multi-threaded junit tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 17471: HIVE-6325: Enable using multiple concurrent sessions in tez

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17471/
---

(Updated Feb. 10, 2014, 10:34 p.m.)


Review request for hive.


Changes
---

Fix failing tests.


Bugs: HIVE-6325
https://issues.apache.org/jira/browse/HIVE-6325


Repository: hive-git


Description
---

Enable using multiple concurrent sessions in tez.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ec1f09e 
  conf/hive-default.xml.template e1ce436 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 9b2e219 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b8552a3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 22a21c9 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 845ff77 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 8700c5b 

Diff: https://reviews.apache.org/r/17471/diff/


Testing
---

Added multi-threaded junit tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 17471: HIVE-6325: Enable using multiple concurrent sessions in tez

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17471/
---

(Updated Feb. 11, 2014, 7:50 a.m.)


Review request for hive.


Bugs: HIVE-6325
https://issues.apache.org/jira/browse/HIVE-6325


Repository: hive-git


Description
---

Enable using multiple concurrent sessions in tez.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ec1f09e 
  conf/hive-default.xml.template e1ce436 
  hcatalog/core/.gitignore 0a7a9c5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b8552a3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 22a21c9 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 845ff77 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 8700c5b 

Diff: https://reviews.apache.org/r/17471/diff/


Testing
---

Added multi-threaded junit tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 17471: HIVE-6325: Enable using multiple concurrent sessions in tez

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17471/
---

(Updated Feb. 8, 2014, 2:49 a.m.)


Review request for hive.


Changes
---

Addressed review comments.


Bugs: HIVE-6325
https://issues.apache.org/jira/browse/HIVE-6325


Repository: hive-git


Description
---

Enable using multiple concurrent sessions in tez.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3f50361 
  conf/hive-default.xml.template 420d959 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 9b2e219 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b8552a3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 22a21c9 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 64a8a60 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 8700c5b 

Diff: https://reviews.apache.org/r/17471/diff/


Testing
---

Added multi-threaded junit tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 17471: HIVE-6325: Enable using multiple concurrent sessions in tez

[Vikram Dixit Kumaraswamy]

 On Feb. 3, 2014, 9:10 p.m., Gunther Hagleitner wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java, 
  line 28
  https://reviews.apache.org/r/17471/diff/1/?file=453262#file453262line28
 
  BlockingQueue should be able to tell you length, right?

Blocking queue has a size that reflects the number of elements in the queue not 
the size of the queue itself. The length can be computed using the size and the 
remaining capacity from the queue in a synchronized method which seems somewhat 
round-about to me.


 On Feb. 3, 2014, 9:10 p.m., Gunther Hagleitner wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java, 
  line 65
  https://reviews.apache.org/r/17471/diff/1/?file=453262#file453262line65
 
  don't think this is needed

Addressed above.


 On Feb. 3, 2014, 9:10 p.m., Gunther Hagleitner wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java, 
  line 73
  https://reviews.apache.org/r/17471/diff/1/?file=453262#file453262line73
 
  is this the right name? shouldn't that be a yarn var?

The tez queue name is the variable used by tez to allocate containers on a 
specific queue. This is again dependent on the yarn queues already. So 
according to the tez team we can just depend on this variable. Also, mapreduce 
has the behavior that its variable will return the queue 'default' if no queue 
is specified. We would like to choose a set of queues as default and not depend 
on yarn's default queue.


 On Feb. 3, 2014, 9:10 p.m., Gunther Hagleitner wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java, 
  line 106
  https://reviews.apache.org/r/17471/diff/1/?file=453262#file453262line106
 
  It doesn't look like you're keeping track of this sessionstate here. I 
  think we should. The user should always get/return sessions and we handle 
  the alloc/dealloc. (why can't return close the session for non default for 
  instance?)

We want to retain the efficiencies in having pre-launched containers. Closing 
the non-default sessions would let go of that.


- Vikram


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17471/#review33496
---


On Jan. 28, 2014, 10:34 p.m., Vikram Dixit Kumaraswamy wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/17471/
 ---
 
 (Updated Jan. 28, 2014, 10:34 p.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-6325
 https://issues.apache.org/jira/browse/HIVE-6325
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Enable using multiple concurrent sessions in tez.
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 84ee78f 
   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 9ad5986 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b8552a3 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionStateFactory.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java c6f431c 
   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java d7edda1 
   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
 PRE-CREATION 
   service/src/java/org/apache/hive/service/server/HiveServer2.java fa13783 
 
 Diff: https://reviews.apache.org/r/17471/diff/
 
 
 Testing
 ---
 
 Added multi-threaded junit tests.
 
 
 Thanks,
 
 Vikram Dixit Kumaraswamy
 

Re: Review Request 17471: HIVE-6325: Enable using multiple concurrent sessions in tez

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17471/
---

(Updated Feb. 6, 2014, 2:08 a.m.)


Review request for hive.


Changes
---

Updated with review feedback addressed.


Bugs: HIVE-6325
https://issues.apache.org/jira/browse/HIVE-6325


Repository: hive-git


Description
---

Enable using multiple concurrent sessions in tez.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 89c9349 
  conf/hive-default.xml.template 420d959 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 9b2e219 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b8552a3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 22a21c9 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 64a8a60 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 8700c5b 

Diff: https://reviews.apache.org/r/17471/diff/


Testing
---

Added multi-threaded junit tests.


Thanks,

Vikram Dixit Kumaraswamy

Review Request 17471: HIVE-6325: Enable using multiple concurrent sessions in tez

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17471/
---

Review request for hive.


Bugs: HIVE-6325
https://issues.apache.org/jira/browse/HIVE-6325


Repository: hive-git


Description
---

Enable using multiple concurrent sessions in tez.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 84ee78f 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 9ad5986 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b8552a3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionStateFactory.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java c6f431c 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java d7edda1 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java fa13783 

Diff: https://reviews.apache.org/r/17471/diff/


Testing
---

Added multi-threaded junit tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 16728: Implement non-staged MapJoin

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16728/#review32885
---



itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
https://reviews.apache.org/r/16728/#comment61876

This would break tez tests.



itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
https://reviews.apache.org/r/16728/#comment61875

This would eliminate tez unit tests. Was this intentional?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LocalMapJoinProcFactory.java
https://reviews.apache.org/r/16728/#comment61880

Could you raise a jira for this.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LocalMapJoinProcFactory.java
https://reviews.apache.org/r/16728/#comment61882

can it be only these 2 operators? Maybe common join operator can be used?


- Vikram Dixit Kumaraswamy


On Jan. 20, 2014, 5 a.m., Navis Ryu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/16728/
 ---
 
 (Updated Jan. 20, 2014, 5 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-6144
 https://issues.apache.org/jira/browse/HIVE-6144
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 For map join, all data in small aliases are hashed and stored into temporary 
 file in MapRedLocalTask. But for some aliases without filter or projection, 
 it seemed not necessary to do that. For example.
 
 {noformat}
 select a.* from src a join src b on a.key=b.key;
 {noformat}
 
 makes plan like this.
 {noformat}
 STAGE PLANS:
   Stage: Stage-4
 Map Reduce Local Work
   Alias - Map Local Tables:
 a 
   Fetch Operator
 limit: -1
   Alias - Map Local Operator Tree:
 a 
   TableScan
 alias: a
 HashTable Sink Operator
   condition expressions:
 0 {key} {value}
 1 
   handleSkewJoin: false
   keys:
 0 [Column[key]]
 1 [Column[key]]
   Position of Big Table: 1
 
   Stage: Stage-3
 Map Reduce
   Alias - Map Operator Tree:
 b 
   TableScan
 alias: b
 Map Join Operator
   condition map:
Inner Join 0 to 1
   condition expressions:
 0 {key} {value}
 1 
   handleSkewJoin: false
   keys:
 0 [Column[key]]
 1 [Column[key]]
   outputColumnNames: _col0, _col1
   Position of Big Table: 1
   Select Operator
 File Output Operator
   Local Work:
 Map Reduce Local Work
   Stage: Stage-0
 Fetch Operator
 {noformat}
 
 table src(a) is fetched and stored as-is in MRLocalTask. With this patch, 
 plan can be like below.
 {noformat}
   Stage: Stage-3
 Map Reduce
   Alias - Map Operator Tree:
 b 
   TableScan
 alias: b
 Map Join Operator
   condition map:
Inner Join 0 to 1
   condition expressions:
 0 {key} {value}
 1 
   handleSkewJoin: false
   keys:
 0 [Column[key]]
 1 [Column[key]]
   outputColumnNames: _col0, _col1
   Position of Big Table: 1
   Select Operator
   File Output Operator
   Local Work:
 Map Reduce Local Work
   Alias - Map Local Tables:
 a 
   Fetch Operator
 limit: -1
   Alias - Map Local Operator Tree:
 a 
   TableScan
 alias: a
   Has Any Stage Alias: false
   Stage: Stage-0
 Fetch Operator
 {noformat}
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a78b72f 
   conf/hive-default.xml.template 7cd8a1f 
   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 9ad5986 
   
 itests/util/src/main/java/org/apache/hadoop/hive/ql/hooks/MapJoinCounterHook.java
  1b0d57e 
   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 
 d8f4eb4 
   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableLoader.java a080fcc 
   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 
 fc08b28 
   ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java 1e0314d 
   ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java bdc85b9 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 56676df 
   ql/src/java/org/apache/hadoop/hive/ql/exec/TemporaryHashSinkOperator.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java

Re: Review Request 16440: HIVE-6098: Merge Tez branch into trunk

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16440/#review31347
---



trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java
https://reviews.apache.org/r/16440/#comment59818

Need to update java doc.



trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java
https://reviews.apache.org/r/16440/#comment59817

No param conf.



trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java
https://reviews.apache.org/r/16440/#comment59831

spelling error commone



trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java
https://reviews.apache.org/r/16440/#comment59832

nit: reword the sentence.



trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java
https://reviews.apache.org/r/16440/#comment59834

Could rename the mrTask to be more generic.



trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
https://reviews.apache.org/r/16440/#comment59836

Why doesn't the same apply for the value table desc below?



trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
https://reviews.apache.org/r/16440/#comment59835

Comment needs to be fixed.



trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java
https://reviews.apache.org/r/16440/#comment59838

Could add some documentation explaining the use of this class.



trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java
https://reviews.apache.org/r/16440/#comment59839

Could it ever be a non-tez task?


- Vikram Dixit Kumaraswamy


On Jan. 7, 2014, 12:50 a.m., Gunther Hagleitner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/16440/
 ---
 
 (Updated Jan. 7, 2014, 12:50 a.m.)
 
 
 Review request for hive.
 
 
 Repository: hive
 
 
 Description
 ---
 
 I think the Tez branch is at a point where we can consider merging it back 
 into trunk after review.
 Tez itself has had its first release, most hive features are available on Tez 
 and the test coverage is decent. There are a few known limitations, all of 
 which can be handled in trunk as far as I can tell (i.e.: None of them are 
 large disruptive changes that still require a branch.)
 Limitations:
 Union all is not yet supported on Tez
 SMB is not yet supported on Tez
 Bucketed map-join is executed as broadcast join (bucketing is ignored)
 Since the user is free to toggle hive.optimize.tez, it's obviously possible 
 to just run these on MR.
 I am hoping to follow the approach that was taken with vectorization and 
 shoot for a merge instead of single commit. This would retain history of the 
 branch. Also in vectorization we required at least three +1s before merge, 
 I'm hoping to go with that as well.
 I will add a combined patch to this ticket for review purposes (not for 
 commit). I'll also attach instructions to run on a cluster if anyone wants to 
 try.
 
 
 Diffs
 -
 
   trunk/ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 1556067 
   trunk/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 
 1556067 
   trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1556067 
   trunk/conf/hive-default.xml.template 1556067 
   trunk/data/conf/tez/hive-site.xml UNKNOWN 
   trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm 1556067 
   trunk/hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm 
 1556067 
   
 trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java
  1556067 
   trunk/itests/qtest/pom.xml 1556067 
   
 trunk/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java
  1556067 
   trunk/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
 1556067 
   trunk/pom.xml 1556067 
   trunk/ql/pom.xml 1556067 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1556067 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 1556067 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java 
 1556067 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredContext.java 1556067 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 
 1556067 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1556067 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1556067 
   
 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 
 1556067 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 
 PRE-CREATION 
   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 
 PRE-CREATION 
   
 trunk/ql/src/java/org/apache/hadoop/hive

Re: Review Request 16213: HIVE-5973: SMB joins produce incorrect results with multiple partitions and buckets

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16213/
---

(Updated Dec. 13, 2013, 11:10 p.m.)


Review request for hive, Navis Ryu and Harish Butani.


Changes
---

Addressed Harish's comments.


Bugs: HIVE-5973
https://issues.apache.org/jira/browse/HIVE-5973


Repository: hive-git


Description
---

SMB joins produce incorrect results with multiple partitions and buckets


Diffs (updated)
-

  itests/qtest/pom.xml 8c249a0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DummyStoreOperator.java acdb040 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_16.q PRE-CREATION 
  ql/src/test/results/clientpositive/auto_sortmerge_join_16.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/16213/diff/


Testing
---

New test case added.


Thanks,

Vikram Dixit Kumaraswamy

Review Request 16213: HIVE-5973: SMB joins produce incorrect results with multiple partitions and buckets

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16213/
---

Review request for hive, Navis Ryu and Harish Butani.


Bugs: HIVE-5973
https://issues.apache.org/jira/browse/HIVE-5973


Repository: hive-git


Description
---

SMB joins produce incorrect results with multiple partitions and buckets


Diffs
-

  itests/hive-unit/pom.xml dae4e50 
  itests/qtest/pom.xml 8c249a0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DummyStoreOperator.java acdb040 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_16.q PRE-CREATION 
  ql/src/test/results/clientpositive/auto_sortmerge_join_16.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/16213/diff/


Testing
---

New test case added.


Thanks,

Vikram Dixit Kumaraswamy

Review Request 15543: HIVE-5827: Incorrect location of logs for failed tests.

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15543/
---

Review request for hive.


Bugs: HIVE-5827
https://issues.apache.org/jira/browse/HIVE-5827


Repository: hive-git


Description
---

HIVE-5827: Incorrect location of logs for failed tests.


Diffs
-

  ql/src/test/templates/TestCliDriver.vm f2623f8 
  ql/src/test/templates/TestNegativeCliDriver.vm 7159faf 
  ql/src/test/templates/TestParse.vm e28e901 
  ql/src/test/templates/TestParseNegative.vm 1aec083 

Diff: https://reviews.apache.org/r/15543/diff/


Testing
---

Ran failing tests and verified the message.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 14651: HIVE-5543 - Running the mini tez cluster for tez unit tests

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14651/#review27559
---



ql/build.xml
https://reviews.apache.org/r/14651/#comment53488





ql/build.xml
https://reviews.apache.org/r/14651/#comment53489

I will do it. Just to understand though, does this affect mavenization or 
is an aesthetic concern?



ql/build.xml
https://reviews.apache.org/r/14651/#comment53490

log dir is the directory where results are put temporarily before being 
compared against the results directory. It will not prefix files with 
TestMiniTez.



ql/ivy.xml
https://reviews.apache.org/r/14651/#comment53487

No. But they use classes in this jar to execute. Since in the case of mini 
tez cluster, the jars are not available in TEZ_HOME etc., there is no access to 
this jar unless it is put on the classpath. Hence the ivy dependency to pull in 
this jar.



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
https://reviews.apache.org/r/14651/#comment53486

The for loop below only updates configurations that are null and 
respectively non-null within the hive conf. This was wiping out some of the 
configurations because of defaults coming into play. The new update fixes this 
as this is how the api used in the latest code.



ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
https://reviews.apache.org/r/14651/#comment53491

Fixed.



ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
https://reviews.apache.org/r/14651/#comment53492

Fixed



shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
https://reviews.apache.org/r/14651/#comment53493

Fixed.



shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
https://reviews.apache.org/r/14651/#comment53494

Fixed.


- Vikram Dixit Kumaraswamy


On Oct. 15, 2013, 9:50 p.m., Vikram Dixit Kumaraswamy wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/14651/
 ---
 
 (Updated Oct. 15, 2013, 9:50 p.m.)
 
 
 Review request for hive and Gunther Hagleitner.
 
 
 Bugs: HIVE-5543
 https://issues.apache.org/jira/browse/HIVE-5543
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Running the mini tez cluster for tez unit tests.
 
 
 Diffs
 -
 
   build-common.xml ee6a3ce 
   build.properties 3dbd33f 
   hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 
 0558048 
   hbase-handler/src/test/templates/TestHBaseCliDriver.vm 191bbc7 
   hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm e6bd211 
   ql/build.xml be8e10f 
   ql/ivy.xml d0159d9 
   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java cee95fd 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 1726da6 
   ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java ddcc544 
   ql/src/test/org/apache/hadoop/hive/ql/TestLocationQueries.java 0ddc1d5 
   ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION 
   ql/src/test/results/clientpositive/tez_join_tests.q.out PRE-CREATION 
   ql/src/test/results/clientpositive/tez_joins_explain.q.out PRE-CREATION 
   ql/src/test/templates/TestCliDriver.vm f2623f8 
   ql/src/test/templates/TestNegativeCliDriver.vm 7159faf 
   ql/src/test/templates/TestParse.vm e28e901 
   ql/src/test/templates/TestParseNegative.vm 1aec083 
   shims/build.xml cbfba5a 
   shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 6368395 
   shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java a4714ae 
   shims/src/tez/java/org/apache/hadoop/hive/shims/TezShims.java PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/14651/diff/
 
 
 Testing
 ---
 
 Added new tests for running the tez mini cluster and a new test target.
 
 
 Thanks,
 
 Vikram Dixit Kumaraswamy
 

Review Request 15069: HIVE-5685: partition column type validation doesn't work in some cases

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15069/
---

Review request for hive.


Bugs: HIVE-5685
https://issues.apache.org/jira/browse/HIVE-5685


Repository: hive-git


Description
---

HIVE-5685: partition column type validation doesn't work in some cases


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 46d1fac 
  ql/src/test/queries/clientnegative/illegal_partition_type3.q PRE-CREATION 
  ql/src/test/results/clientnegative/illegal_partition_type3.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/15069/diff/


Testing
---


Thanks,

Vikram Dixit Kumaraswamy

Review Request 14877: HIVE-5625: Fix issue with metastore version restriction test.

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14877/
---

Review request for hive, Ashutosh Chauhan and Brock Noland.


Bugs: HIVE-5625
https://issues.apache.org/jira/browse/HIVE-5625


Repository: hive-git


Description
---

Fix issue with metastore version restriction test.


Diffs
-

  metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java 
d7761f4 

Diff: https://reviews.apache.org/r/14877/diff/


Testing
---

Ran all metastore tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 14892: HIVE-5633: Perflogger broken due to HIVE-5403

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14892/
---

(Updated Oct. 23, 2013, 10:55 p.m.)


Review request for hive and Gunther Hagleitner.


Bugs: HIVE-5633
https://issues.apache.org/jira/browse/HIVE-5633


Repository: hive-git


Description
---

Perflogger broken due to HIVE-5403.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 8276d4d 

Diff: https://reviews.apache.org/r/14892/diff/


Testing
---


Thanks,

Vikram Dixit Kumaraswamy

Review Request 14892: HIVE-5633: Perflogger broken due to HIVE-5403

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14892/
---

Review request for hive and Gunther Hagleitner.


Bugs: HIVE-5633
https://issues.apache.org/jira/browse/HIVE-5633


Repository: hive-git


Description
---

Perflogger broken due to HIVE-5403.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 8276d4d 

Diff: https://reviews.apache.org/r/14892/diff/


Testing
---


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 14892: HIVE-5633: Perflogger broken due to HIVE-5403

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14892/
---

(Updated Oct. 24, 2013, 12:05 a.m.)


Review request for hive and Gunther Hagleitner.


Changes
---

Updated.


Bugs: HIVE-5633
https://issues.apache.org/jira/browse/HIVE-5633


Repository: hive-git


Description
---

Perflogger broken due to HIVE-5403.


Diffs (updated)
-

  metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java 
d7761f4 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 8276d4d 

Diff: https://reviews.apache.org/r/14892/diff/


Testing
---


Thanks,

Vikram Dixit Kumaraswamy

Review Request 14651: HIVE-5543 - Running the mini tez cluster for tez unit tests

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14651/
---

Review request for hive and Gunther Hagleitner.


Bugs: HIVE-5543
https://issues.apache.org/jira/browse/HIVE-5543


Repository: hive-git


Description
---

Running the mini tez cluster for tez unit tests.


Diffs
-

  build-common.xml ee6a3ce 
  build.properties 3dbd33f 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 
0558048 
  hbase-handler/src/test/templates/TestHBaseCliDriver.vm 191bbc7 
  hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm e6bd211 
  ql/build.xml be8e10f 
  ql/ivy.xml d0159d9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java cee95fd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 1726da6 
  ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java ddcc544 
  ql/src/test/org/apache/hadoop/hive/ql/TestLocationQueries.java 0ddc1d5 
  ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION 
  ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION 
  ql/src/test/results/clientpositive/tez_join_tests.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez_joins_explain.q.out PRE-CREATION 
  ql/src/test/templates/TestCliDriver.vm f2623f8 
  ql/src/test/templates/TestNegativeCliDriver.vm 7159faf 
  ql/src/test/templates/TestParse.vm e28e901 
  ql/src/test/templates/TestParseNegative.vm 1aec083 
  shims/build.xml cbfba5a 
  shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 6368395 
  shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java a4714ae 
  shims/src/tez/java/org/apache/hadoop/hive/shims/TezShims.java PRE-CREATION 

Diff: https://reviews.apache.org/r/14651/diff/


Testing
---

Added new tests for running the tez mini cluster and a new test target.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 14425: HIVE-5403: Move loading of filesystem, ugi, metastore client to hive session

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14425/
---

(Updated Oct. 1, 2013, 9:04 p.m.)


Review request for hive and Gunther Hagleitner.


Changes
---

Addressed Gunther's comments.


Bugs: HIVE-5403
https://issues.apache.org/jira/browse/HIVE-5403


Repository: hive-git


Description
---

Move loading of filesystem, ugi, metastore client to hive session.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 0491f8b 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java f6b1491 

Diff: https://reviews.apache.org/r/14425/diff/


Testing
---

Does not affect any unit tests but all of them exercise this code path.


Thanks,

Vikram Dixit Kumaraswamy

Review Request 14425: HIVE-5403: Move loading of filesystem, ugi, metastore client to hive session

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14425/
---

Review request for hive and Gunther Hagleitner.


Bugs: HIVE-5403
https://issues.apache.org/jira/browse/HIVE-5403


Repository: hive-git


Description
---

Move loading of filesystem, ugi, metastore client to hive session.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 0491f8b 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java f6b1491 

Diff: https://reviews.apache.org/r/14425/diff/


Testing
---

Does not affect any unit tests but all of them exercise this code path.


Thanks,

Vikram Dixit Kumaraswamy

Review Request 14349: HIVE-5271: Convert join op to a map join op in the planning phase

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14349/
---

Review request for hive and Gunther Hagleitner.


Repository: hive-git


Description
---

 Convert join op to a map join op in the planning phase.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 31ca07a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ConvertJoinMapJoin.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 7e51310 

Diff: https://reviews.apache.org/r/14349/diff/


Testing
---


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 14155: HIVE-5297 Hive does not honor type for partition columns

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14155/
---

(Updated Sept. 26, 2013, 6:55 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Fix failing test.


Bugs: HIVE-5297
https://issues.apache.org/jira/browse/HIVE-5297


Repository: hive-git


Description
---

Hive does not consider the type of the partition column while writing 
partitions. Consider for example the query:

create table tab1 (id1 int, id2 string) PARTITIONED BY(month string,day int) 
row format delimited fields terminated by ',';
alter table tab1 add partition (month='June', day='second');

Hive accepts this query. However if you try to select from this table and 
insert into another expecting schema match, it will insert nulls instead.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 350c348 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 393ef57 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java a704462 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 0c01749 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 767f545 
  ql/src/test/queries/clientnegative/illegal_partition_type.q PRE-CREATION 
  ql/src/test/queries/clientnegative/illegal_partition_type2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/alter_partition_coltype.q 4d2e02f 
  ql/src/test/queries/clientpositive/partition_type_check.q PRE-CREATION 
  ql/src/test/results/clientnegative/alter_table_add_partition.q.out bd9c148 
  ql/src/test/results/clientnegative/alter_view_failure5.q.out 4edb82c 
  ql/src/test/results/clientnegative/illegal_partition_type.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/illegal_partition_type2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/alter_partition_coltype.q.out fe16fdd 
  ql/src/test/results/clientpositive/partition_type_check.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/14155/diff/


Testing
---

Ran all tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 14349: HIVE-5271: Convert join op to a map join op in the planning phase

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14349/
---

(Updated Sept. 26, 2013, 7:39 p.m.)


Review request for hive and Gunther Hagleitner.


Changes
---

Address Gunther's comments.


Repository: hive-git


Description
---

 Convert join op to a map join op in the planning phase.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 31ca07a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 7e51310 

Diff: https://reviews.apache.org/r/14349/diff/


Testing
---


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 14155: HIVE-5297 Hive does not honor type for partition columns

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14155/
---

(Updated Sept. 25, 2013, 8:22 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Fix unit test failures when enabling type check by default.


Bugs: HIVE-5297
https://issues.apache.org/jira/browse/HIVE-5297


Repository: hive-git


Description
---

Hive does not consider the type of the partition column while writing 
partitions. Consider for example the query:

create table tab1 (id1 int, id2 string) PARTITIONED BY(month string,day int) 
row format delimited fields terminated by ',';
alter table tab1 add partition (month='June', day='second');

Hive accepts this query. However if you try to select from this table and 
insert into another expecting schema match, it will insert nulls instead.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 350c348 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 393ef57 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java a704462 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 0c01749 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 767f545 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 6c354d3 
  ql/src/test/queries/clientnegative/illegal_partition_type.q PRE-CREATION 
  ql/src/test/queries/clientnegative/illegal_partition_type2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/alter_partition_coltype.q 4d2e02f 
  ql/src/test/queries/clientpositive/partition_type_check.q PRE-CREATION 
  ql/src/test/results/clientnegative/alter_table_add_partition.q.out bd9c148 
  ql/src/test/results/clientnegative/alter_view_failure5.q.out 4edb82c 
  ql/src/test/results/clientnegative/illegal_partition_type.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/illegal_partition_type2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/alter_partition_coltype.q.out fe16fdd 
  ql/src/test/results/clientpositive/partition_type_check.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/14155/diff/


Testing
---

Ran all tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 14155: HIVE-5297 Hive does not honor type for partition columns

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14155/
---

(Updated Sept. 23, 2013, 9:51 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Refreshed against latest trunk and enabled check by default. Passes my tests.


Bugs: HIVE-5297
https://issues.apache.org/jira/browse/HIVE-5297


Repository: hive-git


Description
---

Hive does not consider the type of the partition column while writing 
partitions. Consider for example the query:

create table tab1 (id1 int, id2 string) PARTITIONED BY(month string,day int) 
row format delimited fields terminated by ',';
alter table tab1 add partition (month='June', day='second');

Hive accepts this query. However if you try to select from this table and 
insert into another expecting schema match, it will insert nulls instead.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1af68a6 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 393ef57 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java a704462 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java fb79823 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 767f545 
  ql/src/test/queries/clientnegative/illegal_partition_type.q PRE-CREATION 
  ql/src/test/queries/clientnegative/illegal_partition_type2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_type_check.q PRE-CREATION 
  ql/src/test/results/clientnegative/alter_table_add_partition.q.out bd9c148 
  ql/src/test/results/clientnegative/alter_view_failure5.q.out 4edb82c 
  ql/src/test/results/clientnegative/illegal_partition_type.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/illegal_partition_type2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/partition_type_check.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/14155/diff/


Testing
---

Ran all tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 14155: HIVE-5297 Hive does not honor type for partition columns

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14155/
---

(Updated Sept. 23, 2013, 9:57 p.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-5297
https://issues.apache.org/jira/browse/HIVE-5297


Repository: hive-git


Description
---

Hive does not consider the type of the partition column while writing 
partitions. Consider for example the query:

create table tab1 (id1 int, id2 string) PARTITIONED BY(month string,day int) 
row format delimited fields terminated by ',';
alter table tab1 add partition (month='June', day='second');

Hive accepts this query. However if you try to select from this table and 
insert into another expecting schema match, it will insert nulls instead.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e971644 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 393ef57 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java a704462 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 07b271c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 767f545 
  ql/src/test/queries/clientnegative/illegal_partition_type.q PRE-CREATION 
  ql/src/test/queries/clientnegative/illegal_partition_type2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_type_check.q PRE-CREATION 
  ql/src/test/results/clientnegative/alter_table_add_partition.q.out bd9c148 
  ql/src/test/results/clientnegative/alter_view_failure5.q.out 4edb82c 
  ql/src/test/results/clientnegative/illegal_partition_type.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/illegal_partition_type2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/partition_type_check.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/14155/diff/


Testing
---

Ran all tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 14155: HIVE-5297 Hive does not honor type for partition columns

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14155/
---

(Updated Sept. 20, 2013, 1:49 a.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Addressed Harish and Ashutosh's comments. Took the approach of not updating 
with converted value because of Ashutosh's comments.


Bugs: HIVE-5297
https://issues.apache.org/jira/browse/HIVE-5297


Repository: hive-git


Description
---

Hive does not consider the type of the partition column while writing 
partitions. Consider for example the query:

create table tab1 (id1 int, id2 string) PARTITIONED BY(month string,day int) 
row format delimited fields terminated by ',';
alter table tab1 add partition (month='June', day='second');

Hive accepts this query. However if you try to select from this table and 
insert into another expecting schema match, it will insert nulls instead.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1af68a6 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 393ef57 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java a704462 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java fb79823 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 767f545 
  ql/src/test/queries/clientnegative/illegal_partition_type.q PRE-CREATION 
  ql/src/test/queries/clientnegative/illegal_partition_type2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_type_check.q PRE-CREATION 
  ql/src/test/results/clientnegative/alter_table_add_partition.q.out bd9c148 
  ql/src/test/results/clientnegative/alter_view_failure5.q.out 4edb82c 
  ql/src/test/results/clientnegative/illegal_partition_type.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/illegal_partition_type2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/partition_type_check.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/14155/diff/


Testing
---

Ran all tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 14155: HIVE-5297 Hive does not honor type for partition columns

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14155/
---

(Updated Sept. 17, 2013, 9:08 a.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Updated test results.


Bugs: HIVE-5297
https://issues.apache.org/jira/browse/HIVE-5297


Repository: hive-git


Description
---

Hive does not consider the type of the partition column while writing 
partitions. Consider for example the query:

create table tab1 (id1 int, id2 string) PARTITIONED BY(month string,day int) 
row format delimited fields terminated by ',';
alter table tab1 add partition (month='June', day='second');

Hive accepts this query. However if you try to select from this table and 
insert into another expecting schema match, it will insert nulls instead.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1af68a6 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 393ef57 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 2ece97e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java a704462 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java fb79823 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g ca667d4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 767f545 
  ql/src/test/queries/clientnegative/illegal_partition_type.q PRE-CREATION 
  ql/src/test/queries/clientnegative/illegal_partition_type2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_type_check.q PRE-CREATION 
  ql/src/test/results/clientnegative/alter_table_add_partition.q.out bd9c148 
  ql/src/test/results/clientnegative/alter_view_failure5.q.out 4edb82c 
  ql/src/test/results/clientnegative/illegal_partition_type.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/illegal_partition_type2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/parititon_type_check.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/partition_type_check.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/14155/diff/


Testing
---

Ran all tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 14155: HIVE-5297 Hive does not honor type for partition columns

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14155/
---

(Updated Sept. 17, 2013, 8:55 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Fixed a minor nit.


Bugs: HIVE-5297
https://issues.apache.org/jira/browse/HIVE-5297


Repository: hive-git


Description
---

Hive does not consider the type of the partition column while writing 
partitions. Consider for example the query:

create table tab1 (id1 int, id2 string) PARTITIONED BY(month string,day int) 
row format delimited fields terminated by ',';
alter table tab1 add partition (month='June', day='second');

Hive accepts this query. However if you try to select from this table and 
insert into another expecting schema match, it will insert nulls instead.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1af68a6 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 393ef57 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 2ece97e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java a704462 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java fb79823 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g ca667d4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 767f545 
  ql/src/test/queries/clientnegative/illegal_partition_type.q PRE-CREATION 
  ql/src/test/queries/clientnegative/illegal_partition_type2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_type_check.q PRE-CREATION 
  ql/src/test/results/clientnegative/alter_table_add_partition.q.out bd9c148 
  ql/src/test/results/clientnegative/alter_view_failure5.q.out 4edb82c 
  ql/src/test/results/clientnegative/illegal_partition_type.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/illegal_partition_type2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/parititon_type_check.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/partition_type_check.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/14155/diff/


Testing
---

Ran all tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 14155: HIVE-5297 Hive does not honor type for partition columns

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14155/#review26164
---



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
https://reviews.apache.org/r/14155/#comment51125

Folks may be depending on this 'feature'. Debatable, open to others 
opinions.



ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
https://reviews.apache.org/r/14155/#comment51126

Yes. 

Updated the check as well.



ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
https://reviews.apache.org/r/14155/#comment51127

Cannot as I need to look up the key in different maps and the ordering 
cannot be guaranteed.


- Vikram Dixit Kumaraswamy


On Sept. 17, 2013, 9:08 a.m., Vikram Dixit Kumaraswamy wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/14155/
 ---
 
 (Updated Sept. 17, 2013, 9:08 a.m.)
 
 
 Review request for hive and Ashutosh Chauhan.
 
 
 Bugs: HIVE-5297
 https://issues.apache.org/jira/browse/HIVE-5297
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Hive does not consider the type of the partition column while writing 
 partitions. Consider for example the query:
 
 create table tab1 (id1 int, id2 string) PARTITIONED BY(month string,day int) 
 row format delimited fields terminated by ',';
 alter table tab1 add partition (month='June', day='second');
 
 Hive accepts this query. However if you try to select from this table and 
 insert into another expecting schema match, it will insert nulls instead.
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1af68a6 
   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 393ef57 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 2ece97e 
   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
 a704462 
   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
 fb79823 
   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g ca667d4 
   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
 767f545 
   ql/src/test/queries/clientnegative/illegal_partition_type.q PRE-CREATION 
   ql/src/test/queries/clientnegative/illegal_partition_type2.q PRE-CREATION 
   ql/src/test/queries/clientpositive/partition_type_check.q PRE-CREATION 
   ql/src/test/results/clientnegative/alter_table_add_partition.q.out bd9c148 
   ql/src/test/results/clientnegative/alter_view_failure5.q.out 4edb82c 
   ql/src/test/results/clientnegative/illegal_partition_type.q.out 
 PRE-CREATION 
   ql/src/test/results/clientnegative/illegal_partition_type2.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/parititon_type_check.q.out PRE-CREATION 
   ql/src/test/results/clientpositive/partition_type_check.q.out PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/14155/diff/
 
 
 Testing
 ---
 
 Ran all tests.
 
 
 Thanks,
 
 Vikram Dixit Kumaraswamy
 

Review Request 14155: HIVE-5297 Hive does not honor type for partition columns

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14155/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-5297
https://issues.apache.org/jira/browse/HIVE-5297


Repository: hive-git


Description
---

Hive does not consider the type of the partition column while writing 
partitions. Consider for example the query:

create table tab1 (id1 int, id2 string) PARTITIONED BY(month string,day int) 
row format delimited fields terminated by ',';
alter table tab1 add partition (month='June', day='second');

Hive accepts this query. However if you try to select from this table and 
insert into another expecting schema match, it will insert nulls instead.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1af68a6 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 393ef57 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 2ece97e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java a704462 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g ca667d4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 767f545 
  ql/src/test/queries/clientnegative/illegal_partition_type.q PRE-CREATION 
  ql/src/test/queries/clientnegative/illegal_partition_type2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_type_check.q PRE-CREATION 
  ql/src/test/results/clientnegative/illegal_partition_type.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/illegal_partition_type2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/parititon_type_check.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/partition_type_check.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/14155/diff/


Testing
---

Ran all tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 14155: HIVE-5297 Hive does not honor type for partition columns

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14155/
---

(Updated Sept. 17, 2013, 1:14 a.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Addressed Sergey's comments.


Bugs: HIVE-5297
https://issues.apache.org/jira/browse/HIVE-5297


Repository: hive-git


Description
---

Hive does not consider the type of the partition column while writing 
partitions. Consider for example the query:

create table tab1 (id1 int, id2 string) PARTITIONED BY(month string,day int) 
row format delimited fields terminated by ',';
alter table tab1 add partition (month='June', day='second');

Hive accepts this query. However if you try to select from this table and 
insert into another expecting schema match, it will insert nulls instead.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1af68a6 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 393ef57 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 2ece97e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java a704462 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java fb79823 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g ca667d4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 767f545 
  ql/src/test/queries/clientnegative/illegal_partition_type.q PRE-CREATION 
  ql/src/test/queries/clientnegative/illegal_partition_type2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/partition_type_check.q PRE-CREATION 
  ql/src/test/results/clientnegative/illegal_partition_type.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/illegal_partition_type2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/parititon_type_check.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/partition_type_check.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/14155/diff/


Testing
---

Ran all tests.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 13697: HIVE-5129: Multiple table insert fails on count(distinct)

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13697/
---

(Updated Aug. 23, 2013, 4:37 p.m.)


Review request for hive and Navis Ryu.


Changes
---

Made changes to retain parallelism. Added new test case.


Bugs: HIVE-5129
https://issues.apache.org/jira/browse/HIVE-5129


Repository: hive-git


Description
---

Hive fails with class cast exception on multiple table insert fails on 
count(distinct).


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java a39fd21 
  ql/src/test/queries/clientpositive/multi_insert_gby3.q PRE-CREATION 
  ql/src/test/results/clientpositive/multi_insert_gby3.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/13697/diff/


Testing
---

Runs the test from HIVE-4692 successfully.


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 13697: HIVE-5129: Multiple table insert fails on count(distinct)

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13697/
---

(Updated Aug. 23, 2013, 10:44 p.m.)


Review request for hive and Navis Ryu.


Changes
---

Addressed Harish's comments. Added to the test case and fixed a unit test 
failure.


Bugs: HIVE-5129
https://issues.apache.org/jira/browse/HIVE-5129


Repository: hive-git


Description
---

Hive fails with class cast exception on multiple table insert fails on 
count(distinct).


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1d6e6fe 
  ql/src/test/queries/clientpositive/multi_insert_gby3.q PRE-CREATION 
  ql/src/test/results/clientpositive/multi_insert_gby3.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/13697/diff/


Testing
---

Runs the test from HIVE-4692 successfully.


Thanks,

Vikram Dixit Kumaraswamy

Review Request 13787: HIVE-5095: Hive needs new operator walker for parallelization/optimization for tez

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13787/
---

Review request for hive and Gunther Hagleitner.


Bugs: HIVE-5095
https://issues.apache.org/jira/browse/HIVE-5095


Repository: hive-git


Description
---

For tez to compute the number of reducers, we should be walking the operator 
tree in a topological fashion so that the reducers down the tree get the 
estimate from all parents. However, the current walkers in hive only walk the 
operator tree in a depth-first fashion. We need to add a new walker for the 
topological walk. Also, since information about the parent operators needs to 
be propagated on a per parent basis, we need to retain some context across 
operators to be passed to the child which the walker will co-ordinate.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7408a5a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java ca48f5e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 6a538e8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 6ee13ec 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java f3c34d1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 49a0ee3 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7433ddc 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/OpProcessor.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/OpProcContext.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenOpGraphWalker.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java 827637a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java ff8b17b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java c1c1da5 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 248eb03 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 5abedfe 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 5fd8d828 

Diff: https://reviews.apache.org/r/13787/diff/


Testing
---


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 13697: HIVE-5129: Multiple table insert fails on count(distinct)

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13697/
---

(Updated Aug. 22, 2013, 2:16 a.m.)


Review request for hive and Navis Ryu.


Changes
---

Made changes according to Navis' suggestion.


Bugs: HIVE-5129
https://issues.apache.org/jira/browse/HIVE-5129


Repository: hive-git


Description
---

Hive fails with class cast exception on multiple table insert fails on 
count(distinct).


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java a39fd21 

Diff: https://reviews.apache.org/r/13697/diff/


Testing
---

Runs the test from HIVE-4692 successfully.


Thanks,

Vikram Dixit Kumaraswamy

Review Request 13697: HIVE-5129: Multiple table insert fails on count(distinct)

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13697/
---

Review request for hive and Navis Ryu.


Bugs: HIVE-5129
https://issues.apache.org/jira/browse/HIVE-5129


Repository: hive-git


Description
---

Hive fails with class cast exception on multiple table insert fails on 
count(distinct).


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java a39fd21 

Diff: https://reviews.apache.org/r/13697/diff/


Testing
---

Runs the test from HIVE-4692 successfully.


Thanks,

Vikram Dixit Kumaraswamy

Review Request 13555: HIVE-5052: Set parallelism when generating the tez tasks

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13555/
---

Review request for hive.


Bugs: HIVE-5052
https://issues.apache.org/jira/browse/HIVE-5052


Repository: hive-git


Description
---

Set parallelism when generating the tez tasks.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7408a5a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java edb55fa 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java 48145ad 

Diff: https://reviews.apache.org/r/13555/diff/


Testing
---


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 13555: HIVE-5052: Set parallelism when generating the tez tasks

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13555/
---

(Updated Aug. 14, 2013, 7:15 a.m.)


Review request for hive.


Changes
---

Removed cruft.


Bugs: HIVE-5052
https://issues.apache.org/jira/browse/HIVE-5052


Repository: hive-git


Description
---

Set parallelism when generating the tez tasks.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7408a5a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java edb55fa 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java 48145ad 

Diff: https://reviews.apache.org/r/13555/diff/


Testing
---


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 12827: HIVE-4611 - SMB joins fail based on bigtable selection policy.

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12827/
---

(Updated Aug. 14, 2013, 7:21 p.m.)


Review request for hive, Ashutosh Chauhan, Brock Noland, and Gunther Hagleitner.


Changes
---

Addressed Ashutosh's comments.


Bugs: HIVE-4611
https://issues.apache.org/jira/browse/HIVE-4611


Repository: hive-git


Description
---

SMB joins fail based on bigtable selection policy. The default setting for 
hive.auto.convert.sortmerge.join.bigtable.selection.policy will choose the big 
table as the one with largest average partition size. However, this can result 
in a query failing because this policy conflicts with the big table candidates 
chosen for outer joins. This policy should just be a tie breaker and not have 
the ultimate say in the choice of tables.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 12e9334 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractSMBJoinProc.java 
fda2f84 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/AvgPartitionSizeBasedBigTableSelectorForAutoSMJ.java
 1bed28f 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/BigTableSelectorForAutoSMJ.java 
db5ff0f 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LeftmostBigTableSelectorForAutoSMJ.java
 db3c9e7 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java cd1b4ad 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/TableSizeBasedBigTableSelectorForAutoSMJ.java
 d33ea91 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationOptimizer.java
 3071713 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java
 e214807 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java
 da5115b 
  ql/src/test/queries/clientnegative/auto_sortmerge_join_1.q c858254 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_15.q PRE-CREATION 
  ql/src/test/results/clientnegative/auto_sortmerge_join_1.q.out 0eddb69 
  ql/src/test/results/clientnegative/smb_bucketmapjoin.q.out 7a5b8c1 
  ql/src/test/results/clientpositive/auto_sortmerge_join_15.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/12827/diff/


Testing
---

All tests pass on hadoop 1.


Thanks,

Vikram Dixit Kumaraswamy

Review Request 13507: HIVE-5003: Localize hive exec jar for tez

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13507/
---

Review request for hive.


Bugs: HIVE-5003 and HIVE-5004
https://issues.apache.org/jira/browse/HIVE-5003
https://issues.apache.org/jira/browse/HIVE-5004


Repository: hive-git


Description
---

Tez localization of exec and additional jars.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 79c38c1 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 12e9334 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java faa99f7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java ac536e2 

Diff: https://reviews.apache.org/r/13507/diff/


Testing
---


Thanks,

Vikram Dixit Kumaraswamy

Re: Review Request 12827: HIVE-4611 - SMB joins fail based on bigtable selection policy.

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12827/
---

(Updated Aug. 5, 2013, 5:57 p.m.)


Review request for hive, Ashutosh Chauhan, Brock Noland, and Gunther Hagleitner.


Changes
---

Addressed Gunther's comments.


Bugs: HIVE-4611
https://issues.apache.org/jira/browse/HIVE-4611


Repository: hive-git


Description
---

SMB joins fail based on bigtable selection policy. The default setting for 
hive.auto.convert.sortmerge.join.bigtable.selection.policy will choose the big 
table as the one with largest average partition size. However, this can result 
in a query failing because this policy conflicts with the big table candidates 
chosen for outer joins. This policy should just be a tie breaker and not have 
the ultimate say in the choice of tables.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 12e9334 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractSMBJoinProc.java 
cc9de54 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/AvgPartitionSizeBasedBigTableSelectorForAutoSMJ.java
 5320143 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/BigTableSelectorForAutoSMJ.java 
db5ff0f 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LeftmostBigTableSelectorForAutoSMJ.java
 db3c9e7 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java cd1b4ad 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/TableSizeBasedBigTableSelectorForAutoSMJ.java
 b882f87 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationOptimizer.java
 3071713 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java
 e214807 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java
 da5115b 
  ql/src/test/queries/clientnegative/auto_sortmerge_join_1.q c858254 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_15.q PRE-CREATION 
  ql/src/test/results/clientnegative/auto_sortmerge_join_1.q.out 0eddb69 
  ql/src/test/results/clientnegative/smb_bucketmapjoin.q.out 7a5b8c1 
  ql/src/test/results/clientpositive/auto_sortmerge_join_15.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/12827/diff/


Testing
---

All tests pass on hadoop 1.


Thanks,

Vikram Dixit Kumaraswamy

Review Request 13032: HIVE-4826 Setup build infrastructure for tez

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13032/
---

Review request for hive, Ashutosh Chauhan and Gunther Hagleitner.


Bugs: HIVE-4826
https://issues.apache.org/jira/browse/HIVE-4826


Repository: hive-git


Description
---

Setup build infrastructure for tez.


Diffs
-

  build-common.xml 0807827 
  build.xml 016d363 
  eclipse-templates/.classpath 7114b90 
  ivy/libraries.properties 4a8edce 
  ql/ivy.xml bfb3116 
  shims/ivy.xml 04ef641 

Diff: https://reviews.apache.org/r/13032/diff/


Testing
---

All unit tests pass.


Thanks,

Vikram Dixit Kumaraswamy

Review Request 12894: HIVE-4910: Hadoop 2 archives broken

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12894/
---

Review request for hive, Ashutosh Chauhan and Gunther Hagleitner.


Bugs: HIVE-4910
https://issues.apache.org/jira/browse/HIVE-4910


Repository: hive-git


Description
---

Hadoop 2 archive tests are broken. The issue stems from the fact that har uri 
construction does not really have a port in the URI when unit tests are run. 
This means that an invalid uri is constructed resulting in failures.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 4dcb260 
  ql/src/test/queries/clientpositive/archive_excludeHadoop20.q ddc06a9 
  ql/src/test/results/clientpositive/archive_excludeHadoop20.q.out 5c14851 

Diff: https://reviews.apache.org/r/12894/diff/


Testing
---

Unit tests pass.


Thanks,

Vikram Dixit Kumaraswamy

Review Request 12827: HIVE-4611 - SMB joins fail based on bigtable selection policy.

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12827/
---

Review request for hive, Ashutosh Chauhan, Brock Noland, and Gunther Hagleitner.


Bugs: HIVE-4611
https://issues.apache.org/jira/browse/HIVE-4611


Repository: hive-git


Description
---

SMB joins fail based on bigtable selection policy. The default setting for 
hive.auto.convert.sortmerge.join.bigtable.selection.policy will choose the big 
table as the one with largest average partition size. However, this can result 
in a query failing because this policy conflicts with the big table candidates 
chosen for outer joins. This policy should just be a tie breaker and not have 
the ultimate say in the choice of tables.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 8330f65 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractSMBJoinProc.java 
cc9de54 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/AvgPartitionSizeBasedBigTableSelectorForAutoSMJ.java
 5320143 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/BigTableSelectorForAutoSMJ.java 
db5ff0f 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LeftmostBigTableSelectorForAutoSMJ.java
 db3c9e7 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java d83fb66 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/TableSizeBasedBigTableSelectorForAutoSMJ.java
 b882f87 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationOptimizer.java
 3071713 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java
 f98878c 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java
 af56857 
  ql/src/test/queries/clientnegative/auto_sortmerge_join_1.q c858254 
  ql/src/test/queries/clientpositive/auto_sortmerge_join_15.q PRE-CREATION 
  ql/src/test/results/clientnegative/auto_sortmerge_join_1.q.out 0eddb69 
  ql/src/test/results/clientnegative/join2.q.out b53b3a1 
  ql/src/test/results/clientnegative/smb_bucketmapjoin.q.out 7a5b8c1 
  ql/src/test/results/clientpositive/auto_sortmerge_join_15.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/12827/diff/


Testing
---

All tests pass on hadoop 1.


Thanks,

Vikram Dixit Kumaraswamy