Re: Review Request 24427: HIVE-7616 pre-size mapjoin hashtable based on statistics

2014-08-08 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24427/
---

(Updated Aug. 8, 2014, 11:42 p.m.)


Review request for hive, Gunther Hagleitner, Mostafa Mokhtar, and Prasanth_J.


Repository: hive-git


Description
---

See jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8490558 
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java cf64aa0 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
 cdb5dc5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
5b3b770 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
 629457c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 6d292d0 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
29d895a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 44cb9c0 
  ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 4173ea4 
  ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 3f36851 
  ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 567b0ca 
  ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 9e90ec2 

Diff: https://reviews.apache.org/r/24427/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 24427: HIVE-7616 pre-size mapjoin hashtable based on statistics

2014-08-08 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24427/
---

(Updated Aug. 8, 2014, 11:36 p.m.)


Review request for hive, Gunther Hagleitner, Mostafa Mokhtar, and Prasanth_J.


Repository: hive-git


Description
---

See jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8490558 
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java cf64aa0 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
 cdb5dc5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
5b3b770 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
 629457c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 6d292d0 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
29d895a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 44cb9c0 
  ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 4173ea4 
  ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 3f36851 
  ql/src/test/results/clientpositive/bucket_map_join_1.q.out 63fb0d1 
  ql/src/test/results/clientpositive/bucket_map_join_2.q.out 21f2d5a 
  ql/src/test/results/clientpositive/bucketcontext_1.q.out 5212de3 
  ql/src/test/results/clientpositive/bucketcontext_2.q.out d86c430 
  ql/src/test/results/clientpositive/bucketcontext_3.q.out a536e8b 
  ql/src/test/results/clientpositive/bucketcontext_4.q.out 26c8720 
  ql/src/test/results/clientpositive/bucketcontext_5.q.out 2619cfb 
  ql/src/test/results/clientpositive/bucketcontext_6.q.out 4c42ca7 
  ql/src/test/results/clientpositive/bucketcontext_7.q.out 7e5afb5 
  ql/src/test/results/clientpositive/bucketcontext_8.q.out 243b67a 
  ql/src/test/results/clientpositive/bucketmapjoin1.q.out 10f1af4 
  ql/src/test/results/clientpositive/bucketmapjoin10.q.out f852cde 
  ql/src/test/results/clientpositive/bucketmapjoin11.q.out 97e80fb 
  ql/src/test/results/clientpositive/bucketmapjoin12.q.out e486ca5 
  ql/src/test/results/clientpositive/bucketmapjoin2.q.out 297412f 
  ql/src/test/results/clientpositive/bucketmapjoin3.q.out 7f307a0 
  ql/src/test/results/clientpositive/bucketmapjoin4.q.out f0f9aee 
  ql/src/test/results/clientpositive/bucketmapjoin5.q.out 79e1c3d 
  ql/src/test/results/clientpositive/bucketmapjoin8.q.out e504c9d 
  ql/src/test/results/clientpositive/bucketmapjoin9.q.out 18f350a 
  ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out 751e32f 
  ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out 3eb70d1 
  ql/src/test/results/clientpositive/bucketmapjoin_negative3.q.out 34abe4f 
  ql/src/test/results/clientpositive/join26.q.out bf8cf57 
  ql/src/test/results/clientpositive/join32.q.out ff0d7cc 
  ql/src/test/results/clientpositive/join33.q.out ff0d7cc 
  ql/src/test/results/clientpositive/join34.q.out b52777a 
  ql/src/test/results/clientpositive/join35.q.out 20c69ea 
  ql/src/test/results/clientpositive/join_map_ppr.q.out 51fb6c6 
  ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 567b0ca 
  ql/src/test/results/clientpositive/sample8.q.out e0c0f9e 
  ql/src/test/results/clientpositive/smb_mapjoin_11.q.out d59b801 
  ql/src/test/results/clientpositive/sort_merge_join_desc_5.q.out ba8928b 
  ql/src/test/results/clientpositive/sort_merge_join_desc_6.q.out d51a54e 
  ql/src/test/results/clientpositive/sort_merge_join_desc_7.q.out fcb6367 
  ql/src/test/results/clientpositive/stats11.q.out c5531c5 
  ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 9e90ec2 
  ql/src/test/results/clientpositive/transform_ppr1.q.out 6f908fa 
  ql/src/test/results/clientpositive/transform_ppr2.q.out 9285151 
  ql/src/test/results/clientpositive/union22.q.out 884c106 
  ql/src/test/results/clientpositive/union_ppr.q.out ee209c7 

Diff: https://reviews.apache.org/r/24427/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 24427: HIVE-7616 pre-size mapjoin hashtable based on statistics

2014-08-08 Thread Gunther Hagleitner 


> On Aug. 7, 2014, 12:04 a.m., Gunther Hagleitner wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java, 
> > line 136
> > 
> >
> > curlies per coding standard
> 
> Sergey Shelukhin wrote:
> added; next time I review your patch, I'll enforce C variable 
> declarations (all variables in the beginning of the block) which are also 
> part of the same Sun standard Hive wiki page point to :P

I do that anyways.


- Gunther


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24427/#review49830
---


On Aug. 7, 2014, 11:53 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24427/
> ---
> 
> (Updated Aug. 7, 2014, 11:53 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Mostafa Mokhtar, and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8490558 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 
> cf64aa0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
>  cdb5dc5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
> 5b3b770 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
>  629457c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 6d292d0 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
> d42e1f7 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
> 29d895a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 44cb9c0 
> 
> Diff: https://reviews.apache.org/r/24427/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 24427: HIVE-7616 pre-size mapjoin hashtable based on statistics

2014-08-08 Thread Gunther Hagleitner 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24427/#review50071
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java


I'm not sure about this one. But shouldn't the condition be: 
joinConf.isBucketMapJoin()?


- Gunther Hagleitner


On Aug. 7, 2014, 11:53 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24427/
> ---
> 
> (Updated Aug. 7, 2014, 11:53 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Mostafa Mokhtar, and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8490558 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 
> cf64aa0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
>  cdb5dc5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
> 5b3b770 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
>  629457c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 6d292d0 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
> d42e1f7 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
> 29d895a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 44cb9c0 
> 
> Diff: https://reviews.apache.org/r/24427/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 24427: HIVE-7616 pre-size mapjoin hashtable based on statistics

2014-08-07 Thread Lefty Leverenz 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24427/#review4
---


HiveConf.java parameter descriptions look good.  I just mentioned two nano-nits 
to prove I'd done the review.


common/src/java/org/apache/hadoop/hive/conf/HiveConf.java


utterly ignorable nitpick:  leading space isn't necessary (" of the number 
of keys ...)



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java


even more ignorable:  trailing space isn't necessary ("Initial capacity of 
")


- Lefty Leverenz


On Aug. 7, 2014, 11:53 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24427/
> ---
> 
> (Updated Aug. 7, 2014, 11:53 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Mostafa Mokhtar, and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8490558 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 
> cf64aa0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
>  cdb5dc5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
> 5b3b770 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
>  629457c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 6d292d0 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
> d42e1f7 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
> 29d895a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 44cb9c0 
> 
> Diff: https://reviews.apache.org/r/24427/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 24427: HIVE-7616 pre-size mapjoin hashtable based on statistics

2014-08-07 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24427/
---

(Updated Aug. 7, 2014, 11:53 p.m.)


Review request for hive, Gunther Hagleitner, Mostafa Mokhtar, and Prasanth_J.


Repository: hive-git


Description
---

See jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8490558 
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java cf64aa0 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
 cdb5dc5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
5b3b770 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
 629457c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 6d292d0 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
d42e1f7 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
29d895a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 44cb9c0 

Diff: https://reviews.apache.org/r/24427/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 24427: HIVE-7616 pre-size mapjoin hashtable based on statistics

2014-08-07 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24427/
---

(Updated Aug. 7, 2014, 11:51 p.m.)


Review request for hive, Gunther Hagleitner, Mostafa Mokhtar, and Prasanth_J.


Repository: hive-git


Description
---

See jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8490558 
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java cf64aa0 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
 cdb5dc5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
5b3b770 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
 629457c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 6d292d0 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
d42e1f7 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
29d895a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 44cb9c0 

Diff: https://reviews.apache.org/r/24427/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 24427: HIVE-7616 pre-size mapjoin hashtable based on statistics

2014-08-07 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24427/
---

(Updated Aug. 7, 2014, 11:48 p.m.)


Review request for hive, Gunther Hagleitner, Mostafa Mokhtar, and Prasanth_J.


Repository: hive-git


Description
---

See jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8490558 
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java cf64aa0 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
 cdb5dc5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
5b3b770 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
 629457c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 6d292d0 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
29d895a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 44cb9c0 

Diff: https://reviews.apache.org/r/24427/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 24427: HIVE-7616 pre-size mapjoin hashtable based on statistics

2014-08-07 Thread Sergey Shelukhin 


> On Aug. 7, 2014, 12:04 a.m., Gunther Hagleitner wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java, 
> > line 136
> > 
> >
> > curlies per coding standard

added; next time I review your patch, I'll enforce C variable declarations (all 
variables in the beginning of the block) which are also part of the same Sun 
standard Hive wiki page point to :P


> On Aug. 7, 2014, 12:04 a.m., Gunther Hagleitner wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java, 
> > line 146
> > 
> >
> > i think this number needs to be adjusted for bucketed map join. 
> > otherwise you'll over allocate in that case, but we take the bucketing into 
> > consideration when we do size estimation for the overall operator...

can you elaborate on "but we take the bucketing into consideration when we do 
size estimation for the overall operator..."? 


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24427/#review49830
---


On Aug. 6, 2014, 10 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24427/
> ---
> 
> (Updated Aug. 6, 2014, 10 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Mostafa Mokhtar, and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8490558 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 
> cf64aa0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
>  cdb5dc5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
> 5b3b770 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
>  629457c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 6d292d0 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
> 29d895a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 44cb9c0 
> 
> Diff: https://reviews.apache.org/r/24427/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 24427: HIVE-7616 pre-size mapjoin hashtable based on statistics

2014-08-06 Thread Gunther Hagleitner 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24427/#review49830
---



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java


I think once you cross 1000 characters some underscores help readability. 
Or drop stats and estimate from the name.



ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java


this would be good to know at the info level i think. also, you've copied 
the lines above for the wrapper but not the logging.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java


You initialize with null - why use Long.MAX_VALUE here?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java


ditto



ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java


curlies per coding standard



ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java


i think this number needs to be adjusted for bucketed map join. otherwise 
you'll over allocate in that case, but we take the bucketing into consideration 
when we do size estimation for the overall operator...



ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java


todo... put jira number, fix it or drop.



ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java


confusing name. we already have "stats" in each desc, which has multiple 
values. how about parentToNumberKeyEstimate


- Gunther Hagleitner


On Aug. 6, 2014, 10 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24427/
> ---
> 
> (Updated Aug. 6, 2014, 10 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Mostafa Mokhtar, and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8490558 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 
> cf64aa0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
>  cdb5dc5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
> 5b3b770 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
>  629457c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 6d292d0 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
> 29d895a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 44cb9c0 
> 
> Diff: https://reviews.apache.org/r/24427/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 24427: HIVE-7616 pre-size mapjoin hashtable based on statistics

2014-08-06 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24427/
---

(Updated Aug. 6, 2014, 10 p.m.)


Review request for hive, Gunther Hagleitner, Mostafa Mokhtar, and Prasanth_J.


Repository: hive-git


Description
---

See jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8490558 
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java cf64aa0 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
 cdb5dc5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
5b3b770 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
 629457c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 6d292d0 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
29d895a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 44cb9c0 

Diff: https://reviews.apache.org/r/24427/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 24427: HIVE-7616 pre-size mapjoin hashtable based on statistics

2014-08-06 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24427/#review49820
---



ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java


should also divide by HIVEHASHTABLELOADFACTOR here and in another place


- Sergey Shelukhin


On Aug. 6, 2014, 9:26 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24427/
> ---
> 
> (Updated Aug. 6, 2014, 9:26 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Mostafa Mokhtar, and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8490558 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 
> cf64aa0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
>  cdb5dc5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
> 5b3b770 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
>  629457c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 6d292d0 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
> 29d895a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 44cb9c0 
> 
> Diff: https://reviews.apache.org/r/24427/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 24427: HIVE-7616 pre-size mapjoin hashtable based on statistics

2014-08-06 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24427/#review49818
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java


this is wrong


- Sergey Shelukhin


On Aug. 6, 2014, 9:26 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24427/
> ---
> 
> (Updated Aug. 6, 2014, 9:26 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Mostafa Mokhtar, and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8490558 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 
> cf64aa0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
>  cdb5dc5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
> 5b3b770 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
>  629457c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 6d292d0 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
> 29d895a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 44cb9c0 
> 
> Diff: https://reviews.apache.org/r/24427/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 24427: HIVE-7616 pre-size mapjoin hashtable based on statistics

2014-08-06 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24427/#review49814
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java


Can you add a null check here?


- Prasanth_J


On Aug. 6, 2014, 9:26 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24427/
> ---
> 
> (Updated Aug. 6, 2014, 9:26 p.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Mostafa Mokhtar, and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8490558 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 
> cf64aa0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
>  cdb5dc5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
> 5b3b770 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
>  629457c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 6d292d0 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
> 29d895a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 44cb9c0 
> 
> Diff: https://reviews.apache.org/r/24427/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 24427: HIVE-7616 pre-size mapjoin hashtable based on statistics

2014-08-06 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24427/
---

Review request for hive, Gunther Hagleitner, Mostafa Mokhtar, and Prasanth_J.


Repository: hive-git


Description
---

See jira


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8490558 
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java cf64aa0 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
 cdb5dc5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
5b3b770 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
 629457c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 6d292d0 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
29d895a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 44cb9c0 

Diff: https://reviews.apache.org/r/24427/diff/


Testing
---


Thanks,

Sergey Shelukhin