[ 
https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785687#action_12785687
 ] 

Raghotham Murthy commented on HIVE-963:
---------------------------------------

Actually, after thinking about this a little more, we dont need a hashmap or 
jdbm for skewed joins. We run out of memory in the reducer when data is skewed. 
And in the reducer, data is already sorted. 

So, we just need to read the first table's data into an array and when we run 
out of memory, just dump the remaining data into a tmp file. Then, while 
streaming through the second table, we first join with the data in memory and 
then swap the contents of the tmp file for the rest of the join. We could 
optimize by reading in several tuples from the second table into memory instead 
of processing one tuple at a time.

does that make sense?

> no out of memory errors for skewed join
> ---------------------------------------
>
>                 Key: HIVE-963
>                 URL: https://issues.apache.org/jira/browse/HIVE-963
>             Project: Hadoop Hive
>          Issue Type: Improvement
>          Components: Query Processor
>            Reporter: Namit Jain
>            Assignee: Ning Zhang
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of 
> values for a given key exceed a given number.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to