[
https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15279701#comment-15279701
]
Wei Zheng commented on HIVE-13730:
----------------------------------
The root cause is that in some cases (e.g. when
hive.auto.convert.join.noconditionaltask.size is set very small), a hash
partition can be empty when it's being spilled (the memory estimation logic is
conservative and strict, so even without loading any row into a hash partition,
it can still assume the memory is about to get full, thus choose a partition to
spill).
Still, spilling an empty hash partition is OK. The problem happens during
deserialization of the spilled hash partition (BytesBytesMultiHashMap). If the
hash partition is empty, it will result in the refs array in the hashmap to
have a length of only 1. This causes problem of putRow as the backtrace above
shows, because it couldn't find a proper slot for inserting.
The solution is to instantiate a new BytesBytesMultiHashMap manually if we
figure out the deserialized hashmap is empty. This way we can have a properly
constructed refs array for it.
> hybridgrace_hashjoin_1.q test gets stuck
> ----------------------------------------
>
> Key: HIVE-13730
> URL: https://issues.apache.org/jira/browse/HIVE-13730
> Project: Hive
> Issue Type: Bug
> Components: Tez
> Affects Versions: 2.1.0
> Reporter: Vikram Dixit K
> Assignee: Wei Zheng
> Priority: Blocker
>
> I am seeing hybridgrace_hashjoin_1.q getting stuck on master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
