[
https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943745#comment-15943745
]
Gopal V edited comment on HIVE-16296 at 3/27/17 6:03 PM:
---------------------------------------------------------
The auto-reducer parallelism is entirely disabled for hash-joins (because when
scaling down the reducers, the hash can run out of memory if the streaming side
turns out to be tiny like in a LOJ).
Instead of checking for ARP and bailing, this needs the other ReducerTrait to
be passed on (ReducerTraits.UNIFORM). For reduce sinks which have UNIFORM
setup, the reducer counts can be increased (but not decreased).
ReduceWork can carry a boolean for UNIFORM (which is pretty much the opposite
of FIXED) and scale those reducers up as well.
was (Author: gopalv):
The auto-reducer parallelism is entirely disabled for hash-joins.
Instead of checking for ARP and bailing, this needs the other ReducerTrait to
be passed on (ReducerTraits.UNIFORM). For reduce sinks which have UNIFORM
setup, the reducer counts can be increased (but not decreased).
ReduceWork can carry a boolean for UNIFORM (which is pretty much the opposite
of FIXED) and scale those reducers up as well.
> use LLAP executor count to configure reducer auto-parallelism
> -------------------------------------------------------------
>
> Key: HIVE-16296
> URL: https://issues.apache.org/jira/browse/HIVE-16296
> Project: Hive
> Issue Type: Bug
> Reporter: Sergey Shelukhin
> Assignee: Sergey Shelukhin
> Attachments: HIVE-16296.01.patch, HIVE-16296.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
