Gopal V created HIVE-20973:
------------------------------
Summary: Optimizer: Reduce de-dup changes the hash-function of a
reducer edge
Key: HIVE-20973
URL: https://issues.apache.org/jira/browse/HIVE-20973
Project: Hive
Issue Type: Bug
Reporter: Gopal V
{code}
private static void
propagateMaxNumReducers(ReduceSinkJoinDeDuplicateProcCtx dedupCtx,
ReduceSinkOperator rsOp, int maxNumReducers) throws
SemanticException {
if (rsOp == null) {
// Bail out
return;
}
if (rsOp.getChildOperators().get(0) instanceof MapJoinOperator ||
rsOp.getChildOperators().get(0) instanceof
CommonMergeJoinOperator) {
for (Operator<?> p :
rsOp.getChildOperators().get(0).getParentOperators()) {
ReduceSinkOperator pRSOp = (ReduceSinkOperator) p;
pRSOp.getConf().setReducerTraits(EnumSet.of(ReducerTraits.FIXED));
pRSOp.getConf().setNumReducers(maxNumReducers);
LOG.debug("Set {} to FIXED parallelism: {}", pRSOp, maxNumReducers);
if (pRSOp.getConf().isForwarding()) {
ReduceSinkOperator newRSOp =
CorrelationUtilities.findFirstPossibleParent(
pRSOp, ReduceSinkOperator.class, dedupCtx.trustScript());
propagateMaxNumReducers(dedupCtx, newRSOp, maxNumReducers);
}
}
{code}
FIXED used to mean AUTOPARALLEL=false, but now FIXED means a different hash
function from UNIFORM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
