zabetak commented on a change in pull request #1035:
URL: https://github.com/apache/hive/pull/1035#discussion_r433508844
##########
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplicationUtils.java
##########
@@ -197,6 +186,27 @@ public static boolean merge(ReduceSinkOperator cRS,
ReduceSinkOperator pRS, int
return true;
}
+ private static long estimateReducers(HiveConf conf, ReduceSinkOperator rs) {
+ // TODO: Check if we can somehow exploit the logic in SetReducerParallelism
+ if (rs.getConf().getNumReducers() > 0) {
+ return rs.getConf().getNumReducers();
+ }
+ int constantReducers = conf.getIntVar(HiveConf.ConfVars.HADOOPNUMREDUCERS);
+ if (constantReducers > 0) {
+ return constantReducers;
+ }
+ long inputTotalBytes = 0;
+ List<Operator<?>> rsSiblings =
rs.getChildOperators().get(0).getParentOperators();
+ for (Operator<?> sibling : rsSiblings) {
+ if (sibling.getStatistics() != null) {
Review comment:
In the caller of this method, before checking if the new parallelism is
very low, we are checking if the parallelism changes (`newParallelism <
oldParallelism`). If the parallelism does not change then we proceed in the
deduplication as usual. If we don't have stats then the snippet below will
return 1. The same will happen for both parent RS and child RS so the
parallelism does not change since `1 < 1` returns false.
If you prefer to have an explicit check for the presence/absence of stats I
can try to add it. Let me know.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
