[
https://issues.apache.org/jira/browse/PIG-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415911#comment-13415911
]
Bill Graham commented on PIG-2779:
----------------------------------
Jie, these changes make sense to me, assuming tests pass. Just a few questions
and minor nits:
{{MRCompiler}}
1. You removed some logic which seems to now be in JCC. What about the removed
call to {{eng.getJobConf().getNumReduceTasks()}}? Is that still being picked up
elsewhere? It appears like that check was dropped.
2. {{HExecutionEngine}} inport no longer needed.
{{JobControlCompiler}}
3. Should {{ParallelConstantVisitor}} be called with {{mro.reducePlan}} or
{{nextMro.reducePlan}}?
4. {{calculateRuntimeReducers}} doesn't need {{MROperPlan plan}} in it's
signature.
5. Line 804, if statements should have curly braces.
6. Line 819, should have space between {{else}} and trailing curly bracket.
{{test/smoke-tests}}
7. Did you mean to include this in the patch?
> Refactoring the code for setting number of reducers
> ---------------------------------------------------
>
> Key: PIG-2779
> URL: https://issues.apache.org/jira/browse/PIG-2779
> Project: Pig
> Issue Type: Bug
> Reporter: Jie Li
> Assignee: Jie Li
> Fix For: 0.11
>
> Attachments: PIG-2779.0.patch, PIG-2779.1.patch,
> TestNumberOfReducers.java, TestNumberOfReducers.java
>
>
> As PIG-2652 observed, currently the code for setting number of reducers is a
> little messy. MapReduceOper.requestedParallelism seems being misused in some
> plases, and now we support runtime estimation of #reducer which further
> complicates the problem.
> For example, if we specify parallel 1 for the order-by, the estimated
> #reducer will be used. If we specify parallel 2 while it estimates 4,
> order-by will fail due to "Illegal partition for Null". If we specify
> parallel 4 while it estimates 2, then some reducers will have nothing to do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
