[
https://issues.apache.org/jira/browse/PIG-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424070#comment-13424070
]
Bill Graham commented on PIG-2779:
----------------------------------
Thanks Jie, I've been running the test suite on our CI server and so far things
look good. We're close. A few more comments though:
- In {{JobControlCompiler}} we should only call {{estimateNumberOfReducers}} if
we need to, since we don't need it if {{requestedParallelism}} or
{{defaultParallel}} will govern and the call could be expensive. So we'd have
this:
{noformat}
} else {
mro.estimatedParallelism = estimateNumberOfReducers(conf, lds, nwJob);
jobParallelism = mro.estimatedParallelism;
}
{noformat}
- The semantics of {{pig.info.reducers.requested.parallel}} is a bit misleading
as implemented. I would expect that would be the value set via the {{PARALLEL}}
statement, but that's not the case, since {{requestedParallel}} gets set to
{{jobParallelism}} on line 796 of JCC. Would you please add to the tests in
{{TestJobSubmission}} that each of the {{pig.info.reducers.*}} fields are set
as expected (or not set) after each of the scenarios. I suspect there are cases
where {{pig.info.reducers.requested.parallel}} is being set when {{PARALLEL}}
isn't used.
> Refactoring the code for setting number of reducers
> ---------------------------------------------------
>
> Key: PIG-2779
> URL: https://issues.apache.org/jira/browse/PIG-2779
> Project: Pig
> Issue Type: Bug
> Reporter: Jie Li
> Assignee: Jie Li
> Fix For: 0.11
>
> Attachments: PIG-2779.0.patch, PIG-2779.1.patch, PIG-2779.2.patch,
> PIG-2779.3.patch, TestNumberOfReducers.java, TestNumberOfReducers.java
>
>
> As PIG-2652 observed, currently the code for setting number of reducers is a
> little messy. MapReduceOper.requestedParallelism seems being misused in some
> plases, and now we support runtime estimation of #reducer which further
> complicates the problem.
> For example, if we specify parallel 1 for the order-by, the estimated
> #reducer will be used. If we specify parallel 2 while it estimates 4,
> order-by will fail due to "Illegal partition for Null". If we specify
> parallel 4 while it estimates 2, then some reducers will have nothing to do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
