[
https://issues.apache.org/jira/browse/PIG-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424105#comment-13424105
]
Jie Li commented on PIG-2779:
-----------------------------
Hi Bill, regarding the first comment, I agree we can avoid estimating and
should be adjusted before. But now would you need it for later analysis in any
way? (e.g. comparing estimated #reducer and PALALLEL keyword). If not I'll fix
it then.
Regarding the second comment, yes requestedParallel has been used for multiple
purposes for a long time and adjusted across logical/physical/mr phases, and
also set as the final number of reducers. Many unit tests also assume it as the
final number of reducers. I'm afraid cleaning it up can be another ticket? Or
maybe an easier approach is to add another field just recording the PARALLEL
keyword?
> Refactoring the code for setting number of reducers
> ---------------------------------------------------
>
> Key: PIG-2779
> URL: https://issues.apache.org/jira/browse/PIG-2779
> Project: Pig
> Issue Type: Bug
> Reporter: Jie Li
> Assignee: Jie Li
> Fix For: 0.11
>
> Attachments: PIG-2779.0.patch, PIG-2779.1.patch, PIG-2779.2.patch,
> PIG-2779.3.patch, TestNumberOfReducers.java, TestNumberOfReducers.java
>
>
> As PIG-2652 observed, currently the code for setting number of reducers is a
> little messy. MapReduceOper.requestedParallelism seems being misused in some
> plases, and now we support runtime estimation of #reducer which further
> complicates the problem.
> For example, if we specify parallel 1 for the order-by, the estimated
> #reducer will be used. If we specify parallel 2 while it estimates 4,
> order-by will fail due to "Illegal partition for Null". If we specify
> parallel 4 while it estimates 2, then some reducers will have nothing to do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
