[
https://issues.apache.org/jira/browse/PIG-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13792969#comment-13792969
]
Aniket Mokashi commented on PIG-3497:
-------------------------------------
Thanks Daniel!
I found some related test failures-
junit.framework.AssertionFailedError: Unexpected value found in configs for
pig.info.reducers.estimated.parallel expected:<6> but was:<-1>
at org.apache.pig.test.Util.assertConfLong(Util.java:1223)
at org.apache.pig.test.Util.assertParallelValues(Util.java:1218)
at
org.apache.pig.test.TestJobSubmission.testReducerNumEstimation(TestJobSubmission.java:570)
> JobControlCompiler should only do reducer estimation when the job has a
> reduce phase
> ------------------------------------------------------------------------------------
>
> Key: PIG-3497
> URL: https://issues.apache.org/jira/browse/PIG-3497
> Project: Pig
> Issue Type: Bug
> Reporter: Akihiro Matsukawa
> Assignee: Akihiro Matsukawa
> Priority: Minor
> Fix For: 0.12.0
>
> Attachments: reducer_estimation.patch
>
>
> Currently, JobControlCompiler makes an estimation for the number of reducers
> required (by default based on input size into mappers) regardless of whether
> there is a reduce phase in the job. This is unnecessary, especially when
> running more complex custom reducer estimators.
> Change to only estimate reducers when necessary.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
