[
https://issues.apache.org/jira/browse/PIG-304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated PIG-304:
---------------------------
Attachment: movelocalrearrange.patch
This patch adds a post MR compile visitor that moves local rearranges from the
reducer to the next mapper. This is not the best solution. The best solution
would be to make the maps somehow no-ops, so that we can avoid that stage.
Ways to do that need further investigation. The next best solution would be to
correct the logic in MRCompiler to place the local rearranges in the subsequent
mapper instead of the reducer. But Shravan is out until Aug 4 and I don't want
to change that code without his input. And seems to work.
> Distinct fail if previous map plan is closed
> --------------------------------------------
>
> Key: PIG-304
> URL: https://issues.apache.org/jira/browse/PIG-304
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Daniel Dai
> Fix For: types_branch
>
> Attachments: distinct.patch, movelocalrearrange.patch
>
>
> Distinct will fail when the previous map plan is closed. For example, the
> following script fail:
> a = load 'a';
> b = group a by $0;
> c = foreach b generate $1;
> d = distinct c;
> dump d;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
