[
https://issues.apache.org/jira/browse/PIG-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916538#action_12916538
]
Thejas M Nair commented on PIG-1629:
------------------------------------
Similar optimization can be done for inner filter as well -
C = foreach B{ D = filter A by x > 0; generate group, MyUDF(D);}
Changes required-
- group physical/MR plan implementation to have an inner limit/filter.
- logical optimizer rules to make the load/filter an inner plan of groupp
> Need ability to limit bags produced during GROUP + LIMIT
> --------------------------------------------------------
>
> Key: PIG-1629
> URL: https://issues.apache.org/jira/browse/PIG-1629
> Project: Pig
> Issue Type: Improvement
> Reporter: Olga Natkovich
> Assignee: Thejas M Nair
> Fix For: 0.9.0
>
>
> Currently, the code below will construct the full group in memory and then
> trim it. This requires in use of more memory than needed.
> A = load 'data' as (x, y, z);
> B = group A by x;
> C = foreach B{
> D = limit A 100;
> generate group, MyUDF(D);}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
