Limit can not push in front of ForEach with flatten
---------------------------------------------------
Key: PIG-362
URL: https://issues.apache.org/jira/browse/PIG-362
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: types_branch
Reporter: Daniel Dai
Assignee: Daniel Dai
Fix For: types_branch
Currently logical optimizer will push Limit in front of ForEach with flatten.
It is based on the assumption that ForEach with a flatten always increase the
number of records. However, this is a false assumption. In the case that there
is empty bags inside input tuple, the number of output records can be 0, thus
less than input records.
We have no way to know whether there is an empty bag in the input at
optimization time. So the only solution is not to push Limit in front of
ForEach with flatten
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
