[jira] Updated: (PIG-1446) OOME in a query having a bincond in the inner plan of a Foreach.
[ https://issues.apache.org/jira/browse/PIG-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated PIG-1446: -- Status: Resolved (was: Patch Available) Fix Version/s: 0.8.0 0.7.0 Resolution: Fixed As usual, hudson is not responding. I manually ran all the unit tests, all of them passed. Committed to both trunk and 0.7 OOME in a query having a bincond in the inner plan of a Foreach. Key: PIG-1446 URL: https://issues.apache.org/jira/browse/PIG-1446 Project: Pig Issue Type: Bug Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.0, 0.7.0 Attachments: pig-1446.patch This is seen when For Each is following a group-by and there is a bin cond as an inner plan of For Each. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1446) OOME in a query having a bincond in the inner plan of a Foreach.
[ https://issues.apache.org/jira/browse/PIG-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated PIG-1446: -- Attachment: pig-1446.patch Sequence of event is as follows: 1) MultiQuery optimizer combined 30 group-bys in one reducer. So, there are 30 pipelines in a reducer. 2) Each of these group-by has a ForEach after them. 3) ForEach has a bincond in its own plan. 4) Group-by resulted in large bags (10s of million of records). 5) Tuple containing group and bag is attached to the roots of inner plan of FE. 6) FE pulled the tuples through its leaves. 7) Due to short-circuiting in bin-cond, one branch of the plan is never pulled resulting in stray reference of bag which actually was not needed. 8) Due to MQ optimized 30 group-bys, we had many such bags now hanging in there, eating up all the memory. Fix: Detach tuples from the roots once you are done in FE. OOME in a query having a bincond in the inner plan of a Foreach. Key: PIG-1446 URL: https://issues.apache.org/jira/browse/PIG-1446 Project: Pig Issue Type: Bug Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Attachments: pig-1446.patch This is seen when For Each is following a group-by and there is a bin cond as an inner plan of For Each. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1446) OOME in a query having a bincond in the inner plan of a Foreach.
[ https://issues.apache.org/jira/browse/PIG-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated PIG-1446: -- Status: Patch Available (was: Open) OOME in a query having a bincond in the inner plan of a Foreach. Key: PIG-1446 URL: https://issues.apache.org/jira/browse/PIG-1446 Project: Pig Issue Type: Bug Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: pig-1446.patch This is seen when For Each is following a group-by and there is a bin cond as an inner plan of For Each. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
