[
https://issues.apache.org/jira/browse/PIG-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated PIG-1815:
-------------------------------
Attachment: PIG-1815.1.patch
I have tested the patch using the query that was running out of memory. Patch
does not have any unit tests, as I can't think of a good/easy way to test the
leak in unit test.
All unit tests except TestScriptLanguage passed. TestScriptLanguage is failing
even without the patch.
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new
or modified tests.
[exec] Please justify why no tests are needed for
this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
> pig task retains used instances of PhysicalPlan
> -----------------------------------------------
>
> Key: PIG-1815
> URL: https://issues.apache.org/jira/browse/PIG-1815
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.8.0, 0.9.0
>
> Attachments: PIG-1815.1.patch, yourkit_combiner_hprof.jpg
>
>
> map tasks of a pig query ran out of memory because there were too many
> (thousands) instances of combiner PhysicalPlan in memory. Each physical plan
> (except the last?) was linked to older one as shown in the yourkit snapshot
> that I am attaching.
> This problem was noticed with 0.8 because of the split combination feature,
> that resulted in each map having larger inputs. The query also had large
> physical plan because of multi-query, it had 17 MR jobs merged into one
> during the multi-query optimization phase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
