[ https://issues.apache.org/jira/browse/PIG-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-4043: ------------------------------- Attachment: PIG-4043-1.patch One thing to note is that the size of TaskReport object seems to have increased significantly in Hadoop 2 as compared to Hadoop 1. The same job can run with no problem in Hadoop 1 using the same size of heap, but it fails in Hadoop 2. The attached patch introduces a new property {{pig.stats.noTaskReport}} via which retrieving TaskReports can be disabled for large jobs. By default, it is set to false, so JobStats will still use TaskReports. I also documented this new property in {{pig.properties}}. > JobClient.getMap/ReduceTaskReports() causes OOM for jobs with a large number > of tasks > ------------------------------------------------------------------------------------- > > Key: PIG-4043 > URL: https://issues.apache.org/jira/browse/PIG-4043 > Project: Pig > Issue Type: Bug > Reporter: Cheolsoo Park > Assignee: Cheolsoo Park > Fix For: 0.14.0 > > Attachments: PIG-4043-1.patch, heapdump.png > > > With Hadoop 2.4, I often see Pig client fails due to OOM when there are many > tasks (~100K) with 1GB heap size. > The heap dump (attached) shows that TaskReport[] occupies about 80% of heap > space at the time of OOM. > The problem is that JobClient.getMap/ReduceTaskReports() returns an array of > TaskReport objects, which can be huge if the number of task is large. -- This message was sent by Atlassian JIRA (v6.2#6252)