Well , the OOM issue about Pig Bag has been fixed by me. I've push it into production and no OOM about Bag any more. I've contributed a patch here: https://issues.apache.org/jira/browse/PIG-2812 . Hope this can draw someone's attention and merge it into trunk.
Haitao Yao [email protected] weibo: @haitao_yao Skype: haitao.yao.final 在 2012-8-22，上午6:05， Jonathan Coveney 写道： > Make sure that any Pig process has at least 1gb. I've seen bag-related > OOM's when people don't give enough heap to Pig. > > 2012/8/21 Haitao Yao <[email protected]> > >> Thank you very much for this. >> But I still can not find which snippet of the script caused this OOM. The >> heap dump is generated in the midnight, nobody is standing by. >> >> I 've added some scripts to capture more information about the job. If >> succeed, I 'll share with you tomorrow. >> Thanks again. >> >> >> >> Haitao Yao >> [email protected] >> weibo: @haitao_yao >> Skype: haitao.yao.final >> >> 在 2012-8-21，下午5:22， Subir S 写道： >> >>> I think it helps to get some context, if you can spin a small snippet >> from your script which causes this issue. This may give a better >> understanding of what you are trying to achieve, and if there is some work >> around. However this is upto you! >>> >>> Thanks, Subir >>> >>> On Tue, Aug 21, 2012 at 2:12 PM, Haitao Yao <[email protected]> wrote: >>> I 'v found the reason: org.apache.hadoop.io.DataInputBuffer , there's a >> big byte array referenced by DataInputBuffer. >>> >>> But there's no any way to close the buffer. >>> >>> Is there any other solution? >>> >>> BTW, my script is more than 500 lines, sorry i can not show you. >>> >>> here's the screenshot: >>> >>> <aa.jpg> >>> >>> >>> >>> >>> <bb.jpg> >>> >>> >>> >>> Haitao Yao >>> [email protected] >>> weibo: @haitao_yao >>> Skype: haitao.yao.final >>> >>> 在 2012-8-20，上午9:49， Haitao Yao 写道： >>> >>>> Hi, all, >>>> I got an OOME , on org.apache.hadoop.mapreduce.Reducer$Context, >> here's the snapshot of the heap dump: >>>> <aa.jpg> >>>> >>>> Well, does pig have to report so many data through the Reducer$Context? >>>> Can this be closed? >>>> >>>> thanks. >>>> >>>> >>>> >>>> Haitao Yao >>>> [email protected] >>>> weibo: @haitao_yao >>>> Skype: haitao.yao.final >>>> >>> >>> >> >>
