The pie chart is generated by MemoryAnalyzer(http://www.eclipse.org/mat/) from 
the heap dump when OOME happened.

I've increased all the parallelisms and set default_parallel to 3. It does not 
work.

Still I don't know what the first MR job compiled by Pig is doing . Only 1 
reducer all the time ... 

Thank you. 

Haitao Yao
[email protected]
weibo: @haitao_yao
Skype:  haitao.yao.final

On 2012-9-17, at 下午5:07, Dmitriy Ryaboy wrote:

> Neat pie chart! What produces this?
> 
> Trunk is not entirely stable right now, but it's stabilizing pretty rapidly 
> (as long as you don't go using DateTime types and Cube operations.. don't 
> think they've been put through the paces quite yet, and will be considered 
> experimental when 0.11 does roll out).
> 
> It would be interesting to know if we fixed the memory issue you are 
> encountering, though.
> 
> 0.10 will, I suspect, actually perform somewhat worse in your case than 0.9, 
> due to the massive amount of distincting you appear to be doing.
> 
> Since we now see that the memory is spent on actual data, not general 
> overhead, I think your best bet is to increase parallelism (via the 
> "parallel" keyword) where possible, and where not possible -- say, if you are 
> doing a "group all" -- perform distinct counting via 2 MR jobs, as I outlined 
> earlier.
> 
> Pig 0.11 will also have an experimental option to codegen custom tuples for 
> known schemas, which may drop the memory footprint significantly, but that's 
> a different conversation.
> 
> Hope this helps,
> Dmitriy
> 
> On Mon, Sep 17, 2012 at 1:53 AM, Haitao Yao <[email protected]> wrote:
> Thank you very much for the reply. I've checked the latest heapdump again,and 
> you're right: the OOME is not caused by split, but DefaultDataBag. Sorry for 
> the misleading, I've got too many heap dumps and I ' confused.
> 
> Here's  the latest screen shot of the heap dump: 
> <aa.jpg>
> There's a lot of DefaultDataBag. 
> 
> I've tried your suggestion: set pig.exec.mapPartAgg true; set 
> pig.exec.mapPartAgg.minReduction 3; But the reducer count is still 1. 
> The pig version I'm using is 0.9.2.
> 
> BTW, is "the latest trunk" stable enough for production deployment? If not, 
> does 0.10.0 provide this optimization rule? When will  0.11 release?
> 
> Thank you again. 
> 
> Haitao Yao
> [email protected]
> weibo: @haitao_yao
> Skype:  haitao.yao.final
> 
> On 2012-9-17, at 下午1:01, Dmitriy Ryaboy wrote:
> 
>> Ok, then it's not POSplit that's holding the memory -- it does not
>> participate in any of the reduce stages, according the the plan you
>> attached.
>> 
>> To set parallelism, you can hardcode it on every operation that causes
>> an MR boundary, with the exception of "group all"  and "limit" since
>> those by definition require a single reducer. So, you can alter your
>> script to explicitly request parallelism to be greater than what is
>> estimated: "join .. parallel $P", "group by .. parallel $P", "order
>> ... parallel $P", etc.
>> 
>> I would recommend two things:
>> 1) Make sure you are running the latest trunk, and have enabled
>> in-memory aggregation ( set pig.exec.mapPartAgg true; set
>> pig.exec.mapPartAgg.minReduction 3 ). I just made some significant
>> improvements to Distinct's Initial phase (not requiring it to register
>> with SpillableMemoryManager at all), and also improved in-mem
>> aggregation performance.
>> 
>> 2) It seems like you are doing a lot of "group, distinct the group,
>> count" type operations. If you do have a distinct group that is very
>> large, loading it all into ram is bound to cause problems. When the
>> size of distinct sets is expected to be fairly high, we usually
>> recommend a different pattern for count(distinct x):
>> 
>> Instead of :
>> results = foreach (group data by country) {
>>  distinct_ids = distinct data.id;
>>  generate group as country, COUNT(distinct_ids) as num_dist,
>> COUNT(data) as total;
>> }
>> 
>> Do the following:
>> 
>> results_per_id = foreach (group data by (country, id))
>>  generate flatten(group) as (country, id), COUNT(data) as num_repeats;
>> results = foreach (group results_per_id by country)
>>  generate group as country, COUNT(results_per_id) as num_dist,
>> SUM(results_per_id.num_repeats) as total;
>> 
>> This will introduce an extra MR step, but it's much more scalable when
>> you get into millions of distincts in a single dimension.
>> 
>> D
>> 
>> On Sun, Sep 16, 2012 at 2:18 AM, Haitao Yao <[email protected]> wrote:
>>> The map output of the first MR job is over 500MB, and only 1 reducer 
>>> processes it. So OutOfMemoryError is caused.
>>> 
>>> After set the child memory to 1GB, the first job succeeded. But most of our 
>>> jobs does not need that much memory. 512MB is enough if I can set the 
>>> reducer to more than 1.
>>> 
>>> 
>>> 
>>> 
>>> Haitao Yao
>>> [email protected]
>>> weibo: @haitao_yao
>>> Skype:  haitao.yao.final
>>> 
>>> On 2012-9-16, at 下午5:05, Haitao Yao wrote:
>>> 
>>>> here's the explain result compressed.(The apache mail server does not 
>>>> allow big attachments.)
>>>> <explain.tar.gz>
>>>> 
>>>> 
>>>> Haitao Yao
>>>> [email protected]
>>>> weibo: @haitao_yao
>>>> Skype:  haitao.yao.final
>>>> 
>>>> On 2012-9-16, at 下午4:41, Dmitriy Ryaboy wrote:
>>>> 
>>>>> Still would like to see the script or the explain plan..
>>>>> 
>>>>> D
>>>>> 
>>>>> On Sat, Sep 15, 2012 at 7:50 PM, Haitao Yao <[email protected]> wrote:
>>>>>> No, I also thought it is a mapper , but It surely is a reducer. all the 
>>>>>> mappers succeeded and the reducer failed.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Haitao Yao
>>>>>> [email protected]
>>>>>> weibo: @haitao_yao
>>>>>> Skype:  haitao.yao.final
>>>>>> 
>>>>>> On 2012-9-16, at 上午10:08, Haitao Yao wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>>     I 'v encountered a problem: the job failed because of POSplit 
>>>>>>> retained too much memory in the reducer. How can I specify more 
>>>>>>> reducers for the spill?
>>>>>>> 
>>>>>>>     Here's the screen snapshot of the Heap dump.
>>>>>>>     <aa.jpg>
>>>>>>> 
>>>>>>> 
>>>>>>> And here's the snippet of my split script:
>>>>>>> 
>>>>>>>     split RawData into AURawData if type == 2, NURawData if type == 1, 
>>>>>>> InRawData if type == 9, GCData if type == 61, HCData if type == 71, 
>>>>>>> TutorialRawData if type == 3 or t    ype == 15;
>>>>>>> 
>>>>>>> There's 3 similar split clause in my script. The reducer count is 
>>>>>>> always 1. How can I increase it?
>>>>>>> 
>>>>>>> Thanks.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Haitao Yao
>>>>>>> [email protected]
>>>>>>> weibo: @haitao_yao
>>>>>>> Skype:  haitao.yao.final
>>>>>>> 
>>>>>> 
>>>> 
>>> 
> 
> 

Reply via email to