[
https://issues.apache.org/jira/browse/PIG-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated PIG-4314:
----------------------------
Attachment: PIG-4314-4.patch
Some refine on memory usage.
> BigData_5 hang on some machine
> ------------------------------
>
> Key: PIG-4314
> URL: https://issues.apache.org/jira/browse/PIG-4314
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.14.0
>
> Attachments: PIG-4314-1.patch, PIG-4314-2.patch, PIG-4314-3.patch,
> PIG-4314-4.patch
>
>
> Hang for both tez and mr. Here is the log:
> {code}
> 2014-11-07 19:49:45,799 INFO [TezChild] relationalOperators.POPartialAgg:
> Aggregated 10000 raw tuples. Processed tuples before aggregation = 0, after
> aggregation = 676
> 2014-11-07 19:49:45,799 INFO [TezChild] relationalOperators.POPartialAgg:
> After reduction, processed map: 676; raw map: 0
> 2014-11-07 19:49:45,800 INFO [TezChild] relationalOperators.POPartialAgg:
> Observed reduction factor: from 10000 to 676 => 14.
> 2014-11-07 19:49:45,992 INFO [TezChild] relationalOperators.POPartialAgg:
> Getting mem limits; considering 1 POPArtialAgg objects. with memory
> percentage 0.2
> 2014-11-07 19:49:46,290 INFO [TezChild] relationalOperators.POPartialAgg:
> Estimated total tuples to buffer, based on 10000 tuples that took up 2364864
> bytes: 1052574
> 2014-11-07 19:49:46,290 INFO [TezChild] relationalOperators.POPartialAgg:
> Setting thresholds. Primary: 977390. Secondary: 75184
> 2014-11-07 19:49:50,641 INFO [Service Thread] util.SpillableMemoryManager:
> first memory handler call - Collection threshold init = 216530944(211456K)
> used = 724849000(707860K) committed = 927989760(906240K) max =
> 1146093568(1119232K)
> 2014-11-07 19:49:50,644 INFO [Service Thread]
> relationalOperators.POPartialAgg: Spill triggered by SpillableMemoryManager
> 2014-11-07 19:49:50,993 INFO [TezChild] relationalOperators.POPartialAgg:
> Aggregated 346545 raw tuples. Processed tuples before aggregation = 676,
> after aggregation = 1352
> 2014-11-07 19:49:50,998 INFO [TezChild] relationalOperators.POPartialAgg:
> Aggregated 1352 processed tuples to 676 tuples
> 2014-11-07 19:49:50,998 INFO [TezChild] relationalOperators.POPartialAgg:
> Avoided emitting records during spill memory call.
> 2014-11-07 19:49:51,046 INFO [Service Thread]
> relationalOperators.POPartialAgg: Finished spill for SpillableMemoryManager
> call
> 2014-11-07 19:49:51,851 INFO [Service Thread] util.SpillableMemoryManager:
> Spilled an estimate of 81943920 bytes from 1 objects. init =
> 216530944(211456K) used = 724849000(707860K) committed = 927989760(906240K)
> max = 1146093568(1119232K)
> 2014-11-07 19:49:59,644 INFO [Service Thread]
> relationalOperators.POPartialAgg: Spill triggered by SpillableMemoryManager
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
