[
https://issues.apache.org/jira/browse/PIG-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai reopened PIG-4314:
-----------------------------
The patch does not completely solve the issue. See job hanging again. Here is
the stack:
{code}
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.pig.impl.util.SpillableMemoryManager.registerSpillable(SpillableMemoryManager.java:327)
- waiting to lock <0x00000000d9d02648> (a java.util.LinkedList)
at
org.apache.pig.data.DefaultAbstractBag.markSpillableIfNecessary(DefaultAbstractBag.java:129)
at
org.apache.pig.data.DefaultAbstractBag.add(DefaultAbstractBag.java:118)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPartialAgg.createValueTuple(POPartialAgg.java:482)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPartialAgg.aggregate(POPartialAgg.java:419)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPartialAgg.aggregateFirstLevel(POPartialAgg.java:453)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPartialAgg.aggregateBothLevels(POPartialAgg.java:436)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPartialAgg.getNextTuple(POPartialAgg.java:207)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNextTuple(POLocalRearrange.java:291)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:283)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
{code}
> BigData_5 hang on some machine
> ------------------------------
>
> Key: PIG-4314
> URL: https://issues.apache.org/jira/browse/PIG-4314
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.14.0
>
> Attachments: PIG-4314-1.patch, PIG-4314-2.patch
>
>
> Hang for both tez and mr. Here is the log:
> {code}
> 2014-11-07 19:49:45,799 INFO [TezChild] relationalOperators.POPartialAgg:
> Aggregated 10000 raw tuples. Processed tuples before aggregation = 0, after
> aggregation = 676
> 2014-11-07 19:49:45,799 INFO [TezChild] relationalOperators.POPartialAgg:
> After reduction, processed map: 676; raw map: 0
> 2014-11-07 19:49:45,800 INFO [TezChild] relationalOperators.POPartialAgg:
> Observed reduction factor: from 10000 to 676 => 14.
> 2014-11-07 19:49:45,992 INFO [TezChild] relationalOperators.POPartialAgg:
> Getting mem limits; considering 1 POPArtialAgg objects. with memory
> percentage 0.2
> 2014-11-07 19:49:46,290 INFO [TezChild] relationalOperators.POPartialAgg:
> Estimated total tuples to buffer, based on 10000 tuples that took up 2364864
> bytes: 1052574
> 2014-11-07 19:49:46,290 INFO [TezChild] relationalOperators.POPartialAgg:
> Setting thresholds. Primary: 977390. Secondary: 75184
> 2014-11-07 19:49:50,641 INFO [Service Thread] util.SpillableMemoryManager:
> first memory handler call - Collection threshold init = 216530944(211456K)
> used = 724849000(707860K) committed = 927989760(906240K) max =
> 1146093568(1119232K)
> 2014-11-07 19:49:50,644 INFO [Service Thread]
> relationalOperators.POPartialAgg: Spill triggered by SpillableMemoryManager
> 2014-11-07 19:49:50,993 INFO [TezChild] relationalOperators.POPartialAgg:
> Aggregated 346545 raw tuples. Processed tuples before aggregation = 676,
> after aggregation = 1352
> 2014-11-07 19:49:50,998 INFO [TezChild] relationalOperators.POPartialAgg:
> Aggregated 1352 processed tuples to 676 tuples
> 2014-11-07 19:49:50,998 INFO [TezChild] relationalOperators.POPartialAgg:
> Avoided emitting records during spill memory call.
> 2014-11-07 19:49:51,046 INFO [Service Thread]
> relationalOperators.POPartialAgg: Finished spill for SpillableMemoryManager
> call
> 2014-11-07 19:49:51,851 INFO [Service Thread] util.SpillableMemoryManager:
> Spilled an estimate of 81943920 bytes from 1 objects. init =
> 216530944(211456K) used = 724849000(707860K) committed = 927989760(906240K)
> max = 1146093568(1119232K)
> 2014-11-07 19:49:59,644 INFO [Service Thread]
> relationalOperators.POPartialAgg: Spill triggered by SpillableMemoryManager
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
