[
https://issues.apache.org/jira/browse/PIG-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15317267#comment-15317267
]
Rohini Palaniswamy commented on PIG-4922:
-----------------------------------------
{code}
Found one Java-level deadlock:
=============================
"TezChild":
waiting to lock monitor 0x00007fbe097278b8 (object 0x00000000c034d2a0, a
java.util.LinkedList),
which is held by "Service Thread"
"Service Thread":
waiting to lock monitor 0x00007fbdec6468d8 (object 0x00000000f029ef28, a
java.util.ArrayList),
which is held by "TezChild"
Java stack information for the threads listed above:
===================================================
"TezChild":
at
org.apache.pig.impl.util.SpillableMemoryManager.registerSpillable(SpillableMemoryManager.java:378)
- waiting to lock <0x00000000c034d2a0> (a java.util.LinkedList)
at
org.apache.pig.data.DefaultAbstractBag.markSpillableIfNecessary(DefaultAbstractBag.java:129)
at
org.apache.pig.data.DefaultAbstractBag.add(DefaultAbstractBag.java:118)
at org.apache.pig.data.BinInterSedes.readBag(BinInterSedes.java:242)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:354)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:318)
at
org.apache.pig.data.BinInterSedes.addColsToTuple(BinInterSedes.java:664)
at org.apache.pig.data.BinSedesTuple.readFields(BinSedesTuple.java:48)
at
org.apache.pig.data.InternalSortedBag$SortedDataBagIterator.addToQueue(InternalSortedBag.java:314)
at
org.apache.pig.data.InternalSortedBag$SortedDataBagIterator.preMerge(InternalSortedBag.java:388)
at
org.apache.pig.data.InternalSortedBag$SortedDataBagIterator.<init>(InternalSortedBag.java:201)
- locked <0x00000000f029ef28> (a java.util.ArrayList)
at
org.apache.pig.data.InternalSortedBag.iterator(InternalSortedBag.java:149)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSort.getNextTuple(POSort.java:294)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:305)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLimit.getNextTuple(POLimit.java:121)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNextDataBag(PhysicalOperator.java:411)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.processInputBag(POProject.java:566)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.PORelationToExprProject.getNextDataBag(PORelationToExprProject.java:106)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.processInputBag(POProject.java:566)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.getNextDataBag(POProject.java:231)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:335)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:389)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:309)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:305)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNextTuple(POLocalRearrange.java:287)
at
org.apache.pig.backend.hadoop.executionengine.tez.plan.operator.POLocalRearrangeTez.getNextTuple(POLocalRearrangeTez.java:127)
at
org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor.runPipeline(PigProcessor.java:369)
at
org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor.run(PigProcessor.java:234)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:362)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1679)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"Service Thread":
at
org.apache.pig.data.DefaultAbstractBag.getMemorySize(DefaultAbstractBag.java:164)
- waiting to lock <0x00000000f029ef28> (a java.util.ArrayList)
at
org.apache.pig.impl.util.SpillableMemoryManager.handleNotification(SpillableMemoryManager.java:244)
- locked <0x00000000c034d2a0> (a java.util.LinkedList)
- locked <0x00000000c034d2c0> (a java.lang.Object)
at
sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:156)
at sun.management.MemoryImpl.createNotification(MemoryImpl.java:168)
at
sun.management.MemoryPoolImpl$CollectionSensor.triggerAction(MemoryPoolImpl.java:329)
at sun.management.Sensor.trigger(Sensor.java:137)
Found 1 deadlock.
{code}
> Deadlock between SpillableMemoryManager and
> InternalSortedBag$SortedDataBagIterator
> -----------------------------------------------------------------------------------
>
> Key: PIG-4922
> URL: https://issues.apache.org/jira/browse/PIG-4922
> Project: Pig
> Issue Type: Bug
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.17.0, 0.16.1
>
>
> This one ran into a deadlock, when the data was really huge and
> InternalSortedBag was reading spilled data from disk.
> {code}
> grpd = FOREACH (GROUP data BY $0){
> sorted = ORDER data BY timestamp DESC;
> latest = LIMIT sorted 1;
> GENERATE latest;
> };
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
