[
https://issues.apache.org/jira/browse/FLINK-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369815#comment-14369815
]
Sebastian Kruse commented on FLINK-1754:
----------------------------------------
Here is a thread dump. If you need any further information to reenact the
problem, let me know.
{noformat}
2015-03-19 18:50:43
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.7-b01 mixed mode):
"Flink-IPC Client (1110866967) connection to localhost/127.0.0.1:6498 from an
unknown user" daemon prio=6 tid=0x000000000dc2e000 nid=0x1b00 in Object.wait()
[0x0000000012fdf000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.flink.runtime.ipc.Client$Connection.waitForWork(Client.java:404)
- locked <0x00000007dbf833d8> (a
org.apache.flink.runtime.ipc.Client$Connection)
at org.apache.flink.runtime.ipc.Client$Connection.run(Client.java:444)
"Temp writer" daemon prio=6 tid=0x000000000ee3d000 nid=0x1880 waiting on
condition [0x0000000013c8f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007d13d99a8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.flink.runtime.io.network.gates.InputGate.waitForAnyChannelToBecomeAvailable(InputGate.java:238)
at
org.apache.flink.runtime.io.network.gates.InputGate.readRecord(InputGate.java:173)
at
org.apache.flink.runtime.io.network.api.MutableRecordReader.next(MutableRecordReader.java:51)
at
org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:53)
at
org.apache.flink.runtime.operators.TempBarrier$TempWritingThread.run(TempBarrier.java:177)
"SortMerger spilling thread" daemon prio=6 tid=0x000000000ee3c000 nid=0x1510
waiting on condition [0x0000000013adf000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007d141f218> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1217)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:801)
"SortMerger sorting thread" daemon prio=6 tid=0x000000000ee3b800 nid=0x598
waiting on condition [0x00000000138bf000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007d141f130> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SortingThread.go(UnilateralSortMerger.java:1113)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:801)
"SortMerger Reading Thread" daemon prio=6 tid=0x000000000dff4000 nid=0x9b0
waiting on condition [0x000000001088f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007d13d9c38> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.flink.runtime.io.network.gates.InputGate.waitForAnyChannelToBecomeAvailable(InputGate.java:238)
at
org.apache.flink.runtime.io.network.gates.InputGate.readRecord(InputGate.java:173)
at
org.apache.flink.runtime.io.network.api.MutableRecordReader.next(MutableRecordReader.java:51)
at
org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:53)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ReadingThread.go(UnilateralSortMerger.java:1034)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:801)
"CoGroup (CoGroup at calculateTransitiveReduction1(Wendy.java:299)) (1/1)"
daemon prio=6 tid=0x000000000dffb000 nid=0x191c in Object.wait()
[0x00000000137be000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.flink.runtime.operators.TempBarrier.getIterator(TempBarrier.java:93)
- locked <0x00000007d1422650> (a java.lang.Object)
at
org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1127)
at
org.apache.flink.runtime.operators.CoGroupDriver.prepare(CoGroupDriver.java:93)
at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:463)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
at java.lang.Thread.run(Thread.java:722)
"Temp writer" daemon prio=6 tid=0x000000000dff8000 nid=0x1868 waiting on
condition [0x000000001354e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007d13d2718> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.flink.runtime.io.network.gates.InputGate.waitForAnyChannelToBecomeAvailable(InputGate.java:238)
at
org.apache.flink.runtime.io.network.gates.InputGate.readRecord(InputGate.java:173)
at
org.apache.flink.runtime.io.network.api.MutableRecordReader.next(MutableRecordReader.java:51)
at
org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:53)
at
org.apache.flink.runtime.operators.TempBarrier$TempWritingThread.run(TempBarrier.java:177)
"Join (Join at calculateTransitiveReduction1(Wendy.java:292)) (1/1)" daemon
prio=6 tid=0x000000000dc31800 nid=0x1368 in Object.wait() [0x000000001320f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.flink.runtime.operators.TempBarrier.getIterator(TempBarrier.java:93)
- locked <0x00000007d13da348> (a java.lang.Object)
at
org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1127)
at
org.apache.flink.runtime.operators.MatchDriver.prepare(MatchDriver.java:97)
at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:463)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
at java.lang.Thread.run(Thread.java:722)
"FlatMap (FlatMap at calculateTransitiveReduction1(Wendy.java:292)) (1/1)"
daemon prio=6 tid=0x000000000dc31000 nid=0x11bc waiting on condition
[0x000000001273e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007d13d17d8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.flink.runtime.io.network.gates.InputGate.waitForAnyChannelToBecomeAvailable(InputGate.java:238)
at
org.apache.flink.runtime.io.network.gates.InputGate.readRecord(InputGate.java:173)
at
org.apache.flink.runtime.io.network.api.MutableRecordReader.next(MutableRecordReader.java:51)
at
org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:53)
at
org.apache.flink.runtime.operators.FlatMapDriver.run(FlatMapDriver.java:83)
at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:493)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
at java.lang.Thread.run(Thread.java:722)
"RMI TCP Connection(idle)" daemon prio=6 tid=0x000000000dffb800 nid=0x1aa0
waiting on condition [0x0000000011c7e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000780badbd8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
"CHAIN Reduce(Reduce at intersectCandidates(Wendy.java:346)) -> Filter (Filter
at intersectCandidates(Wendy.java:349)) (1/1)" daemon prio=6
tid=0x000000000dc2f800 nid=0x1164 in Object.wait() [0x000000001331e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.flink.runtime.io.network.bufferprovider.LocalBufferPool.requestBuffer(LocalBufferPool.java:166)
- locked <0x00000007d13d8858> (a java.util.ArrayDeque)
at
org.apache.flink.runtime.io.network.bufferprovider.LocalBufferPool.requestBufferBlocking(LocalBufferPool.java:107)
at
org.apache.flink.runtime.io.network.gates.InputGate.requestBufferBlocking(InputGate.java:338)
at
org.apache.flink.runtime.io.network.channels.InputChannel.requestBufferBlocking(InputChannel.java:415)
at
org.apache.flink.runtime.io.network.ChannelManager.dispatchFromOutputChannel(ChannelManager.java:467)
at
org.apache.flink.runtime.io.network.channels.OutputChannel.sendBuffer(OutputChannel.java:81)
at
org.apache.flink.runtime.io.network.gates.OutputGate.sendBuffer(OutputGate.java:56)
at
org.apache.flink.runtime.io.network.api.BufferWriter.sendBuffer(BufferWriter.java:41)
at
org.apache.flink.runtime.io.network.api.RecordWriter.emit(RecordWriter.java:86)
at
org.apache.flink.runtime.operators.shipping.OutputCollector.collect(OutputCollector.java:88)
at
org.apache.flink.api.java.operators.translation.PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:48)
at
org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
at
org.apache.flink.runtime.operators.ReduceDriver.run(ReduceDriver.java:126)
at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:493)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
at java.lang.Thread.run(Thread.java:722)
"RMI Scheduler(0)" daemon prio=6 tid=0x000000000dc32800 nid=0x15f4 waiting on
condition [0x000000001236f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000780baeb38> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
"RMI TCP Connection(idle)" daemon prio=6 tid=0x000000000dc2e800 nid=0x1610
waiting on condition [0x0000000012cdf000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000780badbd8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
"Flink Executor Thread - 8" daemon prio=6 tid=0x000000000dc30000 nid=0x15ec
waiting on condition [0x000000001246e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007840cdf80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
"Flink Executor Thread - 7" daemon prio=6 tid=0x000000000dffd000 nid=0x1b1c
waiting on condition [0x0000000012ece000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007840cdf80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
"Flink Executor Thread - 6" daemon prio=6 tid=0x000000000dc2c000 nid=0x1a68
waiting on condition [0x0000000012acf000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007840cdf80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
"Flink Executor Thread - 5" daemon prio=6 tid=0x000000000dffc800 nid=0x91c
waiting on condition [0x00000000125cf000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007840cdf80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
"Flink Executor Thread - 4" daemon prio=6 tid=0x000000000dff9800 nid=0x55c
waiting on condition [0x000000001225e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007840cdf80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
"Flink Executor Thread - 3" daemon prio=6 tid=0x000000000dffe000 nid=0x1ab0
waiting on condition [0x000000001283e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007840cdf80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
"Flink Executor Thread - 2" daemon prio=6 tid=0x000000000dffa000 nid=0x1638
waiting on condition [0x00000000120ef000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007840cdf80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
"Flink Executor Thread - 1" daemon prio=6 tid=0x000000000dff8800 nid=0x1a20
waiting on condition [0x0000000011f4f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007840cdf80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
"Heartbeat Thread" prio=6 tid=0x000000000dff7000 nid=0x1aac waiting on
condition [0x0000000011b5f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.flink.runtime.taskmanager.TaskManager.registerAndRunHeartbeatLoop(TaskManager.java:844)
at
org.apache.flink.runtime.taskmanager.TaskManager.access$000(TaskManager.java:114)
at
org.apache.flink.runtime.taskmanager.TaskManager$1.run(TaskManager.java:373)
"IOManager reader thread #1" daemon prio=6 tid=0x000000000dff6800 nid=0xe64
waiting on condition [0x000000001196f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007c8418090> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$ReaderThread.run(IOManagerAsync.java:306)
"IOManager writer thread #1" daemon prio=6 tid=0x000000000dff5800 nid=0x176c
waiting on condition [0x000000001165e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007c8417e98> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$WriterThread.run(IOManagerAsync.java:412)
"Timer-3" daemon prio=6 tid=0x000000000dff5000 nid=0x1a88 in Object.wait()
[0x00000000117cf000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007c8473d28> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007c8473d28> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"Flink-IPC Server handler 0 on 6501" daemon prio=6 tid=0x000000000dff3800
nid=0x1478 waiting on condition [0x000000001141f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000078409bdf0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.flink.runtime.ipc.Server$Handler.run(Server.java:941)
"Flink-IPC Server listener on 6501" daemon prio=6 tid=0x000000000dff2800
nid=0x1874 runnable [0x00000000112bf000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
at
sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000007840b4650> (a sun.nio.ch.Util$2)
- locked <0x00000007840b4640> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000078409eb18> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.apache.flink.runtime.ipc.Server$Listener.run(Server.java:341)
"Flink-IPC Server Responder" daemon prio=6 tid=0x000000000dff2000 nid=0x15e0
runnable [0x0000000010e6e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
at
sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000007840b81a8> (a sun.nio.ch.Util$2)
- locked <0x00000007840b8198> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007840b7f40> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.flink.runtime.ipc.Server$Responder.run(Server.java:506)
"Timer-2" daemon prio=6 tid=0x000000000dff1000 nid=0x1ba0 in Object.wait()
[0x0000000010f6f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007840a7040> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"Flink-IPC Server handler 1 on 6498" daemon prio=6 tid=0x000000000dff0800
nid=0x1930 waiting on condition [0x0000000010a8f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007840bc4f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.flink.runtime.ipc.Server$Handler.run(Server.java:941)
"Flink-IPC Server handler 0 on 6498" daemon prio=6 tid=0x000000000dfef800
nid=0x1568 waiting on condition [0x0000000010cde000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007840bc4f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.flink.runtime.ipc.Server$Handler.run(Server.java:941)
"Flink-IPC Server listener on 6498" daemon prio=6 tid=0x000000000dfef000
nid=0xe6c runnable [0x0000000010b9e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
at
sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000007840bf7b8> (a sun.nio.ch.Util$2)
- locked <0x00000007840bf7a8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007840bf268> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.apache.flink.runtime.ipc.Server$Listener.run(Server.java:341)
"Flink-IPC Server Responder" daemon prio=6 tid=0x000000000dfee000 nid=0x1484
runnable [0x000000001098f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
at
sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000007840c32f0> (a sun.nio.ch.Util$2)
- locked <0x00000007840c32e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007840c3088> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.flink.runtime.ipc.Server$Responder.run(Server.java:506)
"Timer-1" daemon prio=6 tid=0x000000000e6c6800 nid=0x194 in Object.wait()
[0x000000001074f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000078409b908> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x000000078409b908> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"Thread-2" prio=6 tid=0x000000000e6c6000 nid=0xf38 runnable [0x000000000fbaf000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at
java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:121)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:183)
- locked <0x00000007840a6d00> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at org.apache.flink.runtime.blob.BlobServer.run(BlobServer.java:186)
"Timer-0" prio=6 tid=0x000000000e6c3000 nid=0x1af0 in Object.wait()
[0x00000000105cf000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007840ce3a0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"RMI TCP Accept-63695" daemon prio=6 tid=0x000000000de8d800 nid=0x440 runnable
[0x00000000103de000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at
java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:121)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:183)
- locked <0x0000000780baf6d8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:722)
"RMI TCP Accept-63692" daemon prio=6 tid=0x000000000de83000 nid=0x3c8 runnable
[0x00000000101ee000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at
java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:121)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:183)
- locked <0x0000000780bafb28> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:722)
"GC Daemon" daemon prio=2 tid=0x000000000d9c3800 nid=0x1b60 in Object.wait()
[0x000000001005f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000780a08a40> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:117)
- locked <0x0000000780a08a40> (a sun.misc.GC$LatencyLock)
"RMI Reaper" prio=6 tid=0x000000000de2f800 nid=0x730 in Object.wait()
[0x000000000febe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000780a16c90> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x0000000780a16c90> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:351)
at java.lang.Thread.run(Thread.java:722)
"RMI TCP Accept-0" daemon prio=6 tid=0x000000000de2f000 nid=0x12f0 runnable
[0x000000000fcce000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at
java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:121)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:183)
- locked <0x0000000780a0f708> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:722)
"Monitor Ctrl-Break" daemon prio=6 tid=0x000000000dbb4800 nid=0x1590 runnable
[0x000000000e34e000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
- locked <0x00000007dc1261b8> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:154)
at java.io.BufferedReader.readLine(BufferedReader.java:317)
- locked <0x00000007dc1261b8> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(BufferedReader.java:382)
at com.intellij.rt.execution.application.AppMain$1.run(AppMain.java:88)
at java.lang.Thread.run(Thread.java:722)
"Service Thread" daemon prio=6 tid=0x000000000b956000 nid=0x1780 runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x000000000b953800 nid=0x1bc8 waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x000000000b93f000 nid=0x930 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener" daemon prio=10 tid=0x000000000b93e000 nid=0x1018 runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x000000000b937000 nid=0x14d4 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=0x000000000235c000 nid=0x1340 in Object.wait()
[0x000000000cc9f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x0000000780a08cc0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
"Reference Handler" daemon prio=10 tid=0x0000000002354000 nid=0x1b7c in
Object.wait() [0x000000000cb8f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x0000000780a08a20> (a java.lang.ref.Reference$Lock)
"main" prio=6 tid=0x000000000229c000 nid=0x4b8 waiting on condition
[0x000000000260e000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.flink.runtime.client.JobClient.submitJobAndWait(JobClient.java:356)
at
org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:239)
- locked <0x0000000780baffc0> (a java.lang.Object)
at
org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:51)
at
de.hpi.isg.sodap.flink.jobs.AbstractFlinkProgram.executePlan(AbstractFlinkProgram.java:68)
at de.hpi.isg.sodap.sindy.jobs.Wendy.buildAndExecuteJob(Wendy.java:280)
at de.hpi.isg.sodap.sindy.jobs.Wendy.executeProgramLogic(Wendy.java:99)
at
de.hpi.isg.sodap.flink.jobs.AbstractProgram.run(AbstractProgram.java:115)
at de.hpi.isg.sodap.sindy.jobs.Wendy.main(Wendy.java:460)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
"VM Thread" prio=10 tid=0x000000000b8b2800 nid=0x116c runnable
"GC task thread#0 (ParallelGC)" prio=6 tid=0x00000000022aa000 nid=0x1ab8
runnable
"GC task thread#1 (ParallelGC)" prio=6 tid=0x00000000022ab800 nid=0x9fc
runnable
"GC task thread#2 (ParallelGC)" prio=6 tid=0x00000000022ad000 nid=0x14cc
runnable
"GC task thread#3 (ParallelGC)" prio=6 tid=0x00000000022af000 nid=0x1604
runnable
"VM Periodic Task Thread" prio=10 tid=0x000000000b991800 nid=0x1960 waiting on
condition
JNI global references: 251
Heap
PSYoungGen total 500224K, used 110065K [0x00000007d58b0000,
0x0000000800000000, 0x0000000800000000)
eden space 303616K, 36% used
[0x00000007d58b0000,0x00000007dc42c4d8,0x00000007e8130000)
from space 196608K, 0% used
[0x00000007f4000000,0x00000007f4000000,0x0000000800000000)
to space 195392K, 0% used
[0x00000007e8130000,0x00000007e8130000,0x00000007f4000000)
ParOldGen total 1391296K, used 1321400K [0x0000000780a00000,
0x00000007d58b0000, 0x00000007d58b0000)
object space 1391296K, 94% used
[0x0000000780a00000,0x00000007d146e3c8,0x00000007d58b0000)
PSPermGen total 21504K, used 15478K [0x000000077b800000,
0x000000077cd00000, 0x0000000780a00000)
object space 21504K, 71% used
[0x000000077b800000,0x000000077c71d920,0x000000077cd00000)
{noformat}
> Deadlock in job execution
> -------------------------
>
> Key: FLINK-1754
> URL: https://issues.apache.org/jira/browse/FLINK-1754
> Project: Flink
> Issue Type: Bug
> Reporter: Sebastian Kruse
>
> I have encountered a reproducible deadlock in the execution of one of my
> jobs. The part of the plan, where this happens, is the following:
> {code:java}
> /** Performs the reduction via creating transitive INDs and removing them
> from the original IND set. */
> private DataSet<Tuple2<Integer, int[]>>
> calculateTransitiveReduction1(DataSet<Tuple2<Integer, int[]>>
> inclusionDependencies) {
> // Concatenate INDs (only one hop).
> DataSet<Tuple2<Integer, int[]>> transitiveInds = inclusionDependencies
> .flatMap(new SplitInds())
> .joinWithTiny(inclusionDependencies)
> .where(1).equalTo(0)
> .with(new ConcatenateInds());
> // Remove the concatenated INDs to come up with a transitive
> reduction of the INDs.
> return inclusionDependencies
> .coGroup(transitiveInds)
> .where(0).equalTo(0)
> .with(new RemoveTransitiveInds());
> }
> {code}
> Seemingly, the flatmap operator waits infinitely for a free buffer to write
> on.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
