[
https://issues.apache.org/jira/browse/FLINK-19964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17226066#comment-17226066
]
Robert Metzger commented on FLINK-19964:
----------------------------------------
>From the logs, you can see that the job seems to make progress without any
>issues, until we reach this point, where we don't see any new log events:
{code}
19:58:21,930 [CHAIN CoGroup (Hub) -> Combine (Sum) (3/4)#0] INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] -
finishing iteration [1]: CHAIN CoGroup (Hub) -> Combine (Sum) (3/4)
19:58:21,930 [flink-akka.actor.default-dispatcher-2] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Reduce (Sum)
(3/4) (852e861d142fcff331c85ee4b12949b9_9d7788ce41d2b7d2b242bfb2589af1cb_2_0)
switched from RUNNING to FINISHED.
19:58:21,931 [Reduce (Sum) (2/4)#0] INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] -
finishing iteration [1]: Reduce (Sum) (2/4)
19:58:21,931 [Reduce (Sum) (1/4)#0] INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] -
finishing iteration [1]: Reduce (Sum) (1/4)
19:58:21,931 [CHAIN Map (Square) -> Combine (Sum) (2/4)#0] INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] -
finishing iteration [1]: CHAIN Map (Square) -> Combine (Sum) (2/4)
19:58:21,931 [Reduce (Sum) (4/4)#0] INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] -
finishing iteration [1]: Reduce (Sum) (4/4)
19:58:21,931 [Reduce (Sum) (3/4)#0] INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] -
finishing iteration [1]: Reduce (Sum) (3/4)
19:58:21,931 [CHAIN Map (Square) -> Combine (Sum) (1/4)#0] INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] -
finishing iteration [1]: CHAIN Map (Square) -> Combine (Sum) (1/4)
19:58:21,931 [CHAIN Map (Square) -> Combine (Sum) (4/4)#0] INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] -
finishing iteration [1]: CHAIN Map (Square) -> Combine (Sum) (4/4)
19:58:21,931 [CHAIN Map (Square) -> Combine (Sum) (3/4)#0] INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] -
finishing iteration [1]: CHAIN Map (Square) -> Combine (Sum) (3/4)
19:58:21,931 [Reduce (Sum) (1/1)#0] INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] -
finishing iteration [1]: Reduce (Sum) (1/1)
19:58:21,931 [CHAIN CoGroup (Authority) -> Combine (Sum) (3/4)#0] INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] - starting
iteration [1]: CHAIN CoGroup (Authority) -> Combine (Sum) (3/4)
19:58:21,932 [CHAIN CoGroup (Authority) -> Combine (Sum) (1/4)#0] INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] - starting
iteration [1]: CHAIN CoGroup (Authority) -> Combine (Sum) (1/4)
19:58:21,932 [CHAIN CoGroup (Authority) -> Combine (Sum) (2/4)#0] INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] - starting
iteration [1]: CHAIN CoGroup (Authority) -> Combine (Sum) (2/4)
19:58:21,932 [CHAIN CoGroup (Authority) -> Combine (Sum) (4/4)#0] INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] - starting
iteration [1]: CHAIN CoGroup (Authority) -> Combine (Sum) (4/4)
19:58:21,932 [CHAIN CoGroup (Authority) -> Combine (Sum) (3/4)#0] INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] -
finishing iteration [1]: CHAIN CoGroup (Authority) -> Combine (Sum) (3/4)
19:58:21,933 [CHAIN CoGroup (Authority) -> Combine (Sum) (1/4)#0] INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] -
finishing iteration [1]: CHAIN CoGroup (Authority) -> Combine (Sum) (1/4)
{code}
At this point, it seems that two iterations are finished:
- CHAIN CoGroup (Authority) -> Combine (Sum) (3/4)
- CHAIN CoGroup (Authority) -> Combine (Sum) (1/4)
and two are not finished:
- CHAIN CoGroup (Authority) -> Combine (Sum) (2/4)
- CHAIN CoGroup (Authority) -> Combine (Sum) (4/4)
Searching for "CHAIN CoGroup (Authority) -> Combine (Sum) (4/4)" in the
threaddumps reveals:
{code}
2020-11-03T20:15:40.9815769Z "CHAIN CoGroup (Authority) -> Combine (Sum)
(4/4)#0" #10722 prio=5 os_prio=0 tid=0x00007f8bf00db000 nid=0x6ce8 waiting on
condition [0x00007f8a257d5000]
2020-11-03T20:15:40.9816219Z java.lang.Thread.State: WAITING (parking)
2020-11-03T20:15:40.9816467Z at sun.misc.Unsafe.park(Native Method)
2020-11-03T20:15:40.9817005Z - parking to wait for <0x000000008728caf8> (a
java.util.concurrent.CompletableFuture$Signaller)
2020-11-03T20:15:40.9817404Z at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
2020-11-03T20:15:40.9818064Z at
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
2020-11-03T20:15:40.9818484Z at
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
2020-11-03T20:15:40.9819044Z at
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
2020-11-03T20:15:40.9819455Z at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
2020-11-03T20:15:40.9819938Z at
org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestMemorySegmentBlocking(LocalBufferPool.java:313)
2020-11-03T20:15:40.9820498Z at
org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBuilderBlocking(LocalBufferPool.java:286)
2020-11-03T20:15:40.9821095Z at
org.apache.flink.runtime.io.network.partition.BufferWritingResultPartition.requestNewBufferBuilderFromPool(BufferWritingResultPartition.java:315)
2020-11-03T20:15:40.9821765Z at
org.apache.flink.runtime.io.network.partition.BufferWritingResultPartition.requestNewUnicastBufferBuilder(BufferWritingResultPartition.java:292)
2020-11-03T20:15:40.9822490Z at
org.apache.flink.runtime.io.network.partition.BufferWritingResultPartition.appendUnicastDataForNewRecord(BufferWritingResultPartition.java:232)
2020-11-03T20:15:40.9823432Z at
org.apache.flink.runtime.io.network.partition.BufferWritingResultPartition.emitRecord(BufferWritingResultPartition.java:131)
2020-11-03T20:15:40.9823980Z at
org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:100)
2020-11-03T20:15:40.9824612Z at
org.apache.flink.runtime.io.network.api.writer.ChannelSelectorRecordWriter.emit(ChannelSelectorRecordWriter.java:53)
2020-11-03T20:15:40.9825143Z at
org.apache.flink.runtime.operators.shipping.OutputCollector.collect(OutputCollector.java:65)
2020-11-03T20:15:40.9825627Z at
org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
2020-11-03T20:15:40.9826167Z at
org.apache.flink.runtime.operators.hash.InPlaceMutableHashTable$ReduceFacade.emit(InPlaceMutableHashTable.java:1060)
2020-11-03T20:15:40.9826721Z at
org.apache.flink.runtime.operators.chaining.ChainedReduceCombineDriver.close(ChainedReduceCombineDriver.java:269)
2020-11-03T20:15:40.9827201Z at
org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:517)
2020-11-03T20:15:40.9827637Z at
org.apache.flink.runtime.iterative.task.AbstractIterativeTask.run(AbstractIterativeTask.java:159)
2020-11-03T20:15:40.9828152Z at
org.apache.flink.runtime.iterative.task.IterationIntermediateTask.run(IterationIntermediateTask.java:107)
2020-11-03T20:15:40.9828606Z at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:370)
2020-11-03T20:15:40.9828998Z at
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
2020-11-03T20:15:40.9829371Z at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
2020-11-03T20:15:40.9829680Z at java.lang.Thread.run(Thread.java:748)
{code}
Both threads ran into this. It seems that the {{LocalBufferPool}} ran out of
buffers?
[~AHeise] can you take a look?
> Gelly ITCase stuck on Azure
> ---------------------------
>
> Key: FLINK-19964
> URL: https://issues.apache.org/jira/browse/FLINK-19964
> Project: Flink
> Issue Type: Bug
> Components: Library / Graph Processing (Gelly), Tests
> Affects Versions: 1.12.0
> Reporter: Chesnay Schepler
> Assignee: Robert Metzger
> Priority: Blocker
> Labels: test-stability
> Fix For: 1.12.0
>
>
> The HITSITCase has gotten stuck on Azure. Chances are that something in the
> scheduling or network has broken it.
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8919&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
