Robert Metzger created FLINK-16417:
--------------------------------------
Summary: ConnectedComponents iterations with high parallelism
end-to-end test fails with OutOfMemoryError: Direct buffer memory
Key: FLINK-16417
URL: https://issues.apache.org/jira/browse/FLINK-16417
Project: Flink
Issue Type: Bug
Components: API / DataSet, Tests
Reporter: Robert Metzger
Logs:
https://dev.azure.com/georgeryan1322/Flink/_build/results?buildId=74&view=logs&j=1f3ed471-1849-5d3c-a34c-19792af4ad16&t=ce095137-3e3b-5f73-4b79-c42d3d5f8283
{code}
2020-03-04T08:03:46.0786078Z 2020-03-04 08:03:42,628 INFO
org.apache.flink.runtime.iterative.task.IterationIntermediateTask [] - starting
iteration [1]: Reduce (MIN(1), at
main(HighParallelismIterationsTestProgram.java:61) (12/25)
2020-03-04T08:03:46.0787503Z 2020-03-04 08:03:42,875 ERROR
org.apache.flink.runtime.io.network.netty.PartitionRequestQueue [] -
Encountered error while consuming partitions
2020-03-04T08:03:46.0788060Z java.lang.OutOfMemoryError: Direct buffer memory
2020-03-04T08:03:46.0788460Z at java.nio.Bits.reserveMemory(Bits.java:175)
~[?:?]
2020-03-04T08:03:46.0788904Z at
java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[?:?]
2020-03-04T08:03:46.0789537Z at
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?]
2020-03-04T08:03:46.0790381Z at
org.apache.flink.shaded.netty4.io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:772)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0791491Z at
org.apache.flink.shaded.netty4.io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:748)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0792483Z at
org.apache.flink.shaded.netty4.io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:245)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0793416Z at
org.apache.flink.shaded.netty4.io.netty.buffer.PoolArena.allocate(PoolArena.java:215)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0794359Z at
org.apache.flink.shaded.netty4.io.netty.buffer.PoolArena.allocate(PoolArena.java:147)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0795385Z at
org.apache.flink.shaded.netty4.io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:342)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0796471Z at
org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0797575Z at
org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0798718Z at
org.apache.flink.shaded.netty4.io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0799951Z at
org.apache.flink.shaded.netty4.io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0801172Z at
org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0802572Z at
org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:779)
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0803719Z at
org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424)
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0804763Z at
org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0806007Z at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0807050Z at
org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0807612Z at java.lang.Thread.run(Thread.java:834) [?:?]
2020-03-04T08:03:46.0808499Z 2020-03-04 08:03:43,572 ERROR
org.apache.flink.runtime.operators.BatchTask [] - Error in task
code: Reduce (MIN(1), at main(HighParallelismIterationsTestProgram.java:61)
(5/25)
2020-03-04T08:03:46.0810179Z java.lang.Exception: The data preparation for task
'Reduce (MIN(1), at main(HighParallelismIterationsTestProgram.java:61)' ,
caused an error: Error obtaining the sorted input: Thread 'SortMerger Reading
Thread' terminated due to an exception: readAddress(..) failed: Connection
reset by peer (connection to '10.1.0.4/10.1.0.4:44453')
2020-03-04T08:03:46.0811472Z at
org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:480)
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0813477Z at
org.apache.flink.runtime.iterative.task.AbstractIterativeTask.run(AbstractIterativeTask.java:157)
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0814813Z at
org.apache.flink.runtime.iterative.task.IterationIntermediateTask.run(IterationIntermediateTask.java:107)
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0816257Z at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:369)
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0817111Z at
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:717)
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0817911Z at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:541)
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0818381Z at java.lang.Thread.run(Thread.java:834) [?:?]
2020-03-04T08:03:46.0819353Z Caused by: java.lang.RuntimeException: Error
obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due
to an exception: readAddress(..) failed: Connection reset by peer (connection
to '10.1.0.4/10.1.0.4:44453')
2020-03-04T08:03:46.0820498Z at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:650)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0821448Z at
org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1110)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0822376Z at
org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:99)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0823248Z at
org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:474)
[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0823661Z ... 6 more
2020-03-04T08:03:46.0824426Z Caused by: java.io.IOException: Thread 'SortMerger
Reading Thread' terminated due to an exception: readAddress(..) failed:
Connection reset by peer (connection to '10.1.0.4/10.1.0.4:44453')
2020-03-04T08:03:46.0825507Z at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:831)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0826579Z Caused by:
org.apache.flink.runtime.io.network.netty.exception.LocalTransportException:
readAddress(..) failed: Connection reset by peer (connection to
'10.1.0.4/10.1.0.4:44453')
2020-03-04T08:03:46.0827970Z at
org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.exceptionCaught(CreditBasedPartitionRequestClientHandler.java:165)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0829232Z at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0830423Z at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:276)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0831611Z at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:268)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0832773Z at
org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1388)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0834969Z at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0836413Z at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:276)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0838310Z at
org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:918)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0839629Z at
org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.handleReadException(AbstractEpollStreamChannel.java:730)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0841070Z at
org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:820)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0842211Z at
org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0843214Z at
org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0844284Z at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0845351Z at
org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
~[flink-dist_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
2020-03-04T08:03:46.0845828Z ... 1 more
2020-03-04T08:03:46.0846253Z Caused by:
org.apache.flink.shaded.netty4.io.netty.channel.unix.Errors$NativeIoException:
readAddress(..) failed: Connection reset by peer
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
