大家好,请教一下,flink任务读取kafka数据进行聚集操作后将结果写回kafka,flink版本为1.8.0。任务运行一段时间后出现如下异常,之后flink任务异常挂掉,请问一下这个问题该如何解决呢?多谢
2020-02-19 10:45:45,314 ERROR
org.apache.flink.runtime.io.network.netty.PartitionRequestQueue - Encountered
error while consuming partitions
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at
org.apache.flink.shaded.netty4.io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
at
org.apache.flink.shaded.netty4.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
at
org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347)
at
org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
at
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591)
at
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508)
at
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
at java.lang.Thread.run(Thread.java:748)
2020-02-19 10:45:45,317 INFO org.apache.kafka.clients.producer.KafkaProducer
- [Producer clientId=producer-1] Closing the Kafka producer with
timeoutMillis = 9223372036854775807 ms.
2020-02-19 10:45:45,412 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Memory usage
stats: [HEAP: 98/6912/6912 MB, NON HEAP: 81/83/-1 MB (used/committed/max)]
2020-02-19 10:45:45,413 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Direct memory
stats: Count: 24596, Total Capacity: 806956211, Used Memory: 806956212
2020-02-19 10:50:31,351 WARN akka.remote.transport.netty.NettyTransport
- Remote connection to [null] failed with
java.net.ConnectException: Connection refused: aj-flinknode01/9.186.36.80:56983
2020-02-19 10:50:31,351 WARN akka.remote.ReliableDeliverySupervisor
- Association with remote system
[akka.tcp://flink@aj-flinknode01:56983] has failed, address is now gated for
[50] ms. Reason: [Association failed with
[akka.tcp://flink@aj-flinknode01:56983]] Caused by: [Connection refused:
aj-flinknode01/9.186.36.80:56983]
2020-02-19 10:50:55,419 WARN akka.remote.ReliableDeliverySupervisor
- Association with remote system
[akka.tcp://flink@aj-flinknode01:45703] has failed, address is now gated for
[50] ms. Reason: [Disassociated]
2020-02-19 10:50:56,370 INFO org.apache.flink.yarn.YarnResourceManager
- Closing TaskExecutor connection
container_1578492316659_0830_01_000006 because: Container
[pid=30031,containerID=container_1578492316659_0830_01_000006] is running
beyond physical memory limits. Current usage: 10.0 GB of 10 GB physical memory
used; 11.8 GB of 21 GB virtual memory used. Killing container.
Dump of the process-tree for container_1578492316659_0830_01_000006 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 30068 30031 30031 30031 (java) 277668 18972 12626370560 2630988
/data/jdk1.8.0_211/bin/java -Xms6912m -Xmx6912m -XX:MaxDirectMemorySize=3328m
-XX:+UseG1GC
-Dlog.file=/data/hadoop-2.6.0-cdh5.16.1/logs/userlogs/application_1578492316659_0830/container_1578492316659_0830_01_000006/taskmanager.log
-Dlogback.configurationFile=file:./logback.xml
-Dlog4j.configuration=file:./log4j.properties
org.apache.flink.yarn.YarnTaskExecutorRunner --configDir .
|- 30031 30027 30031 30031 (bash) 0 0 11001856 329 /bin/bash -c
/data/jdk1.8.0_211/bin/java -Xms6912m -Xmx6912m -XX:MaxDirectMemorySize=3328m
-XX:+UseG1GC
-Dlog.file=/data/hadoop-2.6.0-cdh5.16.1/logs/userlogs/application_1578492316659_0830/container_1578492316659_0830_01_000006/taskmanager.log
-Dlogback.configurationFile=file:./logback.xml
-Dlog4j.configuration=file:./log4j.properties
org.apache.flink.yarn.YarnTaskExecutorRunner --configDir . 1>
/data/hadoop-2.6.0-cdh5.16.1/logs/userlogs/application_1578492316659_0830/container_1578492316659_0830_01_000006/taskmanager.out
2>
/data/hadoop-2.6.0-cdh5.16.1/logs/userlogs/application_1578492316659_0830/container_1578492316659_0830_01_000006/taskmanager.err