flink1.12.4 写入hdfs报错 java.lang.OutOfMemoryError: Direct buffer memory

Mon, 08 Nov 2021 06:20:32 -0800 

通过flink 1.12.4 streaming file sink 写入hdfs，运行过程中抛出以下异常：


2021-11-08 20:39:05
java.io.IOException: java.lang.OutOfMemoryError: Direct buffer memory
    at 
org.apache.hadoop.hdfs.DataStreamer$LastExceptionInStreamer.set(DataStreamer.java:299)
    at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:820)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:694)
    at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
    at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241)
    at sun.nio.ch.IOUtil.write(IOUtil.java:58)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
    at 
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
    at 
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
    at 
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
    at 
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
    at java.io.DataOutputStream.write(DataOutputStream.java:107)
    at org.apache.hadoop.hdfs.DFSPacket.writeTo(DFSPacket.java:180)
    at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:765)

回复