[
https://issues.apache.org/jira/browse/FLINK-14124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933568#comment-16933568
]
zhijiang commented on FLINK-14124:
----------------------------------
I think it is better for you to upgrade the flink version. The 1.4.2 is so old
and there are many improvements in network stack after 1.5 version. Especially
on netty server, we avoid the data copy from flink buffer to netty ByteBuffer
which could save some direct memory used by netty server side.
So my suggestion is upgrading the flink latest version if possible to verify
whether this issue still exist.
> potential memory leak in netty server
> -------------------------------------
>
> Key: FLINK-14124
> URL: https://issues.apache.org/jira/browse/FLINK-14124
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Affects Versions: 1.6.3
> Reporter: YufeiLiu
> Priority: Critical
> Attachments: image-2019-09-19-15-53-32-294.png, screenshot-1.png,
> screenshot-2.png
>
>
> I have a job running in flink 1.4.2, end of the task is use Phoenix jdbc
> driver write record into Apache Phoenix.
> _mqStream
> .keyBy(0)
> .window(TumblingProcessingTimeWindows.of(Time.of(300,
> TimeUnit.SECONDS)))
> .process(new MyProcessWindowFunction())
> .addSink(new PhoenixSinkFunction());_
> But the TaskManager of sink subtask off-heap memory keep increasing,
> precisely is might case by DirectByteBuffer.
> I analyze heap dump, find there are hundreds of DirectByteBuffer object, each
> of them reference to over 3MB memory address, they are all link to Flink
> Netty Server Thread.
> !image-2019-09-19-15-53-32-294.png!
> It only happened in sink task, other nodes just work fine. I think is problem
> of Phoenix at first, but heap dump show memory is consume by netty. I didn't
> know much about flink network, I will be appreciated if someone can tell me
> the might causation or how to dig further.
> !screenshot-1.png!
> yarn.heap-cutoff-ratio: 0.2
> taskmanager.memory.fraction: 0.6
> taskmanager.network.numberOfBuffers: 32240
> !screenshot-2.png!
> I have Zookeeper, Kafka, Phoenix(Hbase), Flume dependency in package, they
> all might use direct memory, but when direct memory get free, is there
> something block the Cleaner progress.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
