[ https://issues.apache.org/jira/browse/FLINK-14124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
YufeiLiu updated FLINK-14124: ----------------------------- Description: I have a job running in flink 1.4.2, end of the task is use Phoenix jdbc driver write record into Apache Phoenix. _mqStream .keyBy(0) .window(TumblingProcessingTimeWindows.of(Time.of(300, TimeUnit.SECONDS))) .process(new MyProcessWindowFunction()) .addSink(new PhoenixSinkFunction());_ But the TaskManager of sink subtask off-heap memory keep increasing, precisely is might case by DirectByteBuffer. I analyze heap dump, find there are hundreds of DirectByteBuffer object, each of them reference to over 3MB memory address, they are all link to Flink Netty Server Thread. !image-2019-09-19-15-53-32-294.png! It only happened in sink task, other nodes just work fine. I think is problem of Phoenix at first, but heap dump show memory is consume by netty. I didn't know much about flink network, I will be appreciated if someone can tell me the might causation or how to dig further. was: I have a job running in flink 1.4.2, end of the task is use Phoenix jdbc driver write record into Apache Phoenix. _mqStream .keyBy(0) .window(TumblingProcessingTimeWindows.of(Time.of(300, TimeUnit.SECONDS))) .process(new MyProcessWindowFunction()) .addSink(new PhoenixSinkFunction());_ But the TaskManager of sink subtask off-heap memory keep increasing, precisely is might case by DirectByteBuffer. I analyze heap dump, find there are hundreds of DirectByteBuffer object, each of them reference to over 3MB memory address, they are all link to Flink Netty Server Thread. !image-2019-09-19-15-53-32-294.png! It only happened in sink task, other nodes just work fine. I think is problem of Phoenix at first, but heap dump show memory is consume by netty. I didn't know much about flink network, I will be appreciated if someone can tell me the might causation or how to dig durther. > potential memory leak in netty server > ------------------------------------- > > Key: FLINK-14124 > URL: https://issues.apache.org/jira/browse/FLINK-14124 > Project: Flink > Issue Type: Improvement > Components: Runtime / Network > Affects Versions: 1.6.3 > Reporter: YufeiLiu > Priority: Critical > Attachments: image-2019-09-19-15-53-32-294.png > > > I have a job running in flink 1.4.2, end of the task is use Phoenix jdbc > driver write record into Apache Phoenix. > _mqStream > .keyBy(0) > .window(TumblingProcessingTimeWindows.of(Time.of(300, > TimeUnit.SECONDS))) > .process(new MyProcessWindowFunction()) > .addSink(new PhoenixSinkFunction());_ > But the TaskManager of sink subtask off-heap memory keep increasing, > precisely is might case by DirectByteBuffer. > I analyze heap dump, find there are hundreds of DirectByteBuffer object, each > of them reference to over 3MB memory address, they are all link to Flink > Netty Server Thread. > !image-2019-09-19-15-53-32-294.png! > It only happened in sink task, other nodes just work fine. I think is problem > of Phoenix at first, but heap dump show memory is consume by netty. I didn't > know much about flink network, I will be appreciated if someone can tell me > the might causation or how to dig further. -- This message was sent by Atlassian Jira (v8.3.4#803005)