I am not using the Kafka SSL port. On Thu, Jul 15, 2021 at 9:48 PM Alexey Trenikhun <yen...@msn.com> wrote:
> Just in case, make sure that you are not using Kafka SSL port without > setting security protocol, see [1] > > [1] https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-4090 > ------------------------------ > *From:* bat man <tintin0...@gmail.com> > *Sent:* Wednesday, July 14, 2021 10:55:54 AM > *To:* Timo Walther <twal...@apache.org> > *Cc:* user <user@flink.apache.org> > *Subject:* Re: High DirectByteBuffer Usage > > Hi Timo, > > I am looking at these options. > However, I had a couple of questions - > 1. The off-heap usage grows overtime. My job does not do any off-heap > operations so I don't think there is a leak there. Even after GC it keeps > adding a few MBs after hours of running. > 2. Secondly, I am seeing as the incoming record volume increases the > off-heap usage grows. What's the reason for this? > > I am using 1.9. Is there any known bug which is causing this issue? > > Thanks, > Hemant > > On Wed, Jul 14, 2021 at 7:30 PM Timo Walther <twal...@apache.org> wrote: > > Hi Hemant, > > did you checkout the dedicated page for memory configuration and > troubleshooting: > > > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/memory/mem_trouble/#outofmemoryerror-direct-buffer-memory > > > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/memory/mem_trouble/#container-memory-exceeded > > It is likely that the high number of output streams could cause your > issues. > > Regards, > Timo > > > > > On 14.07.21 08:46, bat man wrote: > > Hi, > > I have a job which reads different streams from 5 kafka topics. It > > filters data and then data is streamed to different operators for > > processing. This step involves data shuffling. > > > > Also, once data is enriched in 4 joins(KeyedCoProcessFunction) > > operators. After joining the data is written to different kafka topics. > > There are a total of 16 different output streams which are written to 4 > > topics. > > > > I have been facing some issues with yarn killing containers. I took the > > heap dump and ran it through JXray [1]. Heap usage is not high. One > > thing which stands out is off-heap usage which is very high. My guess is > > this is what is killing the containers as the data inflow increases. > > > > Screenshot 2021-07-14 at 11.52.41 AM.png > > > > > > From the stack above is this usage high because of many output streams > > being written to kafka topics. As the stack shows RecordWriter holding > > off this DirectByteBuffer. I have assigned Network Memory as 1GB, and > > --MaxDirectMemorySize also shows ~1GB for task managers. > > > > From here[2] I found that setting -Djdk.nio.maxCachedBufferSize=262144 > > limits the temp buffer cache. Will it help in this case? > > jvm version used is - JVM: OpenJDK 64-Bit Server VM - Red Hat, Inc. - > > 1.8/25.282-b08 > > > > [1] - https://jxray.com <https://jxray.com> > > [2] - > > > https://dzone.com/articles/troubleshooting-problems-with-native-off-heap-memo > > < > https://dzone.com/articles/troubleshooting-problems-with-native-off-heap-memo > > > > > > Thanks, > > Hemant > >
