[
https://issues.apache.org/jira/browse/FLINK-35192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842312#comment-17842312
]
chenyuzhi edited comment on FLINK-35192 at 4/30/24 9:19 AM:
------------------------------------------------------------
The op image we use is to modify the basic image of the DockerFile file from
eclipse-temurin:11-jre-jammy to maven:3.8.4-eclipse-temurin-11 (convenient to
use jdk tools). There is no modification at the code level [~bgeng777]
Recently, we using pmap cmd observed the memory growth of operator and made a
comparison and checking the process rss monitoring of the op host. The
comparison time was 04.24 20:00-04.25 01:20.
The process rss occupation increased by 95 MB(34.725G-34.630G):
!https://office.netease.com/api/admin/file/download?path=cowork/2024/04/25/7b7c4c18e6ad44baa9cebec9c3ee4dae.png|width=1395,height=802!
!https://office.netease.com/api/admin/file/download?path=cowork/2024/04/25/918f54f737d8430bab99f8a8536d287b.png|width=1302,height=464!
Observe the pmap results at these two time points and compare:
!https://office.netease.com/api/admin/file/download?path=cowork/2024/04/25/38d9386430814e8282683930ab72daea.png|width=1448,height=314!
It can be found that the growth mainly comes from the front memory area, and
there are many 65536KB memory blocks. At the same time, the front memory area
increased from 74 MB (35444 KB). + 39668 KB) grows to
165 MB (65536 KB + 65536 KB + 35708 KB), the increaing result 91MB（165MB -
74MB）basically matching the memory growth of the process.
Referring to an issue in hadoop, it is suspected that glibc has a memory leak
due to memory fragmentation in multiple arenas under multi-threading. At
present, we refer to the
[DockerFile|https://github.com/apache/flink-docker/blob/master/1.18/scala_2.12-java11-ubuntu/Dockerfile#L24]
of flink-docker, which using jemalloc as memory assigner.
We replacing the glibc used by the operator with jemalloc4.5, and adjust the
Xmx of the operator from 32g to 5.25g.
Here is the memory usage of the process after adjustment:
!image-2024-04-30-17-11-24-974.png|width=1243,height=489!
Therefore, it is suspected that it is caused by a memory leak in glibc in a
multi-threaded environment. Does the community consider changing the memory
allocation of the op from glibc to jemalloc? If so, I will be happy to provide
a PR to optimize this issue.
[~gyfora]
was (Author: stupid_pig):
The op image we use is to modify the basic image of the DockerFile file from
eclipse-temurin:11-jre-jammy to maven:3.8.4-eclipse-temurin-11 (convenient to
use jdk tools). There is no modification at the code level [~bgeng777]
Recently, we using pmap cmd observed the memory growth of operator and made a
comparison and checking the process rss monitoring of the op host. The
comparison time was 04.24 20:00-04.25 01:20.
The process rss occupation increased by 95 MB(34.725G-34.630G):
!https://office.netease.com/api/admin/file/download?path=cowork/2024/04/25/7b7c4c18e6ad44baa9cebec9c3ee4dae.png|width=1607,height=924!
!https://office.netease.com/api/admin/file/download?path=cowork/2024/04/25/918f54f737d8430bab99f8a8536d287b.png|width=1743,height=621!
Observe the pmap results at these two time points and compare:
!https://office.netease.com/api/admin/file/download?path=cowork/2024/04/25/38d9386430814e8282683930ab72daea.png!
It can be found that the growth mainly comes from the front memory area, and
there are many 65536KB memory blocks. At the same time, the front memory area
increased from 74 MB (35444 KB). + 39668 KB) grows to
165 MB (65536 KB + 65536 KB + 35708 KB), the increaing result 91MB（165MB -
74MB）basically matching the memory growth of the process.
Referring to an issue in hadoop, it is suspected that glibc has a memory leak
due to memory fragmentation in multiple arenas under multi-threading. At
present, we refer to the
[DockerFile|https://github.com/apache/flink-docker/blob/master/1.18/scala_2.12-java11-ubuntu/Dockerfile#L24]
of flink-docker, which using jemalloc as memory assigner.
We replacing the glibc used by the operator with jemalloc4.5, and adjust the
Xmx of the operator from 32g to 5.25g.
Here is the memory usage of the process after adjustment:
!image-2024-04-30-17-11-24-974.png|width=1583,height=623!
Therefore, it is suspected that it is caused by a memory leak in glibc in a
multi-threaded environment. Does the community consider changing the memory
allocation of the op from glibc to jemalloc? If so, I will be happy to provide
a PR to optimize this issue.
[~gyfora]
> operator oom
> ------------
>
> Key: FLINK-35192
> URL: https://issues.apache.org/jira/browse/FLINK-35192
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Affects Versions: kubernetes-operator-1.6.1
> Environment: jdk: openjdk11
> operator version: 1.6.1
> Reporter: chenyuzhi
> Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-04-22-15-47-49-455.png,
> image-2024-04-22-15-52-51-600.png, image-2024-04-22-15-58-23-269.png,
> image-2024-04-22-15-58-42-850.png, image-2024-04-30-16-47-07-289.png,
> image-2024-04-30-17-11-24-974.png, screenshot-1.png, screenshot-2.png,
> screenshot-3.png
>
>
> The kubernetest operator docker process was killed by kernel cause out of
> memory(the time is 2024.04.03: 18:16)
> !image-2024-04-22-15-47-49-455.png!
> Metrics:
> the pod memory (RSS) is increasing slowly in the past 7 days:
> !screenshot-1.png!
> However the jvm memory metrics of operator not shown obvious anomaly：
> !image-2024-04-22-15-58-23-269.png!
> !image-2024-04-22-15-58-42-850.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
