zhanghaou opened a new issue, #16564:
URL: https://github.com/apache/pulsar/issues/16564
**Environment**
1. broker run in kubernetes, set memory request to 20GB and limit to 20GB
2. pulsar version 2.8.1
3. os:ubuntu:20.04
4. jdk: openjdk version "11.0.11" 2021-04-20
5. PULSAR_GC: -XX:+UseG1GC -XX:MaxGCPauseMillis=10
-Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024
-XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions
-XX:+DoEscapeAnalysis -XX:ParallelGCThreads=4 -XX:ConcGCThreads=4
-XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB
-XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem
6. PULSAR_MEM: -Xms8G -Xmx8G -XX:MaxDirectMemorySize=8G
**Describe the bug**
When running broker in our cluster, some broker pod may occur memory usage
over 85%, and then cause to unloading namespace bundles from this broker, but
from monitor of pod, we found the usage of memory is not highï¼›on the other
side, we found that when unloading bundles, jvm heap memory is very high, and
the monitoring curve of `jvm_memory_bytes_used` is very similar to
`pulsar_lb_memory_usage`
**Question**
Does LinuxBrokerHostUsageImpl.getMemUsage() return error results? Why get
different result in same pod?
**To Reproduce**
1. test in the broker pod
```
import com.sun.management.OperatingSystemMXBean;
import java.lang.management.ManagementFactory;
public class MemTest {
public static void main(String[] args) {
OperatingSystemMXBean systemBean = (OperatingSystemMXBean)
ManagementFactory.getOperatingSystemMXBean();
double total = ((double) systemBean.getTotalPhysicalMemorySize()) /
(1024 * 1024);
double free = ((double) systemBean.getFreePhysicalMemorySize()) /
(1024 * 1024);
System.out.println("total:" + total);
System.out.println("free:" + free);
}
}
```
and result of limit is 20GB.
2. use arthas to see the real return of
LinuxBrokerHostUsageImpl.getMemUsage, and can see that the limit is 8GB.
**Expected behavior**
LinuxBrokerHostUsageImpl.getMemUsage return the real memory of pod rather
than the heap memory of jvm.
**Screenshots**
1.monitor of broker pod
2.monitor of pulsar_lb_memory_usage
Thanks for any suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
