[
https://issues.apache.org/jira/browse/HDDS-11240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17892731#comment-17892731
]
Ivan Andika edited comment on HDDS-11240 at 10/25/24 9:38 AM:
--------------------------------------------------------------
[~weiming] Thanks, from our side we tried to use ZGC and run the OM with JDK
21, but seems there are still a lot of entries in the ThreadLocalMap
The OZONE_OM_OPTS
export OZONE_OM_OPTS="-server -Xmn4G -Xmx64G -XX:+UnlockExperimentalVMOptions
-XX:+UseZGC -XX:+ZGenerational
-XX:ErrorFile=/var/log/ozone/$USER/hs_err_pid%p.log
-Xloggc:/var/log/ozone/$USER/gc.log`date +'%Y%m%d%H%M'` -verbose:gc
-XX:+PrintGCDetails"
We use the following command
{code:java}
jmap -histo:live <PID> | head -100 {code}
!image-2024-10-25-17-35-20-821.png|width=586,height=387!
>From the picture there are around 2.5 millions ThreadLocalMap#Entry instances,
>which I don't think is normal. Additionally, there seems to be some increase
>in the RPC latency, but we're still investigating the validity.
was (Author: JIRAUSER298977):
[~weiming] Thanks, from our side we tried to use ZGC and run the OM with JDK
21, but seems there are still a lot of entries in the ThreadLocalMap
The OZONE_OM_OPTS
export OZONE_OM_OPTS="-server -Xmn4G -Xmx64G -XX:+UnlockExperimentalVMOptions
-XX:+UseZGC -XX:+ZGenerational
-XX:ErrorFile=/var/log/ozone/$USER/hs_err_pid%p.log
-Xloggc:/var/log/ozone/$USER/gc.log`date +'%Y%m%d%H%M'` -verbose:gc
-XX:+PrintGCDetails"
We use the following command
{code:java}
jmap -histo:live <PID> | head -100 {code}
!image-2024-10-25-17-35-20-821.png|width=586,height=387!
>From the picture there are around 5 millions ThreadLocalMap#Entry, which I
>don't think is normal. Additionally, there seems to be some increase in the
>RPC latency, but we're still investigating the validity.
> High cpu usage on ReadWrite locks in JDK17
> ------------------------------------------
>
> Key: HDDS-11240
> URL: https://issues.apache.org/jira/browse/HDDS-11240
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Affects Versions: 1.4.0
> Environment: JDK：
> openjdk 17.0.2 2022-01-18
> OpenJDK Runtime Environment (build 17.0.2+8-86)
> OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)
> Ozone:
> 1.4.0
>
> Reporter: weiming
> Assignee: Tanvi Penumudy
> Priority: Major
> Attachments: flamegraph.profile.html,
> image-2024-07-28-20-17-58-466.png, image-2024-07-30-09-32-16-320.png,
> image-2024-10-25-17-35-20-821.png
>
>
> That will cause threads on the following stack trace to consume a lot of CPU:
> "IPC Server handler 7 on default port 9862" #3994 daemon prio=5 os_prio=0
> cpu=5403833.36ms elapsed=653145.54s tid=0x00007fa03fdd2a00 nid=0x921f9
> runnable [0x00007fa0ca3fd000]
> java.lang.Thread.State: RUNNABLE
> at
> java.lang.ThreadLocal$ThreadLocalMap.expungeStaleEntry([email protected]/ThreadLocal.java:632)
> at
> java.lang.ThreadLocal$ThreadLocalMap.remove([email protected]/ThreadLocal.java:516)
> at java.lang.ThreadLocal.remove([email protected]/ThreadLocal.java:242)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared([email protected]/ReentrantReadWriteLock.java:430)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared([email protected]/AbstractQueuedSynchronizer.java:1094)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock([email protected]/ReentrantReadWriteLock.java:897)
> at
> org.apache.hadoop.ozone.upgrade.AbstractLayoutVersionManager.needsFinalization(AbstractLayoutVersionManager.java:182)
> at
> org.apache.hadoop.ozone.om.request.validation.ValidationCondition$1.shouldApply(ValidationCondition.java:39)
> at
> org.apache.hadoop.ozone.om.request.validation.RequestValidations.lambda$0(RequestValidations.java:110)
> at
> org.apache.hadoop.ozone.om.request.validation.RequestValidations$$Lambda$839/0x00000008013cda80.test(Unknown
> Source)
>
> [^flamegraph.profile.html]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
