Tao Yang created HADOOP-16850:
---------------------------------
Summary: Support getting thread info from thread group for
JvmMetrics to improve the performance
Key: HADOOP-16850
URL: https://issues.apache.org/jira/browse/HADOOP-16850
Project: Hadoop Common
Issue Type: Improvement
Components: metrics
Affects Versions: 2.8.6, 2.9.3, 3.1.4, 3.2.2, 2.10.1, 3.3.1
Reporter: Tao Yang
Recently we found jmx request taken almost 5s+ to be done when there were 1w+
threads in a stressed datanode process, meanwhile other http requests were
blocked and some disk operations were affected (we can see many "Slow
manageWriterOsCache" messages in DN log, and these messages were hard to be
seen again after we stopped sending jxm requests)
The excessive time is spent in getting thread info via ThreadMXBean inside
which ThreadImpl#getThreadInfo native method is called, the time complexity of
ThreadImpl#getThreadInfo is O(n*n) according to JDK-8185005 and it may held
global thread lock (prevent creation or termination of threads) for a long time.
To improve this, I propose to support getting thread info from thread group
which will improve a lot by default, also support using original approach when
"-Dhadoop.metrics.jvm.use-thread-mxbean=true" is configured in the startup
command.
An example of performance tests between these two approaches is as follows:
{noformat}
#Threads=100, ThreadMXBean=382372 ns, ThreadGroup=72046 ns, ratio: 5
#Threads=200, ThreadMXBean=776619 ns, ThreadGroup=83875 ns, ratio: 9
#Threads=500, ThreadMXBean=3392954 ns, ThreadGroup=216269 ns, ratio: 15
#Threads=1000, ThreadMXBean=9475768 ns, ThreadGroup=220447 ns, ratio: 42
#Threads=2000, ThreadMXBean=53833729 ns, ThreadGroup=579608 ns, ratio: 92
#Threads=3000, ThreadMXBean=196829971 ns, ThreadGroup=1157670 ns, ratio: 170
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
