[
https://issues.apache.org/jira/browse/HDFS-13265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397335#comment-16397335
]
Erik Krogen commented on HDFS-13265:
------------------------------------
Attached a test, [^TestMiniDFSClusterThreads.java] , which tries to estimate
how many threads are used by NNs and DNs within a {{MiniDFSCluster}}. It is not
perfect but will provide a rough estimate.
Running this without any modifications results in the following:
{code}
MiniDFSCluster Thread Counts: Namenode 41 // Datanode 44 // Datanode #2 39
{code}
Most recently I was able to drop the thread counts down to:
{code}
MiniDFSCluster Thread Counts: Namenode 24 // Datanode 23 // Datanode #2 18
{code}
This was mostly done via (a) disabling unnecessary services on the
MiniDFSCluster (b) leveraging HDFS-13272 and HADOOP-15311 to decrease the
number of threads consumed by Jetty. For a 1-DN MiniDFSCluster (most common
AFAIK), this would be a reduction of about 40 threads per MiniDFSCluster, so
hopefully pretty beneficial to the testing pipeline.
> MiniDFSCluster should set reasonable defaults to reduce resource consumption
> ----------------------------------------------------------------------------
>
> Key: HDFS-13265
> URL: https://issues.apache.org/jira/browse/HDFS-13265
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, test
> Reporter: Erik Krogen
> Assignee: Erik Krogen
> Priority: Major
> Attachments: TestMiniDFSClusterThreads.java
>
>
> MiniDFSCluster takes its defaults from {{DFSConfigKeys}} defaults, but many
> of these are not suitable for a unit test environment. For example, the
> default handler thread count of 10 is definitely more than necessary for
> (almost?) any unit test. We should set reasonable, lower defaults unless a
> test specifically requires more.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
