Erik Krogen commented on HDFS-13265:

Attached a test,  [^TestMiniDFSClusterThreads.java] , which tries to estimate 
how many threads are used by NNs and DNs within a {{MiniDFSCluster}}. It is not 
perfect but will provide a rough estimate.
Running this without any modifications results in the following:
MiniDFSCluster Thread Counts: Namenode 41 // Datanode 44 // Datanode #2 39

Most recently I was able to drop the thread counts down to:
MiniDFSCluster Thread Counts: Namenode 24 // Datanode 23 // Datanode #2 18
This was mostly done via (a) disabling unnecessary services on the 
MiniDFSCluster (b) leveraging HDFS-13272 and HADOOP-15311 to decrease the 
number of threads consumed by Jetty. For a 1-DN MiniDFSCluster (most common 
AFAIK), this would be a reduction of about 40 threads per MiniDFSCluster, so 
hopefully pretty beneficial to the testing pipeline.

> MiniDFSCluster should set reasonable defaults to reduce resource consumption
> ----------------------------------------------------------------------------
>                 Key: HDFS-13265
>                 URL: https://issues.apache.org/jira/browse/HDFS-13265
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode, namenode, test
>            Reporter: Erik Krogen
>            Assignee: Erik Krogen
>            Priority: Major
>         Attachments: TestMiniDFSClusterThreads.java
> MiniDFSCluster takes its defaults from {{DFSConfigKeys}} defaults, but many 
> of these are not suitable for a unit test environment. For example, the 
> default handler thread count of 10 is definitely more than necessary for 
> (almost?) any unit test. We should set reasonable, lower defaults unless a 
> test specifically requires more.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to