[
https://issues.apache.org/jira/browse/HADOOP-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
YUBI LEE updated HADOOP-17861:
------------------------------
Description:
There are some points to improve the performance of YARN Registry DNS Server.
- Do not print unnecessary logs (It just needs change log4j.properties)
- log4j.logger.org.apache.hadoop.registry.server.dns=WARN
- Change some loglevels at points which can affect performance degradation.
- Use "newFixedThreadPool" instead of "newCachedThreadPool" to prevent OOM
- Use Blocking on TCP handler. Using non-blocking and sleeping some
time("Thread.sleep(500)") is meaningless.
In our environment, QPS of original yarn dns server is about 5000 (UDP), 100
(TCP).
Now, QPS of our improved yarn dns server is about 47000 (UDP), 500 (TCP).
I will make a pull request at https://github.com/apache/hadoop soon.
was:
There are some points to improve the performance of YARN Registry DNS Server.
- Do not print unnecessary logs (It just needs change log4j.properties)
- log4j.logger.org.apache.hadoop.registry.server.dns=WARN
- Change some loglevels at points which can affect performance degradation.
- Use "newFixedThreadPool" instead of "newCachedThreadPool" to prevent OOM
- Use Blocking on TCP handler. Using non-blocking and sleeping some
time("Thread.sleep(500)") is meaningless.
In our environment, QPS of original yarn dns server is about 5000 (UDP), 100
(TCP).
Now, QPS of our improved yarn dns server is about 47000 (UDP), 500 (TCP).
I will make a pull request at https://github.com/apache/hadoop soon.
> improve YARN Registry DNS Server qps
> ------------------------------------
>
> Key: HADOOP-17861
> URL: https://issues.apache.org/jira/browse/HADOOP-17861
> Project: Hadoop Common
> Issue Type: Improvement
> Components: registry
> Affects Versions: 3.1.2
> Reporter: YUBI LEE
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There are some points to improve the performance of YARN Registry DNS Server.
> - Do not print unnecessary logs (It just needs change log4j.properties)
> - log4j.logger.org.apache.hadoop.registry.server.dns=WARN
> - Change some loglevels at points which can affect performance degradation.
> - Use "newFixedThreadPool" instead of "newCachedThreadPool" to prevent OOM
> - Use Blocking on TCP handler. Using non-blocking and sleeping some
> time("Thread.sleep(500)") is meaningless.
> In our environment, QPS of original yarn dns server is about 5000 (UDP), 100
> (TCP).
> Now, QPS of our improved yarn dns server is about 47000 (UDP), 500 (TCP).
> I will make a pull request at https://github.com/apache/hadoop soon.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
