[
https://issues.apache.org/jira/browse/HADOOP-15743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16610746#comment-16610746
]
Daryn Sharp commented on HADOOP-15743:
--------------------------------------
Out of the box, throughput under production load was ~1-2k ops/sec with such a
high failure rate that production incidents robbed over a week of my life. The
cited settings increase the throughput to an "acceptable" yet dismal ~5k
ops/sec. At least it's stable. Ie. No client failures and excessive latency
that caused client read timeouts. That's embarrassing compared to the NN which
is capable of _hundreds of thousands_ of open ops/sec.
We directly use the 2.8 KMS war (not embedded) with jetty 9. I don't have the
cycles, so I'd suggest someone translate the configuration into HttpServer2 and
continue to refine the tunings.
> Jetty and SSL tunings to stabilize KMS performance
> ---------------------------------------------------
>
> Key: HADOOP-15743
> URL: https://issues.apache.org/jira/browse/HADOOP-15743
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
> Affects Versions: 2.8.0
> Reporter: Daryn Sharp
> Priority: Major
>
> The KMS has very low throughput with high client failure rates. The
> following config options will "stabilize" the KMS under load:
> # Disable ECDH algos because java's SSL engine is inexplicably HORRIBLE.
> # Reduce SSL session cache size (unlimited) and ttl (24h). The memory cache
> has very poor performance and causes extreme GC collection pressure. Load
> balancing diminishes the effectiveness of the cache to 1/N-hosts anyway.
> ** -Djavax.net.ssl.sessionCacheSize=1000
> ** -Djavax.net.ssl.sessionCacheTimeout=6
> # Completely disable thread LowResourceMonitor to stop jetty from
> immediately closing incoming connections during connection bursts. Client
> retries cause jetty to remain in a low resource state until many clients fail
> and cause thousands of sockets to linger in various close related states.
> # Set min/max threads to 4x processors. Jetty recommends only 50 to 500
> threads. Java's SSL engine has excessive synchronization that limits
> performance anyway.
> # Set https idle timeout to 6s.
> # Significantly increase max fds to at least 128k. Recommend using a VIP
> load balancer with a lower limit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
