[
https://issues.apache.org/jira/browse/HADOOP-15743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16611430#comment-16611430
]
Wei-Chiu Chuang edited comment on HADOOP-15743 at 9/12/18 1:17 AM:
-------------------------------------------------------------------
Thanks [~daryn]. For the most part, your observation is consistent with what we
observed in Hadoop 3 (where KMS was ported using Jetty) although we didn't dive
into that deep and wide. IMHO, the biggest problem is KMS client doesn't reuse
a connection. How about designing a connection reuse mechanism that's similar
to Hadoop IPC? i.e. reuse an idle connection that belongs to the same user?
was (Author: jojochuang):
Thanks [~daryn]. For the most part, your observation is consistent to what we
observed in Hadoop 3 (where KMS was ported using Jetty) although we didn't dive
into that deep and wide. IMHO, the biggest problem is KMS client doesn't reuse
a connection. How about designing a connection reuse mechanism that's similar
to Hadoop IPC? i.e. reuse an idle connection that belongs to the same user?
> Jetty and SSL tunings to stabilize KMS performance
> ---------------------------------------------------
>
> Key: HADOOP-15743
> URL: https://issues.apache.org/jira/browse/HADOOP-15743
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
> Affects Versions: 2.8.0
> Reporter: Daryn Sharp
> Priority: Major
>
> The KMS has very low throughput with high client failure rates. The
> following config options will "stabilize" the KMS under load:
> # Disable ECDH algos because java's SSL engine is inexplicably HORRIBLE.
> # Reduce SSL session cache size (unlimited) and ttl (24h). The memory cache
> has very poor performance and causes extreme GC collection pressure. Load
> balancing diminishes the effectiveness of the cache to 1/N-hosts anyway.
> ** -Djavax.net.ssl.sessionCacheSize=1000
> ** -Djavax.net.ssl.sessionCacheTimeout=6
> # Completely disable thread LowResourceMonitor to stop jetty from
> immediately closing incoming connections during connection bursts. Client
> retries cause jetty to remain in a low resource state until many clients fail
> and cause thousands of sockets to linger in various close related states.
> # Set min/max threads to 4x processors. Jetty recommends only 50 to 500
> threads. Java's SSL engine has excessive synchronization that limits
> performance anyway.
> # Set https idle timeout to 6s.
> # Significantly increase max fds to at least 128k. Recommend using a VIP
> load balancer with a lower limit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
