[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-12765: --- Fix Version/s: (was: 2.9.0) > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: HADOOP-12765-branch-2.patch, HADOOP-12765.001.patch, > HADOOP-12765.001.patch, HADOOP-12765.002.patch, HADOOP-12765.003.patch, > HADOOP-12765.004.patch, HADOOP-12765.005.patch, blocking_1.png, > blocking_2.png, unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-12765: --- Resolution: Fixed Status: Resolved (was: Patch Available) > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Fix For: 2.8.0, 2.9.0, 2.7.4, 3.0.0-alpha2 > > Attachments: HADOOP-12765-branch-2.patch, HADOOP-12765.001.patch, > HADOOP-12765.001.patch, HADOOP-12765.002.patch, HADOOP-12765.003.patch, > HADOOP-12765.004.patch, HADOOP-12765.005.patch, blocking_1.png, > blocking_2.png, unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-12765: --- Fix Version/s: 2.7.4 > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Fix For: 2.8.0, 2.9.0, 2.7.4, 3.0.0-alpha2 > > Attachments: HADOOP-12765-branch-2.patch, HADOOP-12765.001.patch, > HADOOP-12765.001.patch, HADOOP-12765.002.patch, HADOOP-12765.003.patch, > HADOOP-12765.004.patch, HADOOP-12765.005.patch, blocking_1.png, > blocking_2.png, unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-12765: --- Fix Version/s: 2.9.0 > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Fix For: 2.8.0, 2.9.0, 3.0.0-alpha2 > > Attachments: HADOOP-12765-branch-2.patch, HADOOP-12765.001.patch, > HADOOP-12765.001.patch, HADOOP-12765.002.patch, HADOOP-12765.003.patch, > HADOOP-12765.004.patch, HADOOP-12765.005.patch, blocking_1.png, > blocking_2.png, unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-12765: --- Fix Version/s: 2.8.0 > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-12765-branch-2.patch, HADOOP-12765.001.patch, > HADOOP-12765.001.patch, HADOOP-12765.002.patch, HADOOP-12765.003.patch, > HADOOP-12765.004.patch, HADOOP-12765.005.patch, blocking_1.png, > blocking_2.png, unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-12765: --- Target Version/s: 2.7.4 (was: 2.9.0) > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-12765-branch-2.patch, HADOOP-12765.001.patch, > HADOOP-12765.001.patch, HADOOP-12765.002.patch, HADOOP-12765.003.patch, > HADOOP-12765.004.patch, HADOOP-12765.005.patch, blocking_1.png, > blocking_2.png, unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-12765: --- Fix Version/s: 3.0.0-alpha2 > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-12765-branch-2.patch, HADOOP-12765.001.patch, > HADOOP-12765.001.patch, HADOOP-12765.002.patch, HADOOP-12765.003.patch, > HADOOP-12765.004.patch, HADOOP-12765.005.patch, blocking_1.png, > blocking_2.png, unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-12765: - Target Version/s: 2.9.0 > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Attachments: HADOOP-12765-branch-2.patch, HADOOP-12765.001.patch, > HADOOP-12765.001.patch, HADOOP-12765.002.patch, HADOOP-12765.003.patch, > HADOOP-12765.004.patch, HADOOP-12765.005.patch, blocking_1.png, > blocking_2.png, unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-12765: - Attachment: HADOOP-12765-branch-2.patch There's a conflict due to HADOOP-10588. Attached a branch-2 patch. > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Attachments: HADOOP-12765-branch-2.patch, HADOOP-12765.001.patch, > HADOOP-12765.001.patch, HADOOP-12765.002.patch, HADOOP-12765.003.patch, > HADOOP-12765.004.patch, HADOOP-12765.005.patch, blocking_1.png, > blocking_2.png, unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[
https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-12765:
-
Attachment: HADOOP-12765.005.patch
While Min's patch looks good to me, it did not address [~vinayrpet]'s comment.
So I updated the patch to remove the changes in pom.xml. The code compiles in
my local tree. If it passes precommit I'll +1 and commit the patch.
{quote}
I wonder whether following change required in both hadoop-kms and https, as
dependency will be already propogated from hadoop-common.
+ org.mortbay.jetty
+ jetty-sslengine
+ test
+
{quote}
> HttpServer2 should switch to using the non-blocking SslSelectChannelConnector
> to prevent performance degradation when handling SSL connections
> --
>
> Key: HADOOP-12765
> URL: https://issues.apache.org/jira/browse/HADOOP-12765
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.2, 2.6.3
>Reporter: Min Shen
>Assignee: Min Shen
> Attachments: HADOOP-12765.001.patch, HADOOP-12765.001.patch,
> HADOOP-12765.002.patch, HADOOP-12765.003.patch, HADOOP-12765.004.patch,
> HADOOP-12765.005.patch, blocking_1.png, blocking_2.png, unblocking.png
>
>
> The current implementation uses the blocking SslSocketConnector which takes
> the default maxIdleTime as 200 seconds. We noticed in our cluster that when
> users use a custom client that accesses the WebHDFS REST APIs through https,
> it could block all the 250 handler threads in NN jetty server, causing severe
> performance degradation for accessing WebHDFS and NN web UI. Attached
> screenshots (blocking_1.png and blocking_2.png) illustrate that when using
> SslSocketConnector, the jetty handler threads are not released until the 200
> seconds maxIdleTime has passed. With sufficient number of SSL connections,
> this issue could render NN HttpServer to become entirely irresponsive.
> We propose to use the non-blocking SslSelectChannelConnector as a fix. We
> have deployed the attached patch within our cluster, and have seen
> significant improvement. The attached screenshot (unblocking.png) further
> illustrates the behavior of NN jetty server after switching to using
> SslSelectChannelConnector.
> The patch further disables SSLv3 protocol on server side to preserve the
> spirit of HADOOP-11260.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-12765: - Hadoop Flags: Reviewed > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Attachments: HADOOP-12765.001.patch, HADOOP-12765.001.patch, > HADOOP-12765.002.patch, HADOOP-12765.003.patch, HADOOP-12765.004.patch, > blocking_1.png, blocking_2.png, unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Shen updated HADOOP-12765: -- Attachment: HADOOP-12765.004.patch > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Attachments: HADOOP-12765.001.patch, HADOOP-12765.001.patch, > HADOOP-12765.002.patch, HADOOP-12765.003.patch, HADOOP-12765.004.patch, > blocking_1.png, blocking_2.png, unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Shen updated HADOOP-12765: -- Attachment: HADOOP-12765.003.patch Attaching revised patch to address [~jojochuang]'s comment. > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Attachments: HADOOP-12765.001.patch, HADOOP-12765.001.patch, > HADOOP-12765.002.patch, HADOOP-12765.003.patch, blocking_1.png, > blocking_2.png, unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Shen updated HADOOP-12765: -- Attachment: HADOOP-12765.002.patch Attach patch rebased on trunk > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Attachments: HADOOP-12765.001.patch, HADOOP-12765.001.patch, > HADOOP-12765.002.patch, blocking_1.png, blocking_2.png, unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HADOOP-12765: --- Attachment: HADOOP-12765.001.patch Re-attaching same patch to be picked up by yetus > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Attachments: HADOOP-12765.001.patch, HADOOP-12765.001.patch, > blocking_1.png, blocking_2.png, unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Shen updated HADOOP-12765: -- Affects Version/s: 2.7.2 2.6.3 > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Attachments: HADOOP-12765.001.patch, blocking_1.png, blocking_2.png, > unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Shen updated HADOOP-12765: -- Status: Patch Available (was: Open) > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Reporter: Min Shen >Assignee: Min Shen > Attachments: HADOOP-12765.001.patch, blocking_1.png, blocking_2.png, > unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Shen updated HADOOP-12765: -- Attachment: HADOOP-12765.001.patch > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Reporter: Min Shen >Assignee: Min Shen > Attachments: HADOOP-12765.001.patch > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Shen updated HADOOP-12765: -- Attachment: unblocking.png blocking_2.png blocking_1.png > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Reporter: Min Shen >Assignee: Min Shen > Attachments: HADOOP-12765.001.patch, blocking_1.png, blocking_2.png, > unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
