[
https://issues.apache.org/jira/browse/HTTPCORE-483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139198#comment-16139198
]
Nick Cabatoff commented on HTTPCORE-483:
----------------------------------------
The responses are 11 bytes long. Also, as you can see in the flame graphs, the
load is basically all in SelectorImpl.select(), which for some reason in the
chunked mode is doing a lot more epolling. I also see this with strace, e.g.
{noformat}
121911 17:19:50.617523 epoll_wait(103, {{EPOLLOUT, {u32=112,
u64=16899750248723775600}}}, 4096, 1000) = 1
121911 17:19:50.617599 epoll_wait(103, {{EPOLLOUT, {u32=112,
u64=16899750248723775600}}}, 4096, 1000) = 1
121911 17:19:50.617678 epoll_wait(103, {{EPOLLOUT, {u32=112,
u64=16899750248723775600}}}, 4096, 1000) = 1
121911 17:19:50.617740 epoll_wait(103, {{EPOLLOUT, {u32=112,
u64=16899750248723775600}}}, 4096, 1000) = 1
121911 17:19:50.617808 epoll_wait(103, {{EPOLLOUT, {u32=112,
u64=16899750248723775600}}}, 4096, 1000) = 1
{noformat}
Obviously there's still some epoll_waits happening in the non-chunked version,
but it's not constant like this.
> High CPU usage after enabling chunked https responses
> -----------------------------------------------------
>
> Key: HTTPCORE-483
> URL: https://issues.apache.org/jira/browse/HTTPCORE-483
> Project: HttpComponents HttpCore
> Issue Type: Bug
> Components: HttpCore NIO
> Affects Versions: 4.4.6
> Environment: Centos 7
> Reporter: Nick Cabatoff
> Attachments: fg1-initial-unchunked.svg, fg2-initial-chunked.svg,
> fg3-patched-unchunked.svg, fg4-patched-chunked.svg
>
>
> I wanted persistent connections with keep-alives, and I discovered that the
> reason I wasn't getting them was because my responses didn't have a
> content-length or transfer-encoding: chunked. So I did body.setChunked( true
> ), passed that to response.setEntity(), and it worked. But I noticed that my
> app was pegged at 100% CPU in top. I found that the app was fine when it
> started, but after servicing a single HTTPS request with chunked responses,
> it got stuck in this 100% cpu state. It turns out that the reactor thread
> was busy looping: despite there being no connections open after servicing
> that one request, in AbstractIOReactor.execute() the selector.select() call
> continued to return non-zero, and processEvent() always found a key that was
> writable. Stepping through the code I noticed that
> SSLIOSession.updateEventMask() wasn't closing the session:
> {noformat}
> // Graceful session termination
> if (this.status == CLOSING && this.sslEngine.isOutboundDone()
> && (this.endOfStream || this.sslEngine.isInboundDone())) {
> this.status = CLOSED;
> }
> {noformat}
> The status was CLOSING and endOfStream was true, but
> sslEngine.isOutboundDone() returned false.
> I looked at httpcore 5 and found that a new check had been added just above:
> {noformat}
> diff --git
> a/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
>
> b/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
> ---
> a/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
> +++
> b/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
> @@ -371,6 +371,11 @@
>
> private void updateEventMask() {
> // Graceful session termination
> + if (this.status == CLOSING && !this.outEncrypted.hasData()) {
> + this.sslEngine.closeOutbound();
> + }
> +
> + // Graceful session termination
> if (this.status == CLOSING && this.sslEngine.isOutboundDone()
> && (this.endOfStream || this.sslEngine.isInboundDone())) {
> this.status = CLOSED;
> {noformat}
> I applied this patch to my httpcore 4.4.6 and the problem went away. Now
> after servicing a request with chunked responses the session gets closed
> properly and the reactor is properly idle.
> There's still another problem outstanding though. Request processing now
> consumes way more CPU than it used to. Doing one request per second without
> chunked responses, my app consumes ~20% of one core; with chunked responses
> it's closer to 100%.
> I'm attaching flame graphs (see
> [http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html]): (1) my
> starting point without chunked respones; (2) with chunked respones; (3) after
> applying the above patch, without chunked responses; (4) after applying
> patch, with chunked responses. Note the similarity between (2) and (4): in
> both cases, much of the Java code (green) is mostly consumed with epoll calls.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
