[
https://issues.apache.org/jira/browse/HTTPCORE-483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16140517#comment-16140517
]
Nick Cabatoff commented on HTTPCORE-483:
----------------------------------------
{quote} chunk-coding may cause messages to get split into multiple IP frames,
which would explain extra epolling.{quote}
That's plausible, but again I don't think there's any way it could account for
the volume. To demonstrate, I switched the request frequency to 1/minute and
stopped everything else on the system besides this one app (request issuer is
on another host). As soon as the first request is serviced, the CPU goes from
0 to 100 and stays there so long as the connection is open. So it doesn't look
to have anything to do with the chunking per se, except insofar as using
chunked responses allows persistent connections, and in my setup the persistent
connection causes the IO dispatcher thread to continuously wake up thinking it
has work to do. As with my original problem, it always lands in
processEvent(), having found a session that was writable.
I've tried suspending the IO dispatch thread in the debugger, and stepping
through what happens. It reaches ThrottlingHttpServiceHandler.responseReady(),
then the
{noformat}
if ( connState.getOutputState() == ServerConnState.READY
&& response != null
&& !conn.isResponseSubmitted())
{noformat}
condition fails, I guess because the response has already been sent.
> High CPU usage after enabling chunked https responses
> -----------------------------------------------------
>
> Key: HTTPCORE-483
> URL: https://issues.apache.org/jira/browse/HTTPCORE-483
> Project: HttpComponents HttpCore
> Issue Type: Bug
> Components: HttpCore NIO
> Affects Versions: 4.4.6
> Environment: Centos 7
> Reporter: Nick Cabatoff
> Attachments: fg1-initial-unchunked.svg, fg2-initial-chunked.svg,
> fg3-patched-unchunked.svg, fg4-patched-chunked.svg
>
>
> I wanted persistent connections with keep-alives, and I discovered that the
> reason I wasn't getting them was because my responses didn't have a
> content-length or transfer-encoding: chunked. So I did body.setChunked( true
> ), passed that to response.setEntity(), and it worked. But I noticed that my
> app was pegged at 100% CPU in top. I found that the app was fine when it
> started, but after servicing a single HTTPS request with chunked responses,
> it got stuck in this 100% cpu state. It turns out that the reactor thread
> was busy looping: despite there being no connections open after servicing
> that one request, in AbstractIOReactor.execute() the selector.select() call
> continued to return non-zero, and processEvent() always found a key that was
> writable. Stepping through the code I noticed that
> SSLIOSession.updateEventMask() wasn't closing the session:
> {noformat}
> // Graceful session termination
> if (this.status == CLOSING && this.sslEngine.isOutboundDone()
> && (this.endOfStream || this.sslEngine.isInboundDone())) {
> this.status = CLOSED;
> }
> {noformat}
> The status was CLOSING and endOfStream was true, but
> sslEngine.isOutboundDone() returned false.
> I looked at httpcore 5 and found that a new check had been added just above:
> {noformat}
> diff --git
> a/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
>
> b/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
> ---
> a/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
> +++
> b/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
> @@ -371,6 +371,11 @@
>
> private void updateEventMask() {
> // Graceful session termination
> + if (this.status == CLOSING && !this.outEncrypted.hasData()) {
> + this.sslEngine.closeOutbound();
> + }
> +
> + // Graceful session termination
> if (this.status == CLOSING && this.sslEngine.isOutboundDone()
> && (this.endOfStream || this.sslEngine.isInboundDone())) {
> this.status = CLOSED;
> {noformat}
> I applied this patch to my httpcore 4.4.6 and the problem went away. Now
> after servicing a request with chunked responses the session gets closed
> properly and the reactor is properly idle.
> There's still another problem outstanding though. Request processing now
> consumes way more CPU than it used to. Doing one request per second without
> chunked responses, my app consumes ~20% of one core; with chunked responses
> it's closer to 100%.
> I'm attaching flame graphs (see
> [http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html]): (1) my
> starting point without chunked respones; (2) with chunked respones; (3) after
> applying the above patch, without chunked responses; (4) after applying
> patch, with chunked responses. Note the similarity between (2) and (4): in
> both cases, much of the Java code (green) is mostly consumed with epoll calls.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
