Nick Cabatoff created HTTPCORE-483:
--------------------------------------
Summary: High CPU usage after enabling chunked https responses
Key: HTTPCORE-483
URL: https://issues.apache.org/jira/browse/HTTPCORE-483
Project: HttpComponents HttpCore
Issue Type: Bug
Components: HttpCore NIO
Affects Versions: 4.4.6
Environment: Centos 7
Reporter: Nick Cabatoff
Attachments: fg1-initial-unchunked.svg, fg2-initial-chunked.svg,
fg3-patched-unchunked.svg, fg4-patched-chunked.svg
I wanted persistent connections with keep-alives, and I discovered that the
reason I wasn't getting them was because my responses didn't have a
content-length or transfer-encoding: chunked. So I did body.setChunked( true
), passed that to response.setEntity(), and it worked. But I noticed that my
app was pegged at 100% CPU in top. I found that the app was fine when it
started, but after servicing a single HTTPS request with chunked responses, it
got stuck in this 100% cpu state. It turns out that the reactor thread was
busy looping: despite there being no connections open after servicing that one
request, in AbstractIOReactor.execute() the selector.select() call continued to
return non-zero, and processEvent() always found a key that was writable.
Stepping through the code I noticed that SSLIOSession.updateEventMask() wasn't
closing the session:
{noformat}
// Graceful session termination
if (this.status == CLOSING && this.sslEngine.isOutboundDone()
&& (this.endOfStream || this.sslEngine.isInboundDone())) {
this.status = CLOSED;
}
{noformat}
The status was CLOSING and endOfStream was true, but sslEngine.isOutboundDone()
returned false.
I looked at httpcore 5 and found that a new check had been added just above:
{noformat}
diff --git
a/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
b/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
---
a/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
+++
b/httpcore-nio/src/main/java/org/apache/http/nio/reactor/ssl/SSLIOSession.java
@@ -371,6 +371,11 @@
private void updateEventMask() {
// Graceful session termination
+ if (this.status == CLOSING && !this.outEncrypted.hasData()) {
+ this.sslEngine.closeOutbound();
+ }
+
+ // Graceful session termination
if (this.status == CLOSING && this.sslEngine.isOutboundDone()
&& (this.endOfStream || this.sslEngine.isInboundDone())) {
this.status = CLOSED;
{noformat}
I applied this patch to my httpcore 4.4.6 and the problem went away. Now after
servicing a request with chunked responses the session gets closed properly and
the reactor is properly idle.
There's still another problem outstanding though. Request processing now
consumes way more CPU than it used to. Doing one request per second without
chunked responses, my app consumes ~20% of one core; with chunked responses
it's closer to 100%.
I'm attaching flame graphs (see
[http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html]): (1) my starting
point without chunked respones; (2) with chunked respones; (3) after applying
the above patch, without chunked responses; (4) after applying patch, with
chunked responses. Note the similarity between (2) and (4): in both cases,
much of the Java code (green) is mostly consumed with epoll calls.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
