Hi Oleg, Also, fix[1] you attached, resolves the issue in our server.
[1] - https://github.com/ok2c/httpcomponents-core/commit/bf12c7dd9eabf0cd75bd6c70ac6b285b0d4430a5 Thanks & Regards, Arunan On Wed, Oct 21, 2020 at 10:57 PM Arunan Sugunakumar <[email protected]> wrote: > Hi Oleg, > > Thank you for confirming the bug. Please find the JIRA[1]. > > [1] - https://issues.apache.org/jira/browse/HTTPCORE-644 > > Regards, > Arunan > > On Wed, Oct 21, 2020 at 9:46 PM Oleg Kalnichevski <[email protected]> > wrote: > >> Arunan, >> >> I confirm this is a bug in HttpCore likely affecting all versions >> (4.4.x, 5.0.x and 5.1.x). >> >> Please raise a JIRA for this defect. >> >> Please try out this fix and let me know if that resolves the issue for >> you. >> >> >> https://github.com/ok2c/httpcomponents-core/commit/bf12c7dd9eabf0cd75bd6c70ac6b285b0d4430a5 >> >> >> Oleg >> >> On Wed, 2020-10-21 at 12:09 +0200, Oleg Kalnichevski wrote: >> > On Wed, 2020-10-21 at 00:25 +0530, Arunan Sugunakumar wrote: >> > > Hi Oleg, >> > > >> > > I was able to write a simple Test Server that reproduces this >> > > issue. >> > > I changed the NHttpFileServer example to come up with an example. >> > > If >> > > you send a POST request with a dummy payload and a >> > > "Connection:close" >> > > header to this server in 8443 port, you can see that the connection >> > > will not be closed (JDK11 & JDK8_261) and the CPU will be spinning >> > > continuously processing Write Events. If you start the same server >> > > in >> > > JDK8 (JDK8_251), the session will close normally. >> > > >> > > Please go through the example and provide your suggestions. >> > > >> > > Regards, >> > > Arunan >> > > >> > >> > Hi Arunan >> > >> > I was able to reproduce the issue with Oracle Java 11.0.8 which is a >> > reasonably up to date Java 11 runtime. However at the moment I do not >> > know whether this is a bug in Oracle's TLSv1.3 implementation or that >> > in HttpCore. It will likely take me a few days to complete the >> > analysis >> > and come up a fix. >> > >> > Oleg >> > >> > >> > > On Tue, Oct 13, 2020 at 11:07 AM Oleg Kalnichevski < >> > > [email protected]> >> > > wrote: >> > > > On Tue, 2020-10-13 at 08:42 +0530, Arunan Sugunakumar wrote: >> > > > > Hi Oleg, >> > > > > >> > > > > I upgraded to 4.4.13. Still the issue is there. >> > > > > >> > > > >> > > > Provide a reproducer. >> > > > >> > > > Oleg >> > > > >> > > > > Regards, >> > > > > Arunan >> > > > > >> > > > > On Mon, Oct 12, 2020, 20:56 Oleg Kalnichevski <[email protected] >> > > > > > >> > > > > wrote: >> > > > > >> > > > > > On Sun, 2020-10-11 at 22:16 +0530, Arunan Sugunakumar wrote: >> > > > > > > Hi, >> > > > > > > >> > > > > > > We have used httpcore-NIO to implement a non-blocking >> > > > > > > server >> > > > and >> > > > > > > with >> > > > > > > a recent JAVA upgrade, we are finding a high CPU usage in >> > > > > > > our >> > > > > > > servers. While we debugged, we found out that when we close >> > > > the >> > > > > > > HTTPS >> > > > > > > connection on the server-side, the WRITE event is getting >> > > > > > > fired continuously and the connection close is >> > > > > > > unsuccessful. >> > > > > > > >> > > > > > > To explain the code, we have extended >> > > > DefaultNHttpConnectionBase >> > > > > > > with >> > > > > > > our own ServerConnection, and inside close() we simply just >> > > > call >> > > > > > > the >> > > > > > > close() method of the NHttpConnectionBase[1]. If we look at >> > > > the >> > > > > > > code >> > > > > > > below, the close method sets the status as CLOSING and >> > > > > > > after >> > > > that >> > > > > > > either sets the WRITE event in the session or tries to >> > > > > > > close >> > > > the >> > > > > > > session (SSLIOSession in our case). >> > > > > > > >> > > > > > > @Override >> > > > > > > public void close() throws IOException { >> > > > > > > if (this.status != ACTIVE) { >> > > > > > > return; >> > > > > > > } >> > > > > > > this.status = CLOSING; >> > > > > > > if (this.outbuf.hasData()) { >> > > > > > > this.session.setEvent(EventMask.WRITE); >> > > > > > > } else { >> > > > > > > this.session.close(); >> > > > > > > this.status = CLOSED; >> > > > > > > } >> > > > > > > } >> > > > > > > >> > > > > > > Inside SSLIOSession close(), the updateEventMask() [2] is >> > > > called. >> > > > > > > @Override >> > > > > > > public synchronized void close() { >> > > > > > > if (this.status >= CLOSING) { >> > > > > > > return; >> > > > > > > } >> > > > > > > this.status = CLOSING; >> > > > > > > if (this.session.getSocketTimeout() == 0) { >> > > > > > > this.session.setSocketTimeout(1000); >> > > > > > > } >> > > > > > > try { >> > > > > > > updateEventMask(); >> > > > > > > } catch (final CancelledKeyException ex) { >> > > > > > > shutdown(); >> > > > > > > } >> > > > > > > } >> > > > > > > >> > > > > > > Inside updateEventMask() method, we get the HandshakeStatus >> > > > from >> > > > > > > the >> > > > > > > JAVA SSLEngine, and set a new Mask based on the status. >> > > > > > > switch (this.sslEngine.getHandshakeStatus()) { >> > > > > > > case NEED_WRAP: >> > > > > > > newMask = EventMask.READ_WRITE; >> > > > > > > break; >> > > > > > > case NEED_UNWRAP: >> > > > > > > newMask = EventMask.READ; >> > > > > > > break; >> > > > > > > case NOT_HANDSHAKING: >> > > > > > > newMask = this.appEventMask; >> > > > > > > break; >> > > > > > > case NEED_TASK: >> > > > > > > break; >> > > > > > > case FINISHED: >> > > > > > > break; >> > > > > > > } >> > > > > > > Before JDK upgrade, when outbound is closed, we used to get >> > > > the >> > > > > > > status as NEED_UNWRAP which sets the newMask as READ mask >> > > > which >> > > > > > > closes the connection in our server perfectly. >> > > > > > > But with a recent JDK commit[3], the new status is returned >> > > > as >> > > > > > > NOT_HANDSHAKING. >> > > > > > > Since mask is not set to READ because of the >> > > > > > > NOT_HANDSHAKING >> > > > > > > status, >> > > > > > > WRITE event is continuously getting fired which makes the >> > > > Server >> > > > > > > connection close unsuccessful. >> > > > > > > When the close is called multiple times for multiple >> > > > connections, >> > > > > > > we >> > > > > > > could observe that CPU usage increased continuosly and >> > > > reached >> > > > > > > 100 >> > > > > > > percent and stays there continuously. >> > > > > > > When I look at the stack trace, the execution flow was >> > > > completely >> > > > > > > inside the httpcore-nio component, which made our attempt >> > > > > > > to >> > > > > > > clear >> > > > > > > the Event from the server implementation, unsuccessful. >> > > > > > > >> > > > > > > >> > > > > > > Earlier we were using Httpcore-NIO version 4.3.3. With this >> > > > > > > issue, we >> > > > > > > tried upgrading to 4.4.13. But still the issue exists. >> > > > > > >> > > > > > I have seen no evidence supporting that. Please upgrade to >> > > > 4.4.13. >> > > > > > Oleg >> > > > > > >> > > > > > >> > > > > > >> > > > > > ------------------------------------------------------------- >> > > > ---- >> > > > > > ---- >> > > > > > To unsubscribe, e-mail: [email protected] >> > > > > > For additional commands, e-mail: [email protected] >> > > > > > >> > > > > > >> > > > >> > > > ----------------------------------------------------------------- >> > > > ---- >> > > > To unsubscribe, e-mail: [email protected] >> > > > For additional commands, e-mail: [email protected] >> > > > >> > > >> > > ----------------------------------------------------------------- >> > > ---- >> > > To unsubscribe, e-mail: [email protected] >> > > For additional commands, e-mail: [email protected] >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [email protected] >> > For additional commands, e-mail: [email protected] >> > >> >> > > -- > *Sugunakumar Arunan* > Software Engineer | WSO2 > > Email : [email protected] > Mobile : +94766016272 > Web : http://wso2.com > <https://wso2.com/signature> > -- *Sugunakumar Arunan* Software Engineer | WSO2 Email : [email protected] Mobile : +94766016272 Web : http://wso2.com <https://wso2.com/signature>
