Re: HTTP/2 connection broken (RST_STREAM) when multiple timeouts waiting for data from client
> > > > On 18/11/2022 10:21, Mark Thomas wrote: > > On 17/11/2022 20:01, Gonzalo Fernandez wrote: > > > > > > > >>> The problem happens when a client with an open TCP / HTTP2 connection > >>> sends multiple incomplete streams, which seems to block the connection > >>> forever and not be able to accept new streams. > >>> > >>> In detail, using tcpdump, we are seeing that the Tomcat Server > >>> receives the HEADER frame, but never receives the following DATA > >>> frames for a stream. After 20 seconds (streamReadTimeout default > >>> value), the Tomcat Server throws a "client timeout" exception > >>> (stacktrace at the end of the mail), and returns "400 Bad Request" > >>> followed by a RST_STREAM frame. > >>> The problem is that when the amount of incomplete streams surpasses > >>> the value of "maxConcurrentStreams", the connection starts to return > >>> RST_STREAM to any new stream indefinitely, it never recovers. I > >>> verified this by changing the value of that property and looking at > >>> the number of streams in every connection. When the value wasn't > >>> defined, it took 100 incomplete streams over the same connection to > >>> break it, then I changed to 20 and it took 20 incomplete streams to > >>> break it. > >>> > >>> This makes me suspicious that the concurrent streams counter isn't > >>> being decreased when this happens, and could possibly be a bug. I > >>> tried to identify this in the tomcat codebase but I am not familiar > >>> enough with it. > > > > Thanks for the very detailed description. > > > > Your suspicion is correct. There is a Tomcat bug here. I have been able > > to write a test case that demonstrates the bug and am working on a fix. > > > > I just wanted to say how much the effort you put into researching this > > is appreciated. Your report has exactly the right amount of detail we > > need. Enough to see exactly what the problem is and to recreate the > > problem and no unnecessary extra information. > > > > I should have a patch for this shortly. > > Fixed in: > - 10.1.x for 10.1.3 onwards > - 9.0.x for 9.0.70 onwards > - 8.5.x for 8.5.85 onwards > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > Thanks for the quick response. I appreciate your work. Have a great day. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HTTP/2 connection broken (RST_STREAM) when multiple timeouts waiting for data from client
On 18/11/2022 10:21, Mark Thomas wrote: On 17/11/2022 20:01, Gonzalo Fernandez wrote: The problem happens when a client with an open TCP / HTTP2 connection sends multiple incomplete streams, which seems to block the connection forever and not be able to accept new streams. In detail, using tcpdump, we are seeing that the Tomcat Server receives the HEADER frame, but never receives the following DATA frames for a stream. After 20 seconds (streamReadTimeout default value), the Tomcat Server throws a "client timeout" exception (stacktrace at the end of the mail), and returns "400 Bad Request" followed by a RST_STREAM frame. The problem is that when the amount of incomplete streams surpasses the value of "maxConcurrentStreams", the connection starts to return RST_STREAM to any new stream indefinitely, it never recovers. I verified this by changing the value of that property and looking at the number of streams in every connection. When the value wasn't defined, it took 100 incomplete streams over the same connection to break it, then I changed to 20 and it took 20 incomplete streams to break it. This makes me suspicious that the concurrent streams counter isn't being decreased when this happens, and could possibly be a bug. I tried to identify this in the tomcat codebase but I am not familiar enough with it. Thanks for the very detailed description. Your suspicion is correct. There is a Tomcat bug here. I have been able to write a test case that demonstrates the bug and am working on a fix. I just wanted to say how much the effort you put into researching this is appreciated. Your report has exactly the right amount of detail we need. Enough to see exactly what the problem is and to recreate the problem and no unnecessary extra information. I should have a patch for this shortly. Fixed in: - 10.1.x for 10.1.3 onwards - 9.0.x for 9.0.70 onwards - 8.5.x for 8.5.85 onwards Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HTTP/2 connection broken (RST_STREAM) when multiple timeouts waiting for data from client
On 17/11/2022 20:01, Gonzalo Fernandez wrote: The problem happens when a client with an open TCP / HTTP2 connection sends multiple incomplete streams, which seems to block the connection forever and not be able to accept new streams. In detail, using tcpdump, we are seeing that the Tomcat Server receives the HEADER frame, but never receives the following DATA frames for a stream. After 20 seconds (streamReadTimeout default value), the Tomcat Server throws a "client timeout" exception (stacktrace at the end of the mail), and returns "400 Bad Request" followed by a RST_STREAM frame. The problem is that when the amount of incomplete streams surpasses the value of "maxConcurrentStreams", the connection starts to return RST_STREAM to any new stream indefinitely, it never recovers. I verified this by changing the value of that property and looking at the number of streams in every connection. When the value wasn't defined, it took 100 incomplete streams over the same connection to break it, then I changed to 20 and it took 20 incomplete streams to break it. This makes me suspicious that the concurrent streams counter isn't being decreased when this happens, and could possibly be a bug. I tried to identify this in the tomcat codebase but I am not familiar enough with it. Thanks for the very detailed description. Your suspicion is correct. There is a Tomcat bug here. I have been able to write a test case that demonstrates the bug and am working on a fix. I just wanted to say how much the effort you put into researching this is appreciated. Your report has exactly the right amount of detail we need. Enough to see exactly what the problem is and to recreate the problem and no unnecessary extra information. I should have a patch for this shortly. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HTTP/2 connection broken (RST_STREAM) when multiple timeouts waiting for data from client
> > Hello. > > We are experiencing a problem in the following tomcat versions: > Tomcat 9.0.63 on OpenJDK 11.0.13 > Tomcat 9.0.65 on OpenJDK 17.0.5 > > The problem happens when a client with an open TCP / HTTP2 connection > sends multiple incomplete streams, which seems to block the connection > forever and not be able to accept new streams. > > In detail, using tcpdump, we are seeing that the Tomcat Server > receives the HEADER frame, but never receives the following DATA > frames for a stream. After 20 seconds (streamReadTimeout default > value), the Tomcat Server throws a "client timeout" exception > (stacktrace at the end of the mail), and returns "400 Bad Request" > followed by a RST_STREAM frame. > The problem is that when the amount of incomplete streams surpasses > the value of "maxConcurrentStreams", the connection starts to return > RST_STREAM to any new stream indefinitely, it never recovers. I > verified this by changing the value of that property and looking at > the number of streams in every connection. When the value wasn't > defined, it took 100 incomplete streams over the same connection to > break it, then I changed to 20 and it took 20 incomplete streams to > break it. > > This makes me suspicious that the concurrent streams counter isn't > being decreased when this happens, and could possibly be a bug. I > tried to identify this in the tomcat codebase but I am not familiar > enough with it. > > I managed to reproduce this but the setup is a bit tricky because the > http client must support overriding the "Content-Length" header to > cause a "timeout" exception while waiting for stream data. The way I > managed to do it is with an Envoy Proxy in the middle (which actually > is the way this is happening in our environment). > I set an Envoy Proxy configured to forward the requests to a Tomcat > Server using a HTTP2 connection. The application in the Tomcat Server > expects a POST request. Then I use curl to send a HTTP/1.1 request, > setting the header "Content-Length: 40", but without sending any data > in the body. > > curl --http1.1 -X POST http://localhost:1/ -H "Content-Length: 40" -H > > "Content-Type: application/json" > Monitoring this with wireshark, I observe that when the number of > streams (not necessarily sent concurrently) reaches the value of > maxConcurrentStreams, that connection stays broken forever (replies > RST_STREAM to any subsequent request). > > Would like to get help verifying if this is an unexpected behavior, so > I proceed to create the bug report, or is working as expected and we > should mitigate it in some other way. > > I read on the mailing list docs that must not attach anything to the > mail. In case you want the tcpdump capture, let me know. > > Thanks in advance! > > Stacktraces: > > Resolved > > [org.springframework.http.converter.HttpMessageNotReadableException: I/O > > error while reading input message; nested exception is > > org.apache.catalina.connector.ClientAbortException: java.io.IOException: > > Stream reset] > > > java.lang.ClassCastException: class > > org.apache.catalina.connector.ClientAbortException cannot be cast to class > > com.fasterxml.jackson.databind.exc.InvalidFormatException > > (org.apache.catalina.connector.ClientAbortException and > > com.fasterxml.jackson.databind.exc.InvalidFormatException are in unnamed > > module of loader org.springframework.boot.loader.LaunchedURLClassLoader > > @76f2b07d) > >at > > ar.com.nbch.exceptions.CustomGlobalExceptionHandler.handleHttpMessageNotReadable(CustomGlobalExceptionHandler.java:223) > >at > > org.springframework.web.servlet.mvc.method.annotation.ResponseEntityExceptionHandler.handleException(ResponseEntityExceptionHandler.java:161) > >at jdk.internal.reflect.GeneratedMethodAccessor185.invoke(Unknown Source) > >at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > Source) > >at java.base/java.lang.reflect.Method.invoke(Unknown Source) > >at > > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > >at > > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) > >at > > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) > >at > > org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException(ExceptionHandlerExceptionResolver.java:428) > >at > > org.springframework.web.servlet.handler.AbstractHandlerMethodExceptionResolver.doResolveException(AbstractHandlerMethodExceptionResolver.java:75) > >at > > org.springframework.web.servlet.handler.AbstractHandlerExceptionResolver.resolveException(AbstractHandlerExceptionResolver.java:142) > >at > >
