Hello. We are experiencing a problem in the following tomcat versions: Tomcat 9.0.63 on OpenJDK 11.0.13 Tomcat 9.0.65 on OpenJDK 17.0.5
The problem happens when a client with an open TCP / HTTP2 connection sends multiple incomplete streams, which seems to block the connection forever and not be able to accept new streams. In detail, using tcpdump, we are seeing that the Tomcat Server receives the HEADER frame, but never receives the following DATA frames for a stream. After 20 seconds (streamReadTimeout default value), the Tomcat Server throws a "client timeout" exception (stacktrace at the end of the mail), and returns "400 Bad Request" followed by a RST_STREAM frame. The problem is that when the amount of incomplete streams surpasses the value of "maxConcurrentStreams", the connection starts to return RST_STREAM to any new stream indefinitely, it never recovers. I verified this by changing the value of that property and looking at the number of streams in every connection. When the value wasn't defined, it took 100 incomplete streams over the same connection to break it, then I changed to 20 and it took 20 incomplete streams to break it. This makes me suspicious that the concurrent streams counter isn't being decreased when this happens, and could possibly be a bug. I tried to identify this in the tomcat codebase but I am not familiar enough with it. I managed to reproduce this but the setup is a bit tricky because the http client must support overriding the "Content-Length" header to cause a "timeout" exception while waiting for stream data. The way I managed to do it is with an Envoy Proxy in the middle (which actually is the way this is happening in our environment). I set an Envoy Proxy configured to forward the requests to a Tomcat Server using a HTTP2 connection. The application in the Tomcat Server expects a POST request. Then I use curl to send a HTTP/1.1 request, setting the header "Content-Length: 40", but without sending any data in the body. > curl --http1.1 -X POST http://localhost:10000/ -H "Content-Length: 40" -H > "Content-Type: application/json" Monitoring this with wireshark, I observe that when the number of streams (not necessarily sent concurrently) reaches the value of maxConcurrentStreams, that connection stays broken forever (replies RST_STREAM to any subsequent request). Would like to get help verifying if this is an unexpected behavior, so I proceed to create the bug report, or is working as expected and we should mitigate it in some other way. I read on the mailing list docs that must not attach anything to the mail. In case you want the tcpdump capture, let me know. Thanks in advance! Stacktraces: > Resolved [org.springframework.http.converter.HttpMessageNotReadableException: > I/O error while reading input message; nested exception is > org.apache.catalina.connector.ClientAbortException: java.io.IOException: > Stream reset] > java.lang.ClassCastException: class > org.apache.catalina.connector.ClientAbortException cannot be cast to class > com.fasterxml.jackson.databind.exc.InvalidFormatException > (org.apache.catalina.connector.ClientAbortException and > com.fasterxml.jackson.databind.exc.InvalidFormatException are in unnamed > module of loader org.springframework.boot.loader.LaunchedURLClassLoader > @76f2b07d) > at > ar.com.nbch.exceptions.CustomGlobalExceptionHandler.handleHttpMessageNotReadable(CustomGlobalExceptionHandler.java:223) > at > org.springframework.web.servlet.mvc.method.annotation.ResponseEntityExceptionHandler.handleException(ResponseEntityExceptionHandler.java:161) > at jdk.internal.reflect.GeneratedMethodAccessor185.invoke(Unknown Source) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > Source) > at java.base/java.lang.reflect.Method.invoke(Unknown Source) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) > at > org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException(ExceptionHandlerExceptionResolver.java:428) > at > org.springframework.web.servlet.handler.AbstractHandlerMethodExceptionResolver.doResolveException(AbstractHandlerMethodExceptionResolver.java:75) > at > org.springframework.web.servlet.handler.AbstractHandlerExceptionResolver.resolveException(AbstractHandlerExceptionResolver.java:142) > at > org.springframework.web.servlet.handler.HandlerExceptionResolverComposite.resolveException(HandlerExceptionResolverComposite.java:80) > at > org.springframework.web.servlet.DispatcherServlet.processHandlerException(DispatcherServlet.java:1327) > at > org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1138) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1084) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) > at > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) > at > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at brave.servlet.TracingFilter.doFilter(TracingFilter.java:68) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at > ar.com.nbch.canal.configuration.VersionHeadersFilter.doFilterInternal(VersionHeadersFilter.java:77) > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at ar.com.nbch.canal.filter.LogFilter.doFilter(LogFilter.java:52) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at > org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at > org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at brave.servlet.TracingFilter.doFilter(TracingFilter.java:87) > at > org.springframework.cloud.sleuth.instrument.web.LazyTracingFilter.doFilter(TraceWebServletAutoConfiguration.java:141) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at > org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at > org.springframework.web.filter.ForwardedHeaderFilter.doFilterInternal(ForwardedHeaderFilter.java:156) > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) > at > org.apache.coyote.http2.StreamProcessor.service(StreamProcessor.java:426) > at > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) > at org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:87) > at org.apache.coyote.http2.StreamRunnable.run(StreamRunnable.java:35) > at > org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) > at > org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.base/java.lang.Thread.run(Unknown Source) > org.apache.catalina.connector.ClientAbortException: > org.apache.coyote.CloseNowException: Connection [0], Stream [373], This > stream is not writable > at > org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:310) > at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:273) > at > org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118) > at java.base/java.io.FilterOutputStream.flush(Unknown Source) > at > com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1193) > at > com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1008) > at > org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:456) > at > org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:104) > at > org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:290) > at > org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor.handleReturnValue(HttpEntityMethodProcessor.java:219) > at > org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:78) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:135) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) > at > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) > at > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at > org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at brave.servlet.TracingFilter.doFilter(TracingFilter.java:68) > at > org.springframework.cloud.sleuth.instrument.web.LazyTracingFilter.doFilter(TraceWebServletAutoConfiguration.java:141) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at > org.springframework.web.filter.ForwardedHeaderFilter.doFilterInternal(ForwardedHeaderFilter.java:156) > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > at > org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:711) > at > org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461) > at > org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:385) > at > org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:313) > at > org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:403) > at > org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:249) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) > at > org.apache.coyote.http2.StreamProcessor.service(StreamProcessor.java:426) > at > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) > at org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:87) > at org.apache.coyote.http2.StreamRunnable.run(StreamRunnable.java:35) > at > org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) > at > org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.base/java.lang.Thread.run(Unknown Source) > Caused by: org.apache.coyote.CloseNowException: Connection [0], Stream [373], > This stream is not writable > at org.apache.coyote.http2.Stream.doStreamCancel(Stream.java:257) > at > org.apache.coyote.http2.Http2UpgradeHandler.reserveWindowSize(Http2UpgradeHandler.java:892) > at org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:940) > at org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:886) > at > org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:1009) > at > org.apache.coyote.http2.Http2OutputBuffer.flush(Http2OutputBuffer.java:77) > at org.apache.coyote.http2.StreamProcessor.flush(StreamProcessor.java:254) > at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:402) > at org.apache.coyote.Response.action(Response.java:209) > at > org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:306) > ... 62 common frames omitted > Caused by: org.apache.coyote.http2.StreamException: Connection [0], Stream > [373], This stream is not writable > at org.apache.coyote.http2.Stream.doStreamCancel(Stream.java:249) > ... 71 common frames omitted --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
