> > Hello. > > We are experiencing a problem in the following tomcat versions: > Tomcat 9.0.63 on OpenJDK 11.0.13 > Tomcat 9.0.65 on OpenJDK 17.0.5 > > The problem happens when a client with an open TCP / HTTP2 connection > sends multiple incomplete streams, which seems to block the connection > forever and not be able to accept new streams. > > In detail, using tcpdump, we are seeing that the Tomcat Server > receives the HEADER frame, but never receives the following DATA > frames for a stream. After 20 seconds (streamReadTimeout default > value), the Tomcat Server throws a "client timeout" exception > (stacktrace at the end of the mail), and returns "400 Bad Request" > followed by a RST_STREAM frame. > The problem is that when the amount of incomplete streams surpasses > the value of "maxConcurrentStreams", the connection starts to return > RST_STREAM to any new stream indefinitely, it never recovers. I > verified this by changing the value of that property and looking at > the number of streams in every connection. When the value wasn't > defined, it took 100 incomplete streams over the same connection to > break it, then I changed to 20 and it took 20 incomplete streams to > break it. > > This makes me suspicious that the concurrent streams counter isn't > being decreased when this happens, and could possibly be a bug. I > tried to identify this in the tomcat codebase but I am not familiar > enough with it. > > I managed to reproduce this but the setup is a bit tricky because the > http client must support overriding the "Content-Length" header to > cause a "timeout" exception while waiting for stream data. The way I > managed to do it is with an Envoy Proxy in the middle (which actually > is the way this is happening in our environment). > I set an Envoy Proxy configured to forward the requests to a Tomcat > Server using a HTTP2 connection. The application in the Tomcat Server > expects a POST request. Then I use curl to send a HTTP/1.1 request, > setting the header "Content-Length: 40", but without sending any data > in the body. > > curl --http1.1 -X POST http://localhost:10000/ -H "Content-Length: 40" -H > > "Content-Type: application/json" > Monitoring this with wireshark, I observe that when the number of > streams (not necessarily sent concurrently) reaches the value of > maxConcurrentStreams, that connection stays broken forever (replies > RST_STREAM to any subsequent request). > > Would like to get help verifying if this is an unexpected behavior, so > I proceed to create the bug report, or is working as expected and we > should mitigate it in some other way. > > I read on the mailing list docs that must not attach anything to the > mail. In case you want the tcpdump capture, let me know. > > Thanks in advance! > > Stacktraces: > > Resolved > > [org.springframework.http.converter.HttpMessageNotReadableException: I/O > > error while reading input message; nested exception is > > org.apache.catalina.connector.ClientAbortException: java.io.IOException: > > Stream reset] > > > java.lang.ClassCastException: class > > org.apache.catalina.connector.ClientAbortException cannot be cast to class > > com.fasterxml.jackson.databind.exc.InvalidFormatException > > (org.apache.catalina.connector.ClientAbortException and > > com.fasterxml.jackson.databind.exc.InvalidFormatException are in unnamed > > module of loader org.springframework.boot.loader.LaunchedURLClassLoader > > @76f2b07d) > > at > > ar.com.nbch.exceptions.CustomGlobalExceptionHandler.handleHttpMessageNotReadable(CustomGlobalExceptionHandler.java:223) > > at > > org.springframework.web.servlet.mvc.method.annotation.ResponseEntityExceptionHandler.handleException(ResponseEntityExceptionHandler.java:161) > > at jdk.internal.reflect.GeneratedMethodAccessor185.invoke(Unknown Source) > > at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > Source) > > at java.base/java.lang.reflect.Method.invoke(Unknown Source) > > at > > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > > at > > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) > > at > > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) > > at > > org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException(ExceptionHandlerExceptionResolver.java:428) > > at > > org.springframework.web.servlet.handler.AbstractHandlerMethodExceptionResolver.doResolveException(AbstractHandlerMethodExceptionResolver.java:75) > > at > > org.springframework.web.servlet.handler.AbstractHandlerExceptionResolver.resolveException(AbstractHandlerExceptionResolver.java:142) > > at > > org.springframework.web.servlet.handler.HandlerExceptionResolverComposite.resolveException(HandlerExceptionResolverComposite.java:80) > > at > > org.springframework.web.servlet.DispatcherServlet.processHandlerException(DispatcherServlet.java:1327) > > at > > org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1138) > > at > > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1084) > > at > > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) > > at > > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) > > at > > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) > > at > > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at brave.servlet.TracingFilter.doFilter(TracingFilter.java:68) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at > > ar.com.nbch.canal.configuration.VersionHeadersFilter.doFilterInternal(VersionHeadersFilter.java:77) > > at > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at ar.com.nbch.canal.filter.LogFilter.doFilter(LogFilter.java:52) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at > > org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) > > at > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at > > org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) > > at > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at brave.servlet.TracingFilter.doFilter(TracingFilter.java:87) > > at > > org.springframework.cloud.sleuth.instrument.web.LazyTracingFilter.doFilter(TraceWebServletAutoConfiguration.java:141) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at > > org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) > > at > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at > > org.springframework.web.filter.ForwardedHeaderFilter.doFilterInternal(ForwardedHeaderFilter.java:156) > > at > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) > > at > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) > > at > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) > > at > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) > > at > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) > > at > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) > > at > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) > > at > > org.apache.coyote.http2.StreamProcessor.service(StreamProcessor.java:426) > > at > > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) > > at > > org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:87) > > at org.apache.coyote.http2.StreamRunnable.run(StreamRunnable.java:35) > > at > > org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) > > at > > org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) > > at > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > > at java.base/java.lang.Thread.run(Unknown Source) > > > org.apache.catalina.connector.ClientAbortException: > > org.apache.coyote.CloseNowException: Connection [0], Stream [373], This > > stream is not writable > > at > > org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:310) > > at > > org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:273) > > at > > org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118) > > at java.base/java.io.FilterOutputStream.flush(Unknown Source) > > at > > com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1193) > > at > > com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1008) > > at > > org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:456) > > at > > org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:104) > > at > > org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:290) > > at > > org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor.handleReturnValue(HttpEntityMethodProcessor.java:219) > > at > > org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:78) > > at > > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:135) > > at > > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) > > at > > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) > > at > > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) > > at > > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) > > at > > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) > > at > > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) > > at > > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) > > at > > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at > > org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) > > at > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at brave.servlet.TracingFilter.doFilter(TracingFilter.java:68) > > at > > org.springframework.cloud.sleuth.instrument.web.LazyTracingFilter.doFilter(TraceWebServletAutoConfiguration.java:141) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at > > org.springframework.web.filter.ForwardedHeaderFilter.doFilterInternal(ForwardedHeaderFilter.java:156) > > at > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) > > at > > org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:711) > > at > > org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461) > > at > > org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:385) > > at > > org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:313) > > at > > org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:403) > > at > > org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:249) > > at > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) > > at > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) > > at > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) > > at > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) > > at > > org.apache.coyote.http2.StreamProcessor.service(StreamProcessor.java:426) > > at > > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) > > at > > org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:87) > > at org.apache.coyote.http2.StreamRunnable.run(StreamRunnable.java:35) > > at > > org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) > > at > > org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) > > at > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > > at java.base/java.lang.Thread.run(Unknown Source) > > Caused by: org.apache.coyote.CloseNowException: Connection [0], Stream > > [373], This stream is not writable > > at org.apache.coyote.http2.Stream.doStreamCancel(Stream.java:257) > > at > > org.apache.coyote.http2.Http2UpgradeHandler.reserveWindowSize(Http2UpgradeHandler.java:892) > > at > > org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:940) > > at > > org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:886) > > at > > org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:1009) > > at > > org.apache.coyote.http2.Http2OutputBuffer.flush(Http2OutputBuffer.java:77) > > at > > org.apache.coyote.http2.StreamProcessor.flush(StreamProcessor.java:254) > > at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:402) > > at org.apache.coyote.Response.action(Response.java:209) > > at > > org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:306) > > ... 62 common frames omitted > > Caused by: org.apache.coyote.http2.StreamException: Connection [0], Stream > > [373], This stream is not writable > > at org.apache.coyote.http2.Stream.doStreamCancel(Stream.java:249) > > ... 71 common frames omitted
Sorry, I think I posted the wrong exceptions. This is the related one: > Resolved [org.springframework.http.converter.HttpMessageNotReadableException: > I/O error while reading input message; nested exception is > org.apache.catalina.connector.ClientAbortException: > org.apache.coyote.CloseNowException: Timeout waiting to read data from client] --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
