https://bz.apache.org/bugzilla/show_bug.cgi?id=64574
Bug ID: 64574 Summary: Some HTTP/1.1 responses are broken (missing headers, empty chunks) Product: Tomcat 8 Version: 8.5.55 Hardware: PC OS: Linux Status: NEW Severity: normal Priority: P2 Component: Connectors Assignee: dev@tomcat.apache.org Reporter: martin.bon...@prime-sign.com Target Milestone: ---- We are experiencing a very strange behaviour with our application running on Tomcat 8.5.55, since we changed our frontend reverse proxy from Nginx to Traefik. Traefik access logs show HTTP result code 500 for responses from Tomcat where Tomcat was reporting them with HTTP 200 in its own logs. This was only concerning a few requests per day on our production servers and we where not able to reproduce the effect on test systems. So we started a network capture on the production servers and where able to capture a few broken responses from Tomcat. It turns out, that Tomcat is producing malformed HTTP responses every now and then. The HTTP headers are missing and the response immediately starts with a chunked response: GET / HTTP/1.1 Host: *** User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36 Accept: image/webp,image/apng,image/*,*/*;q=0.8 Accept-Encoding: gzip, deflate, br Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7 Cookie: JSESSIONID=**** Referer: https://*** Sec-Fetch-Dest: image Sec-Fetch-Mode: no-cors Sec-Fetch-Site: same-origin X-Forwarded-For: *** X-Forwarded-Host: *** X-Forwarded-Port: 443 X-Forwarded-Proto: https X-Forwarded-Server: *** X-Real-Ip: *** 741 <!DOCTYPE HTML> <html> <head> ... We also captured responses where only an empty chunk and no headers were in the response. Since the issue first appeared with the change from Nginx to Traefik, the most obvious change was that Nginx is using HTTP/1.0 in its requests to Tomcat and Traefik is using HTTP/1.1. So we where suspecting keep-alive and pipelining to trigger the issue. We disabled keep-alive in Traefik and still had the same issues. We found only one bug report on Stackoverflow describing a similar issue: https://stackoverflow.com/questions/53496598/tomcat-occasionally-returns-a-response-without-http-headers. (In the bug report Nginx was used with HTTP/1.1.) But the proposed solution was not appropriate for our situation. Code reading and further experimenting showed, that setting the system property org.apache.catalina.connector.RECYCLE_FACADES to 'true' is preventing the issue. The property name is actually misleading since setting it to 'true' is effectively disabling recycling of facade objects: https://github.com/apache/tomcat/blob/8.5.x/java/org/apache/catalina/connector/Connector.java#L171 It seems when facades (especially response' OutputSteam) is recycled there may be some kind of concurrency issue or the code obtaining the OutputStream from the response is not flushing or closing it correctly before the same OutputStream is handed over to another thread in processing the next request. We have not yet fully understand what is actually triggering the issue - it may also be in the framework and our own code processing the request - however, we would like to report for other users experiencing the same issue. Also we are wondering, if recycling the OutputStream has any relevant performance advantage or for what other reason it is done by default. Since it may cause issues not easy to track down, it may be a better idea to disable it by default. Interestingly it only causes broken responses when HTTP/1.1 is used, but there are differences in the request processing in HTTP11Processor depending on the protocol used. -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org