Re: Tomcat occasionally returns a response without HTTP headers
On 16/12/2018 03:01, Kohei Nozaki wrote:
> Hello,
>
> It turned out that the "sjsxp" library which JAX-WS RI v2.1.3 uses makes
> Tomcat behave this way. I tried a different version of JAX-WS RI (v2.1.7)
> which doesn't use the "sjsxp" library anymore and it solved the issue.
Thanks for reporting back when you found a solution.
> A very similar issue posted on the Metro mailing list:
> http://metro.1045641.n5.nabble.com/JAX-WS-RI-2-1-5-returning-malformed-response-tp1063518.html
>
> It's surprising that a bug of a framework which is built on the Servlet API
> can make such an issue happen, but anyway thank you very much everyone who
> helped me out.
I'm guessing it held onto a reference and re-used it when it shouldn't.
That can cause all sorts of chaos.
Mark
>
> Regards,
> Kohei
>
>> On Dec 4, 2018, at 3:39, Christopher Schultz
>> wrote:
>>
> Kohei,
>
> On 12/1/18 10:06, Kohei Nozaki wrote:
Hello Konstantin, thanks for sharing the valuable information.
> On Dec 1, 2018, at 19:00, Konstantin Kolinko
> wrote:
>
>> * Our downstream Nginx instance (The client of our Tomcat
>> instance) recorded the error "upstream sent no valid HTTP/1.0
>> header while reading response header from upstream" at that
>> time and the error makes perfect sense concerning the response
>> which has neither HTTP status line nor HTTP headers.
>
> 1. See the official FAQ / Troubleshoting page:
> https://wiki.apache.org/tomcat/FAQ/Troubleshooting_and_Diagnostics
>
>
>
> Especially pay attention to
> 1) configuring an access log 2) setting system property
> org.apache.catalina.connector.RECYCLE_FACADES=true
I've investigated "RECYCLE_FACADES" and understand that an
improperly implemented application which keeps a reference to
Request or Response objects outside of their lifecycle can make
such an issue like mine happen (please correct me if I'm wrong..).
But I still don't quite understand what does "RECYCLE_FACADES=true"
do. The Wiki page says "This makes it easier to spot illegal access
when it happens, instead of waiting until side effects of such
access become visible" but how does it make easier? Does this
property make Tomcat produce an Exception or make Tomcat produce
some warning message to a log file or something when such accesses
happen, for example?
>
> Tomcat usually handles requests something like this. Imagine a
> single-threaded server where Tomcat only accepts a single connection
> at a time (just to simplify the code to the point where it fits into a
> ML post).
>
> Many of these methods are made-up. There is no
> TomcatHttpServletRequest class or a .setRequestLine method in it
> (though there are ... siilar concepts in there, way down deep). The
> point is how the objects are used, or rather *re* used.
>
> HttpServletRequest request = new TomcatHttpServletRequest();
> HttpServletResponse response = new TomcatHttpServletResponse();
>
> Connection conn = null;
>
> while(null != (conn = socket.acceptConnection()) {
>request.setRequestLine(conn.getRequestLine());
>request.setInputStream(conn.getInputStream());
>response.setOutputStream(conn.getOutputStream());
>
>Servlet servlet = getServletForRequest(request);
>if(null == servlet)
>servlet = defaultServlet;
>
>servlet.service(request, response);
>
>request.reset();
>response.reset();
> }
>
> In "real" Tomcat, each Connection object holds its own Request and
> Response objects ad manages them in a similar way, and of course,
> Tomcat can accept multiple simultaneous connections -- including
> multiple requests over a single connection -- simultaneously -- in the
> case of HTTP/2.
>
> If you enable the RECYCLE_FACADES in Tomcat, the code changes to
> behave like this:
>
> Connection conn = null;
>
> while(null != (conn = socket.acceptConnection()) {
>HttpServletRequest request = new TomcatHttpServletRequest();
>HttpServletResponse response = new TomcatHttpServletResponse();
>request.setRequestLine(conn.getRequestLine());
>request.setInputStream(conn.getInputStream());
>response.setOutputStream(conn.getOutputStream());
>
>Servlet servlet = getServletForRequest(request);
>if(null == servlet)
>servlet = defaultServlet;
>
>servlet.service(request, response);
>
>request.dispose();
>response.dispose();
> }
>
> Note how the request and response objects are no longer re-used across
> requests. This represents a trade-off between security/stability
> (always getting a fresh object) versus performance (less
> garbage-collection for a given request). An application can do things
> to the request or response that can break the way the server works, so
> an untrusted application should always be run with RECYCLE_FACADES set
> to ON.
>
> If the application keeps a reference to a request or response object
> after the
Re: Tomcat occasionally returns a response without HTTP headers
Hello,
It turned out that the "sjsxp" library which JAX-WS RI v2.1.3 uses makes Tomcat
behave this way. I tried a different version of JAX-WS RI (v2.1.7) which
doesn't use the "sjsxp" library anymore and it solved the issue.
A very similar issue posted on the Metro mailing list:
http://metro.1045641.n5.nabble.com/JAX-WS-RI-2-1-5-returning-malformed-response-tp1063518.html
It's surprising that a bug of a framework which is built on the Servlet API can
make such an issue happen, but anyway thank you very much everyone who helped
me out.
Regards,
Kohei
> On Dec 4, 2018, at 3:39, Christopher Schultz
> wrote:
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Kohei,
>
> On 12/1/18 10:06, Kohei Nozaki wrote:
>> Hello Konstantin, thanks for sharing the valuable information.
>>
>>> On Dec 1, 2018, at 19:00, Konstantin Kolinko
>>> wrote:
>>>
* Our downstream Nginx instance (The client of our Tomcat
instance) recorded the error "upstream sent no valid HTTP/1.0
header while reading response header from upstream" at that
time and the error makes perfect sense concerning the response
which has neither HTTP status line nor HTTP headers.
>>>
>>> 1. See the official FAQ / Troubleshoting page:
>>> https://wiki.apache.org/tomcat/FAQ/Troubleshooting_and_Diagnostics
>>>
>>>
>>>
> Especially pay attention to
>>> 1) configuring an access log 2) setting system property
>>> org.apache.catalina.connector.RECYCLE_FACADES=true
>>
>> I've investigated "RECYCLE_FACADES" and understand that an
>> improperly implemented application which keeps a reference to
>> Request or Response objects outside of their lifecycle can make
>> such an issue like mine happen (please correct me if I'm wrong..).
>>
>>
>> But I still don't quite understand what does "RECYCLE_FACADES=true"
>> do. The Wiki page says "This makes it easier to spot illegal access
>> when it happens, instead of waiting until side effects of such
>> access become visible" but how does it make easier? Does this
>> property make Tomcat produce an Exception or make Tomcat produce
>> some warning message to a log file or something when such accesses
>> happen, for example?
>
> Tomcat usually handles requests something like this. Imagine a
> single-threaded server where Tomcat only accepts a single connection
> at a time (just to simplify the code to the point where it fits into a
> ML post).
>
> Many of these methods are made-up. There is no
> TomcatHttpServletRequest class or a .setRequestLine method in it
> (though there are ... siilar concepts in there, way down deep). The
> point is how the objects are used, or rather *re* used.
>
> HttpServletRequest request = new TomcatHttpServletRequest();
> HttpServletResponse response = new TomcatHttpServletResponse();
>
> Connection conn = null;
>
> while(null != (conn = socket.acceptConnection()) {
>request.setRequestLine(conn.getRequestLine());
>request.setInputStream(conn.getInputStream());
>response.setOutputStream(conn.getOutputStream());
>
>Servlet servlet = getServletForRequest(request);
>if(null == servlet)
>servlet = defaultServlet;
>
>servlet.service(request, response);
>
>request.reset();
>response.reset();
> }
>
> In "real" Tomcat, each Connection object holds its own Request and
> Response objects ad manages them in a similar way, and of course,
> Tomcat can accept multiple simultaneous connections -- including
> multiple requests over a single connection -- simultaneously -- in the
> case of HTTP/2.
>
> If you enable the RECYCLE_FACADES in Tomcat, the code changes to
> behave like this:
>
> Connection conn = null;
>
> while(null != (conn = socket.acceptConnection()) {
>HttpServletRequest request = new TomcatHttpServletRequest();
>HttpServletResponse response = new TomcatHttpServletResponse();
>request.setRequestLine(conn.getRequestLine());
>request.setInputStream(conn.getInputStream());
>response.setOutputStream(conn.getOutputStream());
>
>Servlet servlet = getServletForRequest(request);
>if(null == servlet)
>servlet = defaultServlet;
>
>servlet.service(request, response);
>
>request.dispose();
>response.dispose();
> }
>
> Note how the request and response objects are no longer re-used across
> requests. This represents a trade-off between security/stability
> (always getting a fresh object) versus performance (less
> garbage-collection for a given request). An application can do things
> to the request or response that can break the way the server works, so
> an untrusted application should always be run with RECYCLE_FACADES set
> to ON.
>
> If the application keeps a reference to a request or response object
> after the request has completed (as defined by returning from
> servlet.service()), then Bad Things can happen. If you write to that
> response's OutputStream, for example, you might write into the
> response of *another request* that is being handled after
Re: Tomcat occasionally returns a response without HTTP headers
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Kohei,
On 12/1/18 10:06, Kohei Nozaki wrote:
> Hello Konstantin, thanks for sharing the valuable information.
>
>> On Dec 1, 2018, at 19:00, Konstantin Kolinko
>> wrote:
>>
>>> * Our downstream Nginx instance (The client of our Tomcat
>>> instance) recorded the error "upstream sent no valid HTTP/1.0
>>> header while reading response header from upstream" at that
>>> time and the error makes perfect sense concerning the response
>>> which has neither HTTP status line nor HTTP headers.
>>
>> 1. See the official FAQ / Troubleshoting page:
>> https://wiki.apache.org/tomcat/FAQ/Troubleshooting_and_Diagnostics
>>
>>
>>
Especially pay attention to
>> 1) configuring an access log 2) setting system property
>> org.apache.catalina.connector.RECYCLE_FACADES=true
>
> I've investigated "RECYCLE_FACADES" and understand that an
> improperly implemented application which keeps a reference to
> Request or Response objects outside of their lifecycle can make
> such an issue like mine happen (please correct me if I'm wrong..).
>
>
> But I still don't quite understand what does "RECYCLE_FACADES=true"
> do. The Wiki page says "This makes it easier to spot illegal access
> when it happens, instead of waiting until side effects of such
> access become visible" but how does it make easier? Does this
> property make Tomcat produce an Exception or make Tomcat produce
> some warning message to a log file or something when such accesses
> happen, for example?
Tomcat usually handles requests something like this. Imagine a
single-threaded server where Tomcat only accepts a single connection
at a time (just to simplify the code to the point where it fits into a
ML post).
Many of these methods are made-up. There is no
TomcatHttpServletRequest class or a .setRequestLine method in it
(though there are ... siilar concepts in there, way down deep). The
point is how the objects are used, or rather *re* used.
HttpServletRequest request = new TomcatHttpServletRequest();
HttpServletResponse response = new TomcatHttpServletResponse();
Connection conn = null;
while(null != (conn = socket.acceptConnection()) {
request.setRequestLine(conn.getRequestLine());
request.setInputStream(conn.getInputStream());
response.setOutputStream(conn.getOutputStream());
Servlet servlet = getServletForRequest(request);
if(null == servlet)
servlet = defaultServlet;
servlet.service(request, response);
request.reset();
response.reset();
}
In "real" Tomcat, each Connection object holds its own Request and
Response objects ad manages them in a similar way, and of course,
Tomcat can accept multiple simultaneous connections -- including
multiple requests over a single connection -- simultaneously -- in the
case of HTTP/2.
If you enable the RECYCLE_FACADES in Tomcat, the code changes to
behave like this:
Connection conn = null;
while(null != (conn = socket.acceptConnection()) {
HttpServletRequest request = new TomcatHttpServletRequest();
HttpServletResponse response = new TomcatHttpServletResponse();
request.setRequestLine(conn.getRequestLine());
request.setInputStream(conn.getInputStream());
response.setOutputStream(conn.getOutputStream());
Servlet servlet = getServletForRequest(request);
if(null == servlet)
servlet = defaultServlet;
servlet.service(request, response);
request.dispose();
response.dispose();
}
Note how the request and response objects are no longer re-used across
requests. This represents a trade-off between security/stability
(always getting a fresh object) versus performance (less
garbage-collection for a given request). An application can do things
to the request or response that can break the way the server works, so
an untrusted application should always be run with RECYCLE_FACADES set
to ON.
If the application keeps a reference to a request or response object
after the request has completed (as defined by returning from
servlet.service()), then Bad Things can happen. If you write to that
response's OutputStream, for example, you might write into the
response of *another request* that is being handled after your code
should be finished.
>
> p.s. Speaking of the possibility of an improper implementation,
> I've found some discussions which seem to have some similarity to
> my case like the following:
>
> *
> http://tomcat.10.x6.nabble.com/NullPointerException-in-MimeHeaders-td2
054107.html
> : I've seen some similar Exceptions at the "MimeHeaders" class
> which were recorded in my Tomcat's log too
>
> *
> http://tomcat.10.x6.nabble.com/Tomcat-occasionally-duplicating-respons
es-td5034710.html
> : A case where a bug in a Filter make Tomcat produce an invalid
> HTTP response
>
> Those discussions made me think of this possibility where my app
> might be improperly implemented. I'm going to check the application
> code from this perspective on Monday.
When you use RECYCLE_FACADES, your
Re: Tomcat occasionally returns a response without HTTP headers
Hello Konstantin, thanks for sharing the valuable information. > On Dec 1, 2018, at 19:00, Konstantin Kolinko wrote: > >> * Our downstream Nginx instance (The client of our Tomcat instance) recorded >> the error "upstream sent no valid HTTP/1.0 header while reading response >> header from upstream" at that time and the error makes perfect sense >> concerning the response which has neither HTTP status line nor HTTP headers. > > 1. See the official FAQ / Troubleshoting page: > https://wiki.apache.org/tomcat/FAQ/Troubleshooting_and_Diagnostics > > Especially pay attention to > 1) configuring an access log > 2) setting system property > org.apache.catalina.connector.RECYCLE_FACADES=true I've investigated "RECYCLE_FACADES" and understand that an improperly implemented application which keeps a reference to Request or Response objects outside of their lifecycle can make such an issue like mine happen (please correct me if I'm wrong..). But I still don't quite understand what does "RECYCLE_FACADES=true" do. The Wiki page says "This makes it easier to spot illegal access when it happens, instead of waiting until side effects of such access become visible" but how does it make easier? Does this property make Tomcat produce an Exception or make Tomcat produce some warning message to a log file or something when such accesses happen, for example? p.s. Speaking of the possibility of an improper implementation, I've found some discussions which seem to have some similarity to my case like the following: * http://tomcat.10.x6.nabble.com/NullPointerException-in-MimeHeaders-td2054107.html : I've seen some similar Exceptions at the "MimeHeaders" class which were recorded in my Tomcat's log too * http://tomcat.10.x6.nabble.com/Tomcat-occasionally-duplicating-responses-td5034710.html : A case where a bug in a Filter make Tomcat produce an invalid HTTP response Those discussions made me think of this possibility where my app might be improperly implemented. I'm going to check the application code from this perspective on Monday.. Best Regards, Kohei - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat occasionally returns a response without HTTP headers
сб, 1 дек. 2018 г. в 01:30, Kohei Nozaki : > > Hello Christopher, thank you for your help. > > * Our downstream Nginx instance (The client of our Tomcat instance) recorded > the error "upstream sent no valid HTTP/1.0 header while reading response > header from upstream" at that time and the error makes perfect sense > concerning the response which has neither HTTP status line nor HTTP headers. > > Speaking of a possibility of a bug, a person commented on the Stackoverflow > question and said that there might be something in the request that possibly > downgrades the connection to HTTP/0.9. Do you think it's possible? The > comments can be seen from the URL below: > > https://stackoverflow.com/questions/53496598/tomcat-occasionally-returns-a-response-without-http-headers#comment93976313_53552752 1. See the official FAQ / Troubleshoting page: https://wiki.apache.org/tomcat/FAQ/Troubleshooting_and_Diagnostics Especially pay attention to 1) configuring an access log 2) setting system property org.apache.catalina.connector.RECYCLE_FACADES=true (and Java ImageIO stream handling bug) 2. HTTP 0.9 is a valid response format (a feature, not a bug) that does not contain status line nor headers, sends just the requested document and closes the connection afterwards. The "Specifications" page in the wiki has a link to the original specification, if you are interested. But HTTP 0.9 should never mix itself with chunked encoding. (The "5d + CRLF" chunk size that you are seeing). 3. If somebody calls "out.flush()" (or response.flushBuffer()) before writing a response body, the headers and the body will be sent as separate packets and may appear separately in wireshark. It is a valid behaviour. (But your client should see the headers. It shouldn't report that headers are missing). Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat occasionally returns a response without HTTP headers
Hello Christopher, thank you for your help.
> On Dec 1, 2018, at 1:52, Christopher Schultz
> wrote:
>
>> I'm investigating a problem where Tomcat (7.0.92) returns a
>> response with no HTTP headers very occasionally. It happened during
>> a load test.
>>
>> According to the captured packets by Wireshark, after Tomcat
>> receives a request it just returns only a response body. It returns
>> neither a status line nor HTTP response headers.
>>
>> This is a screenshot of Wireshark's "Follow HTTP Stream" which is
>> showing the problematic response:
>> https://i.stack.imgur.com/E6ttG.png
>
> It's difficult to tell what's being shown, here. Too much has been
> redacted. It looks like both request and response are being shown, and
> the response appears to be part of a chunked response.
The problem here is that the HTTP status line and the HTTP response headers
which should have been at before the beginning of the chunked response ("5d")
are missing.
I'm sorry about the too much redaction but that's what I only can do due to
security reasons.. If more information is needed in order to narrow down the
problem, please tell me and I will do the best.
>> This is a screen shot of "TCP Stream" of the relevant part (only
>> response): https://i.stack.imgur.com/TLKl3.png
>
> This looks like a standard chunked response. I see no problems, there.
Actually, this screenshot shows 2 responses. There are two terminating chunk
("0" and an empty line) there. Those 2 responses were sent from a SOAP endpoint
and that's why there are 2 XML documents.
And the problem here is that the HTTP status line and the HTTP response headers
are missing in the latter response.
>> In my opinion, neither an application nor a framework can make
>> Tomcat behave this way, so I'm wondering if it's a bug or something
>> of Tomcat. Do you think it can be a bug of Tomcat or is there
>> something which can make this happen? Was there a relevant change
>> of Tomcat in recent versions?
>
> I agree that it should not be possible for an application (and a
> framework counts as "the application" in this sense) to break Tomcat's
> handling of the HTTP spec, but I don't see direct evidence of a bug,
> yet, here.
>
> Are your samples (shown in your images) definitely showing only a
> single TCP connection, or is it possible that wireshark is merging the
> requests/responses of multiple connections together in the same log
> stream?
I believe my samples are showing only one single TCP connection because:
* They are filtered by a "tcp.stream eq xxx" condition as shown in the title
bars in the screen shots
* Our downstream Nginx instance (The client of our Tomcat instance) recorded
the error "upstream sent no valid HTTP/1.0 header while reading response header
from upstream" at that time and the error makes perfect sense concerning the
response which has neither HTTP status line nor HTTP headers.
Speaking of a possibility of a bug, a person commented on the Stackoverflow
question and said that there might be something in the request that possibly
downgrades the connection to HTTP/0.9. Do you think it's possible? The comments
can be seen from the URL below:
https://stackoverflow.com/questions/53496598/tomcat-occasionally-returns-a-response-without-http-headers#comment93976313_53552752
Best regards,
Kohei
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat occasionally returns a response without HTTP headers
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Kohei, On 11/29/18 20:19, Kohei Nozaki wrote: > I'm investigating a problem where Tomcat (7.0.92) returns a > response with no HTTP headers very occasionally. It happened during > a load test. > > > According to the captured packets by Wireshark, after Tomcat > receives a request it just returns only a response body. It returns > neither a status line nor HTTP response headers. > > > This is a screenshot of Wireshark's "Follow HTTP Stream" which is > showing the problematic response: > https://i.stack.imgur.com/E6ttG.png It's difficult to tell what's being shown, here. Too much has been redacted. It looks like both request and response are being shown, and the response appears to be part of a chunked response. > This is a screen shot of "TCP Stream" of the relevant part (only > response): https://i.stack.imgur.com/TLKl3.png This looks like a standard chunked response. I see no problems, there. > In my opinion, neither an application nor a framework can make > Tomcat behave this way, so I'm wondering if it's a bug or something > of Tomcat. Do you think it can be a bug of Tomcat or is there > something which can make this happen? Was there a relevant change > of Tomcat in recent versions? I agree that it should not be possible for an application (and a framework counts as "the application" in this sense) to break Tomcat's handling of the HTTP spec, but I don't see direct evidence of a bug, yet, here. Are your samples (shown in your images) definitely showing only a single TCP connection, or is it possible that wireshark is merging the requests/responses of multiple connections together in the same log stream? - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlwBarQACgkQHPApP6U8 pFisaw/9G3l7InffOzzOQcTLWE7b+38dFKxrB0ejpLqoG70C1w2VgwwSGBfzrpxA ucpA+SJeyLujI+NLyeANtB01uY/B9Ba3f0XgdQGOSl+RcT4vOQMs5qGtfS8ImitE slCjiogr077Lp/sKMXSCkSDzEvHaJ/8jIITqGElo05q4HZqSHsHAan4crndP0lE6 9mIHfN477NmOhoHHg+HwuSzbgGbawPlNwjkFLZvQlQQIBUEc00DyGFC+avnie5nf 0z40UFULlR+rzsdXYsrfmOtzYpAV1lzksxDx7RqPbKjrOdqNRzeCkcdE2yzoi8CF rzacG2anhxWOuuE5zKoFzcxxAv2uNDnl9sDfIcNkl00D5pBBeGAQ6fib27oDNtJ/ zZW9UsbwiiUzDOnpzNTFLNzf/TTF3NuHNpVXyVq9rBXwdqzqCQXHaI7zOSyY6NDr 6V09eP2xwsXlaDMXp7hl8hea3Ci0gIUXFm8wTq8LBjNtteoWrJ4vkx5qoVFhzORe D0TgmZgwtablmzb76hLYMJP1t6gsnx0nbcob1YcxAx9k1pCTKEiRsdFHYO1fH0KE 8TpR4q/3xmINNhU2vb8SefvoTXIvOcI8LJpy6u9YuTAIiTlmwUQouqtdZ99U9EjN EdG5eUVv2XRJvlKLRovIOAoJl1LL7dDWO1II6hRdBbqulmeQXHE= =eDT+ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat occasionally returns a response without HTTP headers
Hello, I'm investigating a problem where Tomcat (7.0.92) returns a response with no HTTP headers very occasionally. It happened during a load test. According to the captured packets by Wireshark, after Tomcat receives a request it just returns only a response body. It returns neither a status line nor HTTP response headers. This is a screenshot of Wireshark's "Follow HTTP Stream" which is showing the problematic response: https://i.stack.imgur.com/E6ttG.png This is a screen shot of "TCP Stream" of the relevant part (only response): https://i.stack.imgur.com/TLKl3.png In my opinion, neither an application nor a framework can make Tomcat behave this way, so I'm wondering if it's a bug or something of Tomcat. Do you think it can be a bug of Tomcat or is there something which can make this happen? Was there a relevant change of Tomcat in recent versions? This question was originally posted on Stackoverflow. There are some discussions and a bit of more information there: https://stackoverflow.com/q/53496598/3591946 Thanks, Kohei
