Re: POST request fails if content is ignored?
Very grateful for your reply, this does indeed solve my issue (and I learned something new too). Thanks, Leon On 10/09/2019 12:03, Mark Thomas wrote: > On 09/09/2019 16:41, Leon Atherton wrote: >> Our use case is rejecting the request based on IP. >> >> In the browser the status code is 0, and the network tab in developer >> tools is showing no response to the request. It's the same in Chrome and >> Firefox. >> >> The request works fine when I send from Node.JS. >> >> It seems to me that Tomcat responds to the request before the upload has >> completed, and calling request.getParameter() fixes the problem because >> it causes Tomcat to read the full request before the response is sent. >> >> Some clients are fine with the early response (e.g. Node.JS), but both >> Chrome and Firefox don't like it. >> >> I'm not sure if it's an issue with how Tomcat handles the request, or >> how the browsers are handling the response (but I suspect it can be >> fixed on the Tomcat side as the problem does not occur with Payara). > The configuration attribute you want is maxSwallowSize. You need to set > that to greater than the size of the uploaded file. > > Clients could handle this better but many don't read the response until > the request is fully written. > > Tomcat limit's maxSwallowSize as a DoS protection measure. Without it, a > client could just continue uploading a slow trickle of data and tie up a > server thread. > > For the record, maPostSize applies *only* to requests using > application/x-www-form-urlencoded > > The test provided by the OP uses multipart/form-data. The applicable > limits are defined by javax.servlet.annotation.MultipartConfig and the > defaults are unlimited. > > Any call to getPart(), getParameter() and friends will trigger the > reading of the request body. > > Hence, without the call to getParameter() Tomcat doesn't read the > request body. With small uploads there is enough network buffering > between the client and the server for the client to be able to write the > full request so it reads the response. (Tomcat's maxSwallowSize > effectively acts as a buffer.) With larger uploads the client fills the > buffers before the request is fully written so it never sees the response. > > Increasing the maxSwallowSize will allow the client to write the full > request and then read the response. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: POST request fails if content is ignored?
On 09/09/2019 16:41, Leon Atherton wrote: > Our use case is rejecting the request based on IP. > > In the browser the status code is 0, and the network tab in developer > tools is showing no response to the request. It's the same in Chrome and > Firefox. > > The request works fine when I send from Node.JS. > > It seems to me that Tomcat responds to the request before the upload has > completed, and calling request.getParameter() fixes the problem because > it causes Tomcat to read the full request before the response is sent. > > Some clients are fine with the early response (e.g. Node.JS), but both > Chrome and Firefox don't like it. > > I'm not sure if it's an issue with how Tomcat handles the request, or > how the browsers are handling the response (but I suspect it can be > fixed on the Tomcat side as the problem does not occur with Payara). The configuration attribute you want is maxSwallowSize. You need to set that to greater than the size of the uploaded file. Clients could handle this better but many don't read the response until the request is fully written. Tomcat limit's maxSwallowSize as a DoS protection measure. Without it, a client could just continue uploading a slow trickle of data and tie up a server thread. For the record, maPostSize applies *only* to requests using application/x-www-form-urlencoded The test provided by the OP uses multipart/form-data. The applicable limits are defined by javax.servlet.annotation.MultipartConfig and the defaults are unlimited. Any call to getPart(), getParameter() and friends will trigger the reading of the request body. Hence, without the call to getParameter() Tomcat doesn't read the request body. With small uploads there is enough network buffering between the client and the server for the client to be able to write the full request so it reads the response. (Tomcat's maxSwallowSize effectively acts as a buffer.) With larger uploads the client fills the buffers before the request is fully written so it never sees the response. Increasing the maxSwallowSize will allow the client to write the full request and then read the response. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: POST request fails if content is ignored?
Our use case is rejecting the request based on IP. In the browser the status code is 0, and the network tab in developer tools is showing no response to the request. It's the same in Chrome and Firefox. The request works fine when I send from Node.JS. It seems to me that Tomcat responds to the request before the upload has completed, and calling request.getParameter() fixes the problem because it causes Tomcat to read the full request before the response is sent. Some clients are fine with the early response (e.g. Node.JS), but both Chrome and Firefox don't like it. I'm not sure if it's an issue with how Tomcat handles the request, or how the browsers are handling the response (but I suspect it can be fixed on the Tomcat side as the problem does not occur with Payara). Thanks On 09/09/2019 15:48, André Warnier (tomcat) wrote: > On 09.09.2019 15:21, Leon Atherton wrote: >> Thank you for the suggestion. >> >> I have just tried playing with this value. Setting it to -1, and setting >> it to 100x larger than the default. >> In both cases, the behaviour seems unchanged. >> >> Without touching this value, Tomcat will accept multipart POST requests >> much larger than 2MB (which is the default) as long as you call >> request.getPart() or request.getParameter(). > > I haven't tried that, but as per the documentation that at least looks > wrong. > >> >> The problem I am seeing is that if a response is sent without calling >> one of those methods, the browsers see it as a failed request. > > To help me (or us) better understand the issue, could you provide a > bit of context ? > Such as : why would the client send content in a POST, if the service > at the end of the target URL is not going to read that content ? > What would be a use case ? > > (Mainly because my interpretation is that, if the browser does that, > then at least in some way it /is/ a bad request, which /should/ fail). > > And maybe another question : have you tried monitoring such a > request/response using a browser-based tracing tool, to show exactly > what the server is sending back ? > (is it e.g. some kind of 4xx response ?) > > >> >> Thanks >> >> On 09/09/2019 12:59, André Warnier (tomcat) wrote: >>> Hi. >>> Did you check : >>> https://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Common_Attributes >>> >>> >>> --> maxPostSize >>> >>> Note : normally, the browser will encode (Base64 or similar) the >>> content of a file and send the encoded content, which tends to be >>> significantly larger (in bytes) than the original file (say + 30%). I >>> do not know (and the doc does not say) if the maxPostSize attribute >>> refers to the POST content still encoded or already decoded. >>> >>> >>> On 09.09.2019 12:53, Leon Atherton wrote: Hello, I've discovered an interesting issue where POST requests fail when uploading a file over about ~6MB if the server ignores the request content. I've put together a simple project to reproduce it: https://github.com/leonatherton/tomcat-request-issue Serverside code: https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/java/DemoServlet.java Clientside code: https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/webapp/index.html If you'd like to try it yourself there are a couple of pre-built war files on the releases page, and there are steps to reproduce in the README.md file. The problem does not occur when uploading a small file, and the problem can be "fixed" by simply getting a parameter from the request object. It reproduces in Tomcat 8 & 9. The problem does not reproduce on Payara, but I am seeing similar on Jetty. My guess is that the server responds before the client has finished uploading the file. The browsers see the incomplete upload and report this as an error, despite content being sent in response. And my guess is that inspecting a request parameter causes the server to wait for the full upload before sending the response. It's a slightly odd workflow, but it's not too unreasonable to sometimes respond early and ignore the request content. Is this expected behavior, or a bug in Tomcat? Thanks! Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> >>> - >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail:
Re: POST request fails if content is ignored?
On 09.09.2019 15:21, Leon Atherton wrote: Thank you for the suggestion. I have just tried playing with this value. Setting it to -1, and setting it to 100x larger than the default. In both cases, the behaviour seems unchanged. Without touching this value, Tomcat will accept multipart POST requests much larger than 2MB (which is the default) as long as you call request.getPart() or request.getParameter(). I haven't tried that, but as per the documentation that at least looks wrong. The problem I am seeing is that if a response is sent without calling one of those methods, the browsers see it as a failed request. To help me (or us) better understand the issue, could you provide a bit of context ? Such as : why would the client send content in a POST, if the service at the end of the target URL is not going to read that content ? What would be a use case ? (Mainly because my interpretation is that, if the browser does that, then at least in some way it /is/ a bad request, which /should/ fail). And maybe another question : have you tried monitoring such a request/response using a browser-based tracing tool, to show exactly what the server is sending back ? (is it e.g. some kind of 4xx response ?) Thanks On 09/09/2019 12:59, André Warnier (tomcat) wrote: Hi. Did you check : https://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Common_Attributes --> maxPostSize Note : normally, the browser will encode (Base64 or similar) the content of a file and send the encoded content, which tends to be significantly larger (in bytes) than the original file (say + 30%). I do not know (and the doc does not say) if the maxPostSize attribute refers to the POST content still encoded or already decoded. On 09.09.2019 12:53, Leon Atherton wrote: Hello, I've discovered an interesting issue where POST requests fail when uploading a file over about ~6MB if the server ignores the request content. I've put together a simple project to reproduce it: https://github.com/leonatherton/tomcat-request-issue Serverside code: https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/java/DemoServlet.java Clientside code: https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/webapp/index.html If you'd like to try it yourself there are a couple of pre-built war files on the releases page, and there are steps to reproduce in the README.md file. The problem does not occur when uploading a small file, and the problem can be "fixed" by simply getting a parameter from the request object. It reproduces in Tomcat 8 & 9. The problem does not reproduce on Payara, but I am seeing similar on Jetty. My guess is that the server responds before the client has finished uploading the file. The browsers see the incomplete upload and report this as an error, despite content being sent in response. And my guess is that inspecting a request parameter causes the server to wait for the full upload before sending the response. It's a slightly odd workflow, but it's not too unreasonable to sometimes respond early and ignore the request content. Is this expected behavior, or a bug in Tomcat? Thanks! Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: POST request fails if content is ignored?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 9/9/19 07:59, André Warnier (tomcat) wrote: > Hi. Did you check : > https://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Common_Attri butes > > - --> maxPostSize > > Note : normally, the browser will encode (Base64 or similar) the > content of a file and send the encoded content, which tends to be > significantly larger (in bytes) than the original file (say + > 30%). I know of no browser that encodes a file in any way when performing a standard file-upload via an control. HTTP handled binary content just fine. > I do not know (and the doc does not say) if the maxPostSize > attribute refers to the POST content still encoded or already > decoded. That setting counts bytes on the wire. - -chris > On 09.09.2019 12:53, Leon Atherton wrote: >> Hello, >> >> I've discovered an interesting issue where POST requests fail >> when uploading a file over about ~6MB if the server ignores the >> request content. >> >> I've put together a simple project to reproduce it: >> https://github.com/leonatherton/tomcat-request-issue >> >> Serverside code: >> https://github.com/leonatherton/tomcat-request-issue/blob/master/src/ main/java/DemoServlet.java >> >> >> Clientside code: >> https://github.com/leonatherton/tomcat-request-issue/blob/master/src/ main/webapp/index.html >> >> >> >> If you'd like to try it yourself there are a couple of pre-built war >> files on the releases page, and there are steps to reproduce in >> the README.md file. >> >> The problem does not occur when uploading a small file, and the >> problem can be "fixed" by simply getting a parameter from the >> request object. It reproduces in Tomcat 8 & 9. The problem does >> not reproduce on Payara, but I am seeing similar on Jetty. >> >> My guess is that the server responds before the client has >> finished uploading the file. The browsers see the incomplete >> upload and report this as an error, despite content being sent in >> response. And my guess is that inspecting a request parameter >> causes the server to wait for the full upload before sending the >> response. >> >> It's a slightly odd workflow, but it's not too unreasonable to >> sometimes respond early and ignore the request content. >> >> Is this expected behavior, or a bug in Tomcat? >> >> Thanks! Leon >> >> - >> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl12We8ACgkQHPApP6U8 pFhbsxAAta8C4RjgQ7DPpoZ/gtE9M6oFfICUeX1gRCB4YYu6nWY4O+8r/CK+hGHV yuWUchWQaeTG+sSIrXL2tvaIbQEIUzLLEDd6H/nSbCIL/rOF3AXs4AbGfsyLcUji L7c9G334aBWQoGS6TfYrXAHZfI7YmtPqc4rXgQqJwRkOCGg0oLL8e7YKLyffPb2U BFnSaMFaXHIPIoKEC3lLl39r6fCikpgx/MJ7wqLrU3Oux6tr3srcAFFlnhGpmUG9 vcoXULMxo890gd4uspyfbV5OL5BektFLG4u5t1wu4vVdC+G/+l0uTd1i3J9w6+cM GbBMjeIH1g87/bb+rvrszMGn5XEcEs3i1UwZIy0UNY0oVcGJkV4V5iu2e5VVCp8k ZcrG56bCptKL9CkTQafhPeztfhRPJya4UVUm9MeVKu1mQsKWZrya2840hDeKzv9V FhTsqLlS0GeUn3dFpKItQ5ngH4jpT3Lm4VQ2hrTt4h529YpgD0FMk8i+BdeuFnCF hsHbTxJLnmw8fZ+H5mGLNzB4DTWVVJVTmCa0xcI/zO2S6Arp8q1jyxiSOurlTNZ6 pD0Y45NwTpwtwWhjhgWNQilLHtc1Nc3KOMcERX+0++cqJWrM2bxMcm/YQ/QK9yXA nPWD8BMelyL9MhdvyjeGKoH6zqQ0uAhT6/iYFDqOw3gTMOtwEkk= =1NdL -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: POST request fails if content is ignored?
Thank you for the suggestion. I have just tried playing with this value. Setting it to -1, and setting it to 100x larger than the default. In both cases, the behaviour seems unchanged. Without touching this value, Tomcat will accept multipart POST requests much larger than 2MB (which is the default) as long as you call request.getPart() or request.getParameter(). The problem I am seeing is that if a response is sent without calling one of those methods, the browsers see it as a failed request. Thanks On 09/09/2019 12:59, André Warnier (tomcat) wrote: > Hi. > Did you check : > https://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Common_Attributes > > --> maxPostSize > > Note : normally, the browser will encode (Base64 or similar) the > content of a file and send the encoded content, which tends to be > significantly larger (in bytes) than the original file (say + 30%). I > do not know (and the doc does not say) if the maxPostSize attribute > refers to the POST content still encoded or already decoded. > > > On 09.09.2019 12:53, Leon Atherton wrote: >> Hello, >> >> I've discovered an interesting issue where POST requests fail when >> uploading a file over about ~6MB if the server ignores the request >> content. >> >> I've put together a simple project to reproduce it: >> https://github.com/leonatherton/tomcat-request-issue >> >> Serverside code: >> https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/java/DemoServlet.java >> >> >> Clientside code: >> https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/webapp/index.html >> >> >> >> If you'd like to try it yourself there are a couple of pre-built war >> files on the releases page, and there are steps to reproduce in the >> README.md file. >> >> The problem does not occur when uploading a small file, and the problem >> can be "fixed" by simply getting a parameter from the request object. It >> reproduces in Tomcat 8 & 9. The problem does not reproduce on Payara, >> but I am seeing similar on Jetty. >> >> My guess is that the server responds before the client has finished >> uploading the file. The browsers see the incomplete upload and report >> this as an error, despite content being sent in response. And my guess >> is that inspecting a request parameter causes the server to wait for the >> full upload before sending the response. >> >> It's a slightly odd workflow, but it's not too unreasonable to sometimes >> respond early and ignore the request content. >> >> Is this expected behavior, or a bug in Tomcat? >> >> Thanks! >> Leon >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: POST request fails if content is ignored?
Hi. Did you check : https://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Common_Attributes --> maxPostSize Note : normally, the browser will encode (Base64 or similar) the content of a file and send the encoded content, which tends to be significantly larger (in bytes) than the original file (say + 30%). I do not know (and the doc does not say) if the maxPostSize attribute refers to the POST content still encoded or already decoded. On 09.09.2019 12:53, Leon Atherton wrote: Hello, I've discovered an interesting issue where POST requests fail when uploading a file over about ~6MB if the server ignores the request content. I've put together a simple project to reproduce it: https://github.com/leonatherton/tomcat-request-issue Serverside code: https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/java/DemoServlet.java Clientside code: https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/webapp/index.html If you'd like to try it yourself there are a couple of pre-built war files on the releases page, and there are steps to reproduce in the README.md file. The problem does not occur when uploading a small file, and the problem can be "fixed" by simply getting a parameter from the request object. It reproduces in Tomcat 8 & 9. The problem does not reproduce on Payara, but I am seeing similar on Jetty. My guess is that the server responds before the client has finished uploading the file. The browsers see the incomplete upload and report this as an error, despite content being sent in response. And my guess is that inspecting a request parameter causes the server to wait for the full upload before sending the response. It's a slightly odd workflow, but it's not too unreasonable to sometimes respond early and ignore the request content. Is this expected behavior, or a bug in Tomcat? Thanks! Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
POST request fails if content is ignored?
Hello, I've discovered an interesting issue where POST requests fail when uploading a file over about ~6MB if the server ignores the request content. I've put together a simple project to reproduce it: https://github.com/leonatherton/tomcat-request-issue Serverside code: https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/java/DemoServlet.java Clientside code: https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/webapp/index.html If you'd like to try it yourself there are a couple of pre-built war files on the releases page, and there are steps to reproduce in the README.md file. The problem does not occur when uploading a small file, and the problem can be "fixed" by simply getting a parameter from the request object. It reproduces in Tomcat 8 & 9. The problem does not reproduce on Payara, but I am seeing similar on Jetty. My guess is that the server responds before the client has finished uploading the file. The browsers see the incomplete upload and report this as an error, despite content being sent in response. And my guess is that inspecting a request parameter causes the server to wait for the full upload before sending the response. It's a slightly odd workflow, but it's not too unreasonable to sometimes respond early and ignore the request content. Is this expected behavior, or a bug in Tomcat? Thanks! Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org