Re: POST request fails if content is ignored?

2019-09-10 Thread Leon Atherton
Very grateful for your reply, this does indeed solve my issue (and I 
learned something new too).

Thanks,
Leon

On 10/09/2019 12:03, Mark Thomas wrote:
> On 09/09/2019 16:41, Leon Atherton wrote:
>> Our use case is rejecting the request based on IP.
>>
>> In the browser the status code is 0, and the network tab in developer
>> tools is showing no response to the request. It's the same in Chrome and
>> Firefox.
>>
>> The request works fine when I send from Node.JS.
>>
>> It seems to me that Tomcat responds to the request before the upload has
>> completed, and calling request.getParameter() fixes the problem because
>> it causes Tomcat to read the full request before the response is sent.
>>
>> Some clients are fine with the early response (e.g. Node.JS), but both
>> Chrome and Firefox don't like it.
>>
>> I'm not sure if it's an issue with how Tomcat handles the request, or
>> how the browsers are handling the response (but I suspect it can be
>> fixed on the Tomcat side as the problem does not occur with Payara).
> The configuration attribute you want is maxSwallowSize. You need to set
> that to greater than the size of the uploaded file.
>
> Clients could handle this better but many don't read the response until
> the request is fully written.
>
> Tomcat limit's maxSwallowSize as a DoS protection measure. Without it, a
> client could just continue uploading a slow trickle of data and tie up a
> server thread.
>
> For the record, maPostSize applies *only* to requests using
> application/x-www-form-urlencoded
>
> The test provided by the OP uses multipart/form-data. The applicable
> limits are defined by javax.servlet.annotation.MultipartConfig and the
> defaults are unlimited.
>
> Any call to getPart(), getParameter() and friends will trigger the
> reading of the request body.
>
> Hence, without the call to getParameter() Tomcat doesn't read the
> request body. With small uploads there is enough network buffering
> between the client and the server for the client to be able to write the
> full request so it reads the response. (Tomcat's maxSwallowSize
> effectively acts as a buffer.) With larger uploads the client fills the
> buffers before the request is fully written so it never sees the response.
>
> Increasing the maxSwallowSize will allow the client to write the full
> request and then read the response.
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: POST request fails if content is ignored?

2019-09-10 Thread Mark Thomas
On 09/09/2019 16:41, Leon Atherton wrote:
> Our use case is rejecting the request based on IP.
> 
> In the browser the status code is 0, and the network tab in developer 
> tools is showing no response to the request. It's the same in Chrome and 
> Firefox.
> 
> The request works fine when I send from Node.JS.
> 
> It seems to me that Tomcat responds to the request before the upload has 
> completed, and calling request.getParameter() fixes the problem because 
> it causes Tomcat to read the full request before the response is sent.
> 
> Some clients are fine with the early response (e.g. Node.JS), but both 
> Chrome and Firefox don't like it.
> 
> I'm not sure if it's an issue with how Tomcat handles the request, or 
> how the browsers are handling the response (but I suspect it can be 
> fixed on the Tomcat side as the problem does not occur with Payara).

The configuration attribute you want is maxSwallowSize. You need to set
that to greater than the size of the uploaded file.

Clients could handle this better but many don't read the response until
the request is fully written.

Tomcat limit's maxSwallowSize as a DoS protection measure. Without it, a
client could just continue uploading a slow trickle of data and tie up a
server thread.

For the record, maPostSize applies *only* to requests using
application/x-www-form-urlencoded

The test provided by the OP uses multipart/form-data. The applicable
limits are defined by javax.servlet.annotation.MultipartConfig and the
defaults are unlimited.

Any call to getPart(), getParameter() and friends will trigger the
reading of the request body.

Hence, without the call to getParameter() Tomcat doesn't read the
request body. With small uploads there is enough network buffering
between the client and the server for the client to be able to write the
full request so it reads the response. (Tomcat's maxSwallowSize
effectively acts as a buffer.) With larger uploads the client fills the
buffers before the request is fully written so it never sees the response.

Increasing the maxSwallowSize will allow the client to write the full
request and then read the response.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: POST request fails if content is ignored?

2019-09-09 Thread Leon Atherton
Our use case is rejecting the request based on IP.

In the browser the status code is 0, and the network tab in developer 
tools is showing no response to the request. It's the same in Chrome and 
Firefox.

The request works fine when I send from Node.JS.

It seems to me that Tomcat responds to the request before the upload has 
completed, and calling request.getParameter() fixes the problem because 
it causes Tomcat to read the full request before the response is sent.

Some clients are fine with the early response (e.g. Node.JS), but both 
Chrome and Firefox don't like it.

I'm not sure if it's an issue with how Tomcat handles the request, or 
how the browsers are handling the response (but I suspect it can be 
fixed on the Tomcat side as the problem does not occur with Payara).

Thanks

On 09/09/2019 15:48, André Warnier (tomcat) wrote:
> On 09.09.2019 15:21, Leon Atherton wrote:
>> Thank you for the suggestion.
>>
>> I have just tried playing with this value. Setting it to -1, and setting
>> it to 100x larger than the default.
>> In both cases, the behaviour seems unchanged.
>>
>> Without touching this value, Tomcat will accept multipart POST requests
>> much larger than 2MB (which is the default) as long as you call
>> request.getPart() or request.getParameter().
>
> I haven't tried that, but as per the documentation that at least looks 
> wrong.
>
>>
>> The problem I am seeing is that if a response is sent without calling
>> one of those methods, the browsers see it as a failed request.
>
> To help me (or us) better understand the issue, could you provide a 
> bit of context ?
> Such as : why would the client send content in a POST, if the service 
> at the end of the target URL is not going to read that content ?
> What would be a use case ?
>
> (Mainly because my interpretation is that, if the browser does that, 
> then at least in some way it /is/ a bad request, which /should/ fail).
>
> And maybe another question : have you tried monitoring such a 
> request/response using a browser-based tracing tool, to show exactly 
> what the server is sending back ?
> (is it e.g. some kind of 4xx response ?)
>
>
>>
>> Thanks
>>
>> On 09/09/2019 12:59, André Warnier (tomcat) wrote:
>>> Hi.
>>> Did you check :
>>> https://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Common_Attributes 
>>>
>>>
>>> --> maxPostSize
>>>
>>> Note : normally, the browser will encode (Base64 or similar) the
>>> content of a file and send the encoded content, which tends to be
>>> significantly larger (in bytes) than the original file (say + 30%). I
>>> do not know (and the doc does not say) if the maxPostSize attribute
>>> refers to the POST content still encoded or already decoded.
>>>
>>>
>>> On 09.09.2019 12:53, Leon Atherton wrote:
 Hello,

 I've discovered an interesting issue where POST requests fail when
 uploading a file over about ~6MB if the server ignores the request
 content.

 I've put together a simple project to reproduce it:
 https://github.com/leonatherton/tomcat-request-issue

 Serverside code:
 https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/java/DemoServlet.java
  


 Clientside code:
 https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/webapp/index.html
  



 If you'd like to try it yourself there are a couple of pre-built war
 files on the releases page, and there are steps to reproduce in the
 README.md file.

 The problem does not occur when uploading a small file, and the 
 problem
 can be "fixed" by simply getting a parameter from the request 
 object. It
 reproduces in Tomcat 8 & 9. The problem does not reproduce on Payara,
 but I am seeing similar on Jetty.

 My guess is that the server responds before the client has finished
 uploading the file. The browsers see the incomplete upload and report
 this as an error, despite content being sent in response. And my guess
 is that inspecting a request parameter causes the server to wait 
 for the
 full upload before sending the response.

 It's a slightly odd workflow, but it's not too unreasonable to 
 sometimes
 respond early and ignore the request content.

 Is this expected behavior, or a bug in Tomcat?

 Thanks!
 Leon

 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org

>>>
>>>
>>> -
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: 

Re: POST request fails if content is ignored?

2019-09-09 Thread tomcat

On 09.09.2019 15:21, Leon Atherton wrote:

Thank you for the suggestion.

I have just tried playing with this value. Setting it to -1, and setting
it to 100x larger than the default.
In both cases, the behaviour seems unchanged.

Without touching this value, Tomcat will accept multipart POST requests
much larger than 2MB (which is the default) as long as you call
request.getPart() or request.getParameter().


I haven't tried that, but as per the documentation that at least looks wrong.



The problem I am seeing is that if a response is sent without calling
one of those methods, the browsers see it as a failed request.


To help me (or us) better understand the issue, could you provide a bit of 
context ?
Such as : why would the client send content in a POST, if the service at the end of the 
target URL is not going to read that content ?

What would be a use case ?

(Mainly because my interpretation is that, if the browser does that, then at least in some 
way it /is/ a bad request, which /should/ fail).


And maybe another question : have you tried monitoring such a request/response using a 
browser-based tracing tool, to show exactly what the server is sending back ?

(is it e.g. some kind of 4xx response ?)




Thanks

On 09/09/2019 12:59, André Warnier (tomcat) wrote:

Hi.
Did you check :
https://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Common_Attributes

--> maxPostSize

Note : normally, the browser will encode (Base64 or similar) the
content of a file and send the encoded content, which tends to be
significantly larger (in bytes) than the original file (say + 30%). I
do not know (and the doc does not say) if the maxPostSize attribute
refers to the POST content still encoded or already decoded.


On 09.09.2019 12:53, Leon Atherton wrote:

Hello,

I've discovered an interesting issue where POST requests fail when
uploading a file over about ~6MB if the server ignores the request
content.

I've put together a simple project to reproduce it:
https://github.com/leonatherton/tomcat-request-issue

Serverside code:
https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/java/DemoServlet.java

Clientside code:
https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/webapp/index.html


If you'd like to try it yourself there are a couple of pre-built war
files on the releases page, and there are steps to reproduce in the
README.md file.

The problem does not occur when uploading a small file, and the problem
can be "fixed" by simply getting a parameter from the request object. It
reproduces in Tomcat 8 & 9. The problem does not reproduce on Payara,
but I am seeing similar on Jetty.

My guess is that the server responds before the client has finished
uploading the file. The browsers see the incomplete upload and report
this as an error, despite content being sent in response. And my guess
is that inspecting a request parameter causes the server to wait for the
full upload before sending the response.

It's a slightly odd workflow, but it's not too unreasonable to sometimes
respond early and ignore the request content.

Is this expected behavior, or a bug in Tomcat?

Thanks!
Leon

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: POST request fails if content is ignored?

2019-09-09 Thread Christopher Schultz
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

André,

On 9/9/19 07:59, André Warnier (tomcat) wrote:
> Hi. Did you check : 
> https://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Common_Attri
butes
>
> 
- --> maxPostSize
> 
> Note : normally, the browser will encode (Base64 or similar) the
> content of a file and send the encoded content, which tends to be
> significantly larger (in bytes) than the original file (say +
> 30%).

I know of no browser that encodes a file in any way when performing a
standard file-upload via an  control. HTTP handled
binary content just fine.

> I do not know (and the doc does not say) if the maxPostSize
> attribute refers to the POST content still encoded or already
> decoded.

That setting counts bytes on the wire.

- -chris

> On 09.09.2019 12:53, Leon Atherton wrote:
>> Hello,
>> 
>> I've discovered an interesting issue where POST requests fail
>> when uploading a file over about ~6MB if the server ignores the
>> request content.
>> 
>> I've put together a simple project to reproduce it: 
>> https://github.com/leonatherton/tomcat-request-issue
>> 
>> Serverside code: 
>> https://github.com/leonatherton/tomcat-request-issue/blob/master/src/
main/java/DemoServlet.java
>>
>>
>> 
Clientside code:
>> https://github.com/leonatherton/tomcat-request-issue/blob/master/src/
main/webapp/index.html
>>
>>
>>
>> 
If you'd like to try it yourself there are a couple of pre-built war
>> files on the releases page, and there are steps to reproduce in
>> the README.md file.
>> 
>> The problem does not occur when uploading a small file, and the
>> problem can be "fixed" by simply getting a parameter from the
>> request object. It reproduces in Tomcat 8 & 9. The problem does
>> not reproduce on Payara, but I am seeing similar on Jetty.
>> 
>> My guess is that the server responds before the client has
>> finished uploading the file. The browsers see the incomplete
>> upload and report this as an error, despite content being sent in
>> response. And my guess is that inspecting a request parameter
>> causes the server to wait for the full upload before sending the
>> response.
>> 
>> It's a slightly odd workflow, but it's not too unreasonable to
>> sometimes respond early and ignore the request content.
>> 
>> Is this expected behavior, or a bug in Tomcat?
>> 
>> Thanks! Leon
>> 
>> -
>>
>> 
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
> 
> 
> -
>
> 
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl12We8ACgkQHPApP6U8
pFhbsxAAta8C4RjgQ7DPpoZ/gtE9M6oFfICUeX1gRCB4YYu6nWY4O+8r/CK+hGHV
yuWUchWQaeTG+sSIrXL2tvaIbQEIUzLLEDd6H/nSbCIL/rOF3AXs4AbGfsyLcUji
L7c9G334aBWQoGS6TfYrXAHZfI7YmtPqc4rXgQqJwRkOCGg0oLL8e7YKLyffPb2U
BFnSaMFaXHIPIoKEC3lLl39r6fCikpgx/MJ7wqLrU3Oux6tr3srcAFFlnhGpmUG9
vcoXULMxo890gd4uspyfbV5OL5BektFLG4u5t1wu4vVdC+G/+l0uTd1i3J9w6+cM
GbBMjeIH1g87/bb+rvrszMGn5XEcEs3i1UwZIy0UNY0oVcGJkV4V5iu2e5VVCp8k
ZcrG56bCptKL9CkTQafhPeztfhRPJya4UVUm9MeVKu1mQsKWZrya2840hDeKzv9V
FhTsqLlS0GeUn3dFpKItQ5ngH4jpT3Lm4VQ2hrTt4h529YpgD0FMk8i+BdeuFnCF
hsHbTxJLnmw8fZ+H5mGLNzB4DTWVVJVTmCa0xcI/zO2S6Arp8q1jyxiSOurlTNZ6
pD0Y45NwTpwtwWhjhgWNQilLHtc1Nc3KOMcERX+0++cqJWrM2bxMcm/YQ/QK9yXA
nPWD8BMelyL9MhdvyjeGKoH6zqQ0uAhT6/iYFDqOw3gTMOtwEkk=
=1NdL
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: POST request fails if content is ignored?

2019-09-09 Thread Leon Atherton
Thank you for the suggestion.

I have just tried playing with this value. Setting it to -1, and setting 
it to 100x larger than the default.
In both cases, the behaviour seems unchanged.

Without touching this value, Tomcat will accept multipart POST requests 
much larger than 2MB (which is the default) as long as you call 
request.getPart() or request.getParameter().

The problem I am seeing is that if a response is sent without calling 
one of those methods, the browsers see it as a failed request.

Thanks

On 09/09/2019 12:59, André Warnier (tomcat) wrote:
> Hi.
> Did you check :
> https://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Common_Attributes 
>
> --> maxPostSize
>
> Note : normally, the browser will encode (Base64 or similar) the 
> content of a file and send the encoded content, which tends to be 
> significantly larger (in bytes) than the original file (say + 30%). I 
> do not know (and the doc does not say) if the maxPostSize attribute 
> refers to the POST content still encoded or already decoded.
>
>
> On 09.09.2019 12:53, Leon Atherton wrote:
>> Hello,
>>
>> I've discovered an interesting issue where POST requests fail when
>> uploading a file over about ~6MB if the server ignores the request 
>> content.
>>
>> I've put together a simple project to reproduce it:
>> https://github.com/leonatherton/tomcat-request-issue
>>
>> Serverside code:
>> https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/java/DemoServlet.java
>>  
>>
>> Clientside code:
>> https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/webapp/index.html
>>  
>>
>>
>> If you'd like to try it yourself there are a couple of pre-built war
>> files on the releases page, and there are steps to reproduce in the
>> README.md file.
>>
>> The problem does not occur when uploading a small file, and the problem
>> can be "fixed" by simply getting a parameter from the request object. It
>> reproduces in Tomcat 8 & 9. The problem does not reproduce on Payara,
>> but I am seeing similar on Jetty.
>>
>> My guess is that the server responds before the client has finished
>> uploading the file. The browsers see the incomplete upload and report
>> this as an error, despite content being sent in response. And my guess
>> is that inspecting a request parameter causes the server to wait for the
>> full upload before sending the response.
>>
>> It's a slightly odd workflow, but it's not too unreasonable to sometimes
>> respond early and ignore the request content.
>>
>> Is this expected behavior, or a bug in Tomcat?
>>
>> Thanks!
>> Leon
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: POST request fails if content is ignored?

2019-09-09 Thread tomcat

Hi.
Did you check :
https://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Common_Attributes
--> maxPostSize

Note : normally, the browser will encode (Base64 or similar) the content of a file and 
send the encoded content, which tends to be significantly larger (in bytes) than the 
original file (say + 30%). I do not know (and the doc does not say) if the maxPostSize 
attribute refers to the POST content still encoded or already decoded.



On 09.09.2019 12:53, Leon Atherton wrote:

Hello,

I've discovered an interesting issue where POST requests fail when
uploading a file over about ~6MB if the server ignores the request content.

I've put together a simple project to reproduce it:
https://github.com/leonatherton/tomcat-request-issue

Serverside code:
https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/java/DemoServlet.java
Clientside code:
https://github.com/leonatherton/tomcat-request-issue/blob/master/src/main/webapp/index.html

If you'd like to try it yourself there are a couple of pre-built war
files on the releases page, and there are steps to reproduce in the
README.md file.

The problem does not occur when uploading a small file, and the problem
can be "fixed" by simply getting a parameter from the request object. It
reproduces in Tomcat 8 & 9. The problem does not reproduce on Payara,
but I am seeing similar on Jetty.

My guess is that the server responds before the client has finished
uploading the file. The browsers see the incomplete upload and report
this as an error, despite content being sent in response. And my guess
is that inspecting a request parameter causes the server to wait for the
full upload before sending the response.

It's a slightly odd workflow, but it's not too unreasonable to sometimes
respond early and ignore the request content.

Is this expected behavior, or a bug in Tomcat?

Thanks!
Leon

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org