Re: File size truncated at 1.4GB during download from Tomcat WebApp

[calder]

On Sat, Oct 3, 2020, 11:43 Mauro Tridici  wrote:

>
> > On 3 Oct 2020, at 17:03, calder  wrote:
> >
> > On Sat, Oct 3, 2020, 09:58 calder  wrote:
> >
> >> On Sat, Oct 3, 2020, 09:01 Mauro Tridici  wrote:
> >>
> >>>
> >>>
> >>> On 3 Oct 2020, at 15:14, Christopher Schultz <
> >>> ch...@christopherschultz.net> wrote:
> >>>
> >>> Mauro,
> >>>
> >>> On 10/3/20 08:47, Mauro Tridici wrote
> >>>
> >>> I’m struggling with the problem mentioned in this mail subject.
> >>> When I try to download a 5GB sized file using two different
> >>>
> >>>
> >> < snip >
> >>
> >> In your opinion, if the problem is related to the 32-bit overflow, is
> >>> there something that I can do in order to solve this issue.
> >>> Since I can’t modify any change to these two applications, I would like
> >>> to know if I can do something on the other sides.
> >>>
> >>
> >>
> >> This issue was reported to the IRODs team, they confirmed it, and
> pushed a
> >> fix.  You should check your version.
> >>
> >> The issue, which is now closed.
> >>
> >> https://github.com/irods-contrib/irods-rest/issues/66
> >>
> >
> > I also see your involvement in this issue,  which was marked as a Dup.
> >
> > https://github.com/irods-contrib/metalnx-web/issues/130
> >
> > ... and updated here
> > https://github.com/irods-contrib/metalnx-web/issues/143
>
>
> Hi Calder,
>
> thank you for your help and suggestions.
> Unfortunately, I’m still experiencing the same problem despite I’m using
> the last available version.
>
> Since I used also a different web application (not MetalNX) and I noticed
> the same problem, I was thinking that there is something to be changed in
> tomcat configuration.
> I will try to start again from scentsratch.
>

Just to be sure ... did you check again to be sure you have the patch
installed?   I ask because two people who had the problem reported back
that the patch fixed the issue for them.

Reading the comments, they did find where an "int" was being used for
"contentLength"  in the code.

>

Re: File size truncated at 1.4GB during download from Tomcat WebApp

[Mauro Tridici]

> On 3 Oct 2020, at 17:03, calder  wrote:
> 
> On Sat, Oct 3, 2020, 09:58 calder  wrote:
> 
>> On Sat, Oct 3, 2020, 09:01 Mauro Tridici  wrote:
>> 
>>> 
>>> 
>>> On 3 Oct 2020, at 15:14, Christopher Schultz <
>>> ch...@christopherschultz.net> wrote:
>>> 
>>> Mauro,
>>> 
>>> On 10/3/20 08:47, Mauro Tridici wrote
>>> 
>>> I’m struggling with the problem mentioned in this mail subject.
>>> When I try to download a 5GB sized file using two different
>>> 
>>> 
>> < snip >
>> 
>> In your opinion, if the problem is related to the 32-bit overflow, is
>>> there something that I can do in order to solve this issue.
>>> Since I can’t modify any change to these two applications, I would like
>>> to know if I can do something on the other sides.
>>> 
>> 
>> 
>> This issue was reported to the IRODs team, they confirmed it, and pushed a
>> fix.  You should check your version.
>> 
>> The issue, which is now closed.
>> 
>> https://github.com/irods-contrib/irods-rest/issues/66
>> 
> 
> I also see your involvement in this issue,  which was marked as a Dup.
> 
> https://github.com/irods-contrib/metalnx-web/issues/130
> 
> ... and updated here
> https://github.com/irods-contrib/metalnx-web/issues/143


Hi Calder,

thank you for your help and suggestions.
Unfortunately, I’m still experiencing the same problem despite I’m using the 
last available version.

Since I used also a different web application (not MetalNX) and I noticed the 
same problem, I was thinking that there is something to be changed in tomcat 
configuration.
I will try to start again from scratch.

Mauro

Re: File size truncated at 1.4GB during download from Tomcat WebApp

[calder]

On Sat, Oct 3, 2020, 09:58 calder  wrote:

> On Sat, Oct 3, 2020, 09:01 Mauro Tridici  wrote:
>
>>
>>
>> On 3 Oct 2020, at 15:14, Christopher Schultz <
>> ch...@christopherschultz.net> wrote:
>>
>> Mauro,
>>
>> On 10/3/20 08:47, Mauro Tridici wrote
>>
>> I’m struggling with the problem mentioned in this mail subject.
>> When I try to download a 5GB sized file using two different
>>
>>
>  < snip >
>
> In your opinion, if the problem is related to the 32-bit overflow, is
>> there something that I can do in order to solve this issue.
>> Since I can’t modify any change to these two applications, I would like
>> to know if I can do something on the other sides.
>>
>
>
> This issue was reported to the IRODs team, they confirmed it, and pushed a
> fix.  You should check your version.
>
> The issue, which is now closed.
>
> https://github.com/irods-contrib/irods-rest/issues/66
>

I also see your involvement in this issue,  which was marked as a Dup.

https://github.com/irods-contrib/metalnx-web/issues/130

... and updated here
https://github.com/irods-contrib/metalnx-web/issues/143

Re: File size truncated at 1.4GB during download from Tomcat WebApp

[calder]

On Sat, Oct 3, 2020, 09:01 Mauro Tridici  wrote:

>
>
> On 3 Oct 2020, at 15:14, Christopher Schultz 
> wrote:
>
> Mauro,
>
> On 10/3/20 08:47, Mauro Tridici wrote
>
> I’m struggling with the problem mentioned in this mail subject.
> When I try to download a 5GB sized file using two different
>
>
 < snip >

In your opinion, if the problem is related to the 32-bit overflow, is there
> something that I can do in order to solve this issue.
> Since I can’t modify any change to these two applications, I would like to
> know if I can do something on the other sides.
>


This issue was reported to the IRODs team, they confirmed it, and pushed a
fix.  You should check your version.

The issue, which is now closed.

https://github.com/irods-contrib/irods-rest/issues/66

Re: File size truncated at 1.4GB during download from Tomcat WebApp

[Mauro Tridici]

> On 3 Oct 2020, at 15:14, Christopher Schultz  
> wrote:
> 
> Mauro,
> 
> On 10/3/20 08:47, Mauro Tridici wrote:
>> Dear Users,
>> 
>> I’m struggling with the problem mentioned in this mail subject.
>> When I try to download a 5GB sized file using two different tomcat web
>> applications on two different virtual machines, I noticed that the
>> browser download window shows a wrong file size (please take a look to
>> the attached picture).
>> Download starts regularly but it never ends.
>> 
>> I tried to download different files using different web browsers, but
>> nothing changed.
>> It seems that this problem is not related to the web applications (as I
>> said, I’m testing two applications) and it is not related to a specific
>> tomcat version.
>> 
>> I’m a “tomcat newbie”, could you please help me to understand the cause
>> of this issue?
>> You can find below some additional information about virtual machines,
>> os and tomcat.
>> 
>> *1) virtual machine “serverA"*
>> 
>> OS: CentOS Linux release 7.8.2003
>> 
>> TOMCAT: 
>> Server version: Apache Tomcat/7.0.76
>> Server built:   Mar 17 2020 23:48:55 UTC
>> Server number:  7.0.76.0
>> OS Name:Linux
>> OS Version: 3.10.0-1127.8.2.el7.x86_64
>> Architecture:   amd64
>> JVM Version:1.8.0_252-b09
>> JVM Vendor: Oracle Corporation
>> 
>> *2) virtual machine “serverB”*
>> 
>> OS: CentOS Linux release 7.8.2003
>> 
>> TOMCAT:
>> Server version: Apache Tomcat/8.5.56
>> Server built:   Jun 3 2020 20:18:30 UTC
>> Server number:  8.5.56.0
>> OS Name:Linux
>> OS Version: 3.10.0-1127.8.2.el7.x86_64
>> Architecture:   amd64
>> JVM Version:1.8.0_252-b09
>> JVM Vendor: Oracle Corporation
>> 
>> 
>> Thank you in advance,
>> Mauro
> 
> Can you observe the HTTP response headers sent by the server along with
> the file?
> 
> Also, how are you sending the file(s) from the application to the
> client? Are you using sendfile? Are you using the DefaultServlet? Or do
> you have your own code which reads the file and writes it to the client?
> 
> Does the download always stop at the same point (byte count), or it is
> different each time?
> 
> My guess is:
> 
> 1. The code is in your application (or at least, not using Tomcat's code
> at all)
> 
> 2. The code uses "int" variable to track the file size / set
> Content-Length response header
> 
> 3. The file size is 5.4GiB
> 
> 4. 5.4 GiB will overflow a 32-bit int by 3.4GiB resulting in a value of
> +1.4GiB final value
> 
> 5. This value is being send to the client in the Content-Length header
> 
> 6. The client is terminating the download after 1.4GiB (respecting the
> Content-Length header)
> 
> -chris
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org


HI Christopher,

thank you very much for your answer.

You can find below the HTTP response headers detected during the download fo 
file named disk-0.vmdk

172.16.1.8 - - [03/Oct/2020:15:25:53 +0200] "GET 
/irods-cloud-backend/virtualCollection/root HTTP/1.1" 401 951
172.16.1.8 - - [03/Oct/2020:15:25:53 +0200] "GET 
/irods-cloud-backend/collection/root?offset=0&path=%2Fidas%2Fhome%2Fsysm02 
HTTP/1.1" 401 951
172.16.1.8 - - [03/Oct/2020:15:25:54 +0200] "GET 
/irods-cloud-backend/initialConf HTTP/1.1" 200 138
172.16.1.8 - - [03/Oct/2020:15:26:00 +0200] "OPTIONS 
/irods-cloud-backend/login/ HTTP/1.1" 200 -
172.16.1.8 - - [03/Oct/2020:15:26:00 +0200] "POST /irods-cloud-backend/login/ 
HTTP/1.1" 200 102
172.16.1.8 - - [03/Oct/2020:15:26:01 +0200] "GET 
/irods-cloud-backend/collection/root?offset=0&path=%2Fidas%2Fhome%2Fsysm02 
HTTP/1.1" 200 8117
172.16.1.8 - - [03/Oct/2020:15:26:01 +0200] "GET 
/irods-cloud-backend/virtualCollection/root HTTP/1.1" 200 480
172.16.1.8 - - [03/Oct/2020:15:26:01 +0200] "GET 
/irods-cloud-backend/fileCreatorTemplate HTTP/1.1" 200 787
172.16.1.8 - - [03/Oct/2020:15:26:02 +0200] "GET 
/irods-cloud-backend/virtualCollection HTTP/1.1" 200 2304
172.16.1.8 - - [03/Oct/2020:15:40:29 +0200] "GET 
/irods-cloud-backend/download?path=/idas/home/sysm02/disk-0.vmdk HTTP/1.1" 200 
1532646912

Unfortunately, I can't give you more details about how the sfiles are sent to 
the client because the 2 different web applications have not been created by me 
(this is one of web applications I’m using 
https://github.com/irods-contrib/irods-cloud-browser 
)
I’m sorry, but I’m not a developer.

The file download is always completed and it stops at the same point (1.4GB)



In your opinion, if the problem is related to the 32-bit overflow, is there 
something that I can do in order to solve this issue.
Since I can’t modify any change to these two applications, I would like to know 
if I can do something on the other sides.

Many thanks again for your patience.
Mauro

Re: File size truncated at 1.4GB during download from Tomcat WebApp

[Christopher Schultz]

Mauro,

On 10/3/20 08:47, Mauro Tridici wrote:
> Dear Users,
> 
> I’m struggling with the problem mentioned in this mail subject.
> When I try to download a 5GB sized file using two different tomcat web
> applications on two different virtual machines, I noticed that the
> browser download window shows a wrong file size (please take a look to
> the attached picture).
> Download starts regularly but it never ends.
> 
> I tried to download different files using different web browsers, but
> nothing changed.
> It seems that this problem is not related to the web applications (as I
> said, I’m testing two applications) and it is not related to a specific
> tomcat version.
> 
> I’m a “tomcat newbie”, could you please help me to understand the cause
> of this issue?
> You can find below some additional information about virtual machines,
> os and tomcat.
> 
> *1) virtual machine “serverA"*
> 
> OS: CentOS Linux release 7.8.2003
> 
> TOMCAT: 
> Server version: Apache Tomcat/7.0.76
> Server built:   Mar 17 2020 23:48:55 UTC
> Server number:  7.0.76.0
> OS Name:        Linux
> OS Version:     3.10.0-1127.8.2.el7.x86_64
> Architecture:   amd64
> JVM Version:    1.8.0_252-b09
> JVM Vendor:     Oracle Corporation
> 
> *2) virtual machine “serverB”*
> 
> OS: CentOS Linux release 7.8.2003
> 
> TOMCAT:
> Server version: Apache Tomcat/8.5.56
> Server built:   Jun 3 2020 20:18:30 UTC
> Server number:  8.5.56.0
> OS Name:        Linux
> OS Version:     3.10.0-1127.8.2.el7.x86_64
> Architecture:   amd64
> JVM Version:    1.8.0_252-b09
> JVM Vendor:     Oracle Corporation
> 
> 
> Thank you in advance,
> Mauro

Can you observe the HTTP response headers sent by the server along with
the file?

Also, how are you sending the file(s) from the application to the
client? Are you using sendfile? Are you using the DefaultServlet? Or do
you have your own code which reads the file and writes it to the client?

Does the download always stop at the same point (byte count), or it is
different each time?

My guess is:

1. The code is in your application (or at least, not using Tomcat's code
at all)

2. The code uses "int" variable to track the file size / set
Content-Length response header

3. The file size is 5.4GiB

4. 5.4 GiB will overflow a 32-bit int by 3.4GiB resulting in a value of
+1.4GiB final value

5. This value is being send to the client in the Content-Length header

6. The client is terminating the download after 1.4GiB (respecting the
Content-Length header)

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org