[
https://issues.apache.org/jira/browse/AIRAVATA-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Christie updated AIRAVATA-3679:
--------------------------------------
Description:
When a directory is downloaded in the Django portal, the directory is first
converted to a zip file and then that zip file is returned in the response.
Because the zip file has to be created first, as a temporary file, there is a
limit on how large it can be so it doesn't overfill the temporary storage space
(usually the same as the root partition). To prevent a disk space overfill
there is a cap on directory downloads of 1GB.
Whenever a directory zip download file goes above 1GB, the *Zip file size
exceeds max of 1073741824 bytes* error is generated.
Initially I thought I could do a check first if the directory is larger than
1GB and then disallow download. But a directory might be larger than 1GB and
yet compress to much less than 1GB. It's generally hard to know in advance how
large a zipped directory will be.
Another approach is to eliminate the temporary file creation and just stream
the zip file. That is, write the zip file directly to the HTTP response instead
of having a temporary file intermediary. This is light on memory usage and disk
usage and also benefits the user by starting the download sooner. So I'm going
to explore this option.
was:
When a directory is downloaded in the Django portal, the directory is first
converted to a zip file and then that zip file is returned in the response.
Because the zip file has to be created first, as a temporary file, there is a
limit on how large it can be so it doesn't overfill the temporary storage space
(usually the same as the root partition). To this end there is a cap on
directory downloads of 1GB.
Whenever a directory zip download file goes above 1GB, the *Zip file size
exceeds max of 1073741824 bytes* error is generated.
> directory download error: Zip file size exceeds max of 1073741824 bytes
> -----------------------------------------------------------------------
>
> Key: AIRAVATA-3679
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3679
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
> Reporter: Marcus Christie
> Assignee: Marcus Christie
> Priority: Major
>
> When a directory is downloaded in the Django portal, the directory is first
> converted to a zip file and then that zip file is returned in the response.
> Because the zip file has to be created first, as a temporary file, there is a
> limit on how large it can be so it doesn't overfill the temporary storage
> space (usually the same as the root partition). To prevent a disk space
> overfill there is a cap on directory downloads of 1GB.
> Whenever a directory zip download file goes above 1GB, the *Zip file size
> exceeds max of 1073741824 bytes* error is generated.
> Initially I thought I could do a check first if the directory is larger than
> 1GB and then disallow download. But a directory might be larger than 1GB and
> yet compress to much less than 1GB. It's generally hard to know in advance
> how large a zipped directory will be.
> Another approach is to eliminate the temporary file creation and just stream
> the zip file. That is, write the zip file directly to the HTTP response
> instead of having a temporary file intermediary. This is light on memory
> usage and disk usage and also benefits the user by starting the download
> sooner. So I'm going to explore this option.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
