[
https://issues.apache.org/jira/browse/AIRAVATA-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644949#comment-17644949
]
Marcus Christie commented on AIRAVATA-3679:
-------------------------------------------
Needed to pin Markdown dependency to 3.3.4 for unit tests on Python 3.6
https://github.com/Python-Markdown/markdown/issues/1203#issuecomment-976691959
> directory download error: Zip file size exceeds max of 1073741824 bytes
> -----------------------------------------------------------------------
>
> Key: AIRAVATA-3679
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3679
> Project: Airavata
> Issue Type: Bug
> Components: Django Portal
> Reporter: Marcus Christie
> Assignee: Marcus Christie
> Priority: Major
>
> When a directory is downloaded in the Django portal, the directory is first
> converted to a zip file and then that zip file is returned in the response.
> Because the zip file has to be created first, as a temporary file, there is a
> limit on how large it can be so it doesn't overfill the temporary storage
> space (usually the same as the root partition). To prevent a disk space
> overfill there is a cap on directory downloads of 1GB.
> Whenever a directory zip download file goes above 1GB, the *Zip file size
> exceeds max of 1073741824 bytes* error is generated.
> Initially I thought I could do a check first if the directory is larger than
> 1GB and then disallow download. But a directory might be larger than 1GB and
> yet compress to much less than 1GB. It's generally hard to know in advance
> how large a zipped directory will be.
> Another approach is to eliminate the temporary file creation and just stream
> the zip file. That is, write the zip file directly to the HTTP response
> instead of having a temporary file intermediary. This is light on memory
> usage and disk usage and also benefits the user by starting the download
> sooner. So I'm going to explore this option.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
