Re: [Clamav-devel] scanning large zip files -> file loaded entirely into (RSS) memory during extraction?

2011-12-14 Thread Bram
Quoting aCaB : On 12/12/11 15:34, Bram wrote: Is this implicit relation only for zip files or also for other compression/archive formats? (I tested bzip2 and gzip and there the RSS does not increase - or at least not that much) I'm sure it affects other formats too. But I can't tell which on

Re: [Clamav-devel] scanning large zip files -> file loaded entirely into (RSS) memory during extraction?

2011-12-12 Thread aCaB
On 12/12/11 21:31, Bram wrote: > * a zip file which contains 4 files of 100 MB. > * max-filesize set to 1GB > * max-scansize set to 4GB > What is the expected value of the RSS during the scan? About MIN(zip_size(100MB), 1GB, 4GB), where zip_size() is the size of the compressed stream. Hope it's

Re: [Clamav-devel] scanning large zip files -> file loaded entirely into (RSS) memory during extraction?

2011-12-12 Thread Bram
Is it expected that the RSS memory increases with approximately the size of the zip file before extracting it? No. The RSS memory size will increase at most by the smaller of - compressed internal file size - max-filesize (the file is skipped if the value's exceeded) - max-scansize (the scan is

Re: [Clamav-devel] scanning large zip files -> file loaded entirely into (RSS) memory during extraction?

2011-12-12 Thread aCaB
On 12/12/11 15:34, Bram wrote: > Is this implicit relation only for zip files or also for other > compression/archive formats? (I tested bzip2 and gzip and there the RSS > does not increase - or at least not that much) I'm sure it affects other formats too. But I can't tell which on the top of my

Re: [Clamav-devel] scanning large zip files -> file loaded entirely into (RSS) memory during extraction?

2011-12-12 Thread Bram
Quoting aCaB : Is this necessary? (Tools such as unzip are able to decompress the file without loading the entire file into memory). Not necessary but it makes the code much simpler in the default case scenario as, with the default settings, the allocated memory is way below CLI_MAX_ALLOCATIO

Re: [Clamav-devel] scanning large zip files -> file loaded entirely into (RSS) memory during extraction?

2011-12-12 Thread aCaB
On 12/12/11 09:31, Bram wrote: > Questions: > Is it expected that the RSS memory increases with approximately the size > of the zip file before extracting it? No. The RSS memory size will increase at most by the smaller of - compressed internal file size - max-filesize (the file is skipped if the

[Clamav-devel] scanning large zip files -> file loaded entirely into (RSS) memory during extraction?

2011-12-12 Thread Bram
Hi, I'm having a 'problem' with scanning a large zip file. What appears to be happening is that clamav loads the entire file into memory (RSS not VSZ) and then unzips the file. The increase in VSZ memory is expected since the file is mmap'ed; the increase in RSS memory is/was not expected (