Re: std.zip expand: memory allocation failed
On Tuesday, 26 October 2021 at 13:43:36 UTC, Steven Schveighoffer wrote: On 10/26/21 2:32 AM, bauss wrote: On Monday, 25 October 2021 at 22:38:38 UTC, Imperatorn wrote: On Monday, 25 October 2021 at 20:50:40 UTC, Steven Schveighoffer wrote: On 10/24/21 8:00 AM, Selim Ozel wrote: It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff. Wait, x86 is 32-bit. Max address space is 4GB. So maybe it was just trying to use 4GB and running out of memory? -Steve Good catch, but still, should it use so much memory? Definitely not. It shouldn't use a lot of memory when unzipping as it should be done in chunks! You guys aren't getting it: ``` ubyte[] expand(ArchiveMember de); Decompress the contents of a member. Fills in properties extractVersion, flags, compressionMethod, time, crc32, compressedSize, expandedSize, expandedData[], name[], extra[]. ``` Where is it supposed to store that `ubyte[]`? -Steve It's not supposed, but a new implementation can utilize something like a stream.
Re: std.zip expand: memory allocation failed
On Tuesday, 26 October 2021 at 20:33:17 UTC, Steven Schveighoffer wrote: On 10/26/21 1:38 PM, Imperatorn wrote: That's the current implementation. No, that's the API. You cannot fix the implementation with that API and not end up allocating an array to hold the entire unzipped contents. You can't even decompress to a file, and then mmap those contents -- the address space isn't there. I don't know about *nix, but my Windows machine can easily extract a file bigger than my RAM. zlib does not require decompressing in entirety. This is just the way that std.zip decided to expose the API. e.g. iopipe uses an expandable buffer for decompression and compression, which does not have to contain the entire file. -Steve Yes, that's how it's written. That's the problem.
Re: std.zip expand: memory allocation failed
On 10/26/21 1:38 PM, Imperatorn wrote: That's the current implementation. No, that's the API. You cannot fix the implementation with that API and not end up allocating an array to hold the entire unzipped contents. You can't even decompress to a file, and then mmap those contents -- the address space isn't there. I don't know about *nix, but my Windows machine can easily extract a file bigger than my RAM. zlib does not require decompressing in entirety. This is just the way that std.zip decided to expose the API. e.g. iopipe uses an expandable buffer for decompression and compression, which does not have to contain the entire file. -Steve
Re: std.zip expand: memory allocation failed
On Tuesday, 26 October 2021 at 17:38:22 UTC, Imperatorn wrote: On Tuesday, 26 October 2021 at 13:43:36 UTC, Steven Schveighoffer wrote: On 10/26/21 2:32 AM, bauss wrote: On Monday, 25 October 2021 at 22:38:38 UTC, Imperatorn wrote: [...] Definitely not. It shouldn't use a lot of memory when unzipping as it should be done in chunks! You guys aren't getting it: ``` ubyte[] expand(ArchiveMember de); Decompress the contents of a member. Fills in properties extractVersion, flags, compressionMethod, time, crc32, compressedSize, expandedSize, expandedData[], name[], extra[]. ``` Where is it supposed to store that `ubyte[]`? -Steve That's the current implementation. I don't know about *nix, but my Windows machine can easily extract a file bigger than my RAM. It ofc also depends on the dictionary. The biggest file I've ever decompressed on my own hardware was about 200 GB. Needless to say, it wasn't using the algorithm in std.zip
Re: std.zip expand: memory allocation failed
On Tuesday, 26 October 2021 at 13:43:36 UTC, Steven Schveighoffer wrote: On 10/26/21 2:32 AM, bauss wrote: On Monday, 25 October 2021 at 22:38:38 UTC, Imperatorn wrote: On Monday, 25 October 2021 at 20:50:40 UTC, Steven Schveighoffer wrote: On 10/24/21 8:00 AM, Selim Ozel wrote: [...] Wait, x86 is 32-bit. Max address space is 4GB. So maybe it was just trying to use 4GB and running out of memory? -Steve Good catch, but still, should it use so much memory? Definitely not. It shouldn't use a lot of memory when unzipping as it should be done in chunks! You guys aren't getting it: ``` ubyte[] expand(ArchiveMember de); Decompress the contents of a member. Fills in properties extractVersion, flags, compressionMethod, time, crc32, compressedSize, expandedSize, expandedData[], name[], extra[]. ``` Where is it supposed to store that `ubyte[]`? -Steve That's the current implementation. I don't know about *nix, but my Windows machine can easily extract a file bigger than my RAM. It ofc also depends on the dictionary.
Re: std.zip expand: memory allocation failed
On 10/26/21 2:32 AM, bauss wrote: On Monday, 25 October 2021 at 22:38:38 UTC, Imperatorn wrote: On Monday, 25 October 2021 at 20:50:40 UTC, Steven Schveighoffer wrote: On 10/24/21 8:00 AM, Selim Ozel wrote: It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff. Wait, x86 is 32-bit. Max address space is 4GB. So maybe it was just trying to use 4GB and running out of memory? -Steve Good catch, but still, should it use so much memory? Definitely not. It shouldn't use a lot of memory when unzipping as it should be done in chunks! You guys aren't getting it: ``` ubyte[] expand(ArchiveMember de); Decompress the contents of a member. Fills in properties extractVersion, flags, compressionMethod, time, crc32, compressedSize, expandedSize, expandedData[], name[], extra[]. ``` Where is it supposed to store that `ubyte[]`? -Steve
Re: std.zip expand: memory allocation failed
On Tuesday, 26 October 2021 at 06:32:21 UTC, bauss wrote: On Monday, 25 October 2021 at 22:38:38 UTC, Imperatorn wrote: On Monday, 25 October 2021 at 20:50:40 UTC, Steven Schveighoffer wrote: On 10/24/21 8:00 AM, Selim Ozel wrote: [...] Wait, x86 is 32-bit. Max address space is 4GB. So maybe it was just trying to use 4GB and running out of memory? -Steve Good catch, but still, should it use so much memory? Definitely not. It shouldn't use a lot of memory when unzipping as it should be done in chunks! Exactly my thoughts
Re: std.zip expand: memory allocation failed
On Monday, 25 October 2021 at 22:38:38 UTC, Imperatorn wrote: On Monday, 25 October 2021 at 20:50:40 UTC, Steven Schveighoffer wrote: On 10/24/21 8:00 AM, Selim Ozel wrote: It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff. Wait, x86 is 32-bit. Max address space is 4GB. So maybe it was just trying to use 4GB and running out of memory? -Steve Good catch, but still, should it use so much memory? Definitely not. It shouldn't use a lot of memory when unzipping as it should be done in chunks!
Re: std.zip expand: memory allocation failed
On Monday, 25 October 2021 at 20:50:40 UTC, Steven Schveighoffer wrote: On 10/24/21 8:00 AM, Selim Ozel wrote: It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff. Wait, x86 is 32-bit. Max address space is 4GB. So maybe it was just trying to use 4GB and running out of memory? -Steve Good catch, but still, should it use so much memory?
Re: std.zip expand: memory allocation failed
On 10/24/21 8:00 AM, Selim Ozel wrote: It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff. Wait, x86 is 32-bit. Max address space is 4GB. So maybe it was just trying to use 4GB and running out of memory? -Steve
Re: std.zip expand: memory allocation failed
On Sunday, 24 October 2021 at 14:14:08 UTC, Imperatorn wrote: Create an issue and we can solve it Thanks. I opened an issue. https://issues.dlang.org/show_bug.cgi?id=22436
Re: std.zip expand: memory allocation failed
On Sunday, 24 October 2021 at 12:00:39 UTC, Selim Ozel wrote: On Friday, 15 October 2021 at 20:41:36 UTC, Selim Ozel wrote: [...] It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff. My work around was to call 7z from D Lang and do the compression over there. That worked like a charm. It seems that zip.d [1] calls uncompress routine from zlib.d [2]. Would calling zlib uncompress by chunks solve this memory issue? Any ideas? S [1] https://github.com/dlang/phobos/blob/master/std/zip.d [2] https://github.com/dlang/phobos/blob/master/std/zlib.d Create an issue and we can solve it
Re: std.zip expand: memory allocation failed
On Friday, 15 October 2021 at 20:41:36 UTC, Selim Ozel wrote: I am simply trying to unzip a compressed zip file slightly over 1GB. The de-compressed size is about 4 GB. The code is very similar to what's explained in the documentation [1] and it works for smaller files. Anyone has a solution? Memory mapping [2] previously solved some part of my issue but expand is still throwing memory allocation failure. Selim [1] https://dlang.org/phobos/std_zip.html [2] https://forum.dlang.org/thread/mfnleztnwrbgivjvz...@forum.dlang.org It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff. My work around was to call 7z from D Lang and do the compression over there. That worked like a charm. It seems that zip.d [1] calls uncompress routine from zlib.d [2]. Would calling zlib uncompress by chunks solve this memory issue? Any ideas? S [1] https://github.com/dlang/phobos/blob/master/std/zip.d [2] https://github.com/dlang/phobos/blob/master/std/zlib.d
Re: std.zip expand: memory allocation failed
Did you try the MmFile workaround? I did. I also pinpointed the problem, I use x86_mscoff to run dub and it's specific to that architecture selection. It's related to MapViewOfFileEx [1]. I still haven't found a way around it though. [1] https://stackoverflow.com/questions/12121843/mapviewoffileex-valid-lpbaseaddress
Re: std.zip expand: memory allocation failed
On Friday, 15 October 2021 at 20:41:36 UTC, Selim Ozel wrote: I am simply trying to unzip a compressed zip file slightly over 1GB. The de-compressed size is about 4 GB. The code is very similar to what's explained in the documentation [1] and it works for smaller files. Anyone has a solution? Memory mapping [2] previously solved some part of my issue but expand is still throwing memory allocation failure. Selim [1] https://dlang.org/phobos/std_zip.html [2] https://forum.dlang.org/thread/mfnleztnwrbgivjvz...@forum.dlang.org Did you try the MmFile workaround?
std.zip expand: memory allocation failed
I am simply trying to unzip a compressed zip file slightly over 1GB. The de-compressed size is about 4 GB. The code is very similar to what's explained in the documentation [1] and it works for smaller files. Anyone has a solution? Memory mapping [2] previously solved some part of my issue but expand is still throwing memory allocation failure. Selim [1] https://dlang.org/phobos/std_zip.html [2] https://forum.dlang.org/thread/mfnleztnwrbgivjvz...@forum.dlang.org
