Patches item #647083, was opened at 2002-12-02 10:38 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=300103&aid=647083&group_id=103
Category: command line scripts Group: None Status: Open Resolution: None Priority: 5 Submitted By: John-Mark Gurney (jmg) Assigned to: Nobody/Anonymous (nobody) Summary: compress dies w/ MemoryError Initial Comment: If you have a large mailing list, it is very easy to quickly exhast memory in many conditions. nightly_gzip script tries to load the entire file into memory before compressing it, not only is this terribly ineffecient, it also means you are limited to file sizes that can fit in memory. Some machines also further restrict how much memory each process can access at a time. If that limit is 512megs, and you have a 400+meg file, compress will die. Solution: read/write the file in blocks. ---------------------------------------------------------------------- >Comment By: John-Mark Gurney (jmg) Date: 2002-12-02 16:22 Message: Logged In: YES user_id=27410 Well, from the looks of it, it is best to use the gzip module if we want to be able to use gunzip (cmd line) to uncompress the file. zlib only provides for compression objects, and doesn't say if the output is compatible for direct input to the gunzip program. I also went with the easiest modification, least source modifications. Now that I'm thinking about it, it might of been implemented as a call to writelines(xreadlines()) w/o having to use my own while loop resulting in even less modification of the code. The only problem is that this may produce very long lines for binary code. Also, this requires more scanning for the new lines, while the block while does not have to scan the input. To bad write doesn't support file objects to copy in a block manner. so the code could simply be: outfp.write(infp) P.S. I forgot to mention in the body that this work is supported by Vernier Networks, Inc. ---------------------------------------------------------------------- Comment By: Bryan Fullerton (fehwalker) Date: 2002-12-02 10:53 Message: Logged In: YES user_id=660772 would it make more sense to do this with zlib routines on a stream than with gzip on blocks? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=300103&aid=647083&group_id=103 _______________________________________________ Mailman-Developers mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-developers
