Patches item #1675951, was opened at 2007-03-07 18:57 Message generated for change (Comment added) made by florianfesti You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1675951&group_id=5470
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Florian Festi (florianfesti) Assigned to: Nobody/Anonymous (nobody) Summary: [gzip] Performance for small reads and fix seek problem Initial Comment: This patch improves read performance of the gzip module. We have seen improvments from 20% from reading big blocks up to factor 50 for reading 4 byte blocks. Additionally this patch removes the need of seeking within the file with allows using streams like stdin as input. Details: The whole read(),readline() stack got rewritten. Core part is a new input buffer. It consists of a list of strings (self.buffer), an offset of what has already be consumed from the first string (self.pos) and the length of the (still consumable) buffer (self.bufferlen). This makes adding to and removing from the buffer cheap. It turned out that removing from the old buffer was breaking performance as for reading one byte the whole buffer had to be copied. For reading a 2k buffer in single bytes 2MB had to be copied. readline no longer uses read() but directly works on the buffer. This removes a whole layer of copying strings together. For removing the need of seeking a new readonly filelike class is used (PaddedFile). It just prepends a string to a file and uses the file's read method when the string got used up. There is probably still some space for tweaking when it comes to buffere sizes as we kept this simple. But the great performance improvments show that we can't be off that much. Performance test program and results are attached. ---------------------------------------------------------------------- >Comment By: Florian Festi (florianfesti) Date: 2007-03-07 19:24 Message: Logged In: YES user_id=1736372 Originator: YES Added minor improvement of the unittest: Check for newlines in the readline() test code. File Added: test_gzip.py.diff ---------------------------------------------------------------------- Comment By: Florian Festi (florianfesti) Date: 2007-03-07 19:01 Message: Logged In: YES user_id=1736372 Originator: YES File Added: results.txt ---------------------------------------------------------------------- Comment By: Florian Festi (florianfesti) Date: 2007-03-07 18:58 Message: Logged In: YES user_id=1736372 Originator: YES File Added: test_gzip2.py ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1675951&group_id=5470 _______________________________________________ Patches mailing list Patches@python.org http://mail.python.org/mailman/listinfo/patches
