Patches item #1675951, was opened at 2007-03-07 18:57 Message generated for change (Comment added) made by florianfesti You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1675951&group_id=5470
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Florian Festi (florianfesti) Assigned to: Nobody/Anonymous (nobody) Summary: [gzip] Performance for small reads and fix seek problem Initial Comment: This patch improves read performance of the gzip module. We have seen improvments from 20% from reading big blocks up to factor 50 for reading 4 byte blocks. Additionally this patch removes the need of seeking within the file with allows using streams like stdin as input. Details: The whole read(),readline() stack got rewritten. Core part is a new input buffer. It consists of a list of strings (self.buffer), an offset of what has already be consumed from the first string (self.pos) and the length of the (still consumable) buffer (self.bufferlen). This makes adding to and removing from the buffer cheap. It turned out that removing from the old buffer was breaking performance as for reading one byte the whole buffer had to be copied. For reading a 2k buffer in single bytes 2MB had to be copied. readline no longer uses read() but directly works on the buffer. This removes a whole layer of copying strings together. For removing the need of seeking a new readonly filelike class is used (PaddedFile). It just prepends a string to a file and uses the file's read method when the string got used up. There is probably still some space for tweaking when it comes to buffere sizes as we kept this simple. But the great performance improvments show that we can't be off that much. Performance test program and results are attached. ---------------------------------------------------------------------- >Comment By: Florian Festi (florianfesti) Date: 2007-03-15 18:43 Message: Logged In: YES user_id=1736372 Originator: YES Hm, works one Linux. Try this one File Added: gzip.py.diff ---------------------------------------------------------------------- Comment By: Lucas Malor (lucas_malor) Date: 2007-03-15 17:42 Message: Logged In: YES user_id=1403274 Originator: NO Excuse me, but I can't apply the patch. I have Windows XP without any SP and I tried to do the command patch -u gzip.py gzip.py.diff ---------------------------------------------------------------------- Comment By: Florian Festi (florianfesti) Date: 2007-03-09 10:24 Message: Logged In: YES user_id=1736372 Originator: YES I added checks to test_readline() and test_append(). I left the other read test untouched to keep some filename=filename coverage. BTW: I really forgot special thanks for Phil Knirsch who wrote the initial read() implementation and the performance test cases and did a lot of weaking and testing and without whom this patch would never have existed. File Added: test_gzip.py-noseek.diff ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2007-03-08 21:58 Message: Logged In: YES user_id=849994 Originator: NO The patch looks good, and the tests pass. Can you add a test that ensures that a seek() method is not necessary any more for the underlying stream? (closed #914340 which provided a similar patch as duplicate) ---------------------------------------------------------------------- Comment By: Florian Festi (florianfesti) Date: 2007-03-07 19:24 Message: Logged In: YES user_id=1736372 Originator: YES Added minor improvement of the unittest: Check for newlines in the readline() test code. File Added: test_gzip.py.diff ---------------------------------------------------------------------- Comment By: Florian Festi (florianfesti) Date: 2007-03-07 19:01 Message: Logged In: YES user_id=1736372 Originator: YES File Added: results.txt ---------------------------------------------------------------------- Comment By: Florian Festi (florianfesti) Date: 2007-03-07 18:58 Message: Logged In: YES user_id=1736372 Originator: YES File Added: test_gzip2.py ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1675951&group_id=5470 _______________________________________________ Patches mailing list Patches@python.org http://mail.python.org/mailman/listinfo/patches
