[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
STINNER Victor added the comment: See also issue #15612 for a possible optimization on StringIO (use _PyUnicodeWriter). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Changes by STINNER Victor victor.stin...@gmail.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Changes by Serhiy Storchaka storch...@gmail.com: -- assignee: - storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Serhiy Storchaka storch...@gmail.com added the comment: Hmm. Without the ability to reproduce this effect, it will be difficult for me to get rid of him. What times for StringIO, append/join, unpatched BytesIO? Do this happen with a little different numbers (n=1500, 1600,...,3000)? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Antoine Pitrou pit...@free.fr added the comment: Hmm. Without the ability to reproduce this effect, it will be difficult for me to get rid of him. It's under 64-bit Linux, Intel Core i5 CPU. Are you sure you're testing in non-debug mode? That said, the numbers under Windows suggest me that Eli's original idea (append and then join at the end) would be more robust. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Antoine Pitrou pit...@free.fr added the comment: By the way, here is Matt Mackall's take on realloc(): http://www.selenic.com/pipermail/mercurial-devel/2011-October/034988.html -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Antoine Pitrou pit...@free.fr added the comment: StringIO: $ ./python -m timeit -s import io; n=2000; d=['a'*n,'bb'*n,'ccc'*n]*1000 s=io.StringIO(); w=s.write for x in d: w(x) global y; y = s.getvalue() - Linux: 1000 loops, best of 3: 985 usec per loop - Windows: 100 loops, best of 3: 4.26 msec per loop Unpatched BytesIO: $ ./python -m timeit -s import io; n=2000; d=[b'a'*n,b'bb'*n,b'ccc'*n]*1000 s=io.BytesIO(); w=s.write for x in d: w(x) global y; y = s.getvalue() - Linux: 100 loops, best of 3: 2.44 msec per loop - Windows: 10 loops, best of 3: 38.4 msec per loop b''.join(): $ ./python -m timeit -s import io; n=2000; d=[b'a'*n,b'bb'*n,b'ccc'*n]*1000 l = list(d) global y; y = b''.join(l) - Linux: 1000 loops, best of 3: 821 usec per loop - Windows: 100 loops, best of 3: 4.09 msec per loop bytearray: $ ./python -m timeit -s import io; n=2000; d=[b'a'*n,b'bb'*n,b'ccc'*n]*1000 b = bytearray() for x in d: b += x global y; y = b - Linux: 1000 loops, best of 3: 834 usec per loop - Windows: 10 loops, best of 3: 37.8 msec per loop -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Serhiy Storchaka storch...@gmail.com added the comment: It's under 64-bit Linux, Intel Core i5 CPU. Are you sure you're testing in non-debug mode? I use 32-bit Linux. That said, the numbers under Windows suggest me that Eli's original idea (append and then join at the end) would be more robust. I agree, it would be more robust, but much more complex solution. I will try to implement this approach too. Victor Stinner in their experiments with formatting preferred overallocation approach (_PyUnicodeWriter), therefore, the solution probably will be a hybrid. However, even in itself the patch deserves attention. It not only significant speeds up writing under Linux, but also speeds up the initialization, which leads to faster reading. ./python -m timeit -s import io; d=(b'a'*99+b'\n')*1 s=io.BytesIO(d); r=s.readline while r(): pass Unpatched: 100 loops, best of 3: 5.92 msec per loop Patched: 100 loops, best of 3: 3.95 msec per loop I will try to preserve this advantage. By the way, here is Matt Mackall's take on realloc(): Note, that list also uses realloc() inside and therefore you get the same behavior with other constant factor. Actually, appending to a list less than sizeof(void*) bytes at a time you will run into this sooner. Perhaps this is the cause that append/join approach under Windows slower than under Linux. Thank you for measurements, this shows the scale of the issue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Serhiy Storchaka storch...@gmail.com added the comment: Ok, I understand. You're trying to make the getvalue() call cheaper, right? Yes, it saves up to half of time on large data (on Linux). It would be interesting to see the results of these microbenchmarks on Windows. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Changes by Eli Bendersky eli...@gmail.com: -- assignee: eli.bendersky - ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Antoine Pitrou pit...@free.fr added the comment: There seems to be a problem with the patch: when you store the getvalue() result somewhere (instead of discarding it), things get much slower: $ ./python -m timeit -s import io; n=2000; d=[b'a'*n,b'bb'*n,b'ccc'*n]*1000 s=io.BytesIO(); w=s.write for x in d: w(x) s.getvalue() 1000 loops, best of 3: 913 usec per loop $ ./python -m timeit -s import io; n=2000; d=[b'a'*n,b'bb'*n,b'ccc'*n]*1000 s=io.BytesIO(); w=s.write for x in d: w(x) global y; y = s.getvalue() 100 loops, best of 3: 4.67 msec per loop This does not happen without the patch. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Antoine Pitrou pit...@free.fr added the comment: Under Windows (64-bit Windows 7 on a VirtualBox VM), the patch increases performance slightly but not as much as under Linux: - before patch: C:\t\cpythonpc\VS9.0\amd64\python.exe -m timeit -s import io; n=2000; d=[b'a'* n,b'bb'*n,b'ccc'*n]*1000 s=io.BytesIO(); w=s.write for x in d: w(x) s.g etvalue() 10 loops, best of 3: 49.2 msec per loop - after patch: C:\t\cpythonpc\VS9.0\amd64\python.exe -m timeit -s import io; n=2000; d=[b'a'* n,b'bb'*n,b'ccc'*n]*1000 s=io.BytesIO(); w=s.write for x in d: w(x) s.g etvalue() 10 loops, best of 3: 41.7 msec per loop And the join() approach is 10x faster (!): C:\t\cpythonpc\VS9.0\amd64\python.exe -m timeit -s import io; n=2000; d=[b'a'* n,b'bb'*n,b'ccc'*n]*1000 b''.join(d) 100 loops, best of 3: 4.63 msec per loop ... which points to a much less optimized realloc() under Windows. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Serhiy Storchaka storch...@gmail.com added the comment: The patch updated. Fixed some errors, optimized initialization, added checks and comments. I think that now the patch is ready for review. -- Added file: http://bugs.python.org/file26461/bytesio_resized_bytes-2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Changes by Serhiy Storchaka storch...@gmail.com: Removed file: http://bugs.python.org/file26429/bytesio_resized_bytes.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Serhiy Storchaka storch...@gmail.com added the comment: There seems to be a problem with the patch: when you store the getvalue() result somewhere (instead of discarding it), things get much slower: This problem exists with the new patch? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Serhiy Storchaka storch...@gmail.com added the comment: Under Windows (64-bit Windows 7 on a VirtualBox VM), the patch increases performance slightly but not as much as under Linux: Thank you, Antoine. This is an expected result. And the join() approach is 10x faster (!): C:\t\cpythonpc\VS9.0\amd64\python.exe -m timeit -s import io; n=2000; d=[b'a'* n,b'bb'*n,b'ccc'*n]*1000 b''.join(d) 100 loops, best of 3: 4.63 msec per loop To be fair, test body should be: s=[]; w=s.append for x in d: w(x) b''.join(s) May be tuning resize strategy (overallocate not 1/8, but 1/4, 1/2, or 100%) can help. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Antoine Pitrou pit...@free.fr added the comment: There seems to be a problem with the patch: when you store the getvalue() result somewhere (instead of discarding it), things get much slower: This problem exists with the new patch? I think you can run the benchmark yourself (I ran it under Linux). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Serhiy Storchaka storch...@gmail.com added the comment: I think you can run the benchmark yourself (I ran it under Linux). I don't see any differences. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Antoine Pitrou pit...@free.fr added the comment: Well, with the latest patch I get: $ ./python -m timeit -s import io; n=2000; d=[b'a'*n,b'bb'*n,b'ccc'*n]*1000 s=io.BytesIO(); w=s.write for x in d: w(x) s.getvalue() 1000 loops, best of 3: 982 usec per loop $ ./python -m timeit -s import io; n=2000; d=[b'a'*n,b'bb'*n,b'ccc'*n]*1000 s=io.BytesIO(); w=s.write for x in d: w(x) global y; y = s.getvalue() 100 loops, best of 3: 4.79 msec per loop -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Antoine Pitrou pit...@free.fr added the comment: Here is a preliminary version of the patch. I don't understand the purpose of your patch. It just replaces a direct realloc() call with an indirect one (since _PyBytes_Resize() will call realloc() internally). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Antoine Pitrou pit...@free.fr added the comment: Here is a preliminary version of the patch. I don't understand the purpose of your patch. It just replaces a direct realloc() call with an indirect one (since _PyBytes_Resize() will call realloc() internally). Ok, I understand. You're trying to make the getvalue() call cheaper, right? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Changes by Meador Inge mead...@gmail.com: -- nosy: +meador.inge ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Eli Bendersky eli...@gmail.com added the comment:
I wonder if this is a fair comparison, Serhiy. Strings are unicode underneath,
so they have a large overhead per string (more data to copy around). Increasing
the length of the strings changes the game because due to PEP 393, the overhead
for ASCII-only Unicode strings is constant:
import sys
sys.getsizeof('a')
50
sys.getsizeof(b'a')
34
sys.getsizeof('a' * 1000)
1049
sys.getsizeof(b'a' * 1000)
1033
When re-running your tests with larger chunks, the results are quite
interesting:
$ ./python -m timeit -s import io; d=[b'a'*100,b'bb'*50,b'ccc'*50]*1000
b=io.BytesIO(); w=b.write for x in d: w(x) b.getvalue()
1000 loops, best of 3: 509 usec per loop
$ ./python -m timeit -s import io; d=['a'*100,'bb'*50,'ccc'*50]*1000
s=io.StringIO(); w=s.write for x in d: w(x) s.getvalue()
1000 loops, best of 3: 282 usec per loop
So, it seems to me that BytesIO could use some optimization!
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15381
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Eli Bendersky eli...@gmail.com added the comment: I'd like to take a shot at this, if Antoine doesn't mind. I'll prepare a patch for bytesio.c Question: what set of benchmarks would it be good to run to make sure this doesn't degrade the performance of BytesIO in various cases? -- assignee: - eli.bendersky ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Serhiy Storchaka storch...@gmail.com added the comment: I wonder if this is a fair comparison, Serhiy. I just used your examples. When re-running your tests with larger chunks, the results are quite interesting: Well, now I see, that BytesIO slower StringIO. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Serhiy Storchaka storch...@gmail.com added the comment: Here is a preliminary version of the patch. I am not sure that it is fully correct. Microbenchmark results: $ ./python -m timeit -s import io; n=100; d=['a'*n,'bb'*n,'ccc'*n]*1 s=io.StringIO(); w=s.write for x in d: w(x) s.getvalue() 10 loops, best of 3: 25.5 msec per loop $ ./python -m timeit -s import io; n=100; d=[b'a'*n,b'bb'*n,b'ccc'*n]*1 s=io.BytesIO(); w=s.write for x in d: w(x) s.getvalue() 10 loops, best of 3: 39.9 msec per loop $ ./python-patched -m timeit -s import io; n=100; d=[b'a'*n,b'bb'*n,b'ccc'*n]*1 s=io.BytesIO(); w=s.write for x in d: w(x) s.getvalue() 10 loops, best of 3: 26.1 msec per loop $ ./python -m timeit -s import io; n=1000; d=['a'*n,'bb'*n,'ccc'*n]*1000 s=io.StringIO(); w=s.write for x in d: w(x) s.getvalue() 100 loops, best of 3: 12.1 msec per loop $ ./python -m timeit -s import io; n=1000; d=[b'a'*n,b'bb'*n,b'ccc'*n]*1000 s=io.BytesIO(); w=s.write for x in d: w(x) s.getvalue() 10 loops, best of 3: 26.5 msec per loop $ ./python-patched -m timeit -s import io; n=1000; d=[b'a'*n,b'bb'*n,b'ccc'*n]*1000 s=io.BytesIO(); w=s.write for x in d: w(x) s.getvalue() 100 loops, best of 3: 13.6 msec per loop -- keywords: +patch Added file: http://bugs.python.org/file26429/bytesio_resized_bytes.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
New submission from Eli Bendersky eli...@gmail.com: From this pydev thread: http://mail.python.org/pipermail/python-dev/2012-July/120981.html BytesIO is actually missing an optimisation that is already used in StringIO: the StringIO C implementation uses a fragment accumulator internally, and collapses that into a single string object when getvalue() is called. BytesIO is still using the old resize-the-buffer-as-you-go strategy, and thus ends up repeatedly reallocating the buffer as the data sequence grows incrementally. It should be optimised to work the same way StringIO does (which is effectively the same way that the monkeypatched version works) -- components: Library (Lib) messages: 165715 nosy: eli.bendersky, ncoghlan, pitrou priority: normal severity: normal stage: needs patch status: open title: Optimize BytesIO to so less reallocations when written, similarly to StringIO type: performance versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Changes by Tshepang Lekhonkhobe tshep...@gmail.com: -- nosy: +tshepang ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Eli Bendersky eli...@gmail.com added the comment: This optimization for StringIO was done in issue #13149 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Serhiy Storchaka storch...@gmail.com added the comment: I am not see that BytesIO is slower than StringIO (both are about 30% slower than append/join). $ ./python -m timeit -s import io; d=[b'a'*10,b'bb'*5,b'ccc'*5]*1000 b=[]; w=b.append for x in d: w(x) b''.join(b) 1000 loops, best of 3: 966 usec per loop $ ./python -m timeit -s import io; d=['a'*10,'bb'*5,'ccc'*5]*1000 b=[]; w=b.append for x in d: w(x) ''.join(b) 1000 loops, best of 3: 918 usec per loop $ ./python -m timeit -s import io; d=[b'a'*10,b'bb'*5,b'ccc'*5]*1000 b=io.BytesIO(); w=b.write for x in d: w(x) b.getvalue() 1000 loops, best of 3: 1.22 msec per loop $ ./python -m timeit -s import io; d=['a'*10,'bb'*5,'ccc'*5]*1000 b=io.StringIO(); w=b.write for x in d: w(x) b.getvalue() 1000 loops, best of 3: 1.24 msec per loop -- nosy: +storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Changes by John O'Connor tehj...@gmail.com: -- nosy: +jcon ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15381] Optimize BytesIO to so less reallocations when written, similarly to StringIO
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15381 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
