[issue4194] default subprocess.Popen buffer size
Antoine Pitrou pit...@free.fr added the comment: The subprocess doc now has a note about buffering and performance issues, closing. -- resolution: - out of date status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4194 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4194] default subprocess.Popen buffer size
Changes by Skip Montanaro s...@pobox.com: -- nosy: -skip.montanaro ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4194 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4194] default subprocess.Popen buffer size
Shashwat Anand anand.shash...@gmail.com added the comment: Tested it on mac OSX (Snow leopard) Shashwat-Anands-MacBook-Pro:Desktop l0nwlf$ python2.5 popentest.py time with os.popen : 0.0342061519623 time with subprocess.Popen : 0.0421631336212 Shashwat-Anands-MacBook-Pro:Desktop l0nwlf$ python2.6 --version Python 2.6.1 Shashwat-Anands-MacBook-Pro:Desktop l0nwlf$ python2.6 popentest.py time with os.popen : 0.0282561779022 time with subprocess.Popen : 0.0366630554199 Python 2.5.4s os.popen was faster than subprocess.Popen, the same being the case with Python 2.6.1 I do not think it is a mac issue. -- nosy: +l0nwlf ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4194 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4194] default subprocess.Popen buffer size
Skip Montanaro s...@pobox.com added the comment: Looks good to me: tmp% python3.1 popentest.py time with os.popen : 0.035565 time with subprocess.Popen : 0.031796 tmp% python3.2 popentest.py time with os.popen : 0.03501 time with subprocess.Popen : 0.031168 tmp% python3.1 Python 3.1.1+ (release31-maint:77485M, Jan 13 2010, 19:53:41) [GCC 4.0.1 (Apple Inc. build 5490)] on darwin Type help, copyright, credits or license for more information. tmp% python3.2 Python 3.2a0 (py3k:77484:77485, Jan 13 2010, 19:49:33) [GCC 4.0.1 (Apple Inc. build 5490)] on darwin Type help, copyright, credits or license for more information. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4194 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4194] default subprocess.Popen buffer size
Antoine Pitrou pit...@free.fr added the comment: I just wanna say that buffering can be a problem when writing, but not when reading. If you read() from a buffered file, you still get the available contents immediately, you don't have to wait for the buffer to be full. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4194 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4194] default subprocess.Popen buffer size
Skip Montanaro [EMAIL PROTECTED] added the comment: Victor About Python3, os.popen() is more than two times faster (0.20 Victor sec vs 0.50 sec) than subprocess.Popen()! It's amazing because Victor popen() opens the standard output as unicode file whereas Victor Popen() creates a binary file! Another funny thing: os.popen() Victor calls subprocess.Popen() :-) The difference is just this Victor instruction: Victorstdout = io.TextIOWrapper(stdout) This is a known issue. The default for bufsize in os.popen is -1 (fully buffered? line buffered?). The default for bufsize in subprocess.Popen is 0 (unbuffered). I think it should have been changed but was voted down. I think the best you can do at this point is document the change, perhaps in the Replacing os.popen section. Skip ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4194 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4194] default subprocess.Popen buffer size
STINNER Victor [EMAIL PROTECTED] added the comment: Victor About Python3, os.popen() is more than two times faster (...) Victor The difference is just this instruction: Victorstdout = io.TextIOWrapper(stdout) This is a known issue. The default for bufsize in os.popen is -1 (fully buffered? line buffered?). The default for bufsize in subprocess.Popen is 0 (unbuffered). Wrong, it's not related to the buffer size. With Python3 trunk on Linux, os.popen time is ~0.10 sec whereas subprocess.Popen is ~0.25 sec. Change the buffer size of subprocess doesn't help: - (default) 0.25 - buffering = (-1): 0.25 - buffering = 1: 0.25 - buffering = 8192: 0.26 - buffering = 16384: 0.26 (it's a little big slower with a bigger buffer...) You get the same speed (than os.popen) using TextIOWrapper() adapter: [i for i in read_handle] = 0.25 sec [i for i in io.TextIOWrapper(read_handle)] = 0.10 sec WTF? Unicode is *FASTER* than raw bytes? ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4194 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4194] default subprocess.Popen buffer size
STINNER Victor [EMAIL PROTECTED] added the comment: About Python3, os.popen() is more than two times faster (0.20 sec vs 0.50 sec) than subprocess.Popen()! It's amazing because popen() opens the standard output as unicode file whereas Popen() creates a binary file! Another funny thing: os.popen() calls subprocess.Popen() :-) The difference is just this instruction: stdout = io.TextIOWrapper(stdout) -- nosy: +haypo ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4194 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4194] default subprocess.Popen buffer size
STINNER Victor [EMAIL PROTECTED] added the comment: Summary of unchanged Python (2.4 .. 2.7): * Mac: subprocess is 25 .. 50 times SLOWER than os.popen * Solaris : subprocess is 13 times SLOWER than os.popen * Windows XP : subprocess is 1.5 times FASTER than os.popen * Linux : (results are very close) With a different buffer size: * Solaris : Popen(bufsize=-1) is FASTER than os.popen() * Mac : Popen(bufsize=1) and Popen(bufsize=8192) are a little bit slower than os.popen(), but much FASTER than Popen(bufsize=0) Notes: - PyFile_SetBufSize(bufsize) does nothing if bufsize 0: keep system default (buffer of BUFSIZE bytes) - On Ubuntu Gutsy, system default (BUFSIZ) is 8192 bytes ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4194 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4194] default subprocess.Popen buffer size
Gregory P. Smith [EMAIL PROTECTED] added the comment: If anything for 2.6 lets just highlight this in the docs and mention that bufsize needs to be set to non-zero for good performance on many platforms such as Mac OS X and Solaris. We can consider changing the default for 2.7/3.1. 3.x having poor performance is pretty much another issue entirely of its own.. -- nosy: +gregory.p.smith ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4194 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4194] default subprocess.Popen buffer size
Changes by STINNER Victor [EMAIL PROTECTED]: -- title: Miserable subprocess.Popen performance - default subprocess.Popen buffer size ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4194 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
