[issue11314] Subprocess suffers 40% process creation overhead penalty

Antoine Pitrou added the comment: > That being said, I can't really submit a full-blown monitoring system > against this bug, Perhaps you can take a look at Shinken, it is a Nagios-lookalike written in Python. -- ___ Python tracker

[issue11314] Subprocess suffers 40% process creation overhead penalty

Aaron Sherman added the comment: I think it's still safe to say that high performance applications which need to create many hundreds or thousands of children (e.g. large monitoring systems) will still need another solution that isn't subprocess. That being said, you're right that no one is g

[issue11314] Subprocess suffers 40% process creation overhead penalty

STINNER Victor added the comment: Benchmark on subprocess with a less trivial example. Run 100x python -c pass: 8.63 sec without my patch, 8.53 sec with my patch => only 1% faster, so the patch is just useless on a real world example. Finally, I think that there is just nothing to do on Pytho

[issue11314] Subprocess suffers 40% process creation overhead penalty

Charles-Francois Natali added the comment: > So, even though implemented in C, the file descriptor closing logic is still > quite costly! Yes, see this recent issue: http://bugs.python.org/issue11284 In the reporter's case, it's much worse, because FreeBSD (at least the version he's using) h

[issue11314] Subprocess suffers 40% process creation overhead penalty

Charles-Francois Natali added the comment: > pitrou> I think your analysis is wrong. These mmap() calls are for > pitrou> anonymous memory, most likely they are emitted by the libc's > pitrou> malloc() to get some memory from the kernel. In other words > pitrou> they will be blazingly fast. > >

[issue11314] Subprocess suffers 40% process creation overhead penalty

STINNER Victor added the comment: > os.popen is 41% is slower than subprocess: I suppose that it > is the usage of stdout=PIPE (creation of the pipe) which make > it slower Oh no, it's because os.popen() calls subprocess.Popen() with shell=True: the overhead is the shell. Nothing wrong here.

[issue11314] Subprocess suffers 40% process creation overhead penalty

STINNER Victor added the comment: pitrou> Victor, your patch doesn't even apply on 3.x. pitrou> That code doesn't exist anymore... subprocess.Popen() does still read errpipe_read, but using a buffer of 50,000 bytes instead of 1 MB (the traceback is not more send to the parent process). Benc

[issue11314] Subprocess suffers 40% process creation overhead penalty

Antoine Pitrou added the comment: Interestingly, reducing from the max open file descriptors from 8192 (my default) to 512 halves the runtime and solves the regression: $ ulimit -n 512 $ ./python bench_subprocess.py pid: 31631 Time: 4903.8 ms So, even though implemented in C, the file descr

[issue11314] Subprocess suffers 40% process creation overhead penalty

Antoine Pitrou added the comment: Here is a quick profile under 3.x: Time: 10178.0 ms 320812 function calls (320809 primitive calls) in 10.182 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 15.9370.0015

[issue11314] Subprocess suffers 40% process creation overhead penalty

Antoine Pitrou added the comment: Benchmark under 3.x (without obsolete patch): - fork + execv + waitpid: 4794.4 ms - os.popen: 19792.9 ms - subprocess.popen: 10152.1 ms Benchmark under 2.x (without patch:) - fork + execv + waitpid: 4292.7 ms - os.popen: 12697.6 ms - subprocess.popen: 5496.3 ms

[issue11314] Subprocess suffers 40% process creation overhead penalty

Antoine Pitrou added the comment: Victor, your patch doesn't even apply on 3.x. That code doesn't exist anymore... -- ___ Python tracker ___

[issue11314] Subprocess suffers 40% process creation overhead penalty

STINNER Victor added the comment: Quick benchmark: - fork(), execv(), waitpid(): 19081.1 ms - os.popen(): 25769.8 ms - subprocess.Popen(): 40025.8 ms - subprocess.Popen() patched: 26155.2 ms I tested Python 2.7 (debug build) on Linux (Debian Sid) with a Pentium 4 (Hyperthreading, 2 fake co

[issue11314] Subprocess suffers 40% process creation overhead penalty

Antoine Pitrou added the comment: > subprocess_errpipe_buffer.patch: Patch for 2.7 which improve by 30% > the dummy benchmark (run /bin/false 10,000 times). It avoids the > creation of the 1 MB buffer: start with a buffer of a single byte > (just to check if there is data or if the pipe was clos

[issue11314] Subprocess suffers 40% process creation overhead penalty

STINNER Victor added the comment: subprocess_errpipe_buffer.patch: Patch for 2.7 which improve by 30% the dummy benchmark (run /bin/false 10,000 times). It avoids the creation of the 1 MB buffer: start with a buffer of a single byte (just to check if there is data or if the pipe was closed).

[issue11314] Subprocess suffers 40% process creation overhead penalty

Charles-Francois Natali added the comment: > Aaron Sherman added the comment: > > "That's why I asked for absolute numbers for the overhead difference." > > Did you not follow the link in my first post? I got pretty detailed, there. > By the way, strace has a '-T' flag that can be used to meas

[issue11314] Subprocess suffers 40% process creation overhead penalty

Aaron Sherman added the comment: "That's why I asked for absolute numbers for the overhead difference." Did you not follow the link in my first post? I got pretty detailed, there. "os.popen just calls the popen(3) library call, which just performs a fork/execve and some dup/close in between.

[issue11314] Subprocess suffers 40% process creation overhead penalty

Gregory P. Smith added the comment: fyi - while I still need to update it now that 3.2 final has been released, the http://code.google.com/p/python-subprocess32/ project has a backport of the _posixsubprocess stuff from python 3.2 for use on Python 2.x. -- ___

[issue11314] Subprocess suffers 40% process creation overhead penalty

Antoine Pitrou added the comment: Okay, thanks to Charles-François' measurements, we can deduce that each subprocess launch is at most 0.3ms of user CPU time and 1.2ms of system CPU time. IMO that's not a real problem. -- ___ Python tracker

[issue11314] Subprocess suffers 40% process creation overhead penalty

Antoine Pitrou added the comment: > But in the two scenarios I mentioned (monitoring and Web services such > as CGI, neither of which is particularly rare), this is going to make > quite a lot of difference That's why I asked for absolute numbers for the overhead difference. A percentage does

[issue11314] Subprocess suffers 40% process creation overhead penalty

Changes by Giampaolo Rodola' : -- nosy: +giampaolo.rodola ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http:

[issue11314] Subprocess suffers 40% process creation overhead penalty

Charles-Francois Natali added the comment: > If you stop to think about it, though, this is actually a shockingly huge > percent increase. In any process creation scenario I'm familiar with, its > overhead should be so small that you could bump it up several orders of > magnitude and still no

[issue11314] Subprocess suffers 40% process creation overhead penalty

Aaron Sherman added the comment: "Python 3.2 has a _posixsubprocess: some parts of subprocess are implemented in C. Can you try it?" I don't have a Python 3 installation handy, but I can see what I can do tomorrow evening to get one set up and try it out. "disagree with the idea that spawnin

[issue11314] Subprocess suffers 40% process creation overhead penalty

Antoine Pitrou added the comment: I think your analysis is wrong. These mmap() calls are for anonymous memory, most likely they are emitted by the libc's malloc() to get some memory from the kernel. In other words they will be blazingly fast. I would suggest you try to dig deeper. For example

[issue11314] Subprocess suffers 40% process creation overhead penalty

STINNER Victor added the comment: Python 3.2 has a _posixsubprocess: some parts of subprocess are implemented in C. Can you try it? Python 3.2 uses also pipe2(), if available, to avoid the extra fcntl(4, F_GETFD)+fcntl(4, F_SETFD, FD_CLOEXEC). I suppose that the pipe and mmap(NULL, 1052672,

[issue11314] Subprocess suffers 40% process creation overhead penalty

Changes by Antoine Pitrou : -- nosy: +gregory.p.smith ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://ma

[issue11314] Subprocess suffers 40% process creation overhead penalty

New submission from Aaron Sherman : I wrote some code a while back which used os.popen. I recently got a warning about popen being deprecated so I tried a test with the new subprocess module. In that test, subprocess.Popen appears to have a 40% process creation overhead penalty over os.popen,