[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Change by STINNER Victor : -- nosy: -vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Eryk Sun added the comment: I'm changing this to a documentation issue and removing the Windows component. The documentation doesn't make it clear that communicate() may block indefinitely (in both POSIX and Windows) even after the process has terminated. As currently implemented, this

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Change by Eryk Sun : -- components: +Library (Lib), Windows -Interpreter Core nosy: +paul.moore, steve.dower, tim.golden, zach.ware versions: +Python 3.10, Python 3.8, Python 3.9 -Python 3.6, Python 3.7 ___ Python tracker

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

STINNER Victor added the comment: See also bpo-38207: "subprocess: Popen.kill() + Popen.communicate() is blocking until all processes using the pipe close the pipe". -- nosy: +vstinner ___ Python tracker

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Eryk Sun added the comment: > when I said "also with close_fds=True", I meant that I tried > WITHOUT overriding stdin, stdout, and stderr AND setting > close_fds=True, but it didn't work. Console applications (e.g. python.exe) duplicate their standard handles into a child console process

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Leonardo Francalanci added the comment: That works! But when I said "also with close_fds=True", I meant that I tried WITHOUT overriding stdin, stdout, and stderr AND setting close_fds=True, but it didn't work. What worked was not overriding stdin/out/err and adding os.set_inheritable(0,

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Eryk Sun added the comment: > I tried with stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, > stderr=subprocess.DEVNULL As I said previously, you also need to make the current standard handles non-inheritable. Pending issue 19764, in 3.7 you'll be able to override stdin, stdout, and

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Leonardo Francalanci added the comment: can I at least change the call to: subprocess.run('cmd /S /C waitfor g /t 200', shell=False, timeout=4) in any way to avoid the problem? I tried with stdin=subprocess.DEVNULL,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL; also with

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Leonardo Francalanci added the comment: I'm really sorry, you are 100% correct: it blocks on the pipe (my tests killed the process tree before reading from the pipes). Still, I think there should be a way to actually read the output also in this case... works for me when I kill the whole

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Eryk Sun added the comment: > Now I can read from the "R" pipes with 0 problems (getting the error > and output streams) The pipe handles are inherited by waitfor.exe, just as before. You'll find that reading to EOF will block just as it does when using subprocess.PIPE with communicate().

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Leonardo Francalanci added the comment: I have a workaround, and I guess this means there's a bug in the current implementation of stdout/stderr=subprocess.PIPE; if I open my own pipes instead of using subprocess.PIPE everything seems to work (provided I close the pipe before reading from

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Eryk Sun added the comment: > the process terminates, but "communicate" doesn't exit. It doesn't > say "communicate will hang as long as the pipes are open". I think the problem with leaked handles does warrant a note. It can be a difficult problem to diagnose otherwise. If communicate()

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Martin Panter added the comment: . Isn’t your use of “cmd” similar enough to shell=True? I.e. isn’t that a “cmd” parent process spawning a “waitfor” child? If your 4 s “subprocess.run” call times out, does it kill the “waitfor” process, or leave it running? Could the “waitfor” process write

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Changes by R. David Murray : -- nosy: +r.david.murray ___ Python tracker ___ ___

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Leonardo Francalanci added the comment: (forgot: Issue 26534 is about shell=True, I use shell=False, right?) -- ___ Python tracker ___

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Leonardo Francalanci added the comment: thank you for your replies! I used waitfor because it's the simplest and shortest way to have reproducible code. The issue I'm having is obviously not with waitfor, but with another exe, but this doesn't change the fact that I need a way to exit as soon

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Martin Panter added the comment: I’m not familiar with Windows commands or processes enough, but going by Erik’s comment, this sounds similar to Issue 26534 and Issue 30154. Doesn’t your “waitfor” command inherit the same parent process’s stdout etc pipes? -- nosy: +martin.panter

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Leonardo Francalanci added the comment: The called "C:\\Program Files (x86)\\Anaconda3\\Python.exe" process exits after 4 seconds. The reason why it ends shouldn't matter, right? I expect that a call to communicate should exit as soon as the called process is not running anymore. I don't hold

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Eryk Sun added the comment: I don't see a bug here. The inner run() kills cmd.exe after 4 seconds. But the waitfor.exe process waits for 200 seconds and has handles for the pipes. The OS won't close out the pipes as long as there are potential writers, so the outer communicate() waits the

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

Changes by Leonardo Francalanci : -- components: +Interpreter Core type: -> behavior ___ Python tracker ___

[issue31447] proc communicate not exiting on python subprocess timeout using PIPES

New submission from Leonardo Francalanci: the script below (a python process is called, which calls a waitfor cmd with a timeout of 4 seconds) is supposed to end after 4 seconds. But instead proc.communicate stops after the 20 seconds timeout. Everything works 100% ok if I remove the