[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[STINNER Victor]

STINNER Victor  added the comment:

> I'm still seeing hangs with subprocess.run() in Python 3.7.4

That's not surprising, the fix has been pushed at 2019-09-11. Python 3.7.5 will 
include the fix and it will be released soon:
https://www.python.org/dev/peps/pep-0537/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[Tony Cappellini]

Tony Cappellini  added the comment:

Using Python 3.7.4, I'm calling subprocess.run() with the following arguments. 
.run() still hangs even though a timeout is being passed in.

subprocess.run(cmd_list,
stdout=subprocess.PIPE,   
stderr=subprocess.STDOUT,   
shell=False,   timeout=timeout_val, 
  check=True,   
universal_newlines=True)



cmd_list contains the name of the bash script below, which is
./rescan.sh

--
#!/usr/bin/bash

echo Rescanning system for PCIe devices

echo "Rescan device"
echo 1 > /sys/bus/pci/rescan

sleep 5

if [ `lspci | grep -ic "Non-Volatile memory controller"` -gt 0 ]
then
echo "Device Detected after Rescan"
else 
echo "Device NOT detected after Rescan"
exit 1
fi 

echo Rescan Done


This script is scanning for NVME SSDs, so duplicating the issue is not as 
straightforward as submitting a python script.

The OS is CentOS 7. 

uname  -a shows
 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 
GNU/Linux

I know the Kernel is old, but we have a restriction against updating it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[Tony Cappellini]

Tony Cappellini  added the comment:

I'm still seeing hangs with subprocess.run() in Python 3.7.4
Unfortunately, it involves talking to an NVME SSD on Linux, so I cannot
easily submit code to duplicate it.

--
nosy: +cappy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[STINNER Victor]

STINNER Victor  added the comment:

> On Windows, the following pattern _can_ hang: (...)

I created bpo-38207 "subprocess: On Windows, Popen.kill() + Popen.communicate() 
is blocking until all processes using the pipe close the pipe" to track this 
issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[Gregory P. Smith]

Gregory P. Smith  added the comment:

Thanks.  I believe this issue is fixed but you've identified follow-on issues.  
lets follow up on those in their own bugs.

--
resolution:  -> fixed
stage: patch review -> commit review
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[STINNER Victor]

STINNER Victor  added the comment:

On Windows, the following pattern _can_ hang:

proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
try:
  return proc.communicate(timeout=1.0)
except TimeoutExpired:
  proc.kill()
  return proc.communicate() # <== HERE

Even if the first child process is killed, communicate() waits until the pipe 
is closed. If the child process spawned a 3rd process before being killed, the 
second .communicate() calls hangs until the 3rd process exit or close its 
stdout.

I'm not sure if subprocess.run() should do anything about this case, but it was 
at least for your information.

I'm fighting against this issue in bpo-37531.

IMHO it's an issue of the Windows implementation of Popen.communicate(): it's 
implemented with a blocking call to stdout.read() run in a thread. The thread 
cannot be interrupted in any way and will until complete once stdout is closed.

Again, if the direct child process spawned other processes, stdout is only 
closed in the main parent process once all child processes exit or at least 
closed their stdout.

Maybe another issue should be opened to avoid blocking with the following 
pattern on Windows:

proc.kill()
proc.communicate()

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[miss-islington]

miss-islington  added the comment:


New changeset 5fe153c21de020f37712910b0dcce93341bba272 by Miss Islington (bot) 
in branch '3.7':
bpo-37424: Avoid a hang in subprocess.run timeout output capture (GH-14490)
https://github.com/python/cpython/commit/5fe153c21de020f37712910b0dcce93341bba272


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[miss-islington]

miss-islington  added the comment:


New changeset 872c85a1796290baef89a20dbde819c4da45830c by Miss Islington (bot) 
in branch '3.8':
bpo-37424: Avoid a hang in subprocess.run timeout output capture (GH-14490)
https://github.com/python/cpython/commit/872c85a1796290baef89a20dbde819c4da45830c


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[miss-islington]

Change by miss-islington :


--
pull_requests: +15539
pull_request: https://github.com/python/cpython/pull/15898

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[miss-islington]

Change by miss-islington :


--
keywords: +patch
pull_requests: +15538
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/15897

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[Thomas Wouters]

Thomas Wouters  added the comment:


New changeset 580d2782f70f8e0bed7ec20abb03d740cb83b5da by T. Wouters (Gregory 
P. Smith) in branch 'master':
bpo-37424: Avoid a hang in subprocess.run timeout output capture (GH-14490)
https://github.com/python/cpython/commit/580d2782f70f8e0bed7ec20abb03d740cb83b5da


--
nosy: +twouters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[Gregory P. Smith]

Change by Gregory P. Smith :


--
assignee:  -> gregory.p.smith
resolution: duplicate -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[Gregory P. Smith]

Change by Gregory P. Smith :


--
stage: resolved -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[Gregory P. Smith]

Gregory P. Smith  added the comment:

Issue 30154 that I've marked as a duplicate demonstrates this problem without 
using shell=True.  The solution I proposed handles that via the additional 
small timeout on the cleanup side, but still has the caveat that the grandchild 
processes keep running unless the caller used start_new_session=True.

See the PR.

We cannot reasonably determine when start_new_session=True should be a default 
behavior.  And I worry that doing it when it should not be will cause 
unexpected new problems with existing code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[Gregory P. Smith]

Change by Gregory P. Smith :


--
pull_requests: +14307
pull_request: https://github.com/python/cpython/pull/14490

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[haizaar]

haizaar  added the comment:

Thanks for looking at it.

My original code had "tar" running, which is a child of the shell as well... I 
assume running exec in the shell may help somewhat, but not a cure obviously.

I'm all for killing the process group. "Run something and get it's about" 
should be simple enough without requiring a programmer to know all POSIX 
process semantics.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[Martin Panter]

Martin Panter  added the comment:

Same thing going on as in Issue 30154. The shell is probably spawning the 
“sleep” command as a child process (grandchild of Python), and waiting for it 
to exit. When Python times out, it will kill the shell process, leaving the 
grandchild as an orphan. The “sleep” process will still be running and probably 
holds the “stdout” and/or “stderr” pipes open, and Python will wait 
indefinitely to be sure it has captured all the output to those pipes.

Also see Issue 26534 proposes APIs to kill a process group rather than the 
single child process.

--
nosy: +martin.panter
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> subprocess.run with stderr connected to a pipe won't timeout 
when killing a never-ending shell commanad

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

On mac this exits immediately with 0.1 seconds as timeout but reproducible on 
Ubuntu with master.

--
nosy: +gregory.p.smith, xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37424] subprocess.run timeout does not function if shell=True and capture_output=True

[haizaar]

New submission from haizaar :

Consider the following:

subprocess.run('sleep 10', shell=True, timeout=.1, capture_output=True)

It should raise after 0.1 seconds, but it does not - it waits 10 seconds till 
sleep finishes and only then raises "subprocess.TimeoutExpired: Command 'sleep 
10' timed out after 0.1 seconds"

Removing 'capture_output=True' or converting command string to list (and 
removing shell=True) makes it work.

I'm using Python 3.7.3 on Ubuntu 16.04. Reproduces on official docker Python 
3.7.3 image alpine3.8.

--
components: Library (Lib)
messages: 346712
nosy: haizaar
priority: normal
severity: normal
status: open
title: subprocess.run timeout does not function if shell=True and 
capture_output=True
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com