[issue46454] '0 -> /dev/null' is lost

[Eryk Sun]

Change by Eryk Sun :


--
Removed message: https://bugs.python.org/msg411287

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[Eryk Sun]

Change by Eryk Sun :


--
Removed message: https://bugs.python.org/msg411776

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[Eryk Sun]

Eryk Sun  added the comment:

> stdin is closed
> os.open() is called and creates a new fd=0
> a call to os.dup2(fd, 0) is made but is a noop

The dup2() silent noop case is a problem, but not the problem that's reported 
in msg43. 

The two examples for this issue are fd.py and fd2.py. In fd.py, "/dev/null" is 
opened, and then, if the fd isn't 0, the fd is duped to inheritable fd 0. In 
fd2.py, fd 0 is closed, and then "/dev/null" is opened as fd 0, which isn't 
inheritable in Python 3.

I think the only problems regarding fd2.py are cross-platform inconsistency 
and/or documentation. If a similar example is tested in Windows it actually 
works, assuming os.devnull is used and that "sleep.exe" is available (e.g. from 
MSYS2). The child's stdin will be the same file as the parent's stdin.

In regard to the standard files, the subprocess documentation states the 
following:

 With the default settings of None, no redirection will occur; 
 the child’s file handles will be inherited from the parent.

 If close_fds is true, all file descriptors except 0, 1 and 2 
 will be closed before the child process is executed.

The above applies only to POSIX. It's not wrong in that case, but it should be 
emphasized that any of the standard file descriptors that's not overridden has 
to be inheritable in POSIX. Also, if overriding to a specific file, the file 
descriptor must be inheritable in POSIX. By default, file descriptors for 
opened files are not inheritable, so the subprocess documentation should 
reference os.set_inheritable(). 

For Windows it says the following:

 if close_fds is true then no handles will be inherited by
 the child process unless explicitly passed in the handle_list
 element of STARTUPINFO.lpAttributeList, or by standard handle 
 redirection.

That's mostly right, except the last part about standard handle redirection is 
vague. If close_fds is true, none of the standard handles of the current 
process is ever inherited in Windows. They could be in principle, but 
subprocess wasn't designed that way. When close_fds is true, and the standard 
handles aren't overridden, subprocess relies on the OS to duplicate (not 
inherit) the standard handles to the child when spawning a console application 
(e.g. "python.exe", but not "pythonw.exe"). If at least one is overridden, 
subprocess duplicates all three as inheritable handles via _make_inheritable(). 
(This is a choice. It doesn't have to do this. It could simply ensure that its 
own pipe/null files are inheritable.) Thus, when close_fds is true it never 
matters whether the current standard handles are inheritable, or whether a 
specified override is inheritable. This behavior is in stark contrast to how 
subprocess works in POSIX.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[Brett Randall]

Brett Randall  added the comment:

docker run -i --rm python:3.9 <<-EOF

import os

print(os.get_inheritable(0))

fd_null_before_close = os.open("/dev/null", os.O_RDWR)
os.close(0)
fd_null = os.open("/dev/null", os.O_RDWR)

print(fd_null_before_close)
print(fd_null)

os.dup2(fd_null, 0)
print(os.get_inheritable(0))

os.dup2(fd_null_before_close, 0)
print(os.get_inheritable(0))

EOF
True
3
0
False
True

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[Brett Randall]

Brett Randall  added the comment:

For the possible benefit of future readers of this issue, the sequence and 
behaviour here which may cause confusion:

- say stdin is closed, perhaps along with stdout and stderr via a call to 
os.closerange(0, 3) or otherwise
- os.open() is called and creates a new fd=0, which will always be 0 since 0 is 
now available and lowest.  Per PEP 446 this fd is non-inheritable.
- a call to os.dup2(fd, 0) is made but is a noop, since fd=0 and dup2() ignores 
copy-to-same.  This behaviour may go unnoticed - it may not be noticed that the 
new fd=0 (it should always be this), or that dup2() has this noop behaviour.  
Per PEP 446, dup2() still creates inheritable by-default, so the caller may 
expect that they have just assigned an inheritable fd to stdin, instead of a 
noop.
- subprocess() is called and the subprocess does not have stdin/0 fd assigned, 
since it was not inheritable.

It seems the main way to avoid this is to os.open() the replacement fd _before_ 
closing stdin/0, in which case it will receive an fd >= 3.  The dup2(fd, 0) 
call will then work and the resulting fd 0 will be inheritable, including 
by-default by subprocess() processes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[Brett Randall]

Change by Brett Randall :


--
nosy: +javabrett

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[Eryk Sun]

Eryk Sun  added the comment:

> If some one closes fd 0, then he reopens it. it will not be inherited.

In Windows, when a console process spawns a child console process without 
enabling handle inheritance -- e.g. subprocess.Popen(['python.exe']) -- the OS 
will manually duplicate (not inherit) the standard handles to the child. It 
doesn't matter in this case that a standard handle isn't inheritable. 

The latter also doesn't matter when any of the standard handles is overridden 
by a Popen() call in Windows -- e.g. subprocess.Popen(['python.exe'], 
stderr=DEVNULL). All of the standard handle values have to be overridden 
together. Popen() uses this as an opportunity to duplicate an inheritable 
handle for each that's not overridden, instead of just copying the handle value.

On the other hand, if subprocess.Popen() is called in Windows with handle 
inheritance enabled and without overriding the standard handles -- e.g. 
subprocess.Popen(['python.exe'], close_fds=False) -- then the standard handles 
should be inheritable. If any of the standard handles isn't inheritable, then 
at best the standard handle value in the child will be invalid or used by a 
handle for a kernel object type other than a file (e.g. process, event). By 
coincidence, however, a file open in the child could reuse the standard handle 
value, which can lead to buggy behavior.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[licunlong]

licunlong  added the comment:

> File descriptors 0, 1, 2 are special: stdin, stdout and stderr file 
> descriptors. You should use stdin, stdout, and stderr parameter of subprocess.

I finally find why `fd 0` is inherited. The reason is that os.dup2() has a 
parameter "inheritable" which defaults to True. Not because it's special stdin.

I still think this is a bug. If some one closes fd 0, then he reopens it. it 
will not be inherited.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[STINNER Victor]

STINNER Victor  added the comment:

> this fd 0 `is` inherited.

File descriptors 0, 1, 2 are special: stdin, stdout and stderr file 
descriptors. You should use stdin, stdout, and stderr parameter of subprocess.

Sorry but the bug tracker is not a good place to ask questions about how to use 
Python, it's only to report bugs.

Python works as expected.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[licunlong]

licunlong  added the comment:

But one more question.
Please take a look at fd.py. In this file, we first close fd 0, then open 
/dev/null as fd_null, and dump fd_null to fd 0. After we fork new process, this 
fd 0 `is` inherited.
This seems to be a bug?

--
nosy: +Geass-LL

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[STINNER Victor]

STINNER Victor  added the comment:

That's not a bug, but a deliberate choice. In Python 3, file descriptors are no 
longer inherited by default by child processes (fork+exec). Jelle is right, it 
was changed by my PEP 446.

You must use pass_fds=[fd] parameter of subprocess.

Or at least, make the file descriptor inheritable (worse solution).

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[Jelle Zijlstra]

Jelle Zijlstra  added the comment:

This sounds like a consequence of PEP 446.

--
nosy: +Jelle Zijlstra

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[tongxiaoge]

Change by tongxiaoge :


Added file: https://bugs.python.org/file50573/python3.10.0-out.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[tongxiaoge]

Change by tongxiaoge :


Added file: https://bugs.python.org/file50574/python3.4.6-out.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[tongxiaoge]

Change by tongxiaoge :


Added file: https://bugs.python.org/file50572/python2.7.13-out.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[tongxiaoge]

Change by tongxiaoge :


Added file: https://bugs.python.org/file50571/fd2.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[tongxiaoge]

Change by tongxiaoge :


Added file: https://bugs.python.org/file50570/fd.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46454] '0 -> /dev/null' is lost

[tongxiaoge]

New submission from tongxiaoge :

I found a problem in the environment of python2 and python3, as follows:

in python2:
```
VM-0-13-suse:~ # python2 -V
Python 2.7.13
VM-0-13-suse:~ #
VM-0-13-suse:~ # ps -aux|grep python2
root 29414  0.1  0.7  37096  6632 pts/3S+   19:41   0:00 python2 fd2.py
root 29442  0.0  0.1  10540  1656 pts/5S+   19:41   0:00 grep 
--color=auto python2
VM-0-13-suse:~ #
VM-0-13-suse:~ # ps -aux|grep sleep
root 29415  0.0  0.1   5760  1256 pts/3S+   19:41   0:00 sleep 12345
root 29451  0.0  0.1  10540  1648 pts/5S+   19:41   0:00 grep 
--color=auto sleep
VM-0-13-suse:~ #
VM-0-13-suse:~ # ls -l /proc/29415/fd
total 0
lrwx-- 1 root root 64 Jan 21 19:41 0 -> /dev/null
lrwx-- 1 root root 64 Jan 21 19:41 1 -> /dev/pts/3
lrwx-- 1 root root 64 Jan 21 19:41 2 -> /dev/pts/3
VM-0-13-suse:~ #
VM-0-13-suse:~ # kill -9 29415 29414
VM-0-13-suse:~ #
VM-0-13-suse:~ # ps -aux|grep python2
root 29551  0.1  0.7  37096  6632 pts/3S+   19:42   0:00 python2 fd.py
root 29564  0.0  0.1  10540  1608 pts/5S+   19:42   0:00 grep 
--color=auto python2
VM-0-13-suse:~ # ps -aux|grep sleep
root 29552  0.0  0.1   5760  1260 pts/3S+   19:42   0:00 sleep 12345
root 29576  0.0  0.1  10540  1628 pts/5S+   19:42   0:00 grep 
--color=auto sleep
VM-0-13-suse:~ #
VM-0-13-suse:~ # ls -l /proc/29552/fd
total 0
lrwx-- 1 root root 64 Jan 21 19:42 0 -> /dev/null
lrwx-- 1 root root 64 Jan 21 19:42 1 -> /dev/pts/3
lrwx-- 1 root root 64 Jan 21 19:42 2 -> /dev/pts/3
lrwx-- 1 root root 64 Jan 21 19:42 3 -> /dev/null
VM-0-13-suse:~ #
```

in python3.4.6:
```
VM-0-13-suse:~ # python3 -V
Python 3.4.6
VM-0-13-suse:~ # ps -aux|grep python3
root 29086  0.1  0.9  33628  8136 pts/3S+   19:39   0:00 python3 fd2.py
root 29143  0.0  0.1  10540  1616 pts/5S+   19:39   0:00 grep 
--color=auto python3
VM-0-13-suse:~ #
VM-0-13-suse:~ # ps -aux|grep sleep
root 29087  0.0  0.1   5760  1284 pts/3S+   19:39   0:00 sleep 12345
root 29164  0.0  0.1  10540  1620 pts/5S+   19:39   0:00 grep 
--color=auto sleep
VM-0-13-suse:~ #
VM-0-13-suse:~ # ls -l /proc/29087/fd
total 0
lrwx-- 1 root root 64 Jan 21 19:39 1 -> /dev/pts/3
lrwx-- 1 root root 64 Jan 21 19:39 2 -> /dev/pts/3
VM-0-13-suse:~ #
VM-0-13-suse:~ # kill -9 29087 29086
VM-0-13-suse:~ #
VM-0-13-suse:~ # ps -aux|grep python3
root 29257  0.3  0.9  33628  8076 pts/3S+   19:40   0:00 python3 fd.py
root 29270  0.0  0.1  10540  1620 pts/5S+   19:40   0:00 grep 
--color=auto python3
VM-0-13-suse:~ #
VM-0-13-suse:~ # ps -aux|grep sleep
root 29258  0.0  0.1   5760  1264 pts/3S+   19:40   0:00 sleep 12345
root 29281  0.0  0.1  10540  1628 pts/5S+   19:40   0:00 grep 
--color=auto sleep
VM-0-13-suse:~ #
VM-0-13-suse:~ # ls -l /proc/29258/fd
total 0
lrwx-- 1 root root 64 Jan 21 19:40 0 -> /dev/null
lrwx-- 1 root root 64 Jan 21 19:40 1 -> /dev/pts/3
lrwx-- 1 root root 64 Jan 21 19:40 2 -> /dev/pts/3
```

in python3.10.0:
```
bash-5.1# python3 -V
Python 3.10.0
bash-5.1#
bash-5.1# ps -aux|grep python3
root 28688  0.0  0.9  11664  8732 ?S+   11:36   0:00 python3 fd2.py
root 28725  0.0  0.2   6408  2216 ?S+   11:36   0:00 grep python3
bash-5.1#
bash-5.1# ps -aux|grep sleep
root 28694  0.0  0.1   5524   908 ?S+   11:36   0:00 sleep 12345
root 28729  0.0  0.2   6408  2276 ?S+   11:36   0:00 grep sleep
bash-5.1#
bash-5.1# ls -l /proc/28694/fd
total 0
lrwx-- 1 root root 64 Jan 21 11:36 1 -> /dev/pts/3
lrwx-- 1 root root 64 Jan 21 11:36 2 -> /dev/pts/3
bash-5.1#
bash-5.1#
bash-5.1# kill -9 28694 28688
bash-5.1#
bash-5.1# ps -aux|grep python3
root 28846  0.5  0.9  11680  8428 ?S+   11:37   0:00 python3 fd.py
root 28854  0.0  0.2   6408  2064 ?S+   11:37   0:00 grep python3
bash-5.1#
bash-5.1# ps -aux|grep sleep
root 28847  0.0  0.0   5524   872 ?S+   11:37   0:00 sleep 12345
root 28863  0.0  0.2   6408  2220 ?S+   11:37   0:00 grep sleep
bash-5.1#
bash-5.1# ls -l /proc/28847/fd
total 0
lrwx-- 1 root root 64 Jan 21 11:37 0 -> /dev/null
lrwx-- 1 root root 64 Jan 21 11:37 1 -> /dev/pts/3
lrwx-- 1 root root 64 Jan 21 11:37 2 -> /dev/pts/3
bash-5.1#
```

When we execute the script fd2.py in the python 3 environment, we can find that 
 '0 -> /dev/null' is lost. I wonder if this is a bug in Python3 or a new 
feature? If it is a bug, how to fix it? I will look forward to your reply very 
much.

--
components: Library (Lib)
messages: 43
nosy: shihai1991, sxt1001, vstinner
priority: normal
severity: normal
status: open
title: '0 -> /dev/null' is lost
type: behavior
versions: Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: