[issue44219] Opening a file holds the GIL when it calls "isatty()"

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset 314de5326f8b687a3fa7b19ea8faaa449f59b8fe by Miss Islington (bot) 
in branch '3.9':
bpo-44219: Mention GH-28250 is a deadlock bugfix (GH-28261) (GH-28275)
https://github.com/python/cpython/commit/314de5326f8b687a3fa7b19ea8faaa449f59b8fe


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset 60ddf499e14cc7daba3804e5a3460e4224dacc5c by Miss Islington (bot) 
in branch '3.10':
bpo-44219: Mention GH-28250 is a deadlock bugfix (GH-28261) (GH-28274)
https://github.com/python/cpython/commit/60ddf499e14cc7daba3804e5a3460e4224dacc5c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[miss-islington]

Change by miss-islington :


--
pull_requests: +26692
pull_request: https://github.com/python/cpython/pull/28274

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[miss-islington]

Change by miss-islington :


--
pull_requests: +26693
pull_request: https://github.com/python/cpython/pull/28275

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset 296b7100705ef52aece3378b0ae42c33a58526e1 by Łukasz Langa in 
branch 'main':
bpo-44219: Mention GH-28250 is a deadlock bugfix (GH-28261)
https://github.com/python/cpython/commit/296b7100705ef52aece3378b0ae42c33a58526e1


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[Łukasz Langa]

Change by Łukasz Langa :


--
pull_requests: +26681
pull_request: https://github.com/python/cpython/pull/28261

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[STINNER Victor]

STINNER Victor  added the comment:

> In my case, the file in question is implemented as a FUSE mount which is 
> served by the same process (different thread of course). Thus holding the GIL 
> at this point causes a rather interesting deadlock.

Since the change fixes a deadlock, I agree to backport it to 3.9 and 3.10.

Thanks for the fix!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset 23c46778d65870784cb6d4de30f43aac62d71e73 by Miss Islington (bot) 
in branch '3.10':
bpo-44219: Release the GIL during isatty syscalls (GH-28250) (GH-28255)
https://github.com/python/cpython/commit/23c46778d65870784cb6d4de30f43aac62d71e73


--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[miss-islington]

miss-islington  added the comment:


New changeset 5c65834d801d6b4313eef0684a30e12c22ccfedd by Miss Islington (bot) 
in branch '3.9':
bpo-44219: Release the GIL during isatty syscalls (GH-28250)
https://github.com/python/cpython/commit/5c65834d801d6b4313eef0684a30e12c22ccfedd


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[miss-islington]

Change by miss-islington :


--
pull_requests: +26675
pull_request: https://github.com/python/cpython/pull/28256

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 3.0 -> 4.0
pull_requests: +26674
pull_request: https://github.com/python/cpython/pull/28255

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 06148b1870fceb1a21738761b8e1ac3bf654319b by Vincent Michel in 
branch 'main':
bpo-44219: Release the GIL during isatty syscalls (GH-28250)
https://github.com/python/cpython/commit/06148b1870fceb1a21738761b8e1ac3bf654319b


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[Vincent Michel]

Vincent Michel  added the comment:

There are a couple of reasons why I did not make changes to the stdstream 
related functions. 

The first one is that a PR with many changes is less likely to get reviewed and 
merged than a PR with fewer changes. The second one is that it's hard for me to 
make sure that those functions are always called with the GIL already held. 
Maybe some of them never hold the GIL in the first place, and I'm not familiar 
enough with the code base to tell.

So in the end it will probably be up to the coredev reviewing the PR, but 
better start small.

> Besides, nothing prevents somebody from starting a FUSE file system and then 
> redirecting stdout to it …

I ran some checks and `python -c 'input()' < my-fuse-mountpoint/data_in` does 
indeed trigger an ioctl call to the corresponding fuse file system. But how 
would `my-fuse-mountpoint/data_in` be the stdin for the process that itself 
starts the fuse filesystem? I don't see how to generate this deadlock, apart 
from setting up interprocess communication between the fuse process and the 
`my-fuse-mountpoint/data_in`-as-stdin process.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[Matthias Urlichs]

Matthias Urlichs  added the comment:

Please do. However, I do think that changing the stdstream related ioctl calls 
also is a good idea, if only for code regularity/completeness sake. 

(Besides, nothing prevents somebody from starting a FUSE file system and then 
redirecting stdout to it …)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[Vincent Michel]

Change by Vincent Michel :


--
pull_requests: +26671
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/28250

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[Vincent Michel]

Vincent Michel  added the comment:

Here's a possible patch that fixes the 3 unprotected calls to `isatty` 
mentioned above. It successfully passes the test suite. I can submit a PR with 
this patch if necessary.

--
keywords: +patch
Added file: https://bugs.python.org/file50270/bpo-44219.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[Vincent Michel]

Vincent Michel  added the comment:

My team ran into this issue while developing a fuse application too.

In an effort to help this issue move forward, I tried to list all occurrences 
of the `isatty` C function in the cpython code base. I found 14 of them.

9 of them are directly related to stdin/stdout/stderr, so it's probably not 
crucial to release the GIL for those occurrences:
- `main.c:stdin_is_interactive`
- `main.c:pymain_import_readline`
- `readline.c:setup_readline`
- `bltinmodule.c:builtin_input_impl` (x2)
- `frozenmain.c:Py_FrozenMain`
- `pylifecycle.c:Py_FdIsInteractive` (x2)
- `fileobject.c:stdprinter_isatty` (GIL is actually released for this one)

Out of the remaining 4, only 1 releases the GIL:
- `fileio.c:_io_FileIO_isatty_impl`: used for `FileIO.isatty`

Which gives 3 occurrences of non-stdstream specific usage of `isatty` that do 
not release the GIL:
- `posixmodule.c:os_isatty_impl`: used by `os.isatty`
- `fileutils.c:_Py_device_encoding`: used `TextIOWrapper.__init__`
- `fileutils.c:_Py_write_impl`: windows specific, issue #11395

The first one is used by `os.isatty` which means this call can also deadlock. I 
did manage to reproduce it with a simple fuse loopback file system: 
https://github.com/fusepy/fusepy/blob/master/examples/loopback.py

The second one is the one found by @smurfix and gets triggered when `io.open()` 
is used in text mode.

The third one only triggers on windows when writing more than 32767 bytes to a 
file descriptor. A comment points to issue #11395 
(https://bugs.python.org/issue11395). Also, it seems from the function 
signature that this function might be called with or without the GIL held, 
which might cause the fix to be a bit more complicated than the first two use 
cases.

I hope this helps.

--
nosy: +vxgmichel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44219] Opening a file holds the GIL when it calls "isatty()"

[Matthias Urlichs]

New submission from Matthias Urlichs :

Opening a file calls `isatty` which calls an ioctl with the GIL held.

GDB:
```
#0  __GI___tcgetattr (fd=18, termios_p=termios_p@entry=0x7f618a5df920)
at ../sysdeps/unix/sysv/linux/tcgetattr.c:38
#1  0x7f618bd1ca0c in __isatty (fd=) at 
../sysdeps/posix/isatty.c:27
#2  0x0062b746 in _Py_device_encoding (fd=) at 
../Python/fileutils.c:62
#3  0x0060bf90 in _io_TextIOWrapper___init___impl (write_through=0, 
line_buffering=0, 
newline=0x0, errors='strict', encoding=, 
buffer=<_io.BufferedWriter at remote 0x7f618986aeb0>, self=0x7f618985a860)
at ../Modules/_io/textio.c:1149
...
```

Please don't do that.

In my case, the file in question is implemented as a FUSE mount which is served 
by the same process (different thread of course). Thus holding the GIL at this 
point causes a rather interesting deadlock.

Tested with 3.9.

--
components: Interpreter Core
messages: 394208
nosy: smurfix
priority: normal
severity: normal
status: open
title: Opening a file holds the GIL when it calls "isatty()"
type: crash
versions: Python 3.10, Python 3.11, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com