Re: Undocumented issue: Open system call blocks on named pipes (and a feature request)

[Cameron Simpson]

On 28Dec2018 20:21, Daniel Ojalvo  wrote:
I agree that previous behavior shouldn't be changed, but I would 
suggest updating the documentation to point it out as a footnote. The 
current behavior is correct just unclear. Most people just learning 
about the open command wouldn't have this expectation.


Maybe, maybe not. "Most" is a conjecture. IMO people will only find it 
surprising if they think any filesystem object can be instantly opened.  
However that is a misapprehension on their part.


My personal expectation is that open() will come back when the object is 
open. I don't have a timeframe in mind unless I have a strong 
expectation about _what_ I'm opening.


I came across the issue when I had a program that would open up all the 
files in a directory to read a few bytes from the beginning. My concern 
would be someone just making a named pipe over a file that a program 
would open.


What about a symlink to a magic /dev/tcp/host:port device, initiating a 
TCP connection? Particularly if "host" is down or inaccessible? Etc.


Arguably, anyone affected by that would be shooting themselves in the 
foot to begin with, but I think there are "security" concerns because 
someone could cause a bit of mischief that would be difficult to 
diagnose.


It isn't hard to diagnose at all. Point strace at the hung pogram, see 
it is opening some path, "ls -ld the-path", oooh, it isn't a regular 
file.


The point here is that if a programme opens every file in a directory, 
maybe it should constrain itself to regular files. Opening anything else 
may not just hang, it can have real world side effects. (Usually such 
effect happen at some point after open, for example opening a rewind 
take device will physicially rewind the tape on close, but you've 
committed to that happening by opening it in the first place.)


I think Chris offered the example of a subdirectory to suggest that such 
a programme already has an opnion about what to open and what to leave 
alone (unless is _does_ open() subdirectories, which might be useful but 
is usually misleading and on some OSes unsupported). So the programme 
should be pickier anyway.


That all being said, I think I would like to put in a feature request 
for a non-blocking option. How should I go about doing so?


I agree with the suggestion already made: devise a well thought out 
proposal which fits nicely with the existing open() call (eg an addition 
to the mode argument or something), and describe it clearly in 
python-ideas.


Certainly a number of things can be opened in a "nonblocking" mode, 
which means that reads return instantly if there's no available data, so 
having an open not block isn't unreasonable to want. But it may be 
unreasonable to implement in general: OSes may not support it directly.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Undocumented issue: Open system call blocks on named pipes (and a feature request)

[Chris Angelico]

On Sat, Dec 29, 2018 at 7:21 AM Daniel Ojalvo  wrote:
>
> Thank you for the advice!
>
> I haven't used the opener argument before, but I'll keep it for future 
> reference. I think it's still a little kludge-y, but it works.

It has a very similar effect to what you were proposing, but still
works within the normal open() ecosystem. Its main benefit is that it
works on existing Pythons, whereas any idea proposed today can't get
into 3.7 and maybe not even 3.8.

> I agree that previous behavior shouldn't be changed, but I would suggest 
> updating the documentation to point it out as a footnote. The current 
> behavior is correct just unclear. Most people just learning about the open 
> command wouldn't have this expectation.
>

That's what I'm not sure about. Do people really have an expectation
of nonblocking behaviour?

> I came across the issue when I had a program that would open up all the files 
> in a directory to read a few bytes from the beginning. My concern would be 
> someone just making a named pipe over a file that a program would open. 
> Arguably, anyone affected by that would be shooting themselves in the foot to 
> begin with, but I think there are "security" concerns because someone could 
> cause a bit of mischief that would be difficult to diagnose.
>

What happens if someone has a subdirectory in there? To be resilient
against everything you might come across, you probably need to check
anyway.

> That all being said, I think I would like to put in a feature request for a 
> non-blocking option. How should I go about doing so?

Hmm, there are a few options. If you reckon it's pretty easy, you
could just go straight to a GitHub pull request, and discuss it there.
Or you could open a bugs.python.org tracker issue and hope someone
else does the coding. Alternatively, you could find out who else
supports the idea by posting to the python-id...@python.org mailing
list.

Of those, I think posting to python-ideas is possibly the best, as
there will likely be some bikeshedding about the name and such.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Undocumented issue: Open system call blocks on named pipes (and a feature request)

[Daniel Ojalvo via Python-list]

Thank you for the advice!

I haven't used the opener argument before, but I'll keep it for future 
reference. I think it's still a little kludge-y, but it works.

I agree that previous behavior shouldn't be changed, but I would suggest 
updating the documentation to point it out as a footnote. The current behavior 
is correct just unclear. Most people just learning about the open command 
wouldn't have this expectation. I came across the issue when I had a program 
that would open up all the files in a directory to read a few bytes from the 
beginning. My concern would be someone just making a named pipe over a file 
that a program would open. Arguably, anyone affected by that would be shooting 
themselves in the foot to begin with, but I think there are "security" concerns 
because someone could cause a bit of mischief that would be difficult to 
diagnose.

That all being said, I think I would like to put in a feature request for a 
non-blocking option. How should I go about doing so?

Thanks again,
Dan

-Original Message-
From: Chris Angelico  
Sent: Thursday, December 27, 2018 7:10 PM
To: python-list@python.org
Subject: Re: Undocumented issue: Open system call blocks on named pipes (and a 
feature request)

On Fri, Dec 28, 2018 at 1:38 PM Daniel Ojalvo via Python-list 
 wrote:
>
> Hello,
>
> I've been working on a python3 project and I came across an issue with the 
> open system call that, at the very least, isn't documented. In my humble 
> opinion, the 
> documentation<https://docs.python.org/3/library/functions.html#open> should 
> be updated because folks wouldn't expect open to be a blocking operation and 
> simply error out. Worse yet, open doesn't have an option to make itself 
> non-blocking. You have to use the os system calls to kludge a solution.
>

Hmm. I disagree that the docs are deceptive here; I would normally expect 
open() to block if it needs to. But looking at this as a feature request, it 
seems reasonable. Actually, it's not even that hard to do, since open() is 
already pluggable:

rosuav@sikorsky:~/tmp$ rm rene_magritte
rosuav@sikorsky:~/tmp$ mkfifo rene_magritte rosuav@sikorsky:~/tmp$ ls -l 
rene_magritte
prw-r--r-- 1 rosuav rosuav 0 Dec 28 14:05 rene_magritte rosuav@sikorsky:~/tmp$ 
python3 Python 3.8.0a0 (heads/master:8b9c33ea9c, Nov 20 2018, 02:18:50) [GCC 
6.3.0 20170516] on linux Type "help", "copyright", "credits" or "license" for 
more information.
>>> import os
>>> def nonblock(fn, mode): return os.open(fn, mode | os.O_NONBLOCK)
...
>>> open("rene_magritte", opener=nonblock)
<_io.TextIOWrapper name='rene_magritte' mode='r' encoding='UTF-8'>
>>> _.read(1)
''

> Here is how I reproduced the issue:
>
> root@beefy:~/sandbox# mkfifo this_is_a_pipe

(my example file name is a more subtle reference...)

> I'm doing this to get a fileobject and make it error out if we do have a 
> blocking special file:
> with os.fdopen(os.open(, os.O_RDONLY| os.O_NONBLOCK) , mode='rb') 
> as file_obj:
>
> I think this is mostly a documentation bug because this wouldn't be expected 
> behavior to someone reading the docs, but open is behaving as the fifo man 
> page<http://man7.org/linux/man-pages/man7/fifo.7.html> is documented. The 
> feature request would be to add a non-blocking option to the default open 
> system call.
>

Honestly, I don't think there's a problem with it blocking by default.
Most of Python works that way. But it would be pretty straight-forward to add 
"nonblocking=False" as another keyword-only parameter, and for compatibility 
with existing versions (back as far as 3.3), the opener should work just fine.

ChrisA

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Undocumented issue: Open system call blocks on named pipes (and a feature request)

[Chris Angelico]

On Sat, Dec 29, 2018 at 4:24 AM Grant Edwards  wrote:
>
> On 2018-12-27, Daniel Ojalvo via Python-list  wrote:
>
>  open("this_is_a_pipe")
> >
>
> Opening a tty device can also block[1].  However, if somebody is using
> the open() builtin on tty devices that's probably the least of their
> problems.

Depending on what you mean by "block", pretty much anything can. It's
not indefinite, but this is clearly an example of blocking behaviour:

rosuav@sikorsky:~$ mkdir gideon_home
rosuav@sikorsky:~$ sshfs gideon: gideon_home/
rosuav@sikorsky:~$ python3
Python 3.8.0a0 (heads/master:8b9c33ea9c, Nov 20 2018, 02:18:50)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.time(); open("gideon_home/.bashrc"); time.time()
1546018477.1130645
<_io.TextIOWrapper name='gideon_home/.bashrc' mode='r' encoding='UTF-8'>
1546018477.8339248

Due to the latency introduced by having a completely-out-of-cache
remote access directory, simply opening an ordinary file with an
ordinary path took over half a second. But *this* type of blocking
access is NOT changed by adding os.O_NONBLOCK; it will still take that
half second even if you say "opener=nonblock". OTOH, this form of
blocking is a lot safer - normal file systems [1] might be slow, but
they can't deadlock, whereas a pipe most certainly could.

ChrisA

[1] Of course, you could use fusermount and shenanigans to do
basically anything. But at that point, you're deliberately shooting
yourself in the foot, and all I can advise is "don't do that".
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Undocumented issue: Open system call blocks on named pipes (and a feature request)

[Grant Edwards]

On 2018-12-27, Daniel Ojalvo via Python-list  wrote:

 open("this_is_a_pipe")
>

Opening a tty device can also block[1].  However, if somebody is using
the open() builtin on tty devices that's probably the least of their
problems.

[1] Technically, opening any character-mode device could block --
serial ports are the only "common" example I can think of.

-- 
Grant Edwards   grant.b.edwardsYow! How many retured
  at   bricklayers from FLORIDA
  gmail.comare out purchasing PENCIL
   SHARPENERS right NOW??

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Undocumented issue: Open system call blocks on named pipes (and a feature request)

[Chris Angelico]

On Fri, Dec 28, 2018 at 1:38 PM Daniel Ojalvo via Python-list
 wrote:
>
> Hello,
>
> I've been working on a python3 project and I came across an issue with the 
> open system call that, at the very least, isn't documented. In my humble 
> opinion, the 
> documentation should 
> be updated because folks wouldn't expect open to be a blocking operation and 
> simply error out. Worse yet, open doesn't have an option to make itself 
> non-blocking. You have to use the os system calls to kludge a solution.
>

Hmm. I disagree that the docs are deceptive here; I would normally
expect open() to block if it needs to. But looking at this as a
feature request, it seems reasonable. Actually, it's not even that
hard to do, since open() is already pluggable:

rosuav@sikorsky:~/tmp$ rm rene_magritte
rosuav@sikorsky:~/tmp$ mkfifo rene_magritte
rosuav@sikorsky:~/tmp$ ls -l rene_magritte
prw-r--r-- 1 rosuav rosuav 0 Dec 28 14:05 rene_magritte
rosuav@sikorsky:~/tmp$ python3
Python 3.8.0a0 (heads/master:8b9c33ea9c, Nov 20 2018, 02:18:50)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> def nonblock(fn, mode): return os.open(fn, mode | os.O_NONBLOCK)
...
>>> open("rene_magritte", opener=nonblock)
<_io.TextIOWrapper name='rene_magritte' mode='r' encoding='UTF-8'>
>>> _.read(1)
''

> Here is how I reproduced the issue:
>
> root@beefy:~/sandbox# mkfifo this_is_a_pipe

(my example file name is a more subtle reference...)

> I'm doing this to get a fileobject and make it error out if we do have a 
> blocking special file:
> with os.fdopen(os.open(, os.O_RDONLY| os.O_NONBLOCK) , mode='rb') 
> as file_obj:
>
> I think this is mostly a documentation bug because this wouldn't be expected 
> behavior to someone reading the docs, but open is behaving as the fifo man 
> page is documented. The 
> feature request would be to add a non-blocking option to the default open 
> system call.
>

Honestly, I don't think there's a problem with it blocking by default.
Most of Python works that way. But it would be pretty straight-forward
to add "nonblocking=False" as another keyword-only parameter, and for
compatibility with existing versions (back as far as 3.3), the opener
should work just fine.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list