[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[Benjamin Peterson]

Change by Benjamin Peterson :


--
resolution:  -> out of date
stage: test needed -> resolved
status: pending -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[Irit Katriel]

Irit Katriel  added the comment:

Since this is a python 2 only issue, should this issue be closed as out of date?

--
nosy: +iritkatriel
status: open -> pending

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[Jun Wu]

Jun Wu added the comment:

haypo: The file.__iter__ EINTR issue may be better discussed as a separate bug 
report. It's not related to stdin or EOF or Windows.

Since we have some EINTR fixes for Python 2.7.4, I think it's reasonable to fix 
the remaining EINTR issues for 2.7.13+.

If I have read fileobject.c correctly, readahead() is the only remaining place 
having the EINTR issue.

If you agree that we should fix readahead(), I can prepare the patch.

--
nosy: +quark

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[STINNER Victor]

STINNER Victor added the comment:

I don't see any simple solution to get a 100% reliable I/O stack on Python 2.

Python 3.5 contains a pure Python implementation of the io module: _pyio.FileIO 
uses os.read() and os.write(). In Python 3.4 and older, the _pyio still used 
io.FileIO (implemented in C). But try to recall Python 3.0 which had *very* bad 
I/O performance because its io module was fully implemented in pure Python!

The uvloop project proved that Python can be very efficient for (network) I/O 
using code written with Cython. But I also know that Mercurial cares of PyPy 
which is not really Cython-friendly.

Even if fread() bugs are fixed in Python 2.7.x+1, you will still hit bugs on 
Python 2.7.x and older.

Maybe it can be a strong motivation to pursue your Python 3 efforts :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[STINNER Victor]

STINNER Victor added the comment:

Martijn Pieters: Sadly, Python 2 I/O are full of bugs in corner cases :-/

First of all, in most cases, Python 2 uses the libc for I/O, but the libc has 
known bugs including segfaults:
https://haypo-notes.readthedocs.io/python.html#bugs-in-the-c-stdio-used-by-the-python-i-o

Python 3 is better to handle EINTR. EINTR should now be "fully supported" in 
Python 3.5 thanks for the PEP 475. I mean in the Python core, I don't expect 
that any third party implement the PEP 475. Hopefully, most third party module 
don't implement syscall wrappers themself, but reuse Python which handles EINTR 
for them.

To come back to Python 2: yeah, we still have to fix issues to make the code 
more robust in corner cases, and enhance error reporting. It seems like fread() 
errors are not checked correctly in some places.

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[Martijn Pieters]

Martijn Pieters added the comment:

It looks like readahead was missed when http://bugs.python.org/issue12268 was 
fixed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[Martijn Pieters]

Martijn Pieters added the comment:

This bug affects all use of `file.__iter__` and interrupts (EINTR), not just 
sys.stdin.

You can reproduce the issue by reading from a (slow) pipe in a terminal window 
and resizing that window, for example; the interrupt is not handled and a 
future call ends up raising `IOError: [Errno 0] Error`, a rather confusing 
message.

The Mercurial community is switching away from using direct iteration over this 
bug; Jun's excellent analysis is included and enlightening:

   
https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-November/090522.html

The fix is to use

interrupted = ferror(f->f_fp) && errno == EINTR;
// ..
if (interrupted) {
clearerr(f->f_fp);
if (PyErr_CheckSignals()) {
Py_DECREF(v);
return NULL;
}
}

and check for interrupted == 0 in the chunksize == 0 case after 
Py_UniversalNewlineFread calls, as file_read does, for example, but which 
readahead doesn't (where the only public user of readahead is file_iternext).

--
nosy: +mjpieters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[Don Hatch]

Don Hatch added the comment:

I've reported the unfriendly input withholding that several people have
observed and mentioned here as a separate bug: 
http://bugs.python.org/issue26290 . The symptom is different but I suspect it 
has exactly the same underlying cause (incorrect use of stdio) and fix that 
Ralph Corderoy has described clearly here.

--
nosy: +Don Hatch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[Dan Kegel]

Dan Kegel added the comment:

Still present in python 2.7.9, but fixed in python 3.4.3.
Also, in python 3.4.3, output is immediate, there seems to be no
input buffering.

--
nosy: +dankegel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[Mark Lawrence]

Changes by Mark Lawrence breamore...@yahoo.co.uk:


--
nosy:  -BreamoreBoy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1633941
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[Jaroslav Henner]

Changes by Jaroslav Henner jaroslav.hen...@gmail.com:


--
nosy: +jary

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1633941
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[Ralph Corderoy]

Ralph Corderoy ralph-pythonb...@inputplus.co.uk added the comment:

This most certainly is a bug under Unix and an annoying one.  Since the
issue is with platform's stdio library is wrong;  stdio is being used
incorrectly.  It would be nice to see it fixed in the 2.x line.

I've two test programs.

$ head -42 stdin2.6 stdin3.1
== stdin2.6 ==
#! /usr/bin/python2.6

import sys

for l in sys.stdin:
print repr(l)
print 'end'

== stdin3.1 ==
#! /usr/bin/python3.1

import sys

for l in sys.stdin:
print(repr(l))
print('end')

$

For both of them I will type 1 Enter 2 Enter 3 Enter Ctrl-D without
the spaces, Ctrl-D being my tty's EOF, stty -a.

$ ./stdin2.6
1
2
3
'1\n'
'2\n'
'3\n'

On the EOF the first iteration of sys.stdin returns and then so do the
others with the buffered lines.  The loop doesn't terminate, a second
Ctrl-D is required, giving.

end
$

Next,

$ ./stdin3.1
1
'1\n'
2
'2\n'
3
'3\n'
end
$

perfect output.  Only one Ctrl-D required and better still each line is
returned as it's entered.

ltrace shows python2.6 uses fread(3).  I'm assuming it treats only a
zero return as EOF whereas whenever the return value is less than the
number of requested elements, EOF could have been reached;  feof(3) must
be called afterwards to decide.  Really, ferror(3) should also be called
to see if, as well as returning some elements, an error was detected.

It's this lack of feof() that means the second fread() is required to
trigger the flawed `only 0 return is EOF' logic.

Here's some C that shows stdio works fine if feof() and ferror() are
combined with fread().

#include stdio.h

int main(void)
{
unsigned char s[8192], *p;
size_t n;

while ((n = fread(s, 1, 8192, stdin))) {
printf(%zd, n);
p = s;
while (n--)
printf( %02hhx, *p++);
putchar('\n');

if (feof(stdin)) {
puts(end);
break;
}
if (ferror(stdin)) {
fputs(error, stderr);
return 1;
}
}

return 0;
}

--
nosy: +ralph.corderoy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1633941
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[Vetoshkin Nikita]

Vetoshkin Nikita nikita.vetosh...@gmail.com added the comment:

I guess http://bugs.python.org/issue1195 might be related

--
nosy: +nvetoshkin

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1633941
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[Finkregh]

Changes by Finkregh finkr...@mafia-server.net:


--
nosy: +Finkregh

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1633941
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[Mark Lawrence]

Mark Lawrence breamore...@yahoo.co.uk added the comment:

This is fixed in py3k but still exists in 2.7.

--
nosy: +BreamoreBoy
versions: +Python 2.7 -Python 2.6

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1633941
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

Benjamin, is it too late too have this fixed in 2.7?

--
nosy: +benjamin.peterson, merwok

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1633941
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1633941] for line in sys.stdin: doesn't notice EOF the first time

[Fan Decheng]

Changes by Fan Decheng fandech...@gmail.com:


--
nosy: +r_mosaic

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1633941
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com