[issue21090] File read silently stops after EIO I/O error

[Martijn Pieters]

Martijn Pieters added the comment:

The Python 2.7 issue (using fread without checking for interrupts) looks like a 
duplicate of http://bugs.python.org/issue1633941

--
nosy: +mjpieters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[STINNER Victor]

STINNER Victor added the comment:

On IRC, buck1 asked why the following code behaves differently on Python  3.4 
and Python = 3.4. It is related to this issue in fact.

Code:
---
from __future__ import print_function

from os import openpty
read, write = openpty()

from subprocess import Popen
proc = Popen(
('echo', 'ok'),
stdout=write,
close_fds=True,
)

from os import fdopen
fdopen(write, 'w').close()
with fdopen(read) as stdout:
print('STDOUT', stdout.read())

print('exit code:', proc.wait())
---

Simplified example:
---
import io, os
read, write = os.openpty()
os.write(write, b'ok\n')
os.close(write)
with io.FileIO(read, closefd=False) as fp:
print(fp.readall())
---

On Python  3.4, it displays ok, whereas Python 3.4 and later fail with 
OSError(5, 'Input/output error' on readall().

Another example:
---
import os
read, write = os.openpty()
os.write(write, b'ok\n')
os.close(write)
print(read: %r % os.read(read, 4096))
print(read: %r % os.read(read, 4096))
---

The first read syscall succeed, even if the write end is already called. But 
the second read syscall fails with EIO.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[Charles-François Natali]

Charles-François Natali added the comment:

LGTM.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[Roundup Robot]

Roundup Robot added the comment:

New changeset 652b62213072 by Victor Stinner in branch '3.4':
Issue #21090: io.FileIO.readall() does not ignore I/O errors anymore. Before,
http://hg.python.org/cpython/rev/652b62213072

New changeset 440279cec378 by Victor Stinner in branch 'default':
(Merge 3.4) Issue #21090: io.FileIO.readall() does not ignore I/O errors
http://hg.python.org/cpython/rev/440279cec378

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[Roundup Robot]

Roundup Robot added the comment:

New changeset 1492a42b8308 by Victor Stinner in branch '2.7':
Issue #21090: io.FileIO.readall() does not ignore I/O errors anymore. Before,
http://hg.python.org/cpython/rev/1492a42b8308

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[STINNER Victor]

STINNER Victor added the comment:

For Python 2, file.read() looks wrong: if only checks ferror() if fread() 
returns 0, whereas Py_UniversalNewlineFread() can call fread() more than once, 
and according to fread() manual page, fread() result can be different than 0 on 
error.

If an error occurs, or the end of the file is reached, the return value is a 
short item count (or zero).
http://linux.die.net/man/3/fread

Attached fileobject.c rewrites error handling in fileobject.c. The main change 
if that ferror() is called even if fread() does not return 0.

--
Added file: http://bugs.python.org/file35833/fileobject.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[STINNER Victor]

STINNER Victor added the comment:

Here is a patch for FileIO.readall() which should fix the issue. Currently, 
readall() returns read bytes at the first read() error if a least one call to 
read() succeed.

--
keywords: +patch
Added file: http://bugs.python.org/file35820/fileio_readall_eio.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[ivank]

ivank added the comment:

I managed to reproduce this again, this time by corrupting data on a btrfs 
filesystem.

$ cat read_error_file.py
import os

fname = /usr/bin/Xorg
size = os.stat(fname).st_size
print fname, stat size:, size

f = open(fname, rb)
print len(f.read()): , len(f.read())
f.close()

f = open(fname, rb)
for i in xrange(size):
try:
f.read(1)
except IOError:
print IOError at byte %d % i
break
f.close()

$ python read_error_file.py 
/usr/bin/Xorg stat size: 2331776
len(f.read()):  716800
IOError at byte 716800

Note how the first test does not throw an IOError, but the second one does.

The strace for the first test is:

open(/usr/bin/Xorg, O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=2331776, ...}) = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=2331776, ...}) = 0
lseek(3, 0, SEEK_CUR)   = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=2331776, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 
0x7f1334bd6000
lseek(3, 0, SEEK_CUR)   = 0
mmap(NULL, 2334720, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 
0x7f1332ea6000
read(3, \177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\0\1\0\0\0\265M\4\0\0\0\0\0..., 
2330624) = 716800
read(3, 0x7f1332f55034, 1613824)= -1 EIO (Input/output error)
mremap(0x7f1332ea6000, 2334720, 720896, MREMAP_MAYMOVE) = 0x7f1332ea6000
munmap(0x7f1332ea6000, 720896)  = 0
write(1, len(f.read()):  716800\n, 23len(f.read()):  716800
) = 23

Note the -1 EIO (Input/output error) that gets ignored somewhere.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[ivank]

ivank added the comment:

This problem happens with Python 3.4 as well.

$ cat read_error_file.py 
from __future__ import print_function

import os

fname = /usr/bin/Xorg
size = os.stat(fname).st_size
print(fname, stat size:, size)

f = open(fname, rb)
print(len(f.read()): , len(f.read()))
f.close()

f = open(fname, rb)
for i in range(size):
try:
f.read(1)
except IOError:
print(IOError at byte %d % i)
break
f.close()

$ python3 --version
Python 3.4.1

$ python3 read_error_file.py
/usr/bin/Xorg stat size: 2331776
len(f.read()):  716800
IOError at byte 716800

strace for the first test is:

open(/usr/bin/Xorg, O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=2331776, ...}) = 0
ioctl(3, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 
0x7fff323ac8b0) = -1 ENOTTY (Inappropriate ioctl for device)
fstat(3, {st_mode=S_IFREG|0755, st_size=2331776, ...}) = 0
lseek(3, 0, SEEK_CUR)   = 0
lseek(3, 0, SEEK_CUR)   = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=2331776, ...}) = 0
mmap(NULL, 2334720, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 
0x7f57884cc000
read(3, \177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\0\1\0\0\0\265M\4\0\0\0\0\0..., 
2331777) = 716800
read(3, 0x7f578857b030, 1614977)= -1 EIO (Input/output error)
mremap(0x7f57884cc000, 2334720, 720896, MREMAP_MAYMOVE) = 0x7f57884cc000
munmap(0x7f57884cc000, 720896)  = 0
write(1, len(f.read()):  716800\n, 23len(f.read()):  716800
) = 23
close(3)

--
versions: +Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[Claudiu.Popa]

Changes by Claudiu.Popa pcmantic...@gmail.com:


--
nosy:  -Claudiu.Popa

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[STINNER Victor]

STINNER Victor added the comment:

2014-04-27 5:26 GMT+02:00 ivank rep...@bugs.python.org:
 (I see the `IOError: [Errno 5] Input/output error` exception now.)

Can you please run your test in strace to see system calls?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[Charles-François Natali]

Charles-François Natali added the comment:

I'm with Antoine, it's likely a glibc bug.

We already had a similar issue with fwrite():
http://bugs.python.org/issue17976

--
nosy: +neologix

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[Antoine Pitrou]

Antoine Pitrou added the comment:

ivank, if you know some C, perhaps you could write a trivial program that does 
an fopen() followed by an fread() of 131072 bytes, and see if the fread() 
errors out.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[STINNER Victor]

STINNER Victor added the comment:

@ivank: Can you please answer to questions? It's hard to understand the issue. 
Without more information, I would suggest to close the issue.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[ivank]

ivank added the comment:

I'm finding it hard to reproduce the bug again with more zpool corruption.  (I 
see the `IOError: [Errno 5] Input/output error` exception now.)  I do remember 
that in the reported case, Python 3.4, node.js, and OpenJDK 7 threw an EIO 
exception, but Python 2.7 did not.  I tested this multiple times.

Right now I can only speculate that Python 2.7 silently stops reading only in 
certain cases, e.g. depending on how Python's reads are aligned with the first 
byte that causes EIO.

I'm still working on getting it reproduced, please hold off on closing.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[STINNER Victor]

Changes by STINNER Victor victor.stin...@gmail.com:


--
nosy: +haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[STINNER Victor]

STINNER Victor added the comment:

 Python 2.7 uses C fopen() and fread(), so what happens probably is that 
 fread() silences the error.

I see that file_read() checks ferror() if fread() returned 0. I would nice to 
run the test in strace and attach the output of strace to see if the EIO is 
returneded by the kernel at least.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[Antoine Pitrou]

Antoine Pitrou added the comment:

Python 2.7 uses C fopen() and fread(), so what happens probably is that fread() 
silences the error.

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[Claudiu.Popa]

Changes by Claudiu.Popa pcmantic...@gmail.com:


--
nosy: +Claudiu.Popa

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21090] File read silently stops after EIO I/O error

[ivank]

New submission from ivank:

I intentionally corrupted a zpool to induce an I/O error in a file, in this 
case, /usr/lib/x86_64-linux-gnu/gconv/IBM1390.so

# ls -l /usr/lib/x86_64-linux-gnu/gconv/IBM1390.so
-rw-r--r-- 1 root root 231,496 2014-03-24 06:26 
/usr/lib/x86_64-linux-gnu/gconv/IBM1390.so

# cat /usr/lib/x86_64-linux-gnu/gconv/IBM1390.so  /dev/null
cat: /usr/lib/x86_64-linux-gnu/gconv/IBM1390.so: Input/output error

When I read the file, Python 3.3.5 and 3.4.0 check for EIO and raise an 
exception:

 open(/usr/lib/x86_64-linux-gnu/gconv/IBM1390.so, rb).read()
Traceback (most recent call last):
  File stdin, line 1, in module
OSError: [Errno 5] Input/output error

but Python 2.7.6 does not:

# python2
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] on linux2
Type help, copyright, credits or license for more information.
 x = open(/usr/lib/x86_64-linux-gnu/gconv/IBM1390.so, rb).read()
 len(x)
131072

--
components: IO
messages: 215085
nosy: ivank
priority: normal
severity: normal
status: open
title: File read silently stops after EIO I/O error
versions: Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21090
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com