[issue9611] FileIO not 64-bit safe under Windows

[STINNER Victor]

STINNER Victor  added the comment:

Issue #9015 has been marked as a duplicate of this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[STINNER Victor]

STINNER Victor  added the comment:

I backported fixes to 2.7, and also add a new fix to FileIO.read(). I don't see 
anything else to do on this issue, so I close it.

Note: read() and write() methods the file object in 2.7 are 64-bit safe on any 
OS. They use fread() and frwrite() which take a length in the size_t type, not 
in int type even on Windows.

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Roundup Robot]

Roundup Robot  added the comment:

New changeset 7acdf9f5eb31 by Victor Stinner in branch '3.2':
Issue #9611, #9015: FileIO.read() clamps the length to INT_MAX on Windows.
http://hg.python.org/cpython/rev/7acdf9f5eb31

New changeset e8646f120330 by Victor Stinner in branch 'default':
(merge 3.2) Issue #9611, #9015: FileIO.read() clamps the length to INT_MAX on 
Windows.
http://hg.python.org/cpython/rev/e8646f120330

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Roundup Robot]

Roundup Robot  added the comment:

New changeset 6abbc5f68e20 by Victor Stinner in branch '2.7':
Issue #9611, #9015: FileIO.read(), FileIO.readinto(), FileIO.write() and
http://hg.python.org/cpython/rev/6abbc5f68e20

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[STINNER Victor]

STINNER Victor  added the comment:

r87917 removes (useless and dangerous) conversion to size_t.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[STINNER Victor]

STINNER Victor  added the comment:

r87734 fixes stdprinter.write().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[STINNER Victor]

STINNER Victor  added the comment:

As asked by Antoine, I commited my patch: r87722.

... But I don't know if it fixes the issue or not, I don't have access to a 
Windows with more than 4 GB of memory.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[STINNER Victor]

Changes by STINNER Victor :


Removed file: http://bugs.python.org/file19898/read_write_32bits.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[STINNER Victor]

STINNER Victor  added the comment:

(oops, the patch contained unrelated changes: remove trailing spaces)

--
Added file: http://bugs.python.org/file19899/read_write_32bits.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[STINNER Victor]

STINNER Victor  added the comment:

I agree that clamping is a nice solution, and attached patch implements it.

About the question of the loop: FileIO.readall() uses while(1) without checking 
for signals. It looks like new_buffersize() maximum size is not BIGCHUNK but 
(BIGCHUNK-1)*2:

/* Keep doubling until we reach BIGCHUNK;
   then keep adding BIGCHUNK. */
if (currentsize <= BIGCHUNK)
return currentsize + currentsize;
else
return currentsize + BIGCHUNK;

Is it a bug? But (BIGCHUNK-1)*2 is always smaller than INT_MAX. So readall() 
isn't affected by this bug. Should it be patched?

posix.read() doesn't have the bug because it uses an "int" for the size.

--
keywords: +patch
Added file: http://bugs.python.org/file19898/read_write_32bits.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Martin v . Löwis]

Martin v. Löwis  added the comment:

Am 04.11.2010 22:28, schrieb Amaury Forgeot d'Arc:
> 
> Amaury Forgeot d'Arc  added the comment:
> 
>> Why do you think this would be somehow an example for a blocking stream
>> that does not write all data without raising an exception?
> Well, that's what "clamping" means, isn't it?

Ah, then I misunderstood. We certainly should discard any data.
I understood the proposal as rejecting write attempts for larger blocks
with an exception.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> On a second thought... is there another example where a *blocking*
> stream does not write all the data without raising an exception?

It can happen with pipes or sockets, if some buffer can only store part
of the data. Or any regular stream if a signal is received after part of
the data has been written.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc  added the comment:

> Why do you think this would be somehow an example for a blocking stream
> that does not write all data without raising an exception?
Well, that's what "clamping" means, isn't it?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Martin v . Löwis]

Martin v. Löwis  added the comment:

> On a second thought... is there another example where a *blocking* stream 
> does not write all the data without raising an exception?

Why do you think this would be somehow an example for a blocking stream
that does not write all data without raising an exception?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc  added the comment:

On a second thought... is there another example where a *blocking* stream does 
not write all the data without raising an exception?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Martin v . Löwis]

Martin v. Löwis  added the comment:

> If I'm not mistaken, WriteFile takes the length as a DWORD, which is
> still 32 bits under Win64.

Oops, ignore me, then... I agree that clamping is fine, assuming the
buffering layer takes that into account.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> I propose a different solution: On Windows, instead of calling
> write(), we call WriteFile directly.

If I'm not mistaken, WriteFile takes the length as a DWORD, which is
still 32 bits under Win64.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Martin v . Löwis]

Martin v. Löwis  added the comment:

I propose a different solution: On Windows, instead of calling write(), we call 
WriteFile directly. We try to faithfully follow the CRT implementation as much 
as possible, knowing that what we write to actually is a binary file (in 
particular, the file handle should get locked). We should perhaps make an 
exception for the standard handles (0,1,2), and fall back to call the CRT write 
unless we determine they are also in binary mode.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> About the issue, I'd suggest to just clamp the length to 2**32-1, and
> don't bother retrying; leave this to the buffered layer.

Yes, I think it's reasonable.
(or perhaps clamp to something page-aligned, such as 2**32-4096).
Also, the signal issue which was raised by Martin above has since been fixed in 
r84239.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc  added the comment:

Fortunately, the lower-level write() has no such bug, at least when used in 
binary mode as FileIO does: it's almost a direct call to WriteFile().
This issue is more considered because it's not a bug in the Microsoft CRT, but 
in the Python implementation of the IO stack.

About the issue, I'd suggest to just clamp the length to 2**32-1, and don't 
bother retrying; leave this to the buffered layer.

--
nosy: +amaury.forgeotdarc

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Martin Spacek]

Martin Spacek  added the comment:

We've got a near duplicate Issue9015. This thread seems more considered though 
:) Note that writing more than 2**32-1 bytes at once results in a hung process 
with 100% CPU in 64-bit Windows, which has to be killed with Task Manager. So I 
think that qualifies as a crash. This is apparently due to fwrite limitations 
in msvc, even in win64.

--
components: +IO
nosy: +mspacek

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Le dimanche 15 août 2010 à 18:53 +, Martin v. Löwis a écrit :
> Martin v. Löwis  added the comment:
> 
> > Well, the loop stops when an error status is returned by the raw IO
> > layer. At that point, the buffered IO layer re-raises the error after a
> > bit of internal cleanup.
> 
> Assume the following case:
> 1. writing starts, and writes some data
> 2. Ctrl-C is pressed, raises a signal, and interrupts the current
>system call (EINTR)
> 3. having already written data, the signal is discarded, and the
>number of successfully written bytes is returned.
> 4. the loop retries to write the rest. Not receiving any signal
>anymore, the subsequent write operations wait for completion.

Ok, I guess the loop should run PyErr_CheckSignals() somewhere.

Simulate such a situation in an unit test will be a bit tricky.
Perhaps we can use os.pipe() and depend on the fact that writes greater
than the pipe buffer size will be blocking.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Martin v . Löwis]

Martin v. Löwis  added the comment:

> Well, the loop stops when an error status is returned by the raw IO
> layer. At that point, the buffered IO layer re-raises the error after a
> bit of internal cleanup.

Assume the following case:
1. writing starts, and writes some data
2. Ctrl-C is pressed, raises a signal, and interrupts the current
   system call (EINTR)
3. having already written data, the signal is discarded, and the
   number of successfully written bytes is returned.
4. the loop retries to write the rest. Not receiving any signal
   anymore, the subsequent write operations wait for completion.

End consequence: the signal is discarded without any effect.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> > Most people use buffered I/O, and the buffered layer automatically retries.
> 
> I see. I think this is already slightly problematic: if you send an
> interrupt, it won't oblige. IMO, any such loop ought to be
> interruptable.

Well, the loop stops when an error status is returned by the raw IO
layer. At that point, the buffered IO layer re-raises the error after a
bit of internal cleanup.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Martin v . Löwis]

Martin v. Löwis  added the comment:

> Most people use buffered I/O, and the buffered layer automatically retries.

I see. I think this is already slightly problematic: if you send an
interrupt, it won't oblige. IMO, any such loop ought to be
interruptable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Antoine Pitrou]

Antoine Pitrou  added the comment:

The write() man page says:

   The number of bytes written may be less than count if, for example, 
there is  insufficient  space  on
   the underlying physical medium, or the RLIMIT_FSIZE resource limit is 
encountered (see setrlimit(2)),
   or the call was interrupted by a signal handler after having written 
less  than  count  bytes.   (See
   also pipe(7).)

So, we could return the number of bytes successfully written, and let the next 
call fail.

Another possibility is to only write 2GB-1 and let the caller retry.
Most people use buffered I/O, and the buffered layer automatically retries.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Martin v . Löwis]

Martin v. Löwis  added the comment:

Using chunked writes is tricky. If the first write succeeds, and the second one 
fails, how do you report the result?

--
nosy: +loewis

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +brian.curtin, tim.golden

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Antoine Pitrou]

Antoine Pitrou  added the comment:

os.write() (in posixmodule.c) is also affected. os.read(), however, is limited 
to 32-bit inputs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9611] FileIO not 64-bit safe under Windows

[Antoine Pitrou]

New submission from Antoine Pitrou :

Modules/_io/fileio.c assumes that read() and write() allow a Py_ssize_t length, 
but under Windows the length is an int, limiting chunk size to 2GB.

It should be easy enough to read or write in multiple chunks, although testing 
might be difficult.

--
components: Extension Modules, Windows
messages: 113974
nosy: pitrou
priority: normal
severity: normal
status: open
title: FileIO not 64-bit safe under Windows
type: behavior
versions: Python 2.7, Python 3.1, Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com