[issue24658] open().write() fails on 2 GB+ data (OS X)

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

Nosying myself since I just landed here based on an internal $work bug report.  
We're seeing it with reads.  I'll try to set aside some work time to review the 
PRs.

--
nosy: +barry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[miss-islington]

miss-islington  added the comment:


New changeset 0c15e508baec7e542933db2b31ea950a646cd968 by Miss Islington (bot) 
in branch '3.6':
bpo-24658: os.read() reuses _PY_READ_MAX (GH-10657)
https://github.com/python/cpython/commit/0c15e508baec7e542933db2b31ea950a646cd968


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[miss-islington]

miss-islington  added the comment:


New changeset 18f3327d9a99163a658697465eb00c31f86535eb by Miss Islington (bot) 
in branch '3.7':
bpo-24658: os.read() reuses _PY_READ_MAX (GH-10657)
https://github.com/python/cpython/commit/18f3327d9a99163a658697465eb00c31f86535eb


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[miss-islington]

Change by miss-islington :


--
pull_requests: +9912

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[miss-islington]

Change by miss-islington :


--
pull_requests: +9911

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 9a0d7a7648547ffb77144bf2480155f6d7940dea by Victor Stinner in 
branch 'master':
bpo-24658: os.read() reuses _PY_READ_MAX (GH-10657)
https://github.com/python/cpython/commit/9a0d7a7648547ffb77144bf2480155f6d7940dea


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9910

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[miss-islington]

miss-islington  added the comment:


New changeset 178d1c07778553bf66e09fe0bb13796be3fb9abf by Miss Islington (bot) 
in branch '3.7':
bpo-24658: Fix read/write greater than 2 GiB on macOS (GH-1705)
https://github.com/python/cpython/commit/178d1c07778553bf66e09fe0bb13796be3fb9abf


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Stéphane Wirtel]

Change by Stéphane Wirtel :


--
pull_requests: +9289

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset a5ebc205beea2bf1501e4ac33ed6e81732dd0604 by Victor Stinner 
(Stéphane Wirtel) in branch '3.6':
[3.6] bpo-24658: Fix read/write greater than 2 GiB on macOS (GH-1705) (GH-9937)
https://github.com/python/cpython/commit/a5ebc205beea2bf1501e4ac33ed6e81732dd0604


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Stéphane Wirtel]

Change by Stéphane Wirtel :


--
pull_requests: +9287

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[STINNER Victor]

Change by STINNER Victor :


--
versions: +Python 2.7, Python 3.7, Python 3.8 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[miss-islington]

Change by miss-islington :


--
pull_requests: +9286

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 74a8b6ea7e0a8508b13a1c75ec9b91febd8b5557 by Victor Stinner 
(Stéphane Wirtel) in branch 'master':
bpo-24658: Fix read/write greater than 2 GiB on macOS (GH-1705)
https://github.com/python/cpython/commit/74a8b6ea7e0a8508b13a1c75ec9b91febd8b5557


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Stéphane Wirtel]

Stéphane Wirtel  added the comment:

Hi all,

Could you test the PR with Windows? I don't have a Windows computer.

Thank you,

Stéphane

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[STINNER Victor]

STINNER Victor added the comment:

I don't say that something is broken. Just that it would be nice if someone
could test socket methods.

On Windows, the bug was obvious: the function takes a C int...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Stéphane Wirtel]

Stéphane Wirtel added the comment:

1. in the case of Windows, maybe we could open a new issue because this fix is 
only for MacOS

2. the issue was only for the files and not the sockets

what do you suggest ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Zachary Ware]

Changes by Zachary Ware :


--
nosy: +zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[STINNER Victor]

STINNER Victor added the comment:

I see that we have other clamps on Windows using INT_MAX:

* sock_setsockopt()
* sock_sendto_impl()

Are these functions ok on macOS? If not, a new issue should be opened ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
pull_requests: +1798

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Stéphane Wirtel]

Stéphane Wirtel added the comment:

Hello

I just updated this ticket with a PR on Github.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Harry Li]

Changes by Harry Li :


--
nosy: +Harry Li

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Stéphane Wirtel]

Stéphane Wirtel added the comment:

upload a new version

--
Added file: http://bugs.python.org/file45178/issue24658-3-3.6.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Stéphane Wirtel]

Stéphane Wirtel added the comment:

Victor, could you check the new patch ?

--
Added file: http://bugs.python.org/file45177/issue24658-2-3.6.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Stéphane Wirtel]

Stéphane Wirtel added the comment:

Ned Deily, I added you because you are in the expert for the OSX platform.

--
assignee:  -> ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Stéphane Wirtel]

Stéphane Wirtel added the comment:

ping

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Stéphane Wirtel]

Stéphane Wirtel added the comment:

Sorry, I was busy with a task but here is my patch for 3.5, in fact, it's just 
the same for 3.6

--
Added file: http://bugs.python.org/file44023/issue24658-3.5.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


Added file: http://bugs.python.org/file44024/issue24658-3.5.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


Removed file: http://bugs.python.org/file44023/issue24658-3.5.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Stéphane Wirtel]

Stéphane Wirtel added the comment:

Here is my patch 3.6, I am going to provide the patch for 3.5

--
nosy: +matrixise
Added file: http://bugs.python.org/file44021/issue24658-3.6.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Zachary Ware]

Changes by Zachary Ware :


--
versions: +Python 3.6 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Ian Carroll]

Ian Carroll added the comment:

Write still fails on 3.5.1 and OS X 10.11.2. I'm no dev, so can someone explain 
how to use the patch while it's under review?

--
nosy: +Ian Carroll

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Ronald Oussoren]

Ronald Oussoren added the comment:

Indeed, read(2) has the same problem. I just tested this with a small C 
program. 

I'll rework the patch for this, and will work on patches for 3.4/3.5 and 2.7 as 
well.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24658
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Ronald Oussoren]

Ronald Oussoren added the comment:

This is likely a platform bug, it fails with os.write as well.  Interestingly 
enough file.write works fine on Python 2.7 (which uses stdio), that appearently 
works around this kernel misfeature.

A possible partial workaround is recognise this error in the implementation of 
os.write and then perform a partial write. Problem is: while write(2) is 
documented as possibly writing less data than expected most users writing to 
normal files (as opposed to sockets) probably don’t expect that behavior. On 
the other hand, os.write already limits writes to INT_MAX on Windows (see 
_Py_write in Python/fileutils.c)

Because of this I’m in favour of adding a simular workaround on OSX (and can 
provide a patch).

BTW. the manpage for write says that writev(2) might fail with EINVAL:

 [EINVAL]   The sum of the iov_len values in the iov array over-
flows a 32-bit integer.

I wouldn’t be surprised if write(2) is implemented using writev(2) and that 
this explains the problem.

 On 18 Jul 2015, at 06:05, Serhiy Storchaka rep...@bugs.python.org wrote:
 
 
 Changes by Serhiy Storchaka storch...@gmail.com:
 
 
 --
 components: +Extension Modules, IO -Interpreter Core
 nosy: +haypo, ned.deily, ronaldoussoren
 
 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue24658
 ___

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24658
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Eric O. LEBIGOT]

Eric O. LEBIGOT added the comment:

Thank you for looking into this, Ronald.

What does your patch do, exactly? does it only limit the returned byte count, 
or does it really limit the size of the data written by truncating it?

In any case, it would be very useful to have a warning from the Python 
interpreter. If the data is truncated, I would even prefer an explicit 
exception (e.g. data too big for this platform (= 2 GB)), along with an 
explicit mention of it in the documentation. What do you think?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24658
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Ronald Oussoren]

Ronald Oussoren added the comment:

The attached patch is a first stab at a workaround. It will unconditionally 
limit the write size in os.write to INT_MAX on OSX.

I haven't tested yet if this actually fixes the problem mentioned on stack 
overflow.

--
keywords: +needs review, patch
stage:  - patch review
Added file: http://bugs.python.org/file39960/issue24658.txt

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24658
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Eric O. LEBIGOT]

Eric O. LEBIGOT added the comment:

I see, thanks.

This sounds good to me too: no need for a warning or exception, indeed, since 
file.write() should work and the behavior of os.write() is documented.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24658
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[STINNER Victor]

STINNER Victor added the comment:

The Windows limit to INT_MAX is one many functions:

* os.write()
* io.FileIO.write()
* hum, maybe other, I don't remember

In the default branch, there is now _Py_write(), so only one place should be 
fixed.

See the issue #11395 which fixed the bug on Windows.

If it's a bug, it should be fixed on Python 2.7, 3.4, 3.5 and default branches.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24658
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Ronald Oussoren]

Ronald Oussoren added the comment:

The patch I attached earlier is for the default branch. More work is needed for 
the other active branches.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24658
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Mali Akmanalp]

Mali Akmanalp added the comment:

I don't know how helpful it is at this point, but the issue happens while 
reading also.

Here's some related discussion in the numpy tracker:

https://github.com/numpy/numpy/issues/3858 (The claim was that OSX Mavericks 
fixed this issue, it didn't, and there is an Apple bug ID in there somewhere, 
plus there is a link to a patch the torch folks used)

and also in pandas: https://github.com/pydata/pandas/issues/10641

I'd be happy to try to test patches out.

--
nosy: +Mali Akmanalp

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24658
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Eric O. LEBIGOT]

Eric O. LEBIGOT added the comment:

PS: I should have written 2 GB bytearray (so this looks like a signed 32 bit 
integer issue).

--
title: open().write() fails on 4 GB+ data (OS X) - open().write() fails on 2 
GB+ data (OS X)

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24658
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24658] open().write() fails on 2 GB+ data (OS X)

[Serhiy Storchaka]

Changes by Serhiy Storchaka storch...@gmail.com:


--
components: +Extension Modules, IO -Interpreter Core
nosy: +haypo, ned.deily, ronaldoussoren

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24658
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com