[issue37696] FileIO.read() on a closed TTY throws an exception prematurely

2019-08-08 Thread David Wilson
David Wilson added the comment: A real example of where returning the partial buffer is dangerous would be EBADF. - Repeated reading succeeds, building up a partial buffer. Another thread runs, and buggy code causes an unrelated fd blonging to the file object to be closed. - Original

[issue37696] FileIO.read() on a closed TTY throws an exception prematurely

2019-08-08 Thread David Wilson
David Wilson added the comment: If we treat different errnos specially, the list of 'okay to silently fail' errors seems quite succinct. In another project I treat EIO, EPIPE and ECONNRESET as EOF, and raise all others -- https://github.com/dw/mitogen/blob

[issue37696] FileIO.read() on a closed TTY throws an exception prematurely

2019-08-08 Thread David Wilson
David Wilson added the comment: Interesting, this immediately turns into a little rabbit hole :) The reason read() is failing in this case, is because argument clinic defaults the size parameter to -1, which redirects the call to readall(). So this issue is actually about readall

[issue37696] FileIO.read() on a closed TTY throws an exception prematurely

2019-07-28 Thread David Wilson
David Wilson added the comment: Happy to send a patch for this if we can agree on the semantic being incorrect, and more importantly, someone is happy to review the patch once it reaches GitHub ;) -- ___ Python tracker <https://bugs.python.

[issue37696] FileIO.read() on a closed TTY throws an exception prematurely

2019-07-28 Thread David Wilson
New submission from David Wilson : Given: $ cat tty-failure.py import pty import os master, slave = pty.openpty() master = os.fdopen(master, 'r+b', 0) slave = os.fdopen(slave, 'r+b', 0) slave.write(b'foo') slave.close() print(master.read()) On Python 2

[issue36279] os.wait3() leaks some uninitialized stack when no processes exist

2019-07-26 Thread David Wilson
David Wilson added the comment: The original diff is attached here (per the old process) so others can find it, and the PR+fork are closed, as carrying a fork in my GitHub for 4 months has non-zero cost. I'm presently more interested in having a clean GH account than carrying around

[issue36279] os.wait3() leaks some uninitialized stack when no processes exist

2019-03-12 Thread David Wilson
New submission from David Wilson : Not sure if this is worth reporting.. p = os.popen('sleep 1234') os.wait3(os.WNOHANG) os.wait3(os.WNOHANG) os.wait3(os.WNOHANG) Notice struct rusage return value. When wait3() succeeds on Linux, but no child was waiting to be reaped, is not updated

[issue35486] subprocess module import hooks breaks back compatibility

2019-01-08 Thread David Wilson
David Wilson added the comment: Hi Nick, The purpose of ModuleNotFoundError is clear and unrelated to the problem documented here. The problem is that due to the introduction of ModuleNotFoundError, ImportError's semantics have been changed within a minor release in a breaking, backwards

[issue35486] subprocess module breaks backwards compatibility with import hooks

2018-12-13 Thread David Wilson
David Wilson added the comment: Having considered this for a few hours, it seems the following is clear: - 3.6 introduces ModuleImportError - 3.7 begins using it within importlib - 3.8 first instance of stdlib code catching it - PEP-302 and PEP-451 are the definitive specifications for how

[issue35486] subprocess module breaks backwards compatibility with import hooks

2018-12-13 Thread David Wilson
Change by David Wilson : -- title: subprocess module breaks backwards compatibility with older import hooks -> subprocess module breaks backwards compatibility with import hooks ___ Python tracker <https://bugs.python.org/issu

[issue35486] subprocess module breaks backwards compatibility with older import hooks

2018-12-13 Thread David Wilson
David Wilson added the comment: Corrected GitHub link for the commit: https://github.com/python/cpython/commit/880d42a3b24 -- ___ Python tracker <https://bugs.python.org/issue35

[issue35486] subprocess module breaks backwards compatibility with older import hooks

2018-12-13 Thread David Wilson
New submission from David Wilson : The subprocess package since 880d42a3b24 / September 2018 has begun using this idiom: try: import _foo except ModuleNotFoundError: bar However, ModuleNotFoundError may not be thrown by older import hook implementations, since that subclass was only

[issue26601] Use new madvise()'s MADV_FREE on the private heap

2016-04-21 Thread David Wilson
David Wilson added the comment: @Julian note that ARENA_SIZE is double the threshold after which at least glibc resorts to calling mmap directly, so using malloc in place of mmap on at least Linux would have zero effect -- nosy: +dw ___ Python

[issue22003] BytesIO copy-on-write

2015-03-04 Thread David Wilson
David Wilson added the comment: Hi Piotr, There wasn't an obvious fix that didn't involve changing the buffer interface itself. There is presently ambiguity in the interface regarding the difference between a read only buffer and an immutable buffer, which is crucial for its use in this case

[issue22003] BytesIO copy-on-write

2015-02-09 Thread David Wilson
David Wilson added the comment: Attached trivial patch for whatsnew.rst. -- Added file: http://bugs.python.org/file38058/whatsnew.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22003

[issue14099] ZipFile.open() should not reopen the underlying file

2014-12-02 Thread David Wilson
David Wilson added the comment: Could we also make a small tweak to zipfile.rst indicating the new behaviour? I had made an initial attempt in my patch but wasn't particularly happy with the wording. -- ___ Python tracker rep...@bugs.python.org

[issue14099] ZipFile.open() should not reopen the underlying file

2014-12-02 Thread David Wilson
David Wilson added the comment: Sounds great :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14099 ___ ___ Python-bugs-list mailing list

[issue14099] ZipFile.open() should not reopen the underlying file

2014-11-21 Thread David Wilson
David Wilson added the comment: While in spirit this is a bug fix, it's reasonably complex and affects a popular module -- I'm not sure it should be applied to 2.x, and probably not in a minor release of 3.x either. Would it make sense to include as part of 3.5? (That said, I'd love to see

[issue14099] ZipFile.open() should not reopen the underlying file

2014-11-14 Thread David Wilson
David Wilson added the comment: Hi Serhiy, Thanks for the new patch, it looks better than my attempt. :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14099

[issue14099] ZipFile.open() should not reopen the underlying file

2014-11-13 Thread David Wilson
David Wilson added the comment: Per my comment on issue16569, the overhead of performing one seek before each (raw file data) read is quite minimal. I have attached a new (but incomplete) patch, on which the following microbenchmarks are based. The patch is essentially identical to Stepan's

[issue22842] zipfile simultaneous open broken and/or needlessly(?) consumes unreasonable number of file descriptors

2014-11-10 Thread David Wilson
New submission from David Wilson: There is some really funky behaviour in the zipfile module, where, depending on whether zipfile.ZipFile() is passed a string filename or a file-like object, one of two things happens: a) Given a file-like object, zipfile does not (since it cannot) consume

[issue22842] zipfile simultaneous open broken and/or needlessly(?) consumes unreasonable number of file descriptors

2014-11-10 Thread David Wilson
David Wilson added the comment: As a random side-note, this is another case where I really wish Python had a .pread() function. It's uniquely valuable for coordinating positioned reads in a threaded app without synchronization (at user level anyway) or extraneous system calls

[issue16569] Preventing errors of simultaneous access in zipfile

2014-11-10 Thread David Wilson
David Wilson added the comment: Compared to the cost of everything else ZipExtFile must do (e.g. 4kb string concatenation in a loop, zlib), its surprising that lseek() would measurable at all. The attached file 'patch' is the minimal change I tested. It represents, in terms of computation

[issue15381] Optimize BytesIO to do less reallocations when written, similarly to StringIO

2014-08-13 Thread David Wilson
David Wilson added the comment: Hey Serhiy, The implementation for your readline optimization seems less contentious (and less risky) than the remainder of the patch -- it could perhaps be easily split off into a separate patch, which may be far more easily committed. I love the concept

[issue22125] Cure signedness warnings introduced by #22003

2014-08-02 Thread David Wilson
New submission from David Wilson: The attached patch (hopefully) silences the signedness warnings generated by Visual Studio and reported on python-dev in https://mail.python.org/pipermail/python-dev/2014-July/135603.html. This was sloppiness on my part, I even noted the problem

[issue22003] BytesIO copy-on-write

2014-07-29 Thread David Wilson
David Wilson added the comment: I suspect it's all covered now, but is there anything else I can help with to get this patch pushed along its merry way? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22003

[issue22003] BytesIO copy-on-write

2014-07-28 Thread David Wilson
David Wilson added the comment: Newest patch incorporates Antoine's review comments. The final benchmark results are below. Just curious, what causes e.g. telco to differ up to 7% between runs? That's really huge Report on Linux k2 3.14-1-amd64 #1 SMP Debian 3.14.9-1 (2014-06-30) x86_64 Total

[issue22003] BytesIO copy-on-write

2014-07-27 Thread David Wilson
David Wilson added the comment: Hey Antoine, Thanks for the link. I'm having trouble getting reproducible results at present, and running out of ideas as to what might be causing it. Even after totally isolating a CPU for e.g. django_v2 and with frequency scaling disabled, numbers still jump

[issue22003] BytesIO copy-on-write

2014-07-24 Thread David Wilson
David Wilson added the comment: This new patch abandons the buffer interface and specializes for Bytes per the comments on this issue. Anyone care to glance at least at the general structure? Tests could probably use a little more work. Microbenchmark seems fine, at least for construction

[issue22003] BytesIO copy-on-write

2014-07-22 Thread David Wilson
David Wilson added the comment: Stefan, I like your new idea. If there isn't some backwards compatibility argument about mmap.mmap being hashable, then it could be considered a bug, and fixed in the same hypothetical future release that includes this BytesIO change. The only cost now

[issue22003] BytesIO copy-on-write

2014-07-21 Thread David Wilson
David Wilson added the comment: Stefan, Thanks for digging here. As much as I'd love to follow this interpretation, it simply doesn't match existing buffer implementations, including within the standard library. For example, mmap.mmap(..., flags=mmap.MAP_SHARED, prot=mmap.PROT_READ

[issue22003] BytesIO copy-on-write

2014-07-21 Thread David Wilson
David Wilson added the comment: I'm not sure how much work it would be, or even if it could be made sufficient to solve our problem, but what about extending the buffers interface to include a int stable flag, defaulting to 0? It seems though, that it would just be making the already arcane

[issue22003] BytesIO copy-on-write

2014-07-21 Thread David Wilson
Changes by David Wilson d...@botanicus.net: Added file: http://bugs.python.org/file36016/cow4.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22003

[issue22003] BytesIO copy-on-write

2014-07-21 Thread David Wilson
David Wilson added the comment: Hi Stefan, How does this approach in reinit() look? We first ask for a writable buffer, and if the object obliges, immediately copy it. Otherwise if it refused, ask for a read-only buffer, and this time expect that it will never change. This still does

[issue22003] BytesIO copy-on-write

2014-07-20 Thread David Wilson
David Wilson added the comment: This version is tidied up enough that I think it could be reviewed. Changes are: * Defer `buf' allocation until __init__, rather than __new__ as was previously done. Now upon completion, BytesIO.__new__ returns a valid, closed BytesIO, whereas previously

[issue22003] BytesIO copy-on-write

2014-07-20 Thread David Wilson
David Wilson added the comment: New patch also calls unshare() during getbuffer() -- Added file: http://bugs.python.org/file36005/cow3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22003

[issue22003] BytesIO copy-on-write

2014-07-20 Thread David Wilson
David Wilson added the comment: I'm not sure the read only buffer test is strong enough: having a readonly view is not a guarantee that the data in the view cannot be changed through some other means, i.e. it is read-only, not immutable. Pretty sure this approach is broken. What about

[issue22003] BytesIO copy-on-write

2014-07-18 Thread David Wilson
David Wilson added the comment: Good catch :( There doesn't seem to be way a to ask for an immutable buffer, so perhaps it could just be a little more selective. I think the majority of use cases would still be covered if the sharing behaviour was restricted only to BytesType. In that case

[issue22003] BytesIO copy-on-write

2014-07-17 Thread David Wilson
New submission from David Wilson: This is a followup to the thread at https://mail.python.org/pipermail/python-dev/2014-July/135543.html , discussing the existing behaviour of BytesIO copying its source object, and how this regresses compared to cStringIO.StringI. The goal of posting

[issue22003] BytesIO copy-on-write

2014-07-17 Thread David Wilson
David Wilson added the comment: Submitted contributor agreement. Please consider the demo patch licensed under the Apache 2 licence. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22003