[issue32145] Wrong ExitStack Callback recipe

[Nick Coghlan]

Nick Coghlan  added the comment:

As per the comment at https://bugs.python.org/issue32445#msg309356, there's a 
bug in my suggested changes to `ExitStack.pop_all()`: the right method to call 
is ExitStack.push(), *not* ExitStack.callback() (the latter adds a wrapper 
function to make the signatures match, but our stored callbacks all already 
have the right signature).

I'm not too fussy about the details of the docstring, but we need to be careful 
about the guarantees we make to ExitStack subclasses: if the docstring implies 
that "target.push()" will always be called, then we need to limit the stack 
stealing behaviour to actual ExitStack instances (which may be a good idea 
anyway).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32445] Skip creating redundant wrapper functions in ExitStack.callback

[Nick Coghlan]

Nick Coghlan  added the comment:

Ah, you're right, I wasn't thinking clearly when I filed this - the correct 
public API method to transfer existing exit stack entries to a new stack would 
be "ExitStack.push()".

I'll make a note of that on the original issue.

--
resolution:  -> not a bug
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32478] Add tests for 'break' and 'return' inside 'finally' clause

[Roundup Robot]

Change by Roundup Robot :


--
pull_requests: +4958

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32452] Brackets and Parentheses used in an ambiguous way

[Roundup Robot]

Change by Roundup Robot :


--
pull_requests: +4957

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32452] Brackets and Parentheses used in an ambiguous way

[Roundup Robot]

Change by Roundup Robot :


--
pull_requests: +4956

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32452] Brackets and Parentheses used in an ambiguous way

[R. David Murray]

R. David Murray  added the comment:


New changeset f190eb59e60e2ae7a7cbd396458389a7a076e0d3 by R. David Murray 
(Emily Morehouse) in branch 'master':
bpo-32452: clarify term 'brackets' in generator tutorial (#5079)
https://github.com/python/cpython/commit/f190eb59e60e2ae7a7cbd396458389a7a076e0d3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32478] Add tests for 'break' and 'return' inside 'finally' clause

[Roundup Robot]

Change by Roundup Robot :


--
pull_requests: +4955

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32478] Add tests for 'break' and 'return' inside 'finally' clause

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 7cc42c356b0dc5ad9eaa9392789e84bd4aa1c7de by Serhiy Storchaka in 
branch 'master':
bpo-32478: Add tests for 'break' and 'return' inside 'finally' clause. (#5078)
https://github.com/python/cpython/commit/7cc42c356b0dc5ad9eaa9392789e84bd4aa1c7de


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32452] Brackets and Parentheses used in an ambiguous way

[Emily Morehouse]

Change by Emily Morehouse :


--
nosy: +emilyemorehouse

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32452] Brackets and Parentheses used in an ambiguous way

[Emily Morehouse]

Change by Emily Morehouse :


--
keywords: +patch
pull_requests: +4954
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32453] shutil.rmtree can have O(n^2) performance on large dirs

[Niklas Hambüchen]

Niklas Hambüchen  added the comment:

A better location to view the whole coreutils thread is:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=29921

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32478] Add tests for 'break' and 'return' inside 'finally' clause

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +4953
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32478] Add tests for 'break' and 'return' inside 'finally' clause

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

'break' and 'return' are never used inside 'finally' clause in the stdlib. 
Therefore these cases are not covered by tests even implicitly. The proposed 
patch adds explicit tests for them.

'continue' inside 'finally' clause currently is prohibited (see issue1542451) 
and there are tests for this.

--
components: Tests
messages: 309352
nosy: serhiy.storchaka
priority: normal
severity: normal
status: open
title: Add tests for 'break' and 'return' inside 'finally' clause
type: enhancement
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32477] Move jumps optimization from the peepholer to the compiler

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +4952
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32477] Move jumps optimization from the peepholer to the compiler

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

The proposed patch moves jumps optimization from the peepholer to the compiler. 
The optimization is performed for lists of instructions before generating 
concrete bytecode and is not restricted by the size of bytecode instructions. 
It allows to optimize more long chains of jumps.

This is a step to getting rid of the peepholer.

--
components: Interpreter Core
messages: 309351
nosy: benjamin.peterson, brett.cannon, ncoghlan, serhiy.storchaka, yselivanov
priority: normal
severity: normal
status: open
title: Move jumps optimization from the peepholer to the compiler
type: performance
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17611] Move unwinding of stack for "pseudo exceptions" from interpreter to compiler.

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +4951

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32474] argparse nargs should support string wrapped integers too

[R. David Murray]

R. David Murray  added the comment:

Agreed.  I don't think there is sufficient motivation for doing this, and there 
are downsides as Eric has pointed out.  Rejecting.

--
resolution:  -> rejected
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17611] Move unwinding of stack for "pseudo exceptions" from interpreter to compiler.

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> I just worked on it.

I meant Mark's PR :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17611] Move unwinding of stack for "pseudo exceptions" from interpreter to compiler.

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

> It would be nice to know if it's able to fix the stack size issues as in the 
> following tests:

I just worked on it. See issue24340.

--
dependencies: +co_stacksize estimate can be highly off

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24340] co_stacksize estimate can be highly off

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Tests originally based on Antoine's tests added for PR 2827.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24340] co_stacksize estimate can be highly off

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

With PR 5076 the result of the above example is 10.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24340] co_stacksize estimate can be highly off

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +4950

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32474] argparse nargs should support string wrapped integers too

[Andrew Svetlov]

Andrew Svetlov  added the comment:

-1

--
nosy: +asvetlov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32476] Add concat functionality to ElementTree xpath find

[John Jolly]

New submission from John Jolly :

The current implementation of the xpath specification in ElementTree does not 
allow for predicates that may contain both an apostrophe and a quote. Either 
one or the other is allowed, but escaping (either string escaping using the 
backslash or html escaping) is not supported.

A simple solution would be to implement the xpath concat function for 
predicates. This would allow the combination of quote types in the predicate. 
Attached is a proposed patch for this fix.

--
components: Library (Lib)
files: etree-concat.patch
keywords: patch
messages: 309344
nosy: jjolly
priority: normal
severity: normal
status: open
title: Add concat functionality to ElementTree xpath find
type: enhancement
versions: Python 3.7
Added file: https://bugs.python.org/file47360/etree-concat.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32416] Refactor and add new tests for the f_lineno setter

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 439ce8a93936d92e4f10765c482195a28e93ec76 by Serhiy Storchaka 
(Miss Islington (bot)) in branch '3.6':
bpo-32416: Add two new tests in test_sys_settrace. (GH-5072) (#5073)
https://github.com/python/cpython/commit/439ce8a93936d92e4f10765c482195a28e93ec76


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32416] Refactor and add new tests for the f_lineno setter

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 2de47ca109d0418c7c01e451b5614f748ad76ee9 by Serhiy Storchaka 
(Miss Islington (bot)) in branch '2.7':
bpo-32416: Add two new tests in test_sys_settrace. (GH-5072) (#5074)
https://github.com/python/cpython/commit/2de47ca109d0418c7c01e451b5614f748ad76ee9


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32474] argparse nargs should support string wrapped integers too

[Shubham Sharma]

Change by Shubham Sharma :


--
pull_requests: +4949

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17611] Move unwinding of stack for "pseudo exceptions" from interpreter to compiler.

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I took a quick look at Mark's PR.  The main thing going for it IMHO is that it 
produces simpler bytecode: it removes the complicated with/finally-related 
opcodes, while Serhiy's has non-trivial END_FINALLY and POP_FINALLY.

It would be nice to know if it's able to fix the stack size issues as in the 
following tests:
https://github.com/python/cpython/pull/5006/files?diff=unified#diff-dae68b96e8fdcb924e1ea46c31f51aec

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32416] Refactor and add new tests for the f_lineno setter

[Roundup Robot]

Change by Roundup Robot :


--
pull_requests: +4948

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32416] Refactor and add new tests for the f_lineno setter

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset e8ed96550c6aa9a1e39c36e67e892994e25e2c41 by Serhiy Storchaka in 
branch 'master':
bpo-32416: Add two new tests in test_sys_settrace. (#5072)
https://github.com/python/cpython/commit/e8ed96550c6aa9a1e39c36e67e892994e25e2c41


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32416] Refactor and add new tests for the f_lineno setter

[Roundup Robot]

Change by Roundup Robot :


--
pull_requests: +4947

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17611] Move unwinding of stack for "pseudo exceptions" from interpreter to compiler.

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

New opcodes in PR 5071 look interesting, but I think it is better to leave this 
experiment for a separate issue.

Complexities of PRs (ignoring tests, documentation, and generated files):

PR 4682:
 Include/opcode.h  |  12 +-
 Lib/opcode.py |  21 +-
 Objects/frameobject.c | 241 -
 Objects/genobject.c   |  10 +-
 Python/ceval.c| 426 +++
 Python/compile.c  | 884 +---
 Python/peephole.c |  18 +-
 7 files changed, 979 insertions(+), 633 deletions(-)

PR 5006:
 Include/opcode.h  |   8 +-
 Lib/opcode.py |  11 +-
 Objects/frameobject.c | 203 -
 Objects/genobject.c   |  10 +-
 Python/ceval.c| 344 ---
 Python/compile.c  | 474 +---
 Python/peephole.c |  33 ++--
 7 files changed, 561 insertions(+), 522 deletions(-)

PR 5071:
 Include/frameobject.h |   5 +-
 Include/opcode.h  |  18 +-
 Lib/opcode.py |  25 +-
 Objects/frameobject.c | 224 ++-
 Objects/genobject.c   |  10 +-
 Python/ceval.c| 507 -
 Python/compile.c  | 681 +++-
 Python/peephole.c |  49 +---
 Python/wordcode_helpers.h |  33 ++-
 9 files changed, 827 insertions(+), 725 deletions(-)

PR 4682: +4 opcodes, -4 opcodes, and changed numerical values of 2 opcodes
PR 5006: +4 opcodes, -4 opcodes
PR 5071: +8 opcodes, -8 opcodes, and changed numerical value of 1 opcode

It looks to me that PR 5006 has the lowest complexity and is the most easy for 
reviewing.

I'm working on it since taking Antoine's PR 5 months ago.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27632] build on AIX fails when builddir != srcdir, more than bad path to ld_so_aix

[Michael Felt]

Michael Felt  added the comment:

There were changes made - I know not when - but OOT builds work now. This issue 
may be closed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28290] BETA report: Python-3.6 build messages to stderr: AIX and "not GCC"

[Michael Felt]

Michael Felt  added the comment:

this was just feedback - and should probably be closed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32474] argparse nargs should support string wrapped integers too

[Eric V. Smith]

Change by Eric V. Smith :


--
versions:  -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27643] test_ctypes fails on AIX with xlc

[Michael Felt]

Michael Felt  added the comment:

As I was not responding properly (too verbose) - I'll reread the thread for the 
initial patch suggestion - and hope it still fits the current 'master'.

My apologies for the long silence on this.

--
nosy:  -aixto...@gmail.com

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32416] Refactor and add new tests for the f_lineno setter

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +4946

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17611] Move unwinding of stack for "pseudo exceptions" from interpreter to compiler.

[Mark Shannon]

Mark Shannon  added the comment:

It shouldn't be a surprise that I have submitted a PR. The original patch was 
my work and a month ago I said I would update it over the Christmas break. 

To avoid "competing" PRs in future, simply asking before appropriating other 
peoples work would help. I've signed the CLA, so I don't think there are any 
legal issues with Serhiy's or Neil's PRs containing my code, but it does seem a 
bit presumptive.

Having said that, if someone can convince me that Serhiy's PR is better, then I 
have no problem with it being merged.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32475] Add ability to query number of buffered bytes available on buffered I/O

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Why don't you use BufferedReader.peek()?

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17611] Move unwinding of stack for "pseudo exceptions" from interpreter to compiler.

[Antoine Pitrou]

Antoine Pitrou  added the comment:

So we now have three competing PRs... each of which is not trivial. It would be 
nice if somehow there could be some conciliation, or at least a synthetic 
comparison. I don't think I want to review all three proposals (and I already 
spent time reviewing previous versions of Serhiy's).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32390] AIX (xlc_r) compile error with Modules/posixmodule.c: Function argument assignment between types "unsigned long" and "struct fsid_t" is not allowed

[Michael Felt]

Michael Felt  added the comment:

Did some additional research. The code can work asis when _ALL_SOURCE is 
undefined. Or, in other words - there does not have to be a variable syntax 
issue or separate code for AIX.

As you read this - please remember that since issue #32143 was 'resolved' AIX 
cannot compile.

* 
https://www.gnu.org/software/autoconf/manual/autoconf-2.64/html_node/Posix-Variants.html

describes _ALL_SOURCE as a macro for AIX3.

Looking at AIX: /usr/include/sys/types.h - by undefining _ALL_SOURCE the 
typedefs in the header files are equivalent enough that the new code added for 
os.statvfs(path) is compiled by both gcc and xlc.

As examples (from Centos)
/usr/include/bits/typesizes.h:72:#define __FSID_T_TYPE  struct { int 
__val[2]; }

And from AIX:
 * /usr/include/sys/types.h
  +360  / * typedef for the File System Identifier (fsid).  This must correspond
  +361* to the "struct fsid" structure in _ALL_SOURCE below.
  +362* /
  +363  typedef struct fsid_t {
  +364  #ifdef __64BIT_KERNEL
  +365  unsigned long val[2];
  +366  #else  / * __64BIT_KERNEL * /
  +367  #ifdef _ALL_SOURCE
  +368  unsigned int val[2];
  +369  #else  / * _ALL_SOURCE * /
  +370  unsigned int __val[2];
  +371  #endif / * _ALL_SOURCE * /
  +372  #endif / * __64BIT_KERNEL * /
  +373  } fsid_t;
 */

(Note - a variation of the text above was in the pull request 
https://github.com/python/cpython/pull/4972. Per reviewer request - this is 
removed and a discussion is now open here.)

Thank you for your comments.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17611] Move unwinding of stack for "pseudo exceptions" from interpreter to compiler.

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +4945

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32466] Fix missing test coverage for fractions.Fraction.__new__

[Mark Dickinson]

Mark Dickinson  added the comment:

I've retitled the issue to avoid confusion, since we're no longer proposing 
removing the _gcd function.

--
title: Remove fractions._gcd() -> Fix missing test coverage for 
fractions.Fraction.__new__

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32466] Remove fractions._gcd()

[Mark Dickinson]

Mark Dickinson  added the comment:

> the types in question were added to the numeric tower for gmpy 2.0.9 and 
> 2.1.0.

Ah, good to know. Thanks. I was using 2.0.8.

> * We're back to needing a test for the line in question.

Agreed. It's not currently covered, and with the current behaviour the line 
_is_ necessary.

> * We're eschewing the possibility of changing the behavior of 
> `fractions.Fraction` to force int numerator and denominator.

I think that's a separate issue from this one, and if you want to pursue it 
it's worth its own issue on the tracker. But there would need to be a good 
reason for deliberately changing the existing by-design behaviour.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32474] argparse nargs should support string wrapped integers too

[Shubham Sharma]

Change by Shubham Sharma :


--
keywords: +patch
pull_requests: +4944
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32475] Add ability to query number of buffered bytes available on buffered I/O

[Tim Savannah]

New submission from Tim Savannah :

Hello!

This is my first time submitting to Python bug tracker, so please bear with me 
if I miss/mess something.

So a little bit of relevant background, I'm an avid python developer with many 
open-source projects.

One of the projects I wrote and maintain is called "python-nonblock", which 
provides pure-python non-blocking I/O methods. It is available at:

https://github.com/kata198/python-nonblock


I'll only include the relevant details to this topic.

So, one of the features provided by the python-nonblock in the nonblock_read 
function. This allows you to read from a stream whilst ensuring the operation 
does not block.

It achieves this by basically following this pattern:

1. Call "select" on the stream and see if any data is available. If not, sleep 
and reiterate.

2. If there is data available, it reads a single byte from the stream and 
stores is to return at the end.

It supports most streams and sockets which have a real fd backing (and thus 
support "select").


There are a couple reasons you may need to do this, e.x. certain interactive 
scenarios, I won't go into it too much.


The python-nonblock library also bundles a layer which sits on top of that 
method, called BackgroundRead. This interface launches a thread into the 
background, which reads blocks of arbitrary (user-provided) size into a 
variable on an object. So you could have a processing app which reads blocks of 
data from a source, processes them in the foreground whilst they continue to 
load up in the background.


That's all well and good, but we are getting to the meat of the issue: for 
large sources of available data (like a file on disk), while this method of 
operation is effective, it is SLOW, due to the overhead of a select syscall and 
libpython for every single byte. This is especially true on a loaded system, as 
it makes us a prime candidate for the scheduler to preempt our task and context 
switch us off the cpu!


I've been looking into ways to improve this, and have actually seemed to have 
struck gold. So on a standard linux HDD filesystem, the I/O block size is 4096. 
So, thanks to readahead, on a non-fragmented file, a read call for 1 byte will 
actually load up to 4096 bytes. libpython has this extra data, and calls like 
read1 will return it if available, but it does not expose this number. Thus, 
libraries like mine can't take advantage of it, which means that for a 
filesystem I/O read on linux, 4095 out of 4096 iterations of the two-step loop 
above are wasted effort.

So I've written up an additional function to the C code for BufferedReader, 
"getbuffn", which returns the number bytes currently buffered in libpython, but 
not yet returned to the application, and modified python-nonblock ( in the 
4.0branch_getbuffn branch ) with simple additions to take advantage of this 
extra information, when available. So if we detect that there are 4095 bytes 
already read and pending, we know for certain we can grab all 4095 bytes at 
once without blocking, or even needing a call to select!

So the new pattern for buffered streams that have getbuffn available, we can:

1. Select to see if data is available, if not rest and reiterate

2. Read a single byte off the stream

3. Check getbuffn, and if it returns >0 read that many bytes off the stream 
(Guaranteed to not block)


The performance improvements are * MASSIVE * with this change in place.

   On a 5 meg file from a VM which is running on an SSD, I average the 
following:
 
 Loaded system, non-cached I/O:
 
One-Fell-Swoop file.read() - .3 seconds
 
getbuffn-patched python and this impl - 3.1 seconds
 
unpatched python and this impl - 41 to 55 = 44 seconds. ( avg min - avg 
max)
 
 Unloaded system, cached I/O:
 
One-Fell-Swoop file.read() - .0017 seconds
 
getbuffn-patched python and this impl - .034 seconds
 
unpatched python and this impl - 45 seconds. ( not as variable as 
loaded system )

That's a 13,235% (thirteen-thousand two-hundred and five percent) performance 
boost on just a 5MB file, which just grows exponentially as the size of the 
dataset increases. These gains are just simply not possible without this 
information available (the amount remaining in the buffer).


So I've attached the simple patch (only additions, no modifications to existing 
functions) against python 3.6.4. The version of python-nonblock which supports 
this enhanced approach when available (and more details at the top of the 
README) can be found here: 

https://github.com/kata198/python-nonblock/tree/4.0branch_getbuffn

I've looked at the python 2.7 code too, and it seems that with minimal effort 
this same functionality can be provided there as well!


So, I'm an experienced developer who is willing to put in the legwork. Is this 
something that is possible to get merged upstream? If so, what are the steps I 
would need to take in order to make it so?