[issue26297] Move constant folding to AST level

[STINNER Victor]

STINNER Victor added the comment:

> we need to move some optimizations from bytecode level to AST level

You may be aware that there is a long list of similar (old) issues proposing 
something similar:
https://www.python.org/dev/peps/pep-0511/#usage-1-ast-optimizer

My PEP 511 tries to propose a generic API to support all proposed optimizations.

--

I didn't try yet to do something like importlib to compile an AST optimizer 
written in Python into a frozen module and then use it in Python. It would 
allow to implement an AST optimizer in pure Python to ease its maintenance 
*and* have it builtin into CPython, rather than having to play with AST in C 
(less fun).

My plan for the PEP 511 is more to start to experiment multiple AST optimizers 
and bytecode optimizers outside CPython, wait until the code becomes more 
stable, and later discuss if one optimizer deserves to become part of Python 
stdlib.

"Writing an optimizer or a preprocessor is out of the scope of this PEP."

--

But maybe you would prefer to have a single and simple AST optimizer builtin 
Python without new additional API?

I don't want to block your effort on experimentation :-) I just see a nice 
opportunity to sell you the PEP 511 ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26287] Core dump in f-string with formatting errors

[Roundup Robot]

Roundup Robot added the comment:

New changeset 9095a5787a82 by Eric V. Smith in branch 'default':
Fix issue 26287: While handling FORMAT_VALUE opcode, the top of stack was being 
corrupted if an error occurred in PyObject_Format().
https://hg.python.org/cpython/rev/9095a5787a82

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24683] Type confusion in json encoding

[paul]

paul added the comment:

Sorry, I wasn't clear enough. This POC is a proof that the original bug can be 
used for EIP control. I just checked and it works as advertised on 2.7 
revision: https://hg.python.org/cpython/rev/2d39777f3477 - it's a parent of 
https://hg.python.org/cpython/rev/0a1266ef1b5d containing the patch for this 
issue. I added this file, because I submitted a bug on hackerone claiming EIP 
control.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Yury Selivanov added the comment:

As to weather we want this patch committed or not, here's a 
mini-macro-something benchmark:


$ ./python.exe -m timeit -s "x=2" "x + 10 + x * 20  + x* 10 + 20 -x"
1000 loops, best of 3: 0.115 usec per loop

$ python3.5 -m timeit -s "x=2" "x + 10 + x * 20  + x* 10 + 20 -x"
1000 loops, best of 3: 0.141 usec per loop


$ ./python.exe -m timeit -s "x=2" "x*2.2 + 2 + x*2.5 + 1.0 - x / 2.0 + 
(x+0.1)/(x-0.1)*2 + (x+10)*(x-30)"
100 loops, best of 3: 0.308 usec per loop

$ python3.5 -m timeit -s "x=2" "x*2.2 + 2 + x*2.5 + 1.0 - x / 2.0 + 
(x+0.1)/(x-0.1)*2 + (x+10)*(x-30)"
100 loops, best of 3: 0.652 usec per loop


Still, longs are faster 30-50%, FP are faster 100%.  I think it's a very good 
result.  Please don't block this patch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: [issue21955] ceval.c: implement fast path for integers with a single digit

[M.-A. Lemburg]

On 05.02.2016 16:14, STINNER Victor wrote:
> 
> Please don't. I would like to have time to benchmark all these patches (there 
> are now 9 patches attached to the issue :-)) and I would like to hear 
> Serhiy's feedback on your latest patches.

Regardless of the performance, the fastint5.patch looks like the
least invasive approach to me. It also doesn't incur as much
maintenance overhead as the others do.

I'd only rename the macro MAYBE_DISPATCH_FAST_NUM_OP to
TRY_FAST_NUMOP_DISPATCH :-)

BTW: I do wonder why this approach is as fast as the others. Have
compilers grown smart enough to realize that the number slot
functions will not change and can thus be inlined ?

-- 
Marc-Andre Lemburg
eGenix.com

___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[STINNER Victor]

STINNER Victor added the comment:

My analysis of benchmarks.

Even using CPU isolation to run benchmarks, the results look unreliable for 
very short benchmarks like 3 ** 2.0: I don't think that fastint_alt can make 
the operation 16% slower since it doesn't touch this code, no?

Well... as expected, speedup is quite *small*: the largest difference is on "3 
* 2" ran 100 times: 18% faster with fastint_alt. We are talking about 1.82 us 
=> 1.49 us: delta of 330 ns. I expect a much larger difference is you compile a 
function to machine code using Cython or a JIT like Numba and PyPy. Remember 
that we are running *micro*-benchmarks, so we should not push overkill 
optimizations except if the speedup is really impressive.

It's quite obvious from the tables than fastint_alt.patch only optimize int 
(float is not optimized). If we choose to optimize float too, 
fastintfloat_alt.patch and fastint5.patch look to have the *same* speed.

I don't see any overhead on Decimal + Decimal with any patch: good.

--

Between fastintfloat_alt.patch and fastint5.patch, I prefer 
fastintfloat_alt.patch which is much easier to read, so probably much easier to 
debug. I hate huge macro when I have to debug code in gdb :-( I also like very 
much the idea of *reusing* existing functions, rather than duplicating code.

Even if Antoine doesn't seem interested by optimizations on float, I think that 
it's ok to add a few lines for this type, fastintfloat_alt.patch is not so 
complex. What do *you* think?

Why not optimizing a**b? It's a common operation, especially 2**k, no?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Yury Selivanov added the comment:

Anyways, if it's about macro vs non-macro, I can inline the macro by hand 
(which I think is an inferior approach here).  But I'd like the final code to 
use my approach of using slots directly, instead of modifying 
longobject/floatobject to export lots of *internal* stuff.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Yury Selivanov added the comment:

Unless there are any objections, I'll commit fastint5.patch in a day or two.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Yury Selivanov added the comment:

>> Unless there are any objections, I'll commit fastint5.patch in a day or two.

> Please don't. I would like to have time to benchmark all these patches (there 
> are now 9 patches attached to the issue :-)) and I would like to hear 
> Serhiy's feedback on your latest patches.

Sure, I'd very appreciate a review of fastint5.

I can save you sometime on benchmarking -- it's really about fastint_alt vs 
fastint5.  The latter optimizes ALL ops on longs AND floats.  The former only 
optimizes some ops on longs.  So please be sure you're comparing oranges to 
oranges ;)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[STINNER Victor]

STINNER Victor added the comment:

bench_long2.py: my updated microbenchmark to test many types and more 
operations.

compare.txt: compare Python original, fastint_alt.patch, fastintfloat_alt.patch 
and fastint5.patch. "(*)" marks the minimum of the line, percents are relative 
to the minimum (if larger than +/-5%).

compare_to.txt: similar to compare.txt, but percents are relative to the 
original Python.

--
Added file: http://bugs.python.org/file41821/bench_long2.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19217] Calling assertEquals for moderately long list takes too long

[Chris AtLee]

Chris AtLee added the comment:

Switching this to unified_diff allows the test case to finish nearly instantly.

The downside is that the output is changed in the case of a difference found:

FF
==
FAIL: test_compare (__main__.SCSE)
--
Traceback (most recent call last):
  File "test_case.py", line 14, in test_compare
self.assertEqual(arr1, arr2)
AssertionError: Lists differ: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,[29953 
chars]1, 1] != [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,[29953 chars]2, 2]

First differing element 0:
1
2

Diff is 100034 characters long. Set self.maxDiff to None to see it.

==
FAIL: test_compare_2 (__main__.SCSE)
--
Traceback (most recent call last):
  File "test_case.py", line 24, in test_compare_2
self.assertEqual(arr1, arr2)
AssertionError: Lists differ: [1, 1[29932 chars] 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] != [1, 1[29932 chars] 1, 1, 1, 1, 1, 1, 1, 1, 
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2]

First differing element :
1
2

--- 
+++ 
@@ -9997,4 +9997,4 @@
  1,
  1,
  1,
- 1]
+ 2]

--
Ran 2 tests in 0.098s

FAILED (failures=2)


--
nosy: +Chris AtLee

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Yury Selivanov added the comment:

> Regardless of the performance, the fastint5.patch looks like the
least invasive approach to me. It also doesn't incur as much
maintenance overhead as the others do.

Thanks.  It's a result of an enlightenment that can only come
after running benchmarks all day :)

> I'd only rename the macro MAYBE_DISPATCH_FAST_NUM_OP to
TRY_FAST_NUMOP_DISPATCH :-)

Yeah, your name is better.

> BTW: I do wonder why this approach is as fast as the others. Have
compilers grown smart enough to realize that the number slot
functions will not change and can thus be inlined ?

Looks like so, I'm very impressed myself.  I'd expect fastint3 (which just 
inlines a lot of logic directly in ceval.c) to be the fastest one.  But it 
seems that compiler does an excellent job here.

Victor, BTW, if you want to test fastint3 vs fastint5, don't forget to apply 
the patch from issue #26288 over fastint5 (fixes slow performance of 
PyLong_AsDouble)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Yury Selivanov added the comment:

> Between fastintfloat_alt.patch and fastint5.patch, I prefer 
> fastintfloat_alt.patch which is much easier to read, so probably much easier 
> to debug. I hate huge macro when I have to debug code in gdb :-( I also like 
> very much the idea of *reusing* existing functions, rather than duplicating 
> code.

I disagree.

fastintfloat_alt exports a lot of functions from longobject/floatobject, 
something that I really don't like.  Lots of repetitive code in ceval.c also 
make it harder to make sure everything is correct.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19217] Calling assertEquals for moderately long list takes too long

[Chris AtLee]

Changes by Chris AtLee :


Added file: http://bugs.python.org/file41824/cpython-issue19217.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26280] ceval: Optimize [] operation similarly to CPython 2.7

[Zach Byrne]

Zach Byrne added the comment:

I'm attaching output from a selection of the benchmarks, I'm counting 
non-builtins and slices, but for everything, not just lists and tuples.

Quick observation: math workloads seem list heavy, text workloads seem dict 
heavy, and tuples are usually somewhere in the middle.

--
Added file: http://bugs.python.org/file41826/subscr_stats.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

My patches were just samples. I'm glad that Yury incorporated the main idea and 
that this helps. If apply any patch I would prefer fastint5.patch. But I don't 
quite understand why it adds any gain. Is this just due to overhead of calling 
PyNumber_Add? Then we should test with other compilers and with the LTO option. 
fastint5.patch adds an overhead for type checks and increases the size of ceval 
loop. What is outweigh this overhead?

As for tests, it would be more honest to test data that results out of small 
ints range (-5..256).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23551] IDLE to provide menu link to PIP gui.

[Upendra Kumar]

Upendra Kumar added the comment:

I trying to write the Tk application for the pip package manager. It is in very 
initial stage, with none of the functionalities implemented yet. I have 
attached the current state of code written by me. I have also included the 
wrapper provided by Terri for pip command line. 

Please can someone review my current code , if I am on correct path? If there 
are some objections to my coding style and the structure of program, please do 
tell me.

--
Added file: http://bugs.python.org/file41825/tem_pip_v2.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26287] Core dump in f-string with formatting errors due to refcount bug

[Eric V. Smith]

Eric V. Smith added the comment:

The problem has to do with refcounting when an error occurs. Adjusting the 
title accordingly.

I'm not sure yet if the problem is in PyObject_Format(), or in handling errors 
in the eval loop when processing FORMAT_VALUE opcodes. I'm slowly tracking it 
down. It doesn't happen with format(), so I suspect it's in FORMAT_VALUE.

Here's an example showing the error with a large, non-cached int. You need to 
call it twice to trigger the refcount problem.

>>> f'{1000:j}'
Traceback (most recent call last):
  File "", line 1, in 
ValueError: Unknown format code 'j' for object of type 'int'
>>> f'{1000:j}'
Fatal Python error: Objects/tupleobject.c:233 object at 0x7f904006b360 has 
negative ref count -2604246222170760230

Current thread 0x7f9041278700 (most recent call first):
Aborted (core dumped)

--
title: Core dump in f-string with lambda and format specification -> Core dump 
in f-string with formatting errors due to refcount bug

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[STINNER Victor]

STINNER Victor added the comment:

> Unless there are any objections, I'll commit fastint5.patch in a day or two.

Please don't. I would like to have time to benchmark all these patches (there 
are now 9 patches attached to the issue :-)) and I would like to hear Serhiy's 
feedback on your latest patches.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24683] Type confusion in json encoding

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The same result on 2.7 branch:

$ ./python ../cpython/eip.py 
Traceback (most recent call last):
  File "../cpython/eip.py", line 21, in 
e = j.make_encoder(markers, None, enc, 4, "ks", "is", False, True, True)
TypeError: make_encoder() argument 1 must be dict or None, not str

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22107] tempfile module misinterprets access denied error on Windows

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26296] colorys rgb_to_hls algorithm error

[Shashank Agarwal]

Shashank Agarwal added the comment:

I would like to work on this..

--
nosy: +The_Knight

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Yury Selivanov added the comment:

Thanks, Serhiy,

> But I don't quite understand why it adds any gain. 

Perhaps, and this is just a guess - the fast path does only couple of eq tests 
& one call for the actual op.  If it's long+long then long_add will be called 
directly.

PyNumber_Add has more overhead on:
- at least one extra call
- a few extra checks to guard against NotImplemented
- abstract.c/binary_op1 has a few more checks/slot lookups

So it look that there's just much less instructions to be executed.  If this 
guess is correct, then an LTO build without fast paths will still be somewhat 
slower.

> Is this just due to overhead of calling PyNumber_Add? Then we should test 
> with other compilers and with the LTO option.

I actually tried to compile CPython with LTO -- but couldn't.  Almost all of C 
extension modules failed to link.  Do we compile official binaries with LTO?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26298] Split ceval.c into small files

[STINNER Victor]

New submission from STINNER Victor:

Attached patch splits the huge "switch (opcode)" of ceval.c into smaller 
ceval_xxx.h files. New files:

   93 Python/ceval_stack.h
  142 Python/ceval_condjump.h
  155 Python/ceval_misc.h
  162 Python/ceval_fast.h
  180 Python/ceval_module.h
  238 Python/ceval_ctx.h
  249 Python/ceval_func.h
  262 Python/ceval_iter.h
  268 Python/ceval_build.h
  384 Python/ceval_number.h

Maybe we can put more files per .h file, maybe less. I don't really care.

It will allow to keep the code readable even with new optimizations like the 
issue #21955.

What do you think?

--
files: split_ceval.patch
keywords: patch
messages: 259682
nosy: haypo, pitrou, serhiy.storchaka, yselivanov
priority: normal
severity: normal
status: open
title: Split ceval.c into small files
versions: Python 3.6
Added file: http://bugs.python.org/file41827/split_ceval.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26298] Split ceval.c into small files

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I don't think this will make the code more readable. Rather less readable, 
since macros are defined in different file than used.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26297] Move constant folding to AST level

[Serhiy Storchaka]

New submission from Serhiy Storchaka:

For using more efficient bytecode (either specialized 8-bit opcodes or 16-bit 
opcodes) we need to move some optimizations from bytecode level to AST level, 
since LOAD_CONST variants could have variable size. Now with the Constant node 
this should be easy.

--
components: Interpreter Core
messages: 259680
nosy: abarnert, benjamin.peterson, brett.cannon, georg.brandl, haypo, ncoghlan, 
serhiy.storchaka, yselivanov
priority: normal
severity: normal
stage: needs patch
status: open
title: Move constant folding to AST level
type: enhancement
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24683] Type confusion in json encoding

[paul]

paul added the comment:

Can you try on 2.7 branch?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26299] wsgiref.util FileWrapper raises ValueError: I/O operation on closed file.

[Samwyse]

New submission from Samwyse:

While developing, I am using wsgiref.simple_server.  Using to serve static 
content isn't working.  The attached program demonstrates the issue.  Run it 
and connect to http://127.0.0.1:8000/.  You will see three buttons.  Clicking 
on 'wsgi.filewrapper' causes the FileWrapper class found in wsgiref.util to be 
used, while clicking on 'PEP 0333' uses code suggested in PEP 0333 
(https://www.python.org/dev/peps/pep-0333/#id36).  Both of these fail.  
Clicking on 'slurp' causes the entire file to loaded and returned in a list.  
This works.

When an application returns an instance of environ['wsgi.file_wrapper'], or 
creates it's own iterator, an error is raised during the initial read of the 
file.  Reading the entire file and returning a single-element list (the 'slurp' 
technique) works, but is impractical for larger files.

I've tested this with both Python 2.7.9 and 3.4.3 under Windows 7 (my laptop) 
and 3.4.3 under Alpine Linux (a Docker instance).

--
components: Library (Lib)
files: wsgitest.py
messages: 259685
nosy: samwyse
priority: normal
severity: normal
status: open
title: wsgiref.util FileWrapper raises ValueError: I/O operation on closed file.
versions: Python 2.7, Python 3.4
Added file: http://bugs.python.org/file41828/wsgitest.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24683] Type confusion in json encoding

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I can't reproduce your example paul.

$ ./python eip.py 
Traceback (most recent call last):
  File "eip.py", line 21, in 
e = j.make_encoder(markers, None, enc, 4, "ks", "is", False, True, True)
TypeError: make_encoder() argument 1 must be dict or None, not array.array

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21328] Resize doesn't change reported length on create_string_buffer()

[Tamás Bence Gedai]

Tamás Bence Gedai added the comment:

Thanks for the remarks, I think the issue can be closed as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1927] raw_input behavior incorrect if readline not enabled

[Martin Panter]

Changes by Martin Panter :


--
Removed message: http://bugs.python.org/msg259637

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21328] Resize doesn't change reported length on create_string_buffer()

[Dustin Oprea]

Dustin Oprea added the comment:

I'm closing it. The ticket has been open two-years and no one else seemed to be 
interested in this issue until now, which leads me to believe that it's a 
PEBCAK/understanding issue. The rationale for why it's irrelevant seems sound. 
Thanks for digging through the code, Tamás. Thank you for your comments, Martin 
and Eryk.

--
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[STINNER Victor]

Changes by STINNER Victor :


Added file: http://bugs.python.org/file41823/compare_to.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[STINNER Victor]

Changes by STINNER Victor :


Added file: http://bugs.python.org/file41822/compare.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26292] Raw I/O writelines() broken

[Martin Panter]

Martin Panter added the comment:

For BufferedIOBase, the documentation says write() will not return a short 
number of bytes, so I don’t see how writelines() is a problem there. For 
TextIOBase, the documentation is not so clear 
, 
but I understand the standard library implementations do not do short writes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[STINNER Victor]

STINNER Victor added the comment:

Serhiy Storchaka: "My patches were just samples. I'm glad that Yury 
incorporated the main idea and that this helps."

Oh, if even Serhiy prefers Yury's patches, I should read them again :-)

--

I read fastint5.patch one more time and I finally understood the following 
macros:

+#define NB_SLOT(slot) offsetof(PyNumberMethods, slot)
+#define NB_BINOP(nb_methods, slot) \
+(*(binaryfunc*)(& ((char*)nb_methods)[NB_SLOT(slot)]))
+#define PY_LONG_CALL_BINOP(slot, left, right) \
+(NB_BINOP(PyLong_Type.tp_as_number, slot))(left, right)
+#define PY_FLOAT_CALL_BINOP(slot, left, right) \
+(NB_BINOP(PyFloat_Type.tp_as_number, slot))(left, right)

In short, a+b calls long_add(a, b) with that. At the first read, I understood 
that it casted objects to C long or C double (don't ask me why).


I see a difference between fastint5.patch and fastintfloat_alt.patch: 
fastint5.patch resolves the address of long_add() at runtime, whereas 
fastintfloat_alt.patch gets a direct pointer to _PyLong_Add() at the 
compilation. I expected a sublte speedup, but I'm unable to see it on 
benchmarks (again, both patches have the same speed).

The float path is simpler in fastint5.patch because it uses the same code if 
right is float or long, but it adds more checks for the slow-path. No patch 
looks to have a real impact on the slow-path. Is it worth to change the second 
if to PyFloat_CheckExact() and then check type of right in the if body to avoid 
other checks on the slow-path?

(C checks look very cheap, so I think that I already replied to my own question 
:-))

--

fastint5.patch optimizes a+b, a-b, a*b, a/b and a//b. Why not other operators? 
List of operators from my constant folding optimzation in fatoptimizer:

* int, float: a+b, a-b, a*b, a/b, +x, -x, ~x, a//b, a%b, a**b
* int only: a<>b, a, a|b, a^b

If we optimize a//b, I suggest to also optimize a%b to be consistent. For 
integers, a**b, a<>b would make sense too. Coming from the C language, 
I would prefer a<>b than a*2**k or a//2**k, since I expect better 
performance.

For float, -x and +x may be common, but less a+b, a-b, a*b, a/b.

Well, what I'm trying to say: if choose fastintfloat_alt.patch design, we will 
have to expose like a lot of new C functions in headers, and duplicate a lot of 
code.

To support more than 4 operators, we need a macro.

If we use a macro, it's cheap (in term of code maintenance) to use it for most 
or even all operators.

--

> But I don't quite understand why it adds any gain. Is this just due to 
> overhead of calling PyNumber_Add?

Hum, that's a good question.


> Then we should test with other compilers and with the LTO option.

There are projects (I don't recall the number number) but I would prefer to 
handle LTO separatly. Python supports platforms and compilers which don't 
implement LTO.


> fastint5.patch adds an overhead for type checks and increases the size of 
> ceval loop. What is outweigh this overhead?

I stopped to guess the speedup just by reading the code or a patch. I only 
trust benchmarks :-)

Advice: don't trust yourself! only trust benchmarks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26297] Move constant folding to AST level

[STINNER Victor]

STINNER Victor added the comment:

Yury Selivanov added the comment:
> In fact I'm -1 on using any kind of flag to trigger optimizations on or off.  
> Those flags will only drive confusion.  If the optimizations are any good, 
> people will simply start using the flag all the time.

Yeah, Guido calls this a missed optimization opportunity.

My only concern is:

> ast.Constant for tuples and frozensets has a limitation: it doesn't store the 
> location of items (line number, column offset).

Is it a problem in practice? I didn't have time to investigate this.

Since the PEP 511 is still under discussion, I prefer to focus on the
PEP itself than this specific implementation detail ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26297] Move constant folding to AST level

[STINNER Victor]

STINNER Victor added the comment:

*If* the PEP 511 is accepted, it will be super easy to start with an
implementation in pure Python.

But I think that we should benchmark the overhead of the Python API of
PEP 511, because we need to convert all AST internal objects to Python
objects, run the AST optimizer, and then again convert Python objects
to internal AST objects.

For a long running process, the time to compile a .py file doesn't
matter. For a short script, it matters. At least, we need to compile
the script itself.

By the way, would it be insane to *not* optimize the script when
running "python script.py"?

> Now with the Constant node this should be easy.

ast.Constant is *not* emited by the compiler to not break backward
compatibility. I *know* that there is no stable API on AST, but I
noticed some issues when working on my AST project. For example, pip
doesn't work because an internal library uses AST and the code doesn't
handle ast.Constant (it's probably super easy to fix it).

I'm open to change the compiler to emit ast.Constant directly, but
maybe only in "optimized mode" (ex: python -O?). ast.Constant for
tuples and frozensets has a limitation: it doesn't store the location
of items (line number, column offset).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26300] "unpacked" bytecode

[Andrew Barnert]

New submission from Andrew Barnert:

Currently, the compiler starts with a list of arrays of instructions, packs 
them to 1/3/6-bytes-apiece bytecodes, fixes up all the jumps, and then calls 
PyCode_Optimize on the result. This makes the peephole optimizer much more 
complicated. Assuming PEP 511 is accepted, it will also make plug-in bytecode 
optimizers much more complicated (and probably wasteful--they'll each be 
repeating the same work to re-do the fixups).

The simplest alternative (as suggested by Serhiy on -ideas) is to expose an 
"unpacked" bytecode to the optimizer (in the code parameter and return value 
and lnotab_obj in-out parameter for PyCode_Optimize, and similarly for PEP 511) 
where each instruction takes a fixed 4 bytes. This is much easier to process. 
After the optimizer returns, the compiler packs opcodes into the usual 
1/3/6-byte format, removing NOPs, retargeting jumps, and adjusting the lnotab 
as it goes. (Note that it already pretty much has code to do all of this except 
the NOP removal; it's just doing it before the optimizer instead of after.)

Negatives:

 * Arguments can now only go up to 2**23 instead of 2**31. I don't think that's 
a problem (has anyone ever created a code object with 4 million instructions?).

 * A bit more work for the compiler; we'd need to test to make sure there's no 
measurable performance impact.

We could also expose this functionality through C API PyCode_Pack/Unpack and 
Python dis.pack_code/unpack_code functions (and also make the dis module know 
how to parse unpacked code), which would allow import hooks, post-processing 
decorators, etc. to be simplified as well. This would remove some, but not all, 
of the need for things like byteplay. I think this may be worth doing, but I'm 
not sure until I see how complicated it is.

We could even allow code objects with unpacked bytecode to be executed, but I 
think that's unnecessary complexity. Nobody should want to do that 
intentionally, and if an optimizer lets such code escape by accident, a 
SystemError is fine.

MRAB implied an alternative: exposing some slightly-higher-level label-based 
format. That would be even nicer to work with. But it's also more complicated 
for the compiler and for the API, and I think it's already easy enough to 
handle jumps with fixed-width instructions.

--
components: Interpreter Core
messages: 259693
nosy: abarnert, benjamin.peterson, georg.brandl, haypo, pitrou, 
serhiy.storchaka, yselivanov
priority: normal
severity: normal
status: open
title: "unpacked" bytecode
type: enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26297] Move constant folding to AST level

[Yury Selivanov]

Yury Selivanov added the comment:

> ast.Constant is *not* emited by the compiler to not break backward
compatibility. I *know* that there is no stable API on AST, but I
noticed some issues when working on my AST project. For example, pip
doesn't work because an internal library uses AST and the code doesn't
handle ast.Constant (it's probably super easy to fix it).

Yeah, it's probably very easy to fix.  And if pip stops working on 3.6-alpha it 
will be fixed.  We don't give any guarantees on AST backwards compatibility.  
In fact, we even tell that the AST might change in every release in the docs 
"The abstract syntax itself might change with each Python release".


> I'm open to change the compiler to emit ast.Constant directly, but
maybe only in "optimized mode" (ex: python -O?). 

I'm -1 on using -O flag since some people would prefer to have optimizations 
*and* have their assert statements.

In fact I'm -1 on using any kind of flag to trigger optimizations on or off.  
Those flags will only drive confusion.  If the optimizations are any good, 
people will simply start using the flag all the time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26287] Core dump in f-string with formatting errors due to refcount bug

[Eric V. Smith]

Eric V. Smith added the comment:

Yes, that's my thinking as well. I'll have a patch soon-ish.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26287] Core dump in f-string with formatting errors

[Eric V. Smith]

Eric V. Smith added the comment:

Now I think it's not a refcount problem.

I'm going to switch to POP/PUSH instead of TOP/SET_TOP. I believe the problem 
is that I'm not adjusting the top of stack if there's an error.

--
title: Core dump in f-string with formatting errors due to refcount bug -> Core 
dump in f-string with formatting errors

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26287] Core dump in f-string with formatting errors

[Eric V. Smith]

Eric V. Smith added the comment:

Thanks for the report and the debugging help.

--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26298] Split ceval.c into small files

[Brett Cannon]

Brett Cannon added the comment:

I have a similar worry as Serhiy as I don't know where to find something like 
GET_AWAITABLE with that organization.

--
nosy: +brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25228] Regression in cookie parsing with brackets and quotes

[Guido van Rossum]

Changes by Guido van Rossum :


--
nosy:  -gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22107] tempfile module misinterprets access denied error on Windows

[Mark Lawrence]

Changes by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26287] Core dump in f-string with formatting errors due to refcount bug

[Martin Panter]

Martin Panter added the comment:

I have zero experience with with ceval.c but I have a theory after looking over 
revision 1ddeb2e175df. I suspect “value” is a borrowed reference from TOP(), 
but at  
and  you 
are calling DECREF on it without calling SET_TOP() if there is an error.

I suspected FORMAT_VALUE, because the crash doesn’t happen when you call 
format() directly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26248] Improve scandir DirEntry docs, especially re symlinks and caching

[Guido van Rossum]

Guido van Rossum added the comment:

Thanks very much for all that added information!

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26293] Embedded zipfile fields dependent on absolute position

[spoo]

spoo added the comment:

I'm not familiar with iOS development, but I'd hazard a guess that "some zip 
library implementations" means a (the official?) iOS zip library.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

STINNER Victor added the comment:

Rebased patch combining also pycf_transformed_ast.patch.

--
Added file: http://bugs.python.org/file41816/transformers-5.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26280] ceval: Optimize [] operation similarly to CPython 2.7

[Antoine Pitrou]

Antoine Pitrou added the comment:

The test suite is not an appropriate workload to run benchmarks or statistics. 
Can you run with the benchmarks suite instead?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26293] Embedded zipfile fields dependent on absolute position

[spoo]

New submission from spoo:

Example:

from zipfile import ZipFile
with open('a.zipp', 'wb') as base:
base.write(b'old\n')
with ZipFile(base, 'a') as myzip:
myzip.write('eggs.txt')

If the embedded zip portion of the file is extracted (first four bytes 
deleted), some fields will be incorrect in the resultant file - commenting out 
line 3 produces a file that can serve as a comparison.  These differences cause 
issues opening with some zip library implementations.

My best guess is that this is related to this line: 
https://github.com/python/cpython/blob/master/Lib/zipfile.py#L1459

--
components: Library (Lib)
messages: 259642
nosy: spoo
priority: normal
severity: normal
status: open
title: Embedded zipfile fields dependent on absolute position
type: behavior
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26292] Raw I/O writelines() broken

[STINNER Victor]

STINNER Victor added the comment:

"(it's probably broken with non-blocking streams too, for the same
reason)"

Yes it is :-)

We hitted this issue on eventlet when I changed the socket.send() method in 
eventlet 0.18 to stop sending data on partial write, whereas eventlet < 0.18 
used a loop to ensure that all bytes are written.

The side effect of my change is that (indirectly) 
socket.makefile().writelines() may also use partial write if the underlying 
send() only writes partially data.

The problem is that writelines() has *no* result, so the caller cannot be aware 
of the partial write and data is lost...

eventlet:

* send() change: https://github.com/eventlet/eventlet/issues/274
* writelines() bug: https://github.com/eventlet/eventlet/issues/295


"In the spirit of RawIO.write(), I think RawIO.writelines() could return
the number of bytes written (allowing for partial writes)."

I don't know yet what is the best option for eventlet, but we should probaly 
enhance the Python stdlib io module to return something on writelines() and 
*document* the behaviour on partial write.

The problem writelines() API is that writelines() not only takes a single 
text/bytes string, but a *list* of text/bytes strings.

Another option is to modify to retry on partial write to ensure that all data 
is written. If the underlying write() method returns None (blocking I/O error), 
we must raise an error. It doesn't make sense to retry writing on a 
non-blocking I/O error.

In this case, we must document that writelines() *must not* be used on 
non-blocking I/O (ex: non-blocking socket, pipe, whatever).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26292] Raw I/O writelines() broken

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +pitrou
versions: +Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26293] Embedded zipfile fields dependent on absolute position

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
assignee:  -> serhiy.storchaka
nosy: +alanmcintyre, serhiy.storchaka, twouters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26098] PEP 510: Specialize functions with guards

[STINNER Victor]

STINNER Victor added the comment:

Oh, I missed comments on the code review. Fixed on patch version 8.

--
Added file: http://bugs.python.org/file41818/specialize-8.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26292] Raw I/O writelines() broken

[STINNER Victor]

New submission from STINNER Victor:

Copy of Antoine Pitrou's email (sent in 2012 ;-):
https://mail.python.org/pipermail/python-dev/2012-August/121396.html

Hello,

I was considering a FileIO.writelines() implementation based on
writev() and I noticed that the current RawIO.writelines()
implementation is broken: RawIO.write() can return a partial write but
writelines() ignores the result and happily proceeds to the next
iterator item (and None is returned at the end).

(it's probably broken with non-blocking streams too, for the same
reason)

In the spirit of RawIO.write(), I think RawIO.writelines() could return
the number of bytes written (allowing for partial writes).

Regards

Antoine.

-- 
Software development and contracting: http://pro.pitrou.net

--
components: IO
messages: 259640
nosy: haypo
priority: normal
severity: normal
status: open
title: Raw I/O writelines() broken
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26294] Queue().unfinished_tasks not in docs - deliberate?

[Frank Millman]

New submission from Frank Millman:

dir(queue.Queue()) shows an attribute 'unfinished_tasks'.

It appears to be the counter referred to in the docs to 'join()', but it is not 
documented itself.

I would like to make use of it, but I don't know if it is part of the official 
API for this module.

Please advise.

--
assignee: docs@python
components: Documentation
messages: 259645
nosy: docs@python, frankmillman
priority: normal
severity: normal
status: open
title: Queue().unfinished_tasks not in docs - deliberate?
versions: Python 3.4, Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21328] Resize doesn't change reported length on create_string_buffer()

[Eryk Sun]

Changes by Eryk Sun :


--
resolution:  -> not a bug
stage: patch review -> resolved

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26295] Random failures when running test suite in parallel (-m test -j0) caused by test_regrtest

[STINNER Victor]

New submission from STINNER Victor:

test_regrtest creates temporary test files called test_regrtest_pid_xxx.py in 
Lib/test/. The problem is that some tests like test___all__ and test_zipfile 
haves test relying on the list of Lib/test/test_*.py.

When tests are run in parallel, test_regrtest can creates temporary 
test_regrtest_pid_xxx.py files, test_zipfile sees them, test_regrtest removes 
them, and then test_zipfiles fails.

The best would be to write these temporary files into a temporary directory, 
but I failed to fix Lib/test/regrtest.py to load tests from a different 
directory. In theory, Python 3 supports packages with files splitted into 
multiple directories, in practice it doesn't seem to work :-p

Maybe test_regrtest should use a test submodule like Lib/test/temp_regrtest/ ?

test_regrtest started to create temporary test_xxx.py files since issue #25220. 
(Other changes to test_regrtest: issues #18174, #22806, #25260, #25306,  
#25369, #25373, #25694).

==
ERROR: test_all (test.test___all__.AllTest)
--
Traceback (most recent call last):
  File "/home/haypo/prog/python/default/Lib/test/test___all__.py", line 102, in 
test_all
with open(path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 
'/home/haypo/prog/python/default/Lib/test/test_regrtest_25743_noop20.py'

--

--
components: Tests
messages: 259647
nosy: brett.cannon, haypo
priority: normal
severity: normal
status: open
title: Random failures when running test suite in parallel (-m test -j0) caused 
by test_regrtest
versions: Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26287] Core dump in f-string with lambda and format specification

[Eric V. Smith]

Changes by Eric V. Smith :


--
assignee:  -> eric.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26295] Random failures when running test suite in parallel (-m test -j0) caused by test_regrtest

[STINNER Victor]

STINNER Victor added the comment:

> The directory with test files can be read-only.

test_regrtest skips tests which requires to write in Lib/test/ if the write 
fails.

except PermissionError as exc:
if not sysconfig.is_python_build():
self.skipTest("cannot write %s: %s" % (path, exc))

> test_regrtest should use temporary directory.

Yes! That's the purpose of this issue!

But I was unable to find a technical solution to implement this. I need to try 
harder :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26292] Raw I/O writelines() broken

[STINNER Victor]

STINNER Victor added the comment:

> Is deprecating RawIOBase.writelines() an option, and only recommending 
> BufferedIOBase.writelines() and TextIOBase.writelines()?

IMHO the problem is the same for other classes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26285] Garbage collection of unused input sections from CPython binaries

[Alecsandru Patrascu]

Alecsandru Patrascu added the comment:

I've done again the experiments on larger workloads, such as our OpenStack 
Swift cluster, and it works without any issues.

Also, I've attached an archive with a simple external module in CPython3 that 
uses PyMem_RawMalloc. The output is ok, and it's copied bellow.


u@palecsandru:~/w/experimente/c_ext3$ /home/u/w/cpython3_deadcode/python 
setup.py build_ext --inplace
running build_ext
building 'mytest' extension
creating build
creating build/temp.linux-x86_64-3.6
gcc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g 
-fwrapv -O3 -Wall -Wstrict-prototypes -fdata-sections -ffunction-sections 
-Wl,--gc-sections -fPIC -I/home/u/w/cpython3_deadcode/Include 
-I/home/u/w/cpython3_deadcode -c mytest.c -o 
build/temp.linux-x86_64-3.6/mytest.o
gcc -pthread -shared build/temp.linux-x86_64-3.6/mytest.o -o 
/home/u/w/experimente/c_ext3/mytest.cpython-36m-x86_64-linux-gnu.so

u@palecsandru:~/w/experimente/c_ext3$ ll
total 40
drwxrwxr-x  3 u u  4096 Feb  5 14:29 ./
drwxr-xr-x 12 u u  4096 Feb  5 14:00 ../
drwxrwxr-x  3 u u  4096 Feb  5 14:29 build/
-rw-rw-r--  1 u u   619 Feb  5 14:16 mytest.c
-rwxrwxr-x  1 u u 17856 Feb  5 14:29 mytest.cpython-36m-x86_64-linux-gnu.so*
-rw-rw-r--  1 u u   132 Feb  5 14:15 setup.py

u@palecsandru:~/w/experimente/c_ext3$ /home/u/w/cpython3_deadcode/python
Python 3.6.0a0 (default:87dfadd61e0d+, Feb  5 2016, 14:22:57) 
[GCC 5.2.1 20151010] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import mytest
>>> mytest.mytest()
'test'
>>>

--
Added file: http://bugs.python.org/file41819/c_ext3.zip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26292] Raw I/O writelines() broken

[Martin Panter]

Martin Panter added the comment:

Is deprecating RawIOBase.writelines() an option, and only recommending 
BufferedIOBase.writelines() and TextIOBase.writelines()?

Otherwise, I think it would make most sense to keep retrying until all the data 
is written. This mirrors how I understand readline() and readall() work (keeps 
reading until it gets as much as necessary).

For non-blocking mode, readline() does not support that (see Issue 13858). It 
does not make much sense to me to have writelines() support non-blocking mode 
either.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26296] colorys rgb_to_hls algorithm error

[Mats Luspa]

New submission from Mats Luspa:

In the colorsys library function rgb_to_hls the algorithm is not implemented 
quite correctly.

According to algorithm the correct implementation should be (the error is in 
the condition r == maxc). In the current code g

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26296] colorys rgb_to_hls algorithm error

[STINNER Victor]

STINNER Victor added the comment:

Oh, someone uses colorsys :-)

--
nosy: +haypo, yselivanov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26248] Improve scandir DirEntry docs, especially re symlinks and caching

[STINNER Victor]

STINNER Victor added the comment:

Ben Hoyt added the comment:
> Seeing this has been merged (thanks Victor), can this issue be closed?

As I wrote, I left the issue open for wait for feedback. I know that
some people to read the doc online:
https://docs.python.org/dev/library/os.html#os.DirEntry

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18199] Windows: support path longer than 260 bytes using "\\?\" prefix

[Jens Diemer]

Jens Diemer added the comment:

I also with this problems.

I have made a test script.

There is a problem with os.chdir(): It doesn't work with \\?\ notation.
And there is also a problem, if you use 

```
import os
import pathlib
import tempfile

with tempfile.TemporaryDirectory(prefix="path_test_") as path:
new_path = pathlib.Path(path, "A"*255, "B"*255)
extended_path = "?\\%s" % new_path
os.makedirs(extended_path)
print(len(extended_path), extended_path)
```
os.makedirs() will work, but the cleanup will failed.
Output is:
```
567 
\\?\C:\Users\jens\AppData\Local\Temp\path_test_8fe6utdz\AAA\BBB
Traceback (most recent call last):
  File "D:\test2.py", line 18, in 
print(len(extended_path), extended_path)
  File "C:\Program Files (x86)\Python35-32\lib\tempfile.py", line 807, in 
__exit__
self.cleanup()
  File "C:\Program Files (x86)\Python35-32\lib\tempfile.py", line 811, in 
cleanup
_shutil.rmtree(self.name)
  File "C:\Program Files (x86)\Python35-32\lib\shutil.py", line 488, in rmtree
return _rmtree_unsafe(path, onerror)
  File "C:\Program Files (x86)\Python35-32\lib\shutil.py", line 383, in 
_rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Program Files (x86)\Python35-32\lib\shutil.py", line 381, in 
_rmtree_unsafe
os.unlink(fullname)
FileNotFoundError: [WinError 3] Das System kann den angegebenen Pfad nicht 
finden: 
'C:\\Users\\jens\\AppData\\Local\\Temp\\path_test_8fe6utdz\\AAA'
```

--
nosy: +jens
Added file: http://bugs.python.org/file41820/test.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26248] Improve scandir DirEntry docs, especially re symlinks and caching

[Ben Hoyt]

Ben Hoyt added the comment:

Seeing this has been merged (thanks Victor), can this issue be closed? Guido, 
are you happy with the changes given your comments at 
http://bugs.python.org/issue26032#msg257665?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1927] raw_input behavior incorrect if readline not enabled

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I think we first should fix 3.6 in correct way, and then see what we can 
backport to other branches without breaking too much. Or left them as is. For 
example the output of --version was changed from strerr to stdout (issue18338, 
issue18920) in default branch, but this was not backported.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23551] IDLE to provide menu link to PIP gui.

[Upendra Kumar]

Upendra Kumar added the comment:

@Terry, I am an intended GSOC student and I came through this idea of 
developing a GUI based on Tkinter in the python core-mentorship mailing lists. 
You highlighted/mentioned this issue in the mailing list.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26295] Random failures when running test suite in parallel (-m test -j0) caused by test_regrtest

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The directory with test files can be read-only. test_regrtest should use 
temporary directory.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26292] Raw I/O writelines() broken

[Jakub Stasiak]

Changes by Jakub Stasiak :


--
nosy: +jstasiak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26280] ceval: Optimize [] operation similarly to CPython 2.7

[Antoine Pitrou]

Antoine Pitrou added the comment:

I also suggest counting the number of BINARY_SUBSCR calls that are *not* one of 
the builtin types under consideration. Also, it would be good to distinguish 
slicing from integer indexing, for lists and tuples.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26098] PEP 510: Specialize functions with guards

[STINNER Victor]

STINNER Victor added the comment:

Patch version 7:

* Fix a random crash related to _testcapi.PyGuard: implement tp_traverse on 
PyFuncGuard and "inherit" tp_traverse on PyGuard
* Fix a typo Include/funcobject.h
* (rebase the patch)

--
Added file: http://bugs.python.org/file41817/specialize-7.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26296] colorys rgb_to_hls algorithm error

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
keywords: +easy
stage:  -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25228] Regression in cookie parsing with brackets and quotes

[Will Harris]

Changes by Will Harris :


--
nosy: +harris

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26301] ceval.c: reintroduce fast-path for list[index] in BINARY_SUBSCR

[Yury Selivanov]

Yury Selivanov added the comment:

I think this is a duplicate of http://bugs.python.org/issue26280...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[STINNER Victor]

STINNER Victor added the comment:

msg223186, Serhiy Storchaka about inline.patch: "Confirmed speed up about 20%. 
Surprisingly it affects even integers outside of the of preallocated small 
integers (-5...255)."

The optimization applies to Python int with 0 or 1 digit so in range [-2^30+1; 
2^30-1].

Small integers in [-5; 255] might be faster but for PyLong_FromLong().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26288] Optimize PyLong_AsDouble for single-digit longs

[STINNER Victor]

STINNER Victor added the comment:

Nice enhancement.

/* Fast path; single digit will always fit decimal.
   This improves performance of FP/long operations by at
   least 20%. This is even visible on macro-benchmarks.
*/

I'm not sure that "spectral_norm" can be qualified as macro-benchmark. It's 
more a microbenchmark on numerics, no? I'm just nitpicking, your patch is fine 
;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26240] Docstring of the subprocess module should be cleaned up

[Terry J. Reedy]

Terry J. Reedy added the comment:

I agree. The module docstring should briefly describe the module and maybe list 
the contents, a line for each.  The itertools docstring does the latter; the 
math docstring does not, though I would not mind if it did.

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[STINNER Victor]

STINNER Victor added the comment:

myself> Ok. Now I'm lost. We have so many patches :-) Which one do you prefer?

I read again fully this *old* issue, well, *almost* all messages.

Well, it's clear that no consensus was found yet :-) I see two main trends: 
optimize most cases (optimize most operators for int and float,  ex: 
fastint5_4.patch) versus optimize very few cases to limit changes and to limit 
effects on ceval.c (ex: inline-2.patch).

Marc-Andre and Antoine asked to not stick to micro-optimizations but think 
wider: run macro benchmarks, like perf.py, and suggest to use PyPy, Numba, 
Cython & cie for users who use best performances on numeric functions.

They also warned about subtle side-effects of any kind of change on ceval.c 
which may be counter-productive. It was shown in the long list of patches that 
some of them introduced performance *regressions*.

I don't expect that CPython can beat any compiler emiting machine code. CPython 
will always have to pay the price of boxing/unboxing and its loop evaluating 
bytecode. We can do *better*, the question is "how far?".

I think that we gone far enough on investigation *all* different options to 
optimize 1+2 ;-) Each option was micro-benchmarked very carefully.

Now I suggest to focus on *macro* benchmarks to help use to take a decision. I 
will run perf.py on fastint5_4.patch and inline-2.patch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26301] ceval.c: reintroduce fast-path for list[index] in BINARY_SUBSCR

[STINNER Victor]

STINNER Victor added the comment:

> I think this is a duplicate of http://bugs.python.org/issue26280...

Oh, I missed this one. I didn't understand "[]" in the title. I didn't 
understand the purpose of optimizing BUILD_LIST :-D

--
resolution:  -> duplicate
status: open -> closed
superseder:  -> ceval: Optimize list[int] (subscript) operation similarly to 
CPython 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26301] ceval.c: reintroduce fast-path for list[index] in BINARY_SUBSCR

[STINNER Victor]

New submission from STINNER Victor:

Copy of msg222985 by Raymond Hettinger from issue #21955:

"There also used to be a fast path for binary subscriptions with integer 
indexes.  I would like to see that performance regression fixed if it can be 
done cleanly."

--
messages: 259708
nosy: haypo, rhettinger, yselivanov
priority: normal
severity: normal
status: open
title: ceval.c: reintroduce fast-path for list[index] in BINARY_SUBSCR
type: performance
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26280] ceval: Optimize list[int] (subscript) operation similarly to CPython 2.7

[STINNER Victor]

STINNER Victor added the comment:

Oh, I just created a duplicate with a patch for list[int]: issue #26301.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26280] ceval: Optimize list[int] (subscript) operation similarly to CPython 2.7

[STINNER Victor]

Changes by STINNER Victor :


--
title: ceval: Optimize [] operation similarly to CPython 2.7 -> ceval: Optimize 
list[int] (subscript) operation similarly to CPython 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26288] Optimize PyLong_AsDouble for single-digit longs

[Roundup Robot]

Roundup Robot added the comment:

New changeset 986184c355e8 by Yury Selivanov in branch 'default':
Issue #26288: Optimize PyLong_AsDouble.
https://hg.python.org/cpython/rev/986184c355e8

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26288] Optimize PyLong_AsDouble for single-digit longs

[Yury Selivanov]

Yury Selivanov added the comment:

Thanks a lot for the review, Serhiy!

--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed
type:  -> performance

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10044] small int optimization

[STINNER Victor]

STINNER Victor added the comment:

Issue #21955 is a spin-off of this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Changes by Yury Selivanov :


Added file: http://bugs.python.org/file41831/fastint5_4.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26301] ceval.c: reintroduce fast-path for list[index] in BINARY_SUBSCR

[STINNER Victor]

STINNER Victor added the comment:

(Oops, I attached the wrong patch, fixed in patch v2.)

Quick & dirty micro-benchmark:

Original:

$ ./python -m timeit -s 'lst=list("hello")' 'lst[2]'
1000 loops, best of 3: 0.0261 usec per loop

Patched:

$ ./python -m timeit -s 'lst=list("hello")' 'lst[2]'
1000 loops, best of 3: 0.0186 usec per loop

--
Added file: http://bugs.python.org/file41834/binary_subscr-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26301] ceval.c: reintroduce fast-path for list[index] in BINARY_SUBSCR

[STINNER Victor]

Changes by STINNER Victor :


Removed file: http://bugs.python.org/file41833/binary_subscr.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Yury Selivanov added the comment:

Attached is the new version of fastint5 patch.  I fixed most of the review 
comments.  I also optimized %, << and >> operators.  I didn't optimize other 
operators because they are less common.  I guess we have to draw a line 
somwhere...

Victor, thanks a lot for your suggestion to drop NB_SLOT etc macros!  Without 
them the code is even simpler.

--
Added file: http://bugs.python.org/file41829/fastint5_2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Changes by Yury Selivanov :


Added file: http://bugs.python.org/file41830/fastint5_3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[STINNER Victor]

STINNER Victor added the comment:

inline-2.patch: more complete version of inline.patch.

Optimize the same instructions than Python 2: BINARY_ADD, INPLACE_ADD, 
BINARY_SUBSTRACT, INPLACE_SUBSTRACT.


Quick & *dirty* microbenchmark:

$ ./python -m timeit -s 'x=1' 'x+x+x+x+x+x+x+x+x+x+x+x+x+x+x+x+x+x+x+x+x+x+x'

* Original: 287 ns
* fastint5_2.patch: 261 ns (-9%)
* inline-2.patch: 212 ns (-26%)


$ ./python -m timeit -s 'x=1000; y=1' 
'x-y-y-y-y-y-y-y-y-y-y-y-y-y-y-y-y-y-y-y-y-y-y-y-y'

* Original: 517 ns
* fastint5_2.patch: 469 ns (-9%)
* inline-2.patch: 442 ns (-15%)


Ok. Now I'm lost. We have so many patches :-) Which one do you prefer?

In term of speedup, I expect that Python 2 design (inline-2.patch) cannot be 
beaten in term of performance by another another option since it doesn't need 
any C code and does everything in ceval.c.

--
Added file: http://bugs.python.org/file41832/inline-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Yury Selivanov added the comment:

> Ok. Now I'm lost. We have so many patches :-) Which one do you prefer?

To no-ones surprise I prefer fastint5, because it optimizes almost all binary 
operators on both ints and floats.

inline-2.patch only optimizes just + and - for just ints.  If + and - 
performance of inline-2 is really important, I suggest to merge it in fastint5, 
but i'd keep it simple ;)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26301] ceval.c: reintroduce fast-path for list[index] in BINARY_SUBSCR

[STINNER Victor]

Changes by STINNER Victor :


--
keywords: +patch
Added file: http://bugs.python.org/file41833/binary_subscr.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[STINNER Victor]

STINNER Victor added the comment:

msg222985: Raymond Hettinger
"There also used to be a fast path for binary subscriptions with integer 
indexes.  I would like to see that performance regression fixed if it can be 
done cleanly."

The issue #26280 was opened to track this optimization.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21159] configparser.InterpolationMissingOptionError is not very intuitive

[Ned Deily]

Changes by Ned Deily :


--
resolution: fixed -> 
stage: resolved -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com