[issue26270] Support for read()/write()/select() on asyncio

[STINNER Victor]

STINNER Victor added the comment:

Paulo Costa:
> async def async_read(fd. n):

This method already exists:
* (TCP) 
https://docs.python.org/dev/library/asyncio-protocol.html#asyncio.Protocol.data_received
* (UDP) 
https://docs.python.org/dev/library/asyncio-protocol.html#asyncio.DatagramProtocol.datagram_received
* (TCP) 
https://docs.python.org/dev/library/asyncio-stream.html#asyncio.StreamReader.read


Martin Panter:
> last time I looked they all seemed to buffer your write data and send it in 
> the background. E.g. in the echo-client-using-streams example, there is a 
> writer.write() call that does not use “yield from”. Does asyncio have an easy 
> way for a coroutine to write to a socket (or file descriptor) and block 
> (suspend the coroutine) until the write has finished

Sure, you must call 
https://docs.python.org/dev/library/asyncio-stream.html#asyncio.StreamWriter.drain

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26256] Fast decimalisation and conversion to other bases

[Jurjen N.E. Bos]

Jurjen N.E. Bos added the comment:

That reference you gave says that the binary version is faster than the Python 
version, but here the _complexity_ actually changed by a lot.
Only people who know the library by name will notice that it is this fast.
But you are right, for 99% of the people it doesn't make much of a difference.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21042] ctypes.util.find_library() should return full pathname instead of filename in linux

[Gedai Tamás Bence]

Gedai Tamás Bence added the comment:

Added a small patch that solves this issue on Ubuntu 15.10.

Produces output like:
/lib/x86_64-linux-gnu/libm.so.6
/lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libbz2.so.1.0

I'd be glad to add some test cases if someone can give me some tips on how to 
do that.

--
keywords: +patch
nosy: +beng94
Added file: http://bugs.python.org/file41789/find_lib.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26272] `zipfile.ZipFile` fails reading a file object in specific version(s)

[Pengyu Chen]

Pengyu Chen added the comment:

Oh yes you're right Martin. Thanks for the notice :) (I've closed this ticket 
before seeing your reply)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26274] Add CPU affinity to perf.py

[STINNER Victor]

STINNER Victor added the comment:

I like the feature, but I would prefer to use a syntax like --affinity=0-2 to 
use cores 0, 1 and 2, rather than having to compute manually a binary mask 
(--affinity=3).

It's a builtin feature of taskset: just use --cpu-list.

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26272] `zipfile.ZipFile` fails reading a file object in specific version(s)

[Martin Panter]

Martin Panter added the comment:

I think you are only meant to pass in a “binary” (bytes) file object. Maybe 
BufferedIOBase, I’m not sure. The documentation doesn’t seem to make this very 
clear though. Can you suggest an improvement?

Open the file with

a = open("testfile.zip", "rb")

and all should be well :)

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26273] Expose TCP_CONGESTION and TCP_USER_TIMEOUT to the socket module

[Omar Sandoval]

New submission from Omar Sandoval:

The socket module is missing a couple of TCP socket options: TCP_CONGESTION was 
added to Linux in v2.6.13 and TCP_USER_TIMEOUT was added in v2.6.37. These 
should be exposed.

--
components: Library (Lib)
files: socket_tcp_options.patch
keywords: patch
messages: 259460
nosy: Omar Sandoval
priority: normal
severity: normal
status: open
title: Expose TCP_CONGESTION and TCP_USER_TIMEOUT to the socket module
type: enhancement
versions: Python 3.6
Added file: http://bugs.python.org/file41787/socket_tcp_options.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26256] Fast decimalisation and conversion to other bases

[Jurjen N.E. Bos]

Jurjen N.E. Bos added the comment:

OMG is decimal that fast?
Maybe I should change the issue then to "documentation missing": it nowhere 
says in the documentation that decimal has optimized multiprecision 
computations. It only says that precision "can be as large as needed for a 
given problem", but I never realized that that included millions of digits.
It computed to my complete surprise 2**2**29 to full precision (161 million 
decimal digits) in about a minute. That smells suspiciously like FFT style 
multiplication, which implies that it is way more sophisticated than integer 
multiplication!
I suggest that the documentation of the decimal module recommends using decimal 
for multiprecision computations, as long as you use the builtin version.

--
assignee:  -> docs@python
components: +Documentation -Library (Lib)
nosy: +docs@python
versions: +Python 3.4 -Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26272] `zipfile.ZipFile` fails reading a file object in specific version(s)

[Pengyu Chen]

New submission from Pengyu Chen:

When passing a file object to `zipfile.ZipFile` for reading it fails in Python 
3.5.1 it fails. While the same code in Python 2.7.11 works.
Also loading that specific file directly from file path works in Python 3.5.1.
(Please see the sample testing code below for details)

[pengyu@GLaDOS tmp]$ echo "test message" > testfile
[pengyu@GLaDOS tmp]$ zip testfile.zip testfile
updating: testfile (stored 0%)
[pengyu@GLaDOS tmp]$ file testfile.zip 
testfile.zip: Zip archive data, at least v1.0 to extract
[pengyu@GLaDOS tmp]$ python -c "a = open('testfile.zip'); import zipfile; b = 
zipfile.ZipFile(a, 'r')"
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3.5/zipfile.py", line 1026, in __init__
self._RealGetContents()
  File "/usr/lib/python3.5/zipfile.py", line 1093, in _RealGetContents
raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
[pengyu@GLaDOS tmp]$ python2 -c "a = open('testfile.zip'); import zipfile; b = 
zipfile.ZipFile(a, 'r')"
[pengyu@GLaDOS tmp]$ python -c "import zipfile; b = 
zipfile.ZipFile('testfile.zip', 'r')"
[pengyu@GLaDOS tmp]$ python --version
Python 3.5.1
[pengyu@GLaDOS tmp]$ python2 --version
Python 2.7.11

--
components: Library (Lib)
messages: 259459
nosy: Pengyu Chen
priority: normal
severity: normal
status: open
title: `zipfile.ZipFile` fails reading a file object in specific version(s)
type: crash
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26274] Add CPU affinity to perf.py

[Florin Papa]

New submission from Florin Papa:

Hi all,

This is Florin Papa from the Dynamic Scripting Languages Optimizations Team 
from Intel Corporation.

The patch submitted adds an affinity feature to the Grand Unified Python 
Benchmarks suite to allow running benchmarks on a given CPU/set of CPUs. It is 
implemented for Linux and uses the taskset command to bond a command to the 
CPUs specified. This minimizes run to run variation, as we can get considerable 
differences in measured performance from running a benchmark on different 
cores. The taskset command receives a CPU mask that specifies which cores in 
the system will be used for the command.

Example:
python perf.py –-affinity=0x1
 will use processor #0
python perf.py –-affinity=0x3
will use processors #0 and #1 

Thank you,
Florin

--
components: Benchmarks
files: affinity.patch
keywords: patch
messages: 259464
nosy: brett.cannon, florin.papa, pitrou
priority: normal
severity: normal
status: open
title: Add CPU affinity to perf.py
versions: Python 2.7, Python 3.6
Added file: http://bugs.python.org/file41788/affinity.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25924] investigate if getaddrinfo(3) on OSX is thread-safe

[A. Jesse Jiryu Davis]

A. Jesse Jiryu Davis added the comment:

Thanks Martin. The MAC_OS_X_VERSION_10_5 macro ensures that Python is still 
compatible with Mac OS before version 10.5: if it's built on 10.4 or older, it 
locks around getaddrinfo (since it's not thread-safe there), and on 10.5 and 
later it doesn't lock.

I've built on a Mac OS 10.4 virtual machine and verified we'll still use the 
lock on that OS.

I think this is good enough, without a runtime check. We distribute prebuilt 
binaries for Mac OS 10.5+:

https://www.python.org/downloads/release/python-2711/

So those prebuilt Pythons should no longer lock around getaddrinfo. 

People who still use older Mac OSes (more than 12 years old!) will have to 
build Python themselves in order to get recent versions of Python, since we 
don't distribute binaries for them any more. Since they're building Python on 
old Mac OS, the MAC_OS_X_VERSION_10_5 macro will be undefined and they'll keep 
using the getaddrinfo lock.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26274] Add CPU affinity to perf.py

[STINNER Victor]

Changes by STINNER Victor :


--
type:  -> performance

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26274] Add CPU affinity to perf.py

[Florin Papa]

Florin Papa added the comment:

Thank you for the feedback. Please see the updated patch.

--
Added file: http://bugs.python.org/file41791/affinity_v2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26274] Add CPU affinity to perf.py

[Florin Papa]

Florin Papa added the comment:

I was thinking that a warning might go unnoticed, that is why I chose to 
end the program if taskset is not found. It would force the user to
install missing packages.

Otherwise the patch looks ok.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26256] Fast decimalisation and conversion to other bases

[Stefan Krah]

Stefan Krah added the comment:

On Wed, Feb 03, 2016 at 09:51:53AM +, STINNER Victor wrote:
> Well, nowhere means:
> https://docs.python.org/dev/whatsnew/3.3.html#decimal

Okay, but hardly anyone reads that, and I can't blame them. :)

For example, if I use something like Lua, I won't read a release announcement
from 2012.

> Usually, we don't document performances of a function, maybe only its 
> complexity.

As Jurjen said, the presence of the FFT (actually fast number theoretic
transform here) in Decimal is something completely unexpected, so if there's
a place for a line or two in the docs that makes some users happy I'd be
in favor of it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26275] perf.py: bm_regex_v8 doesn't seem reliable even with --rigorous

[STINNER Victor]

New submission from STINNER Victor:

Hi, I'm working on some optimizations projects like FAT Python (PEP 509: issue 
#26058, PEP 510: issue #26098, and PEP 511: issue #26145) and faster memory 
allocators (issue #26249).

I have the *feeling* that perf.py output is not reliable even if it takes more 
than 20 minutes :-/ Maybe because Yury told that I must use -r (--rigorous) :-)

Example with 5 runs of "python3 perf.py ../default/python 
../default/python.orig -b regex_v8":

---
Report on Linux smithers 4.3.3-300.fc23.x86_64 #1 SMP Tue Jan 5 23:31:01 UTC 
2016 x86_64 x86_64
Total CPU cores: 8

### regex_v8 ###
Min: 0.043237 -> 0.050196: 1.16x slower
Avg: 0.043714 -> 0.050574: 1.16x slower
Significant (t=-19.83)
Stddev: 0.00171 -> 0.00174: 1.0178x larger

### regex_v8 ###
Min: 0.042774 -> 0.051420: 1.20x slower
Avg: 0.043843 -> 0.051874: 1.18x slower
Significant (t=-14.46)
Stddev: 0.00351 -> 0.00176: 2.0009x smaller

### regex_v8 ###
Min: 0.042673 -> 0.048870: 1.15x slower
Avg: 0.043726 -> 0.050474: 1.15x slower
Significant (t=-8.74)
Stddev: 0.00283 -> 0.00467: 1.6513x larger

### regex_v8 ###
Min: 0.044029 -> 0.049445: 1.12x slower
Avg: 0.044564 -> 0.049971: 1.12x slower
Significant (t=-13.97)
Stddev: 0.00175 -> 0.00211: 1.2073x larger

### regex_v8 ###
Min: 0.042692 -> 0.049084: 1.15x slower
Avg: 0.044295 -> 0.050725: 1.15x slower
Significant (t=-7.00)
Stddev: 0.00421 -> 0.00494: 1.1745x larger
---

I'm only care of the "Min", IMHO it's the most interesting information here.

The slowdown is betwen 12% and 20%, for me it's a big difference.


It looks like some benchmarks have very short iterations compare to others. For 
example, bm_json_v2 takes around 3 seconds, whereas bm_regex_v8 only takes less 
than 0.050 second (50 ms).

$ python3 performance/bm_json_v2.py -n 3 --timer perf_counter
3.310384973010514
3.3116717970115133
3.3077902760123834

$ python3 performance/bm_regex_v8.py -n 3 --timer perf_counter
0.0670697659952566
0.04515827298746444
0.045114840992027894

Do you think that bm_regex_v8 is reliable? I see that there is an "iteration 
scaling" to use run the benchmarks with more iterations. Maybe we can start to 
increase the "iteration scaling" for bm_regex_v8?

Instead of a fixed number of iterations, we should redesign benchmarks to use 
time. For example, one iteration must take at least 100 ms and should not take 
more than 1 second (but take longer to get more reliable results). Then the 
benchmark is responsible to ajust internal parameters.

I used this design for my "benchmark.py" script which is written to get 
"reliable" microbenchmarks:
https://bitbucket.org/haypo/misc/src/tip/python/benchmark.py?fileviewer=file-view-default

The script is based on time and calibrate a benchmark. It also uses the 
*effictive* resolution of the clock used by the benchmark to calibrate the 
benchmark.

I will maybe work on such patch, but it would be good to know first your 
opinion on such change.

I guess that we should use the base python to calibrate the benchmark and then 
pass the same parameters to the modified python.

--
components: Benchmarks
messages: 259469
nosy: brett.cannon, haypo, pitrou, yselivanov
priority: normal
severity: normal
status: open
title: perf.py: bm_regex_v8 doesn't seem reliable even with --rigorous
type: performance
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26274] Add CPU affinity to perf.py

[Florin Papa]

Florin Papa added the comment:

I have updated the patch to throw an exception if the taskset command is not 
available. Not checking that taskset works will generate an ambiguous error (if 
there is no taskset on the system) that does not explicitly point to the 
problem.

I believe that users working on performance have knowledge on the platform and 
OS setup they are using and they will draw benefit from this feature because it 
gives more accurate results.

--
Added file: http://bugs.python.org/file41792/affinity_v3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26275] perf.py: bm_regex_v8 doesn't seem reliable even with --rigorous

[STINNER Victor]

STINNER Victor added the comment:

Well, it was simpler than I expected to implement calibration. Here is a PoC 
for regex_v8 and json_dump_v2.

json_dump_v2 takes 3 seconds per run, but it already uses internally 100 loop 
iterations. I divided the number of iterations by 100 in my patch.

--
keywords: +patch
Added file: http://bugs.python.org/file41790/perf_calibration.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26272] `zipfile.ZipFile` fails reading a file object in specific version(s)

[Pengyu Chen]

Pengyu Chen added the comment:

Well I should have noticed that I wasn't using binary mode.. Sorry for the 
mistake. Closing the issue report.

--
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26256] Fast decimalisation and conversion to other bases

[STINNER Victor]

STINNER Victor added the comment:

"It's great to have this stuff, but I don't think it belongs in core Python: 
I'd much rather that the core Python integer implementation remain simple, 
portable and low-maintenance, and work well for the domain it's intended for: 
small-to-medium size integers."

Yeah I agree. Maybe we need to explain that somewhere? In the devguide? Even in 
Python doc?

I know that they are super crazy^W fast algorithm to handle very large numbers 
(hum, what is large? more than 4096 bits?), but they are usually very complex.

Projects like GMP have ultra fast code with optimizations in assemblers. Having 
assembler implementation is clearly out of the scope of Python.


"Maybe I should change the issue then to "documentation missing": it nowhere 
says in the documentation that decimal has optimized multiprecision 
computations."

Well, nowhere means:
https://docs.python.org/dev/whatsnew/3.3.html#decimal

Usually, we don't document performances of a function, maybe only its 
complexity.

--

You may move to the numpy community which is problably more keen on such 
optimization than the Python *core*.

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26275] perf.py: calibrate benchmarks using time, not using a fixed number of iterations

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +florin.papa
title: perf.py: bm_regex_v8 doesn't seem reliable even with --rigorous -> 
perf.py: calibrate benchmarks using time, not using a fixed number of iterations

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25774] [benchmarks] Adjust to allow uploading benchmark data to codespeed

[Zachary Ware]

Changes by Zachary Ware :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26280] ceval: Optimize [] operation similarly to CPython 2.7

[Yury Selivanov]

Yury Selivanov added the comment:

Zach, first I was going to collect some stats on this (as Serhiy also 
suggests).  It would be interesting to collect some stats on how many times 
BINARY_SUBSCR receives lists, tuples, dicts, and other types.  I'd instrument 
the code to collect those stats and run Python tests suite and benchmarks.

I'm pretty sure that optimizing lists (and tuples?) is a great idea.  It would 
also be nice to optimize [-1] lookup.  I'd also measure if we should add a fast 
path for dicts (PyDict_CheckExact + PyDict_GetItem).

If you have time to work on this issue, then by all means go ahead.  We'll be 
glad to assist you and review the patch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26219] implement per-opcode cache in ceval

[Francis MB]

Francis MB added the comment:

>From the two checks on Python/compile.c:

+ expr_ty meth = e->v.Call.func;
[...]
+/* Check [...] that
+   the call doesn't have keyword parameters. */
[...]
+/* Check that there are no *varargs types of arguments. */
[...]

I just wonder how many times those kind of checks/guards are done
on the whole Cpython code base (the second one seems expensive).

(Naive Idea):
Wouldn't be some kind of fast function description (e.g. bit flags
or 'e->v.Call.func.desc') generally helpful? The function description
could have: 'has_keywords' or 'has_varargs', ... 

Thanks in advance!

--
nosy: +francismb

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26280] ceval: Optimize [] operation similarly to CPython 2.7

[Yury Selivanov]

Yury Selivanov added the comment:

Zach, BTW, you can see how I instrumented ceval for stats collection in the 
patch for issue #26219.  That's only for the research purposes, we won't commit 
that... or maybe we will, but in a separate issue :).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26282] Add support for partial keyword arguments in extension functions

[Serhiy Storchaka]

New submission from Serhiy Storchaka:

Currently extension functions either accept only positional-only arguments 
(PyArg_ParseTuple), or keyword arguments (PyArg_ParseTupleAndKeywords). While 
adding support passing by keywords looks good for some arguments, for other 
arguments it doesn't make much sense. For example "encoding" and "errors" 
arguments for str or "base" argument for int are examples of good keyword 
arguments, but it is hard to choose good name for the first argument.

I suggest to allow to add the support of keyword arguments only for the part of 
arguments, while left other arguments positional-only. This issue consists from 
two stages:

1. Allow PyArg_ParseTupleAndKeywords to accept empty string "" as keywords and 
interpret this as positional-only argument.

2. Make Argument Clinic to generate code for partial keyword arguments. The 
syntax already supports this: "/" separates positional-only arguments from 
keyword arguments.

--
messages: 259521
nosy: larry, martin.panter, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Add support for partial keyword arguments in extension functions
type: enhancement
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26275] perf.py: calibrate benchmarks using time, not using a fixed number of iterations

[STINNER Victor]

STINNER Victor added the comment:

> If running time is close to the limit, different run will use different 
> number of repetitions. This will add additional instability.

Maybe, to be honest I don't know. How can we decide if a patch makes
perf.py more and less stable?

> I prefer a stable number of repetitions manually calibrated for average 
> modern computer.

The problem is to define an "average modern computer".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24905] Allow incremental I/O to blobs in sqlite3

[Aviv Palivoda]

Aviv Palivoda added the comment:

I did the needed changes for the pull request at pysqlite for porting to 
python3. I will continue updating the patch if there will be changes in 
pysqlite and vice versa.

--
keywords: +patch
Added file: http://bugs.python.org/file41798/blob.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25774] [benchmarks] Adjust to allow uploading benchmark data to codespeed

[Roundup Robot]

Roundup Robot added the comment:

New changeset 9923b81a1d34 by Zachary Ware in branch 'default':
Issue #25774: Add raw mode to perf.py.
https://hg.python.org/benchmarks/rev/9923b81a1d34

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26280] ceval: Optimize [] operation similarly to CPython 2.7

[Zach Byrne]

Zach Byrne added the comment:

Yury,
Are you going to tackle this one, or would you like me to?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26280] ceval: Optimize [] operation similarly to CPython 2.7

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Would be nice to collect statistics about arguments types during running the 
testsuite. May be list is not the most popular type of collection.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26252] Add an example to importlib docs on setting up an importer

[Simon Cross]

Changes by Simon Cross :


--
nosy: +hodgestar

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26281] Clear sys.path_importer_cache from importlib.invalidate_caches()

[Brett Cannon]

New submission from Brett Cannon:

Would it make sense to clear sys.path_importer_cache when someone calls 
importlib.invalidate_caches()? It is a cache after all.

--
components: Library (Lib)
messages: 259518
nosy: brett.cannon, eric.snow, ncoghlan
priority: normal
severity: normal
stage: test needed
status: open
title: Clear sys.path_importer_cache from importlib.invalidate_caches()
type: enhancement
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26194] Undefined behavior for deque.insert() when len(d) == maxlen

[Martin Panter]

Martin Panter added the comment:

I thought the 3.6.0 NEWS file generally listed bugs fixed since (at least) the 
3.5.0 release. But the NEWS in default has no mention of Issue 26194.

Serhiy is right that there is nothing more to do for the merging problem. I was 
just pointing out that something in your “merge” commit must have not worked 
properly. Normally a merge looks like revision 58266f5101cc (there are two 
parents listed). But in this case I pulled from default expecting to get all 
the 3.5 changes as well, but never got your latest 3.5 change until I pulled it 
explicitly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26274] Add CPU affinity to perf.py

[STINNER Victor]

STINNER Victor added the comment:

Florent: "I was thinking that a warning might go unnoticed, that is why I chose 
to end the program if taskset is not found."

Oh right, I kept your behaviour and pushed a change.

---
changeset:   238:a97acad3bbf7
tag: tip
user:Victor Stinner 
date:Wed Feb 03 15:20:02 2016 +0100
files:   perf.py
description:
Add --affinity command line option

If used, run the benchmark using taskset to pin the process to a set of CPU
cores to reduce context switching. See taskset manual page for more information.

Patch written by Florin Papa.
---

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26209] TypeError in smtpd module with string arguments

[Terry J. Reedy]

Terry J. Reedy added the comment:

Doc fixes unrelated to security fixes are not applied to security-fix-only 
versions.

--
versions:  -Python 3.2, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26276] Inconsistent behaviour of PEP 3101 formatting between versions

[Eric V. Smith]

Changes by Eric V. Smith :


--
nosy: +eric.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26274] Add CPU affinity to perf.py

[STINNER Victor]

STINNER Victor added the comment:

I don't think that it's useful to check at runtime that taskset works. At
least, I expect an error if the option is used and the command fails.

To use CPU affinity you need to know your hardware and your platform. For
most stable results you may even configure the Linux kernel to ignore some
cores. So i expect that users will not play with affinity if they don't
understand it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26277] Allow zipapp to target modules

[flying sheep]

New submission from flying sheep:

currently, zipapp’s notion of __main__.py is very different from the usual.

the root directory of a zipapp is not a module, therefore __init__.py isn’t 
used and relative imports from __main__.py don’t work.

i propose that the zipapp functionality is amended/changed to

1. allow more than one target directory (bundle multiple modules)
2. allow creation of a __main__.py that contains a runpy-powered module runner

so maybe we could specify the module itself as entry point to get this behavior

zipapp -m my_module dependency.py my_module

should create

my_module.pyz
├ __main__.py  →  ...runpy.run_module('my_module')
├ dependency.py
└ my_module
  ├ __init__.py
  └ __main__.py  →  from . import ...

or there would be another option to specify it.

--
components: Extension Modules
messages: 259476
nosy: flying sheep
priority: normal
severity: normal
status: open
title: Allow zipapp to target modules
type: enhancement
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26274] Add CPU affinity to perf.py

[STINNER Victor]

Changes by STINNER Victor :


--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26276] Inconsistent behaviour of PEP 3101 formatting between versions

[Eric V. Smith]

Eric V. Smith added the comment:

Hi, Mark.

Yes, PEP 3101 is very much under-specified in this area. I tried to avoid the 
same mistake in PEP 498, although of course that's a different problem area.

I don't recall why this particular case broke between 3.2.3 and 3.4.3. I'll try 
and track it down. I'm sure we were trying to "fix" some other edge condition.

FWIW, in 3.4.3 this also fails, as expected:
>>> "{ {{ 0} }}".format_map({' {{ 0} }': 'X'})

Can you try it in 3.2.3? I expect it to work, but you never know.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26219] implement per-opcode cache in ceval

[Yury Selivanov]

Yury Selivanov added the comment:

> From the two checks on Python/compile.c:

Stuff done in compile.c is cached in *.pyc files.

The for-loop you saw shouldn't take a lot of time - it iterates through 
function parameters, which an average function doesn't have more than 3-6.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26194] Undefined behavior for deque.insert() when len(d) == maxlen

[Raymond Hettinger]

Raymond Hettinger added the comment:

I can't say that I agree with putting an additional entry the 3.6 NEWS.   In 
the past, I've not made news entries in unreleased versions regarding bug fixes 
in prior versions.  That seems pointless to me and it would tend to clutter a 
document whose size already gets in the way of its usability.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26282] Add support for partial keyword arguments in extension functions

[Martin Panter]

Martin Panter added the comment:

For the examples you gave, they already seem to support keywords for the first 
parameter: int(x=0), str(object=""), bytes(source=b""). Maybe it is a bit 
unfortunate that they are inconsistent, but these names seem reasonable when 
considered separately.

But I agree that in those cases having the option of a keyword would rarely be 
useful. This proposal sounds like it could give a small benefit, and I guess it 
wouldn’t be costly to implement.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25660] tabs don't work correctly in python repl

[Berker Peksag]

Berker Peksag added the comment:

With rlcompleter.patch applied, I see the same behavior on my OS X (with 
libedit) and Ubuntu (with readline) systems.

Thanks!

--
nosy: +berker.peksag
stage: patch review -> commit review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13501] Make libedit support more generic; port readline / libedit to FreeBSD

[Berker Peksag]

Changes by Berker Peksag :


--
nosy: +berker.peksag

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26136] DeprecationWarning for PEP 479 (generator_stop)

[Martin Panter]

Martin Panter added the comment:

FYI I found that changeset by doing annotate on the file that you originally 
changed 
, 
which is an index of the changesets that last modified each line.

Thanks for the patch, I think it is generally good. Perhaps this also needs a 
short notice in What’s New, a bit like 
?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21042] ctypes.util.find_library() should return full pathname instead of filename in linux

[Martin Panter]

Martin Panter added the comment:

IMO this should be treated as a new feature for the next release. But 
consistently returning the path sounds good to me if there is no good reason 
not to.

Left a question on the review. I think you also need to update the 
documentation, and since this is changing documented behaviour it probably 
needs a What’s New entry.

For tests, I would try doing something like find_library("c"), and ensuring 
that the result is an absolute path. We may have to end up skipping the test 
for platforms like Windows where this is not expected to work. Look through the 
files in /Lib/ctypes/test/ for a good place for it to live (if there isn’t 
already a test there you can modify).

--
nosy: +martin.panter
stage:  -> patch review
type:  -> enhancement
versions: +Python 3.6 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26283] zipfile can not handle the path build by os.path.join()

[張伯誠]

New submission from 張伯誠:

I think the built-in library zipfile.py does not handle correctly the path 
build by os.path.join().

For example, assuming we have a zipfile named Test.zip which contanins a member 
xml/hello.xml,
if you want to extract the member out, then you hava to use 'xml/hello.xml', if 
using os.path.join('xml','hello.xml'), you will get an error. 

Platform: Windows7, Python3.4

>>> import zipfile,os
>>> f=zipfile.ZipFile("Test.zip",'r')
>>> f.extract('xml/hello.xml','.') # OK.
>>> f.extract(os.path.join('xml','hello.xml'),'.') # does not work.

If we fixed the zipfile.py, inside the method getinfo(self,name) of class 
ZipFile:

before:
def getinfo(self, name):
"""Return the instance of ZipInfo given 'name'."""  
info = self.NameToInfo.get(name)

if info is None:
raise KeyError(
'There is no item named %r in the archive' % name)

return info

after:
def getinfo(self, name):
"""Return the instance of ZipInfo given 'name'."""
if os.sep=='\\' and os.sep in name:
name=name.replace('\\','/')

info = self.NameToInfo.get(name)
if info is None:
raise KeyError(
'There is no item named %r in the archive' % name)

return info

Then this line work!
>>>  f.extract(os.path.join('xml','hello.xml'),'.') # OK!

of course, this line also work:
>>> f.extract('xml/hello.xml','.') # also OK!


I think it is a bug. Why?
Let's we take a closer look at the method:

info = self.NameToInfo.get(name)

The keys of NameToInfo are always the format "xxx/yyy" according to the 
document of class ZipInfo:

def __init__(self, filename="NoName", date_time=(1980,1,1,0,0,0)):
 
# This is used to ensure paths in generated ZIP files always use
# forward slashes as the directory separator, as required by the
# ZIP format specification.

if os.sep != "/" and os.sep in filename:

filename = filename.replace(os.sep, "/")

Hence the method getinfo(self,name) of class ZipFile always get KeyError for 
the path build os.path.join('xxx','yyy')


Thnaks for reading!
Bocheng.

--
components: Library (Lib)
messages: 259524
nosy: 張伯誠
priority: normal
severity: normal
status: open
title: zipfile can not handle the path build by os.path.join()
type: behavior
versions: Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26279] time.strptime does not properly convert out-of-bounds values

[Emanuel Barry]

Changes by Emanuel Barry :


--
nosy: +ebarry
title: Possible bug in time library -> time.strptime does not properly convert 
out-of-bounds values
versions:  -Python 3.2, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26274] Add CPU affinity to perf.py

[Florin Papa]

Florin Papa added the comment:

No problem :)

--
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26278] BaseTransport.close() does not trigger connection_lost()

[Sümer Cip]

New submission from Sümer Cip:

Hi all,

We have implemented a TCP server based on asyncio. And while doing some 
regression tests we randomly see following error:

1) Client connects to the server.
2) Client is closed ungracefully(without sending a FIN, deplug cable)
3) We have a custom PING handler that sends a PING and waits for PONG message.
4) After a while, we see that we timeout for the PING and we call close() on 
the Transport object. 

Now, most of the time, above just works fine, but at some point, somehow 
connection_lost() is NEVER gets called even though we call close() on the 
socket. As this issue is happening very randomly I don't have any asyncio logs 
for it. But can you think about any scenario that might lead to this somehow? 

Somehow, it seems we have an outgoing data in the TCP buffer when this happens 
and that is why the close() does not call the connection_lost immediately, but 
why it is never calling it is a mystery to me. Can that be following:

1) we call close() and is_closing is set to true we have outgoing data so we 
return.
2) Then a subsequent write occurs and connection ConnectionResetError() is 
raised and this calls _force_close(), but as we have previously set is_closing 
to True, connection_lost() does not get called.

Above is just a very trivial idea which is probably is not the case, I do not 
spend too much time on the code. 

Thanks,

--
components: asyncio
messages: 259487
nosy: Sümer.Cip, gvanrossum, haypo, yselivanov
priority: normal
severity: normal
status: open
title: BaseTransport.close() does not trigger connection_lost()
type: behavior
versions: Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26274] Add CPU affinity to perf.py

[STINNER Victor]

STINNER Victor added the comment:

"Florent" oops, Florin, sorry :-p

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26274] Add CPU affinity to perf.py

[STINNER Victor]

STINNER Victor added the comment:

Thanks for your contribution ;-)

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26280] ceval: Optimize [] operation similarly to CPython 2.7

[Yury Selivanov]

New submission from Yury Selivanov:

See also issue #21955

--
components: Interpreter Core
messages: 259492
nosy: haypo, yselivanov, zbyrne
priority: normal
severity: normal
stage: needs patch
status: open
title: ceval: Optimize [] operation similarly to CPython 2.7
type: performance
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25660] tabs don't work correctly in python repl

[Ned Deily]

Ned Deily added the comment:

LGTM on OS X with both the system libedit and a MacPorts GNU readline.  Let's 
apply this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25660] tabs don't work correctly in python repl

[Roundup Robot]

Roundup Robot added the comment:

New changeset 64417e7a1760 by Yury Selivanov in branch '3.5':
Issue #25660: Fix TAB key behaviour in REPL.
https://hg.python.org/cpython/rev/64417e7a1760

New changeset 87dfadd61e0d by Yury Selivanov in branch 'default':
Merge 3.5 (issue #25660)
https://hg.python.org/cpython/rev/87dfadd61e0d

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25660] tabs don't work correctly in python repl

[Berker Peksag]

Berker Peksag added the comment:

Thanks, Yury!

Look like we forgot to update test_rcompleter:

==
FAIL: test_complete (test.test_rlcompleter.TestRlcompleter)
--
Traceback (most recent call last):
  File 
"/ssd/buildbot/buildarea/3.5.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_rlcompleter.py",
 line 82, in test_complete
self.assertEqual(completer.complete('', 0), '\t')
AssertionError: '' != '\t'
+ 

http://buildbot.python.org/all/builders/ARMv7%20Ubuntu%203.5/builds/576/steps/test/logs/stdio

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26182] Deprecation warnings for the future async and await keywords

[Marco Buttu]

Marco Buttu added the comment:

Thank you Brett, your explanation (msg259409) and the c-api doc are really 
straightforward. I think I can fix it in a few days, just the time to read the 
devguide and be confident about the workflow.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26136] DeprecationWarning for PEP 479 (generator_stop)

[Anish Shah]

Anish Shah added the comment:

Thanks @martin.panter. I learnt something new, I didn't know about annotate 
before.
I have updated the patch.
Thanks again.

--
Added file: http://bugs.python.org/file41800/issue26136_20160204.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26115] pathlib.glob('**') returns only directories

[JitterMan]

JitterMan added the comment:

It may be what the documentation says it will do, but is not what it should do. 
I believe that because:

1. Currently ** in pathlib matches only directories, but **.py matches files. 
That seems inconsistent.
2. In bash, and csh, ** matches files and directories. To get the same in 
pathlib one must use **/*, which is inconsistent with what we have used for 
many decades.
3. With the traditional meaning of **, it is easy to constrain the match to 
directories by adding slash to the end of the glob (just use **/).
4. There is considerable value in supporting the traditional meaning of glob 
strings. Globbing is a very powerful feature, and it is often offered to the 
end user in shell-like situations. For example, sftp offers globbing. When 
offering globbing to the end users it is best to be consistent the globbing 
they are already familiar with.
5. There is no significant advantage to the difference between pathlib globbing 
and traditional globbing.

Globbing in pathlib is different from traditional globbing in another important 
way. pathlib does not distinguish between hidden files and directories and 
normal files and directories.  There may be isolated cases where that is 
preferred, but generally that is not true. Indeed, the primary characteristic 
of being hidden is that it should not be included in globbing. One marks a file 
or directory to be hidden specifically to mean 'do not include this one when 
selecting groups of files or directories'. Once the glob string has been 
expanded, it is possible to filter out the hidden files and directories, but it 
very difficult to do if there are several levels of directories because when 
weeding out the matches that should not be be included you have to look for 
hidden items at all levels of the path.

Globbing has been available and largely unchanged for almost 50 years. I 
encourage you to strongly consider making pathlib globbing more consistent with 
what we have all grown up with.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26115] pathlib.glob('**') returns only directories

[SilentGhost]

Changes by SilentGhost :


--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26283] zipfile can not handle the path build by os.path.join()

[Anish Shah]

Changes by Anish Shah :


--
nosy: +anish.shah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26096] '*' glob string matches dot files in pathlib

[JitterMan]

JitterMan added the comment:

Globbing has been with us for almost 50 years, and in all that time it has 
never matched the hidden files/directories. There may be isolated cases where 
matching the hidden items is preferred, but generally that is not the case. 
Indeed, the primary characteristic of being hidden is that it should not be 
included in globbing. One marks a file or directory to be hidden specifically 
to mean 'do not include this one when selecting groups of files or directories'.

Once the glob string has been expanded, it is possible to filter out the hidden 
files and directories, but it very difficult to do so if there are several 
levels of directories because one has to look for hidden items at all levels of 
the path.

Globbing has been available and largely unchanged for almost 50 years. I am not 
the one that is asking for it to be changed. I am asking for it to be returned 
to what it has always been. Being consistent with bash and other shells is a 
very important feature. It allows us to offer pathlib globbing to the end user 
and have it work the way they expect.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Yury Selivanov added the comment:

Attaching a second version of the patch.  (BTW, Serhiy, I tried your idea of 
using a switch statement to optimize branches 
(https://github.com/1st1/cpython/blob/fastint2/Python/ceval.c#L5390) -- no 
detectable speed improvement).


I decided to add fast path for floats & single-digit longs and their 
combinations.  +, -, *, /, //, and their inplace versions are optimized now.  


I'll have a full result of macro-benchmarks run tomorrow morning, but here's a 
result for spectral_norm (rigorous run, best of 3):

### spectral_norm ###
Min: 0.300269 -> 0.233037: 1.29x faster
Avg: 0.301700 -> 0.234282: 1.29x faster
Significant (t=399.89)
Stddev: 0.00147 -> 0.00083: 1.7619x smaller


Some nano-benchmarks (best of 3):

-m timeit -s "loops=tuple(range(100))" "sum([x + x + 1 for x in loops])"
2.7  7.233.5  8.17  3.6  7.57

-m timeit -s "loops=tuple(range(100))" "sum([x + x + 1.0 for x in loops])"
2.7  9.083.5  11.7  3.6  7.22

-m timeit -s "loops=tuple(range(100))" "sum([x/2.2 + 2 + x*2.5 + 1.0 for x in 
loops])"
2.7  17.93.5  24.3  3.6  11.8

--
Added file: http://bugs.python.org/file41799/fastint2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26252] Add an example to importlib docs on setting up an importer

[Anish Shah]

Changes by Anish Shah :


--
nosy: +anish.shah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26281] Clear sys.path_importer_cache from importlib.invalidate_caches()

[Anish Shah]

Changes by Anish Shah :


--
nosy: +anish.shah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 04.02.2016 07:02, Yury Selivanov wrote:
> Attaching a second version of the patch.  (BTW, Serhiy, I tried your idea of 
> using a switch statement to optimize branches 
> (https://github.com/1st1/cpython/blob/fastint2/Python/ceval.c#L5390) -- no 
> detectable speed improvement).

It would be better to consistently have the fast_*() helpers
return -1 in case of an error, instead of -1 or 1.

Overall, I see two problems with doing too many of these
fast paths:

 * the ceval loop may no longer fit in to the CPU cache on
   systems with small cache sizes, since the compiler will likely
   inline all the fast_*() functions (I guess it would be possible
   to simply eliminate all fast paths using a compile time
   flag)

 * maintenance will get more difficult

In a numerics heavy application it's like that all fast paths
will trigger somewhere, but those will likely be better off
using numpy or numba. For a text heavy application such as a web
server, only few fast paths will trigger and so the various
checks only add overhead.

Since 'a'+'b' is a very often used instruction type in the
latter type of applications, please make sure that this fast
path gets more priority in your patch.

Please also check the effects of the fast paths for cases
where they don't trigger, e.g. 'a'+'b' or 'a'*2.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26282] Add support for partial keyword arguments in extension functions

[Anish Shah]

Changes by Anish Shah :


--
nosy: +anish.shah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26263] Serialize array.array to JSON by default

[Anish Shah]

Changes by Anish Shah :


--
nosy: +anish.shah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26274] Add CPU affinity to perf.py

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +yselivanov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26276] Inconsistent behaviour of PEP 3101 formatting between versions

[Mark Shannon]

New submission from Mark Shannon:

In Python 2.7.6 and 3.2.3:
>>> "{ {{ 0} }}".format(**{' {{ 0} }': 'X'})
'X'

In Python 3.4.3:
>>> "{ {{ 0} }}".format(**{' {{ 0} }': 'X'})
Traceback (most recent call last):
  File "", line 1, in 
ValueError: unexpected '{' in field name


I think the problem is that PEP 3101 is under specified w.r.t. to non 
identifier characters in format units.

--
components: Interpreter Core
messages: 259473
nosy: Mark Shannon
priority: normal
severity: normal
status: open
title: Inconsistent behaviour of PEP 3101 formatting between versions
type: behavior
versions: Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26243] zlib.compress level as keyword argument

[Aviv Palivoda]

Aviv Palivoda added the comment:

Added the "suggested" test

--
Added file: http://bugs.python.org/file41797/zlib-compress-level-keyword3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Antoine Pitrou]

Antoine Pitrou added the comment:

Le 03/02/2016 18:21, Yury Selivanov a écrit :
> 
> Yury Selivanov added the comment:
> 
>> Yury suggested running perf.py twice with the binaries swapped
> 
> Yeah, I had some experience with perf.py when its results were
> skewed
depending on what you test first.

Have you tried disabling turbo on your CPU? (or any kind of power
management that would change the CPU clock depending on the current
workload)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26279] Possible bug in time library

[Ioannis Aslanidis]

New submission from Ioannis Aslanidis:

There seems to be a bug in the time library when performing the following 
conversion:

"""
In [8]: mytime=time.strptime(str([2015, 53, 0]), '[%Y, %U, %w]')

In [9]: mytime
Out[9]: time.struct_time(tm_year=2016, tm_mon=1, tm_mday=3, tm_hour=0, 
tm_min=0, tm_sec=0, tm_wday=6, tm_yday=368, tm_isdst=-1)

In [10]: time.strftime('%Y %U', mytime)
---
ValueError Traceback (most recent call last)
 in ()
> 1 time.strftime('%Y %U', mytime)

ValueError: day of year out of range 
"""

As you can observe, tm_yday got a value of 368 instead of 3. It seems that the 
C function is not properly subtracting 365 or 366 days when converting from 
week 53 of a particular year which, according to ISO 8601 [1] and per python 
documentation [2]. Documentation explicitly says that: "The first week of the 
year is the week with the year's first Thursday in it".

[1] https://en.wikipedia.org/wiki/ISO_8601#Week_dates
[2] https://docs.python.org/2/library/time.html

--
components: Library (Lib)
messages: 259488
nosy: iaslan
priority: normal
severity: normal
status: open
title: Possible bug in time library
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26274] Add CPU affinity to perf.py

[STINNER Victor]

STINNER Victor added the comment:

Hum, I modified your change:

* only display a warning if the option is used but taskset doesn't work
* remove check on Linux: taskset might be available on other platforms, it 
doesn't really matter to check the OS, we already check if the command works
* try to run "taskset --cpu-list 0 python -c pass" instead of "taskset -V" to 
check if taskset works or not

What do you think?

--
Added file: http://bugs.python.org/file41793/affinity_warning.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Yury Selivanov added the comment:

Attaching a new patch -- rewritten to optimize -, *, +, -=, *= and +=.  I also 
removed the optimization of [] operator -- that should be done in a separate 
patch and in a separate issue.

Some nano-benchmarks (best of 3):

python -m timeit  "sum([x + x + 1 for x in range(100)])"
2.7: 7.71 3.5: 8.54  3.6: 7.33

python -m timeit  "sum([x - x - 1 for x in range(100)])"
2.7: 7.81 3.5: 8.59  3.6: 7.57

python -m timeit  "sum([x * x * 1 for x in range(100)])"
2.7: 9.28 3.5: 10.6  3.6: 9.44


Python 3.6 vs 3.5 (spectral_norm, rigorous run):
Min: 0.315917 -> 0.276785: 1.14x faster
Avg: 0.321006 -> 0.284909: 1.13x faster


Zach, thanks a lot for the research!  I'm glad that unpack_sequence finally 
proved to be irrelevant.  Could you please take a look at the updated patch?

--
Added file: http://bugs.python.org/file41796/fastint1.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Antoine Pitrou]

Antoine Pitrou added the comment:

> In this table I've flipped the results for running the modified build > as 
> the reference, but in the new attachment, slower in the right
> column means faster, I think :)

I don't understand what this table means (why 4 columns?). Can you explain what 
you did?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[STINNER Victor]

STINNER Victor added the comment:

> python -m timeit  "sum([x * x * 1 for x in range(100)])"

If you only want to benchmark x*y, x+y and list-comprehension, you
should use a tuple for the iterator.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26273] Expose TCP_CONGESTION and TCP_USER_TIMEOUT to the socket module

[Yury Selivanov]

Changes by Yury Selivanov :


--
nosy: +yselivanov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Yury Selivanov added the comment:

> Yury suggested running perf.py twice with the binaries swapped

Yeah, I had some experience with perf.py when its results were skewed depending 
on what you test first.  Hopefully Victor's new patch will fix that 
http://bugs.python.org/issue26275

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26277] Allow zipapp to target modules

[Brett Cannon]

Changes by Brett Cannon :


--
nosy: +paul.moore

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Zach Byrne]

Zach Byrne added the comment:

> Could you please take a look at the updated patch?
Looks ok to me, for whatever that's worth.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Yury Selivanov added the comment:

> Fast patch is already implemented in long_mul(). May be we should just use 
> this function if both arguments are exact int, and apply the switch 
> optimization inside.

Agree.

BTW, what do you think about using __int128 when available?  That way we can 
also optimize twodigit PyLongs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[STINNER Victor]

STINNER Victor added the comment:

I don't think. I run benchmarks (for __int128) :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Yury Selivanov added the comment:

> I don't think. I run benchmarks (for __int128) :-)

Never mind...  Seems that __int128 is still an experimental feature and some 
versions of clang even had bugs with it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> BTW, what do you think about using __int128 when available?  That way we can 
> also optimize twodigit PyLongs.

__int128 is not always available and it will add too much of complexity for 
possible less gain. There is many ways to optimize the code and we should to 
choose those of them that have the best gain/complexity ratio.

Lets split the patch on smaller parts: 1) direct using long-specialized 
functions in ceval.c, and 2) optimize the fast path in these functions, and 
test them separately and combined. May be only one way will add a gain.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Zach Byrne]

Zach Byrne added the comment:

> I don't understand what this table means (why 4 columns?). Can you explain 
> what you did?

Yury suggested running perf.py twice with the binaries swapped
So "faster" and "slower" underneath "Baseline Reference" are runs where the 
unmodified python binary was the first argument to perf, and the "Modified 
Reference" is where the patched binary is the first argument.

ie. "perf.py -r -b all python patched_python" vs "perf.py -r -b all 
patched_python python"

bench_results.txt has the actual output in it, and the "slower in the right 
column" comment was referring to the contents of that file, not the table. 
Sorry for the confusion.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26194] Undefined behavior for deque.insert() when len(d) == maxlen

[Raymond Hettinger]

Raymond Hettinger added the comment:

[Martin]
>  the default branch never got any NEWS entry in 
> 58266f5101cc, so maybe it needs to be added?

No need for a news entry in default.  We haven't released 3.6 yet, so the 
bugfix from 3.5 is just a carry forward of the corrected behavior.  That isn't 
news ;-)

[Martin]
> revision 0731f097157b didn’t actually merge the 3.5 
> branch (It only had one parent)

I'm not sure I see what was missed here (I am not a mercurial jedi).  My 3.5 
checkout shows the change and so does the 3.6.   What needs to be done at this 
point ?

[Serhiy]
> This isn't new feature, this is just a bugfix, old behavior
> obviously was incorrect. We don't add the versionchanged 
> directive for every bugfix.

I concur with Serhiy.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Yury Selivanov]

Yury Selivanov added the comment:

Antoine, yeah, it's probably turbo boost related.  There is no easy way to turn 
it off on mac os x, though.  I hope Victor's patch to perf.py will help to 
mitigate this. 

Victor, Marc-Andre,

Updated results of nano-bench (best of 10):

-m timeit -s "loops=tuple(range(100))" "sum([x * x * 1 for x in loops])"
2.7  8.5 3.5  10.1 3.6  8.91

-m timeit -s "loops=tuple(range(100))" "sum([x + x + 1 for x in loops])"
2.7  7.273.5  8.2  3.6  7.13

-m timeit -s "loops=tuple(range(100))" "sum([x - x - 1 for x in loops])"
2.7  7.013.5  8.1  3.6  6.95

Antoine, Serhiy, I'll upload a new patch soon.  Probably Serhiy's idea of using 
a switch statement will make it slightly faster.  I'll also add a fast path for 
integer division.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Fast patch is already implemented in long_mul(). May be we should just use this 
function if both arguments are exact int, and apply the switch optimization 
inside.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26275] perf.py: calibrate benchmarks using time, not using a fixed number of iterations

[Yury Selivanov]

Yury Selivanov added the comment:

Here's a very interesting table from Zach Byrne: 
http://bugs.python.org/issue21955#msg259490

It shows that some benchmarks are indeed very unstable.  This also correlates 
with my own experience.

These ones are very unstable: pickle_dict, nbody, regex_v8, etree_parse, telco.

--
nosy: +zbyrne

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Zach Byrne]

Zach Byrne added the comment:

I ran 6 benchmarks on my work machine(not the same one as the last set) 
overnight.
Two with just the BINARY_ADD change, two with the BINARY_SUBSCR change, and two 
with both.
I'm attaching the output from all my benchmark runs, but here are the highlights
In this table I've flipped the results for running the modified build as the 
reference, but in the new attachment, slower in the right column means faster, 
I think :)
|--|---|---|
|Build | Baseline Reference| Modified Reference 
   |
|--||--||--|
|  | Faster | Slower   | Faster 
| Slower   |
|--||--||--|
|BINARY_ADD| chameleon_v2   | etree_parse  | chameleon_v2   
| call_simple  |
|  | chaos  | nbody| fannkuch   
| nbody|
|  | django | normal_startup   | normal_startup 
| pickle_dict  |
|  | etree_generate | pickle_dict  | nqueens
| regex_v8 |
|  | fannkuch   | pickle_list  | regex_compile  
|  |
|  | formatted_logging  | regex_effbot | spectral_norm  
|  |
|  | go |  | unpickle_list  
|  |
|  | json_load  |  |
|  |
|  | regex_compile  |  |
|  |
|  | simple_logging |  |
|  |
|  | spectral_norm  |  |
|  |
|--||--||--|
|BINARY_SUBSCR | chameleon_v2   | call_simple  | 2to3   
| etree_parse  |
|  | chaos  | go   | call_method_slots  
| json_dump_v2 |
|  | etree_generate | pickle_list  | chaos  
| pickle_dict  |
|  | fannkuch   | telco| fannkuch   
|  |
|  | fastpickle |  | formatted_logging  
|  |
|  | hexiom2|  | go 
|  |
|  | json_load  |  | hexiom2
|  |
|  | mako_v2|  | mako_v2
|  |
|  | meteor_contest |  | meteor_contest 
|  |
|  | nbody  |  | nbody  
|  |
|  | regex_v8   |  | normal_startup 
|  |
|  | spectral_norm  |  | nqueens
|  |
|  ||  | pickle_list
|  |
|  ||  | simple_logging 
|  |
|  ||  | spectral_norm  
|  |
|  ||  | telco  
|  |
|--||--||--|
|BOTH  | chameleon_v2   | call_simple  | chameleon_v2   
| fastpickle   |
|  | chaos  | etree_parse  | choas  
| pickle_dict  |
|  | etree_generate | pathlib  | etree_generate 
| pickle_list  |
|  | etree_process  | pickle_list  | etree_process  
| telco|
|  | fannkuch   |  | fannkuch   
|  |
|  | fastunpickle   |  | float  
|  |
|  | float  |  | formatted_logging  
|  |
|  | formatted_logging  |  | go 
|  |
|  | hexiom2|  | hexiom2
|  |
|  | nbody  |  | nbody  
|  |
|  | nqueens|  | normal_startup 
|  |
|  | regex_v8   |  | nqueens
|  |
|  | spectral_norm

Re: [issue21955] ceval.c: implement fast path for integers with a single digit

[M.-A. Lemburg]

On 03.02.2016 18:05, STINNER Victor wrote:
> 
>> python -m timeit  "sum([x * x * 1 for x in range(100)])"
> 
> If you only want to benchmark x*y, x+y and list-comprehension, you
> should use a tuple for the iterator.

... and precalculate that in the setup:

python -m timeit -s "loops=tuple(range(100))" "sum([x * x * 1 for x in loops])"

# python -m timeit "sum([x * x * 1 for x in range(100)])"
10 loops, best of 3: 5.74 usec per loop
# python -m timeit -s "loops=tuple(range(100))" "sum([x * x * 1 for x in 
loops])"
10 loops, best of 3: 5.56 usec per loop

(python = Python 2.7)

-- 
Marc-Andre Lemburg
eGenix.com

___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26136] DeprecationWarning for PEP 479 (generator_stop)

[Anish Shah]

Anish Shah added the comment:

Yes @martin.panter, the tests were failing. But, I didn't know where the tests 
for PendingDeprecationWarning were added. Thanks for telling the changeset. I 
have updated the patch. Can you please have a look?

--
Added file: http://bugs.python.org/file41794/issue26136_20160203.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26275] perf.py: calibrate benchmarks using time, not using a fixed number of iterations

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

If running time is close to the limit, different run will use different number 
of repetitions. This will add additional instability. I prefer a stable number 
of repetitions manually calibrated for average modern computer.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25774] [benchmarks] Adjust to allow uploading benchmark data to codespeed

[Zachary Ware]

Zachary Ware added the comment:

Fair point, I'll leave out the upload script.

Thanks for the review!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com