[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Steve Dower]

Steve Dower added the comment:

> File "C:\Program Files\Python35\lib\asyncio\windows_events.py", line 606, in 
> _register_with_iocp
>   _overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)

I don't think there's any case where this is going to work on an actual file - 
CreateIoCompletionPort needs a handle and not a file descriptor.

Presumably this line of code normally gets used with something else that 
defines fileno()?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Gabriel Mesquita Cangussu]

New submission from Gabriel Mesquita Cangussu:

The documentation of asyncio specifies that the methods connect_read_pipe and 
connect_write_pipe are available on Windows with the ProactorEventLoop. The 
documentation then says that those methods accept file-like objects on the pipe 
parameter. However, the methods doesn't seem to work with stdio or any disk 
file under Windows. The following example catches this problem:

import asyncio
import sys

class MyProtocol(asyncio.Protocol):
def connection_made(self, transport):
print('connection established')

def data_received(self, data):
print('received: {!r}'.format(data.decode()))

def connection_lost(self, exc):
print('lost connection')

if sys.platform.startswith('win32'):
loop = asyncio.ProactorEventLoop()
else:
loop = asyncio.SelectorEventLoop()
coro = loop.connect_read_pipe(MyProtocol, sys.stdin)
loop.run_until_complete(coro)
loop.run_forever()
loop.close()


This code when executed on Ubuntu have the desired behavior, but under Windows 
10 it gives OSError: [WinError 6] The handle is invalid. The complete output is 
this:

c:\Users\Gabriel\Documents\Python Scripts>python async_pipe.py
connection established
Fatal read error on pipe transport
protocol: <__main__.MyProtocol object at 0x01970EB2FAC8>
transport: <_ProactorReadPipeTransport fd=0>
Traceback (most recent call last):
  File "C:\Program Files\Python35\lib\asyncio\proactor_events.py", line 195, in 
_loop_reading
self._read_fut = self._loop._proactor.recv(self._sock, 4096)
  File "C:\Program Files\Python35\lib\asyncio\windows_events.py", line 425, in 
recv
self._register_with_iocp(conn)
  File "C:\Program Files\Python35\lib\asyncio\windows_events.py", line 606, in 
_register_with_iocp
_overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)
OSError: [WinError 6] Identificador inválido
lost connection


I think that the documentation should state that there is no support for disk 
files and stdio with the methods in question and also state what exactly they 
support (an example would be nice). And, of course, better support for watching 
file descriptors on Windows on future Python releases would be nice.

Thank you,
Gabriel

--
components: Windows, asyncio
messages: 264043
nosy: Gabriel Mesquita Cangussu, gvanrossum, haypo, paul.moore, steve.dower, 
tim.golden, yselivanov, zach.ware
priority: normal
severity: normal
status: open
title: ProactorEventLoop doesn't support stdin/stdout nor files with 
connect_read_pipe/connect_write_pipe
type: behavior
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23806] documentation for no_proxy is missing from the python3 urllib documentation

[Senthil Kumaran]

Senthil Kumaran added the comment:

no_proxy is applicable to ProxyHandler of the python 3. I've documented this in 
the corresponding section.

--
nosy: +orsenthil
resolution:  -> fixed
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23806] documentation for no_proxy is missing from the python3 urllib documentation

[Roundup Robot]

Roundup Robot added the comment:

New changeset 5424a559ddea by Senthil Kumaran in branch '3.5':
Issue23806 - Document the no_proxy environment variable in Python 3 docs.
https://hg.python.org/cpython/rev/5424a559ddea

New changeset 103fb8be940b by Senthil Kumaran in branch 'default':
merge 3.5
https://hg.python.org/cpython/rev/103fb8be940b

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

[Terry J. Reedy]

Terry J. Reedy added the comment:

Victor, are you suggesting the following?  "If map has a single iterable, call 
and return as hint, otherwise give up."  Or something else?

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26773] Shelve works inconsistently when carried over to child processes

[Mark Kellogg]

Mark Kellogg added the comment:

The gentoo user's version of gdbm is: gdbm-1.11

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23806] documentation for no_proxy is missing from the python3 urllib documentation

[Berker Peksag]

Changes by Berker Peksag :


--
keywords: +easy
versions: +Python 3.6 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12640] test_ctypes seg fault (test_callback_register_double); armv7; gcc 4.5.1

[Berker Peksag]

Berker Peksag added the comment:

2.7 now uses libffi 3.1 and our ARM buildbots are quite healthy so I think we 
can now close this 'out of date'.

--
nosy: +berker.peksag
resolution:  -> out of date
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26773] Shelve works inconsistently when carried over to child processes

[Mark Kellogg]

Mark Kellogg added the comment:

I have also been running into this issue. 

I am using Debian GNU/Linux 8, it was also reproduced on ubuntu by a coworker. 

It however was not reproducible on Gentoo in our testing.

--
nosy: +mkellogg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26721] Avoid socketserver.StreamRequestHandler.wfile doing partial writes

[Martin Panter]

Changes by Martin Panter :


--
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20077] Format of TypeError differs between comparison and arithmetic operators

[Berker Peksag]

Berker Peksag added the comment:

The exception message has been updated to "TypeError: '<' not supported between 
instances of 'Color' and 'Color'" in 0238eafb68da. Closing this as 'out of 
date'.

--
nosy: +berker.peksag
resolution:  -> out of date
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Roundup Robot]

Roundup Robot added the comment:

New changeset 66e40df31fac by Martin Panter in branch '3.5':
Issue #22359: Disable running cross-compiled _freeze_importlib and pgen
https://hg.python.org/cpython/rev/66e40df31fac

New changeset c36b9ef8ff39 by Martin Panter in branch 'default':
Issue #22359: Merge cross-compilation fix from 3.5
https://hg.python.org/cpython/rev/c36b9ef8ff39

New changeset 0f7a299c6d50 by Martin Panter in branch '2.7':
Issue #22359: Avoid recursive $(MAKE); disable running cross-compiled pgen
https://hg.python.org/cpython/rev/0f7a299c6d50

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7567] Messed up terminal after calling curses.initscr() twice.

[Berker Peksag]

Changes by Berker Peksag :


--
stage: test needed -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16679] Add advice about non-ASCII wsgiref PATH_INFO

[Martin Panter]

Changes by Martin Panter :


--
stage:  -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20962] Rather modest chunk size in gzip.GzipFile

[Martin Panter]

Martin Panter added the comment:

Since there doesn’t seem to be much interest here any more, and the current 
code has changed and now uses 8 KiB buffering, I am closing this. Although in 
theory a buffer or chunk size paramter could still be added to the new code if 
there was a need.

--
resolution:  -> out of date
status: open -> closed
superseder:  -> Limit decompressed data when reading from LZMAFile and BZ2File
versions:  -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24016] Add a Sprints organization/preparation section to devguide

[Brett Cannon]

Brett Cannon added the comment:

This should also include a mention of devinabox.

--
nosy: +brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26446] Mention in the devguide that core dev stuff falls under the PSF CoC

[Brett Cannon]

Brett Cannon added the comment:

The python-dev thread more-or-less petered out w/ the basic agreement that 
adding a short mention in the devguide would be tolerable. When I have a chance 
I will add a very short mention at the end of the opening page.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26446] Mention in the devguide that core dev stuff falls under the PSF CoC

[Brett Cannon]

Changes by Brett Cannon :


--
assignee:  -> brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26831] ConfigParser parsing failures with default_section and ExtendedInterpolation options

[Hans-Peter Jansen]

New submission from Hans-Peter Jansen:

ConfigParser fails in interesting ways, when using default_section and 
ExtendedInterpolation options. Running the attached script results in: 

ConfigParser() with expected result:
global: [('loglevel', 'WARNING'), ('logfile', '-')]
section1: [('key_a', 'value'), ('key_b', 'morevalue')]
section2: [('key_c', 'othervalue'), ('key_d', 'differentvalue')]

ConfigParser(default_section='global') mangles section separation:
section1: [('loglevel', 'WARNING'), ('logfile', '-'), ('key_a', 'value'), 
('key_b', 'morevalue')]
section2: [('loglevel', 'WARNING'), ('logfile', '-'), ('key_c', 'othervalue'), 
('key_d', 'differentvalue')]

ConfigParser(interpolation=ExtendedInterpolation) fails with strange error:
Traceback (most recent call last):
  File "configparser-test.py", line 36, in 
print_sections(cp)
  File "configparser-test.py", line 21, in print_sections
cp.read_string(__doc__)
  File "/usr/lib64/python3.4/configparser.py", line 696, in read_string
self.read_file(sfile, source)
  File "/usr/lib64/python3.4/configparser.py", line 691, in read_file
self._read(f, source)
  File "/usr/lib64/python3.4/configparser.py", line 1089, in _read
self._join_multiline_values()
  File "/usr/lib64/python3.4/configparser.py", line 1101, in 
_join_multiline_values
name, val)
TypeError: before_read() missing 1 required positional argument: 'value'

while it is expected to behave identical.

--
components: Library (Lib)
files: configparser-test.py
messages: 264031
nosy: frispete
priority: normal
severity: normal
status: open
title: ConfigParser parsing failures with default_section and 
ExtendedInterpolation options
versions: Python 3.4
Added file: http://bugs.python.org/file42573/configparser-test.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26830] Refactor Tools/scripts/google.py

[Ned Deily]

Ned Deily added the comment:

FWIW, I'm -0 on removing google.py and +0.5 on the refactoring.  It seems like 
it could still be useful as either an example or a tool.  Georg did a cleanup 
of the Tools directory several years ago (in Issue7962).  Rather than removing 
one script, it might be more useful to review them all for relevance.  In any 
case, thanks for the contribution, Francisco.

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23507] Tuple creation is too slow

[Yury Selivanov]

Changes by Yury Selivanov :


--
nosy: +yselivanov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16113] Add SHA-3 and SHAKE (Keccak) support

[Christian Heimes]

Christian Heimes added the comment:

The authors of Keccak have released a new version of the Keccak Code Package, 
http://keccak.noekeon.org/reorganized_code.html . The new package makes it much 
easier to integrate Keccak in Python. I'm working on a new patch with SHA3 and 
SHAKE support.

--
stage:  -> needs patch
title: Add SHA-3 (Keccak) support -> Add SHA-3 and SHAKE (Keccak) support

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4821] Patches for thread-support in built-in SHA modules

[Christian Heimes]

Changes by Christian Heimes :


--
versions: +Python 3.6 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26830] Refactor Tools/scripts/google.py

[Berker Peksag]

New submission from Berker Peksag:

I don't think google.py is useful anymore. I'd prefer to just remove it from 
the cpython repository.

--
components: +Demos and Tools
nosy: +berker.peksag
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26249] Change PyMem_Malloc to use pymalloc allocator

[Roundup Robot]

Roundup Robot added the comment:

New changeset 104ed24ebbd0 by Victor Stinner in branch 'default':
Issue #26249: Try test_capi on Windows
https://hg.python.org/cpython/rev/104ed24ebbd0

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26830] Refactor Tools/scripts/google.py

[Francisco Couzo]

Changes by Francisco Couzo :


--
files: scripts_google.patch
keywords: patch
nosy: franciscouzo
priority: normal
severity: normal
status: open
title: Refactor Tools/scripts/google.py
type: enhancement
versions: Python 3.6
Added file: http://bugs.python.org/file42572/scripts_google.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23507] Tuple creation is too slow

[STINNER Victor]

STINNER Victor added the comment:

reuse_argtuples_3.patch: rebased reuse_argtuples_2.patch.

--
Added file: http://bugs.python.org/file42571/reuse_argtuples_3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26827] PyObject *PyInit_myextention -> PyMODINIT_FUNC PyInit_myextention

[Brett Cannon]

Brett Cannon added the comment:

And in case my comment wasn't obvious, Herbert is right about the error in the 
docs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26827] PyObject *PyInit_myextention -> PyMODINIT_FUNC PyInit_myextention

[Brett Cannon]

Brett Cannon added the comment:

https://docs.python.org/3/extending/extending.html#the-module-s-method-table-and-initialization-function
 documents what Herbert discovered wrong in the porting docs.

--
keywords: +easy
nosy: +brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22234] urllib.parse.urlparse accepts any falsy value as an url

[R. David Murray]

R. David Murray added the comment:

I just posted about this on the mentors list, where someone brought this issue 
up with a question about our policy on type checking.  The short version is the 
better (preserves duck-typing) and more backward compatibile fix is to change 
the test to be

   x != ''

That will result in an attribute error on 'decode' for values of the incorrect 
type.

But even that should go through a deperecation cycle, since there may be 
working programs depending on the current behavior.  It's worth fixing, though, 
because of the error propogation you report.  I also suggested a rewrite of 
_check_args to get a better error message that would indeed be a type error, 
and I'm anticipating someone from the mentors list will turn that into a patch 
here.

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26823] Shrink recursive tracebacks

[Emanuel Barry]

Emanuel Barry added the comment:

New version with tests now, I test both the C and Python implementations.

--
assignee:  -> ebarry
Added file: http://bugs.python.org/file42570/short_tracebacks_3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] [WIP] Add a new _PyObject_FastCall() function which avoids the creation of a tuple or dict for arguments

[STINNER Victor]

STINNER Victor added the comment:

Results of the CPython benchmark suite on the revision 6c376e866330 of  
https://hg.python.org/sandbox/fastcall compared to CPython 3.6 at the revision 
496e094f4734.

It's surprising than call_simple is 1.08x slower in fastcall. This slowdown is 
not acceptable and should be fixed. It probable explains why many other 
benchmarks are slower.

Hopefully, some benchmarks are faster, between 1.02x and 1.09x faster.

IMHO there are still performance issues in my current implementation that can 
and must be fixed. At least, we have a starting point to compare performances.


$ python3 -u perf.py ../default/python ../fastcall/python -b all
(...)
Report on Linux smithers 4.4.4-301.fc23.x86_64 #1 SMP Fri Mar 4 17:42:42 UTC 
2016 x86_64 x86_64
Total CPU cores: 8

[ slower ]

### 2to3 ###
6.859604 -> 6.985351: 1.02x slower

### call_method_slots ###
Min: 0.308846 -> 0.317780: 1.03x slower
Avg: 0.308902 -> 0.318667: 1.03x slower
Significant (t=-464.83)
Stddev: 0.3 -> 0.00026: 9.8974x larger

### call_simple ###
Min: 0.232594 -> 0.251789: 1.08x slower
Avg: 0.232816 -> 0.252443: 1.08x slower
Significant (t=-911.97)
Stddev: 0.00024 -> 0.00011: 2.2373x smaller

### chaos ###
Min: 0.273084 -> 0.284790: 1.04x slower
Avg: 0.273951 -> 0.293177: 1.07x slower
Significant (t=-7.57)
Stddev: 0.00036 -> 0.01796: 49.9421x larger

### django_v3 ###
Min: 0.549604 -> 0.569982: 1.04x slower
Avg: 0.550557 -> 0.571038: 1.04x slower
Significant (t=-204.09)
Stddev: 0.00046 -> 0.00054: 1.1747x larger

### float ###
Min: 0.261939 -> 0.269224: 1.03x slower
Avg: 0.268475 -> 0.276515: 1.03x slower
Significant (t=-12.22)
Stddev: 0.00301 -> 0.00354: 1.1757x larger

### formatted_logging ###
Min: 0.325786 -> 0.334440: 1.03x slower
Avg: 0.326827 -> 0.335968: 1.03x slower
Significant (t=-34.44)
Stddev: 0.00129 -> 0.00136: 1.0503x larger

### mako_v2 ###
Min: 0.039642 -> 0.044765: 1.13x slower
Avg: 0.040251 -> 0.045562: 1.13x slower
Significant (t=-323.73)
Stddev: 0.00028 -> 0.00024: 1.1558x smaller

### meteor_contest ###
Min: 0.196589 -> 0.203667: 1.04x slower
Avg: 0.197497 -> 0.204782: 1.04x slower
Significant (t=-76.06)
Stddev: 0.00050 -> 0.00045: 1.x smaller

### nqueens ###
Min: 0.274664 -> 0.285866: 1.04x slower
Avg: 0.275285 -> 0.286774: 1.04x slower
Significant (t=-68.34)
Stddev: 0.00091 -> 0.00076: 1.2036x smaller

### pickle_list ###
Min: 0.262687 -> 0.269629: 1.03x slower
Avg: 0.263804 -> 0.270789: 1.03x slower
Significant (t=-50.14)
Stddev: 0.00070 -> 0.00070: 1.0004x larger

### raytrace ###
Min: 1.272960 -> 1.284516: 1.01x slower
Avg: 1.276398 -> 1.368574: 1.07x slower
Significant (t=-3.41)
Stddev: 0.00157 -> 0.19115: 122.0022x larger

### regex_compile ###
Min: 0.335753 -> 0.343820: 1.02x slower
Avg: 0.336273 -> 0.344894: 1.03x slower
Significant (t=-127.84)
Stddev: 0.00026 -> 0.00040: 1.5701x larger

### regex_effbot ###
Min: 0.048656 -> 0.050810: 1.04x slower
Avg: 0.048692 -> 0.051619: 1.06x slower
Significant (t=-69.92)
Stddev: 0.2 -> 0.00030: 16.7793x larger

### silent_logging ###
Min: 0.069539 -> 0.071172: 1.02x slower
Avg: 0.069679 -> 0.071230: 1.02x slower
Significant (t=-124.08)
Stddev: 0.9 -> 0.2: 3.7073x smaller

### simple_logging ###
Min: 0.278439 -> 0.287736: 1.03x slower
Avg: 0.279504 -> 0.288811: 1.03x slower
Significant (t=-52.46)
Stddev: 0.00084 -> 0.00093: 1.1074x larger

### telco ###
Min: 0.012480 -> 0.013104: 1.05x slower
Avg: 0.012561 -> 0.013157: 1.05x slower
Significant (t=-100.42)
Stddev: 0.4 -> 0.2: 1.5881x smaller

### unpack_sequence ###
Min: 0.47 -> 0.48: 1.03x slower
Avg: 0.47 -> 0.48: 1.03x slower
Significant (t=-1170.16)
Stddev: 0.0 -> 0.0: 1.0749x larger

### unpickle_list ###
Min: 0.325310 -> 0.330080: 1.01x slower
Avg: 0.326484 -> 0.333974: 1.02x slower
Significant (t=-24.19)
Stddev: 0.00100 -> 0.00195: 1.9392x larger

[ faster ]

### chameleon_v2 ###
Min: 5.525575 -> 5.263668: 1.05x faster
Avg: 5.541444 -> 5.281893: 1.05x faster
Significant (t=85.79)
Stddev: 0.01107 -> 0.01831: 1.6539x larger

### etree_iterparse ###
Min: 0.212073 -> 0.197146: 1.08x faster
Avg: 0.215504 -> 0.200254: 1.08x faster
Significant (t=61.07)
Stddev: 0.00119 -> 0.00130: 1.0893x larger

### etree_parse ###
Min: 0.282983 -> 0.260390: 1.09x faster
Avg: 0.284333 -> 0.262758: 1.08x faster
Significant (t=77.34)
Stddev: 0.00102 -> 0.00169: 1.6628x larger

### etree_process ###
Min: 0.218953 -> 0.213683: 1.02x faster
Avg: 0.221036 -> 0.215280: 1.03x faster
Significant (t=25.98)
Stddev: 0.00114 -> 0.00108: 1.0580x smaller

### hexiom2 ###
Min: 122.001408 -> 118.967112: 1.03x faster
Avg: 122.108010 -> 119.110115: 1.03x faster
Significant (t=16.81)
Stddev: 0.15076 -> 0.20224: 1.3415x larger

### pathlib ###
Min: 0.088533 -> 0.084888: 1.04x faster
Avg: 0.088916 -> 0.085280: 1.04x faster
Significant (t=257.68)
Stddev: 0.00014 -> 0.00017: 1.1725x larger


The following not significant results are hidden, use -v to show them:
call_method, 

[issue26249] Change PyMem_Malloc to use pymalloc allocator

[Roundup Robot]

Roundup Robot added the comment:

New changeset 68b2a43d8653 by Victor Stinner in branch 'default':
PyMem_Malloc() now uses the fast pymalloc allocator
https://hg.python.org/cpython/rev/68b2a43d8653

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[Stefan Krah]

Stefan Krah added the comment:

#14757 has an implementation of inline caching, which at least seemed to slow 
down some use cases. Then again, whenever someone posts a new speedup 
suggestion, it seems to slow down things I'm working on. At least Case van 
Horsen independently verified the phenomenon in this issue. :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21955] ceval.c: implement fast path for integers with a single digit

[STINNER Victor]

STINNER Victor added the comment:

Maybe we should adopt a difference approach.

There is something called "inline caching": put the cache between instructions, 
in the same memory block. Example of paper on CPython:

"Efficient Inline Caching without Dynamic Translation" by Stefan Brunthaler 
(2009)
https://www.sba-research.org/wp-content/uploads/publications/sac10.pdf

Maybe we can build something on top of the issue #26219 "implement per-opcode 
cache in ceval"?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[Xiang Zhang]

Xiang Zhang added the comment:

With a careful rereading of PEP3123, I think you are right. Actually the title 
of the PEP tells what you mean. I do misunderstand. I am quite sorry and feel 
shamed of making noice here and wasting time of the participants. I will think 
more seriously from now on. :-(

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26829] update docs: when creating classes a new dict is created for the final class object

[Ethan Furman]

New submission from Ethan Furman:

https://docs.python.org/3/reference/datamodel.html#creating-the-class-object

This section should mention that the final class is created with a new dict(), 
and all key/value pairs from the dict used during creation are copied over.

--
assignee: docs@python
components: Documentation
messages: 264016
nosy: docs@python, ethan.furman
priority: normal
severity: normal
status: open
title: update docs: when creating classes a new dict is created for the final 
class object
versions: Python 3.4, Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26809] `string` exposes ChainMap from `collections`

[Martin Panter]

Martin Panter added the comment:

I’m not that fussed if the _ChainMap name is backported. I just thought it is 
safer to not do it; similar reasoning to why I only committed Issue 23883 
patches to 3.6. I certainly think backporting __all__ is reasonable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26601] Use new madvise()'s MADV_FREE on the private heap

[Antoine Pitrou]

Antoine Pitrou added the comment:

All this discussion is in the context of the GNU libc allocator, but please 
remember that Python works on many platforms, including OS X, Windows, the 
*BSDs...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6792] Distutils-based installer does not detect 64bit versions of Python

[sorin]

Changes by sorin :


--
nosy:  -sorin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

[Martin Panter]

Martin Panter added the comment:

Tangentially related: Issue 25156, about using sendfile() to copy files in 
shutil.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23507] Tuple creation is too slow

[STINNER Victor]

STINNER Victor added the comment:

msg264009: Serhiy Storchaka "Could you compare filter(), map() and sorted() 
performance with your patch and with issue23507 patch?"

Here you have the result of bench_builtins.py. The performance look to be the 
same, even if I expect better performance with less trivial callbacks (where 
fastcall would avoid the creation of other temporary tuples).

+-+--
Tests   |   argtuples |  fastcall
+-+--
filter(lambda x: x, range(1000))| 80.2 us (*) | 72.7 us (-9%)
map(lambda x: x, range(1000))   | 79.5 us (*) |   79.9 us
map(lambda x, y: x+y, range(1000), range(1000)) |  111 us (*) |109 us
sorted(list, key=lambda x: x)   | 82.6 us (*) | 75.7 us (-8%)
sorted(list)| 15.7 us (*) |   15.6 us
+-+--
Total   |  369 us (*) |353 us
+-+--

Comparison to the original Python 3.6:

+-++---
Tests   |original |  argtuples 
|   fastcall
+-++---
filter(lambda x: x, range(1000))|  109 us (*) | 80.2 us (-27%) 
| 72.7 us (-33%)
map(lambda x: x, range(1000))   | 96.9 us (*) | 79.5 us (-18%) 
| 79.9 us (-18%)
map(lambda x, y: x+y, range(1000), range(1000)) |  126 us (*) |  111 us (-12%) 
|  109 us (-13%)
sorted(list, key=lambda x: x)   |  114 us (*) | 82.6 us (-28%) 
| 75.7 us (-34%)
sorted(list)| 16.1 us (*) |15.7 us 
|15.6 us
+-++---
Total   |  463 us (*) |  369 us (-20%) 
|  353 us (-24%)
+-++---


The difference is more on the changes in bltinmodule.c. Example with 
filter_next():

Using fastcall:

-good = PyObject_CallFunctionObjArgs(lz->func, item, NULL);
+good = PyObject_CallArg1(lz->func, item);

reuse_argtuples_2.patch:

-good = PyObject_CallFunctionObjArgs(lz->func,
-item, NULL);
+PyObject *argtuple = lz->argtuple;
+lz->argtuple = NULL;
+if (argtuple == NULL) {
+argtuple = PyTuple_New(1);
+if (argtuple == NULL) {
+Py_DECREF(item);
+return NULL;
+}
+Py_INCREF(argtuple);
+}
+assert(Py_REFCNT(argtuple) == 2);
+assert(Py_SIZE(argtuple) == 1);
+assert(PyTuple_GET_ITEM(argtuple, 0) == NULL);
+PyTuple_SET_ITEM(argtuple, 0, item);
+good = PyObject_Call(lz->func, argtuple, NULL);
+if (Py_REFCNT(argtuple) == 2) {
+PyTuple_SET_ITEM(argtuple, 0, NULL);
+if (lz->argtuple == NULL)
+lz->argtuple = argtuple;
+else {
+Py_DECREF(argtuple);
+Py_DECREF(argtuple);
+}
+}
+else {
+Py_INCREF(item);
+Py_DECREF(argtuple);
+Py_DECREF(argtuple);
+}

(reuse_argtuples_2.patch requires a few other changes related to filter_next().)

--
Added file: http://bugs.python.org/file42569/bench_builtins.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26804] Prioritize lowercase proxy variables in urllib.request

[Martin Panter]

Martin Panter added the comment:

I found two bugs; see the comments.

In Python 2, it looks like the proxy_bypass_etc() functions are defined in 
urllib and imported into urllib2, so it makes sense to include the tests in 
test_urllib rather than test_urllib2.

Technically I think proxy_bypass_environment() is meant to be an internal 
function, but it is safer to keep the changes to in minimal for bug fixes. So I 
think the optional proxies parameter should be okay.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6792] Distutils-based installer does not detect 64bit versions of Python

[Brian Curtin]

Changes by Brian Curtin :


--
nosy:  -brian.curtin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

[STINNER Victor]

STINNER Victor added the comment:

> But note that map() and zip() take several iterables, and we should call 
> __length_hint__() for every of them (unless found a one with not implemented 
> __length_hint__()). This can slow down the execution for short sequences.

Oh, there is no slot for __length_hint__(). Maybe we should also try to add a 
new slot for it?

Maybe we can use a fast-path for the most common cases like list(map(func, 
list))?


> It is impossible to implement reasonable filter.__length_hint__(), because 
> the length of resulting sequence can be from 0 to the length of the input 
> sequence, and returning the maximal value would be not correct.

What I see is that Python 2 is much faster and Python 2 reallocates N items if 
the input sequence contains N items. But yeah, we have to benchmark such change 
on Python 3 ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] [WIP] Add a new _PyObject_FastCall() function which avoids the creation of a tuple or dict for arguments

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Could you compare filter(), map() and sorted() performance with your patch and 
with issue23507 patch?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

See also issue14126.

It makes sense to implement map.__length_hint__() and  zip.__length_hint__(). 
But note that map() and zip() take several iterables, and we should call 
__length_hint__() for every of them (unless found a one with not implemented 
__length_hint__()). This can slow down the execution for short sequences.

It is impossible to implement reasonable filter.__length_hint__(), because the 
length of resulting sequence can be from 0 to the length of the input sequence, 
and returning the maximal value would be not correct.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

[STINNER Victor]

New submission from STINNER Victor:

When I compared the performance of filter() and map() between Python 2.7 and 
3.4, I noticed a huge performance drop in Python 3!
http://bugs.python.org/issue26814#msg264003

I didn't analyze yet exactly why Python 3 is so much slower (almost 100% slower 
for the case of fitler!). Maybe it's because filter() returns a list on Python 
2, whereas filter() returns an iterator on Python 3.

In Python 2, filter() and map() use _PyObject_LengthHint(seq, 8) to create the 
result list. Why don't we do the same in Python 3?

filter.__length_hint__() and map.__length_hint__() would return 
seq.__length_hint__() of the input sequence, or return 8. It's a weak 
estimation, but it can help a lot of reduce the number of realloc() when the 
list is slowly growing.

See also the PEP 424 -- A method for exposing a length hint.

Note: the issue #26814 (fastcall) does make filter() and map() faster on Python 
3.6 compared to Python 2.7, but it's not directly related to this issue. IMHO 
using length hint would make list(filter) and list(map) even faster.

--
messages: 264007
nosy: alex, haypo, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Implement __length_hint__() on map() and filter() to optimize list(map) 
and list(filter)
type: performance
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26827] PyObject *PyInit_myextention -> PyMODINIT_FUNC PyInit_myextention

[Herbert]

Herbert added the comment:

The exact error was:

ImportError: dynamic module does not define init function (PyInit_myextension)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26827] PyObject *PyInit_myextention -> PyMODINIT_FUNC PyInit_myextention

[Herbert]

New submission from Herbert:

I think 

PyObject *PyInit_myextention(void)

should be

PyMODINIT_FUNC PyInit_myextention(void)

on https://docs.python.org/3/howto/cporting.html#module-initialization-and-state

It didn't work for me until I replaced this with a message in the about 
'undefined PyInit_myextention'. However, when I used nm to inspect the .so 
object file, I fond the PyInit_myextention (but probably with the wrong return 
type). Moreover, whenever I would remove the same .so importing resulted in a 
different error complaining that the module does not exist (strongly suggesting 
that I did not mix up .so files).

Good luck!

--
assignee: docs@python
components: Documentation
messages: 264005
nosy: docs@python, prinsherbert
priority: normal
severity: normal
status: open
title: PyObject *PyInit_myextention -> PyMODINIT_FUNC PyInit_myextention
versions: Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

[STINNER Victor]

STINNER Victor added the comment:

> We usually wait until syscalls are generally available in common distros and 
> have bindings in glibc. It makes it easier to test the feature.

Usually, yeah. os.urandom() uses syscall() to use the new getrandom() of Linux 
since it's still not exposed in the GNU libc...
https://sourceware.org/bugzilla/show_bug.cgi?id=17252
Status: NEW

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] [WIP] Add a new _PyObject_FastCall() function which avoids the creation of a tuple or dict for arguments

[STINNER Victor]

STINNER Victor added the comment:

For more fun, comparison between Python 2.7 / 3.4 / 3.6 / 3.6 FASTCALL.

--+-+++---
Tests |py27 |   py34 |   
py36 |   fast
--+-+++---
filter|  165 us (*) |  318 us (+93%) |  237 us 
(+43%) | 165 us
map   |  209 us (*) |  258 us (+24%) | 202 
us |  171 us (-18%)
sorted(list, key=lambda x: x) |  272 us (*) |  348 us (+28%) |  237 us 
(-13%) |  163 us (-40%)
sorted(list)  | 33.7 us (*) | 47.8 us (+42%) | 27.3 us 
(-19%) | 27.7 us (-18%)
b=MyBytes(); bytes(b) | 3.31 us (*) |  835 ns (-75%) |  510 ns 
(-85%) |  561 ns (-83%)
namedtuple.attr   | 4.63 us (*) |4.51 us | 1.98 us 
(-57%) | 1.57 us (-66%)
object.__setattr__(obj, "x", 1)   |  463 ns (*) | 440 ns |  343 ns 
(-26%) |  222 ns (-52%)
object.__getattribute__(obj, "x") |  323 ns (*) |  396 ns (+23%) | 316 
ns |  196 ns (-39%)
getattr(1, "real")|  218 ns (*) |   237 ns (+8%) |  264 ns 
(+21%) |  147 ns (-33%)
bounded_pymethod(1, 2)|  213 ns (*) |  244 ns (+14%) |   194 ns 
(-9%) |  188 ns (-12%)
unbound_pymethod(obj, 1, 2)   |  345 ns (*) |  247 ns (-29%) |  196 ns 
(-43%) |  191 ns (-45%)
func()|  161 ns (*) |  211 ns (+31%) | 161 
ns | 157 ns
func(1, 2, 3) |  219 ns (*) |  247 ns (+13%) |  196 ns 
(-10%) |  190 ns (-13%)
--+-+++---
Total |  689 us (*) |  980 us (+42%) | 707 
us |  531 us (-23%)
--+-+++---


I didn't know that Python 3.4 was so much slower than Python 2.7 on function 
calls!?

Note: Python 2.7 and Python 3.4 are system binaries (Fedora 22), wheras Python 
3.6 and Python 3.6 FASTCALL are compiled manually.

Ignore "b=MyBytes(); bytes(b)", this benchmark is written for Python 3.

--

details:

Common platform:
Bits: int=32, long=64, long long=64, size_t=64, void*=64
Platform: Linux-4.4.4-301.fc23.x86_64-x86_64-with-fedora-23-Twenty_Three
CPU model: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz

Platform of campaign py27:
CFLAGS: -fno-strict-aliasing -O2 -g -pipe -Wall -Werror=format-security 
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong 
--param=ssp-buffer-size=4 -grecord-gcc-switches 
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic 
-D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security 
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong 
--param=ssp-buffer-size=4 -grecord-gcc-switches 
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic 
-D_GNU_SOURCE -fPIC -fwrapv
Python unicode implementation: UCS-4
Timer precision: 954 ns
Python version: 2.7.10 (default, Sep 8 2015, 17:20:17) [GCC 5.1.1 20150618 (Red 
Hat 5.1.1-4)]
Timer: time.time

Platform of campaign py34:
Timer info: namespace(adjustable=False, 
implementation='clock_gettime(CLOCK_MONOTONIC)', monotonic=True, 
resolution=1e-09)
CFLAGS: -Wno-unused-result -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g 
-pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions 
-fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches 
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic 
-D_GNU_SOURCE -fPIC -fwrapv
Timer precision: 84 ns
Python unicode implementation: PEP 393
Python version: 3.4.3 (default, Jun 29 2015, 12:16:01) [GCC 5.1.1 20150618 (Red 
Hat 5.1.1-4)]
Timer: time.perf_counter

Platform of campaign py36:
Timer info: namespace(adjustable=False, 
implementation='clock_gettime(CLOCK_MONOTONIC)', monotonic=True, 
resolution=1e-09)
Python version: 3.6.0a0 (default:496e094f4734, Apr 22 2016, 02:18:13) [GCC 
5.3.1 20151207 (Red Hat 5.3.1-2)]
CFLAGS: -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall 
-Wstrict-prototypes
Python unicode implementation: PEP 393
Timer: time.perf_counter

Platform of campaign fast:
Timer info: namespace(adjustable=False, 
implementation='clock_gettime(CLOCK_MONOTONIC)', monotonic=True, 
resolution=1e-09)
CFLAGS: -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall 
-Wstrict-prototypes
Python unicode implementation: PEP 393
Python version: 3.6.0a0 (default:ad4a53ed1fbf, Apr 22 2016, 12:42:15) [GCC 
5.3.1 20151207 (Red Hat 5.3.1-2)]

--
Added file: http://bugs.python.org/file42568/bench_fast-2.py

___
Python tracker 

___

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

[Christian Heimes]

Christian Heimes added the comment:

Thanks, looks interesting.

We usually wait until syscalls are generally available in common distros and 
have bindings in glibc. It makes it easier to test the feature.

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26601] Use new madvise()'s MADV_FREE on the private heap

[Julian Taylor]

Julian Taylor added the comment:

which is exactly what malloc is already doing for, thus my point is by using 
malloc we would fullfill your request.

But do you have an actual real work application where this would help?
it is pretty easy to figure out, just run the application under perf and see if 
there is a relevant amount of time spent in page_fault/clear_pages.

And as mentioned you can already change the allocator for arenas at runtime, so 
you could also try changing it to malloc and see if your application gets any 
faster.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +haypo, neologix, pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26826] Expose new copy_file_range() syscal in os module and use it to improve shutils.copy()

[Marcos Dione]

New submission from Marcos Dione:

copy_file_range() has been introduced in the Linux kernel since version 4.5 
(mid march 2016). This new syscall allows to copy data from one fd to another 
without passing by user space, improving speed in most cases. You can read more 
about it here:

https://lwn.net/Articles/659523/

I intend to start working on adding a binding for it in the os module and then, 
if it's available, use it in shutils.copy() to improve its efficiency. I have a 
couple of questions: If the syscall is not available, should I implement a user 
space alternative or should the method not exist at all?

--
components: Library (Lib)
messages: 264000
nosy: StyXman
priority: normal
severity: normal
status: open
title: Expose new copy_file_range() syscal in os module and use it to improve 
shutils.copy()
type: enhancement
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] [WIP] Add a new _PyObject_FastCall() function which avoids the creation of a tuple or dict for arguments

[STINNER Victor]

STINNER Victor added the comment:

Some microbenchmarks: bench_fast.py.

== Python 3.6 / Python 3.6 FASTCALL ==

--+--+---
Tests | /tmp/default |  /tmp/fastcall
--+--+---
filter|   241 us (*) |  166 us (-31%)
map   |   205 us (*) |  168 us (-18%)
sorted(list, key=lambda x: x) |   242 us (*) |  162 us (-33%)
sorted(list)  |  27.7 us (*) |27.8 us
b=MyBytes(); bytes(b) |   549 ns (*) | 533 ns
namedtuple.attr   |  2.03 us (*) | 1.56 us (-23%)
object.__setattr__(obj, "x", 1)   |   347 ns (*) |  218 ns (-37%)
object.__getattribute__(obj, "x") |   331 ns (*) |  200 ns (-40%)
getattr(1, "real")|   267 ns (*) |  150 ns (-44%)
bounded_pymethod(1, 2)|   193 ns (*) | 190 ns
unbound_pymethod(obj, 1, 2|   195 ns (*) | 192 ns
--+--+---
Total |   719 us (*) |  526 us (-27%)
--+--+---


== Compare Python 3.4 / Python 3.6 / Python 3.6 FASTCALL ==

Common platform:
Timer: time.perf_counter
Python unicode implementation: PEP 393
Timer info: namespace(adjustable=False, 
implementation='clock_gettime(CLOCK_MONOTONIC)', monotonic=True, 
resolution=1e-09)
CPU model: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
Platform: Linux-4.4.4-301.fc23.x86_64-x86_64-with-fedora-23-Twenty_Three
SCM: hg revision=abort: repository . not found! tag=abort: repository . not 
found! branch=abort: repository . not found! date=abort: no repository found in 
'/home/haypo/prog/python' (.hg not found)!
Bits: int=32, long=64, long long=64, size_t=64, void*=64

Platform of campaign /tmp/py34:
Python version: 3.4.3 (default, Jun 29 2015, 12:16:01) [GCC 5.1.1 20150618 (Red 
Hat 5.1.1-4)]
CFLAGS: -Wno-unused-result -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g 
-pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions 
-fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches 
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic 
-D_GNU_SOURCE -fPIC -fwrapv
Timer precision: 78 ns
Date: 2016-04-22 13:37:52

Platform of campaign /tmp/default:
Python version: 3.6.0a0 (default:496e094f4734, Apr 22 2016, 02:18:13) [GCC 
5.3.1 20151207 (Red Hat 5.3.1-2)]
CFLAGS: -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall 
-Wstrict-prototypes
Timer precision: 103 ns
Date: 2016-04-22 13:38:07

Platform of campaign /tmp/fastcall:
Python version: 3.6.0a0 (default:ad4a53ed1fbf, Apr 22 2016, 12:42:15) [GCC 
5.3.1 20151207 (Red Hat 5.3.1-2)]
Timer precision: 99 ns
CFLAGS: -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall 
-Wstrict-prototypes
Date: 2016-04-22 13:38:21

--+-++---
Tests |   /tmp/py34 |   /tmp/default |  
/tmp/fastcall
--+-++---
filter|  325 us (*) |  241 us (-26%) |  166 us 
(-49%)
map   |  260 us (*) |  205 us (-21%) |  168 us 
(-35%)
sorted(list, key=lambda x: x) |  354 us (*) |  242 us (-32%) |  162 us 
(-54%)
sorted(list)  | 46.9 us (*) | 27.7 us (-41%) | 27.8 us 
(-41%)
b=MyBytes(); bytes(b) |  839 ns (*) |  549 ns (-35%) |  533 ns 
(-36%)
namedtuple.attr   | 4.51 us (*) | 2.03 us (-55%) | 1.56 us 
(-65%)
object.__setattr__(obj, "x", 1)   |  447 ns (*) |  347 ns (-22%) |  218 ns 
(-51%)
object.__getattribute__(obj, "x") |  401 ns (*) |  331 ns (-17%) |  200 ns 
(-50%)
getattr(1, "real")|  236 ns (*) |  267 ns (+13%) |  150 ns 
(-36%)
bounded_pymethod(1, 2)|  249 ns (*) |  193 ns (-22%) |  190 ns 
(-24%)
unbound_pymethod(obj, 1, 2|  251 ns (*) |  195 ns (-22%) |  192 ns 
(-23%)
--+-++---
Total |  993 us (*) |  719 us (-28%) |  526 us 
(-47%)
--+-++---

--
Added file: http://bugs.python.org/file42567/bench_fast.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26601] Use new madvise()'s MADV_FREE on the private heap

[Antti Haapala]

Antti Haapala added the comment:

mmap is not the problem, the eagerness of munmap is a source of possible 
problem. 

The munmap eagerness does not show problems in all programs because the arena 
allocation heuristics do not work as intended. A proper solution in Linux and 
other operating systems where it is supported, is to put the freed arenas in a 
list, then mark freed with MADV_FREE. Now if the memory pressure grows, only 
*then* will the OS reclaim these. At any time the application can start reusing 
these arenas/pages; if they're not reclaimed, the old contents will be still 
present there; if operating system reclaimed them, they'd be remapped with 
zeroes.

Really the only downside of all this that I can foresee is that 
`ps/top/whatever` output would see Python using way more memory in its 
RSS/virt/whatever than it is actually using.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26601] Use new madvise()'s MADV_FREE on the private heap

[Julian Taylor]

Julian Taylor added the comment:

I know one can change the allocator, but the default is mmap which I don't 
think is a very good choice for the current arena size.
All the arguments about fragmentation and memory space also apply to pythons 
arena allocator itself and I am not convinced that fragmentation of the libc 
allocator is a real problem for python as pythons allocation pattern is very 
well behaved _due_ to its own arena allocator. I don't doubt it but I think it 
would be very valuable to document the actual real world use case that 
triggered this change, just to avoid people stumbling over this again and again.

But then I also don't think that anything needs to be necessarily be changed 
either, I have not seen the mmaps being a problem in any profiles of 
applications I work with.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] [WIP] Add a new _PyObject_FastCall() function which avoids the creation of a tuple or dict for arguments

[STINNER Victor]

STINNER Victor added the comment:

Related issue: issue #23507, "Tuple creation is too slow".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] [WIP] Add a new _PyObject_FastCall() function which avoids the creation of a tuple or dict for arguments

[STINNER Victor]

STINNER Victor added the comment:

Changes of my current implementation, ad4a53ed1fbf.diff.

The good thing is that all changes are internals (really?). Even if you don't 
modify your C extensions (nor your Python code), you should benefit of the new 
fast call is *a lot* of cases.

IMHO the best tricky part are changes on the PyTypeObject. Is it ok to add a 
new tp_fastcall slot? Should we add even more slots using the fast call 
convention like tp_fastnew and tp_fastinit? How should we handle the 
inheritance of types with that?


(*) Add 2 new public functions:

PyObject* PyObject_CallNoArg(PyObject *func);
PyObject* PyObject_CallArg1(PyObject *func, PyObject *arg);


(*) Add 1 new private function:

PyObject* _PyObject_FastCall(PyObject *func, PyObject **stack, int na, int nk);

_PyObject_FastCall() is the root of the new feature.


(*) type: add a new "tp_fastcall" field to the PyTypeObject structure.

It's unclear to me how inheritance is handled here. Maybe it's simply broken, 
but it's strange because it looks like it works :-) Maybe it's very rare that 
tp_call is overidden in a child class?

TODO: maybe reuse the "tp_call" field? (risk of major backward 
incompatibility...)


(*) slots: add a new "fastwrapper" field to the wrappercase structure. Add a 
fast wrapper to all slots (really all? i should check).

I don't think that consumers of the C API are of this change, or maybe only a 
few projects.

TODO: maybe remove "fastwrapper" and reuse the "wrapper" field? (low risk of 
backward compatibility?)


(*) Implement fast call for Python function (_PyFunction_FastCall) and C 
functions (PyCFunction_FastCall)


(*) Add a new METH_FASTCALL calling convention for C functions. Right now, it 
is used for 4 builtin functions: sorted(), getattr(), iter(), next().

Argument Clinic should be modified to emit C code using this new fast calling 
convention.


(*) Implement fast call in the following functions (types):

- method()
- method_descriptor()
- wrapper_descriptor()
- method_wrapper()
- operator.itemgetter => used by collections.namedtuple to get an item by its 
name


(*) Modify PyObject_Call*() functins to reuse internally the fast call. 
"tp_fastcall" is preferred over "tp_call" (FIXME: is it really useful to do 
that?).

The following functions are able to avoid temporary tuple/dict without having 
to modify the code calling them:

- PyObject_CallFunction()
- PyObject_CallMethod(), _PyObject_CallMethodId()
- PyObject_CallFunctionObjArgs(), PyObject_CallMethodObjArgs()

It's not required to modify code using these functions to use the 3 new shiny 
functions (PyObject_CallNoArg, PyObject_CallArg1, _PyObject_FastCall). For 
example, replacing PyObject_CallFunctionObjArgs(func, NULL) with 
PyObject_CallNoArg(func) is just a micro-optimization, the tuple is already 
avoided. But PyObject_CallNoArg() should use less memory of the C stack and be 
a "little bit" faster.


(*) Add new helpers: new Include/pystack.h file, Py_VaBuildStack(), etc.


Please ignore unrelated changes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] [WIP] Add a new _PyObject_FastCall() function which avoids the creation of a tuple or dict for arguments

[STINNER Victor]

Changes by STINNER Victor :


Added file: http://bugs.python.org/file42566/ad4a53ed1fbf.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26807] mock_open()().readline() fails at EOF

[Robert Collins]

Robert Collins added the comment:

Thanks Yolanda, that looks sane - could you perhaps add a test for this in 
Lib/unittest/tests/test_mock/ ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[Xiang Zhang]

Xiang Zhang added the comment:

Noo!! What I am afraid most seems to happen, a lot of noice due 
to misunderstanding! If that is true I am quite sorry. :-( But before I admit I 
have to figure it out myself.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Seems you misunderstood PEP3123. The bugs described in the PEP are gone and 
can't be reappear. The new PyObject_HEAD fixes the problem, it makes casting to 
PyObject* be defined behaviour. The accessor macros are needed only to write 
the code compatible with 3.x and 2.x. Without Py_TYPE() you would need to use 
`foo->ob_base->ob_type` or `foo->ob_type` in 3.x depending on the type of foo, 
and always use `foo->ob_type` in 2.x.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23507] Tuple creation is too slow

[STINNER Victor]

STINNER Victor added the comment:

Hi, I started to work on a new top-down approach avoiding completly temporary 
tuple/dict: "FASTCALL", issue #26814. As you may expect, the patch is much 
larger that Serhiy's patches attached to his issue, but IMHO the final code is 
much simpler: it doesn't have to use complex tricks on the reference count or 
setting tuple items to NULL (which lead to segfaults like the issue #26811).


Antoine Pitrou: "Maybe we want a facility to create on-stack static-size 
tuples?"

My current implementation of FASTCALL uses a buffer allocated on the stack to 
handle calls with up to 10 parameters (10 positional parameters or 5 keyword 
parameters, since a keyword uses 2 PyObject*).


Antoine Pitrou: "How many functions can benefit from this approach, though?"

In my experimental FASTCALL branch, slot wrappers, PyObject_Call*() functions 
(except of PyObject_Call()!), ceval.c, etc. benefit from FASTCALL. The question 
is not really how much benefit from FASTCALL, but more how much changes are 
worth to avoid temporary tuple/dict. For example, right now I decided to not 
add a FASTCALL flavor of tp_new and tp_init (instanciate a class or create a 
new type) because it becomes much more complex when you have to handle 
inheritance.

Maybe my FASTCALL requires too many changes and is overkill. Maybe we need to 
find a compromise between FASTCALL (issue #26814) and Serhiy's changes which 
are limited to a few functions.

Today, it's too early to decide, but I have fun with my FASTCALL experiment ;-) 
Slot wrappers are 40% faster, getting a namedtuple attribute is 25% faster 
(whereas Serhiy already optimized this specific case!), etc.

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[Xiang Zhang]

Xiang Zhang added the comment:

The current code is OK. But if the macro are used with arguments not of 
PyObject* in the future, the bugs described in PEP3123 can possibly be 
introduced.

The new PyObject_HEAD alone can not fix the problem. It just makes the 
exception happen:

Standard C has one specific exception to its aliasing rules precisely designed 
to support the case of Python: a value of a struct type may also be accessed 
through a pointer to the first field. E.g. if a struct starts with an int , the 
struct * may also be cast to an int * , allowing to write int values into the 
first field. 

You still have to make sure accessing ob_type is via a PyObject* pointer. So 
the PEP also says:

However, all accesses to ob_refcnt and ob_type MUST cast the object pointer to 
PyObject* (unless the pointer is already known to have that type), and SHOULD 
use the respective accessor macros.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6792] Distutils-based installer does not detect 64bit versions of Python

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
Removed message: http://bugs.python.org/msg263973

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19846] Python 3 raises Unicode errors with the C locale

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
Removed message: http://bugs.python.org/msg263975

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14621] Hash function is not randomized properly

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
Removed message: http://bugs.python.org/msg263978

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2466] os.path.ismount doesn't work for mounts the user doesn't have permission to see

[Robin Roth]

Robin Roth added the comment:

Any progress on merging this?
The fix is simple, there is a test; anything else I can do to help?

Ansible integrated the patch posted here as a workaround of an issue this 
caused. So there was some external review of the fix. See
https://github.com/ansible/ansible-modules-core/pull/2737

--
versions: +Python 3.2, Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7980] time.strptime not thread safe

[Martin Panter]

Changes by Martin Panter :


--
Removed message: http://bugs.python.org/msg263977

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20962] Rather modest chunk size in gzip.GzipFile

[Martin Panter]

Changes by Martin Panter :


--
Removed message: http://bugs.python.org/msg263972

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7980] time.strptime not thread safe

[Eryk Sun]

Eryk Sun added the comment:

SilentGhost, msg263977 is spam copied from msg206393 to try to fool people into 
clicking on the link. I've marked all of these messages from editor-buzzfeed as 
spam.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

How this can help reduce the possible errors? I afraid that it rather can 
increase the possibility of error.

There is no undefined behaviour in current code. It was in old code (in the 
casting a pointer to PyObject*), but this was fixed with new PyObject_HEAD in 
3.0+. 2.7 still has this undefined behaviour.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26825] Variable defined in exec(code) unreachable inside function call with visible name in dir() results

[Eryk Sun]

Eryk Sun added the comment:

In Python 2, using the exec statement makes the compiler disable fast locals: 

>>> def foo(): pass
... 
>>> def bar(): exec ''
... 
>>> foo.__code__.co_flags & inspect.CO_OPTIMIZED
1
>>> bar.__code__.co_flags & inspect.CO_OPTIMIZED
0

This has never been the case in Python 3, in which exec() is a function instead 
of a statement. The exec function can be shadowed by a global named "exec", so 
the hack to disable fast locals was removed.

--
nosy: +eryksun
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26825] Variable defined in exec(code) unreachable inside function call with visible name in dir() results

[Larry Hastings]

Changes by Larry Hastings :


--
nosy:  -larry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6792] Distutils-based installer does not detect 64bit versions of Python

[SilentGhost]

Changes by SilentGhost :


--
versions:  -Python 3.1, Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19846] Python 3 raises Unicode errors with the C locale

[SilentGhost]

Changes by SilentGhost :


--
nosy: +Sworddragon, a.badger, bkabrda, haypo, jwilk, larry, lemburg, loewis, 
ncoghlan, pitrou, r.david.murray, serhiy.storchaka, terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14621] Hash function is not randomized properly

[SilentGhost]

Changes by SilentGhost :


--
nosy: +Arfrever, Bob.Ziuchkovski, Giovanni.Bajo, PaulMcMillan, ReneSac, 
Vlado.Boza, alex, arigo, benjamin.peterson, bkabrda, camara, christian.heimes, 
cvrebert, dmalcolm, dstufft, gregory.p.smith, haypo, iElectric, isoschiz, 
konk, lemburg, mark.dickinson, ncoghlan, pconnell, sbermeister, 
serhiy.storchaka, Łukasz.Rekucki

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7980] time.strptime not thread safe

[SilentGhost]

SilentGhost added the comment:

Lisa, please be careful with nosy list.

--
nosy: +Romuald, SilentGhost, William.McBrine, amaury.forgeotdarc, belopolsky, 
brett.cannon, ccorbacho, ced, cptnwillard, davidfraser, dpalms2011, epu, 
eric.smith, flox, fredwheeler, kzsolt, mark.dickinson, matrixise, pitrou, 
r.david.murray, ruseel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26825] Variable defined in exec(code) unreachable inside function call with visible name in dir() results

[Xiang Zhang]

Xiang Zhang added the comment:

This seems to a behaviour change. In 2.7, return ans compiles to LOAD_NAME so 
it can find ans. But in 3.x, it compiles to LOAD_GLOBAL which searches ans from 
global scope. I don't know when this change is introduced.

--
components: +Interpreter Core -Argument Clinic
nosy: +xiang.zhang -324857679
type: crash -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26601] Use new madvise()'s MADV_FREE on the private heap

[STINNER Victor]

STINNER Victor added the comment:

I'm not sure that I understood correctly, but if you are proposing to use 
malloc()/free() instead of mmap()/munmap() to allocate arenas in pymalloc, you 
have to know that we already different allocators depending on the platform:
https://docs.python.org/dev/c-api/memory.html#the-pymalloc-allocator

By the way, it is possible to modify the arena allocator at runtime:
https://docs.python.org/dev/c-api/memory.html#customize-pymalloc-arena-allocator

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[STINNER Victor]

STINNER Victor added the comment:

It is possible to implement type checking in C macros.

Example found on the Internet:

/* inline type checking - can mix in with other macros more easily using the 
comma operator,
 * C11 gives best results here */
#if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L)
#  define CHECK_TYPE_INLINE(val, type) \
(void)((void)(((type)0) != (0 ? (val) : ((type)0))), \
   _Generic((val), type: 0, const type: 0))
#else
#  define CHECK_TYPE_INLINE(val, type) \
((void)(((type)0) != (0 ? (val) : ((type)0
#endif

http://stackoverflow.com/questions/11449632/checking-types-of-macro-parameters-at-compile-time

See also Include/pymacro.h which contains some macros implementing checks 
tested at the compilation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[Xiang Zhang]

Xiang Zhang added the comment:

My intention of the patch is that it helps reduce the possible errors described 
in PEP3123. I agree with Serhiy and Martin's opinion but I think the error 
passing a weird type argument to the macro is easier to find and debug than the 
undefined behaviour.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26601] Use new madvise()'s MADV_FREE on the private heap

[Charles-François Natali]

Charles-François Natali added the comment:

The heap on Linux is still a linear contiguous *address space*. I
agree that MADV_DONTNEED allow's returning committed memory back to
the VM subsystem, but it is still using a large virtual memory area.
Not everyone runs on 64-bit, or can waste address space.
Also, not every Unix is Linux.

But it might make sense to use malloc on Linux, maybe only on 64-bit.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26601] Use new madvise()'s MADV_FREE on the private heap

[Julian Taylor]

Julian Taylor added the comment:

glibcs malloc is not obstack, its not a simple linear heap where one object on 
top means everything below is not freeable. It also uses MADV_DONTNEED give 
sbrk'd memory back to the system. This is the place where MADV_FREE can now be 
used now as the latter does not guarantee a page fault.
But that said of course you can construct workloads which lead to increased 
memory usage also with malloc and maybe python triggers them more often than 
other applications. Is there an existing issues showing the problem? It would 
be a good form of documentation in the source.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14621] Hash function is not randomized properly

[lissacoffey]

lissacoffey added the comment:

How about Python grows a additional btree implementation in its collections 
module? I know that it's not going to fix existing code. However in the long 
run it's the best safeguard against hash collision attacks. I'm thinking about 
a simple, self balancing btree like red-black-tree. A quick search on Wikipedia 
also revealed Scapegoat and Splay tree with interesting properties.

Thanks
Lisa - python expert
mysite: http://www.fixithere.net/asos

--
nosy: +editor-buzzfeed -Arfrever, Bob.Ziuchkovski, Giovanni.Bajo, PaulMcMillan, 
ReneSac, Vlado.Boza, alex, arigo, benjamin.peterson, bkabrda, camara, 
christian.heimes, cvrebert, dmalcolm, dstufft, gregory.p.smith, haypo, 
iElectric, isoschiz, konk, lemburg, mark.dickinson, ncoghlan, pconnell, 
sbermeister, serhiy.storchaka, Łukasz.Rekucki

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7980] time.strptime not thread safe

[lissacoffey]

lissacoffey added the comment:

On the other hand, the way imports are handled in 3.3 and later is a bit 
different, and if I run the unit test (which does still fail for me on 2.7 tip) 
on 3.3 and 3.4 tip it passes.

Thanks
Lisa - python expert
mysite: http://www.followthesteps.net/sky-contact-phone-number/

--
nosy: +editor-buzzfeed -Romuald, William.McBrine, amaury.forgeotdarc, 
belopolsky, brett.cannon, ccorbacho, ced, cptnwillard, davidfraser, dpalms2011, 
epu, eric.smith, flox, fredwheeler, kzsolt, mark.dickinson, matrixise, pitrou, 
r.david.murray, ruseel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[Martin Panter]

Martin Panter added the comment:

Most of Xiang’s changes are to Py_Check() macros. I would expect them to 
be called with a generic PyObject pointer, and they do seem to be documented as 
accepting a PyObject pointer. Py_TYPE() is a macro that uses an unconditional 
cast. In general, these kinds of macros can hide errors that the compiler may 
otherwise pick up. E.g. if you accidentally pass an integer, or pointer to a 
pointer, etc, to Py_TYPE(), I think you will only get a warning or run-time 
crash, rather than a compile-time error. So while I am not that experienced 
with the C API, I suspect the change could have negative consequences.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19846] Python 3 raises Unicode errors with the C locale

[lissacoffey]

lissacoffey added the comment:

Using an environment variable is not the holy grail for this. On writing a 
non-single-user application you can't expect the user to set extra environment 
variables.

If compatibility is the only reason in my opinion it would be much better to 
include something like sys.use_strict_encoding() which decides if 
print()/write() will use sys.getfilesystemencoding() or 
sys.getdefaultencoding().
thanks
Lisa
mysite: http://www.fixithere.net/sky-contact-number/

--
nosy: +editor-buzzfeed -Sworddragon, a.badger, bkabrda, haypo, jwilk, larry, 
lemburg, loewis, ncoghlan, pitrou, r.david.murray, serhiy.storchaka, terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26822] itemgetter/attrgetter/methodcaller objects ignore keyword arguments

[Martin Panter]

Martin Panter added the comment:

This seems like a good change, and the patch looks correct.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6792] Distutils-based installer does not detect 64bit versions of Python

[lissacoffey]

lissacoffey added the comment:

This bug is a really annoying one, any chance it will be fixed in 2.7? It's 
really a matter when you want to deploy a program using distutils (my case), 
because you cannot really require your clients to edit the registry themselves 
:/
Thanks
http://www.fixithere.net/sky-contact-number/

--
nosy: +editor-buzzfeed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20962] Rather modest chunk size in gzip.GzipFile

[lissacoffey]

lissacoffey added the comment:

I measured both the total time of the run, the time to process each input 
records, and time to execute just the seek() call for each record. The bulk of 
the per-record time was in the call to seek(), so by reducing that time, I sped 
up my run-times significantly.
Thanks
http://qwikfix.co.uk/barclays/

--
nosy: +editor-buzzfeed
status: pending -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[STINNER Victor]

STINNER Victor added the comment:

py_refcnt.diff also LGTM, go ahead.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[STINNER Victor]

STINNER Victor added the comment:

change_some_macros_using_Py_TYPE.patch LGTM.

> Direct access to ob_type is used about 300 times in the source code. Changing 
> all this case to use the Py_TYPE() macro causes code churn.

IHMO change_some_macros_using_Py_TYPE.patch makes the code more readable since 
it's harder and easier to read.

Note: Such refactoring patch must only be applied to the development branch 
(default).

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26804] Prioritize lowercase proxy variables in urllib.request

[Hans-Peter Jansen]

Hans-Peter Jansen added the comment:

v5: don't require the proxies argument in proxy_bypass_environment()

--
Added file: 
http://bugs.python.org/file42565/python-urllib-prefer-lowercase-proxies-v5.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26601] Use new madvise()'s MADV_FREE on the private heap

[Charles-François Natali]

Charles-François Natali added the comment:

> Julian Taylor added the comment:
>
> it defaulted to 128kb ten years ago, its a dynamic threshold since ages.

Indeed, and that's what encouraged switching the allocator to use mmap.
The problem with dynamic mmap threshold is that since the Python
allocator uses fixed-size arenas, basically malloc always ends up
allocating from the heap (brk).
Which means that given that we don't use a - compacting - garbage
collector, after a while the heap would end up quite fragmented, or
never shrink: for example let's say you allocate 1GB - on the heap -
and then you free them, but  a single object is allocated at the top
of the heap, you heap never shrinks back.
This has bitten people (and myself a couple times at work).

Now, I see several options:
- revert to using malloc, but this will re-introduce the original problem
- build some form of hysteresis in the arena allocation
- somewhat orthogonally, I'd be interested to see if we couldn't
increase the arena size

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26825] Variable defined in exec(code) unreachable inside function call with visible name in dir() results

[ganix]

New submission from ganix:

here is a code show what happend:
>>> def func():
exec('ans=1')
print(dir())
return ans

>>> func()
['ans']
Traceback (most recent call last):
  File "", line 1, in 
func()
  File "", line 4, in func
return ans
NameError: name 'ans' is not defined

i tried this code in version 2.7,it is ok

--
components: Argument Clinic
messages: 263967
nosy: 324857679, larry
priority: normal
severity: normal
status: open
title: Variable defined in exec(code) unreachable inside function call with 
visible name in dir() results
type: crash
versions: Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[Xiang Zhang]

Xiang Zhang added the comment:

Good point. I don't think about that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com