[issue29519] weakref spewing exceptions during finalization when combined with multiprocessing

2017-02-09 Thread Łukasz Langa

Łukasz Langa added the comment:

I have a pretty minimal repro. You'll have to download both `mod1.py` and 
`mod2.py` and execute `python3.6 mod1.py`. You'll see:

Exception ignored in: .remove at 
0x7fcb56b09400>
Traceback (most recent call last):
  File "/usr/local/fbcode/gcc-4.9-glibc-2.20-fb/lib/python3.6/weakref.py", line 
117, in remove
TypeError: 'NoneType' object is not callable

I can reproduce this both on CentOS 7 and macOS Sierra. I don't quite 
understand what makes this particular combination of instructions in the 
modules trigger the problem, I'd appreciate some explanation. AFAICT any 
additional simplification to the two modules makes it stop producing the spew.

Note: the spew doesn't happen with 3.6.0, only with master because it's 
directly caused by the missing global after the change in #28427.

--
Added file: http://bugs.python.org/file46617/mod1.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29519] weakref spewing exceptions during finalization when combined with multiprocessing

2017-02-09 Thread Łukasz Langa

Changes by Łukasz Langa :


Added file: http://bugs.python.org/file46618/mod2.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29520] Documentation uses deprecated "defindex.html" Sphinx template

2017-02-09 Thread Jim DeLaHunt

New submission from Jim DeLaHunt:

When I build the documentation on the current CPython code, there is a 
deprecation warning on the console. 

= (beginning of output)
% make html
sphinx-build -b html -d build/doctrees -D latex_elements.papersize=  . 
build/html 
Running Sphinx v1.5.2
...[omitted irrelevant output]...
generating indices... genindex py-modindex
writing additional pages... download index
WARNING: Now base template defindex.html is deprecated.
 search opensearch
copying images... [100%] faq/python-video-icon.png  

...[omitted irrelevant output]...
build succeeded, 1 warning.

Build finished. The HTML pages are in build/html.
= (end of output)

This is observed when building documentation from branch master, commit 
b1dc6b6d5fa20f63f9651df2e7986a066c88ff7d . 
The build command is "cd Doc; make html".

There are other warnings in the output, and I'm dealing with them in a 
different issue (number to follow). They are easier to fix than this one.

Diagnosis:
Sphinx config file Doc/conf.py:72 invokes the building of template 
'indexcontent.html'. 
Doc/tools/templates/indexcontent.html:1 contains Sphinx directive 
`{% extends "defindex.html" %}`.
This invokes file sphinx/sphinx/themes/basic/defindex.html 
[See 
https://github.com/sphinx-doc/sphinx/blob/8ecd7ff08249739bbc6d900527fe9306592456ab/sphinx/themes/basic/defindex.html
 ]. Sure enough, it issues a deprecation warning. 

{{ warn('Now base template defindex.html is deprecated.') }}

There's a story behind this file.

Sphinx issue 2986 (https://github.com/sphinx-doc/sphinx/issues/2986) says that 
this is a very old file, from about the 0.2 version of Sphinx. It wasn't HTML 5 
compatible, so they declared it obsolete and threw it out. Well, that lasted 
only about two weeks.  It became apparent that not only Python's docs, but 
thousands of other projects, seem to rely on it. So, defindex.html was 
restored, but with the deprecation warning.

Then, on 1. January 2017, Sphinx deleted defindex.html again. (See 
https://github.com/sphinx-doc/sphinx/commit/45d3f2e8b279efa9d42068d4109cd97eb3f2d899
 ). I can only imagine that, once this change makes it into the public release 
of Sphinx, Python's documentation, and that of thousands of projects, will 
break again.

So, it seems like a good idea to proactively remove the dependence on this 
Sphinx file, before that new Sphinx release comes out. 

Options:

1. Copy the Sphinx defindex.html file into our source tree, and keep using it. 
Plus points: it's simple and easy. Minus points: the Sphinx licence terms may 
not permit this.  And, it is not HTML5 compatible, which we might care about.

2. Identify the template which Sphinx intends as a successor to defindex.html, 
and switch to using that.  I've done a bit of searching, and couldn't find out 
which template that might be.

3. Reimplement our Doc/tools/templates/indexcontent.html to rely on supported 
Sphinx template, and replace whatever intermediate content we were using from 
defindex.html with freshly-written code. 

I don't have a solution in mind for this issue. I just want to get it in the 
bug list, so we know about it.

--
assignee: docs@python
components: Documentation
messages: 287478
nosy: JDLH, docs@python
priority: normal
severity: normal
status: open
title: Documentation uses deprecated "defindex.html" Sphinx template
type: compile error
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29515] socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows

2017-02-09 Thread Eryk Sun

Eryk Sun added the comment:

Unless someone has a better (more automated) way to handle the Winsock IPPROTO 
enum, I suggest we either special-case the individual tests for MS_WINDOWS when 
we know that Winsock defines the value; or just define macros for the values in 
the enum:

#ifdef MS_WINDOWS /* Macros based on the IPPROTO enum. */
#define IPPROTO_ICMP IPPROTO_ICMP
#define IPPROTO_IGMP IPPROTO_IGMP
#define IPPROTO_GGP IPPROTO_GGP
#define IPPROTO_TCP IPPROTO_TCP
#define IPPROTO_PUP IPPROTO_PUP
#define IPPROTO_UDP IPPROTO_UDP
#define IPPROTO_IDP IPPROTO_IDP
#define IPPROTO_ND IPPROTO_ND 
#define IPPROTO_RAW IPPROTO_RAW
#define IPPROTO_MAX IPPROTO_MAX
#if (_WIN32_WINNT >= 0x0501)
#define IPPROTO_HOPOPTS IPPROTO_HOPOPTS
#define IPPROTO_IPV4 IPPROTO_IPV4
#define IPPROTO_IPV6 IPPROTO_IPV6
#define IPPROTO_ROUTING IPPROTO_ROUTING
#define IPPROTO_FRAGMENT IPPROTO_FRAGMENT
#define IPPROTO_ESP IPPROTO_ESP
#define IPPROTO_AH IPPROTO_AH
#define IPPROTO_ICMPV6 IPPROTO_ICMPV6
#define IPPROTO_NONE IPPROTO_NONE
#define IPPROTO_DSTOPTS IPPROTO_DSTOPTS
#define IPPROTO_ICLFXBM IPPROTO_ICLFXBM
#endif /* (_WIN32_WINNT >= 0x0501) */
#if (_WIN32_WINNT >= 0x0600)
#define IPPROTO_ST IPPROTO_ST
#define IPPROTO_CBT IPPROTO_CBT
#define IPPROTO_EGP IPPROTO_EGP
#define IPPROTO_IGP IPPROTO_IGP
#define IPPROTO_RDP IPPROTO_RDP
#define IPPROTO_PIM IPPROTO_PIM
#define IPPROTO_PGM IPPROTO_PGM
#define IPPROTO_L2TP IPPROTO_L2TP
#define IPPROTO_SCTP IPPROTO_SCTP
#endif /* (_WIN32_WINNT >= 0x0600) */
#endif /* MS_WINDOWS */

or call PyModule_AddIntConstant(m, "IPPROTO_ICMP", IPPROTO_ICMP) and so on for 
each enum value.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread INADA Naoki

INADA Naoki added the comment:

> I'm not sure about the change on PyObject_CallMethod*() only for empty format 
> string.

There are many place using _PyObject_CallMethodId() to call method without args.
Maybe, we should recommend to use _PyObject_CallMethodIdObjArgs() when no 
arguments, and replace all caller in cpython?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29519] weakref spewing exceptions during finalization when combined with multiprocessing

2017-02-09 Thread Łukasz Langa

New submission from Łukasz Langa:

Antoine, #28427 introduces a regression. When used with multiprocessing, the 
WeakValueDictionary in `multiprocessing.util._afterfork_registry` causes the 
`remove()` to be invoked during `atexit` and then `sys.meta_path` is None, lots 
of things is None, including the global `_remove_dead_weakref`.

In effect, I'm getting spew like this:

Exception ignored in: .remove at 
0x7fb2b905e2f0>
Traceback (most recent call last):
  File "/usr/local/fbcode/gcc-4.9-glibc-2.20-fb/lib/python3.6/weakref.py", line 
112, in remove
TypeError: 'NoneType' object is not callable
Exception ignored in: .remove at 
0x7fb2b905e2f0>
Traceback (most recent call last):
  File "/usr/local/fbcode/gcc-4.9-glibc-2.20-fb/lib/python3.6/weakref.py", line 
112, in remove
TypeError: 'NoneType' object is not callable
Exception ignored in: .remove at 
0x7fb2b905e2f0>
Traceback (most recent call last):
  File "/usr/local/fbcode/gcc-4.9-glibc-2.20-fb/lib/python3.6/weakref.py", line 
112, in remove
TypeError: 'NoneType' object is not callable
Exception ignored in: .remove at 
0x7fb2b905e2f0>
Traceback (most recent call last):
  File "/usr/local/fbcode/gcc-4.9-glibc-2.20-fb/lib/python3.6/weakref.py", line 
112, in remove
TypeError: 'NoneType' object is not callable

When debugged, this 'NoneType' is `_remove_dead_weakref` (weakref.py:117).

I'm working on a smaller repro, unfortunately this happens as part of a rather 
large multiprocessing app so this might take a while. Just wanted to let you 
know, maybe you'll know right away what the problem is.

--
assignee: pitrou
components: Library (Lib)
keywords: 3.5regression, 3.6regression
messages: 287475
nosy: lukasz.langa, pitrou
priority: normal
severity: normal
status: open
title: weakref spewing exceptions during finalization when combined with 
multiprocessing
type: behavior
versions: Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29518] 'for' loop not automatically breaking (index error on line of loop header)

2017-02-09 Thread Josh Rosenberg

Josh Rosenberg added the comment:

You're going to need to actually provide the source file here. This makes no 
sense as is.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29518] 'for' loop not automatically breaking (index error on line of loop header)

2017-02-09 Thread Justin McNiel

New submission from Justin McNiel:

All for loops are refusing to break (Python 2.7.13) on win32
  - only on execution of '.py' file
an example would be:"
for x in range(5): #Line 1 of main.py
print x#Line 2 of main.py
"
and the resulting error would be:"
File "pathToFile\main.py", line 1, in 
for x in range(5):
IndexError: array index out of range
"
I also have 32-bit python 3.6 installed, but the default for opening '.py' 
files is python 2.7
This error happens on custom made generators too, but never in the interactive 
shell.

--
components: Windows
files: pythonError.png
messages: 287473
nosy: Justin McNiel, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: 'for' loop not automatically breaking (index error on line of loop 
header)
type: behavior
versions: Python 2.7
Added file: http://bugs.python.org/file46616/pythonError.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29515] socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows

2017-02-09 Thread Eryk Sun

Eryk Sun added the comment:

The macro is defined but not defined. If I insert the following line before the 
#ifdef check:

#define IPPROTO_IPV6 IPPROTO_IPV6

then the constant gets added:

>>> import _socket
>>> _socket.IPPROTO_IPV6
41

The same applies to IPPROTO_IPV4 and probably others.

--
nosy: +eryksun
versions: +Python 3.5, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29517] "Can't pickle local object" when uses functools.partial with method and args...

2017-02-09 Thread R. David Murray

Changes by R. David Murray :


--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread Roundup Robot

Roundup Robot added the comment:


New changeset b1dc6b6d5fa20f63f9651df2e7986a066c88ff7d by Victor Stinner in 
branch 'master':
Issue #29507: Fix _PyObject_CallFunctionVa()
https://github.com/python/cpython/commit/b1dc6b6d5fa20f63f9651df2e7986a066c88ff7d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

callmethod2.patch: I like that change on object_vacall(), I'm not sure about 
the change on PyObject_CallMethod*() only for empty format string.

I suggest to split your patch into two parts, and first focus on 
object_vacall(). Do you have a benchmark for this one?

Note: I doesn't like the name I chose for object_vacall(). If we modify it, I 
would suggest to rename it objet_call_vargs() instead.

Anyway, before pushing anything more, I would like to take a decision on the 
repr()/test_exceptions issue. What do you think Naoki?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread Roundup Robot

Roundup Robot added the comment:

New changeset e5cd74868dfc by Victor Stinner in branch 'default':
Issue #29507: Fix _PyObject_CallFunctionVa()
https://hg.python.org/cpython/rev/e5cd74868dfc

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread INADA Naoki

INADA Naoki added the comment:

Thanks for finishing my draft patch, Victor.

callmetohd2.patch is same trick for PyObject_CallMethod* APIs in abstract.c.
It fixes segv in callmethod.patch.
And APIs receiving format string can do same trick when format is empty too.

As I grepping "PyObject_CallMethod", there are many format=NULL callers.

--
Added file: http://bugs.python.org/file46615/callmethod2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

I checked typeobject.c: there is a single case where we use the result of 
lookup_maybe_method()/lookup_method() for something else than calling the 
unbound method: slot_tp_finalize() calls PyErr_WriteUnraisable(del), the case 
discussed in my previous message which caused test_exceptions failure (now 
fixed).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue6926] socket module missing IPPROTO_IPV6, IPPROTO_IPV4

2017-02-09 Thread Maximilian Hils

Maximilian Hils added the comment:

Thanks for the insanely quick feedback. I still think this is a bug, so I filed 
https://bugs.python.org/issue29515 with additional details. Let's continue 
there.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

Oh, I was too lazy to run the full test suite, I only ran a subset and I was 
bitten by buildbots :-) test_unraisable() of test_exceptions fails. IHMO the 
BrokenRepr subtest on this test function is really implementation specific.

To fix buildbots, I removed the BrokenRepr unit test, but kept the other cases 
on test_unraisable(): change be663c9a9e24. See my commit message for the full 
rationale.

In fact, the patch changed the error message logged when a destructor fails. 
Example:
---
class Obj:
def __del__(self):
raise Exception("broken del")

def __repr__(self):
return ""

obj = Obj()
del obj
---

Before, contains "":
---
Exception ignored in: >
Traceback (most recent call last):
  File "x.py", line 3, in __del__
raise Exception("broken del")
Exception: broken del
---

After, "" is gone:
---
Exception ignored in: 
Traceback (most recent call last):
  File "x.py", line 3, in __del__
raise Exception("broken del")
Exception: broken del
---


There is an advantage. The error message is now better when repr(obj) fails. 
Example:
---
class Obj:
def __del__(self):
raise Exception("broken del")

def __repr__(self):
raise Excepiton("broken repr")

obj = Obj()
del obj
---

Before, raw "" with no information on the type:
---
Exception ignored in: 
Traceback (most recent call last):
  File "x.py", line 3, in __del__
raise Exception("broken del")
Exception: broken del
---

After, the error message includes the type:
---
Exception ignored in: 
Traceback (most recent call last):
  File "x.py", line 3, in __del__
raise Exception("broken del")
Exception: broken del
---


Technically, slot_tp_finalize() can call lookup_maybe() to get a bound method 
if the unbound method failed. The question is if it's worth it? In general, I 
dislike calling too much code to log an exception, since it's likely to raise a 
new exception. It's exactly the case here: logging an exception raises a new 
exception (in repr())!

Simpler option: revert the change in slot_tp_finalize() and document that's 
it's deliberate to get a bound method to get a better error message.


The question is a tradeoff between performance and correctness.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread Roundup Robot

Roundup Robot added the comment:

New changeset be663c9a9e24 by Victor Stinner in branch 'default':
Issue #29507: Update test_exceptions
https://hg.python.org/cpython/rev/be663c9a9e24

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread Yury Selivanov

Yury Selivanov added the comment:

patch looks good to me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

Naoki: "method_fastcall4.patch looks clean enough, and performance benefit 
seems nice."

Ok, I pushed the patch with minor changes:

* replace "variants:" with "Variants:"
* rename lookup_maybe_unbound() to lookup_maybe_method()
* rename lookup_method_unbound() to lookup_method()


"I don't know current test suite covers unusual special methods."

What do you mean by "unusual special methods"?


"Maybe, we can extend test_class to cover !unbound (e.g. @classmethod) case."

It's hard to test all cases, since they are a lot of function types in Python, 
and each slot (wrapper in typeobject.c) has its own C implementation.

But yeah, in general more tests don't harm :-)

Since the patch here optimizes the most common case, a regular method 
implemented in Python, I didn't add a specific test with the change. This case 
is already very well tested, like everything in the stdlib, no?

--

I tried to imagine how we could avoid temporary method objects in more cases 
like Python class methods (using @classmethod), but I don't think that it's 
worth it.

It would require more complex code for a less common case. Or do someone see 
other common cases which would benefit of a similar optimization?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

Raymond Hettinger: "+1 Though this is a rather large and impactful patch, I 
think it is a great idea.  It will be one of the highest payoff applications of 
FASTCALL, broadly benefitting a lot of code."

In my experience, avoiding temporary tuple to pass positional arguments 
provides a speedup to up 30% faster in the best case. Here it's 1.5x faster 
because the optimization also avoids the creation of temporary PyMethodObject.


"Let's be sure to be extra careful with this one because it is touching central 
parts of the language, so any errors or subtle behavior changes will be felt by 
a lot of code, some of which is sure to hit the rare corner cases and to rely 
on implementation details."

I reviewed Naoki's patch carefully, but in fact it isn't as big as I expected.

In Python 3.6, call_method() calls tp_descr_get of PyFunction_Type which 
creates PyMethodObject. The tp_call of PyMethodObject calls the function with 
self, nothing crazy.

The patch removes a lot of steps and (IMHO) makes the code simpler than before 
(when calling Python methods).

I'm not saying that such change is bugfree-proof :-) But we are far from Python 
3.7 final, so it's the right time to push such large optimization.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread Roundup Robot

Roundup Robot added the comment:

New changeset 7b8df4a5d81d by Victor Stinner in branch 'default':
Optimize slots: avoid temporary PyMethodObject
https://hg.python.org/cpython/rev/7b8df4a5d81d

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29517] "Can't pickle local object" when uses functools.partial with method and args...

2017-02-09 Thread DAVID ALEJANDRO Pineda

New submission from DAVID ALEJANDRO Pineda:

Hello.

I'm working in a real time data collector project. I'm using asyncio and 
multiprocessing (run_in_executor).
In python 3.5 worked fine but in python 3.6 not, gave to me this error:
 "Traceback (most recent call last):
  File "/usr/local/lib/python3.6/multiprocessing/queues.py", line 241, in _feed
obj = _ForkingPickler.dumps(obj)
  File "/usr/local/lib/python3.6/multiprocessing/reduction.py", line 51, in 
dumps
cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'WeakSet.__init__.._remove'
"

I tracked and the problem is when i run some method in run_in_executor with the 
functools.partial.

Before this version the system works fine.

The main file is local.py (to run with local administration using a socket)
And the engine is in engine.py

The project (in development):
https://gitlab.com/pineiden/collector

--
components: asyncio
files: code_with_bug.png
messages: 287459
nosy: DAVID ALEJANDRO Pineda, gvanrossum, yselivanov
priority: normal
severity: normal
status: open
title: "Can't pickle local object" when uses functools.partial with method and 
args...
type: behavior
versions: Python 3.6
Added file: http://bugs.python.org/file46614/code_with_bug.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29476] Simplify set_add_entry()

2017-02-09 Thread Serhiy Storchaka

Serhiy Storchaka added the comment:

I don't downvote, no. I am just unsure. I don't have enough information to say 
that the net benefit is positive neither that it is negative. In the face of 
hesitance the status quo wins.

But it looks to me that in dominant cases set_add_entry() is used with a set 
that doesn't contain dummies. What do you think about specializing 
set_add_entry() for this case? set_add_entry() could be optimized for most 
common cases (creating and set-to-set operations), but still not be degrading 
in worst cases.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29034] Fix memory leak and use-after-free in path_converter

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

Does the latest commit fixes a regression introduced by the first fix? Too
bad that we missed the refleak in an issue fixing a memory leak :-/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29513] os.scandir(str) reference leak (test_os refleak)

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

Thanks for the quick fix Serhiy! I was working on patch and I like to check
for refleaks. Sadly, it wasn't noticed before. At least my test was pass :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29516] shutil.copytree(symlinks=True) fails when copying symlinks cross-device and there is no alternative

2017-02-09 Thread Darko Poljak

New submission from Darko Poljak:

shutil.copytree() always calls copystat() for symlinks. Copying symlink to 
another volume fails if some attributes cannot be set. And there is no 
alternative to bypass this as it can be done for files and directories using 
copy_function.
By default copy_function equals copy2() which calls copystat(). There is also 
copy() available which calls copymode().

>From the copytree() source code one can see that for symlink copystat()
is called in any case:

if os.path.islink(srcname):
linkto = os.readlink(srcname)
if symlinks:
# We can't just leave it to `copy_function` because legacy
# code with a custom `copy_function` may rely on copytree
# doing the right thing.
os.symlink(linkto, dstname)
copystat(srcname, dstname, follow_symlinks=not symlinks)

An example is running inside a docker container with a zfs based volume
mounted at /root/.cdist. Root filesytem with /tmp is also on zfs based
container root-volume.

And copying files with shutil.move which in this case uses copytree results in 
the following error:

Traceback (most recent call last):
  File "/appenv/lib/python3.5/shutil.py", line 538, in move
os.rename(src, real_dst)
OSError: [Errno 18] Cross-device link: 
'/tmp/tmpuxb_jr3y/936a8745479046ce91a00ee3013fc9b8/data' -> 
'/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/appenv/bin/cdist", line 94, in 
commandline()
  File "/appenv/bin/cdist", line 66, in commandline
args.func(args)
  File "/appenv/lib/python3.5/site-packages/cdist/config.py", line 157, in 
commandline
args, parallel=False)
  File "/appenv/lib/python3.5/site-packages/cdist/config.py", line 227, in 
onehost
c.run()
  File "/appenv/lib/python3.5/site-packages/cdist/config.py", line 255, in run
self.local.save_cache()
  File "/appenv/lib/python3.5/site-packages/cdist/exec/local.py", line 265, in 
save_cache
shutil.move(self.base_path, destination)
  File "/appenv/lib/python3.5/shutil.py", line 549, in move
symlinks=True)
  File "/appenv/lib/python3.5/shutil.py", line 353, in copytree
raise Error(errors)
shutil.Error: 
[('/tmp/tmpuxb_jr3y/936a8745479046ce91a00ee3013fc9b8/data/conf/explorer/network',
 '/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/network', 
"[Errno 95] Not supported: 
'/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/network'"), 
('/tmp/tmpuxb_jr3y/936a8745479046ce91a00ee3013fc9b8/data/conf/explorer/os', 
'/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/os', "[Errno 
95] Not supported: 
'/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/os'"), 
('/tmp/tmpuxb_jr3y/936a8745479046ce91a00ee3013fc9b8/data/conf/explorer/machine',
 '/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/machine', 
"[Errno 95] Not supported: 
'/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/machine'"), 
('/tmp/tmpuxb_jr3y/936a8745479046ce91a00ee3013fc9b8/data/conf/explorer/init-system',
 
'/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/init-system',
 "[Errno 95] Not supported: '/root/.cdist
 /cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/init-system'"), 
('/tmp/tmpuxb_jr3y/936a8745479046ce91a00ee3013fc9b8/data/conf/explorer/interfaces',
 
'/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/interfaces', 
"[Errno 95] Not supported: 
'/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/interfaces'"),
 
('/tmp/tmpuxb_jr3y/936a8745479046ce91a00ee3013fc9b8/data/conf/explorer/os_version',
 
'/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/os_version', 
"[Errno 95] Not supported: 
'/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/os_version'"),
 
('/tmp/tmpuxb_jr3y/936a8745479046ce91a00ee3013fc9b8/data/conf/explorer/cpu_cores',
 '/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/cpu_cores', 
"[Errno 95] Not supported: 
'/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/cpu_cores'"),
 
('/tmp/tmpuxb_jr3y/936a8745479046ce91a00ee3013fc9b8/data/conf/explorer/lsb_description',
 '/root/.cdist/cache/93
 6a8745479046ce91a00ee3013fc9b8/conf/explorer/lsb_description', "[Errno 95] Not 
supported: 
'/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/lsb_description'"),
 
('/tmp/tmpuxb_jr3y/936a8745479046ce91a00ee3013fc9b8/data/conf/explorer/machine_type',
 
'/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/machine_type',
 "[Errno 95] Not supported: 
'/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/machine_type'"),
 ('/tmp/tmpuxb_jr3y/936a8745479046ce91a00ee3013fc9b8/data/conf/explorer/efi', 
'/root/.cdist/cache/936a8745479046ce91a00ee3013fc9b8/conf/explorer/efi', 
"[Errn

[issue29510] gitignore settings files for Eclipse IDE

2017-02-09 Thread Jim DeLaHunt

Jim DeLaHunt added the comment:

I have set up a global gitignore, and it works for me. 
% git config --global core.excludesfile ~/.gitignore_global
(Amusingly, I had _already- set a global gitignore, but I had forgotten it 
existed, and it didn't ignore these IDE files.)

I agree that we should not put IDE entries in the CPython .gitignore. 

I also think it's a good idea to add the points Inada mentions in the developer 
doc somewhere. That should include the Git global ignore idea. I see Berker's 
point that the Devguide shouldn't get too big, but it also shouldn't leave out 
useful information. There's a balance to strike.

I think that's the topic of a different issue, however.  I'll open that when I 
get back to this.

In the meantime, if someone searches the issue database for "gitignore", they 
will find this discussion. That's helpful.

Thank you both for your responses, Inada and Berker.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread Raymond Hettinger

Raymond Hettinger added the comment:

+1 Though this is a rather large and impactful patch, I think it is a great 
idea.  It will be one of the highest payoff applications of FASTCALL, broadly 
benefitting a lot of code.

Let's be sure to be extra careful with this one because it is touching central 
parts of the language, so any errors or subtle behavior changes will be felt by 
a lot of code, some of which is sure to hit the rare corner cases and to rely 
on implementation details.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29034] Fix memory leak and use-after-free in path_converter

2017-02-09 Thread Roundup Robot

Roundup Robot added the comment:


New changeset be85fd4ef41979dbe68262938da12328fb6cfb8c by Serhiy Storchaka in 
branch '3.6':
Issue #29513: Fixed a reference leak in os.scandir() added in issue #29034.
https://github.com/python/cpython/commit/be85fd4ef41979dbe68262938da12328fb6cfb8c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29513] os.scandir(str) reference leak (test_os refleak)

2017-02-09 Thread Roundup Robot

Roundup Robot added the comment:


New changeset be85fd4ef41979dbe68262938da12328fb6cfb8c by Serhiy Storchaka in 
branch '3.6':
Issue #29513: Fixed a reference leak in os.scandir() added in issue #29034.
https://github.com/python/cpython/commit/be85fd4ef41979dbe68262938da12328fb6cfb8c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29070] Integration tests for pty.spawn on Linux and all other platforms

2017-02-09 Thread Cornelius Diekmann

Cornelius Diekmann added the comment:

Uploaded a new version of the patch.

Changelog of this patch (compared to v3):
 * Fixed reliability issue of existing pty tests.
 * pty.fork() should now also work on systems without os.forkpty(). Added code 
to test this backup path of pty.fork().
 * Reverted my flawed changes to pty.py (broken pipe).
 * All new tests which produce output in the pty.spawn()ed child have the 
terminal in a well-defined state w.r.t. termios.
 * Subtle renaming: PtyTest is now PtyBasicTest. The reason is to execute it 
before the integration tests. All tests are independent and the order is 
irrelevant, but if something fails on a platform, it is easier to debug if we 
run the unit tests before we go to the integration tests.
 * Improved, cleaned, documented, and restructured test code. Thank you Xavier 
and Martin for your very helpful feedback :-)

I also added some small comments in the code review tool.

--
Added file: http://bugs.python.org/file46613/pty.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29515] socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows

2017-02-09 Thread Maximilian Hils

New submission from Maximilian Hils:

The latest Windows builds for Python 3.5.3/3.6.0 do not export 
socket.IPPROTO_IPV6, even though e.g. socket.IPV6_V6ONLY is exported. This 
seems to be wrong to me as IPV6_V6ONLY requires the corresponding socket option 
level IPPROTO_IPV6 to be actually useful. The same issue at least also applies 
to IPPROTO_IPV4.

christian.heimes mentioned that this is intended behaviour in 
https://bugs.python.org/issue6926 as Windows would not define the constants. 
Now I am all but an expert on this, but it seems to me that IPPROTO_IPV6 is 
defined in Windows:
https://msdn.microsoft.com/en-us/library/windows/hardware/ff543746(v=vs.85).aspx.
 Apologies if I'm missing something here.

Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 07:18:10) [MSC v.1900 32 bit 
(Intel)] on win32
>>> import socket; socket.IPPROTO_IPV6
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: module 'socket' has no attribute 'IPPROTO_IPV6'

As a workaround, IPPROTO_IPV6 can be substituted with the hardcoded constant 41.

--
components: Windows
messages: 287449
nosy: christian.heimes, mhils, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows
type: behavior
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29513] os.scandir(str) reference leak (test_os refleak)

2017-02-09 Thread Serhiy Storchaka

Changes by Serhiy Storchaka :


--
resolution:  -> fixed
stage: commit review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue26355] Emit major version based canonical URLs for docs

2017-02-09 Thread Matthias Bussonnier

Matthias Bussonnier added the comment:

> turn my "we should do this" idea into a change we've actually made :)

Looking forward to being able to do this more on GitHub as I am more familiar 
with git.

Good luck for the transition, I'm pretty sure there will be some hard time in 
the next few weeks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29513] os.scandir(str) reference leak (test_os refleak)

2017-02-09 Thread Roundup Robot

Roundup Robot added the comment:

New changeset 4e3a16bdadae by Serhiy Storchaka in branch '3.6':
Issue #29513: Fixed a reference leak in os.scandir() added in issue #29034.
https://hg.python.org/cpython/rev/4e3a16bdadae

New changeset a3f8c5d172b4 by Serhiy Storchaka in branch 'default':
Issue #29513: Fix outdated comment and remove redundand code is os.scandir().
https://hg.python.org/cpython/rev/a3f8c5d172b4

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29034] Fix memory leak and use-after-free in path_converter

2017-02-09 Thread Roundup Robot

Roundup Robot added the comment:

New changeset 4e3a16bdadae by Serhiy Storchaka in branch '3.6':
Issue #29513: Fixed a reference leak in os.scandir() added in issue #29034.
https://hg.python.org/cpython/rev/4e3a16bdadae

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue26355] Emit major version based canonical URLs for docs

2017-02-09 Thread Nick Coghlan

Nick Coghlan added the comment:

Matthias - thanks for figuring out how to turn my "we should do this" idea into 
a change we've actually made :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue1353344] python.desktop

2017-02-09 Thread Nick Coghlan

Nick Coghlan added the comment:

>From a downstream redistributor point of view, the key pieces it would be 
>handy to consolidate upstream are the desktop file itself (with all the 
>translations) and the preferred icon. It wouldn't really help much to have it 
>integrated into "make install" since most redistributors split out tkinter and 
>IDLE packages from the base installation anyway.

However, as Terry says, before the change can be accepted, those files need to 
be combined into an actual patch that:

- adds the desktop file and the icon to the Misc directory
- updates Misc/README to describe the new files

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue6926] socket module missing IPPROTO_IPV6, IPPROTO_IPV4

2017-02-09 Thread Christian Heimes

Christian Heimes added the comment:

The constants are defined on Linux. That means Windows does not define the 
constants and therefore the socket module can't export them.

This ticket is closed. If you still think it's a bug, please open a new ticket 
and reference this ticket.

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue6926] socket module missing IPPROTO_IPV6, IPPROTO_IPV4

2017-02-09 Thread Maximilian Hils

Maximilian Hils added the comment:

This still seems to be an issue with the offical Python 3.6 builds on Windows 
10:

Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 07:18:10) [MSC v.1900 32 bit 
(Intel)] on win32
>>> import socket
>>> socket.IPPROTO_IPV6
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: module 'socket' has no attribute 'IPPROTO_IPV6'

--
nosy: +mhils

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue26355] Emit major version based canonical URLs for docs

2017-02-09 Thread Matthias Bussonnier

Matthias Bussonnier added the comment:

> OK, I'm marking this as closed, since it's as resolved as we can make it 
> through a *CPython* change.

Thanks you Nick, I appreciate the time you took to do that. ANd thank you 
Berker for the review.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

method_fastcall4.patch benchmark results. It's not the first time that I notice 
that fannkuch and nbody benchmarks become slower. I guess that it's effect of 
changing code placement because of unrelated change in the C code.

Results don't seem significant on such macro benchmarks (may be random 
performance changes due to code placement). IMHO the change is worth it! "1.46x 
faster (-31%)" on a microbenchmark is significant and the change is small.

$ python3 -m perf compare_to 
/home/haypo/benchmarks/2017-02-08_15-49-default-f507545ad22a.json 
method_fastcall4_ref_f507545ad22a.json -G --min-speed=5
Slower (2):
- fannkuch: 900 ms +- 20 ms -> 994 ms +- 10 ms: 1.10x slower (+10%)
- nbody: 215 ms +- 3 ms -> 228 ms +- 4 ms: 1.06x slower (+6%)

Faster (3):
- scimark_lu: 357 ms +- 23 ms -> 298 ms +- 8 ms: 1.19x faster (-16%)
- scimark_sor: 400 ms +- 11 ms -> 355 ms +- 12 ms: 1.12x faster (-11%)
- raytrace: 1.05 sec +- 0.01 sec -> 984 ms +- 15 ms: 1.07x faster (-6%)

Benchmark hidden because not significant (59): (...)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread INADA Naoki

INADA Naoki added the comment:

method_fastcall4.patch looks clean enough, and performance benefit seems nice.

I don't know current test suite covers unusual special methods.
Maybe, we can extend test_class to cover !unbound (e.g. @classmethod) case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29505] Submit the re, json, & csv modules to oss-fuzz testing

2017-02-09 Thread Brett Cannon

Changes by Brett Cannon :


--
nosy: +brett.cannon, christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29503] Make embedded-Python detectable

2017-02-09 Thread Steve Dower

Steve Dower added the comment:

You can also use the nuget packages (search on nuget.org for Python). These are 
intended to be used is scripted standalone setups, such as CI systems. I've 
been using them as part of most of my build systems recently. They support 
distutils and pip (not venv in the current version, but that's coming).

You can directly download the nuget.exe tool from https://aka.ms/nugetclidl 
(e.g. as part of a script).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue26355] Emit major version based canonical URLs for docs

2017-02-09 Thread Roundup Robot

Roundup Robot added the comment:


New changeset e8455e7137b9bd63d4c0183558161dba6ea00d32 by Nick Coghlan in 
branch '2.7':
Issue #26355: Specify canonical URLs in docs pages
https://github.com/python/cpython/commit/e8455e7137b9bd63d4c0183558161dba6ea00d32


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread STINNER Victor

Changes by STINNER Victor :


Added file: http://bugs.python.org/file46612/bench.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

method_fastcall4.patch: Based on method_fastcall3.patch, I just added 
call_unbound() and call_unbound_noarg() helper functions to factorize code. I 
also modified mro_invoke() to be able to remove lookup_method().

I confirm the speedup with attached bench.py:

Median +- std dev: [ref] 121 ns +- 5 ns -> [patch] 82.8 ns +- 1.0 ns: 1.46x 
faster (-31%)

--
Added file: http://bugs.python.org/file46611/method_fastcall4.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29449] clear() should return prior state in threading.Event

2017-02-09 Thread Jyotirmoy Bhattacharya

Jyotirmoy Bhattacharya added the comment:

Thanks for the comments above. I'm going to rethink my design. Closing this bug.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29476] Simplify set_add_entry()

2017-02-09 Thread Raymond Hettinger

Raymond Hettinger added the comment:

The problem with posting an idea here on the tracker while it is still in the 
research phase is that it will be prematurely downvoted without have fully 
thought it through.

What I'm working on now is that opposite question.  Was it ever worthwhile to 
add this optimization in the first place?   In particular, does it strongly 
preference a single case over normal cases to no real benefit?

Suppose we found a line code in the calendar module that tested for one special 
case. Is it March 3, 1978, and if so used a precomputed result, but for all 
other dates would compute normally.  The right thing to do would be to remove 
that code, but then one commenter would say, "it makes that one case many times 
slower" when in fact it brings that case into line with the other normal cases. 
 And another respondent would say "removing the almost never used special case 
provides too small of an improvement to the rest of the cases".  The group 
would then arrive at the collective incorrect conclusion that yes it is a great 
idea to keep a special case for March 3, 1978.

The test bench I'm working on now is examining whether a repeated add/discard 
of the same element in an otherwise mostly full table is like the calendar 
module scenario described above.  In particular, I want to know whether 
applying the patch makes the new performance for that case about the same as 
other typical cases of add/discard.

Here are some thoughts using timings.  How you time, what you time, and where 
you time it matter a great deal when evaluating whether to keep code that is 
deep in the set insertion logic.  In a complex function that is already having 
register spills, different compilers may have very different results (i.e. 
eliminating the freeslot lookup may allow another important variable to use a 
register rather than spilling and reloading on every iteration).  We can also 
expect different results on 32-bit vs 64-bit builds (the former having many 
fewer registers to work with) and on ARM vs x86 (with the latter having greater 
instruction level parallelism that lets you get away with having blocks of 
useless code running in parallel with the important code).   

The test dataset matters as well. If set insertion timings are dominated by 
hashing or equality tests, then all improvements to the probing logic with look 
insignificant.  Likewise, if the dataset has few hash collisions, then the 
affected code doesn't get run at all, leading to the false conclusion that 
simplifying the code doesn't have any benefit.

For a timing that properly isolates and exercises the affected code, we should 
expect some measurable (and likely non-trivial) benefit.  The GCC-6 x86_64 
disassembly gives us reason to expect a favorable result because the new inner 
loop has a third fewer instructions, two fewer compare/jumps, and half the 
number of memory accesses (because the freeslot is being reloaded from the 
stack on every iteration).

Also, there hasn't been any exploration of the potential indirect benefits in 
cases that I expect to be far more common.  In those cases where we're going to 
have to resize anyway, is it better to do work to have partial early 
reclamation of dummies and defer resizing, or is it better to avoid the work 
for early reclamation so we can resize sooner and eliminate all of the dummies. 
 I don't know the answer to this question yet, but it is far more important and 
usual than a contrived case of repeatedly adding and discarding the exact same 
element but never hitting a resize.

On the other hand, perhaps the early dummy reclamation is worth it.  I don't 
know the answer yet and was hoping that you all would help me find out.   
Downvoting of the basis of two quick and dirty timings on a single machine, 
single compiler, and single dataset isn't helpful.

I've already put a lot of time in looking at these issues and still don't know 
the right thing to do.  So, if other commenters have only put a few minutes 
into thinking about about it and have already drawn a conclusion, then their 
conclusions are suspect.

IMO, there are still open research questions:  Is early freeslot reclamation 
worth it or does it have a net negative benefit in all but the most exotic and 
unlikely cases?  Without the existing optimization, would the exotic case 
perform about the same as other cases, or is the exotic case catastrophically 
worse than the others so that it warrants special handing even to the detriment 
of the other cases?

--
priority: normal -> low

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue26355] Emit major version based canonical URLs for docs

2017-02-09 Thread Nick Coghlan

Nick Coghlan added the comment:

OK, I'm marking this as closed, since it's as resolved as we can make it 
through a *CPython* change.

Since the old branches aren't autobuilt anymore, adding a canonical URL 
reference to them would presumably be a matter of running a script over the 
built docs.

--
resolution:  -> fixed
stage: commit review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue26355] Emit major version based canonical URLs for docs

2017-02-09 Thread Roundup Robot

Roundup Robot added the comment:


New changeset a47e20b636d2a5559e5831c6805df3cba1ddb2a1 by Nick Coghlan in 
branch '3.5':
Issue #26355: Specify canonical URLs in docs pages
https://github.com/python/cpython/commit/a47e20b636d2a5559e5831c6805df3cba1ddb2a1


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue26355] Emit major version based canonical URLs for docs

2017-02-09 Thread Roundup Robot

Roundup Robot added the comment:

New changeset b07d454e45a2 by Nick Coghlan in branch '2.7':
Issue #26355: Specify canonical URLs in docs pages
https://hg.python.org/cpython/rev/b07d454e45a2

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue26355] Emit major version based canonical URLs for docs

2017-02-09 Thread Roundup Robot

Roundup Robot added the comment:


New changeset a47e20b636d2a5559e5831c6805df3cba1ddb2a1 by Nick Coghlan in 
branch 'master':
Issue #26355: Specify canonical URLs in docs pages
https://github.com/python/cpython/commit/a47e20b636d2a5559e5831c6805df3cba1ddb2a1

New changeset 37150972faf660571a3ae7076a623087c06b8791 by Nick Coghlan in 
branch 'master':
Merge issue #26355 fix from Python 3.5
https://github.com/python/cpython/commit/37150972faf660571a3ae7076a623087c06b8791

New changeset d267bc695eaf9422668daedb9e2696e01fe7 by Nick Coghlan in 
branch 'master':
Merge issue #26355 fix from 3.6
https://github.com/python/cpython/commit/d267bc695eaf9422668daedb9e2696e01fe7


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29514] Add a test case that prevents magic number changes in minor releases

2017-02-09 Thread Petr Viktorin

Petr Viktorin added the comment:

For the record, the magic number was changed in issue27286

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29514] Add a test case that prevents magic number changes in minor releases

2017-02-09 Thread Nick Coghlan

New submission from Nick Coghlan:

The magic number change in 3.5.3 broke the world when Fedora attempted to 
rebase from 3.5.2, but upstream CPython developers don't currently get an 
automated notification that it isn't permitted to change the magic number in a 
maintenance release - instead, the current approach relies on people "just 
knowing" not to do that, and we just saw the limitations of relying on the code 
review process to catch this particular problem.


I think we can keep this from happening again by having an importlib test case 
that fails if sys.version_info.releaselevel is "candidate" or "final" and 
either:

- no "expected magic number" is defined; or
- the actual magic number in importlib.util.MAGIC_NUMBER doesn't match the 
expected one

The comments on the new test case would then explain the policy and why it 
isn't OK to break pyc file compatibility in a maintenance release.

We'd also propose an update to the release PEP warning release managers to 
expect this error as part of prepping the first release candidate, and that 
they should lock in the magic number for the release at that time.

--
messages: 287428
nosy: barry, brett.cannon, doko, encukou, ncoghlan
priority: normal
severity: normal
status: open
title: Add a test case that prevents magic number changes in minor releases
versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29506] Incorrect documentation for the copy module

2017-02-09 Thread Marco Buttu

Marco Buttu added the comment:

Serhiy is right about "deep copy" instead of "deepcopy", but IMO the Steven's 
proposal (with the "deep copy" correction) is much clearer than the current doc.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29506] Incorrect documentation for the copy module

2017-02-09 Thread Serhiy Storchaka

Serhiy Storchaka added the comment:

Current documentation looks correct to me. What is the problem with it?

Steven's formulation says about "deepcopy", but current documentation says 
about common issues of "deep copy" and describes how copy.deepcopy() solves 
them.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue26355] Emit major version based canonical URLs for docs

2017-02-09 Thread Roundup Robot

Roundup Robot added the comment:

New changeset c63b09833141 by Nick Coghlan in branch '3.5':
Issue #26355: Specify canonical URLs in docs pages
https://hg.python.org/cpython/rev/c63b09833141

New changeset 80970cf56048 by Nick Coghlan in branch '3.6':
Merge issue #26355 fix from Python 3.5
https://hg.python.org/cpython/rev/80970cf56048

New changeset 26af402c291f by Nick Coghlan in branch 'default':
Merge issue #26355 fix from 3.6
https://hg.python.org/cpython/rev/26af402c291f

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29513] os.scandir(str) reference leak (test_os refleak)

2017-02-09 Thread Xiang Zhang

Xiang Zhang added the comment:

> If PyObject_New(ScandirIterator, &ScandirIteratorType) fails the path should 
> be cleaned up by Argument Clinic.

Ohh yes. My stupid. Then both LGTM.

--
stage: patch review -> commit review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29513] os.scandir(str) reference leak (test_os refleak)

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

scandir-refleak-3.6.patch LGTM. The following removed Py_CLEAR() is just 
redundant, so it's ok to remove it.

-Py_CLEAR(iterator->path.object);

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29513] os.scandir(str) reference leak (test_os refleak)

2017-02-09 Thread Serhiy Storchaka

Serhiy Storchaka added the comment:

If PyObject_New(ScandirIterator, &ScandirIteratorType) fails the path should be 
cleaned up by Argument Clinic.

--
assignee:  -> serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29506] Incorrect documentation for the copy module

2017-02-09 Thread R. David Murray

R. David Murray added the comment:

I prefer Steven's formulation.  The parenthetical is more distracting than 
clarifying, I think.  I agree that 'administrative' is confusing and 
unnecessary.

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29513] os.scandir(str) reference leak (test_os refleak)

2017-02-09 Thread Xiang Zhang

Xiang Zhang added the comment:

3.6 LGTM. 3.7 is technically right to me. But it looks to me AC shouldn't call 
path_cleanup in .c.h. It always do nothing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue26355] Emit major version based canonical URLs for docs

2017-02-09 Thread Nick Coghlan

Nick Coghlan added the comment:

Looks good to me, so I'll apply these right now :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue26355] Emit major version based canonical URLs for docs

2017-02-09 Thread Nick Coghlan

Changes by Nick Coghlan :


--
assignee: docs@python -> ncoghlan
stage: patch review -> commit review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread INADA Naoki

INADA Naoki added the comment:

method_fastcall3.patch implement the trick in more general way.
(I haven't ran test yet since it's midnight.  I'll post result later.)

--
Added file: http://bugs.python.org/file46610/method_fastcall3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29212] Python 3.6 logging thread name regression with concurrent.future threads

2017-02-09 Thread desbma

desbma added the comment:

Ping, so that this has a chance for the 3.6.1 release.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29513] os.scandir(str) reference leak (test_os refleak)

2017-02-09 Thread Serhiy Storchaka

Serhiy Storchaka added the comment:

3.7 doesn't leak, but contains outdated comment and code.

--
versions: +Python 3.7
Added file: http://bugs.python.org/file46609/scandir-refleak-3.7.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29513] os.scandir(str) reference leak (test_os refleak)

2017-02-09 Thread Serhiy Storchaka

Serhiy Storchaka added the comment:

The owning of references in the path_t structure was changed in issue29034, but 
some code still directly manipulates reference counters of path_t fields. 
Proposed patch should fix the leak.

--
keywords: +patch
nosy: +xiang.zhang
stage:  -> patch review
Added file: http://bugs.python.org/file46608/scandir-refleak-3.6.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29506] Incorrect documentation for the copy module

2017-02-09 Thread Pedro

Pedro added the comment:

"Because deepcopy copies everything it may copy too much, such as data which is 
intended to be shared (instead of duplicated) between copies."

A bit more explicit?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread STINNER Victor

Changes by STINNER Victor :


--
nosy: +yselivanov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

> method_fastcall2.patch is tuning same function (call_method() in 
> typeobject.c), and uses trick to bypass temporary method object (same to 
> _PyObject_GetMethod()).

Oh, great idea! That's why I put you in the nosy list ;-) You know better than 
me this area of the code.

> Median +- std dev: [python.default] 155 ns +- 4 ns -> [python] 111 ns +- 1 
> ns: 1.40x faster (-28%)

Wow, much better than my patch. Good job!

Can we implement the same optimization in callmethod() of Objects/abstract.c? 
Maybe add a "is_method" argument to the static function 
_PyObject_CallFunctionVa(), to only enable the optimization for callmehod().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29306] Check usage of Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() in new FASTCALL functions

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

I tested check_recursion_depth.py: Python 2.7, 3.5 and 3.6 have the bug. Python 
3.7 is already fixed.

--
versions: +Python 2.7, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29306] Check usage of Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() in new FASTCALL functions

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

check_recursion_depth.py: script using Python functions and C function 
(functools.partial) to check the recursion depth.

Example with Python 3.6:

haypo@selma$ ./python check_recursion_depth.py   # unpatched
got: 148, expected: 100

haypo@selma$ ./python check_recursion_depth.py   # patched
got: 100, expected: 100

Maybe we need "proxies" in _testcapi for each kind of function, a function 
taking a callback and calling this function. Function types:

* C function, METH_NOARGS
* C function, METH_O
* C function, METH_VARRGS
* C function, METH_VARRGS | METH_KEYWORDS
* C function, METH_FASTCALL
* Python function
* Maybe even most common C functions to call functions: PyObject_Call(), 
_PyObject_FastCallDict(), _PyObject_FastCallKeywords(), etc.

--
Added file: http://bugs.python.org/file46607/check_recursion_depth.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread INADA Naoki

INADA Naoki added the comment:

I'm sorry, callmethod.patch is tuned other place, and causing SEGV.

method_fastcall2.patch is tuning same function (call_method() in typeobject.c), 
and uses trick to bypass temporary method object (same to 
_PyObject_GetMethod()).

$ ./python -m perf timeit --compare-to `pwd`/python.default  -s 'class C:' -s ' 
def __getitem__(self, index): return index' -s 'c=C()' 'c[0]'
python.default: . 155 ns +- 4 ns
python: . 111 ns +- 1 ns

Median +- std dev: [python.default] 155 ns +- 4 ns -> [python] 111 ns +- 1 ns: 
1.40x faster (-28%)

--
Added file: http://bugs.python.org/file46606/method_fastcall2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29513] os.scandir(str) reference leak (test_os refleak)

2017-02-09 Thread STINNER Victor

New submission from STINNER Victor:

The following code leaks one reference on Python 3.6:

def test_attributes(self):
d = dict((entry.name, entry) for entry in os.scandir('.'))

Or try the command:

   ./python -m test -R 3:3 test_os -v -m test_attributes

--
components: Library (Lib)
messages: 287409
nosy: haypo, serhiy.storchaka
priority: normal
severity: normal
status: open
title: os.scandir(str) reference leak (test_os refleak)
type: resource usage
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue11726] clarify that linecache only works on files that can be decoded successfully

2017-02-09 Thread Antti Haapala

Antti Haapala added the comment:

Every now and then there are new questions and answers regarding the use of 
`linecache` module on Stack Overflow for doing random access to text files, 
even though the documentation states that it is meant for Python source code 
files.

One problem is that the title still states: "11.9. linecache — Random access to 
text lines"; the title should really be changed to "Random access to Python 
source code lines" so that the title wouldn't imply that this is a 
general-purpose random access library for text files.

--
nosy: +ztane

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29480] Mac OSX Installer SSL Roots

2017-02-09 Thread Christian Heimes

Changes by Christian Heimes :


--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29504] blake2: compile error with -march=bdver2

2017-02-09 Thread Christian Heimes

Christian Heimes added the comment:

I don't have time to fix the issue right away. Python's copy of blake2 needs 
some manual massaging and tweaking.

--
assignee:  -> christian.heimes
nosy: +ned.deily
priority: normal -> release blocker
stage:  -> needs patch
type:  -> compile error
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29504] blake2: compile error with -march=bdver2

2017-02-09 Thread Christian Heimes

Christian Heimes added the comment:

It's a bug in Blake2's reference implementation with nested C++ comments (/* /* 
*/ */). 
https://github.com/BLAKE2/BLAKE2/commit/259e61dedee5383eac1a90db6ef88f9ccdcf6002#diff-ac1a341ec0ed38dc5d219572f4282287
 fixed the bug. For Python 3.6.1 I'm going to update Python's copy of blake2.

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue1025395] email.Utils.parseaddr fails to parse valid addresses

2017-02-09 Thread Sascha Silbe

Changes by Sascha Silbe :


--
nosy: +sascha_silbe

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29512] regrtest refleak: implement bisection feature

2017-02-09 Thread STINNER Victor

New submission from STINNER Victor:

It would be great to be able to list which methods cause a reference leak.

Example right now on the 3.6 branch:

$ ./python -m test -R 3:3 test_os 
Run tests sequentially
0:00:00 [1/1] test_os
beginning 6 repetitions
123456
..
test_os leaked [124, 124, 124] references, sum=372
test_os leaked [114, 114, 114] memory blocks, sum=342
test_os failed in 32 sec

1 test failed:
test_os

Total duration: 32 sec
Tests result: FAILURE


Which methods caused the refleak?

Is it possible to list test methods of a test file using unittest?

--
components: Tests
messages: 287405
nosy: haypo
priority: normal
severity: normal
status: open
title: regrtest refleak: implement bisection feature
type: enhancement
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29306] Check usage of Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() in new FASTCALL functions

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

enter_recursive_call_36.patch: Patch for Python 3.6.

* Add Py_EnterRecursiveCall() into C functions
* Don't call C functions inside a Py_EnterRecursiveCall() block in 
PyObject_Call()

--
Added file: http://bugs.python.org/file46605/enter_recursive_call_36.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29400] Add instruction level tracing via sys.settrace

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

See aslo issue #29502: "Should PyObject_Call() call the profiler on C 
functions, use C_TRACE() macro?"

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29502] Should PyObject_Call() call the profiler on C functions, use C_TRACE() macro?

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

Output of attached profiler_c_code.py example:
---
haypo@selma$ ./python profiler_c_code.py 
(, 'call', None)
(, 'return', 'h')
(, 'call', None)
(, 'return', 'e')
(, 'call', None)
(, 'return', 'l')
(, 'call', None)
(, 'return', 'l')
(, 'call', None)
(, 'return', 'o')
(, 'c_call', )
---

Except of sys.setprofile(), the profiler doesn't see any C calls! list() and 
map() are simply "hidden".

So tools like coverage miss a lot of C call? Again, is it a compromise between 
performance and correctness, or just a regular bug?

--
Added file: http://bugs.python.org/file46604/profiler_c_code.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread INADA Naoki

INADA Naoki added the comment:

callmethod.patch:

+ ../python.default -m perf compare_to default.json patched2.json -G 
--min-speed=1
Slower (5):
- logging_silent: 717 ns +- 9 ns -> 737 ns +- 8 ns: 1.03x slower (+3%)
- fannkuch: 1.04 sec +- 0.01 sec -> 1.06 sec +- 0.02 sec: 1.02x slower (+2%)
- call_method: 14.5 ms +- 0.1 ms -> 14.7 ms +- 0.1 ms: 1.02x slower (+2%)
- call_method_slots: 14.3 ms +- 0.3 ms -> 14.6 ms +- 0.1 ms: 1.02x slower (+2%)
- scimark_sparse_mat_mult: 8.66 ms +- 0.21 ms -> 8.76 ms +- 0.25 ms: 1.01x 
slower (+1%)

Faster (17):
- scimark_lu: 433 ms +- 28 ms -> 410 ms +- 24 ms: 1.06x faster (-5%)
- unpickle: 32.9 us +- 0.2 us -> 31.7 us +- 0.3 us: 1.04x faster (-4%)
- sqlite_synth: 10.0 us +- 0.2 us -> 9.77 us +- 0.24 us: 1.03x faster (-3%)
- telco: 21.1 ms +- 0.7 ms -> 20.6 ms +- 0.4 ms: 1.03x faster (-2%)
- unpickle_list: 8.22 us +- 0.18 us -> 8.02 us +- 0.17 us: 1.03x faster (-2%)
- json_dumps: 30.3 ms +- 0.8 ms -> 29.6 ms +- 0.4 ms: 1.02x faster (-2%)
- nbody: 245 ms +- 6 ms -> 240 ms +- 5 ms: 1.02x faster (-2%)
- meteor_contest: 207 ms +- 2 ms -> 203 ms +- 2 ms: 1.02x faster (-2%)
- scimark_fft: 738 ms +- 14 ms -> 727 ms +- 17 ms: 1.02x faster (-2%)
- pickle_pure_python: 1.27 ms +- 0.02 ms -> 1.25 ms +- 0.02 ms: 1.01x faster 
(-1%)
- django_template: 401 ms +- 4 ms -> 395 ms +- 5 ms: 1.01x faster (-1%)
- sqlalchemy_declarative: 317 ms +- 3 ms -> 313 ms +- 4 ms: 1.01x faster (-1%)
- json_loads: 64.2 us +- 1.0 us -> 63.4 us +- 1.0 us: 1.01x faster (-1%)
- nqueens: 270 ms +- 2 ms -> 267 ms +- 2 ms: 1.01x faster (-1%)
- crypto_pyaes: 234 ms +- 1 ms -> 231 ms +- 3 ms: 1.01x faster (-1%)
- chaos: 300 ms +- 2 ms -> 297 ms +- 4 ms: 1.01x faster (-1%)
- sympy_expand: 1.01 sec +- 0.01 sec -> 1.00 sec +- 0.01 sec: 1.01x faster (-1%)

Benchmark hidden because not significant (42)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29476] Simplify set_add_entry()

2017-02-09 Thread STINNER Victor

STINNER Victor added the comment:

I vote -1 on set_no_dummy_reuse.diff since it doesn't show any significant 
speedup (3 ns on 130 ns with a std dev of 8 ns is not something significant), 
and makes the set type 2x slower on some corner cases.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29507] Use FASTCALL in call_method() to avoid temporary tuple

2017-02-09 Thread INADA Naoki

INADA Naoki added the comment:

> But PyObject_CallMethod(), _PyObject_CallMethodId(), 
> PyObject_CallMethodObjArgs(), _PyObject_CallMethodIdObjArgs() can use it too.

CallMethod[Id]ObjArgs() can use it easily.
But format support is not so easy.

--
Added file: http://bugs.python.org/file46603/callmethod.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29476] Simplify set_add_entry()

2017-02-09 Thread STINNER Victor

Changes by STINNER Victor :


--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29506] Incorrect documentation for the copy module

2017-02-09 Thread Marco Buttu

Marco Buttu added the comment:

+1 for the Steven's suggestion

--
nosy: +marco.buttu

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29511] Add 'find' as build-in method for lists

2017-02-09 Thread Steven D'Aprano

Steven D'Aprano added the comment:

Only 3.7 can receive new functionality.

Here is a pure Python implementation of a subsequence test:

https://code.activestate.com/recipes/577850-search-sequences-for-sub-sequence/

It appears to be reasonably popular on Activestate: it has about 7000 views, 
but a score of only 1. Take of that what you will. I interpret it as meaning it 
is of moderate but not great interest to people.

--
nosy: +steven.daprano
versions:  -Python 2.7, Python 3.3, Python 3.4, Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29510] gitignore settings files for Eclipse IDE

2017-02-09 Thread INADA Naoki

INADA Naoki added the comment:

I agree with Berker.

But some git/github tips are very useful for CPython core developers while they 
aren't CPython specific.
And some of them are not in beginner's guide.

I think it's worth enough to add link to such tips.

* Use gitignore_global to ignore IDE/Editor specific files: 
https://help.github.com/articles/ignoring-files/
* Checkout pull request without adding remote: 
https://help.github.com/articles/checking-out-pull-requests-locally/
* Readable patch by git diff --indent-heuristic (git 2.11+) or 
--compact-heuristic (git 2.9+).
* Creating lightweight working tree by git-new-workdir (like `hg share`):
 https://github.com/git/git/blob/master/contrib/workdir/git-new-workdir

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29510] gitignore settings files for Eclipse IDE

2017-02-09 Thread Berker Peksag

Berker Peksag added the comment:

Thanks for the report and for the patch, Jim! I agree with Inada. 
global_gitignore is a better way to achieve this.

> Maybe a better solution would be to add something to the devguide about
> excluding IDE settings files, with a link to the Github instructions.

Our goal is to only cover CPython specific topics in devguide. There a lot of 
things that can be mentioned in devguide, but it would be hard to maintain all 
these information and keep devguide short at same time :)

--
nosy: +berker.peksag
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29511] Add 'find' as build-in method for lists

2017-02-09 Thread George Shuklin

New submission from George Shuklin:

I found that Python provides 'find()' and 'in' methods for strings, but lacking 
same functionality for lists.

Because strings and lists are very similar, it's reasonable to expect same 
function available for both.

Here long and rather ugly hack list on stackoverflow about 'reinventing the 
wheel': 
http://stackoverflow.com/questions/16579085/python-verifying-if-one-list-is-a-subset-of-the-other

There are short few proposals, each of them imperfect:
1. Use sets intersection. This looses count and order
2. Use collections.Count. This looses order
3. all(x in two for x in one) - looses order

Propsal: adds a normal 'find' method which will behave the same way as find for 
strings. It should perform normal __cmp__ call on each element, or, may be, 
asking for optional lambda to perform comparison of elements.

--
components: Interpreter Core
messages: 287394
nosy: george-shuklin
priority: normal
severity: normal
status: open
title: Add 'find' as build-in method for lists
type: enhancement
versions: Python 2.7, Python 3.3, Python 3.4, Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29510] gitignore settings files for Eclipse IDE

2017-02-09 Thread Jim DeLaHunt

Jim DeLaHunt added the comment:

gitignore_global is a great idea. I had not heard of it before. 

But here it is: https://help.github.com/articles/ignoring-files/  . This 
instruction also has a link to a gist with a lot of helpful global ignores.

I understand your reluctance to add entries for every IDE. Maybe a better 
solution would be to add something to the devguide about excluding IDE settings 
files, with a link to the Github instructions.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29510] gitignore settings files for Eclipse IDE

2017-02-09 Thread INADA Naoki

INADA Naoki added the comment:

We'll move to github soon.  So no need for caring hgignore.

While there is `.vscode` in gitignore already, I'm not fan of
adding hundreds of IDE specific rule to gitignore.
Why don't you use gitignore_global?

--
nosy: +inada.naoki

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29510] gitignore settings files for Eclipse IDE

2017-02-09 Thread Jim DeLaHunt

Jim DeLaHunt added the comment:

I'm now looking at cpython as retrieved from Mercurial by Eclipse. It appears 
to be concerned by the presence of 
.project
which is also an Eclipse settings file.  

It might be reasonable to extend the scope of this issue to include telling 
Mercurial to ignore settings files. My pull request does not do this, because I 
haven't yet learned the right Mercurial action.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue29503] Make embedded-Python detectable

2017-02-09 Thread Andi Bergmeier

Andi Bergmeier added the comment:

I am not the primary contributor to Bazel, so the decision to support 
embeddable Python is not mine to make.

That said, it seems to me like embeddable Python is a perfect fit if you want 
to have a hermetic Python installation. And I AM actually the person advocating 
for support for embeddable Python :).

We should discuss whether we use -c or a wrapper script. The latter might be 
the better option on the long run for other reasons, too.

My original question stems from the desire to only use e.g. a wrapper script 
for embeddable for the near future. Reason being that it is not yet as well 
tested as the PYTHONPATH approach. So would be hesitant to switch from 
PYTHONPATH to wrapper script for all variants. Also I am not sure yet, the 
wrapper script is an option for all cases.

Thanks for the input.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



  1   2   >