[issue26601] Use new madvise()'s MADV_FREE on the private heap

[Charles-François Natali]

Charles-François Natali added the comment:

> Julian Taylor added the comment:
>
> it defaulted to 128kb ten years ago, its a dynamic threshold since ages.

Indeed, and that's what encouraged switching the allocator to use mmap.
The problem with dynamic mmap threshold is that since the Python
allocator uses fixed-size arenas, basically malloc always ends up
allocating from the heap (brk).
Which means that given that we don't use a - compacting - garbage
collector, after a while the heap would end up quite fragmented, or
never shrink: for example let's say you allocate 1GB - on the heap -
and then you free them, but  a single object is allocated at the top
of the heap, you heap never shrinks back.
This has bitten people (and myself a couple times at work).

Now, I see several options:
- revert to using malloc, but this will re-introduce the original problem
- build some form of hysteresis in the arena allocation
- somewhat orthogonally, I'd be interested to see if we couldn't
increase the arena size

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26825] Variable defined in exec(code) unreachable inside function call with visible name in dir() results

[ganix]

New submission from ganix:

here is a code show what happend:
>>> def func():
exec('ans=1')
print(dir())
return ans

>>> func()
['ans']
Traceback (most recent call last):
  File "", line 1, in 
func()
  File "", line 4, in func
return ans
NameError: name 'ans' is not defined

i tried this code in version 2.7,it is ok

--
components: Argument Clinic
messages: 263967
nosy: 324857679, larry
priority: normal
severity: normal
status: open
title: Variable defined in exec(code) unreachable inside function call with 
visible name in dir() results
type: crash
versions: Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[Xiang Zhang]

Xiang Zhang added the comment:

Good point. I don't think about that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Ah, sorry. This makes sense.

But there is one catch. If allow a macro to be used with an argument not of 
type PyObject*, this will make harder to change the implementation and convert 
the macro to a function.

And this may increase the probability to make an error with calling macros with 
wrong argument.

I have no strong opinion about this patch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[Xiang Zhang]

Xiang Zhang added the comment:

I also inspect Py_REFCNT and Py_SIZE. I don't think there is need to replace 
(obj)->ob_refcnt since they are legal. 

So my intention is only several macros need to be altered and won't make too 
much code churn.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[Xiang Zhang]

Xiang Zhang added the comment:

No, actually I don't mean to change all the (obj)->ob_type in Python repo. I 
know there are more (obj)->ob_type in it, but they are compliant with PEP3123 
since obj is of type PyObject*. You don't have the need to change them. What I 
propose is only to change the macros that not use Py_TYPE since the macros may 
be used later with arguments not of type PyObject*.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26822] itemgetter/attrgetter/methodcaller objects ignore keyword arguments

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> It really seems like methodcaller should allow additional arguments 
> (positional and keyword though)

This is good argument for raising an exception. Wrong expectations can lead to 
incorrect code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

In Python 2.6 PyObject_HEAD was defined as

#define PyObject_HEAD   \
_PyObject_HEAD_EXTRA\
Py_ssize_t ob_refcnt;   \
struct _typeobject *ob_type;

Every extension type structure contained fields ob_refcnt and ob_type. For the 
pointer of type (FooObject *) you used foo->ob_refcnt and foo->ob_type. You 
could also use ob = (PyObject *)foo; ob->ob_type, but this is undefined 
behavior, as estimated in PEP3123.

Now PyObject_HEAD is defined as

#define PyObject_HEAD   PyObject ob_base;

Every extension type structure no longer contain fields ob_refcnt and ob_type, 
and you can't access them directly. For convenience and compatibility with 2.6 
there were added macros Py_TYPE() and Py_REFCNT(). Direct access to ob_refcnt 
and ob_type is legal, just your can't use it with arbitrary pointer type 
without casting it to PyObject*. That just can't be compiled. If you have a 
pointer of type PyObject*, you can use direct access to ob_refcnt and ob_type.

Direct access to ob_type is used about 300 times in the source code. Changing 
all this case to use the Py_TYPE() macro causes code churn.

There is different situation with ob_refcnt. It is less used, and changing the 
code to use Py_REFCNT() causes less code churn. But this can make the code 
looks more pretty. Here is a patch (written a day ago) that makes the code to 
use Py_REFCNT(). I was not sure about the code that directly modifies the 
reference count and left it unchanged. I'm not sure that it is worth to apply 
this patch, may be it still makes too much code churn.

--
components: +Extension Modules, Interpreter Core
nosy: +loewis, ned.deily, rhettinger
stage:  -> patch review
Added file: http://bugs.python.org/file42564/py_refcnt.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26821] array module "minimum size in bytes" table is wrong for int/long

[Georg Brandl]

Georg Brandl added the comment:

Agreed.

--
resolution:  -> not a bug
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26823] Shrink recursive tracebacks

[Nick Coghlan]

Nick Coghlan added the comment:

For testing, you can create a recursive scenario that terminates with an 
exception after a defined number of iterations:

>>> def f(counter):
... if counter:
... f(counter-1)
... else:
... raise RuntimeError
... 
>>> f(3)
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 3, in f
  File "", line 3, in f
  File "", line 3, in f
  File "", line 5, in f
RuntimeError

It's probably also worth checking the handling of two distinct frame 
repetitions by having another recursion counter that terminates itself by 
calling the one that terminates by raising an exception

Beyond that, https://hg.python.org/cpython/file/tip/Lib/test/test_traceback.py 
covers both traceback formatting implementations, so a new test case in 
TracebackFormatTests seems appropriate.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26821] array module "minimum size in bytes" table is wrong for int/long

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> I can't get two arrays one int, one byte with the same backing memory where 
> changes to one effect the other

For this case you can use an array or bytearray + memoryview.

I think this issue can be closed.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26821] array module "minimum size in bytes" table is wrong for int/long

[Georg Brandl]

Georg Brandl added the comment:

Indeed, I don't think the `array` module is much used anymore.

If you're looking to serialize or deserialize fixed-size formats, you'll 
probably want the `struct` module.  It has both native and fixed-size modes.

For anything else involving arrays (mostly, but not exclusively, of numbers), 
just use numpy.  It has a much more developed system of data types for its 
arrays, and supports views.

--
nosy: +georg.brandl

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[Xiang Zhang]

Changes by Xiang Zhang :


--
type:  -> enhancement
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26824] Make some macros use Py_TYPE

[Xiang Zhang]

New submission from Xiang Zhang:

According to PEP3123, all accesses to ob_refcnt and ob_type MUST cast the 
object pointer to PyObject* (unless the pointer is already known to have that 
type), and SHOULD use the respective accessor macros.

I find that there are still some macros in Python use (obj)->ob_type. Though 
right now they may not impose any error, but as macros, they may be used with 
arguments not of type PyObject* later and introduce errors. So I think change 
them to use Py_TYPE is not a bad idea.

--
files: change_some_macros_using_Py_TYPE.patch
keywords: patch
messages: 263956
nosy: serhiy.storchaka, xiang.zhang
priority: normal
severity: normal
status: open
title: Make some macros use Py_TYPE
Added file: 
http://bugs.python.org/file42563/change_some_macros_using_Py_TYPE.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26823] Shrink recursive tracebacks

[Emanuel Barry]

Emanuel Barry added the comment:

Attached patch also modifies Lib/traceback.py to present identical behaviour, 
and changes "Previous message" to "Previous line". I'll postpone the more 
complex implementation of that, and might just not do it as it's indeed better 
to avoid bugs where we're meant to handle bugs =)

Should there be unit tests for that? I'm really not sure how to write tests for 
that particular case...

--
Added file: http://bugs.python.org/file42562/short_tracebacks_2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26823] Shrink recursive tracebacks

[Nick Coghlan]

Nick Coghlan added the comment:

+1 for the simple approach, and deferring the mutual recursion support - it's 
desirable to keep traceback printing simple in order to minimise the chance for 
failures during the display process.

In addition to the C implementation of traceback printing, the standard 
library's traceback module would also need updating.

Regarding the ambiguity of "Previous message", I'd suggest using "Previous 
line" instead - that way it's accurate regardless of whether people read it as 
"previous line of the traceback" or "previous line of quoted source code".

--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22234] urllib.parse.urlparse accepts any falsy value as an url

[Luiz Poleto]

Changes by Luiz Poleto :


--
nosy: +luiz.poleto

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26823] Shrink recursive tracebacks

[Emanuel Barry]

Emanuel Barry added the comment:

The message is mostly a placeholder, but "message" is singular so I figured it 
would be obvious. But alas, if you are confused, others might be too. 
Propositions for a better message are welcome :)

I'll attempt to make it track chained calls (or mutually recursive functions) 
tomorrow, my sleep-deprived brain is unable to find a clean way to do this 
right now :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26823] Shrink recursive tracebacks

[Xiang Zhang]

Xiang Zhang added the comment:

With the current patch, a simple test gives the traceback:

Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in test
  File "", line 2, in test
  File "", line 2, in test
  [Previous message repeated 995 more times]
RecursionError: maximum recursion depth exceeded

I think the message [Previous message repeated 995 more times] is vague. Does 
it refer to the previous line or the whole previous messages? Hmm, at least it 
is vague for me, a non-native English speaker.

--
nosy: +xiang.zhang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26823] Shrink recursive tracebacks

[Ethan Furman]

Changes by Ethan Furman :


--
nosy: +georg.brandl

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26823] Shrink recursive tracebacks

[Ethan Furman]

Ethan Furman added the comment:

If you can, give it a go.  Make it a new patch, though -- don't delete the 
existing one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26823] Shrink recursive tracebacks

[Ethan Furman]

Changes by Ethan Furman :


--
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26823] Shrink recursive tracebacks

[Emanuel Barry]

Emanuel Barry added the comment:

Yes, can't handle mutually recursive functions. I could maybe check for the 
last two or three functions, but that seems like unnecessary work for something 
that might not happen as often (I can see it being the case with e.g. 
__getattr__ though).

If enough people think it should keep track of all or most functions being 
called, I can do it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26823] Shrink recursive tracebacks

[Chris Angelico]

Chris Angelico added the comment:

By "doesn't keep track of call chains", you mean that it can't handle 
mutually-recursive functions, right?

Still useful.

--
nosy: +Rosuav

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26823] Shrink recursive tracebacks

[Ethan Furman]

Changes by Ethan Furman :


--
nosy: +ethan.furman

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26823] Shrink recursive tracebacks

[Emanuel Barry]

New submission from Emanuel Barry:

I recently suggested on Python-ideas ( 
https://mail.python.org/pipermail/python-ideas/2016-April/039899.html ) to 
shrink long tracebacks if they were all the same noise (recursive calls). 
Seeing as the idea had a good reception, I went ahead and implemented a small 
patch for this.

It doesn't keep track of call chains, and I'm not sure if we want to implement 
that, as the performance decrease needed to store all of that might not be 
worth it. But then again, if an error happened performance is probably not a 
concern in this case.

I've never really coded in C before, so feedback is very much welcome.

--
components: Interpreter Core
files: short_tracebacks.patch
keywords: patch
messages: 263948
nosy: ebarry
priority: normal
severity: normal
stage: patch review
status: open
title: Shrink recursive tracebacks
versions: Python 3.6
Added file: http://bugs.python.org/file42561/short_tracebacks.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26822] itemgetter/attrgetter/methodcaller objects ignore keyword arguments

[Xiang Zhang]

Xiang Zhang added the comment:

This makes sense. It makes the C version and Python version consistent.

--
nosy: +xiang.zhang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] [WIP] Add a new _PyObject_FastCall() function which avoids the creation of a tuple or dict for arguments

[STINNER Victor]

STINNER Victor added the comment:

I created a repository. I will work there and make some experiment. It would 
help to have a better idea of the concrete performance. When I will have a 
better view of all requires changes to get best performances everywhere, I will 
start a discussion to see which parts are worth it or not. In my latest 
microbenchmarks, functions calls (C/Python, mixed) are between 8% and 40% 
faster. I'm now running the CPython benchmark suite.

--
title: Add a new _PyObject_FastCall() function which avoids the creation of a 
tuple or dict for arguments -> [WIP] Add a new _PyObject_FastCall() function 
which avoids the creation of a tuple or dict for arguments

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] Add a new _PyObject_FastCall() function which avoids the creation of a tuple or dict for arguments

[STINNER Victor]

Changes by STINNER Victor :


--
hgrepos: +342

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26041] Update deprecation messages of platform.dist() and platform.linux_distribution()

[Luiz Poleto]

Changes by Luiz Poleto :


--
nosy: +luiz.poleto

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26820] Prevent uses of format string based PyObject_Call* that do not produce tuple required by docs

[Jeremy Kloth]

Jeremy Kloth added the comment:

IMHO, this is a documentation bug with PyObject_CallMethod.  The change to its 
documentation to differ from PyObject_CallFunction was changed back in 2004.  
It should have been updated then to reflect the already well-entrenched 
behavior of those 2 (at the time) functions, but that ship has sailed.

It would be a huge mistake to change how they handle the format strings now as 
they have operated they way they have since at least Python 1.5.2 (when I 
learned Python's C API).  There is just too much C code that would potentially 
break (third-party C extensions).

I think that a good change for the docs would be to separate the specification 
of the "building format string" away from the Py_BuildValue function so as to 
allow for differences in how the resulting object is created in those C 
functions that use that specification.  Similar, I suppose, to how the "parsing 
format string" is defined independently from PyArg_ParseTuple.

Now to the "attractive nuisance" that the single argument passed as varargs is 
handled.  I believe it may be best to introduce a new format character just for 
this purpose.  Possibly "T" (for tuple) or "V" (for varargs) using uppercase as 
that seems to be the practice for referencing objects.  And at the same time, 
add a check in the "call" functions (those that *use* Py_VaBuildValue to create 
a argument tuple, of which there are also internal ones).  The check could be 
as simple as:

  if (format[0] == 'O' && format[1] == '\0') {
va_start(va, format);
PyObject *ob = (PyObject *)va_arg(va, PyObject *);
if (ob != NULL) {
  if (PyTuple_Check(ob)) {
if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1,
"use 'V' format code to support varargs")) {
  args = NULL;
}
else {
  args = ob;
}
  }
  else {
args = PyTuple_Pack(1, ob);
  }
}
else if (!PyErr_Occurred()) {
  PyErr_SetString(PyExc_SystemError, "argument is NULL");
  args = NULL;
}
va_end(va);
  }
  else {
args = //...whatever happens now...
  }

--
nosy: +jkloth

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26800] Don't accept bytearray as filenames part 2

[STINNER Victor]

STINNER Victor added the comment:

bytearray is designed for performance. I don't think that the code
creating a filename can be a bottleneck in an application.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26800] Don't accept bytearray as filenames part 2

[Larry Hastings]

Larry Hastings added the comment:

I did the path_converter change.  IIRC some functions supported bytearray, some 
did not, and in my quest for consistency I took the superset of functionality 
and supported bytearray for everything.

Sounds to me like bytearray support should be dropped, but ultimately I don't 
have a strong opinion.  I'm happy to sit back and let the more opinionated 
decide the matter.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26601] Use new madvise()'s MADV_FREE on the private heap

[Julian Taylor]

Julian Taylor added the comment:

it defaulted to 128kb ten years ago, its a dynamic threshold since ages.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26601] Use new madvise()'s MADV_FREE on the private heap

[David Wilson]

David Wilson added the comment:

It defaults to 128kb, and messing with global state like the system allocator 
is a fine way to tempt regressions in third party code

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26601] Use new madvise()'s MADV_FREE on the private heap

[Julian Taylor]

Julian Taylor added the comment:

ARENA_SIZE is 256kb, the threshold in glibc is up to 32 MB

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26601] Use new madvise()'s MADV_FREE on the private heap

[David Wilson]

David Wilson added the comment:

@Julian note that ARENA_SIZE is double the threshold after which at least glibc 
resorts to calling mmap directly, so using malloc in place of mmap on at least 
Linux would have zero effect

--
nosy: +dw

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26821] array module "minimum size in bytes" table is wrong for int/long

[Jonathan Booth]

Jonathan Booth added the comment:

Ugly -- if I know I'm dealing with 4-byte data, I can't just specify 'I' or 'L' 
because it'll be wrong on some platform? Maybe the bug is really the module's 
design. Seems I need to look elsewhere for other reasons (array seems to want 
to copy memory, rather than sharing it, so I can't get two arrays one int, one 
byte with the same backing memory where changes to one effect the other), but 
that's all the more argument for me to switch off array anyway.

In any case, take it as the documentation wasn't particularly clear.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26601] Use new madvise()'s MADV_FREE on the private heap

[Julian Taylor]

Julian Taylor added the comment:

simplest way to fix this would be to not use malloc instead of mmap in the 
allocator, then you also get MADV_FREE for free when malloc uses it.
The rational for using mmap is kind of weak, the source just says "heap 
fragmentation". The usual argument for using mmap is not that but the instant 
return of memory to the system, quite the opposite of what the python memory 
pool does.

--
nosy: +jtaylor

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26822] itemgetter/attrgetter/methodcaller objects ignore keyword arguments

[Josh Rosenberg]

Josh Rosenberg added the comment:

Seems sensible for itemgetter and attrgetter, where all but the first argument 
is nonsensical anyway.

It really seems like methodcaller should allow additional arguments (positional 
and keyword though), a la functools.partial (the difference being the support 
for duck-typed methods, where functools.partial only supports unbound methods 
for a specific type). I suppose #25454 disagrees, but it seems very strange how 
`methodcaller` is like `functools.partial`, but without call time argument 
binding, only definition time.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26821] array module "minimum size in bytes" table is wrong for int/long

[random832]

random832 added the comment:

It says *minimum* size for a reason. The *actual* sizes of the types used in 
array are platform-dependent. 2 is the smallest that an int can be (it is 
probably not that size on any currently supported platforms, but would have 
been in DOS), and 4 is the smallest size a long can be (any 32-bit python 
build, and I think also any build on Windows, will have it be this size).

--
nosy: +random832

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16679] Add advice about non-ASCII wsgiref PATH_INFO

[Graham Dumpleton]

Graham Dumpleton added the comment:

Double back slashes would possibly be an artefact of the some mess that happens 
when logging out through the Apache error log.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26822] itemgetter/attrgetter/methodcaller objects ignore keyword arguments

[Serhiy Storchaka]

New submission from Serhiy Storchaka:

itemgetter(), attrgetter() and methodcaller() objects require one argument. 
They raise TypeError if less or more than one positional argument is provided. 
But they totally ignore any keyword arguments.

>>> import operator
>>> f = operator.itemgetter(1)
>>> f('abc', spam=3)
'b'
>>> f = operator.attrgetter('index')
>>> f('abc', spam=3)

>>> f = operator.methodcaller('upper')
>>> f('abc', spam=3)
'ABC'

Proposed patch makes these objects raise TypeError if keyword arguments are 
provided.

--
components: Extension Modules
files: operator_getters_kwargs.patch
keywords: patch
messages: 263933
nosy: serhiy.storchaka
priority: normal
severity: normal
stage: patch review
status: open
title: itemgetter/attrgetter/methodcaller objects ignore keyword arguments
type: behavior
versions: Python 2.7, Python 3.5, Python 3.6
Added file: http://bugs.python.org/file42560/operator_getters_kwargs.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26689] Add `has_flag` method to `distutils.CCompiler`

[Sylvain Corlay]

Sylvain Corlay added the comment:

Hey, any blocker to getting this in?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26821] array module "minimum size in bytes" table is wrong for int/long

[Jonathan Booth]

New submission from Jonathan Booth:

https://docs.python.org/3.5/library/array.html describes the 'I' and 'i' 
typecodes as being minimum-size in bytes of 2. The interpreter disagrees:

>>> import array
>>> a = array.array('i')
>>> a.itemsize
4

There is also a bug with the 'L' and 'l' long typecodes, which document as 
min-size of 4 bytes but are 8 bytes in the interpreter. That could be a bug in 
cPython itself though, as if 'L' should be 8 bytes, that disagrees with the 
type code sizing from the struct module, where it is 4 bytes, just like 
integers.

I checked documentation for all versions of python and it matches -- I did not 
check all python interpreters to see they match, but 2.7 and 3.5 did.

--
assignee: docs@python
components: Documentation
messages: 263931
nosy: Jonathan Booth, docs@python
priority: normal
severity: normal
status: open
title: array module "minimum size in bytes" table is wrong for int/long
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26820] Prevent uses of format string based PyObject_Call* that do not produce tuple required by docs

[Josh Rosenberg]

Josh Rosenberg added the comment:

The motivation for this change was Mr. STINNER's comment on #26814 ( 
https://bugs.python.org/issue26814#msg263923 ), where he mentioned the 
weirdness of PyObject_CallFunction and friends, which complicates the 
implementation of PyObject_FastCall and alerted me to a second case ( #21209 ) 
in which this silent fix up has caused confusing issues in CPython (I filed 
#26478 so I recognized the issue).

If this fix could be made, it might be possible to eventually make the check 
for non-tuple arguments a debug build only check (or a check only on public 
APIs), allowing the implementation in release mode and/or internal APIs to 
avoid the work involved in constantly checking for and performing this 
workaround to fix doc violating code, and possible simplify PyObject_FastCall 
by removing the corner case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26820] Prevent uses of format string based PyObject_Call* that do not produce tuple required by docs

[Josh Rosenberg]

New submission from Josh Rosenberg:

PyObject_CallMethod explicitly documents that "The C arguments are described by 
a Py_BuildValue() format string that should produce a tuple." While 
PyObject_CallFunction doesn't document this requirement, it has the same 
behavior, and the same failures, as does the undocumented 
_PyObject_CallMethodId.

The issue is that, should someone pass a format string of "O", then the type of 
the subsequent argument determines the effect in a non-obvious way; when the 
argument comes from a caller, and the intent was to pass a single argument, 
this means that if the caller passes a non-tuple sequence, everything works, 
while passing a tuple tries to pass the contents of the tuple as sequential 
arguments. This inconsistency was the cause of both #26478 and #21209 (maybe 
others).

Assuming the API can't/shouldn't be changed, it should still be an error when a 
format string of "O" is passed and the argument is a non-tuple (because you've 
violated the spec; the result of BuildValue was not a tuple). Instead 
call_function_tail is silently rewrapping non-tuple args in a single element 
tuple.

I'm proposing that, in debug builds (and ideally release builds too), 
abstract.c's call_function_tail treat the "non-tuple" case as an error, rather 
than rewrapping in a single element tuple. This still allows the use case where 
the function is used inefficiently, but correctly (where the format string is 
"O" and the value is *always* a tuple that's supposed to be varargs; it should 
really just use 
PyObject_CallFunctionObjArgs/PyObject_CallMethodObjArgs/PyObject_CallObject or 
Id based optimized versions, but it's legal). But it will make the majority of 
cases where a user provided argument could be tuple or not fail fast, rather 
than silently behave themselves *until* they receive a tuple and misbehave.

Downside: It will require code changes for cases like 
PyObject_CallFunction(foo, "k", myunsigned);, where there was no risk of 
misbehavior, but those cases were also violating the spec, and should be 
fixable by changing the format string to wrap the single value in parens, e.g. 
"(k)".

--
components: Interpreter Core
messages: 263929
nosy: haypo, josh.r
priority: normal
severity: normal
status: open
title: Prevent uses of format string based PyObject_Call* that do not produce 
tuple required by docs
versions: Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16679] Add advice about non-ASCII wsgiref PATH_INFO

[Andrew Clover]

Andrew Clover added the comment:

> Why only PATH_INFO is encoded in such a manner, but QUERY_STRING is passed 
> without any changes and does not requires any latin-1 to utf-8 recodings?

Laziness: QUERY_STRING should be pure-ASCII, making any such transcoding a 
no-op.

In principle a user agent *can* submit non-ASCII characters in a query string 
without %-encoding them, but it's not standards-conformant and most browsers 
don't usually do it (exception: apparently curl as above), so it's not worth 
adding a layer of hopefully-fixing-but-potentially-mangling to this variable to 
support a situation that shouldn't arise for normal requests.

PATH_INFO only requires special handling because of the sad, sad historical 
artefact of the CGI spec requiring it to have URL-decoding applied to it at the 
gateway, thus making the non-ASCII characters pop out of the percentage 
woodwork.

@Graham can you share more about how those test results were generated and 
displayed? The Gunicorn results are about what I would expect - the 
double-decoding of PATH_INFO is arguably undesirable when curl submits raw 
bytes, but ultimately that's an unspecified situation so I don't really case.

The output from Apache, on the other hand, is odd - something appears to have 
mangled the results at the reporting stage as not only is there double-decoding 
but also some double-backslashes. It looks like the strings have been put 
through ascii(repr()) or something?

--
nosy: +Andrew Clover

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26819] _ProactorReadPipeTransport pause_reading()/resume_reading() broken if called before any read is perfored

[Fulvio Esposito]

New submission from Fulvio Esposito:

Calling pause_reading()/resume_reading() on a _ProactorReadPipeTransport will 
result in an InvalidStateError('Result is not ready.') from a future if no read 
has been issued yet. The reason is that resume_reading() will schedule 
_loop_reading() a second time on the event loop. For example, currently 
aiomysql always fails to connect using a ProactorEventLoop on Windows because 
it calls pause_reading()/resume_reading() to set TCP_NODELAY on the socket just 
after connecting and before any read is performed.

--
components: asyncio
files: pause_resume_test.py
messages: 263927
nosy: Fulvio Esposito, gvanrossum, haypo, yselivanov
priority: normal
severity: normal
status: open
title: _ProactorReadPipeTransport pause_reading()/resume_reading() broken if 
called before any read is perfored
type: behavior
versions: Python 3.5
Added file: http://bugs.python.org/file42559/pause_resume_test.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] Add a new _PyObject_FastCall() function which avoids the creation of a tuple or dict for arguments

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

PyObject_Call*() implementations with _PyObject_FastCall() look much more 
complex than with PyObject_Call() (even not counting additional complex 
functions in modsupport.c). And I'm not sure there is a benefit. May be for 
first stage we can do without this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7275] CoverageResult fails to merge input file with non-empty callers in trace.py

[Berker Peksag]

Berker Peksag added the comment:

Thanks for the report and for the patch, Christian. Here is a patch with a test.

--
nosy: +berker.peksag
title: CoverageResult fails to merge input file with non-empty callers in 
trace.py (patch) -> CoverageResult fails to merge input file with non-empty 
callers in trace.py
versions: +Python 3.5, Python 3.6 -Python 2.7, Python 3.1, Python 3.2
Added file: http://bugs.python.org/file42558/issue7275.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] Add a new _PyObject_FastCall() function which avoids the creation of a tuple or dict for arguments

[STINNER Victor]

STINNER Victor added the comment:

> I believe I got a very simple version of it working at one point, supporting 
> positional parameters only, with some optional arguments.

Yeah, that would be a nice first step.

> p.s. My last name has two S's.  If you continue to leave off one of them, I 
> shall remove one from yours, Mr. TINNER.

Ooops, I'm sorry Guido Hastings :-(

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] Add a new _PyObject_FastCall() function which avoids the creation of a tuple or dict for arguments

[STINNER Victor]

STINNER Victor added the comment:

Since early microbenchmarks are promising, I wrote a more complete 
implementations which tries to use the fast-path (avoid temporary tuple/dict) 
in all PyObject_Call*() functions.

The next step would be to add a METH_FASTCALL flag. IMHO adding such new flag 
requires to enhance Argument Clinic to be able to use it, at least when a 
function doesn't accept keyword parameters.

PyObject_CallFunction() & friends have a weird API: if call with the format 
string "O", the behaviour depends if the object parameter is a tuple or not. If 
it's a tuple, the tuple is unpacked. It's a little bit weird. I recall that it 
led to a bug in the implementation in generators in Python: issue #21209! 
Moreover, if the format string is "(...)", parenthesis are ignored. If you want 
to call a function with one argument which is a tuple, you have to write 
"((...))". It's a little bit weird, but we cannot change that without breaking 
the (Python) world :-)

call_stack-3.patch:

* I renamed the main function to _PyObject_FastCall()
* I added PyObject_CallNoArg(): call a function with no parameter
* I added Py_VaBuildStack() and _Py_VaBuildStack_SizeT() helpers for 
PyObject_Call*() functions using a format string
* I renamed the new slot to tp_fastcall

Nice change in the WITH_CLEANUP_START opcode (ceval.c):

-/* XXX Not the fastest way to call it... */
-res = PyObject_CallFunctionObjArgs(exit_func, exc, val, tb, NULL);

+arg_stack[0] = exc;
+arg_stack[1] = val;
+arg_stack[2] = tb;
+res = _PyObject_FastCall(exit_func, arg_stack, 3, 0);

I don't know if it's a common byetcode, nor if the change is really faster.

--
title: Add a new _PyObject_CallStack() function which avoids the creation of a 
tuple or dict for arguments -> Add a new _PyObject_FastCall() function which 
avoids the creation of a tuple or dict for arguments
Added file: http://bugs.python.org/file42557/call_stack-3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26818] trace CLI doesn't respect -s option

[Berker Peksag]

New submission from Berker Peksag:

I noticed this while triaging issue 9317. Using traceme.py from that issue,

$ ./python -m trace -c -s traceme.py

returns nothing. It seems like it's a regression caused by 
https://github.com/python/cpython/commit/17e5da46a733a1a05072a277bc81ffa885f0c204

With trace_cli_summary.diff applied:

$ ./python -m trace -c -s traceme.py
lines   cov%   module   (path)
1   100%   trace   (/home/berker/projects/cpython/default/Lib/trace.py)
6   100%   traceme   (traceme.py)

trace_cli_summary.diff also fixes issue 10541.

--
components: Library (Lib)
files: trace_cli_cummary.diff
keywords: patch
messages: 263921
nosy: belopolsky, berker.peksag
priority: normal
severity: normal
stage: patch review
status: open
title: trace CLI doesn't respect -s option
type: behavior
versions: Python 3.5, Python 3.6
Added file: http://bugs.python.org/file42556/trace_cli_cummary.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9317] Incorrect coverage file from trace test_pickle.py

[Berker Peksag]

Berker Peksag added the comment:

A patch similar to issue9317.2.patch has been applied in 0aa46b9ffba3. However, 
I noticed a regression and created issue 26818. I can confirm that this issue 
is fixed with the patch from issue 26818 applied.

--
nosy: +berker.peksag
resolution:  -> fixed
stage: test needed -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] Add a new _PyObject_CallStack() function which avoids the creation of a tuple or dict for arguments

[Larry Hastings]

Larry Hastings added the comment:

Yes, I've been working on a patch to do this as well.  I called the calling 
convention METH_RAW, to go alongside METH_ZERO METH_O etc.  My calling 
convention was exactly the same as yours: PyObject *(PyObject *o, PyObject 
**stack, int na, int nk).  I only had to modify two functions in ceval.c to 
support it: ext_do_call() and call_function().

And yes, the overarching goal was to have Argument Clinic generate custom 
argument parsing code for every function.  Supporting the calling convention 
was the easy part; generating code was quite complicated.  I believe I got a 
very simple version of it working at one point, supporting positional 
parameters only, with some optional arguments.  Parsing arguments by hand gets 
very complicated indeed when you introduce keyword arguments.

I haven't touched this patch in most of a year.  I hope to return to it 
someday.  In the meantime it's fine by me if you add support for this and 
rewrite some functions by hand to use it.

p.s. My last name has two S's.  If you continue to leave off one of them, I 
shall remove one from yours, Mr. TINNER.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12743] C API marshalling doc contains XXX

[Berker Peksag]

Berker Peksag added the comment:

Since 4059e871e74e, PyMarshal_ReadLongFromFile and PyMarshal_ReadShortFromFile 
can return -1 on error. Return values of those functions were already 
documented in acb4d43955f6. Some of the usages also check return value of 
PyErr_Occurred(): https://hg.python.org/cpython/rev/11958c69a4b2#l2.7

I removed the outdated paragraph and add a sentence about using 
PyErr_Occurred().

--
keywords: +patch
nosy: +berker.peksag, haypo
stage: needs patch -> patch review
versions: +Python 3.5, Python 3.6
Added file: http://bugs.python.org/file42555/issue12743.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] Add a new _PyObject_CallStack() function which avoids the creation of a tuple or dict for arguments

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

With call_stack-2.patch attribute access in namedtuple is only 25% slower than 
attribute access in ordinary Python object! Definitely this this worth to 
continue to experiment!

But adding new slot to PyTypeObject sets the bar too high. Try to use your 
function to speed up all cases mentioned in issue23507: sorted()/list.sort(), 
min() and max() with the key argument, filter(), map(), some iterators from 
itertools (groupby(), dropwhile(), takewhile(), accumulate(), filterfalse()), 
thin wrappers around special method (round(), math.floor(), etc). Use it in 
wrappers around PyObject_Call() like PyObject_CallFunctionObjArgs(). May be 
this will cause an effect even on some macrobenchmarks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26817] Docs for StringIO should link to io.BytesIO

[Thomas Guettler]

New submission from Thomas Guettler:

I think a warning at the top of StringIO docs is needed.

And it should link to io.BytesIO.

Maybe even deprecate StringIO and cStringIO in Python2?

StringIO docs: https://docs.python.org/2/library/stringio.html

io.BytesIO docs: https://docs.python.org/2/library/io.html#io.BytesIO

I would like to see this at the top of StringIO:

{{{
Please use io.BytesIO and io.StringIO since this module is not supported in 
Python3
}}

--
assignee: docs@python
components: Documentation
messages: 263917
nosy: docs@python, guettli
priority: normal
severity: normal
status: open
title: Docs for StringIO should link to io.BytesIO
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26815] SIGBUS in test_ssl.test_dealloc_warn() on "AMD64 FreeBSD 10.0 3.x" buildbot

[STINNER Victor]

STINNER Victor added the comment:

Another different failure.

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.x/builds/4362/steps/test/logs/stdio

==
FAIL: test_refcycle (test.test_ssl.BasicSocketTests)
--
Traceback (most recent call last):
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_ssl.py", 
line 331, in test_refcycle
self.assertEqual(wr(), None)
AssertionError:  != None

--

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26816] Make concurrent.futures.Executor an abc

[Xiang Zhang]

Changes by Xiang Zhang :


--
type:  -> enhancement
versions: +Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Xavier de Gaye]

Xavier de Gaye added the comment:

_freeze_importlib and pgen are cross-built in this patch. Patch tested with a 
run of the testsuite after a cross-build.

--
Added file: http://bugs.python.org/file42554/crossbuild-sources-readonly_2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26816] Make concurrent.futures.Executor an abc

[Xiang Zhang]

Xiang Zhang added the comment:

Update the patch to remove more unnecessary base object.

--
Added file: 
http://bugs.python.org/file42553/make_concurrent_futures_Executor_an_abc_v2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26750] Mock autospec does not work with subclasses of property()

[Berker Peksag]

Changes by Berker Peksag :


--
components: +Library (Lib) -Tests
nosy: +berker.peksag
stage:  -> patch review
type: enhancement -> behavior
versions: +Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26803] syslog logging handler fails with address in unix abstract namespace

[Xavier de Gaye]

Xavier de Gaye added the comment:

In makesockaddr(), the current implementation does not do any decoding of 
AF_UNIX addresses in the abstract namespace in the struct sockaddr_un to bytes 
direction, i.e. system to python direction, but does encode a string or bytes 
object to struct sockaddr_un in the reverse direction, in getsockaddrarg(). 
This is not correct.

So the patch does more than fixing the type of the addresses as it also fixes 
this in a non backward compatible way. Maybe this should be indicated in 
Misc/NEWS.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26804] Prioritize lowercase proxy variables in urllib.request

[Hans-Peter Jansen]

Hans-Peter Jansen added the comment:

Here's the finalized version of this patch, including unit tests.

--
Added file: 
http://bugs.python.org/file42552/python-urllib-prefer-lowercase-proxies-v4.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26816] Make concurrent.futures.Executor an abc

[Xiang Zhang]

New submission from Xiang Zhang:

The documentation tells that concurrent.futures.Executor is an abstract class. 
Also PEP3148 tells so and says concurrent.futures.Executor.submit is an 
abstract method and must be implemented by Executor subclasses. I think using 
abc.ABCMeta here is a good choice.

I propose a patch. The patch also remove some unnecessary object base class.

--
files: make_concurrent_futures_Executor_an_abc.patch
keywords: patch
messages: 263911
nosy: bquinlan, xiang.zhang
priority: normal
severity: normal
status: open
title: Make concurrent.futures.Executor an abc
Added file: 
http://bugs.python.org/file42551/make_concurrent_futures_Executor_an_abc.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] Add a new _PyObject_CallStack() function which avoids the creation of a tuple or dict for arguments

[STINNER Victor]

STINNER Victor added the comment:

call_stack-2.patch: A little bit more complete patch, it adds a tp_call_stack 
field to PyTypeObject an use it in _PyObject_CallStack().

Updated microbenchmark on Python 3.6, best of 3 runs:

./python -m timeit -r 11 -s "from collections import namedtuple as n; a = 
n('n', 'a b c')(1, 2, 3)" -- "a.a; a.a; a.a; a.a; a.a; a.a; a.a; a.a; a.a; a.a; 
a.a; a.a; a.a; a.a; a.a; a.a; a.a; a.a; a.a"

* Python 3.6 unpatched: 0.968 usec
* call_stack.patch: 1.27 usec
* Python 3.6 with property_descr_get() of Python 3.4: 1.32 usec
* call_stack-2.patch: 0.664 usec

call_stack-2.patch makes this micro-benchmark 31% faster, not bad! It also 
makes calls to C functions almost 2x as fast if you replace current unoptimized 
calls with _PyObject_CallStack()!!

IHMO we should continue to experiment, making function calls 2x faster is worth 
it ;-)


Serhiy: "See also issue23507. May be your function help to optimize filter(), 
map(), sorted()?"

IMHO the API is generic enough to be usable in a lot of cases.


Serhiy: "Is there any use of this function with keyword arguments?"

Calling functions with keywords is probably the least common case for function 
calls in C code. But I would like to provide a fast function to call with 
keywords. Maybe we need two functions just to make the API cleaner? The 
difference would just be that "int k" would be omitted?

I proposed an API (PyObject **stack, int na, int nk) based on the current code 
in Python/ceval.c. I'm not sure that it's the best API ever :-)

In fact, there is already PyObject_CallFunctionObjArgs() which can be modified 
to reuse internally _PyObject_CallStack(), and its API is maybe more convenient 
than my proposed API.

--
Added file: http://bugs.python.org/file42550/call_stack-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] Add a new _PyObject_CallStack() function which avoids the creation of a tuple or dict for arguments

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

See also issue23507. May be your function help to optimize filter(), map(), 
sorted()?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] Add a new _PyObject_CallStack() function which avoids the creation of a tuple or dict for arguments

[STINNER Victor]

STINNER Victor added the comment:

Microbenchmark on Python 3.6, best of 3 runs:

./python -m timeit -r 11 -s "from collections import namedtuple as n; a = 
n('n', 'a b c')(1, 2, 3)" -- "a.a; a.a; a.a; a.a; a.a; a.a; a.a; a.a; a.a; a.a; 
a.a; a.a; a.a; a.a; a.a; a.a; a.a; a.a; a.a"

* Python 3.6 unpatched: 0.968 usec
* call_stack.patch: 1.27 usec
* Python 3.6 with property_descr_get() of Python 3.4: 1.32 usec

"Python 3.6 with property_descr_get() of Python 3.4": replace the current 
optimization with "return PyObject_CallFunctionObjArgs(gs->prop_get, obj, 
NULL);".

Oh, in fact the tested code calls a property where the final function is 
operator.itemgetter(0). _PyObject_CallStack() creates a temporary tuple to call 
PyObject_Call() which calls func->ob_type->tp_call, itemgetter_call().

Problem: tp_call API uses (PyObject *args, PyObject *kwargs). It doesn't accept 
directly a stack (a C array of PyObject*). And it may be more difficult to 
modify tp_call.

In short, my patch disables the optimization on property with my current 
incomplete implementation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] Add a new _PyObject_CallStack() function which avoids the creation of a tuple or dict for arguments

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

"Stack" in the function name looks a little confusing. I understand that this 
is related to the stack of bytecode interpreter, but this looks as raising 
pretty deep implementation detail. The way of packing positional and keyword 
arguments in the continuous array is not clear. Wouldn't be better to provide 
separate arguments for positional and keyword arguments?

What is the performance effect of using this function? For example compare the 
performance of namedtuple's attribute access of current code, the code with 
with this patch, and unoptimized code in 3.4:

./python -m timeit -r 11 -s "from collections import namedtuple as n; a = 
n('n', 'a b c')(1, 2, 3)" -- "a.a"

Is there any use of this function with keyword arguments?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26815] SIGBUS in test_ssl.test_dealloc_warn() on "AMD64 FreeBSD 10.0 3.x" buildbot

[STINNER Victor]

STINNER Victor added the comment:

The test:

def test_dealloc_warn(self):
ss = ssl.wrap_socket(socket.socket(socket.AF_INET))
r = repr(ss)
with self.assertWarns(ResourceWarning) as cm:
ss = None
support.gc_collect()   <~ SIGBUG occurred here
self.assertIn(r, str(cm.warning.args[0]))

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26815] SIGBUS in test_ssl.test_dealloc_warn() on "AMD64 FreeBSD 10.0 3.x" buildbot

[STINNER Victor]

New submission from STINNER Victor:

Oh oh, that's not good. test_ssl crashed in test_dealloc_warn() on the "AMD64 
FreeBSD 10.0 3.x" buildbot.

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.x/builds/4348/steps/test/logs/stdio

[ 39/400] test_ssl
Fatal Python error: Bus error

Current thread 0x000802006400 (most recent call first):
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/support/__init__.py",
 line 1444 in gc_collect
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_ssl.py", 
line 599 in test_dealloc_warn
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/case.py", 
line 600 in run
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/case.py", 
line 648 in __call__
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/suite.py", 
line 122 in run
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/suite.py", 
line 84 in __call__
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/suite.py", 
line 122 in run
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/suite.py", 
line 84 in __call__
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/unittest/runner.py", 
line 176 in run
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/support/__init__.py",
 line 1802 in _run_suite
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/support/__init__.py",
 line 1836 in run_unittest
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_ssl.py", 
line 3364 in test_main
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py",
 line 162 in runtest_inner
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/runtest.py",
 line 115 in runtest
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/runtest_mp.py",
 line 69 in run_tests_slave
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/main.py",
 line 379 in main
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/main.py",
 line 433 in main
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/main.py",
 line 455 in main_in_temp_cwd
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/regrtest.py", 
line 39 in 
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/runpy.py", line 
85 in _run_code
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/runpy.py", line 
184 in _run_module_as_main
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/runpy.py", line 
184, in _run_module_as_main
"__main__", mod_spec)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/runpy.py", line 
85, in _run_code
exec(code, run_globals)
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/__main__.py", 
line 3, in 
regrtest.main_in_temp_cwd()
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/main.py",
 line 455, in main_in_temp_cwd
main()
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/main.py",
 line 433, in main
Regrtest().main(tests=tests, **kwargs)
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/main.py",
 line 392, in main
self.run_tests()
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/main.py",
 line 354, in run_tests
run_tests_multiprocess(self)
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/libregrtest/runtest_mp.py",
 line 212, in run_tests_multiprocess
raise Exception(msg)
Exception: Child error on test_ssl: Exit code -10
*** Error code 1

--
messages: 263905
nosy: haypo
priority: normal
severity: normal
status: open
title: SIGBUS in test_ssl.test_dealloc_warn() on "AMD64 FreeBSD 10.0 3.x" 
buildbot
type: crash
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26811] segfault due to null pointer in tuple

[STINNER Victor]

STINNER Victor added the comment:

Oh, I see "The property() getter calls are up to 25% faster.
(Contributed by Joe Jevnik in issue 23910.)" in
https://docs.python.org/dev/whatsnew/3.5.html#optimizations

Hum... This is embarrassing :-/ Ok, let's keep it, but we must fix it ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26811] segfault due to null pointer in tuple

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> I suggest to remove the micro-optimization from Python 3.5 for safety.

Then we should remove the promise from What's New In Python 3.5. That will 
cause a regression in namedtuple attributes access about 30%.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Martin Panter]

Martin Panter added the comment:

For the record, to fix this in 2.7 will involve backporting revision 
c2a53aa27cad, plus Xavier’s patch (except the freeze_importlib parts which is 
not relevant to 2.7).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26799] gdb support fails with "Invalid cast."

[STINNER Victor]

STINNER Victor added the comment:

Great!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26811] segfault due to null pointer in tuple

[STINNER Victor]

STINNER Victor added the comment:

I created the issue #26814: "Add a new _PyObject_CallStack() function which 
avoids the creation of a tuple or dict for arguments".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] Add a new _PyObject_CallStack() function which avoids the creation of a tuple or dict for arguments

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +yselivanov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26814] Add a new _PyObject_CallStack() function which avoids the creation of a tuple or dict for arguments

[STINNER Victor]

New submission from STINNER Victor:

Attached patch adds the following new function:

   PyObject* _PyObject_CallStack(PyObject *func,
 PyObject **stack, 
 int na, int nk);

where na is the number of positional arguments and nk is the number of (key, 
pair) arguments stored in the stack.

Example of C code to call a function with one positional argument:

PyObject *stack[1];
stack[0] = arg;
return _PyObject_CallStack(func, stack, 1, 0);

Simple, isn't it?

The difference with PyObject_Call() is that its API avoids the creation of a 
tuple and a dictionary to pass parameters to functions when possible. 
Currently, the temporary tuple and dict can be avoided to call Python functions 
(nice, isn't it?) and C function declared with METH_O (not the most common API, 
but many functions are declared like that).

The patch only modifies property_descr_set() to test the feature, but I'm sure 
that *a lot* of C code can be modified to use this new function to beneift from 
its optimization.

Should we make this new _PyObject_CallStack() function official: call it 
PyObject_CallStack() (without the understand prefix) or experiment it in 
CPython 3.6 and decide later to make it public? If it's made private, it will 
require a large replacement patch later to replace all calls to 
_PyObject_CallStack() with PyObject_CallStack() (strip the underscore prefix).

The next step is to add a new METH_STACK flag to pass parameters to C functions 
using a similar API (PyObject **stack, int na, int nk) and modify the argument 
clinic to use this new API.

Thanks to Larry Hasting who gave me the idea in a previous edition of Pycon US 
;-)

This issue was created after the discussion on issue #26811 which is an issue 
in a micro-optimization in property_descr_set() to avoid the creation of a 
tuple: it caches a private tuple inside property_descr_set().

--
files: call_stack.patch
keywords: patch
messages: 263899
nosy: haypo, larry, rhettinger, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Add a new _PyObject_CallStack() function which avoids the creation of a 
tuple or dict for arguments
type: performance
versions: Python 3.6
Added file: http://bugs.python.org/file42549/call_stack.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Martin Panter]

Martin Panter added the comment:

I see. I guess it would keep the makefile simpler if we cross-compiled both 
programs, and never used them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26804] Prioritize lowercase proxy variables in urllib.request

[Hans-Peter Jansen]

Hans-Peter Jansen added the comment:

Here we go:

v3 fixes following issues:
 * prefer lowercase proxy environment settings, if multiple (disagreeing) 
settings are specified with differing case schemes (e.g. HTTP_PROXY vs. 
http_proxy)
 * an empty proxy variable value resets the related setting correctly 
 * apply this logic to no_proxy, too
 * document this behaviour 
 * fix misleading docstrings related to proxy_bypass

--
Added file: 
http://bugs.python.org/file42548/python-urllib-prefer-lowercase-proxies-v3.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1322] Deprecate platform.dist() and platform.linux_distribution() functions

[Andy Maier]

Andy Maier added the comment:

Just for completeness:

The "ld" package is now called "distro" and its v0.6.0 is on PyPI: 
https://pypi.python.org/pypi/distro

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26811] segfault due to null pointer in tuple

[STINNER Victor]

STINNER Victor added the comment:

I suggest to remove the micro-optimization from Python 3.5 for safety.

I'm ok to experiment a new safer implementation on Python 3.6 ;-) We have more 
time to fix the code in Python 3.6 if new issues are found. Setting the tuple 
size to zero looks simple and safe, but the overall hack deserves a comment to 
explain:

* why you use a cached tuple
* why the reference count can be different than 2: recursive calls
* why do you change the tuple size

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26811] segfault due to null pointer in tuple

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

References:

Issue23910 -- added original optimization (661cdbd617b8).
Issue24276 -- it was significant rewritten (5dbf3d932a59).

My suggestion in msg263886 to revert the optimization was related to the 
original code. Now, after reminding the background, I think it is worth to keep 
the optimization and try to fix the code.

Making the cached tuple to save None adds additional complexity and overhead 
(about 5%). Here is a patch that uses different trick. The size of saved tuple 
is set to 0.

--
stage:  -> patch review
type:  -> crash
Added file: 
http://bugs.python.org/file42547/property_cached_args_set_size_0.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26811] segfault due to null pointer in tuple

[STINNER Victor]

STINNER Victor added the comment:

> Maybe a new C function can be designed to call a function: it would uses 
> PyEval_EvalCodeEx() to call functions implemented in Python, or create the 
> tuple for functions implemented in C.

I would expect C code to be more optimized that Python functions, but for the 
specific case of function calls: Python are more optimized with the fast-path 
fast_function()!

I recall vaguely that Larry Hasting shared me his idea of passing the Python 
stack to C functions to avoid the creation of a tuple.

Maybe we can add a new API for C functions accepted a C array of PyObject*, a 
number of positional arguments and a number of (key, value) pairs for keyword 
arguments. Something similar to PyEval_EvalCodeEx() but for C functions. It 
means that we should add a new flavor of 
PyArg_ParseTuple/PyArg_ParseTupleAndKeywords to accepts this format.

Larry's idea was to use the argument clinic to handle that for us.

--
nosy: +larry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26811] segfault due to null pointer in tuple

[STINNER Victor]

STINNER Victor added the comment:

Serhiy: "Ah, yes, 5dbf3d932a59 partially fixed this optimization."

Since we already talking about hacks, another hack is to hide this private 
tuple from the GC using _PyObject_GC_UNTRACK(): see attached 
property_descr_get_gc_untrack.patch.

Note: I dislike my own patch :-D


Serhiy: "661cdbd617b8 can cause problems with recursion and with cached 
getters. I suggest to revert it."

Even if I don't understand the issue, I concur that the micro-optimization must 
be reverted. A micro-optimization must not introduce any behaviour change nor 
segfault :-)


Raymond Hettinger: "Serhiy, I'll thinking the whole optimization ought to be 
reverted as being too tricky and error-prone for the small benefit.  A less 
aggressive approach would be to have the tuple save a valid object (such as 
None) instead of NULL.  What do you think?"

PyEval_EvalCodeEx() accepts a C array of PyObject* for function arguments, it's 
used by fast_function() in Pythn/ceval.c. Maybe a new C function can be 
designed to call a function: it would uses PyEval_EvalCodeEx() to call 
functions implemented in Python, or create the tuple for functions implemented 
in C.

It looks like the issue #23910 which introduced the optimization was created 
for functions implemented in C. So I don't know it's acceptable. But it would 
be safer and more generic, no?

--
keywords: +patch
nosy: +haypo
Added file: http://bugs.python.org/file42546/property_descr_get_gc_untrack.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22359] Remove incorrect uses of recursive make

[Xavier de Gaye]

Xavier de Gaye added the comment:

> Why do you make the linking of _freeze_importlib conditional, but always 
> build $(PGEN)?

Yes, this is not consistent. The cross-build is correct when both are linked or 
when both are not linked. And $(PGENOBJS) are cross-compiled in both cases.

Should both programs not be created ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26799] gdb support fails with "Invalid cast."

[Thomas]

Thomas added the comment:

Thank you for the quick integration and fixing the return. I have signed the 
electronic form yesterday.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com