[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Terry J. Reedy]

Change by Terry J. Reedy :


--
versions: +Python 3.10, Python 3.9 -Python 2.7, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Mark Dickinson]

Mark Dickinson  added the comment:

> The other way to solve my problem would be to provide an operator module 
> function (operator.as_float?) that does a duck-typed conversion of an 
> arbitrary Python object to a float.

This does feel like the *right* solution to me. See #40801 and the linked PR. 
If we can do something like this, I'd be happy to drop the expectation that 
__float__ return something of exact type float, and similarly for __index__.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Mark Dickinson]

Mark Dickinson  added the comment:

[Serhiy]

> * Undeprecate accepting __index__ and __int__ returning instances of int 
> sublasses. There is no difference from the side of using int and index(), but 
> it can simplify user implementations of __index__ and __int__.

I'm not sure about this. Thinking about the bigger picture, we have a similar 
deprecation in place for __float__ returning an instance of a float subclass. 
That one I'd like to keep (and probably make an error for 3.10).

A problem I've run into in Real Code (TM) is needing to convert something 
float-like to a float, using the same mechanisms that (for example) something 
like `math.sqrt` uses.

One option is to call "float", but that requires explicitly excluding str, 
bytes and bytearray, which feels ugly and not very future-proof.

So the code ends up calling __float__. But because __float__ can return an 
instance of a float subclass, it then still needs some way to convert the 
return value to an actual float. And that's surprisingly tricky.

So I really *do* want to see the ability of __float__ to return a non-float 
eventually removed.

Similarly for __int__, there's no easy Python-side way to mimic the effect of 
calling __int__, followed by converting to an exact int. We have to:

1. Do an explicit check for non-numbers (str, bytes, bytearray)
2. Call int

Or:

1. Call __int__
2. Convert an instance of a possible subclass of int to something of exact type 
int. I don't know how to do this cleanly in general in Python, and end up 
resorting to evil tricks like adding `0`.

Deprecating allowing __int__ to return a non-int helps here, because it lets me 
simply call __int__.

I care much more about the __float__ case than the __int__ case, because the 
"right way" to duck-type integers is to use __index__ rather than __int__, and 
for __index__ we have operator.index as a solution.

But it would seem odd to have the rule in place for __float__ but not for 
__int__ and __index__.


The other way to solve my problem would be to provide an operator module 
function (operator.as_float?) that does a duck-typed conversion of an arbitrary 
Python object to a float.

--
nosy: +mark.dickinson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

The current status:

* Decimal and Fraction are no longer automatically converted to int when pass 
to functions implemented in C. PyLong_AsLong() etc no longer call __int__. (see 
issue36048 and issue37999)
* operator.index() and PyNumber_Index() always return an instance of exact type 
int. (see issue40792)
* int() and PyNumber_Long() always return an instance of exact type int. (see 
issue26984)
* __index__ is used as a fallback if __int__ is not defined. (see issue20092)

But:

* __index__ and __int__ are not called for int subclasses in operator.index() 
and int() (also in the C API PyNumber_Index(), PyNumber_Long(), 
PyLong_AsLong(), etc).
* Instances of int sublasses are accepted as result of __index__ and __int__ 
(but it is deprecated).
* The Python implementation of operator.index() differs from the C 
implementation in many ways. (see issue18712)

What I prefer as solutions of the remaining issues:

* It is good to not call __index__ and __int__ for int subclasses. __index__ 
and __int__ were   designed for converting non-integers to int. There are no 
good use cases for overriding __index__ and __int__ in int subclasses, and 
calling them is just a waste of time. We should just document this behavior.

* Undeprecate accepting __index__ and __int__ returning instances of int 
sublasses. There is no difference from the side of using int and index(), but 
it can simplify user implementations of __index__ and __int__.

* Either sync the pure Python implementation of operator.index() with the C 
implementation or get rid of Python implementation of the operator module at 
all.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[STINNER Victor]

STINNER Victor  added the comment:

It started to write a new issue, but then I found this issue issue (created in 
2013!) which is still open. So let me write my comment here instead.

The code to convert a number to an integer is quite complex in Python. There 
are *many* ways to do that and each way has subtle behavior differences (ex: 
__index__ vs __int__). Python tolerates some behavior which lead to even more 
confusion. For example, some functions explicitly reject the float type but 
accept Decimal:

* bpo-20861
* bpo-35707

PyLong_Long(obj) calls type(obj).__index__() if it's defined, but it accepts 
subtypes of int, not only exactly the int type (type(x) == int). This feature 
is deprecated since Python 3.3 (released in 2012), since this change:

commit 31a655411a79b00517cdcd0a2752824d183db792
Author: Serhiy Storchaka 
Date:   Wed Dec 11 21:07:54 2013 +0200

Issue #17576: Deprecation warning emitted now when __int__() or __index__()
return not int instance.  Introduced _PyLong_FromNbInt() and refactored
PyLong_As*() functions.

I propose to now fail with an exception if __int__() or __index__() return type 
is not exactly int.

Note: My notes on Python numbers: https://pythondev.readthedocs.io/numbers.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Mark, I think you can reopen the PR and merge it in 3.9 now.

As for my proposition to use the FutureWarning first, I think it is not 
necessary. The behavior change is very subtle and will affects only int 
subclasses with overridden __index__. Similar changes (preferring __index__ 
over __int__) have been made in 3.8 without preceding FutureWarning. And 
similar minor changes were made in the past.

On other hand, I am not sure that __index__ should be used for int subclasses. 
We already have the int content, so we can create an exact int with 
_PyLong_Copy().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Mark Dickinson]

Mark Dickinson  added the comment:

I've closed the PR. Reassigning back to Ethan.

--
assignee: mark.dickinson -> ethan.furman
nosy: +ethan.furman

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Mark Dickinson]

Change by Mark Dickinson :


--
nosy:  -mark.dickinson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

> Can we at least switch to PyLong_CheckExact?

This is a behavior change and as such should be preceded by a period of warning.

If we go this way I propose to add a FutureWarning for int subclasses with 
overridden __index__.

As for turning the deprecated behaviour into TypeErrors, we added yet few 
deprecation warnings in 3.8. Would not be better to turn all of them into 
TypeErrors at the same time?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Mark Dickinson]

Mark Dickinson  added the comment:

> Can we at least switch to PyLong_CheckExact?

+1

> I am not sure that converting to an exact int in low-level C API functions is 
> the best option.

I am sure. :-)  The number of naturally-occurring cases where we're actually 
passing a subtype of `int` that's not exactly `int` should be tiny. So long as 
there's a PyLong_CheckExact fast path, I don't think there are really any 
performance concerns here.

And we definitely shouldn't let performance concerns dictate API; get the API 
right first, _then_ see what can be done about performance without changing the 
API. It's clear to me that `operator.index(obj)` _should_ give the exact same 
results as `obj.__index__()`.

I'll split my PR up into two pieces, one for turning the deprecated behaviour 
into TypeErrors, and a second one that just makes the PyLong_CheckExact change. 
(I likely won't have time before feature freeze, though. OTOH, the 
PyLong_CheckExact change could be considered a bugfix.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Can we at least switch to PyLong_CheckExact?  That would fix Barry's original 
issue and should run slightly faster.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I am not sure that raising an error is the best option. We can just convert an 
integer subclass to an exact int using _PyLong_Copy().

I am not sure that converting to an exact int in low-level C API functions is 
the best option. In many cases we use only the content of the resulting object 
ignoring its type (when convert it to the C integer or float, to bytes array, 
to new instance of int subclass). Creating a new exact int is a waste of time.

This is why I withdrawn my patches and this issue is still open.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Mark Dickinson]

Change by Mark Dickinson :


--
pull_requests: +13623
pull_request: https://github.com/python/cpython/pull/13740

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Mark Dickinson]

Change by Mark Dickinson :


--
assignee: ethan.furman -> mark.dickinson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Mark Dickinson]

Mark Dickinson  added the comment:

I'm working on a PR that finally changes the DeprecationWarnings that Serhiy 
introduced to TypeErrors; I think that should be acceptable, four Python 
versions and some years later. With that PR:

- int will always return something of exact type `int` (or raise)
- operator.index will always return something of exact type `int` (or raise)
- PyNumber_Index will always use `__index__` for int subclasses, so this should 
fix the issue that Barry originally reported (mismatch between 
`obj.__index__()` and `operator.index(obj)`).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

See also issue33039.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Ethan Furman]

Changes by Ethan Furman et...@stoneleaf.us:


--
nosy:  -ethan.furman

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Nick Coghlan]

Nick Coghlan added the comment:

OK, something appears to have gotten confused along the way here. Barry's 
original problem report was that operator.index() was returning a different 
answer than operator.__index__() for int subclasses. Absolutely nothing to do 
with the int builtin at all. While the fact int() *may* return int subclasses 
isn't especially good, it's also a longstanding behaviour.

The problem Barry reports, where a subclassing based proxy type isn't reverting 
to a normal integer when accessed via operator.index() despite defining 
__index__() to do exactly that should be possible to fix just by applying the 
stricter check specifically in PyNumber_Index.

Expanding the scope to cover __int__() and __trunc__() as well would be much, 
much hairier, as those are much older interfaces, and used in a wider variety 
of situations. We specifically invented __index__() to stay away from that mess 
while making it possible to explicitly indicate that a type supports a lossless 
conversion to int rather than a potentially lossy one.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The tests are checking that they are the same value (8) and the same type (int)?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Manuel Jacob]

Manuel Jacob added the comment:

Maybe I'm missing something, but it seems to me that 
test_int_subclass_with_index() is testing for the exactly wrong behaviour.  
Isn't the point of this issue that operator.index(a) should be equal to 
a.__index__()?  Why are the tests checking that they are different?

--
nosy: +mjacob

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Ah, it just checks current behavior. So we will know when this will be changed.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Ethan Furman]

Changes by Ethan Furman et...@stoneleaf.us:


--
assignee:  - ethan.furman

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I have doubts about this issue, so I have unassigned it from myself.

--
assignee: serhiy.storchaka - 

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Roundup Robot]

Roundup Robot added the comment:

New changeset a3de2b3881c1 by Serhiy Storchaka in branch '3.3':
Issue #17576: Removed deprecation warnings added in changeset 618cca51a27e.
http://hg.python.org/cpython/rev/a3de2b3881c1

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

PyLong_AsUnsignedLong() and PyLong_AsLong() can return different values for 
same object. Why __int__ and __index__ should be used for int subclasses at 
all? I'm not sure this is right solution.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Roundup Robot]

Roundup Robot added the comment:

New changeset 618cca51a27e by Serhiy Storchaka in branch '3.3':
Issue #17576: Deprecation warning emitted now when __int__() or __index__()
http://hg.python.org/cpython/rev/618cca51a27e

New changeset 59fb79d0411e by Serhiy Storchaka in branch 'default':
Issue #17576: Deprecation warning emitted now when __int__() or __index__()
http://hg.python.org/cpython/rev/59fb79d0411e

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I first committed safe part of patch, which doesn't change behavior, only adds 
warnings and tests. Here is other part (very simple), which change behavior (in 
sum they are equal to issue17576_v3.patch with minor changes).

* PyNumber_Index() now calls __index__() for int subclasses.

 class A(int):
... def __index__(self): return 1
... 
 import operator
 operator.index(A())

Returns 0 without patch and 1 with patch.

* PyNumber_Long() now calls __int__() on result of __trunc__() if it is int 
subclass instance.

 class A:
... def __trunc__(self): return True
... 
 int(A())

Returns True without patch and 1 with patch.

* PyLong_As*() functions (but not all) call __int__() for int subclasses (I'm 
not sure this is right).

 class A(int):
... def __int__(self): return 42
... 
 chr(A(43))

Returns '+' without patch and '*' with patch.

--
Added file: http://bugs.python.org/file33093/issue17576_v4.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Even with last patch int() can return non-int:

 class A(int):
... def __int__(self): return True
... def __repr__(self): return 'A'
... 
 class B:
... def __int__(self): return A()
... 
 class C:
... def __trunc__(self): return A()
... 
 int(B())
A
 int(C())
True

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Here is updated patch. There is no more overhead in PyLong_As* functions. 
Simplified PyNumber_Index(). assertWarns() now used instead of 
support.check_warnings(). Added new tests.

--
Added file: http://bugs.python.org/file33077/issue17576_v3.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Nick Coghlan]

Nick Coghlan added the comment:

Took me a while to figure out that one of the code paths was being deleted as 
redundant because the type machinery will always fill in nb_int for int 
subclasses, but Serhiy's patch looks good to me.

--
assignee: mark.dickinson - serhiy.storchaka

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Mark Dickinson]

Mark Dickinson added the comment:

Thanks, Serhiy.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

What about PyLong_AsSsize_t(), PyLong_AsUnsignedLong(), and PyLong_AsSize_t()? 
They are ignore __int__().

PyLong_AsVoidPtr() calls PyLong_AsLong(), PyLong_AsUnsignedLong(), 
PyLong_AsLongLong(), or PyLong_AsUnsignedLongLong() (depending on pointer's 
size and it's sign) and therefore can call or not call __int__, can raise or 
not raise TypeError on non-int subclasses with defined __int__().

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Nick Coghlan]

Nick Coghlan added the comment:

The ssize_t functions deliberately ignore lossy int conversions (e.g. from
floats) - that's why they only work on types that implement __index__.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

 The ssize_t functions deliberately ignore lossy int conversions (e.g. from
 floats) - that's why they only work on types that implement __index__.

They ignore __index__.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Mark Dickinson]

Mark Dickinson added the comment:

Yes, the PyLong_As... conversions are a mess.  In theory, they should all use 
__index__ if available, and ignore __int__.  In practice, it's a mess that's 
difficult to change without breaking things.  I think there are already some 
other issues open on this subject.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Ping.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Mark Dickinson]

Mark Dickinson added the comment:

 Ping.

Bah.  Sorry; I haven't had time to deal with this.  Serhiy: are you interested 
in taking over?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Mark Dickinson]

Mark Dickinson added the comment:

I still need to act on some of Serhiy's comments.  I do plan to get this in for 
3.4.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Ethan Furman]

Ethan Furman added the comment:

Where do we stand with this issue?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Yet some nitpicks.

Currently the code of _PyLong_FromNbInt() is inlined and the do_decref flag is 
used to prevent needless change refcounts of int objects (see also issue18797). 
In proposed patch common code is extracted into the _PyLong_FromNbInt() 
function and int objects increfed and decrefed. Doesn't it affect a 
performance? PyLong_As* functions used in arguments parsing in a lot of 
builtins and their .performance is important.

If the patch slowdowns PyLong_As* functions we perhaps should check 
PyLong_CheckExact() before calling _PyLong_FromNbInt() and use the do_decref 
flag.

In general the idea and the patch LGTM.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

On PyPy 1.8.0 operator.index(True) returns 1.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

And yet one nitpick. For int subclasses which doesn't overload the __int__ 
method the patch calls default int.__int__ which creates a copy of int object. 
This is redundant in PyLong_As* functions because they only extract C int value 
and drop Python int object. So we can use int subclass object itself as good as 
int object.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Nick Coghlan]

Nick Coghlan added the comment:

On 21 Aug 2013 15:47, Mark Dickinson rep...@bugs.python.org wrote:


 Mark Dickinson added the comment:

  Shouldn't it be PendingDeprecationWarning?

 Hmm.  Possibly.  I'm not sure what the policy is any more regarding
DeprecationWarning versus PendingDeprecationWarning.  Nick?

Not sure if this is written down anywhere, but I use
PendingDeprecationWarning for definitely still around next release, may
not have a set date for removal and DeprecationWarning for may be removed
next release.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Changes by Serhiy Storchaka storch...@gmail.com:


--
stage:  - patch review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Shouldn't it be PendingDeprecationWarning?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Mark Dickinson]

Mark Dickinson added the comment:

 Shouldn't it be PendingDeprecationWarning?

Hmm.  Possibly.  I'm not sure what the policy is any more regarding 
DeprecationWarning versus PendingDeprecationWarning.  Nick?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Mark Dickinson]

Mark Dickinson added the comment:

See the related python-dev discussion started by Mark Shannon here:

http://mail.python.org/pipermail/python-dev/2013-March/125022.html

and continuing well into April here:

http://mail.python.org/pipermail/python-dev/2013-April/125042.html

The consensus that emerged from that thread seems to be that calls to 
operator.index and to int() should always return something of exact type int.

The attached patch:

- Raises TypeError for implicit calls to nb_int that fail to return something 
of exact type int.  (Results of direct calls to __int__ are not checked.)

- Ensures that *all* conversions from a non-int to an int via nb_int make use 
of the nb_int slot, even for int subclasses.  Prior to this patch, some of the 
PyLong_As... functions would bypass __int__ for int subclasses.

- Adds a new private _PyLong_FromNbInt function to Objects/longobject.c, so 
that we have a single place for performing these conversions and making type 
checks, and refactors existing uses of the nb_int slot to go via this function.

- Makes corresponding changes for nb_index, which should address the original 
bug report.

I guess this may be too dangerous a change for Python 3.4.  In that case, I 
propose raising warnings instead of TypeErrors for Python 3.4 and turning those 
into TypeErrors in Python 3.5.

One other question:  should direct calls to __int__ and __index__ also have 
their return values type-checked?  That doesn't seem to happen at the moment 
for other magic methods (see below), so it would seem consistent to only do the 
type checking for interpreter-generated implicit calls to __int__ and 
__index__.  Nick: any opinion?

 class A:
... def __len__(self): return a string
... def __bool__(self): return another string
... 
 a = A()
 a.__len__()
'a string'
 a.__bool__()
'another string'
 len(a)
Traceback (most recent call last):
  File stdin, line 1, in module
TypeError: 'str' object cannot be interpreted as an integer
 bool(a)
Traceback (most recent call last):
  File stdin, line 1, in module
TypeError: __bool__ should return bool, returned str

--
assignee: docs@python - mark.dickinson
components: +Interpreter Core -Documentation
keywords: +patch
nosy: +ncoghlan
Added file: http://bugs.python.org/file31147/issue17576.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Mark Dickinson]

Mark Dickinson added the comment:

New patch that replaces the TypeErrors with warnings and fixes a refleak in the 
original patch.

--
Added file: http://bugs.python.org/file31149/issue17576_v2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Nick Coghlan]

Nick Coghlan added the comment:

The deprecation warning version looks good to me.

Something I'll mention explicitly (regarding the PyCon discussions that Eric 
mentioned above), is that we unfortunately couldn't do something like this for 
the various concrete APIs with overly permissive subclass checks. For those 
APIs, calling them directly was often the *right* thing for simple subtypes 
implemented in C to use to call up to the parent implementation.

This case is different, as it's the *abstract* APIs that currently have the 
overly permissive checks.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

if (PyLong_CheckExact(item) || (PyLong_Check(item) 
item-ob_type-tp_as_number-nb_index == 
PyLong_Type.tp_as_number-nb_index))

--
nosy: +serhiy.storchaka

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[STINNER Victor]

STINNER Victor added the comment:

Alex In my opinion that should use PyLong_CheckExact

+1

--
nosy: +haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Terry J. Reedy]

Changes by Terry J. Reedy tjre...@udel.edu:


--
versions:  -Python 3.1, Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Ethan Furman]

Changes by Ethan Furman et...@stoneleaf.us:


--
nosy: +stoneleaf

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Mark Dickinson]

Changes by Mark Dickinson dicki...@gmail.com:


--
nosy: +mark.dickinson

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

You also end up with this nice bit of inconsistency:

 x = myint(7)
 from operator import index
 range(10)[6:x]
range(6, 7)
 range(10)[6:x.__index__()]
range(6, 8)
 range(10)[6:index(x)]
range(6, 7)
 

Granted, it's insane to have __index__() return a different value like this, 
but in my specific use case, it's the type of object returned from 
operator.index() that's the problem.  operator.index() returns the subclass 
instance while obj.__index__() returns the int.

(The use case is the IntEnum of PEP 435.)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Barry A. Warsaw]

Changes by Barry A. Warsaw ba...@python.org:


--
title: PyNumber_Index() is not int-subclass friendly - PyNumber_Index() is not 
int-subclass friendly (or operator.index() docos lie)

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Eric Snow]

Eric Snow added the comment:

Would it be okay to do a check on __index__ after the PyLong_Check() succeeds?  
Something like this:

if (PyLong_Check(item) 
item-ob_type-tp_as_number-nb_index == 
PyLong_Type.tp_as_number-nb_index) {
Py_INCREF(item);
return item;
}

This is something Nick and I were talking about at the sprints regarding fast 
paths in the abstract API (for mappings and sequences in our case).

--
nosy: +eric.snow

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Alex Gaynor]

Alex Gaynor added the comment:

In my opinion that should use PyLong_CheckExact

--
nosy: +alex

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17576] PyNumber_Index() is not int-subclass friendly (or operator.index() docos lie)

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

On Mar 30, 2013, at 12:29 AM, Eric Snow wrote:

Would it be okay to do a check on __index__ after the PyLong_Check()
succeeds?  Something like this:

if (PyLong_Check(item) 
item-ob_type-tp_as_number-nb_index == 
 PyLong_Type.tp_as_number-nb_index) {
Py_INCREF(item);
return item;
}

This is something Nick and I were talking about at the sprints regarding fast
paths in the abstract API (for mappings and sequences in our case).

I think that would work, yes.  With this extra check, overriding __index__()
in the subclass should fail this condition and fall back to the
PyIndex_Check() clause.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17576
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com