[issue26906] format(object.__reduce__) fails intermittently

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Yet one demonstration of this bug:

$ ./python -IS
>>> import operator
>>> operator.length_hint(iter("abc"))
0
>>> import collections.abc
>>> operator.length_hint(iter("abc"))
3

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Guido van Rossum]

Guido van Rossum added the comment:

Serhiy -- please do what do you think we should do. At this point I'm open to 
just about anything, but I don't feel comfortable creating or reviewing patches 
any more.

--
assignee: gvanrossum -> serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Yet one similar bug: issue11702.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Similar bug just was introduced in issue21124.

--
priority: normal -> high
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Antti Haapala]

Antti Haapala added the comment:

Could it be possible to to make the debug build absolutely abort on any usage 
of PyType's that are not readied, usage including instantiating them. Then, 
instead of changing all `static` linkages to `extern`s (as in Serhiy's first 
patch) one could rather make per-compilation unit initialization functions that 
are called from objects.c; that way it would be easier to use preprocessor to 
turn on and off the very existence of certain types in a compilation unit based 
on a preprocessor flag.

Likewise the C-API docs for PyType_Ready should perhaps say "This must be 
called on all type objects to finish their initialization." instead of "should"

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Guido van Rossum]

Guido van Rossum added the comment:

Because the data structure that defines a type is just data, and at some
point PyType_Ready() must be called. The question is how to do this, given
that nobody can (or needs to) produce a definitive list of all types.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Antti Haapala]

Antti Haapala added the comment:

I am not an expert on PyType internals, so I am wondering why is the 
PyType_Ready'ing done implicitly at all?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Guido van Rossum]

Guido van Rossum added the comment:

Probably.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

There is one test 
(ClassPropertiesAndMethods.test_mutable_bases_with_failing_mro in test_descr) 
that crashes with the code from issue551412 because _PyType_Lookup() is 
recursive called from PyType_Ready(). Is this the reason? My patch prevents 
recursive calls.

Here is minimal example (for Python 3):

class M(type):
def mro(self):
hasattr(self, 'foo')
return type.mro(self)

class C(metaclass=M):
pass

When class C is created, C.mro() is called while C still is not ready. 
Resolving an attribute calls _PyType_Lookup() which calls PyType_Ready() which 
calls mro() etc.

--
keywords:  -patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Guido van Rossum]

Guido van Rossum added the comment:

But the problem isn't limited to format()... Why would format() be special?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Terry J. Reedy]

Terry J. Reedy added the comment:

Is there a way to have format() try to force the initialization, by explicitly 
doing the equivalent of obj.__format__, at least for types, instead of raising 
the TypeError?

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Guido van Rossum]

Guido van Rossum added the comment:

Sadly it's been a very long time since I wrote that code and I don't recall
much about it. I presume there was a good reason for not to do it in
_PyType_Lookup(), but who knows -- maybe the oroginal approach was just too
naive and nobody cared? I'm not excited by a patch that does this for 38
types -- invariably there will be another type that still surfaces the same
bug.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Antti Haapala]

Antti Haapala added the comment:

And to the other things failing, I was trying to find out which of the magic 
method ops fail, and for that tried to find out the `dir` of list iterator. 
Well...

% python3.5 -S  
Python 3.5.0+ (default, Oct 11 2015, 09:05:38) 
[GCC 5.2.1 20151010] on linux
>>> dir(iter([]))
Traceback (most recent call last):
  File "", line 1, in 
TypeError: object does not provide __dir__

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Antti Haapala]

Antti Haapala added the comment:

I can reproduce the bug in 3.5.0+ Ubuntu with list iterator, if I execute 
python with -S:

% python3.5 -S
Python 3.5.0+ (default, Oct 11 2015, 09:05:38) 
[GCC 5.2.1 20151010] on linux
>>> format(iter([]))
Traceback (most recent call last):
  File "", line 1, in 
TypeError: Type list_iterator doesn't define __format__

Thus here it depends on the stuff that site does or doesn't do. Iterating over 
a list iterator does *not* trigger the initialization. Printing it doesn't help 
either, or anything else that does not touch the non-magic attributes. I am not 
even sure what the site.py and such are doing to the list iterator class to 
trigger the initialization.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The problem is that format() fails for instances of some classes, because the 
type still is not initialized. The simplest example -- list iterator.

>>> format(iter([]))
Traceback (most recent call last):
  File "", line 1, in 
TypeError: Type listiterator doesn't define __format__

After forcing type initialization (for example by getting any type's 
attribute), format() becomes working.

>>> type(iter([])).foo
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: type object 'listiterator' has no attribute 'foo'
>>> format(iter([]))
''

I afraid that format() is just one example, and there are other functions or 
operators that don't work or work incorrectly if the type was not initialized.

init_types-2.7.patch adds explicit initialization of 38 types (I didn't check 
that all of them need this, but I suppose they do). This is large patch, and 
I'm not sure that it fixes all types.

Other way is to try to initialize the type just in _PyType_Lookup if it is not 
initialized. This is simple change, but a comment in _PyType_Lookup warns me. I 
found that this solution was already applied as an attempt to fix issue551412, 
but then reverted. Since you seem to be the most knowledgeable with this code, 
I'm asking you what was wrong with this approach and how we can fix this.

Python 3.x also suffers from this bug, but it is reproduced with less types. 
For example it isn't reproduced for list iterator. I don't know why.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Added a check for Py_TPFLAGS_READYING to prevent recursive calling.

--
Added file: http://bugs.python.org/file42718/init_type_in_pytype_lookup.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


Removed file: http://bugs.python.org/file42717/init_type_in_pytype_lookup.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Guido van Rossum]

Guido van Rossum added the comment:

Serhiy, I'm happy to help, but I'm not sure what you're asking me to do. Decide 
between different patches? I can't even repro the issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

An alternative way is just call PyType_Ready from _PyType_Lookup if 
type->tp_mro is NULL.

Here is a patch against 2.7 that restores the solution from issue551412, but 
returns NULL if type->tp_mro is still NULL after calling PyType_Ready. I found 
one place in tests when this is happened (CIOTest.test_IOBase_finalize in 
test_io).

--
assignee:  -> gvanrossum
nosy: +gvanrossum
Added file: http://bugs.python.org/file42717/init_type_in_pytype_lookup.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

A number of other types are not initialized until you request an attribute.

Here is larger patch for 2.7 that makes 38 types to be explicitly initialized.

--
Added file: http://bugs.python.org/file42716/init_types-2.7.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

There is similar issue on 3.x:

>>> import array
>>> it = iter(array.array('i'))
>>> format(it)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: Type arrayiterator doesn't define __format__
>>> type(it).__format__

>>> format(it)
''

--
versions: +Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Proposed patch makes method descriptors types to be explicitly initialized as 
in 3.x.

--
components: +Interpreter Core
keywords: +patch
nosy: +serhiy.storchaka
stage:  -> patch review
Added file: http://bugs.python.org/file42685/init_method_descr_types.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Eric V. Smith]

Changes by Eric V. Smith :


--
nosy: +eric.smith
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Antti Haapala]

Antti Haapala added the comment:

s/explicitly do/explicitly access/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26906] format(object.__reduce__) fails intermittently

[Antti Haapala]

New submission from Antti Haapala:

This is an annoying heisenbug; it seems that some objects cannot be formatted 
until you explicitly do obj.__format__. For example `object.__reduce__` behaves 
like this:

Python 2.7.10 (default, Oct 14 2015, 16:09:02)
[GCC 5.2.1 20151010] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> format(object.__reduce__)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: Type method_descriptor doesn't define __format__
>>> format(object.__reduce__)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: Type method_descriptor doesn't define __format__
>>> object.__reduce__.__format__

>>> format(object.__reduce__)
""

I can replicate this in 2.7.9, .10 and .11 on Ubuntu and Debian, though it 
works on Windows Python, works in 2.6.6, and Pythons 3 wherever I've tried, but 
I've heard this also failing on Python 3.

--
title: __reduce__ format -> format(object.__reduce__) fails intermittently

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com