[issue21919] Changing cls.__bases__ must ensure proper metaclass inheritance
Eldar Abusalimov added the comment: @Mark, that OK, for the issue without a patch. :) I could make a patch, but I'm not sure whether the proposed behavior is right. It could be considered arguable, I guess... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21919 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Eldar Abusalimov added the comment: You're welcome, and thank you too for reviewing and merging it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Eldar Abusalimov added the comment: I feel a bit uneasy, but... any news on this? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Eldar Abusalimov added the comment: ping? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Eldar Abusalimov added the comment: Thank you for your replies. I don't think forbidding reentrancy is a good idea, and I'm against it for the following reasons. First of all, naive prohibition of type_set_bases within mro() on a class doesn't save from mro() reentrancy: def mro(cls): # cls is B, # B extends A A.__bases__ = ... # What to do? Changing bases of a class provokes MRO of all its subclasses to be updated as well. Therefore, one would also need to forbid changing __bases__ of any direct or indirect base of the class up to the 'object', and this doesn't feel natural at all. Why should mro() of some specialized class B prevent changing bases of its general parent class A? Otherwise, there must be a check in mro_subclasses (mro_hierarchy in the patch), which will not recurse into a class flagged as being inside mro(). But in this case a running mro(B) will be unable to notice a change of MRO of the parent class A, in case if mro() involves some non-trivial logic, and A.__bases__ assignment occurs somewhere deep in the call stack and is done by a code from some unknown black-box module. Deferring mro() recalculation upon exiting from the outermost mro() in such case doesn't seem to be a good solution either, especially concerning its (most likely) tricky implementation. def mro(cls): # some complicated calculation based of MROs of bases: parent_mros = [base.__mro__ for base in cls.__bases__] # The line below does this: # parent.__bases__ = ... # (deep-deep-deep in the call stack) third_party.do_mro(cls) # Either the line above raises an error, # or parent_mros becomes invalid. Another example. Setting __bases__ from inside mro() may be useful for changing class behavior on the fly using a proxy base class (e.g. for debugging, logging, testing frameworks). The only place to do this except mro() is __new__, but in that case it is only possible to fix up bases at the moment of class creation. That is, a tricky class that change its __bases__ would break such framework by overriding a proxy base, which is necessary for this framework to function properly. def mro(cls): if cls or one of cls.__bases__ is not a proxy class: class Proxy(*cls.__bases__): ... cls.__bases__ = (Proxy,) # reenter return type.mro(cls) In other words, there should be a loophole to alter __bases__ upon assignment, and I suppose mro() is a good option. Also, __bases__ assignment is known as the only reliable way to invoke MRO recalculation (http://stackoverflow.com/a/20832588/545027). Forbidding it from the inside mro() makes it not so obvious and reliable. Actually, I encountered one of these bugs while working on a real metaclass providing an auto-inheriting attributes feature needed for a custom DSL built on top of Python. In a nutshell, if B extends A, then make B.V extend A.V automatically (https://github.com/abusalimov/mybuild/blob/6c7c89521b856c798b46732501adb5e06dec7e03/util/inherit.py, still work in progress). I know, some of these use cases may sound a bit unreal (or even crazy), but neither me nor you can imagine all scenarios, that Python programmers could ever invent. Actually, there could already exist some: it is possible to workaround most of reentrancy issues through holding a reference to old_mro from inside a Python code. That is, backward compatibility comes into play too. Finally, why do forbid something that was not prohibited before? I think of it as a feature with some issues to fix, not to remove at all. After all, there is a fix provided, it is more or less straightforward (I hope so), with tests showing a desired behavior. The desired behavior is also more or less obvious: take a __mro__ calculated by mro() invoked due to the very last __bases__ assignment, regardless reentrancy (the innermost one in such case). Summarizing, that is why I believe that reentrancy should not be forbidden. Furthermore, I considered that way, and I'm pretty sure it is a wrong one. It implies too many corner cases, it has a non-obvious behavior from the point of view of a Python code, and a possible implementation doesn't seem to be more simple or robust than it is now. I would be happy to hear opposite arguments, and if I convinced you, get a feedback on the patch, of course. I could also prepare a backport patch fixing 2.7 line as well. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20440] Use Py_REPLACE/Py_XREPLACE macros
Changes by Eldar Abusalimov eldar.abusali...@gmail.com: -- nosy: +abusalimov ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20440 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3453] PyType_Ready doesn't ensure that all bases are ready
Eldar Abusalimov added the comment: It is possible to get a partially initialized class from inside a custom mro(). And extending such type results in passing NULL to PySequence_List inside mro_implementation, which in turn leads to PyErr_BadInternalCall. #22735 has a test reproducing it (http://bugs.python.org/file37036) and a corresponding fix of mro_implementation (http://bugs.python.org/file37038). However, I'm not sure that the proper way is to call PyType_Ready on each uninitialized class from tp_bases. In particular, the test case from the link above would end up with infinite recursion (PyType_Ready(cls) - mro(cls) - class X(cls): ... - PyType_Ready(X) - PyType_Ready(cls) - ...). Moreover, whether a type is initialized or not is determized by checking its tp_dict, which is initialized before filling in tp_mro, so that may be the test above is not the case of Roger. -- nosy: +abusalimov ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3453 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
New submission from Eldar Abusalimov: The attached patch fixes several bugs revealed by providing a custom mro(). Most of these bugs are reentrancy issues (mainly, cls.__bases__ assignment within mro() which causes incorrect refcounting), but there are also some issues when using incomplete types with uninitialized MRO (dereferencing NULL when extending such types, attribute lookup on super, etc.). The patch is made against the default branch (py3k), however all these bugs exist in Python 2 as well. I also tried to break the patch into smaller pieces (commits, in fact) to ease reviewing: a common pattern is test-minor-fix series. The patch set is an output of `git format-patch`, and most patches have a detailed commit message describing a change inside. Adding memory mgmt and object model guys to nosy list. -- components: Interpreter Core files: mro-crashers-fix-v1.diff keywords: patch messages: 230044 nosy: abusalimov, benjamin.peterson, lemburg, tim.peters priority: normal severity: normal status: open title: Fix various crashes exposed through mro() customization type: crash versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5 Added file: http://bugs.python.org/file37025/mro-crashers-fix-v1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Changes by Eldar Abusalimov eldar.abusali...@gmail.com: Added file: http://bugs.python.org/file37026/0001-minor-test_mro-test-case-stub-and-helpers.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Changes by Eldar Abusalimov eldar.abusali...@gmail.com: Added file: http://bugs.python.org/file37029/0004-minor-type_set_bases-move-undoing-logic-to-the-end.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Changes by Eldar Abusalimov eldar.abusali...@gmail.com: Added file: http://bugs.python.org/file37027/0002-test-crashers-NULL-old_mro-and-over-decref-on-reent.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Changes by Eldar Abusalimov eldar.abusali...@gmail.com: Added file: http://bugs.python.org/file37030/0005-minor-type_set_bases-extract-add_all_subclasses.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Changes by Eldar Abusalimov eldar.abusali...@gmail.com: Added file: http://bugs.python.org/file37028/0003-minor-mro_internal-extract-mro_invoke-and-mro_check.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Changes by Eldar Abusalimov eldar.abusali...@gmail.com: Added file: http://bugs.python.org/file37031/0006-minor-mro_subclasses-loop-over-a-list-of-subclasses.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Eldar Abusalimov added the comment: This is a patch with most significant changes, please review it carefully. -- Added file: http://bugs.python.org/file37032/0007-fix-handle-tp_mro-overwritten-through-reentrancy.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Changes by Eldar Abusalimov eldar.abusali...@gmail.com: Added file: http://bugs.python.org/file37033/0008-test-crashers-tp_base-tp_subclasses-inherit-cycles.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Changes by Eldar Abusalimov eldar.abusali...@gmail.com: Added file: http://bugs.python.org/file37034/0009-minor-PyType_IsSubtype-extract-a-check-using-tp_base.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Changes by Eldar Abusalimov eldar.abusali...@gmail.com: Added file: http://bugs.python.org/file37035/0010-fix-type_set_bases-inherit-cycles-and-reent-checks.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Changes by Eldar Abusalimov eldar.abusali...@gmail.com: Added file: http://bugs.python.org/file37037/0012-minor-mro_implementation-pmerge-refactory.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Changes by Eldar Abusalimov eldar.abusali...@gmail.com: Added file: http://bugs.python.org/file37038/0013-fix-mro_implementation-check-base-tp_mro-is-not-NULL.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Changes by Eldar Abusalimov eldar.abusali...@gmail.com: Added file: http://bugs.python.org/file37036/0011-test-behavior-error-on-extending-an-incomplete-type.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Changes by Eldar Abusalimov eldar.abusali...@gmail.com: Added file: http://bugs.python.org/file37039/0014-test-crasher-attr-lookup-on-super-with-uninitialized.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Changes by Eldar Abusalimov eldar.abusali...@gmail.com: Added file: http://bugs.python.org/file37040/0015-fix-super_getattro-check-type-tp_mro-and-refactory.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22735] Fix various crashes exposed through mro() customization
Eldar Abusalimov added the comment: Just in case, the previous message about reviewing is about [PATCH 07/15] (fix) handle tp_mro overwritten through reentrancy -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22735 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21919] Changing cls.__bases__ must ensure proper metaclass inheritance
New submission from Eldar Abusalimov: When a new class is constructed Python checks for possible metaclass conflicts within bases and an explicitly specified one, if any, choosing the best available (the most specialized) one. That is the following implication is expected: issubclass(B, A) = issubclass(type(B), type(A)) However, changing __bases__ attribute can break this invariant silently without an error. class O(object): ... pass ... class M(type): ... pass ... class N(type): ... pass ... class A(O, metaclass=M): ... pass ... class B(O, metaclass=N): ... pass ... B.__bases__ = (A,) B.__mro__ (class '__main__.B', class '__main__.A', class '__main__.O', class 'object') type(B) class '__main__.N' type(A) class '__main__.M' issubclass(B, A) True issubclass(type(B), type(A)) False Trying to derive from B now makes things look pretty weird: class C(A, metaclass=N): ... pass ... Traceback (most recent call last): File stdin, line 1, in module TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases class D(B, A): pass ... Traceback (most recent call last): File stdin, line 1, in module TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases class E(B, metaclass=N): ... pass ... type(E) class '__main__.N' That is one can extend a class but not its base (and not a class along its base). This effectively allows to bypass metaclass checks (by introducing a dummy class with the default metaclass, deriving it from a desired class with an inappropriate metaclass by changing __bases__ and using it instead of the desired class). This behavior is observed in 2.7, 3.2 and 3.4. I would expect the same check for metaclass conflicts when changing __bases__ as upon creating a new class: # EXPECTED: ... B.__bases__ = (A,) Traceback (most recent call last): File stdin, line 1, in module TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases -- components: Interpreter Core messages: 222312 nosy: Eldar.Abusalimov priority: normal severity: normal status: open title: Changing cls.__bases__ must ensure proper metaclass inheritance type: behavior versions: Python 2.7, Python 3.2, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21919 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
