[issue41174] asyncio.coroutine decorator returns a non-generator function when using PYTHONASYNCIODEBUG
New submission from Allan Feldman :
This code behaves differently when PYTHONASYNCIODEBUG=1
import asyncio
import inspect
@asyncio.coroutine
def foo():
yield from asyncio.sleep(0)
print("isgeneratorfunction:", inspect.isgeneratorfunction(foo))
PYTHONASYNCIODEBUG:
isgeneratorfunction: False
non-debug mode:
isgeneratorfunction: True
When in debug mode, the `asyncio.coroutine` decorator returns a function that
is not a generator function
(https://github.com/python/cpython/blob/bd4a3f21454a6012f4353e2255837561fc9f0e6a/Lib/asyncio/coroutines.py#L144)
The result is that introspection of functions is changed when
PYTHONASYNCIODEBUG is enabled.
--
components: asyncio
messages: 372706
nosy: a-feld, asvetlov, yselivanov
priority: normal
severity: normal
status: open
title: asyncio.coroutine decorator returns a non-generator function when using
PYTHONASYNCIODEBUG
type: behavior
versions: Python 3.10, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python
3.9
___
Python tracker
<https://bugs.python.org/issue41174>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40312] Weakref callbacks running before finalizers in GC collection
Allan Feldman added the comment: Yup agree with all the points above, just wanted to point out that I think self.value is strongly referenced (even though it's just internal references and will be collected by the gc) during Foo.__del__ execution (annotated code below), yet the WeakValueDictionary entry is cleared: import gc import sys import weakref cache = weakref.WeakValueDictionary() class Bar: pass class Foo: def __init__(self): self._self = self self.value = Bar() cache[id(self.value)] = self.value def __del__(self): print(sys.getrefcount(self.value)) # -> 2 # the cache may or may not have self.value at this point # even though self.value is strongly referenced! print(list(cache.items())) # -> [] o = Foo() del o gc.collect() -- ___ Python tracker <https://bugs.python.org/issue40312> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40312] Weakref callbacks running before finalizers in GC collection
Allan Feldman added the comment: I definitely understand the possibility that some code is relying on the current gc behavior of weakref callbacks being invoked after finalizers. That being said, the behavior is currently inconsistent between gc and reference counted paths. The language doesn't have to define the explicit ordering but the internal inconsistency was definitely unexpected for us. The result is that behavioral consistency becomes more difficult in application code when using language provided structures such as WeakValueDictionary. cache = weakref.WeakValueDictionary() class Bar: pass class Foo: def __init__(self): self._self = self self.value = Bar() cache[id(self.value)] = self.value def __del__(self): # the cache may or may not have self.value at this point # even though self.value is strongly referenced! print(list(cache.items())) >From the weakref docs: > Entries in the dictionary will be discarded when no strong reference to the > value exists any more. But doesn't the code above imply that the entry is discarded even though there are strong references to the value? In any case, I definitely appreciate all the eyes on this Tim + Pablo! At the very least, documentation updates do sound like a good idea if we're moving forward with leaving the behavior of weakrefs as currently specified. In particular, it would be worth pointing out that weakrefs callbacks can run even when the object is referenced. -- ___ Python tracker <https://bugs.python.org/issue40312> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40312] Weakref callbacks running before finalizers in GC collection
Allan Feldman added the comment: Also I just noticed this statement: > In current CPython, for your ForeverObject(False), `del o` does not make the > object trash "for real". __del__ runs immediately (due to deterministic, > synchronous reference counting) and resurrects it. That cuts off the "about > to have its memory destroyed and recycled" part, so the callback doesn't run. The problem is the callback _does_ run even though the object is resurrected! :) (Only if going through gc) -- ___ Python tracker <https://bugs.python.org/issue40312> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40312] Weakref callbacks running before finalizers in GC collection
Allan Feldman added the comment: Thanks for the explanation! I agree that "about to be finalized" is unclear in the docs :) I still believe that having different behavior for the ordering of finalizers versus weakref callbacks depending on whether the path is through gc versus reference counting is a bug. The callback should be able to assume that when it's running, the referent is actually dead. The execution of a weakref callback in our case results in items being dropped from a WeakValueDictionary prematurely (the object is still referenced, accessible, and alive at the time the weakref callback runs). I've attached a patch that would cause weakref callbacks to run consistently after finalizers. With the patch applied, all tests in cpython appear to pass, but the code examples above now work consistently. -- keywords: +patch Added file: https://bugs.python.org/file49076/0001-Run-finalizers-before-invoking-weakref-callbacks.patch ___ Python tracker <https://bugs.python.org/issue40312> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40312] Weakref callbacks running before finalizers in GC collection
Allan Feldman added the comment: Reading more carefully, I may have jumped to conclusions here :) Looking at the weakref docs, I see the following description of the callback functionality: > If callback is provided and not None, and the returned weakref object is > still alive, the callback will be called when the object is about to be > finalized; the weak reference object will be passed as the only parameter to > the callback; the referent will no longer be available. This description seems to imply that even if an object is resurrected, the callback will be run. Using the `ForeverObject` example above, I see the weakref callback behavior is different when going through gc versus going through `_Py_Dealloc`. The behavior being different seems to imply that a contract is broken somewhere. In this case I think I assumed it was gc, but it looks like it might actually be that the contract (as currently defined) is broken by dealloc. Finalizers are always called before weakref callbacks on the reference counted path: https://github.com/python/cpython/blob/482259d0dcf27714a84cf56b93977320bea7e093/Objects/typeobject.c#L1245 Here is the output from the `ForeverObject` example above: --- Circular reference: True --- callback running -- --- Circular reference: False --- -- For my own understanding, why is the API documented as running the callback prior to finalizers running? The referent is unavailable when the callback runs, so isn't it safe to run the weakref callbacks consistently after the finalizers? -- ___ Python tracker <https://bugs.python.org/issue40312> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40312] Weakref callbacks running before finalizers in GC collection
Allan Feldman added the comment: Thanks for the response! > if a weakref W refers to object O, and W and O are _both_ in cyclic trash I believe that in the examples W is not in cyclic trash, but remains referenced as a global in the frame. Only O is in cyclic trash (O references itself). I would expect that W's callback would be invoked in this case, but only after O is guaranteed to be deleted. In some cases O can be resurrected in the finalizer. -- ___ Python tracker <https://bugs.python.org/issue40312> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40312] Weakref callbacks running before finalizers in GC collection
New submission from Allan Feldman :
Our team is making use of a weakref.WeakValueDictionary() that is accessed
through the finalizer of a class. We observed that in Python 3 occasionally
values that are directly referenced by an object being finalized were missing
from the WeakValueDictionary.
Example:
import weakref
cache = weakref.WeakValueDictionary()
class Foo(object):
pass
class Bar(object):
def __init__(self, foo):
self.foo = foo
cache['foo'] = foo
def __del__(self):
del cache['foo']
bar = Bar(Foo())
del bar
Upon further investigation, we realized that this had to do with the weakref
callback within WeakValueDictionary being called (removing the key from the
dict) before the finalizer for Foo was called.
Reproduction:
import gc
import weakref
cache = weakref.WeakValueDictionary()
class Foo(object):
pass
class Bar(object):
def __init__(self, foo):
# Force a reference cycle to run del only on gc.collect
self._self = self
self.foo = foo
cache["foo"] = foo
def __del__(self):
# foo is missing from the cache because the weakref callback has
# already run. KeyError is raised.
del cache["foo"]
bar = Bar(Foo())
del bar
gc.collect()
Expected behavior:
The weakref callback should only be called when the object is known to be
deleted (after the finalizer runs). Running weakref callbacks before then means
that the weakref callback can run on objects being ressurected by the finalizer.
Example:
import gc
import weakref
class ForeverObject(object):
def __init__(self, circular):
# Introduce a circular reference so that gc must collect the object
if circular:
self._self = self
def __del__(self):
global o
o = self
def callback(wr):
print("callback running", wr)
for circular in (True, False):
print("--- Circular reference:", circular, "---")
o = ForeverObject(circular)
wr = weakref.ref(o, callback)
del o
gc.collect()
print("--")
Note: Python 2.7 appears to have the opposite behavior - weakref callbacks are
only invoked when dealloc occurs outside of gc. The Python 2.7 behavior hasn't
yet been investigated.
If the expected behavior above is confirmed, I would be happy to submit a patch
for this issue!
--
components: Interpreter Core
messages: 366675
nosy: a-feld
priority: normal
severity: normal
status: open
title: Weakref callbacks running before finalizers in GC collection
type: behavior
versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9
___
Python tracker
<https://bugs.python.org/issue40312>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33380] Update module attribute on namedtuple methods for introspection.
Allan Feldman <allan.d.feld...@gmail.com> added the comment: That explanation makes sense to me. Thanks for taking the time to look into this! -- resolution: -> works for me stage: -> resolved status: open -> closed ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33380> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33380] Update module attribute on namedtuple methods for introspection.
Allan Feldman <allan.d.feld...@gmail.com> added the comment: Attached is a proposed change. -- keywords: +patch Added file: https://bugs.python.org/file47555/0001-bpo-33380-Update-module-attribute-on-namedtuple-meth.patch ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33380> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33380] Update module attribute on namedtuple methods for introspection.
New submission from Allan Feldman <allan.d.feld...@gmail.com>:
Python 3.7 made several performance improvements to the namedtuple class as
part of https://bugs.python.org/issue28638
Prior to the implementation of bpo-28638, the __module__ attribute for a
namedtuple's methods (e.g. _asdict) would return the value 'namedtuple_%s' %
typename (e.g. namedtuple_Point).
Due to the optimizations made, the __module__ attribute for a namedtuple's
methods now returns 'collections'.
The proposed change as part of this issue is to report the more accurate
derived module name for the namedtuple methods. Updating the __module__
attribute should help debug and introspection tools more accurately report the
details of executing calls (in profilers for example).
Example from Python 3.6:
>>> from collections import namedtuple
>>> Point = namedtuple('Point', ('x', 'y'))
>>> p1 = Point(1,2)
>>> p1._asdict.__module__
'namedtuple_Point'
Example from Python 3.7.0b3:
>>> from collections import namedtuple
>>> Point = namedtuple('Point', ('x', 'y'))
>>> p1 = Point(1,2)
>>> p1._asdict.__module__
'collections'
Desired behavior:
>>> from collections import namedtuple
>>> Point = namedtuple('Point', ('x', 'y'))
>>> p1 = Point(1,2)
>>> p1._asdict.__module__
'__main__'
--
components: Library (Lib)
messages: 315869
nosy: a-feld, rhettinger
priority: normal
severity: normal
status: open
title: Update module attribute on namedtuple methods for introspection.
type: enhancement
versions: Python 3.7, Python 3.8
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue33380>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
