[issue43252] deepcopy of weakref proxies
Konrad Schwarz added the comment: Well, in the example code, the memo dictionary contains the (hard) reference to newly created objects but ultimately, at the close of the deepcopy, the objects are (hard) referenced using the normal "child" attributes. I don't however know how this would work if this was implemented within the weakref code but I expect that their __deepcopy__ method also has access to the memo variable. If, at the close of deepcopy and the destruction of memo, no (hard) references exist to an object, then it would be collected, but that matches standard weak reference semantics. Even if it were not possible to solve this programmatically within Python, I think the problem (and e.g. the workaround given below) should be mentioned in the documentation of deepcopy and/or weakref. -- ___ Python tracker <https://bugs.python.org/issue43252> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43383] imprecise handling of weakref callbacks
Change by Konrad Schwarz : -- resolution: -> postponed stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue43383> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43383] imprecise handling of weakref callbacks
Konrad Schwarz added the comment: Unfortunately, my management has impressed other priorities upon me; I can't delve deeper into this subject at the moment. My takeaway is that the error very likely lies on my side; maybe I need to re-check local variables and del them explicitly. In any case, I can work around the situation by being extra careful in the callback. I don't know how well all of this would work in a truly multi-threaded environment, but the application doesn't require that at all. In any case, thank you for the insightful comments and the willingness to understand my problem! I certainly have learned a lot. -- ___ Python tracker <https://bugs.python.org/issue43383> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43383] imprecise handling of weakref callbacks
Konrad Schwarz added the comment: > If I understand correctly, the reported bug is that you're seeing a weakref > callback being executed > after the weakref.ref instance it's attached to is deleted. Is that correct? Exactly. I del what should be the only reference to the weakref.ref/proxy, then del the weakref.ref's referent. The weakref.ref's callback is executed. > Do you have a minimal example we can use to reproduce the effect? Without > such an example, there's not > much of a realistic path to moving this forward. Unfortunately, not at the moment. > However, my suspicion is that that's not what's actually happening here. I > suspect that when you say > "after the corresponding weak reference has been deleted with del", that the > "del" statement you refer > to is not actually deleting the last reference to the weakref.ref object, so > the weakref still exists > after the "del". One easy way that this could happen is if the weakref is > part of a reference cycle > (and I know from personal experience that it's horribly easy to accidentally > *create* reference cycles > via weakref callbacks, especially if those callbacks refer to instance > methods). I tried to be as punctilious as I could to prevent this sort of thing from happening. The weird thing is that the error does not occur when I single step the code in PDB. It must be something internal to CPython, some sort of optimization. Are local variables/stack frames cleaned lazily? Is the last accessed value of a dictionary cached internally, increasing its reference count? I've worked around the problem by double checking in the callback method that it actually should execute; i.e. in my case, that the object has not been removed (del) from a dictionary (which should be the single reference to the object). If the problem does indeed lie with CPython, but has too much of a performance/maintenance impact, perhaps a caveat could be added to the documentation to the effect of "The lifetime of an object may be longer than what a programmer expects causing a weakref's callback to be invoked at a surprising time; it is good practice to program the callback defensively." -- ___ Python tracker <https://bugs.python.org/issue43383> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43383] imprecise handling of weakref callbacks
New submission from Konrad Schwarz :
I am seeing the following non-deterministic behavior:
My code processes DeviceTree, a tree-based specification format for hardware
descriptions, that includes cross-references ("phandles"). For all intents and
purposes, this format is similar to XML; phandles are analog to ID/IDREFS.
To prevent reference cycles and avoid the need for garbage collection, my code
uses weakref.proxy for parent pointers and weakref.ref for cross-references.
My goal is to provide a "projection" operation on a DeviceTree: creating
derived DeviceTrees that model subsets of the hardware (this is to partition
the hardware into multiple independent sub-machines). The projection is
specified by newly introduced nodes and attributes (aka properties) in the
tree; phandles are used to indicate which part belongs to which partition.
Python weak references provide a callback interface to indicate the demise of
their referents and my code uses that to prune the tree: e.g., if a node
modeling a partition is deleted, nodes that reference that node (i.e., indicate
they belong to that partition) are deleted in the corresponding weakref
callback. So technically, the code implicitly uses the interpreters list of
weak referrers (__weakref__) to find and execute code on them when the
referent's state changes.
This works exactly as envisioned when single-stepping in PDB.
When running at full speed however, I see that weak reference callbacks are
being triggered after the corresponding weak reference has been deleted with
del (the weak reference is a value of a Python dict holding a node's
attributes.)
I suspect that this is because of some batching or deferred processing in the
Python interpreter.
Ultimately, this is a violation of the semantics and must be classified as a
bug.
However, in my case, it would suffice to have a "memory barrier" type of
operation that flushes the queue of deferred deletions before continuing.
Something like that must exist, because single stepping in PDB is successful.
Initial tests of calling the garbage collector to this end were inconclusive,
unfortunately.
--
components: Interpreter Core
messages: 387999
nosy: konrad.schwarz
priority: normal
severity: normal
status: open
title: imprecise handling of weakref callbacks
type: behavior
versions: Python 3.7
___
Python tracker
<https://bugs.python.org/issue43383>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43252] deepcopy of weakref proxies
New submission from Konrad Schwarz : copy.deepcopy()-ing a tree structure that has internal parent and cross-reference links implemented as weakref.proxy() objects causes the weak reference proxies themselves to be copied (still refering to their original referents) rather than weak references to deep-copied referents to be created. A workaround is to add the following __deepcopy__ method to affected classes: def __deepcopy__ (self, memo): # taken from Stackoverflow: "How to override the # copy deepcopy operations for a python object" # and "Python: dereferencing weakproxy" cls = self.__class__ result = cls.__new__ (cls) memo [id (self)] = result for k, v in self.__dict__.items (): if isinstance (v, weakref.ProxyType): new_v = weakref.proxy (copy.deepcopy ( v.__repr__.__self__, memo)) else: new_v = copy.deepcopy (v, memo) setattr (result, k, new_v) return result -- components: Library (Lib) messages: 387226 nosy: konrad.schwarz priority: normal severity: normal status: open title: deepcopy of weakref proxies type: behavior versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue43252> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42324] Doctest: directives
Konrad Schwarz added the comment: Yes it is. On Wed, Nov 11, 2020 at 1:06 PM Karthikeyan Singaravelan < rep...@bugs.python.org> wrote: > > Karthikeyan Singaravelan added the comment: > > Is this similar to https://bugs.python.org/issue36675 ? > > -- > nosy: +xtreak > > ___ > Python tracker > <https://bugs.python.org/issue42324> > _______ > -- Mit freundlichen Grüßen Konrad Schwarz -- ___ Python tracker <https://bugs.python.org/issue42324> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42324] Doctest: directives
New submission from Konrad Schwarz : In both PDF and HTML, the examples for Doctest directives don't actually show the directives themselves, perhaps because they syntactically start with a #. -- assignee: docs@python components: Documentation messages: 380752 nosy: docs@python, konrad.schwarz priority: normal severity: normal status: open title: Doctest: directives type: enhancement versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue42324> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
