Hi, Python 3 added a __traceback__ attribute to exception objects. I guess that it was added to be able to get the original traceback when an exception is re-raised. Artifical example (real code is more complex with subfunctions and conditional code):
try: ... except Exception as exc: ... raise exc # keep original traceback The problem is that Exception.__traceback__ creates reference cycles. If you store an exception in a local variable, suddently, all local variables of all frames (of the current traceback) become part of a giant reference cycle. Sometimes all these variables are kept alive very long (until you exit Python?), at least, much longer than the "expected" lifetime (destroyed "magically" local variables when you exit the function). The reference cycle is: 1) frame -> local variable -> variable which stores the exception 2) exception -> traceback -> frame exception -> ... -> frame -> ... -> same exception Breaking manually the reference cycle is complex. First, you must be aware of the reference cycle! Second, you have to identify which functions of your large application create the reference cycle: this task is long and painful even with good tooling. Finally, you have to explicitly clear variables or attributes to break the reference cycle manually. asyncio.Future.set_exception() keeps an exception object and its traceback object alive: asyncio creates reference cycles *by design*. Enjoy! asyncio tries hard to reduce the consequence of reference cycles, or even try to break cycles, using hacks like "self = None" in methods... Setting self to None is really surprising and requires a comment explainaing the hack. Last years, I fixed many reference cycles in various parts of the Python 3 standard library. Sometimes, it takes years to become aware of the reference cycle and finally fix it. For example, recently, I worked on fixing all "dangling threads" leaked by tests of the Python test suite, and I found and fixed many reference cycles which probably existed since Python 3 was created (forked from Python 2): * socket.create_connection(): commit acb9fa79fa6453c2bbe3ccfc9cad2837feb90093, bpo-31234 * concurrent.futures.ThreadPoolExecutor: commit bc61315377056fe362b744d9c44e17cd3178ce54, bpo-31249 * pydoc: commit 4cab2cd0c05fcda5fcb128c9eb230253fff88c21, bpo-31238 * xmlrpc.server: commit 84524454d0ba77d299741c47bd0f5841ac3f66ce, bpo-31247 Other examples: * test_ssl: commit 868710158910fa38e285ce0e6d50026e1d0b2a8c, bpo-31323 * test_threading: commit 3d284c081fc3042036adfe1bf2ce92c34d743b0b, bpo-31234 Another example of a recent change fixing a reference cycle, by Antoine Pitrou: * multiprocessing: commit 79d37ae979a65ada0b2ac820279ccc3b1cd41ba6, bpo-30775 For socket.create_connection(), I discovered the reference cycle because a test started to log a warning about dangling thred. The warning was introduced indirectly by a change which modified support.HOST value from '127.0.0.1' to 'localhost'... It's hard to see to link between support.HOST value and a reference cycle. Full story: https://bugs.python.org/issue29639#msg302087 Again, it's just yet another random example of a very tricky reference cycle bug caused by Exception.__traceback__. Ideally, CPython 3.x should never create reference cycles. Removing Exception.__traceback__ is the obvious "fix" for the issue. But I expect that slowly, a lot of code started to rely on the attribute, maybe even for good reasons :-) A more practical solution would be to log a warning. Maybe the garbage collector can emit a warning if it detects an exception part of a reference cycle? Or maybe detect frames? If the GC cannot do it, maybe we might use a debug thread (enabled manually) which checks manually if an exception is part of a reference cycle using gc.get_objects(): check if an exception remains alive longer than X seconds? I had the same idea for asyncio, to detect reference cycles or if a task is never "awaited", but I never implemented the idea. Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
