[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[STINNER Victor]

STINNER Victor added the comment:

traceback.clear_frames() doesn't really clear all frames: see bpo-31321.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[A.M. Kuchling]

Changes by A.M. Kuchling a...@amk.ca:


--
resolution:  - fixed
stage: patch review - committed/rejected
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[A.M. Kuchling]

A.M. Kuchling added the comment:

Ping!  I'd like to change the function name to clear_frames() and then commit 
this.  Antoine or anyone, want to disagree with using clear_frames() as the 
name?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Antoine Pitrou]

Antoine Pitrou added the comment:

 Ping!  I'd like to change the function name to clear_frames() and then
 commit this.  Antoine or anyone, want to disagree with using
 clear_frames() as the name?

clear_frames() sounds fine to me :-)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Roundup Robot]

Roundup Robot added the comment:

New changeset 100606ef02cf by Andrew Kuchling in branch 'default':
#1565525: Add traceback.clear_frames() helper function to clear locals ref'd by 
a traceback
http://hg.python.org/cpython/rev/100606ef02cf

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[A.M. Kuchling]

A.M. Kuchling added the comment:

I'm happy to change the function name, though I'll note that the traceback 
module does have print_tb(), format_tb() and extract_tb(). 

I'm OK with both of Victor's suggestions but personally slightly prefer 
traceback.clear_frames(tb).

Rationale: People who are keeping tracebacks around and want to save memory are 
probably using other functions from the traceback module, and the module has 
fairly short names (print_tb, format_exception) so I doubt they'll often do 
'from traceback import clear_traceback_frames'.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Antoine Pitrou]

Antoine Pitrou added the comment:

 Revised version of the patch: catches RuntimeError instead of
 skipping the first frame; adds versionadded tag; adds entry to NEWS
 and whatsnew files.

Looks good to me, thank you.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[STINNER Victor]

STINNER Victor added the comment:

I tried to implement the feature as a new traceback.clear_frames() method. I 
tried to follow the chain of frame objects (using frame.f_back), but it does 
not work as expected. The method needs to follow the chain of traceback objects 
(tb.tb_next). So it makes sense to define a function instead of a method (a 
method usually only affect the object, not a chain of objects).

clear-tb-frames-2.txt:

- I didn't see the tb abbreviation in other places in Python, except for 
traceback attributes. I prefer clear_traceback_frames(). The name 
clear_frames() is maybe better because traceback is already known by the 
context (the module is called tracback. Example: traceback.clear_frames(tb) 
instead of traceback.clear_traceback_frames(tb).

- The documentation is wrong: frame.clear() does not guarantee to clear *all* 
locals, but only *most* locals:

F.clear(): clear most references held by the frame);

So I suggest a more permissive documentation:

Clear most reference held by frames.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Antoine Pitrou]

Antoine Pitrou added the comment:

 - The documentation is wrong: frame.clear() does not guarantee to
 clear *all* locals, but only *most* locals:
 
 F.clear(): clear most references held by the frame);

Actually, the documentation is right: references != locals ;-)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[A.M. Kuchling]

A.M. Kuchling added the comment:

Here's a patch implementing traceback.clear_tb_frames().  (Feel free to 
bikeshed about the name.)

One more substantial question: the top frame of the traceback is possibly still 
running.  Currently the code skips it by doing an initial 'tb = tb.tb_next'.  
Would it be better to catch and ignore the RuntimeError 
from frame.clear()?

--
stage: needs patch - patch review
Added file: http://bugs.python.org/file31668/clear-tb-frames.txt

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Antoine Pitrou]

Antoine Pitrou added the comment:

 One more substantial question: the top frame of the traceback is
 possibly still running.  Currently the code skips it by doing an
 initial 'tb = tb.tb_next'.  Would it be better to catch and ignore the
 RuntimeError 
 from frame.clear()?

Yes, I think it would be better.
Other than that, the doc lacks a versionadded tag.
Thanks!

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[A.M. Kuchling]

A.M. Kuchling added the comment:

Revised version of the patch: catches RuntimeError instead of skipping the 
first frame; adds versionadded tag; adds entry to NEWS and whatsnew files.

--
Added file: http://bugs.python.org/file31692/clear-tb-frames-2.txt

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[STINNER Victor]

STINNER Victor added the comment:

I would prefer a clear_frames() method on the traceback object rather than
a function.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Phil Connell]

Changes by Phil Connell pconn...@gmail.com:


--
nosy: +pconnell

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
keywords: +easy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Antoine Pitrou]

Antoine Pitrou added the comment:

 We need maybe an helper to clear all frames referenced by a traceback?

Yes. Something in the traceback module would be fine.

--
components: +Library (Lib) -Interpreter Core
versions: +Python 3.4 -Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Antoine Pitrou]

Antoine Pitrou added the comment:

frame.clear() was committed in issue17934.

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[STINNER Victor]

STINNER Victor added the comment:

 frame.clear() was committed in issue17934.

How should it be used to workaround this issue (tracebacks eat up memory by 
holding references to locals and globals when they are not wanted)?

We need maybe an helper to clear all frames referenced by a traceback?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Terry J. Reedy]

Terry J. Reedy tjre...@udel.edu added the comment:

This seems to be about reducing internal resource usage in a way that would be 
mostly invisible to the normal user. A core surface feature request would have 
to be put off to 3.3, but I do not see that as such.

--
type: feature request - resource usage

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

It is true that your proposed feature would provide a way for a programmer to 
manually resolve the cycle issue; however, the open issue in the pep is how to 
do this automatically.

But if you hold on to the traceback reference, you should *expect* all those 
values to persist, so that shouldn't be surprising.

I repeat my recommendation that you take this to python-ideas for feedback, and 
then work on a patch if the feedback is positive.

(By the way, I checked with a twisted developer, and what he wanted was a 
convenient way to manually create traceback objects.)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Greg Hazel]

Greg Hazel gha...@users.sourceforge.net added the comment:

 you should *expect* all those values to persist, so that shouldn't be 
 surprising.

It's not something I expected, and the warnings around traceback objects are a 
good indication that other developers have not expected it either. One poster 
on python-ideas said Working with traceback objects can easily introduce 
hidden circular references, so it usually better not access them at all.  
Since these 'hidden' references are not used in many cases, it is surprising 
that they would be required.

 I repeat my recommendation that you take this to python-ideas for feedback, 
 and then work on a patch if the feedback is positive.

I have, and it has been so far.

 (By the way, I checked with a twisted developer, and what he wanted was a 
 convenient way to manually create traceback objects.)

When does Twisted want to manually create traceback objects? Failure has 
specific functions to stringify the traceback to remove the references 
mentioned here. Creating a fake traceback would be one way to achieve that, but 
if the references did not exist I'm not sure what the goal would be.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Excellent.

As for twisted, I'm just repeating what I understood of what he said when I 
asked.  It could well be that this feature would help them, I don't know.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Greg Hazel]

Greg Hazel gha...@users.sourceforge.net added the comment:

This is still an issue.

The bug I'm reporting had been explained well, I thought, but I'll repeat it in 
summary:
There is no way to pass around traceback objects without holding references to 
an excessive number of objects.

Traceback raising typically does not use these references at all, so having 
some way to discard them would be very valuable. This allows storing and 
passing tracebacks between threads (or coroutines or async tasks) without dying 
quickly due to memory bloat. The simple-minded way to fix this is to allow the 
user to break the reference themselves.

Fixing this bug would invalidate the need for hacks like the one Twisted has 
come up with in their twisted.python.Failure object which stringifies the 
traceback object, making it impossible to re-raise the exception. Failure has a 
lot of re-implementations of Exceptions and traceback objects as a result.

--
status: closed - open
title: gc allowing tracebacks to eat up memory - tracebacks eat up memory by 
holding references to locals and globals when they are not wanted
versions: +Python 2.6, Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Martin v . Löwis]

Martin v. Löwis mar...@v.loewis.de added the comment:

I still don't understand the issue. You say that you want a traceback, but then 
you say you don't want the objects in the traceback. So what *precisely* is it 
that you want, and what is it that you don't want?

In any case, whatever the solution, it is likely a new feature, which aren't 
acceptable anymore for 2.x release. So please don't target this report for any 
2.x version.

--
versions: +Python 3.2 -Python 2.5, Python 2.6, Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Greg Hazel]

Greg Hazel gha...@users.sourceforge.net added the comment:

The objects I do want in the traceback are the objects necessary to print a 
traceback, but not the locals and globals of each frame.

For example:

def bar():
  x = stuff
  raise Exception(example!)
bar()

prints: 
Traceback (most recent call last):
  Line 4, in module
bar()
  Line 3, in bar
raise Exception(example!)
Exception: example!

There is no reason in that example to have a reference to x in the traceback, 
since it's not used in the output. This becomes important when I try to save a 
reference to the traceback object and raise it later:

import sys
def bar():
  x = stuff
  raise Exception(example!)
try:
  bar()
except:
  exc_info = sys.exc_info()
def foo(e):
  raise e[0], e[1], e[2]
# sometime possibly much later...
foo(exc_info)

Traceback (most recent call last):
  Line 12, in module
foo(exc_info)
  Line 6, in module
bar()
  Line 4, in bar
raise Exception(example!)
Exception: example!


During that sometime possibly much later... comment, a reference to x is 
held, when it will not be used in printing the traceback later. So, I would not 
like to keep a reference to x, and currently there is no way to do that 
without also dropping a reference to the data needed to print the traceback.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Terry J. Reedy]

Terry J. Reedy tjre...@udel.edu added the comment:

To call this a bug for tracker purposes, there would have to be a specific 
discrepancy between doc promise and observed behavior. Every feature request 
fixes a 'design bug' ;-).

--
resolution: wont fix - 
stage:  - needs patch
type:  - feature request

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Greg Hazel]

Greg Hazel gha...@users.sourceforge.net added the comment:

It seems this is partially addressed in a big red Warning section of the docs 
on sys.exc_info: 
http://docs.python.org/release/3.1/library/sys.html#sys.exc_info
and is captured in the Open Issue: Garbage Collection section for PEP-3134: 
http://www.python.org/dev/peps/pep-3134/

Bug or feature request, this a well understood and real issue.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Those two references have nothing to do with your request.  They are talking 
about cycles that get created by grabbing the traceback variable, and the fact 
that it may take a while before garbage collection reclaims them *after you 
have let go of the traceback reference*.  You are talking about wanting to 
release the locals in the stack frame and still hold on to the traceback.  
Completely different issue.

Note that while the standard traceback print doesn't reference the locals in 
the frame, extended tracebacks such as cgitb do.  There is also a proposal to 
add such locals printing to the standard traceback.

I'm pretty sure this issue isn't going to go anywhere without a patch proposal, 
but before you work on that you might want to raise the idea on python-ideas 
and see if a wider audience thinks it is a good idea.

--
nosy: +r.david.murray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1565525] tracebacks eat up memory by holding references to locals and globals when they are not wanted

[Greg Hazel]

Greg Hazel gha...@users.sourceforge.net added the comment:

It depends on how you look at it. Those two issues describe the surprising 
behavior of the same references I'm talking about here, when the lifetime of 
the traceback reference is only inside the same frame. This ticket describes 
the surprising behavior of those references when the lifetime of the traceback 
is any number of frames. My example eat_memory.py is much closer to the issue 
described in those links - the lifetime of the traceback object is 
insignificantly one frame higher, not the lifetime of the application.

Either way, a feature to discard those references would resolve both.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1565525
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com