[issue17911] Extracting tracebacks does too much work

[STINNER Victor]

STINNER Victor added the comment:

While trying to fix a similar issue for the asyncio project, I wrote the 
following code:
https://bitbucket.org/haypo/misc/src/ce48d7b3ea1d223691e496e41aca8f5784671cd5/python/suppress_locals.py?at=default

I was not aware that a similar approach (attached traceback2.patch) was already 
implemented.

Asyncio issues:
https://code.google.com/p/tulip/issues/detail?id=155
https://code.google.com/p/tulip/issues/detail?id=42

See also the Python issue #20032.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Raymond Hettinger]

Raymond Hettinger added the comment:

>  I think it makes sense to treat this as a completely new
> traceback introspection API and ignore the low level details
> of the legacy API.

That would likely be the cleanest approach.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[STINNER Victor]

Changes by STINNER Victor :


--
components: +asyncio
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Eric Snow]

Changes by Eric Snow :


--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Yury Selivanov]

Changes by Yury Selivanov :


--
keywords:  -easy
nosy: +yselivanov
versions: +Python 3.5 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Nick Coghlan]

Nick Coghlan added the comment:

For dis, we introduced a new rich bytecode introspection API. Ditto for
inspect.Signature. I think it makes sense to treat this as a completely new
traceback introspection API and ignore the low level details of the legacy
API.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Antoine Pitrou]

Antoine Pitrou added the comment:

> When it comes to deferral, we could wrap each line in a different
> class than tuple, but this constitutes a public API change (and this
> is a *very* widely used library). Even changing to a namedtuple
> would cause a lot of grief, since we'd break lots of doctests, and
> subclassing tuple is a lot of effort for little benefit. If we only
> do it for the deferred usage, it would "only" be inconsistent. I
> suppose we could have a completely separate way of doing things for
> ExceptionSummary, but again, inconsistent, and I think if one
> extract_xxx method provides the functionality, users would expect it
> to be available in the others.

YMMV, but I think we should go for inconsistency here. The other APIs
in the traceback module are low-level and clearly limited by the type
of objects they return.

This feature request is a good opportunity to design something a little
more future-proof. I'd love to know what other developers/contributors
think here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Björn Sandberg Lynch]

Björn Sandberg Lynch added the comment:

I was trying to stay with the established pattern of the existing methods. 
There are two unrelated issues to solve here - deferring linecache access, and 
the extract_exception functionality.

When it comes to deferral, we could wrap each line in a different class than 
tuple, but this constitutes a public API change (and this is a *very* widely 
used library). Even changing to a namedtuple would cause a lot of grief, since 
we'd break lots of doctests, and subclassing tuple is a lot of effort for 
little benefit. If we only do it for the deferred usage, it would "only" be 
inconsistent. I suppose we could have a completely separate way of doing things 
for ExceptionSummary, but again, inconsistent, and I think if one extract_xxx 
method provides the functionality, users would expect it to be available in the 
others.

For ExceptionSummary, the same summary instance is used more than once if you 
have a cause set, so object creation had to be separated from the graph 
population. Either this is done in a constructor function or we would need some 
obscure tricks with the class kwargs. This way, there's a separation of 
concerns - the class wraps *one* exception, and the function creates a bunch 
and links them together as needed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Antoine Pitrou]

Antoine Pitrou added the comment:

I think ExceptionSummary could be the visible API, instead of adding an 
indirection through a separate global function.
Also, I don't think having the "exc_traceback" list of tuples is future-proof. 
What if we want to add some information to each traceback entry, or refactor it 
to take __loader__ into account?
Instead, ExceptionSummary could expose the desired operations (e.g. iterate 
over traceback lines and the associated source code lines) without being 
constrained by some implementation details.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Björn Sandberg Lynch]

Changes by Björn Sandberg Lynch :


Added file: http://bugs.python.org/file30901/traceback2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Björn Sandberg Lynch]

Björn Sandberg Lynch added the comment:

After thinking about it, I decided to defer instead of suppress line fetching. 
Suppressing would still give a traceback later, but not the same as the 
original.

extract_exception is where the meat is. There's a slight quirk in how context 
works - if cause is set to something else than the actual exception, context 
still gets set even though it won't be printed. This is to enable later 
analysis. However, if you explicitly 'raise Exception from None' neither of 
them will be set - the formatting code would have issues otherwise.

I'm not too happy about the code duplication between the format_exception 
method and the formatting that works on exceptions directly, but there are 
plenty of tests to ensure that the output is identical.

Feel free to dispute my naming choices. I'm admittedly influenced by twisted.

--
keywords: +patch
Added file: http://bugs.python.org/file30895/traceback.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Antoine Pitrou]

Antoine Pitrou added the comment:

> If we keep a reference to f_globals for each line in the traceback, we 
> can defer this to later (ideally we'd just keep the __loader__ value,
> but that would require changing the linecache interface as well).

Or you could add a *new* linecache interface.
Keeping __loader__ sounds much less hackish than keeping f_globals to me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Guido van Rossum]

Guido van Rossum added the comment:

In terms of how much freedom you have about changing the internal, I'd check 
how long ago they were changed.  "Internal" APIs that have been stable for many 
versions tend to have illicit external uses -- but internal APIs that were 
introduced recently (e.g. in 3.2) are usually safe to use -- nobody is going to 
make too much of a stink if you break their code.

As for saving f_globals, if you're going to save an extra pointer anyways, why 
not just save the frame pointer?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Björn Sandberg Lynch]

Björn Sandberg Lynch added the comment:

I've been looking into this as an easy piece to bite off. If I understand Guido 
correctly, he'd like to defer or suppress the linecache call when getting the 
tb summary. The problem with deferring is that you need to access f_globals for 
the loader to work correctly when the module source is a non-file import 
source. If we keep a reference to f_globals for each line in the traceback, we 
can defer this to later (ideally we'd just keep the __loader__ value, but that 
would require changing the linecache interface as well).

My inclination would be to have another keyword argument to 
_extract_tb_or_stack_iter (terse=False or verbose=True - either style works). 
In terse mode, no source lines would be available, and the formatted output 
would be the same as if the source wasn't available at all. This would work, 
although the traceback module is structured so that I'd need to pass it through 
quite a few wrapped iterator calls.

I'm not sure how free a hand I have when it comes to refactoring the internal 
implementation. I'm not fond of the extractor callbacks - I'd prefer a 
generator-based approach on the lines of:

def _tb_iter(tb, limit):
i = 0
while tb is not None:
if limit is not None and limit < i:
break
yield tb.tb_frame, tb.tb_lineno
tb = tb.tb_next
i += 1

def _extract_frame_iter(frames, terse=False):
...
for frame, lineno in frames:
...

--
nosy: +adaptivelogic

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Antoine Pitrou]

Antoine Pitrou added the comment:

Ok, I've spinned off the frame clearing suggestion to issue17934.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Antoine Pitrou]

Antoine Pitrou added the comment:

You may want two pieces of stack: the piece of stack that is above and 
including the catch point, and the piece of stack that is below it. The former 
is what gets traditionally printed in tracebacks, but the latter can be useful 
as well (see issue1553375 for a related feature request).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Guido van Rossum]

Changes by Guido van Rossum :


--
keywords: +easy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Guido van Rossum]

Guido van Rossum added the comment:

That sounds great, except I'd expect the exception type and str(), since a 
formatted traceback uses the str() of the exception, not its repr().

Personally I don't think the tuples need to be named, but I won't hold up 
progress either. :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Nick Coghlan]

Nick Coghlan added the comment:

Antoine - I like your idea, but I think it's a separate issue. I agree with 
Guido that exposing some lower level non-formatting APIs in the traceback 
module would be helpful.

I see Guido's suggestion here as similar to the change we just made in the dis 
module to expose a dis.get_instructions iterator. That change makes it much 
easier to work with the disassembler output programmatically, whereas the 
module was previously too focused on displaying text with a specific format.

My current thoughts: define a "TracebackSummary" with the following fields:

exc_type
exc_repr
stack_summary
cause
context

stack_summary would be a list of (filename, lineno, functionname) triples as 
Guido suggested (probably a named tuple)

cause and context would be either None or a reference to an appropriate 
TracebackSummary object

Basically, the summary should contain all of the information needed to create 
the formatted traceback output, without actually doing any formatting (aside 
from calling repr() on the exception)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Antoine Pitrou]

Antoine Pitrou added the comment:

> On Mon, May 6, 2013 at 6:06 AM, Antoine Pitrou
>  wrote:
> > Note that generator cleanup through the frame has a patch in
> > issue17807.
> 
> Sadly that doesn't help with my issue.

It won't. It's just a building block for the change I've proposed here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Guido van Rossum]

Guido van Rossum added the comment:

On Mon, May 6, 2013 at 6:06 AM, Antoine Pitrou  wrote:
> Note that generator cleanup through the frame has a patch in issue17807.

Sadly that doesn't help with my issue. I applied path gen3.patch and
ran various versions of the code I have. There's still something
holding on to the Task or the exception.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Guido van Rossum]

Guido van Rossum added the comment:

I'm not snubbing my nose at something that breaks these types of cycles more 
easily, but I still think that the abstractions offered by the traceback module 
could be improved.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Antoine Pitrou]

Antoine Pitrou added the comment:

I think instead we may want to add a finalize() or close() method on frame 
objects which would clear all local variables (as well as dereference the 
globals dict, perhaps), after having optionally run a generator's close() 
method (if the frame belongs to a generator).

If I'm not mistaken, it should allow breaking reference cycles, and remove the 
need for complex traceback processing, which Twisted currently also does: 
http://twistedmatrix.com/trac/browser/trunk/twisted/python/failure.py#L89

Note that generator cleanup through the frame has a patch in issue17807.

--
nosy: +ncoghlan, pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Guido van Rossum]

New submission from Guido van Rossum:

For Tulip I may have to extract tracebacks and store the extracted data, in 
order to avoid cycles created by the frames referenced in the traceback.  I'll 
be extracting the traceback every time an exception is raised, and in most 
cases I won't be using the extracted info, because usually the exception is 
caught (or logged) by user code.  But it's very important to ensure that if the 
user code doesn't catch or log it, I can log a traceback, and I won't know that 
this is the case until a destructor is called, which may be quite a bit later.  
(Reference: http://code.google.com/p/tulip/source/browse/tulip/futures.py#38)

Unfortunately it looks like the public APIs do a lot more work than needed.  
Ideally, there'd be something similar to _extract_tb_or_stack_iter() that 
doesn't call linecache.getline() -- it should just return triples of (filename, 
lineno, functionname), and enough structure to tell apart the __cause__, 
__context__, and __traceback__ (the first two possibly repeated).  Given this 
info it would be simple enough to format and log the actual traceback, and 
storing just this would take less space and time than computing the lines of 
the fully-formatted traceback.

--
components: Library (Lib)
messages: 188467
nosy: gvanrossum
priority: normal
severity: normal
status: open
title: Extracting tracebacks does too much work
type: enhancement
versions: Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com