Re: [Python-Dev] tokenize string literal problem

2009-10-23 Thread C or L Smith
C or L Smith wrote: > PROBLEM > I need to find code snippets which are located in docstrings. > Docstrings, being string literals should be able to be parsed out > with tokenize. But tokenize is giving the wrong results (or I am > doing something wrong) for this (pathological) case: > > foo

[Python-Dev] tokenize string literal problem

2009-10-23 Thread C or L Smith
BACKGROUND I'm trying to modify the doctest DocTestParser so it will parse docstring code snippets out of a *py file. (Although doctest can parse these with another method out of *pyc, it is missing certain decorated functions and we would also like to insist of import of needed modules rath