Dingyuan Wang added the comment:

The new patch should now pass all tests correctly.

The main idea is:
* if the token is INDENT, push it on the `indents` stack and continue
* if a new line starts, AND the position of the first token >= the length of 
the last indent level, we assume the indent is contained in the leading 

The new test_tokenize.py fails:


Line 1244 should be:
codelines = self.roundtrip(code).split(b'\n')

It seems that the tokens generated by tokenize.tokenize don't contain enough 
information to restore the original file.

* Tabs between tokens are not preserved.
* Space before backslash as line continuation are not preserved.

(From test/tokenize_tests.txt)
 # Backslash means line continuation:
-x = 1 \
+x = 1\
 + 1

My roundtrip test code copied here from #24447:

python2 -c 'import sys, tokenize; 
python3 -c 'import sys, tokenize; 

Added file: http://bugs.python.org/file39763/tokenize.patch

Python tracker <rep...@bugs.python.org>
Python-bugs-list mailing list

Reply via email to