New submission from Jared Grubb:
tokenize recognizes '\n' and '\r\n' as newlines, but does not tolerate '\r':
>>> s = "print 1\nprint 2\r\nprint 3\r"
>>> open('temp.py','w').write(s)
>>> exec(open('temp.py','r'))
1
2
3
>>> tokenize.tokenize(open('temp.py','r').readline)
1,0-1,5: NAME 'print'
1,6-1,7: NUMBER '1'
1,7-1,8: NEWLINE '\n'
2,0-2,5: NAME 'print'
2,6-2,7: NUMBER '2'
2,7-2,9: NEWLINE '\r\n'
3,0-3,5: NAME 'print'
3,6-3,7: NUMBER '3'
3,7-3,8: ERRORTOKEN '\r'
4,0-4,0: ENDMARKER ''
----------
components: Extension Modules
messages: 62959
nosy: jaredgrubb
severity: minor
status: open
title: tokenize: does not allow CR for a newline
type: behavior
versions: Python 2.5
__________________________________
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue2182>
__________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
