Michael Foord <mich...@voidspace.org.uk> added the comment: Note from Nick Coghlan from the Python-dev discussion:
A very quick scan of _tokenize suggests it is designed to support detect_encoding returning None to indicate the line iterator will return already decoded lines. This is confirmed by the fact the standard library uses it that way (via generate_tokens). An API that accepts a string, wraps a StringIO around it, then calls _tokenise with an encoding of None would appear to be the answer here. A feature request on the tracker is the best way to make that happen. ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue9969> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
