Hi All, I was going through some of the open issues related to 'tokenize' and ran across 'issue2180'. The reproduction case for this issue is along the lines of:
>>> tokenize.tokenize(io.StringIO("if 1:\n \\\n #hey\n print 1").readline) but, with 'py3k' I get: >>> tokenize.tokenize(io.StringIO("if 1:\n \\\n #hey\n print 1").readline) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/minge/Code/python/py3k/Lib/tokenize.py", line 360, in tokenize encoding, consumed = detect_encoding(readline) File "/Users/minge/Code/python/py3k/Lib/tokenize.py", line 316, in detect_encoding if first.startswith(BOM_UTF8): TypeError: Can't convert 'bytes' object to str implicitly which, as seen in the trace, is because the 'detect_encoding' function in 'Lib/tokenize.py' searches for 'BOM_UTF8' (a 'bytes' object) in the string to tokenize 'first' (a 'str' object). It seems to me that strings should still be able to be tokenized, but maybe I am missing something. Is the implementation of 'detect_encoding' correct in how it attempts to determine an encoding or should I open an issue for this? --- Meador
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com