[issue17125] tokenizer.tokenize passes a bytes object to str.startswith
New submission from Tyler Crompton:
Line 402 in lib/python3.3/tokenize.py, contains the following line:
if first.startswith(BOM_UTF8):
BOM_UTF8 is a bytes object. str.startswith does not accept bytes objects. I was
able to use tokenize.tokenize only after making the following changes:
Change line 402 to the following:
if first.startswith(BOM_UTF8.decode()):
Add these two lines at line 374:
except AttributeError:
line_string = line
Change line 485 to the following:
try:
line = line.decode(encoding)
except AttributeError:
pass
I do not know if these changes are correct as I have not fully tested this
module after these changes, but it started working for me. This is the meat of
my invokation of tokenize.tokenize:
import tokenize
with open('example.py') as file: # opening a file encoded as UTF-8
for token in tokenize.tokenize(file.readline):
print(token)
I am not suggesting that these changes are correct, but I do believe that the
current implementation is incorrect. I am also unsure as to what other versions
of Python are affected by this.
--
components: Library (Lib)
messages: 181349
nosy: Tyler.Crompton
priority: normal
severity: normal
status: open
title: tokenizer.tokenize passes a bytes object to str.startswith
type: behavior
versions: Python 3.3
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17125
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17125] tokenizer.tokenize passes a bytes object to str.startswith
R. David Murray added the comment: The docs could certainly be more explicit...currently they state that tokenize is *detecting* the encoding of the file, which *implies* but does not make explicit that the input must be binary, not text. The doc problem will get fixed as part of the fix to issue 12486, so I'm closing this as a duplicate. If you want to help out with a patch review and doc patch suggestions on that issue, that would be great. -- nosy: +r.david.murray resolution: - duplicate stage: - committed/rejected status: open - closed superseder: - tokenize module should have a unicode API ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17125 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
