[issue23297] Clarify error when ‘tokenize.detect_encoding’ receives text

Ben Finney added the comment: On 28-Apr-2019, Berker Peksag wrote: > The original problem has already been solved by making > tokenize.generate_tokens() public in issue 12486. I don't understand how that would affect the resolution of this issue. Isn't the correct resolution here going to

[issue23297] Clarify error when ‘tokenize.detect_encoding’ receives text

Berker Peksag added the comment: The original problem has already been solved by making tokenize.generate_tokens() public in issue 12486. However, the same exception can be raised when tokenize.open() is used with tokenize.tokenize(), because it returns a text stream:

[issue23297] Clarify error when ‘tokenize.detect_encoding’ receives text

Berker Peksag added the comment: It looks like this can also be fixed by issue 12486. -- nosy: +berker.peksag ___ Python tracker ___

[issue23297] Clarify error when ‘tokenize.detect_encoding’ receives text

Changes by Martin Panter : -- stage: -> needs patch title: ‘tokenize.detect_encoding’ is confused between text and bytes: no ‘startswith’ method on a byte string -> Clarify error when ‘tokenize.detect_encoding’ receives text type: crash -> behavior versions: