Matthias Bussonnier <> added the comment:

> Why not just bless the existing generate_tokens() function as a public API, 

Yes please, or just make the private `_tokenize` public under another name. The 
`tokenize.tokenize` method try to magically detect encoding which may be 

nosy: +mbussonn

Python tracker <>
Python-bugs-list mailing list

Reply via email to