[issue24194] tokenize yield an ERRORTOKEN if an identifier uses Other_ID_Start or Other_ID_Continue

Meador Inge added the comment: Attached is a first cut patch for this. (CC'd haypo as a unicode expert). -- assignee: -> meador.inge keywords: +patch nosy: +haypo, meador.inge stage: needs patch -> patch review Added file: http://bugs.python.org/file42808/issue24194-v0.patch _

[issue24194] tokenize yield an ERRORTOKEN if an identifier uses Other_ID_Start or Other_ID_Continue

Changes by Serhiy Storchaka : -- dependencies: +Request for property support in Python re lib, python lib re uses obsolete sense of \w in full violation of UTS#18 RL1.2a ___ Python tracker

[issue24194] tokenize yield an ERRORTOKEN if an identifier uses Other_ID_Start or Other_ID_Continue

Changes by Serhiy Storchaka : -- stage: -> needs patch versions: +Python 3.5, Python 3.6 -Python 3.4 ___ Python tracker ___ ___ Pytho

[issue24194] tokenize yield an ERRORTOKEN if an identifier uses Other_ID_Start or Other_ID_Continue

New submission from Joshua Landau: This is valid: ℘· = 1 print(℘·) #>>> 1 But this gives an error token: from io import BytesIO from tokenize import tokenize stream = BytesIO("℘·".encode("utf-8")) print(*tokenize(stream.read), sep="\n") #>>> TokenInfo(type=56 (