[issue19091] ast.parse gives wrong position for some Names when non-ascii characters occur before

Changes by Terry J. Reedy tjre...@udel.edu: -- stage: - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19091 ___

[issue19091] ast.parse gives wrong position for some Names when non-ascii characters occur before

New submission from Aivar Annamaa: ast.parse gives col_offset=4 for Name x when given program a + x vs. col_offset=5 for x when Name a is replaced with a-umlaut (I can't write that character here, because it seems that the issue system doesn't handle non-ascii characters either). See the

[issue19091] ast.parse gives wrong position for some Names when non-ascii characters occur before

Serhiy Storchaka added the comment: This looks similar to issue2382 and issue10382. -- nosy: +serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19091 ___

[issue19091] ast.parse gives wrong position for some Names when non-ascii characters occur before

Aivar Annamaa added the comment: Serhiy, it's similar in that it has to do with encodings, but I think it's caused by different bug. I suspect it has something to do with the fact that tokenizer gives positions as offsets of characters and ast as offsets of UTF-8 bytes. --

[issue19091] ast.parse gives wrong position for some Names when non-ascii characters occur before

Aivar Annamaa added the comment: I should explain more -- the string containing the program is read in correctly, the trouble occurs during parse. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19091

[issue19091] ast.parse gives wrong position for some Names when non-ascii characters occur before

Aivar Annamaa added the comment: ast [gives positions as] offsets of UTF-8 bytes Oops, I'm sorry, I realized only now that this is the explanation of this behaviour. So, after all it seems to be a feature (albeit very weird), not a bug. -- resolution: - invalid