[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Terry J. Reedy added the comment: On Windows with 8.6.9, I see '\U0001f4bb' on 3.7.5 and '' on 3.8.0 and 3.9.0a0. I don't know why the difference as Serhiy's patch was backported. I will upgrade 3.7 and 3.8 and try again. -- ___ Python tracker

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Aivar Annamaa added the comment: >>> '\N{PERSONAL COMPUTER}' freezes IDLE 3.7.6 (64-bit, downloaded from python.org) on macOS 10.15 Can it be because Tk 8.6.8 is still used there? -- nosy: +Aivar.Annamaa ___ Python tracker

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Tal Einat added the comment: Indeed, Serhiy, you've done an amazing job with this change and it will greatly benefit many people. -- ___ Python tracker ___

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

STINNER Victor added the comment: > bpo-13153: Use OS native encoding for converting between Python and Tcl. > (GH-16545) WOW. That's huge. The issue with non-BMP characters has been fixed? Finally? The issue was haunting the bug tracker for at least 8 years!!! -- nosy: +vstinner

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Change by Serhiy Storchaka : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

miss-islington added the comment: New changeset 6c3fbbc177f5e1867ab09a315dbf58554a80accd by Miss Islington (bot) in branch '3.7': bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545)

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

miss-islington added the comment: New changeset dc191245d8f63f5ab41afff0468b7463a07e7b00 by Miss Islington (bot) in branch '3.8': bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545)

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Change by miss-islington : -- pull_requests: +16171 pull_request: https://github.com/python/cpython/pull/16580 ___ Python tracker ___

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Change by miss-islington : -- pull_requests: +16172 pull_request: https://github.com/python/cpython/pull/16581 ___ Python tracker ___

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Serhiy Storchaka added the comment: New changeset 06cb94bc8419b9a24df6b0d724fcd8e40c6971d6 by Serhiy Storchaka in branch 'master': bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545)

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Terry J. Reedy added the comment: After remembering to recompile (sorry), the test passes and clipgoard_get returns the rocket. Very nice, thank you. -- ___ Python tracker

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Serhiy Storchaka added the comment: What is the result of new tests? python.bat -m test -v -uall test_tk -m test_clipboard* -- ___ Python tracker ___

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Terry J. Reedy added the comment: What do you mean by fixed? After deleting and remaking a pr_16545 branch, I see the same result for clipboard_get. -- ___ Python tracker

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Serhiy Storchaka added the comment: Thank you for your example Terry. There was one dubious place which I did not change because I did not know how to trigger the execution of it. Now the clipboard is fixed. -- ___ Python tracker

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Terry J. Reedy added the comment: A week ago, I thought that the astral solution was to always replace with the \U escape. With this patch, we can and should send them to read-only text windows, and let the OS and font display it or a substitute. On Windows, at least, the emoji which

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Ma Lin added the comment: > Thus this breaks editing the physical line past the astral character. We > cannot do anything with this. I tried, it's sad the experience is not very good. -- nosy: +Ma Lin ___ Python tracker

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Serhiy Storchaka added the comment: >From the point of view of Tk, the astral character "" looks like either two >invisible characters "\ud83d\udcbb" or as four characters "ð\x9f\x92»" (two of >them are invisible). Thus this breaks editing the physical line past the >astral character. We

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Terry J. Reedy added the comment: The revised PR appears to fix this and other issues, although the presence of astral chars in code being edited messes up tk's cursor positioning. Assuming that this cannot be changed, we could add the the ability to replace astral chars with \U escapes.

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Serhiy Storchaka added the comment: Sorry, I did not test the last version on Windows. There was a bug which caused using the Linux version on Windows. Now it should be fixed. -- ___ Python tracker

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Tal Einat added the comment: More info: >>> '\N{PERSONAL COMPUTER}'.encode('utf-8').decode('latin-1') == '💻' True -- ___ Python tracker ___

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Tal Einat added the comment: Not sure if this helps, but a bit of experimentation brought this up: >>> '\N{PERSONAL COMPUTER}'.encode('utf-8') b'\xf0\x9f\x92\xbb' >>> '💻'.encode('utf-16le') b'\xf0\x00\x9f\x00\x92\x00\xbb\x00' >>> '💻'.encode('utf-16')

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Tal Einat added the comment: Serhiy, this looks like a great step in the right direction! Tested on Win10 with PR GH-16545 (commit f4db0e7e00). Here is a copy/paste from an IDLE shell session: >>> '\N{PERSONAL COMPUTER}' '💻' >>> print('') SyntaxError: 'utf-8' codec can't encode

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Serhiy Storchaka added the comment: PR 16545 solves the problem by using OS specific methods for converting between Python and Tcl strings. It is not ideal, but is good enough for most real cases. Now you can paste, copy and print non-BMP characters. The code containing them can be displayed

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Change by Serhiy Storchaka : -- pull_requests: +16135 pull_request: https://github.com/python/cpython/pull/16545 ___ Python tracker ___

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Serhiy Storchaka added the comment: I now have an access to Windows (I did not have it 5 years ago), so I'm going to finish this issue if I have a time. -- ___ Python tracker

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Terry J. Reedy added the comment: Recap: IDLE 3.x on Windows exits with UnicodeDecodeError when pasting into editor, grep, or shell window a non-BMP astral character such as Ң '\U000104a2', 퐇,  '\U0001F40D', or  '\U0001F431' UTF-8 b'\xf0\x9f\x90\xb1', UTF-16LI b'\x3d\xd8\x31\xdc'. Display

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Change by Terry J. Reedy : -- pull_requests: +15945 pull_request: https://github.com/python/cpython/pull/16365 ___ Python tracker ___

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Change by Terry J. Reedy : -- pull_requests: +15943 pull_request: https://github.com/python/cpython/pull/16363 ___ Python tracker ___

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Tal Einat added the comment: I can confirm that the crash from pasting these characters happens when trying to fetch the clipboard contents. We can override the built-in <> event, but then we have to get the clipboard's contents directly, and the only portable way to do that in the stdlib

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Terry J. Reedy added the comment: Another report today on idle-dev that pasting emoji exits IDLE. Serhiy, I applied the _tkinter part of your...args_2.patch to a branch of current master -- see serhiy_tkinter.patch. (Could push branch if helpful.). After recompiling _tkinter.c, pasting 

[issue13153] IDLE 3.x on Windows exits when pasting non-BMP unicode

Terry J. Reedy added the comment: Closed #37614 in favor of this. We now have only Python with FSR and mostly only tcl 8.6 to worry about. But I presume the Windows clipboard still uses uft-16le. Experimenting with pasting Ң or 'Ң', I usually get the 'ed' message as before, but with the