[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Change by Terry J. Reedy : -- versions: +Python 3.6, Python 3.7 -Python 2.7, Python 3.4, Python 3.5 ___ Python tracker ___

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by irdb : -- nosy: +irdb ___ Python tracker ___ ___ Python-bugs-list mailing

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Terry J. Reedy added the comment: I am puzzled at the following. Some reported today that IDLE crashed when pasting the Snake emoji U+1F40D . I copied from Thunderbird and pasted in IDLE on Win10, with same UnicodeDecodeError as before. I then ran this simple code from tkinter import *

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: There is no the Snake emoji in my font, I use the Cat Face emoji U+1F431  (\xf0\x9f\x90\xb1 in UTF-8, \x3d\xd8\x31\xdc in UTF-16LE). Move cursor or press Backspace. I had needed to press Left 2 times to move cursor to the begin of the line, press Right 4

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Terry J. Reedy tjre...@udel.edu: -- type: crash - behavior versions: +Python 3.5 -Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13153 ___

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Here is updated patch (after committing issue20368 which made main refactoring). It makes PythonCmd never fail due to arguments decoding error. -- Added file: http://bugs.python.org/file34016/tkinter_pythoncmd_args_2.patch

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: As far as the patch becomes too complicated, I propose minimalist patch which fixes only this issue. I.e. IDLE will no longer silently closed when paste some unusual text (non-BMP on Windows or illegal UTF-8 on Linux). It fixes converting Tcl string to

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Martin v. Löwis added the comment: I completely lost track which problem is being solved here. Is it still IDLE crashes when pasting non-BMP unicode char on Py3? If so, how is this patch solving it? IMO, the issue shouhldn't have been reopened. Instead, a new issue should have started.

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Yes, it is still the same issue. The root of issue is in converting strings when passed to Python-implemented callbacks. When a text is pasted in IDLE window, the callback is called (for highlighting). The callback is a command created by Tcl_CreateCommand

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Martin, could you please review this patch? This bug affects not only IDLE, but any Tkinter application which uses callbacks with arguments. Encoding/decoding error during converting arguments from Tcl to Python causes immediate finishing Tcl main loop (and

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka storch...@gmail.com: -- priority: normal - high ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13153 ___ ___

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka storch...@gmail.com: -- components: +Unicode ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13153 ___ ___

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +loewis ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13153 ___ ___ Python-bugs-list

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Previous patch has a regression, it breaks decoding NUL which Tcl encodes in modified UTF-8 as \xc0\x80. However this part of code already broken, because it handles only singular NUL and not a NUL embedded in larger string. Here is a patch which also fixes

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka storch...@gmail.com: Removed file: http://bugs.python.org/file31185/tkinter_string_conv.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13153 ___

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka storch...@gmail.com: Removed file: http://bugs.python.org/file31592/tkinter_string_conv_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13153 ___

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: This bug can be reproduced on Linux too. Just copy and paste illegal UTF-8 sequence. I.e. b'\xed\xb2\x80' or b'\xc0\x80'. My patch works with first example but failed with second. When change the error handler in fromTclStringAndSize() to replace it works

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: u'\U000104a2' == u'\ud801\udca2' on narrow build. u'\ud801'.encode('utf-8', 'surrogatepass') == b'\xed\xa0\x81' u'\udca2'.encode('utf-8', 'surrogatepass') == b'\xed\xb2\xa2' Hope it will help. -- ___ Python

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Seems as Tk stores pasted \U000104a2 as surrogate pair \ud801\udca2. Then it encoded in UTF-8 as \xed\xa0\x81\xed\xb2\xa2 end passed to Python. Python converts char* to Unicode object with PyUnicode_FromString() which forbids invalid UTF-8 including encoded

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka storch...@gmail.com: Removed file: http://bugs.python.org/file31184/tkinter_string_conv.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13153 ___

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka storch...@gmail.com: Added file: http://bugs.python.org/file31185/tkinter_string_conv.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13153 ___

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Ezio Melotti added the comment: 0xed is the start byte of a 3 bytes sequence (i.e. a BMP char), and it should be followed by two continuation bytes. For some reason the traceback you pasted is missing the last part, that might provide some insight. It could be one of these:

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Terry J. Reedy added the comment: Byte 0, not byte 1, is the start byte, and it should be F0, as in output below. However, I now see invalid continuation byte'. In 2.7.5, # -*- coding: utf-8 -*- s = b'Ң' # output same if uncomment following lines #s = u'Ң'.encode('utf-8') # 'Ң' pasted in from

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka storch...@gmail.com: -- assignee: serhiy.storchaka - stage: committed/rejected - ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13153 ___

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Terry J. Reedy added the comment: In 3.3.2, 3.4.0 the traceback says that the invalid continuation byte (immediately, when 'Ң' is pasted) is ED. Snipped version is File F:\Python\dev\py33\lib\tkinter\__init__.py, line 1071, self.tk.mainloop(n) UnicodeDecodeError: 'utf-8' codec can't

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

William Schwartz added the comment: Looks like this issue is closed, but I got IDLE to crash. On Python 3.3.2, Windows 7, and Tk version 8.5, IDLE crashes when pasting \U0001F382 (Unicode birthday cake character). Below is the version string for the Python I'm running. Python 3.3.2

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Unfortunately I can't reproduce a crash on Linux. Perhaps this is Windows only issue. -- status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13153

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Terry J. Reedy added the comment: I verified problem continuing problem windows. There is only a 'crash' when running under pythonw, which does not happenon linux, as far as I know. When running on a console, the error traceback is the same as in msg145581 (with line numbers altered).

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Ned Deily added the comment: Serhiy, I think your patch is ready to commit and close this issue as it prevents the crash. A test would be nice if a reliable test could be devised without too much effort but it's not mandatory, IMO. Any tangential issues or more complex solutions can be

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Roundup Robot added the comment: New changeset bb5a8564e186 by Serhiy Storchaka in branch '2.7': Issue #13153: Tkinter functions now raise TclError instead of ValueError when http://hg.python.org/cpython/rev/bb5a8564e186 New changeset 9904f245c3f0 by Serhiy Storchaka in branch '3.2': Issue

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka storch...@gmail.com: -- assignee: asvetlov - serhiy.storchaka resolution: - fixed stage: commit review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13153

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Ned Deily added the comment: LGTM. The patch does prevent the crash in IDLE which is certainly an improvement until such time as someone investigates having Tk/tkinter fully support non-BMP characters. -- ___ Python tracker rep...@bugs.python.org

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Ramchandra Apte added the comment: @Ned Deily Tk, at least on my system, doesn't render Unicode characters, even within BMP correctly but the characters are kept (cut-and-paste works correctly) What you mean by support. -- nosy: +Ramchandra.Apte ___

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Terry J. Reedy added the comment: The characters tk can render depends on the font you tell it to use. On my Windows IDLE, I have Options Font Face set to Lucida Sans Unicode, though I am not sure what has the widest coverage. This page

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Ezio Melotti added the comment: The font used shouldn't affect the errors. Usually if a glyph is missing in the current font, either a placeholder (usually a box) is showed instead or the missing glyph is taken from another font (if possible). If you still want to do some tests, you can take

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Ned Deily added the comment: Also, there are differences in behavior among the various flavors of Tk. I know of at least four main flavors in use by current Python builds: Unix X11-based Tk 8.5, Windows Tk 8.5, OS X Cocoa Tk 8.5, OS X Carbon Tk 8.4. Some third-party distributors are

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Tkinter is not compatible with Tcl/Tk 8.6 yet (issue16809). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13153 ___

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Ned Deily added the comment: Serhiy, I'm aware of that; regardless, Tk 8.6 is starting to be used out in the field with tkinter. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13153 ___

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Ramchandra Apte added the comment: I have set it to Ubuntu, which supports the Unicode characters. Maybe Tkinter doesn't work properly with all the fonts. On 16 February 2013 01:57, Terry J. Reedy rep...@bugs.python.org wrote: Terry J. Reedy added the comment: The characters tk can render

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Ping. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13153 ___ ___ Python-bugs-list mailing list

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Ping. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13153 ___ ___ Python-bugs-list mailing list

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: A simplest solution is to raise a TclError instead of ValueError for non-BMP characters. This should not break any existing code, because a user code should be ready to catch a TclError in any case. Here is a patch. A more complicated solution is to add

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Terry J. Reedy added the comment: Same on 64-bit 3.3. Changed title since 3.3 is no longer '16-bit build'. -- title: IDLE crashes when pasting non-BMP unicode char on UCS-16 build - IDLE crashes when pasting non-BMP unicode char on Py3 versions: +Python 3.4