[issue14304] Implement utf-8-bmp codec

Change by STINNER Victor : -- nosy: -vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue14304] Implement utf-8-bmp codec

Terry J. Reedy added the comment: October 2019, Serhiy solved the display issue with a _tkinter patch for #13153. bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545) https://github.com/python/cpython/commit/06cb94bc8419b9a24df6b0d724fcd8e40c6971d6 In Windows

[issue14304] Implement utf-8-bmp codec

Terry J. Reedy added the comment: In 3.6, Python's use of the Windows console was changed to work much better with unicode. As a result, IDLE is now worse rather than better than the console on Windows. I plan to do something before 3.7.0. -- components: +IDLE versions: +Python 3.6,

[issue14304] Implement utf-8-bmp codec

Changes by Alexander Belopolsky alexander.belopol...@gmail.com: -- nosy: +belopolsky ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14304 ___ ___

[issue14304] Implement utf-8-bmp codec

Terry J. Reedy added the comment: Pending doing some experiments with current and patched code, and reading the rpc code, I believe I would like to see the patch applied. I don't care about whether the patch defines a 'codec' or what its name would be. What i do want is for the Idle Shell to

[issue14304] Implement utf-8-bmp codec

STINNER Victor added the comment: Tkinter (and IDLE specially) can use only UCS-2 characters. Is it always the case, or does depend on a compilation flag of Tcl or Tk? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14304

[issue14304] Implement utf-8-bmp codec

Serhiy Storchaka added the comment: In theory Tcl/Tk can be built with 32-bit Tcl_Char. But I doubt that this option is well tested. In any case on Linux Python depends on system Tcl/Tk. -- ___ Python tracker rep...@bugs.python.org

[issue14304] Implement utf-8-bmp codec

STINNER Victor added the comment: In theory Tcl/Tk can be built with 32-bit Tcl_Char. Would it make sense to compile Tcl/Tk with 32-bit Tcl_Char on Windows? I think that we embed our own build ot Tcl/Tk, right? -- ___ Python tracker

[issue14304] Implement utf-8-bmp codec

Serhiy Storchaka storch...@gmail.com added the comment: Any chance to commit the patch before final feature freeze? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14304 ___

[issue14304] Implement utf-8-bmp codec

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14304 ___

[issue14304] Implement utf-8-bmp codec

Martin v. Löwis mar...@v.loewis.de added the comment: The patch is incorrect, i.e. it deviates from what the command line interface does. When you try to write to sys.stdout, and the characters are not supported you get UnicodeError. Only when it is interactive mode, and tries to represent

[issue14304] Implement utf-8-bmp codec

Serhiy Storchaka storch...@gmail.com added the comment: I don't see what the patch worse than the current behavior. Unpatched: ''.join(map(chr, [76, 246, 119, 105, 115])) 'Löwis' ''.join(map(chr, [76, 246, 119, 105, 115, 65536])) 'L\xf6wis\U0001' Patched: ''.join(map(chr, [76, 246, 119,

[issue14304] Implement utf-8-bmp codec

Martin v. Löwis mar...@v.loewis.de added the comment: In the case of the Cyrillic alphabet all text becomes unreadable, if there are some non-bmp characters in it. And indeed, that's the correct, desired behavior, as it models what the interactive shell does. If you want to change this,

[issue14304] Implement utf-8-bmp codec

Martin v. Löwis mar...@v.loewis.de added the comment: I take that back; the interactive shell uses the backslashescape error handler. Still, I don't think IDLE should setup a displayhook in the first place. What if an application replaces the displayhook? --

[issue14304] Implement utf-8-bmp codec

Serhiy Storchaka storch...@gmail.com added the comment: Still, I don't think IDLE should setup a displayhook in the first place. What if an application replaces the displayhook? IDLE *is* the application. If another application that uses the idlelib, replace displayhook, it must itself to

[issue14304] Implement utf-8-bmp codec

Andrew Svetlov andrew.svet...@gmail.com added the comment: Serhiy, I like to fix tkinter itself, not only IDLE. There are other problems like idle is crashing if non-bmp char will be pasted from clipboard. Moreover, non-bmp behavior is different from one Tk widget to other. I still want to make

[issue14304] Implement utf-8-bmp codec

Martin v. Löwis mar...@v.loewis.de added the comment: IDLE *is* the application. No, IDLE is the development environment. The application is whatever is being developed with IDLE. -- ___ Python tracker rep...@bugs.python.org

[issue14304] Implement utf-8-bmp codec

Serhiy Storchaka storch...@gmail.com added the comment: I don't understand how the utf-8-bmp codec will help to fix the tkinter. To fix the tkinter, you need to fix the Tcl/Tk, but it is outside of Python. While Tcl does not support non-bmp characters, correct and non-ambiguous working with

[issue14304] Implement utf-8-bmp codec

Serhiy Storchaka storch...@gmail.com added the comment: No, IDLE is the development environment. The application is whatever is being developed with IDLE. If the application replaces the displayhook, than it is the development environment too. --

[issue14304] Implement utf-8-bmp codec

Serhiy Storchaka storch...@gmail.com added the comment: Andrew, imagine that the utf-8-bmp codec is already there (I will do it for you, if I see its necessity). How are you going to use it? Show a patch that fixes IDLE and tkinter using this codec. It seems to me that any result can be achieved

[issue14304] Implement utf-8-bmp codec

Serhiy Storchaka storch...@gmail.com added the comment: Andrew, the patch solves your issue? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14304 ___

[issue14304] Implement utf-8-bmp codec

Martin v. Löwis mar...@v.loewis.de added the comment: This codec is one that is equal to UTF-8, but restricted to the BMP. For non-BMP character, the error handler is called. It will be the stdout codec for the IDLE interactive shell, causing non-BMP results to be ascii() escaped. --

[issue14304] Implement utf-8-bmp codec

Andrew Svetlov andrew.svet...@gmail.com added the comment: Tkinter (as Tcl itself) has no support of non-BMP characters in any form. It looks like support of UTF-16 without surrogates. I like to implement codec for that which will process different error modes (strict, replace, ignore etc) as

[issue14304] Implement utf-8-bmp codec

Serhiy Storchaka storch...@gmail.com added the comment: Example: '\u0100' 'Ā' '\u0100\U0001' '\u0100\U0001' print('\u0100') Ā print('\u0100\U0001') Traceback (most recent call last): File pyshell#33, line 1, in module print('\u0100\U0001') UnicodeEncodeError: 'UCS-2'

[issue14304] Implement utf-8-bmp codec

Andrew Svetlov andrew.svet...@gmail.com added the comment: The way is named 'codec'. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14304 ___

[issue14304] Implement utf-8-bmp codec

Martin v. Löwis mar...@v.loewis.de added the comment: But I think that it is too specific problem and too specific solution. It would be better if IDLE itself escapes the string in the most appropriate way. That is not implementable correctly. If you think otherwise, please submit a patch.

[issue14304] Implement utf-8-bmp codec

Serhiy Storchaka storch...@gmail.com added the comment: May be I did not correctly understand the problem, but I can assume, that this patch solves it. 'Агов!\U0001' -- keywords: +patch Added file: http://bugs.python.org/file25244/idle_escape_nonbmp.patch

[issue14304] Implement utf-8-bmp codec

Serhiy Storchaka storch...@gmail.com added the comment: Sorry, the mail daemon has eaten a piece of example. '\u0410\u0433\u043e\u0432!\U0001' 'Агов!\U0001' -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14304

[issue14304] Implement utf-8-bmp codec

STINNER Victor victor.stin...@gmail.com added the comment: What is this codec? What do you mean by escpe non-ascii? -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14304 ___

[issue14304] Implement utf-8-bmp codec

Changes by Ezio Melotti ezio.melo...@gmail.com: -- nosy: +ezio.melotti ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14304 ___ ___

[issue14304] Implement utf-8-bmp codec

Serhiy Storchaka storch...@gmail.com added the comment: ''.join(c if ord(c) 0x1 else escape(c) for c in s) -- nosy: +storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14304 ___

[issue14304] Implement utf-8-bmp codec

Antoine Pitrou pit...@free.fr added the comment: The solution outlined in the issue title (utf-8-bmp codec) sounds like a rather dubious idea. -- nosy: +loewis, pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14304

[issue14304] Implement utf-8-bmp codec

Martin v. Löwis mar...@v.loewis.de added the comment: pitrou: can you elaborate? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14304 ___ ___

[issue14304] Implement utf-8-bmp codec

New submission from Andrew Svetlov andrew.svet...@gmail.com: Tkinter (and IDLE specially) can use only UCS-2 characters. In PyShell IDLE tries to escape non-ascii. To better result we should to escape only non-BMP chars leaving BMP characters untouched. -- assignee: asvetlov messages:

[issue14304] Implement utf-8-bmp codec

Changes by Andrew Svetlov andrew.svet...@gmail.com: -- components: +Tkinter ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14304 ___ ___

[issue14304] Implement utf-8-bmp codec

Changes by Roger Serwy roger.se...@gmail.com: -- nosy: +serwy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14304 ___ ___ Python-bugs-list mailing