[issue44217] Tkinter/IDLE: literal astral char discombobulates text editing

[Shreyan Avigyan]

Shreyan Avigyan  added the comment:

Sorry, 1.0 and 1.1 refer to  not 1.1 and 1.2

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44217] Tkinter/IDLE: literal astral char discombobulates text editing

[Shreyan Avigyan]

Shreyan Avigyan  added the comment:

I executed the code Tal provided and yes both Serhiy and Tal you're right. It 
seems the backspace problem is related to Tcl/Tk's indexing. 1.1 and 1.2 both 
refer to . Also the dancing I told actually tells why this is happening. Since 
Tcl/Tk uses BMP it treats  also as 2 bytes. Now when we move the typing cursor 
from right to left using arrow keys the  gets split into ??. Notice there are 
two ?. So it's all coming together now.

--
components: +Tkinter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44217] Tkinter/IDLE: literal astral char discombobulates text editing

[Tal Einat]

Tal Einat  added the comment:

> It is partially an IDLE issue. The code expects that indices in Python string 
> correspond indices in Tcl string, but this is not true in case of astral 
> characters which are encoded as 2 (or maybe even 4) characters in Tcl.

It's not just that - Tk's Text widget is the indexing in the line itself wrong. 
In the string from Terry's example, which has 11 characters in a line including 
three smiley emojis, the can be fetch using t.get('1.1'), t.get('1.2') etc. 
through t.get('1.11'). t.get('1.12') returns '\n' since it is at or after the 
end of the line. So, as far as indexing is concerned, each of those emoji 
characters is treated as a single character.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44217] Tkinter/IDLE: literal astral char discombobulates text editing

[Shreyan Avigyan]

Shreyan Avigyan  added the comment:

The first and third behavior is only occurring for IDLE. I believe the third 
one is a cause of the first one. 

The second behavior of dancing is a Tcl/Tk problem.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44217] Tkinter/IDLE: literal astral char discombobulates text editing

[Erlend E. Aasland]

Erlend E. Aasland  added the comment:

STM like the most reasonable thing to do is to enhance the docs, as Terry 
suggested.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44217] Tkinter/IDLE: literal astral char discombobulates text editing

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

It is partially an IDLE issue. The code expects that indices in Python string 
correspond indices in Tcl string, but this is not true in case of astral 
characters which are encoded as 2 (or maybe even 4) characters in Tcl.

To fix it we need to translate between Python and Tcl indices every time when 
we pass indices from one language to other. It is virtually impossible to do in 
general (in Tkinter code) because there are tons of methods which return or 
accept indices. It can be fixed in IDLE specifically, but it is still a lot of 
work. I'll try to fix at least some code (backspace and highlighting).

And every Tkinter application which works with string indices and lengths and 
can support astral characters should fix it separately. It can help if helper 
function for conversion between indices in IDLE be exposed as public API in 
Tkinter.

On other hand, the problem will gone in Tcl/Tk 8.7 or 9.0. So we can just wait 
several years.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44217] Tkinter/IDLE: literal astral char discombobulates text editing

[Tal Einat]

Tal Einat  added the comment:

This is at least partly a tcl/tk issue.

Using Terry's last example, the Text widget reports that the length of the line 
is 14 (t.index('1.end') -> '1.14'), despite it only including 11 characters. It 
appears that each emoji character adds an extra character.

Minimal reproducer:

>>> t.delete('1.0', 'end')
>>> t.insert('1.0', 'a')
>>> t.index('1.end')
'1.1'
>>> t.delete('1.0', 'end')
>>> t.insert('1.0', '')
>>> t.index('1.end')
'1.2'


The same happens when using tcl/tk directly, so it's not a tkinter issue:

$ wish
% tk::text .t -width 40 -height 5  -wrap none -yscrollcommand ".ys set" 
-xscrollcommand ".xs set"
.t
% ttk::scrollbar .ys -orient vertical -command ".t yview"
.ys
% ttk::scrollbar .xs -orient horizontal -command ".t xview".t
extra characters after close-quote
% ttk::scrollbar .xs -orient horizontal -command ".t xview"
.xs
% .t insert end "a"
% grid .t -column 0 -row 0 -sticky nwes
% grid .t -column 0 -row 0 -sticky nwes
% grid .ys -column 1 -row 0 -sticky ns
% grid columnconfigure . 0 -weight 1
% grid rowconfigure . 0 -weight 1
% .t delete 1.0 end  
% .t insert end ""  
% .t index 1.end
1.2
% .t get 1.0


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44217] Tkinter/IDLE: literal astral char discombobulates text editing

[Terry J. Reedy]

Terry J. Reedy  added the comment:

This is fundamentally a tk text widget issue.  Prior to fall 2019, attempts to 
insert an astral (non-BMP) chars in tkinter widgets raised a unicode decode 
exception.  Then Serhiy fixed tkinter so that such chars could be displayed, 
either as the actual chars or as a replacement box for unavailable chars.  
There were 2 remaining problems:

1. The presence of the literal astral char in an editable widget, especially a 
Text widget, make editing past the char weird.  It can be demonstrated with, 
for instance, the following.

import tkinter as tk
r = tk.Tk()
t.insert = tk.Text(r); t.pack()
t.insert('insert', "ab'ab'ab)\n"

We decided that beings able to output such chars to the view-only part of shell 
more than made up for confusion such as Shreyan experienced.

Re-experimenting now, it appears that the behavior is worse in IDLE.  In 
particular, backspace delete does not work (nothing happens).  IDLE intercepts 
 in order to invoke its smart backspace routine.  I will try to find 
out why nothing happens.  Whatever the result, the behavior should be better 
documented.

The 8.7a3 was released Nov 2019, so I have no idea how soon 8.7 will arrive.  
Maybe the devs found enough bugs to work on, especially on Mac.  And then 
8.6.11 was needed to have tcl/tk work on the new Apple hardware and OS version.

2. On some Linux systems with some fonts with some XWindows, attempts to 
display some colored chars (such as particular emoji), in color, causes an 
XWindows error ('too complex').  This is a Linux-Xwindows-font bug.

--
nosy: +taleinat
title: [IDLE] Weird behaviour in IDLE when printing non-BMP unicode characters 
-> Tkinter/IDLE: literal astral char discombobulates text editing

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com