[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[Terry J. Reedy]

Terry J. Reedy  added the comment:

As noted on #13153, files with astral chars can now be read without an 
exception, but the presence of astral chars messes up editing text that follows 
at least on the same line by misplacing the cursor.  I will open a new issue 
about replacing such with \U escapes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Fixed by PR 16545 (see issue13153).

--
resolution:  -> fixed
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[buhtz]

Change by buhtz :


--
nosy: +buhtz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[Mark Lawrence]

Changes by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[Terry J. Reedy]

Terry J. Reedy added the comment:

Tk Text (and other widgets, but Text is the main issue) has two display 
problems: astral chars and long lines (over a thousand chars, say).  These 
problems can manifest in various places: file names, shell input (keyboard or 
clipboard), shell output, editor input (keyboard, clipboard, or file).  IDLE 
needs to take more control over what is displayed to work around both problems.

Tk Text also has a display feature: substring tagging.  I have been heistant to 
simple replace astral chars with their \U000h expansion because of the 
aliasing problem: in shell output, for instance, the user would not know if the 
program wrote 1 char or 10.  It would also be impossible to know if a reverse 
transformation might be needed.  Tagging astral expansions would solve both 
problems.

import re

astral = re.compile(r'([^\x00-\u])')
s = 'X\U0001Y\U0002\U0003Z'
for i, ss in enumerate(re.split(astral, s)):
if not i%2:
print(ss, end='')
else:
print(r'\\U%08x' % ord(ss), end='')
# prints
X\\U0001Y\\U0002\\U0003Z

Now replace print with test.insert, with an 'astral' tag for the second.  tk 
will not double '\'s.  Astral tag could switch, for instance, to underline 
version of current font.  This should work with any color scheme.

[Separate but related issue: augment Format or context menu with functions to 
convert between literal char, escape string, and name representation (using 
unicodedatabase).]

--
nosy: +terry.reedy
resolution: duplicate -> 
versions: +Python 3.6 -Python 2.7, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[irdb]

Changes by irdb :


--
nosy: +irdb

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[wjssz]

wjssz added the comment:

I wrote this code, but I don't know how to make a patch.

Insert these codes in C:\Python34\Lib\idlelib\IOBinding.py
Around line 234, before this line:
self.text.delete(1.0, end)


# check non-bmp characters
line_count = 1
position_count = 1
for char in chars:
if char == '\n':
line_count += 1
position_count = 1
if ord(char)  0x:
nonbmp_msg = (IDLE can't display non-BMP characters 
  (codepoint above 0x).\n
  A non-BMP character found at line %d, 
  position %d of file %s, codepoint 0x%X.\n
  Please open this file with another editor.)
tkMessageBox.showerror(non-BMP character,
nonbmp_msg %
   (line_count, position_count,
filename, ord(char)),
   parent=self.text)
return False
position_count += 1

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21084
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[wjssz]

wjssz added the comment:

Changing the second if to elif is better.

I'm sorry, I have never submitted patch.
If somebody gives a hand, feel free to modify those codes.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21084
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[Ezio Melotti]

Ezio Melotti added the comment:

See https://docs.python.org/devguide/patch.html

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21084
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[wjssz]

wjssz added the comment:

Feel free to modify this patch.

--
Added file: http://bugs.python.org/file36080/nonbmp_except_check.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21084
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[wjssz]

wjssz added the comment:

nonbmp_except_check_v2.patch changes character numbers to 0-based, same as IDLE.

Quote from www.tkdocs.com :
for historical conventions related to how programmers normally refer to lines 
and characters, line numbers are 1-based, and character numbers are 0-based.

--
Added file: http://bugs.python.org/file36082/nonbmp_except_check_v2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21084
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[Serhiy Storchaka]

Changes by Serhiy Storchaka storch...@gmail.com:


--
stage:  - needs patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21084
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[wjssz]

wjssz added the comment:

I suggest don't change the content of file, just give a message such as:

IDLE can't display non-BMP character (codepoint above 0x).
A non-BMP character found in Line 23, position 8 of .py, please open this 
file with other editor.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21084
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[Raymond Hettinger]

Changes by Raymond Hettinger raymond.hettin...@gmail.com:


--
priority: normal - high

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21084
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Yes, this is very similar to issue13153. Both these issues can have same 
solution or can have different solutions. This issue relates to more realistic 
situation and therefore is more important.

Here is simple and almost working solution for this issue. Unfortunately it 
works incorrectly when astral characters are encountered in raw string 
literals. More mature solution should parse sources and convert raw string 
literals containing astral characters to non-raw string literals. But this will 
not work with invalid Python files and non-Python files.

I afraid this issue has not perfect solution. The question is which imperfect 
solution and compromise we will decided enough acceptable.

--
assignee:  - serhiy.storchaka
components: +Tkinter, Unicode
keywords: +patch
nosy: +haypo
versions: +Python 2.7, Python 3.5 -Python 3.3
Added file: http://bugs.python.org/file35929/idle_fix_non_bmp.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21084
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[Mark Lawrence]

Mark Lawrence added the comment:

Accidentally set to pending I take it.

--
nosy: +BreamoreBoy
status: pending - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21084
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[wjssz]

New submission from wjssz:

When open a file with characters above the range (U+-U+), IDLE quit 
without any report. For example, open this file \Lib\test\test_re.py

The below is Traceback info, the last line tells the reason. I just hope IDLE 
say something before quit, so we can know what happend.

I have checked Python 3.3.5 and 3.4.0, they have the same problem. I didn't 
find a 3.5 build, so I can't test this problem under 3.5.

=
Exception in Tkinter callback
Traceback (most recent call last):
  File C:\Python33\lib\tkinter\__init__.py, line 1489, in __call__
return self.func(*args)
  File C:\Python33\lib\idlelib\IOBinding.py, line 186, in open
flist.open(filename)
  File C:\Python33\lib\idlelib\FileList.py, line 36, in open
edit = self.EditorWindow(self, filename, key)
  File C:\Python33\lib\idlelib\PyShell.py, line 126, in __init__
EditorWindow.__init__(self, *args)
  File C:\Python33\lib\idlelib\EditorWindow.py, line 288, in __init__
if io.loadfile(filename):
  File C:\Python33\lib\idlelib\IOBinding.py, line 236, in loadfile
self.text.insert(1.0, chars)
  File C:\Python33\lib\idlelib\Percolator.py, line 25, in insert
self.top.insert(index, chars, tags)
  File C:\Python33\lib\idlelib\UndoDelegator.py, line 81, in insert
self.addcmd(InsertCommand(index, chars, tags))
  File C:\Python33\lib\idlelib\UndoDelegator.py, line 116, in addcmd
cmd.do(self.delegate)
  File C:\Python33\lib\idlelib\UndoDelegator.py, line 219, in do
text.insert(self.index1, self.chars, self.tags)
  File C:\Python33\lib\idlelib\ColorDelegator.py, line 85, in insert
self.delegate.insert(index, chars, tags)
  File C:\Python33\lib\idlelib\WidgetRedirector.py, line 104, in __call__
return self.tk_call(self.orig_and_operation + args)
_tkinter.TclError: character U+1d518 is above the range (U+-U+) allowed 
by Tcl

--
components: IDLE
messages: 215038
nosy: wjssz
priority: normal
severity: normal
status: open
title: IDLE can't deal with characters above the range (U+-U+)
type: crash
versions: Python 3.3, Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21084
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[wjssz]

wjssz added the comment:

When open a file with characters above the range (U+-U+), IDLE quit 
without any report. For example, open this file C:\Python33\lib\test\test_re.py

The below is Traceback info, the last line tells the reason. I just hope IDLE 
say something before quit, so we can know what happend.

I have checked Python 3.3.5 and 3.4.0, they have the same problem. I didn't 
find a 3.5 build, so I can't test this problem under 3.5.

=
Exception in Tkinter callback
Traceback (most recent call last):
  File C:\Python33\lib\tkinter\__init__.py, line 1489, in __call__
return self.func(*args)
  File C:\Python33\lib\idlelib\IOBinding.py, line 186, in open
flist.open(filename)
  File C:\Python33\lib\idlelib\FileList.py, line 36, in open
edit = self.EditorWindow(self, filename, key)
  File C:\Python33\lib\idlelib\PyShell.py, line 126, in __init__
EditorWindow.__init__(self, *args)
  File C:\Python33\lib\idlelib\EditorWindow.py, line 288, in __init__
if io.loadfile(filename):
  File C:\Python33\lib\idlelib\IOBinding.py, line 236, in loadfile
self.text.insert(1.0, chars)
  File C:\Python33\lib\idlelib\Percolator.py, line 25, in insert
self.top.insert(index, chars, tags)
  File C:\Python33\lib\idlelib\UndoDelegator.py, line 81, in insert
self.addcmd(InsertCommand(index, chars, tags))
  File C:\Python33\lib\idlelib\UndoDelegator.py, line 116, in addcmd
cmd.do(self.delegate)
  File C:\Python33\lib\idlelib\UndoDelegator.py, line 219, in do
text.insert(self.index1, self.chars, self.tags)
  File C:\Python33\lib\idlelib\ColorDelegator.py, line 85, in insert
self.delegate.insert(index, chars, tags)
  File C:\Python33\lib\idlelib\WidgetRedirector.py, line 104, in __call__
return self.tk_call(self.orig_and_operation + args)
_tkinter.TclError: character U+1d518 is above the range (U+-U+) allowed 
by Tcl

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21084
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21084] IDLE can't deal with characters above the range (U+0000-U+FFFF)

[Ezio Melotti]

Ezio Melotti added the comment:

See #13153.

--
nosy: +ezio.melotti, serhiy.storchaka
resolution:  - duplicate
status: open - pending
type: crash - behavior

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21084
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com