[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
Terry J. Reedy added the comment: From msg248192 (Serhiy - which ... File /home/serhiy/py/cpython/Lib/idlelib/EditorWindow.py, line 899, in update_recent_files_list if '\0' in path or not os.path.exists(path[0:-1]): File /home/serhiy/py/cpython/Lib/genericpath.py, line 19, in exists os.stat(path) UnicodeEncodeError: 'latin-1' codec can't encode character '\U0001d53c' in position 22: ordinal not in range(256) On Windows, os.stat(astralpath) raises FileNotFoundError: [WinError 2] The system cannot find the file specified: 'as\U0001.py' and os.path.exists(astralpath) catches the exception and returns False. It is a linux issue, and perhaps a bug, that os.path.exists does not catch the exception. To me, as I read the docstring, it should always return True or False, with the latter the default, for any input string. The EditorWindow line is testing for 'badfiles' to be excluded from the recent files list. We can work around the linux behavior with try-except. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23672 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
Roundup Robot added the comment: New changeset d54aa163e4ec by Terry Jan Reedy in branch '2.7': Issue #23672: ACKS https://hg.python.org/cpython/rev/d54aa163e4ec New changeset c1031eb12aa1 by Terry Jan Reedy in branch '3.4': Issue #23672: ACKS https://hg.python.org/cpython/rev/c1031eb12aa1 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23672 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
Terry J. Reedy added the comment: It appears that the failures in msg248192 and msg248365 are issues with non-latin1 chars in general, not with astral chars in particular. Anyone who wants filenames with astral chars should be using a utf-8 locale. This issue is about Idle working around the tk BMP limit where it can so as to not prevent editing and running files with system-legal names. According to msg248188, the patch works. So I am closing the issue as fixed. -- assignee: - terry.reedy resolution: - fixed stage: commit review - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23672 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
sanad added the comment: These observations are noted when the system locale is set to 'iso-88591' 1. Ok, for some reason I'm able to execute this command without any error in Linux (idle window is opening with a file name as 'astralE.py'). Because the character '피' is automatically being shown and treated as 'E' in both terminal and python command line interpreter(similar for chars '풵' = 'z' and '풫' = 'P'). 2. But i'm unable to save/make a file with filname 'astral피.py' and hence unable to run it. The following errors are thrown then : Exception in Tkinter callback Traceback (most recent call last): File /home/sanad/devpy/pessoc/cpython/Lib/tkinter/__init__.py, line 1549, in __call__ return self.func(*args) File /home/sanad/devpy/pessoc/cpython/Lib/idlelib/MultiCall.py, line 176, in handler r = l[i](event) File /home/sanad/devpy/pessoc/cpython/Lib/idlelib/IOBinding.py, line 339, in save self.save_as(event) File /home/sanad/devpy/pessoc/cpython/Lib/idlelib/IOBinding.py, line 353, in save_as if self.writefile(filename): File /home/sanad/devpy/pessoc/cpython/Lib/idlelib/IOBinding.py, line 379, in writefile with open(filename, wb) as f: UnicodeEncodeError: 'ascii' codec can't encode character '\U0001d53c' in position 39: ordinal not in range(128) I'm trying to figure out the root of the problem, feel free to give your inputs -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23672 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
Serhiy Storchaka added the comment: Interesting, this doesn't work with non-UTF-8 locale. $ touch astral피.py $ LC_ALL=en_US.iso88591 ./python -m idlelib.idle -e astral피.py Traceback (most recent call last): File /home/serhiy/py/cpython/Lib/runpy.py, line 170, in _run_module_as_main __main__, mod_spec) File /home/serhiy/py/cpython/Lib/runpy.py, line 85, in _run_code exec(code, run_globals) File /home/serhiy/py/cpython/Lib/idlelib/idle.py, line 11, in module idlelib.PyShell.main() File /home/serhiy/py/cpython/Lib/idlelib/PyShell.py, line 1553, in main if flist.open(filename) is None: File /home/serhiy/py/cpython/Lib/idlelib/FileList.py, line 36, in open edit = self.EditorWindow(self, filename, key) File /home/serhiy/py/cpython/Lib/idlelib/PyShell.py, line 123, in __init__ EditorWindow.__init__(self, *args) File /home/serhiy/py/cpython/Lib/idlelib/EditorWindow.py, line 288, in __init__ if io.loadfile(filename): File /home/serhiy/py/cpython/Lib/idlelib/IOBinding.py, line 244, in loadfile self.updaterecentfileslist(filename) File /home/serhiy/py/cpython/Lib/idlelib/IOBinding.py, line 525, in updaterecentfileslist self.editwin.update_recent_files_list(filename) File /home/serhiy/py/cpython/Lib/idlelib/EditorWindow.py, line 899, in update_recent_files_list if '\0' in path or not os.path.exists(path[0:-1]): File /home/serhiy/py/cpython/Lib/genericpath.py, line 19, in exists os.stat(path) UnicodeEncodeError: 'latin-1' codec can't encode character '\U0001d53c' in position 22: ordinal not in range(256) -- nosy: +serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23672 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
sanad added the comment: Hey Terry, after testing the committed patch on my Linux Mint, I have found the following behaviour: 1. The issue of IDLE not starting when Recent File list has name outside BMP has been fixed. 2. The File Name is correctly formatted and displayed in the file editor window title and in the recent file list.(astral chars replaced by diamond question mark symbol) 3. Files with astral char names now RUN perfectly. 4. You can create a file with astral char in its name and run it effectively and re run it from the recent file list. PS : since you are unable to test it own windows and many other developers might face the same I'm uploading the screenshot of the IDLE windows after the patch is applied. In the given test , I had kept my file name as 피피hello풵My피Name풫.py and it was displayed as ��hello�My�Name�.py -- Added file: http://bugs.python.org/file40143/idlerun.png ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23672 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
Terry J. Reedy added the comment: I made a different fix for avoid the error posted when running. Sanad, PLEASE test running a file with astral char, the same way you did before, to see is there are any other problems. I cannot get such a file into an Idle editor on Windows. I *think* this patch is enough, but cannot be sure. I am leaving this open both for your test and a possible 2.7 backport. -- stage: test needed - commit review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23672 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
Roundup Robot added the comment: New changeset dda625798111 by Terry Jan Reedy in branch '3.4': Issue #23672: Allow Idle to edit and run files with astral chars in name. https://hg.python.org/cpython/rev/dda625798111 New changeset 97d50e6247e1 by Terry Jan Reedy in branch '3.5': Issue #23672:Merge with 3.4 https://hg.python.org/cpython/rev/97d50e6247e1 New changeset 180bfaa7cdf8 by Terry Jan Reedy in branch 'default': Issue #23672:Merge with 3.5 https://hg.python.org/cpython/rev/180bfaa7cdf8 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23672 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
Terry J. Reedy added the comment: The problem with astral chars and open/save dialogs is the subject of #21084. The problem with printing astral chars generated by programs is #22742 and maybe #21084. I added a fix for the very new display of filenames on the run separator bar, which is the problem discovered above. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23672 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
sanad added the comment: On the lines of changes proposed by Terry and others in the comments, this is the patch I'm submitting. The function _filename_to_unicode() plays the major part in the plot. The function can get two types of filename ,'str' and 'bytes'. When its str, all the astral chars(characters that are outside BMP ) are replaced by the '�'(diamond question mark) character by performing a regex substitution in the return statement. If the received filename is in the form of b'..' (bytes) or any other encoding ,it is first decoded(as was earlier) into str and before returning, all the out of range Unicode characters are replaced by the '�'(diamond question mark) character . The effect on behavior is : the 1.IDLE is able to display correctly,the filename in the title-bar and in the file open dialog. 2. Any file with name that have astral chars in them are easily imported without any crash. This is my first patch,please review it ,if any errors found I will correct them and upload again :) -- keywords: +patch nosy: +sanad Added file: http://bugs.python.org/file40098/issue23672(updated third).patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23672 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
sanad added the comment:
Correction:
This patch fixes the problem of idle not opening when recent file list has
filenames outside BMP and the crashing of idle on using filenames with astral
characters.
The added benefit is that you can create a file with such chars and save it and
access it again from the Recent Files List.
What this patch doesn't fix is :
1. The File Open Dialog displays the filename incorrectly.
2. The File Open Dialog doesn't allows the file with such filename to be opened.
3. When trying the to run the file with such name, it gives an Traceback to
Tkinter. The following is the error message :
Exception in Tkinter callback
Traceback (most recent call last):
File /home/sanad/devpy/pessoc/cpython/Lib/tkinter/__init__.py, line 1549,
in __call__
return self.func(*args)
File /home/sanad/devpy/pessoc/cpython/Lib/idlelib/ScriptBinding.py, line
124, in run_module_event
return self._run_module_event(event)
File /home/sanad/devpy/pessoc/cpython/Lib/idlelib/ScriptBinding.py, line
145, in _run_module_event
interp.restart_subprocess(with_cwd=False, filename=code.co_filename)
File /home/sanad/devpy/pessoc/cpython/Lib/idlelib/PyShell.py, line 502, in
restart_subprocess
console.write(\n{0} {1} {0}.format(halfbar, tag))
File /home/sanad/devpy/pessoc/cpython/Lib/idlelib/PyShell.py, line 1294, in
write
'Non-BMP character not supported in Tk')
UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 49-49:
Non-BMP character not supported in Tk
Which I guess is another related issue.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23672
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
Terry J. Reedy added the comment: _filename_to_unicode returns unicode names as is. In 3.x, are filenames ever not unicode? They come from either a file dialog (via tk, hence unicode), or sys.argv. I do not know about the latter, which is possibly OS-dependent. This function is used in three places in idlelib, all within EditorWindow: 1.def update_recent_files_list(self, new_file=None): ... ufile_name = self._filename_to_unicode(file_name) 2.def short_title(self): # reduce filename to basename return self._filename_to_unicode(filename) 3.def long_title(self): return self._filename_to_unicode(self.io.filename or ) The _f2u output is not saved to disk or used to open files; it is display only. So replacing astral chars with either \Un escapes or the BMP box char should be fine. 1. The callback associated with each ufile_name encloses the original file_name, which is used to open the file. The original filename is also saved back to disk before the _f2u call. 23. The titles are display only. WindowList.py displays the long name for editor windows, but the callback is a wakeup function tied to the Window itself. -- stage: needs patch - test needed versions: +Python 3.6 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23672 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
Ned Deily added the comment: At least on some platforms (e.g. OS X), it is easy to create files with legitimate names containing code points above the BMP limit (= U+) currently imposed by Tcl/Tk. For IDLE 3, I suspect _filename_to_unicode() in EditorWindow could be modified to check for such cases to prevent problems if such file names end up in recent-files.lst. That might not be sufficient: there may be other problematic places. I also was able to crash a current IDLE 2.7 just opening a file with such a name. -- stage: - needs patch title: IDLE for osx 10.8.5 won't run - IDLE can crash if file name contains non-BMP Unicode characters versions: +Python 2.7, Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23672 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
Changes by Ned Deily n...@acm.org: -- nosy: +roger.serwy, terry.reedy -ned.deily ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23672 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
Terry J. Reedy added the comment: The full path of a file being edited also ends up in the title bar and the Window menu. I do not know whether the title bar is displayed by tk or the OS (Windows obviously displays the title of taskbar icons) but the Window list is definitely by tk. It seems to me that files need two names: the system name used to open (and re-open) a file (utf-8 bytes on *nix?) and a tk name (BMD unicode) for display in the various places. The current limitation to BMP names is a limitation of the tcl/tk gui framework. I would like to add a workaround but do not consider its absence a bug. I am proposing on python-list the addition of some builtin means to replace non-BMP chars with \U000x escapes for display purposes. This would be generally useful for tkinter programming. The thread is Add str.bmp() to only expand non-BMP chars, for tkinter use. If this does not happen in 3.5, I would consider a patch to add a private function to EditorWindow.py to do the same. It would be far less efficient, but fast enough for short path names. The EditorWindow.py line numbers are slight different from those in 3.4.3 (and 3.4.2, I believe), so I presume this is with 3.4.0 or 3.4.1. The result with 3.4.3 should be unchanged. -- type: behavior - enhancement versions: -Python 2.7, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23672 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
Terry J. Reedy added the comment: Thinking more, there are two issue here. One is the fact that Idle stops when fed a filename with astral chars. This *is* a bug and should be fixed in all versions, even if that fix is to display a message box saying that Idle cannot work properly with such files. The second is the one I addressed in the first message, the inability to work properly with files. Fixing that would obviate the need to display a message, but might be more work, especially on 2.7. Kamisky, if you launch Idle and then try to open the file, do you see the name in the Open dialog? I presume that if you do, and select it, Idle would stop just as it did in your report. -- type: enhancement - behavior versions: +Python 2.7, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23672 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23672] IDLE can crash if file name contains non-BMP Unicode characters
Changes by Raymond Hettinger raymond.hettin...@gmail.com: -- priority: normal - high ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23672 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
