[issue16218] Python launcher does not support non ascii characters
Stefan Krah added the comment:
Some of the buildbots are failing with the new test:
==
FAIL: test_non_utf8 (test.test_cmd_line_script.CmdLineTest)
--
Traceback (most recent call last):
File
/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/test_cmd_line_script.py,
line 373, in test_non_utf8
importlib.machinery.SourceFileLoader)
File
/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/test_cmd_line_script.py,
line 126, in _check_script
rc, out, err = assert_python_ok(*run_args)
File
/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/script_helper.py,
line 54, in assert_python_ok
return _assert_python(True, *args, **env_vars)
File
/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/script_helper.py,
line 46, in _assert_python
stderr follows:\n%s % (rc, err.decode('ascii', 'ignore')))
AssertionError: Process return code is 1, stderr follows:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 15-20:
ordinal not in range(128)
--
Ran 23 tests in 8.959s
--
nosy: +skrah
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16218
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Changes by Jesús Cea Avión j...@jcea.es: -- nosy: +jcea ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Jesús Cea Avión added the comment: Reopening bug. Quite a few buildbots are failing with this patch. Please, commit a new version or revert. -- resolution: fixed - stage: committed/rejected - commit review status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Andrew Svetlov added the comment: I see. Sorry, my fault. Give me weekend to figure out why it fails. Thanks. -- assignee: - asvetlov ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Changes by Brian Curtin br...@python.org: -- nosy: -brian.curtin ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Serhiy Storchaka added the comment: I was not able to reproduce this error, I got other errors. The issue not in Python interpreter, the test is broken. Here is a patch that might solve the issue on some platforms (need to test on Windows). I guess failing of all command line tests when the path to temporary directory contains non-ascii. -- Added file: http://bugs.python.org/file27846/pythonrun_filename_decoding_test.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Stefan Krah added the comment:
Serhiy, your original example from msg173373 still fails on
FreeBSD:
$ name=$(printf \xff)
$ echo print('Hello, world') $name
$ ./python $name
UnicodeEncodeError: 'ascii' codec can't encode character '\xff' in position 0:
ordinal not in range(128)
[41257 refs]
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16218
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Changes by Vinay Sajip vinay_sa...@yahoo.co.uk: -- nosy: -vinay.sajip ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Serhiy Storchaka added the comment: Serhiy, your original example from msg173373 still fails on FreeBSD: Thank you for a report. I have not any ideas what happened (note that error on encoding, not decoding). Can you please show me the results of sys.getdefaultencoding(), sys.getfilesystemencoding(), locale.getpreferredencoding(True), locale.getpreferredencoding(False), the output of locale command? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Stefan Krah added the comment: This is it: sys.getdefaultencoding() 'utf-8' sys.getfilesystemencoding() 'ascii' locale.getpreferredencoding(True) 'US-ASCII' locale.getpreferredencoding(False) 'US-ASCII' $ locale LANG= LC_CTYPE=C LC_COLLATE=C LC_TIME=C LC_NUMERIC=C LC_MONETARY=C LC_MESSAGES=C LC_ALL= -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Andrew Svetlov added the comment: Perhaps we have to skip tests if filesystem encoding doesn't support wide characters. Not sure about the way: should we skip if sys.getfilesystemencoding() is not utf8 or better to try encode path and skip if it fails? I think the later is better. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Stefan Krah added the comment: On FreeBSD both Serhiy's original test case as well as the unit test work if the locale is ISO8859-15: sys.getdefaultencoding() 'utf-8' sys.getfilesystemencoding() 'iso8859-15' locale.getpreferredencoding(True) 'ISO8859-15' locale.getpreferredencoding(False) 'ISO8859-15' Naturally, if the locale is utf-8 the test works as well. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Andrew Svetlov added the comment: Looking on the last message from Stefan I think we have to check cmdpath to be encoded via sys.getfilesystemencoding() first and skip test if fails. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Stefan Krah added the comment:
That sounds good for Unix.
For Windows I'm getting a more informative error message than from the
buildbot output if I run the test via an ssh client:
==
FAIL: test_non_utf8 (test.test_cmd_line_script.CmdLineTest)
--
Traceback (most recent call last):
File C:\Users\stefan\pydev\cpython\lib\test\test_cmd_line_script.py, line
373, in test_non_utf8
importlib.machinery.SourceFileLoader)
File C:\Users\stefan\pydev\cpython\lib\test\test_cmd_line_script.py, line
129, in _check_script
expected_package, expected_loader)
File C:\Users\stefan\pydev\cpython\lib\test\test_cmd_line_script.py, line
113, in _check_output
self.assertIn(printed_file.encode('utf-8'), data)
AssertionError:
b__file__=='c:usersstefanappdatalocaltemptmpr6shx4\\udcf1\\udce
a\\udcf0\\udce8\\udcef\\udcf2.py' not found in b__loader__==class
'_frozen_importlib.SourceFileLoader'\r
\n__file__=='encoding
error'\r\n__package__==None\r\nsys.argv[0]=='c:usersstefanappdataloc
altemptmpr6shx4\\udcf1\\udcea\\udcf0\\udce8\\udcef\\udcf2.py'\r\nsys.path[0]=='c:usersst
efanappdatalocaltemptmpr6shx4'\r\ncwd=='C:Usersstefanpydevcpythonbuild\\
\\test_python_2424'\r\n
It looks to me as if on Windows perhaps some utf-8 encoding steps should
be skipped because the file name *is* unicode on Windows.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16218
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Andrew Svetlov added the comment: I will fix it tomorrow at Kiev Python sprint. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Serhiy Storchaka added the comment:
For Windows I'm getting a more informative error message than from the
buildbot output if I run the test via an ssh client:
Try with my last patch (pythonrun_filename_decoding_test.patch). It
fixes also fail on Linux with 8-bit locale.
$ LC_ALL=en_US.ISO-8859-1 LANG=en_US.ISO-8859-1 LANGUAGE= ./python -m
test -m test_non_utf8 test_cmd_line_script
[1/1] test_cmd_line_script
test test_cmd_line_script failed -- Traceback (most recent call last):
File /home/serhiy/py/cpython/Lib/test/test_cmd_line_script.py, line
373, in test_non_utf8
importlib.machinery.SourceFileLoader)
File /home/serhiy/py/cpython/Lib/test/test_cmd_line_script.py, line
129, in _check_script
expected_package, expected_loader)
File /home/serhiy/py/cpython/Lib/test/test_cmd_line_script.py, line
113, in _check_output
self.assertIn(printed_file.encode('utf-8'), data)
AssertionError: b__file__=='/tmp/tmpda64hd/\\udcf1\\udcea\\udcf0\\udce8
\\udcef\\udcf2.py' not found in b__loader__==class
'_frozen_importlib.SourceFileLoader'\n__file__=='/tmp/tmpda64hd/\\xf1\
\xea\\xf0\\xe8\\xef\\xf2.py'\n__package__==None
\nsys.argv[0]=='/tmp/tmpda64hd/\\xf1\\xea\\xf0\\xe8\\xef\
\xf2.py'\nsys.path[0]=='/tmp/tmpda64hd'\ncwd=='/home/serhiy/py/cpython/build/test_python_3546'\n
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16218
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Stefan Krah added the comment:
Serhiy Storchaka rep...@bugs.python.org wrote:
Try with my last patch (pythonrun_filename_decoding_test.patch). It
fixes also fail on Linux with 8-bit locale.
Unfortunately your last patch does not work on Windows. -- I'm too lazy
to step through the domain specific language of test_cmd_line_script.py.
Is this what is supposed to be tested:
Python 3.4.0a0 (default:b2bd62d1644f+, Nov 2 2012, 22:56:48) [MSC v.1600 64
bit (AMD64)] on win32
Type help, copyright, credits or license for more information.
s = '\udcf1\udcea\udcf0\udce8\udcef\udcf2'
f = open(s, w)
f.write('print(hello world)\n')
f.close()
C:\Users\stefan\pydev\cpythonPCbuild\amd64\python_d.exe ï�
hello world
Because that just works without the complex test machinery. :)
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16218
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Roundup Robot added the comment: New changeset 02d25098ad57 by Andrew Svetlov in branch '3.3': Issue #16218: Support non ascii characters in python launcher. http://hg.python.org/cpython/rev/02d25098ad57 New changeset 1267d64c14b3 by Andrew Svetlov in branch 'default': Merge issue #16218: Support non ascii characters in python launcher. http://hg.python.org/cpython/rev/1267d64c14b3 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Andrew Svetlov added the comment: Fixed. Thanks, Serhiy. -- nosy: +asvetlov resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Vinay Sajip added the comment: I'm not especially familiar with this code, but just trying to understand - how come filename_obj isn't decref'd on normal exit? -- nosy: +vinay.sajip ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Andrew Svetlov added the comment: Vinay, it's processed in PyObject_CallFunction(loader_type, sN, __main__, filename_obj) Please note sN format istead sO. N means PyObject* is passed but unlike sO that object is not increfed. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Vinay Sajip added the comment: Please note sN format istead sO. I see. Thanks. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Serhiy Storchaka added the comment: Test added. -- stage: test needed - patch review Added file: http://bugs.python.org/file27707/pythonrun_filename_decoding_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Serhiy Storchaka added the comment:
I can reproduce this on Linux (3.3+ only):
$ name=$(printf \xff)
$ echo print('Hello, world') $name
$ ./python $name
python: failed to set __main__.__loader__
The issue is in PyRun_SimpleFileExFlags() function, which gets raw char * as
the file name (the documentation says about the filesystem encoding
(sys.getfilesystemencoding())), but then this name decoded from UTF-8 in
set_main_loader().
--
components: +Interpreter Core -Windows
keywords: +3.3regression
nosy: +serhiy.storchaka
versions: +Python 3.4
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16218
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Serhiy Storchaka added the comment: Here is a patch which fixes filename decoding error in PyRun_SimpleFileExFlags(). -- keywords: +patch Added file: http://bugs.python.org/file27630/pythonrun_filename_decoding.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
STINNER Victor added the comment: The patch looks correct, but a test is missing. -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Changes by Ezio Melotti ezio.melo...@gmail.com: -- nosy: +ezio.melotti stage: - test needed type: crash - behavior ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Serhiy Storchaka added the comment: Where we have tests for Python launch? I can't find. runpy is not affected. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Changes by Gertjan Klein gkl...@xs4all.nl: -- nosy: +gklein ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Tim Golden added the comment: Confirming that this doesn't happen on 2.7 py -2 £.py succeeds py -3 £.py gives: python: failed to set __main__.__loader__ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Changes by Éric Araujo mer...@netwok.org: -- nosy: +brian.curtin, tim.golden ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
New submission from Turn: If there are non ASCII character in the py.exe arguments, the execution will fail. The script file name or path may contain non ASCII characters. -- components: Windows messages: 172807 nosy: turncc priority: normal severity: normal status: open title: Python launcher does not support non ascii characters type: crash versions: Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16218] Python launcher does not support non ascii characters
Changes by Jeremy Kloth jeremy.kloth+python-trac...@gmail.com: -- nosy: +jkloth ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16218 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
