[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: Starting at r85691, the full test suite of Python 3.2 pass with ASCII, ISO-8859-1 and UTF-8 locale encodings in a non-ascii directory. The work on this issue is done. -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: Updated list of failing test with py3k and a non-ascii path: * Linux, LANG=C: test_gc test_gdb test_runpy test_zipimport * Windows: test_email test_httpservers test_zipimport Possible reasons: * test_httpservers (CGIHTTPServerTestCase.setUp): test should be skipped if sys.executable is not pure ASCII (and it's not possible to create ASCII path using a symlink) * test_zipimport: zipimport uses utf-8 (in strict mode) for the prefix, instead of the filesystem encoding * test_gc (test_get_count): The following two tests are fragile: ... :-/ * test_gdb: libpython doesn't support surrogates if paths * test_email: issue with the end of line (\n vs \r\n?) * test_runpy: ? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: r85655 fixed test_gdb failure. test_runpy failure looks to be linked to test_zipimport problems. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: r85659 + r85662 + r85663 fixed test_httpservers. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
R. David Murray rdmur...@bitdance.com added the comment: Victor, can you paste or attach the error for email? My MSDN subscription has expired so I can't set up to test it myself (I've submitted the renewal, but who knows how long it will take to process :) -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: Victor, can you paste or attach the error for email? It doesn't look to be related to the path name (same failure with py3ké or py3k directory name), so I opened #10134. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: Status of this issue, 5 months later: most tests pass except test_gc test_gdb test_runpy test_sys test_wsgiref test_zipimport. Said differently, 95% of the task (or more?) is done. It's possible to run Python installed in a non-ascii directory with any locale (I tested ascii, iso-8859-1 and utf-8). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
Changes by Georg Brandl ge...@python.org: -- priority: deferred blocker - release blocker ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
Georg Brandl ge...@python.org added the comment: Deferring once again. -- priority: release blocker - deferred blocker ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: See also #9713 (Py_CompileString fails on non decode-able paths) and #9738 (Document the encoding of functions bytes arguments of the C API). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
Changes by Georg Brandl ge...@python.org: -- priority: deferred blocker - release blocker ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
Georg Brandl ge...@python.org added the comment: The Unicode import system won't be put in place before 3.2a2, deferring. -- priority: release blocker - deferred blocker ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
Georg Brandl ge...@python.org added the comment: This will have to wait until after alpha1, as well. -- dependencies: +Rewrite import machinery to work with unicode paths nosy: +georg.brandl priority: release blocker - deferred blocker ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
Changes by Georg Brandl ge...@python.org: -- priority: deferred blocker - release blocker ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: I posted a patch to fix this issue: see #9425. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: See also #3080. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
Changes by Nick Coghlan ncogh...@gmail.com: -- priority: normal - release blocker ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: I think that #8988 is a duplicate of this issue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: asvetlov I'm skeptical about surrogates particularly for that asvetlov problem. From my perspective the solution is only to use asvetlov native unicode support for windows file operation functions. It's not exclusive. We can use surrogates on POSIX and then convert to bytes at the system calls, and use the unicode version of the Windows API. In both cases, filenames are unicode. asvetlov Conversions utf-8 - mbcs - utf8 will loose encoding asvetlov information thanks to tricky Microsoft mbcs encoding schema. asvetlov If I'm wrong please correct me. On Windows, Python3 *does* convert unicode to bytes with the mbcs encoding in the import machinery. I tested and Python3 has the same problem on Windows with non decodable filenames than Python3 on Unix. Eg. add \u0809 character (random non encodable character) to the Python directory name: Python3 doesn't start if the code page cannot encode/decode it. To fix all OS (Windows and POSIX), Python3 import machinery should not convert filenames to bytes but manipulate unicode characters and only convert filenames to bytes on POSIX at the last moment (at system calls). -- mbcs codec ignores the error handler: it replaces unknown characters by ? by default, see #850997. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
Andrew Svetlov andrew.svet...@gmail.com added the comment: I'm skeptical about surrogates particularly for that problem. From my perspective the solution is only to use native unicode support for windows file operation functions. Conversions utf-8 - mbcs - utf8 will loose encoding information thanks to tricky Microsoft mbcs encoding schema. If I'm wrong please correct me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
Andrew Svetlov andrew.svet...@gmail.com added the comment: After looking in #4352 deep I figured out what true separation of filesystem default encoding and utf8 python namespace is really too complicated. For example import call stack chain converts module name from utf-8 to filesystem in import.c:find_module. After that converted name used by PyImport_ExecCodeModule* as utf-8 name while actually it has filesystem encoding. That problem cannot be solved by five-line patch and Martin von Loevis suggested me to stop potentially dangerous big import.c changes in python 3.1 beta. I like importlib way (with maybe C implementation as next step) in terms of true way reorganization of python import machinery, but unfortunatelly Cannon has no time for that. From my perspective only big refactoring can solve encoding issues (and we can use excellent io implementation to open utf-8 named files in Windows using native unicode functions). We need to split 'module names' from 'filesystem pathes' clean. Maybe pure python importing is not easy - not sure. But reorganizing of current 'import spaghetti' is required. importlib (and PEP 302) introduced a nice way to do that. I like to be volunteer for this task and I feel enough knowledge to implement and test cover at least windows and linux (MacOs is not big problem also). But I need a mentor (Petrou, Cannon - you are welcome) to make it done, done clear and stable, done in resonable time period. -- nosy: +asvetlov ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
Changes by Andrew Svetlov andrew.svet...@gmail.com: -- nosy: +brett.cannon ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: As I wrote, I have an huge patch somewhere in my harddrive fixing this issue. But I don't want to publish it because it's really huge. I prefer to fix the problem step by step. I fixed most related issues: see the dependency list of #8242. I will publish the big patch shortly. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: See also #4352. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: If I understood correctly, this issue is a regression introduced by r67055 (to fix #4213). Read: http://bugs.python.org/issue4213#msg75387 See also r67057 (issue #3723). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
Changes by STINNER Victor victor.stin...@haypocalc.com: Removed file: http://bugs.python.org/file17251/pyunicode_encodefsdefault.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: I opened a separated issue for the new function PyUnicode_EncodeFSDefault(): #8715. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: Let's try with something: pyunicode_asencodefsdefault.patch adds PyUnicode_EncodeFSDefault() function to uniformize how a unicode is converted to bytes. Fallback to UTF-8 if Py_FileSystemEncoding is not set (I should be ASCII, not UTF-8) and use surrogateescape error handler. -- keywords: +patch Added file: http://bugs.python.org/file17251/pyunicode_encodefsdefault.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
Antoine Pitrou pit...@free.fr added the comment: We could have a separate list storing the original bytes form of sys.path; this list would be used by find_module() as long as Py_FileSystemDefaultEncoding isn't initialized. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
Antoine Pitrou pit...@free.fr added the comment: Or find_module() could use wcstombs() as long as Py_FileSystemDefaultEncoding is NULL. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
STINNER Victor victor.stin...@haypocalc.com added the comment: I have a patch implementation most of the point described in my first message. I have to rework on it before submit it. The patch depends on other issues, and I prefer to first fix all related issues. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8611] Python3 doesn't support locale different than utf8 and an non-ASCII path (POSIX)
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8611 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
