[issue11619] On Windows, don't encode filenames in the import machinery
STINNER Victor added the comment: I updated parser_unicode.patch to the last Python version. The new patch has just a minor nit: test_symtable does crash :-D Fixed in new patch: parser_unicode-3.patch -- Added file: http://bugs.python.org/file31472/parser_unicode-3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
Changes by STINNER Victor victor.stin...@gmail.com: Removed file: http://bugs.python.org/file21759/parser_unicode.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
Changes by STINNER Victor victor.stin...@gmail.com: Removed file: http://bugs.python.org/file31446/parser_unicode-2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
Roundup Robot added the comment: New changeset df2fdd42b375 by Victor Stinner in branch 'default': Close #11619: The parser and the import machinery do not encode Unicode http://hg.python.org/cpython/rev/df2fdd42b375 -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
STINNER Victor added the comment: I reopen the issue because some users are now requesting this feature. I updated parser_unicode.patch to the last Python version. The new patch has just a minor nit: test_symtable does crash :-D I will investigate the crash later. -- resolution: wont fix - status: closed - open Added file: http://bugs.python.org/file31446/parser_unicode-2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
Steven Velez added the comment: This may be a small use case, but a use case none-the less. In my situation, I am distributing a frozen python package and it runs under the users home directory. If the user's name has international characters, this will fail. I expect we will have similar problems when dealing with our application which embeds python and is also running from within the user directory... -- nosy: +Steven.Velez ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
Changes by Eric Snow ericsnowcurren...@gmail.com: -- nosy: +eric.snow ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
Drekin added the comment: Is there a chance this will be fixed at least in Python 4? -- nosy: +Drekin ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
STINNER Victor added the comment: The patch is really huge for such a very rare use case, so I prefer to close the issue as wont fix. Common cases with non-ASCII names are already handled correctly in Python 3.3. -- resolution: - wont fix status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
STINNER Victor victor.stin...@haypocalc.com added the comment: Another huge patch to support Unicode filenames: parser_unicode.patch Doc/c-api/exceptions.rst | 26 +++--- Include/ast.h|5 ++ Include/compile.h| 15 +++- Include/parsetok.h | 42 ++- Include/pyerrors.h |7 +++ Include/pythonrun.h | 16 Include/symtable.h |6 ++- Include/warnings.h |8 Modules/parsermodule.c | 49 +-- Modules/symtablemodule.c | 10 +++-- Parser/parsetok.c| 82 + Python/_warnings.c | 31 +++-- Python/ast.c | 40 -- Python/compile.c | 69 +- Python/errors.c | 57 ++- Python/future.c | 27 +++--- Python/import.c | 20 +++ Python/pythonrun.c | 85 +-- Python/symtable.c| 73 +++- 19 files changed, 480 insertions(+), 188 deletions(-) It creates new functions of the following functions which are undocumented: - PyAST_FromNode - PyFuture_FromAST - PyAST_Compile - PyParser_ParseFileFlagsEx - PyParser_ParseStringFlagsFilenameEx - PyErr_ProgramText - PyParser_ASTFromString - PyParser_ASTFromFile - PySymtable_Build We might remove these functions, but they are part of the public API (but they are undocumented). -- Added file: http://bugs.python.org/file21759/parser_unicode.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
Changes by STINNER Victor victor.stin...@haypocalc.com: Removed file: http://bugs.python.org/file21743/compile_filename.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
STINNER Victor victor.stin...@haypocalc.com added the comment: compile_filename.patch: - Add PyErr_ProgramTextObject() and PyErr_WarnExplicitObject() functions - Store the filename as Unicode in compile.c - Remove the filename from get_ref_type() fatal error (I never see such fatal error, and I hope that it does never happen, so the filename should not really matter here) The patch prepares the work to pass the filename to the compiler directly as Unicode. -- Added file: http://bugs.python.org/file21743/compile_filename.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
Roundup Robot devnull@devnull added the comment: New changeset e4e92d68ba3a by Victor Stinner in branch 'default': Close #11619: write_compiled_module() doesn't encode the filename http://hg.python.org/cpython/rev/e4e92d68ba3a -- resolution: - fixed stage: - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
STINNER Victor victor.stin...@haypocalc.com added the comment: c) parse_source_module() = covered by the issue #10785. Issue #10785 didn't change parse_source_module(): it does still encode the filename. We need Unicode version of PyParser_ASTFromFile() and PyAST_Compile(): a new version of these functions accepting a filename as a Unicode string. For PyParser_ASTFromFile(): #10785 prepared the work. For PyAST_Compile(): struct compiler stores the filename as a byte string, the filename should be stored as Unicode. -- resolution: fixed - status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
Roundup Robot devnull@devnull added the comment: New changeset 1b7f484bab6e by Victor Stinner in branch 'default': Issue #11619: _PyImport_LoadDynamicModule() doesn't encode the path to bytes http://hg.python.org/cpython/rev/1b7f484bab6e -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
STINNER Victor victor.stin...@haypocalc.com added the comment: open_exclusive() was created by: changeset: 14708:89b2aee43e0b branch: legacy-trunk user:Guido van Rossum gu...@python.org date:Wed Sep 20 20:31:38 2000 + files: Python/import.c description: On Unix, use O_EXCL when creating the .pyc/.pyo files, to avoid a race condition -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
STINNER Victor victor.stin...@haypocalc.com added the comment: dynload_win.patch: Fix part (a), _PyImport_LoadDynamicModule(). -- keywords: +patch Added file: http://bugs.python.org/file21321/dynload_win.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
New submission from STINNER Victor victor.stin...@haypocalc.com: With #3080, Python 3.3 does now manipulate module paths and names as Unicode in the import machinery. But in 3 remaining places, it does encode filenames (to the ANSI code page): a) _PyImport_LoadDynamicModule() It should pass directly the PyObject* (instead of a char*) to _PyImport_GetDynLoadFunc(), but only on Windows (we may change the function name for Windows). _PyImport_GetDynLoadFunc() of dynload_win.c has to be patched to use the Unicode API (eg. LoadLibraryEx = LoadLibraryExW). b) write_compiled_module() The problem is to implement open_exclusive() for Windows using Unicode. open_exclusive() uses open() on Windows, but open() expects the filename as a byte string. We may use _Py_fopen() (_wfopen), but this function doesn't have an option to open the file in exclusive mode (O_EXCL flag). GNU has an extension: x flag in the file mode, but Windows doesn't support it. The file is passed to marshal functions like PyMarshal_WriteLongToFile(), and so the file have to be a FILE*. c) parse_source_module() = covered by the issue #10785. -- components: Interpreter Core, Unicode, Windows messages: 131574 nosy: amaury.forgeotdarc, haypo, pitrou priority: normal severity: normal status: open title: On Windows, don't encode filenames in the import machinery versions: Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11619] On Windows, don't encode filenames in the import machinery
Changes by Terry J. Reedy tjre...@udel.edu: -- nosy: +terry.reedy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11619 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
