[issue37570] `distutils.util.byte_compile` fails indirect byte compiling with non-ASCII full-path.
Suzumizaki added the comment: Thanks, but Those cannot resolve this issue. This is install time problem. In other words, whatever the default encoding is, even some users change it, `distutils.util.byte_compile` must always succeed. There's no relation between using non-ASCII path and the default encoding. No logically reason to fail. -- ___ Python tracker <https://bugs.python.org/issue37570> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37570] `distutils.util.byte_compile` fails indirect byte compiling with non-ASCII full-path.
Suzumizaki added the comment: ... and please check "batch.diff" (Oops, that's typo of 'patch'). I just added "encoding" parameter to calling open() function, and added the temp file to 'utf-8' code declaration. This bug could happen where the default encoding of open() function is not utf-8(may not only Windows). -- ___ Python tracker <https://bugs.python.org/issue37570> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37570] `distutils.util.byte_compile` fails indirect byte compiling with non-ASCII full-path.
Suzumizaki added the comment: Terribly Sorry! I forgot to write this is only Windows Issue! Thanks! -- components: +Unicode, Windows nosy: +ezio.melotti, paul.moore, steve.dower, tim.golden, vstinner, zach.ware ___ Python tracker <https://bugs.python.org/issue37570> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37570] `distutils.util.byte_compile` fails indirect byte compiling with non-ASCII full-path.
Suzumizaki added the comment: Here's test code. Note: I marked just Python 3.8 and 3.7 I tested, but maybe all versions. -- Added file: https://bugs.python.org/file48473/my_test_case.py ___ Python tracker <https://bugs.python.org/issue37570> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37570] `distutils.util.byte_compile` fails indirect byte compiling with non-ASCII full-path.
New submission from Suzumizaki : `distutils.util.byte_compile` fails _indirect_ byte compiling when the full-path of the .py file contains non-ASCII characters. Because there is the project (cx_Freeze) which directly uses `distutils.util.byte_compile`, the problem would happen while installing it under the non-ASCII path. Fortunately, it is easy to fix. Ofcourse, the files included in external libraries should be named ASCII only, but `distutils.util.byte_compile` makes and uses full path of them. I learned many of libraries uses setuptools instead of distutils, but please consider the following situations: 1) The case that venv creates the virtual environment under or with non-ASCII named folder. 2) And special cases of 1), on Windows, the name of end-users OFTEN be composed with non-ASCII characters, this means including theirs Documents, Desktop, or Downloads etc. The more they are not professional,they make easily the situation above. -- components: Distutils files: batch.diff keywords: patch messages: 347718 nosy: Suzumizaki, dstufft, eric.araujo priority: normal severity: normal status: open title: `distutils.util.byte_compile` fails indirect byte compiling with non-ASCII full-path. type: behavior versions: Python 3.7, Python 3.8 Added file: https://bugs.python.org/file48472/batch.diff ___ Python tracker <https://bugs.python.org/issue37570> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24307] pip error on windows whose current user name contains non-ascii characters
Changes by Suzumizaki suzumiz...@free.japandesign.ne.jp: -- nosy: +Suzumizaki ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24307 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20485] Enable 'import Non-ASCII.pyd'
Suzumizaki added the comment: Thank you Nick about msg210209. I would like to try making PEP, but the work looks somewhat difficult. It may take the time. BTW, C/C++ Standards only allow the encoding of source code as platform dependent. They don't define the standard encoding of source codes... This means we have to choose to resolve this issue, one is giving up readability, the other is allowing platform-dependent feature, using UTF-8 to write the C code. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20485 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20485] Enable 'import Non-ASCII.pyd'
Suzumizaki added the comment: Both Visual Studio 2012 and 2013 CANNOT install on Windows Vista. That's OK for you even Vista alive until April 2017? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20485 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20485] Enable 'import Non-ASCII.pyd'
Suzumizaki added the comment: Thanks for taking into account this issue for PEP 451. Honestly to say, I can't imagine why or/and how this issue(or my patches) causes any problems especially compatibility issues. If someone can point them, I will try to resolve. Note that I extend only the definition of PyInit_. I don't touch the code for loading modules. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20485 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20485] Enable 'import Non-ASCII.pyd'
Suzumizaki added the comment: Thank you Nick, I understand the behavior of this issue should be written on PEP. By the way, Can I continue the discussion here? or is there elsewhere suitable place for the PEP? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20485 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20485] Enable 'import Non-ASCII.pyd'
New submission from Suzumizaki: Currently, the name of .pyd modules is limited within 7 bit US-ASCII. I want to do import X to import X.pyd, where X contains Unicode characters. I tried to make the patch and my work seems to be done successfully. I will post the patch with this issue, and next what should I do? About the solution: To make the export entry 'PyInit_xxx' kept inside 7 bit, I use the simple encoding 'szm62' for unicode, called in the patch. 1) 'szm62' is used once per module(.pyd), only for 'PyInit_xxx'. 2) 'szm62' is used only when non-ASCII characters are in the name of the module. 3) 'szm62' generates short string as UTF-8, except 0-9A-Za-z are encoded to 2 bytes. 4) 'szm62' is very simple, much easier than UTF-8. 5) I tested it only with MS VC++, but I believe highly compatible with other environments. 6) 'szm62' also can decode 7 bits to unicode, but only the encoding is used for this issue. Notes: The simplicity is important for the project like Cython -- it generates .pyd files. The codepoints over 16bits are also simply supported. They will be encoded to 4 alnum(0-9A-Za-z) characters. 0-9A-Za-z are (always) encoded to 2 alnums. They will be simply prefixed with '0'(U+30). When the generating 'Non-ASCII.pyd' with MSVC toolkit, the report 'LINK : warning LNK4232' will be raised on linking. But no problem. The warning says When someone try to link with LoadLibraryEx'A' function, it may or may not fail, depends on the user locale. Our Python.exe uses always LoadLibraryEx'W', it never fail with locale issue. Or if you have any question, please tell me that. Regards, Suzumizaki-Kimitaka -- components: Interpreter Core files: 20140202_patch_for_python_default_branch_5.patch keywords: patch messages: 209988 nosy: Suzumizaki priority: normal severity: normal status: open title: Enable 'import Non-ASCII.pyd' type: behavior versions: Python 3.3, Python 3.4, Python 3.5 Added file: http://bugs.python.org/file33866/20140202_patch_for_python_default_branch_5.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20485 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20485] Enable 'import Non-ASCII.pyd'
Changes by Suzumizaki suzumiz...@free.japandesign.ne.jp: Added file: http://bugs.python.org/file33867/20140202_patch_for_python3.3_4.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20485 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20485] Enable 'import Non-ASCII.pyd'
Changes by Suzumizaki suzumiz...@free.japandesign.ne.jp: Added file: http://bugs.python.org/file33868/sample_importing_non_ascii_pyd.zip ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20485 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20485] Enable 'import Non-ASCII.pyd'
Suzumizaki added the comment:
Thank you for reply.
The hack msg209998 is interesting, but how to name submodule with non latin
like languages, especially keeping native reable? X(
The reason I don't use like name.encode('unicode-escape').replace(b'\\',
b'_') is the length limits of the identifiers.
In fact, Visual C++ can accept 2047 chars(bytes) and gcc have no logical
limits. But the PEP 7 says we should use C89. And even C99 assumes first 63
bytes are significant. I don't know what C89 says, And my C99 reference is
below, this means real-C99 is possibly different:
http://www.open-std.org/JTC1/SC22/wg14/www/docs/n1124.pdf
If we should keep C99 order above, 63 chars are too short to use
'unicode-escape' like. 'PyInit_' takes 7, remains 56. When each characters
encoded as 5 chars like '_3010', only we can use 11 unicode-codepoints. When 6
chars, only 9 chars.
a) If we can break C99 or real-C89/C99 don't have 63 chars rule, we can simply
use as Amaury Forgeot d'Arc says.
b) If we should keep 63 chars rules, the encodings longer than 'szm62' is not
acceptable.
c) Of course, when there is no reason to make entry point 'PyInit_modulename',
Make enable stable constant name 'PyInit' or 'PyInitUnicodeNamedModule' without
individual module name is another acceptable idea.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20485
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20485] Enable 'import Non-ASCII.pyd'
Suzumizaki added the comment: Thank you for reply, STINNER. You encode it to UTF-8 and use \xHH\xHH\xHH... syntax to keep ASCII encoding for the C file? The NAME may also be mentionned in docstrings, C comments, type names, etc. The main purpose of this issue is I want use Cython like Python without any trouble. You don't have to worry about mentioned above. I made, and will fix when needed, the patch for Cython to convert them automatically. The sample C codes I posted uses UTF-8 directly only OUTSIDE of the quotation, but they can be fixed if we really have to fix. I don't like the idea of a new encoding just for one very specific function in C. There are already too many encodings in the world :-( Of course I will accept any encoding and/or any solution to resolve this issue. I made new encoding only to keep the condition as possible as I can, and not to limit the naming too short when using non-ASCII characters. The patch don't include encoding-module for any purpose. For this issue, decoding is not required inside the Python. The C language supports non-ASCII identifiers, but I don't know how they are encoded in the symbol table. That's why we should resolve this problem, shouldn't we? Also the standards don't define about the symbol table. I would prefer to rely on the C compiler if you would like to play in the playground of non-ASCII identifiers. The problem is we CAN'T as you say. Or, at least, if you really think that, any ASCII limiting against dynamic loading should be removed. In Python/dynload_win.c, _PyImport_GetDynLoadWindows() uses GetProcAddressA(). _PyImport_GetDynLoadWindows() seems to be called only to resolve PyInit_xxx entry from _PyImport_LoadDynamicModule() in Python/importdl.c. I have already resolved with the posted patch before. Is it a theorical feature request, or you really have a Python module with a non-ASCII name? As I told, NO to 1st, YES to 2nd. I have many 'non-ASCII.py' which I want to convert using Cython to 'non-ASCII.pyd' files. I'm not sure that it's really useful to support non-ASCII module names for C modules, even if I spend many months to support non-ASCII module names for Python modules :-) Because you are both English and Python expert. Thanks a lot to daily Python work! Thank you for reading this long description. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20485 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9291] mimetypes initialization fails on Windows because of non-Latin characters in registry
Suzumizaki added the comment: There is possibility that the installation of setuptools fails with any Windows machine because of this bug. I want change the priority of this issue higher... I failed the installation of setuptools with Python 2.7.6 on my machine, Windows 8.1 Pro Japanese Edition 64bit, but no problem with both Python 2.7.4 and Python 3.3.3. -- nosy: +Suzumizaki ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9291 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6102] When the package has non-ascii path and .pyc file, we cannot import them.
Suzumizaki suzumiz...@free.japandesign.ne.jp added the comment: Thanks for reply. I tried Python 3.1 Beta, and the testcode goes fine. (and my current work also.) I checked with Windows XP, I told before. My problem seems resolved, and here is answers only to make sure. (What is the value of sys.maxunicode there though?) 65535. I ran the script from the terminal with python3 test.py twice and it worked both the time (is this the correct way to use the script?). Yes. If you delete the .pyc an reload the script again, does it work? You mean and insted of an, right? The answer is Yes, even Python 3.0.1. (Next time upload the files separately, it's easier for us to see them.) OK, I know. Thank you very much for all. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6102 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6102] When the package has non-ascii path and .pyc file, we cannot import them.
New submission from Suzumizaki suzumiz...@free.japandesign.ne.jp: When the path of the package has non-ascii characters, importing such packages always fails except the first time (in other words, always fails when compiled .pyc file exists). The problem exists even the names of such modules only consist of us-ascii characters. Please follow the test code I create, which can test the problem even with English version Windows(, I expected). Target: Windows XP (I tested Home Ed., SP3, 32bit, Japanase version) Python: 3.0.1 The priority of this issue might be CRITICAL or higher by 2 reasons: 1) When the end-users meet this condition, the users cannot find what's wrong easily, because UnicodeDecodeError is raised (not ImportError). 2) On non-English version Windows, Desktop, My Documents or etc. folders have non-English name as default. For example, Katakana is used in Japanese version. This means all python packages in the desktop meet this problem for not a few users. -- components: None files: testcode.zip messages: 88299 nosy: Suzumizaki severity: normal status: open title: When the package has non-ascii path and .pyc file, we cannot import them. type: behavior versions: Python 3.0 Added file: http://bugs.python.org/file14063/testcode.zip ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6102 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5924] When setting complete PYTHONPATH on Python 3.x, paths in the PYTHONPATH are ignored
Changes by Suzumizaki suzumiz...@free.japandesign.ne.jp: -- nosy: +Suzumizaki ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5924 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
