[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode
JGoutin added the comment:
A little encoding cache benchmark.
Current Code:
=
import sys
def _fscodec():
encoding = sys.getfilesystemencoding()
errors = sys.getfilesystemencodeerrors()
def fsencode(filename):
filename = fspath(filename) # Does type-checking of `filename`.
if isinstance(filename, str):
return filename.encode(encoding, errors)
else:
return filename
def fsdecode(filename):
filename = fspath(filename) # Does type-checking of `filename`.
if isinstance(filename, bytes):
return filename.decode(encoding, errors)
else:
return filename
return fsencode, fsdecode
fsencode, fsdecode = _fscodec()
del _fscodec
-
import os
%timeit os.fsdecode(b'\xc3\xa9')
The slowest run took 21.59 times longer than the fastest. This could mean that
an intermediate result is being cached.
100 loops, best of 3: 449 ns per loop
%timeit os.fsencode('é')
The slowest run took 24.20 times longer than the fastest. This could mean that
an intermediate result is being cached.
100 loops, best of 3: 412 ns per loop
Modified Code:
==
from sys import getfilesystemencoding, getfilesystemencodeerrors
def fsencode(filename):
filename = fspath(filename) # Does type-checking of `filename`.
if isinstance(filename, str):
return filename.encode(getfilesystemencoding(),
getfilesystemencodeerrors())
else:
return filename
def fsdecode(filename):
filename = fspath(filename) # Does type-checking of `filename`.
if isinstance(filename, bytes):
return filename.decode(getfilesystemencoding(),
getfilesystemencodeerrors())
else:
return filename
-
import os
%timeit os.fsdecode(b'\xc3\xa9')
The slowest run took 15.88 times longer than the fastest. This could mean that
an intermediate result is being cached.
100 loops, best of 3: 541 ns per loop
%timeit os.fsencode('é')
The slowest run took 19.32 times longer than the fastest. This could mean that
an intermediate result is being cached.
100 loops, best of 3: 502 ns per loop
Result:
===
Cache is a 17% speed up optimization.
--
___
Python tracker
<http://bugs.python.org/issue29241>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode
JGoutin added the comment: Yes, I reported this encoding issue to some of them. But, there is still some problems : - Some libraries are not updated frequently (Or not still maintained), and still use fsencode. - Tests and CI don't see this problem if they don't have a test case for filename with accents or other uncommon characters in english. This problem will not be easy to eliminate totally... -- ___ Python tracker <http://bugs.python.org/issue29241> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode
JGoutin added the comment: Personally, I call "sys._enablelegacywindowsfsencoding()" for only one reason : Temporary fixing issues with some third party libraries which use C code for files I/O (With filename as "mbcs" encoded bytes internally). Theses libraries generally call "filename.encode(sys.getfilesystemencoding())" or "os.fsencode(filename)" before sending filenames from Python to C code. Actually, I didn't see any side effect for using this function. Maybe because I call it at start before anything else. Using the environment variable is not easy in my case. I can look if encoding caching in fsencode is efficient (On Windows). And eventually propose a code change for this. -- ___ Python tracker <http://bugs.python.org/issue29241> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode
JGoutin added the comment:
import sys
# Force the use of legacy encoding like versions of Python prior to 3.6.
sys._enablelegacywindowsfsencoding()
# Show actual file system encoding
encoding = sys.getfilesystemencoding()
print('File system encoding:', encoding)
# os.fsencode(filename) VS filename.encode(File system encoding)
import os
print(os.fsencode('é'), 'é'.encode(encoding))
>>> File system encoding: mbcs
>>> b'\xc3\xa9' b'\xe9'
The result is the same.
--
___
Python tracker
<http://bugs.python.org/issue29241>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29241] sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and os.fsdecode
New submission from JGoutin:
The doc say that calling "sys._enablelegacywindowsfsencoding()" is equivalent
to use "PYTHONLEGACYWINDOWSFSENCODING" environment variable.
In fact, this no apply to "os.fsencode" and "os.fsdecode".
Example with Python 3.6 64Bits on Windows 7 64 bits :
EXAMPLE CODE 1 (sys._enablelegacywindowsfsencoding()):
import sys
import os
# Force the use of legacy encoding like versions of Python prior to 3.6.
sys._enablelegacywindowsfsencoding()
# Show actual file system encoding
encoding = sys.getfilesystemencoding()
print('File system encoding:', encoding)
# os.fsencode(filename) VS filename.encode(File system encoding)
print(os.fsencode('é'), 'é'.encode(encoding))
>>> File system encoding: mbcs
>>> b'\xc3\xa9' b'\xe9'
First is encoded with "utf-8" and not "mbcs" (The actual File system encoding)
EXAMPLE CODE 2 (PYTHONLEGACYWINDOWSFSENCODING):
import sys
import os
# Force the use of legacy encoding like versions of Python prior to 3.6.
# "PYTHONLEGACYWINDOWSFSENCODING" environment variable set before running
Python.
# Show actual file system encoding
encoding = sys.getfilesystemencoding()
print('File system encoding:', encoding)
# os.fsencode(filename) VS filename.encode(File system encoding)
print(os.fsencode('é'), 'é'.encode(encoding))
>>> File system encoding: mbcs
>>> b'\xe9' b'\xe9'
Everything encoded with "mbcs" (The actual File system encoding)
In "os.fsencode" and "os.fsdecode" encoding and errors are cached on start and
never updated by "sys._enablelegacywindowsfsencoding()" after.
--
components: Windows
messages: 285220
nosy: JGoutin, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: sys._enablelegacywindowsfsencoding() don't apply to os.fsencode and
os.fsdecode
type: behavior
versions: Python 3.6
___
Python tracker
<http://bugs.python.org/issue29241>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25646] Distutils and Windows SDK 7.1
JGoutin added the comment: OK, I'll look that. -- resolution: -> out of date status: open -> closed ___ Python tracker <http://bugs.python.org/issue25646> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25646] Distutils and Windows SDK 7.1
JGoutin added the comment: Better compatibility for Architectures names conversion between Vcvarsall.bat and SetEnv.Cmd for cross compilation. This seem to work good on classic compilation, but I still have difficulties with cross-compilation. -- title: Distutils and Windows SDK 7.0 / Windows SDK 7.1 / Visual C++ Build Tools 2015 -> Distutils and Windows SDK 7.1 Added file: http://bugs.python.org/file41097/msvc9compiler.py ___ Python tracker <http://bugs.python.org/issue25646> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25646] Distutils and Windows SDK 7.0 / Windows SDK 7.1 / Visual C++ Build Tools 2015
JGoutin added the comment: I updated the file. I did some new tests with cross compilation and I see that the "libpath" environment variable may be missing in some case. Fortunately, this one is not used in the following of the code. So I set it as optional result for "query_vcvarsall". -- Added file: http://bugs.python.org/file41093/msvc9compiler.py ___ Python tracker <http://bugs.python.org/issue25646> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25646] Distutils and Windows SDK 7.0 / Windows SDK 7.1 / Visual C++ Build Tools 2015
JGoutin added the comment: Hello, I modified "msvc9compiler.py" with following changes : - Add automatic compatibility with standalone VC++ compilers from Windows SDK 7.1. (Not for 6.1/7.0, because that finally don't work as good as it seem on first view and there is already MSVC++ for Python 2.7). - Changed error when "include", "lib", "libpath" or "path" environment variable is missing to be more clear. - Modified some comments and strings to not specify exclusively VS2008 (This file is also compatible with more recent VS and that was not really clear before) - Remove "gen_preprocess_options" from import because it was unused. - Fixed PEP8. The file was tested with SDK7.1 on a Cython compilation + a Pip install from source on Python 3.4 and seem to work perfectly. Ps: Forget my previous comment on VS2015, I didn't see that code for it was moved on "_msvccompiler.py". I'll wait the official release for the fix on Visual C++ Build Tools 2015 (Which is, anyway, only in technical preview since october) -- versions: +Python 3.3, Python 3.4 -Python 2.7, Python 3.5, Python 3.6 Added file: http://bugs.python.org/file41076/msvc9compiler.py ___ Python tracker <http://bugs.python.org/issue25646> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25646] Distutils and Windows SDK 7.0 / Windows SDK 7.1 / Visual C++ Build Tools 2015
JGoutin added the comment: For SDK 7.1, with "DISTUTILS_USE_SDK=1" (And "MSSdk=1"), I still have the problem. The error is on "include", "lib", "libpath", "path" environment variables which are not set by "vcvarsall.bat" (And are difficult to set manually). I looked on "msvc9compiler.py" file, I see that DISTUTILS_USE_SDK set the names for some compiler executables but not these variables. A solution should be to call "SetEnv.cmd" in place of "vcvarsall.bat" if SDK is installed and if environment variables listed above are missing even after the call of "vcvarsall.bat" (Or if "vcvarsall.bat" is missing). "SetEnv.cmd" properly set the environment and I successfully compiled files after launched it. I can do this change and commit it. This will add fully automatic compatibility for SDK 6.1/7.0/7.1, so all possibles version for VC++2008 and 2010 (Is it also possible to add "vcbuildtools.bat" for VC++ Build Tools 2015, but you say it's already fixed for it.) -- ___ Python tracker <http://bugs.python.org/issue25646> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25646] Distutils and Windows SDK 7.0 / Windows SDK 7.1 / Visual C++ Build Tools 2015
New submission from JGoutin: Hello, This issue is related to Visual C++ standalone distributions (Without any Visual Studio version installed). Distutils don't properly detect theses compilers. It try to work with "vcvarsall.bat" but this file : - is missing with Visual C++ Build Tools 2015. - don't set correctly the environment with Windows SDK 7.0/7.1 I fixed this issue by modifying "vcvarsall.bat" for redirect to the goods files and set properly the environment. The procedure is detailed here : https://wiki.python.org/moin/WindowsCompilers I have tested this issue with : - Python 3.5 + Visual C++ Tools 2015 - Python 3.4 + Windows SDK 7.1 I have not tested yet (But it work exactly as SDK 7.1): - Python 2.7 + Windows SDK 7.0 I can eventually work to fix this directly on distutils (Or help to fix it). -- components: Distutils messages: 254788 nosy: JGoutin, dstufft, eric.araujo priority: normal severity: normal status: open title: Distutils and Windows SDK 7.0 / Windows SDK 7.1 / Visual C++ Build Tools 2015 type: enhancement versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 3.6 ___ Python tracker <http://bugs.python.org/issue25646> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
