[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:

boost-python was using the removed private _Py_fopen() function, I proposed 
https://github.com/boostorg/python/pull/344 to replace _Py_fopen() with fopen().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:


New changeset a12491681f08a33abcca843f5150330740c9 by Victor Stinner in 
branch 'master':
bpo-32381: pymain_run_command() uses PyCF_IGNORE_COOKIE (GH-23724)
https://github.com/python/cpython/commit/a12491681f08a33abcca843f5150330740c9


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:


New changeset ceb420251c1d635520049fbb7b5269a73d63fb58 by Victor Stinner in 
branch 'master':
bpo-32381: pymain_run_file() uses PySys_FormatStderr() (GH-23778)
https://github.com/python/cpython/commit/ceb420251c1d635520049fbb7b5269a73d63fb58


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:

My PR 23778 fix the encoding/error handler when writing the filename into 
stderr, when the file does not exist:

$ LANG= PYTHONCOERCECLOCALE=0 ./python -X utf8=0 héllo.py
./python: can't open file '/home/vstinner/python/master/h\udcc3\udca9llo.py': 
[Errno 2] No such file or directory

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22636
pull_request: https://github.com/python/cpython/pull/23778

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22585
pull_request: https://github.com/python/cpython/pull/23724

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:


New changeset a82f63f5af027a0eab0f0812d750b804368cbd25 by Victor Stinner in 
branch 'master':
bpo-32381: Add _PyRun_AnyFileObject() (GH-23723)
https://github.com/python/cpython/commit/a82f63f5af027a0eab0f0812d750b804368cbd25


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22584
pull_request: https://github.com/python/cpython/pull/23723

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:


New changeset ca064402079f889226cb107b26b329891431c319 by Victor Stinner in 
branch 'master':
bpo-32381: Remove unused _Py_fopen() function (GH-23711)
https://github.com/python/cpython/commit/ca064402079f889226cb107b26b329891431c319


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22574
pull_request: https://github.com/python/cpython/pull/23711

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 550e4673be538d98b6ddf5550b3922539cf5c4b2 by Victor Stinner in 
branch 'master':
bpo-32381: Add _PyRun_SimpleFileObject() (GH-23709)
https://github.com/python/cpython/commit/550e4673be538d98b6ddf5550b3922539cf5c4b2


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22573
pull_request: https://github.com/python/cpython/pull/23709

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 815506d852daabc40e14ff0987c1142c0205fbe7 by Victor Stinner in 
branch 'master':
bpo-32381: Rewrite PyErr_ProgramText() (GH-23700)
https://github.com/python/cpython/commit/815506d852daabc40e14ff0987c1142c0205fbe7


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[Alexey Izbyshev]

Change by Alexey Izbyshev :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions:  -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

Thanks for the fix and backports!

--
resolution: fixed -> 
stage: resolved -> patch review
status: closed -> open
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22567
pull_request: https://github.com/python/cpython/pull/23700

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:

It's now fixed in 3.8, 3.9 and master branches.

Thanks for the bug report tianjg.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions:  -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:


New changeset b5cf308de8b19bf8f77053013f7e8a944159e1aa by Victor Stinner in 
branch '3.8':
bpo-32381: Fix PyRun_SimpleFileExFlags() encoding (GH-23642) (GH-23692) 
(GH-23696)
https://github.com/python/cpython/commit/b5cf308de8b19bf8f77053013f7e8a944159e1aa


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22563
pull_request: https://github.com/python/cpython/pull/23696

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:


New changeset f0e42ae03c41ec32fcb3064772f46ff7f2c5ff3b by Victor Stinner in 
branch '3.9':
bpo-32381: Fix PyRun_SimpleFileExFlags() encoding (GH-23642) (GH-23692)
https://github.com/python/cpython/commit/f0e42ae03c41ec32fcb3064772f46ff7f2c5ff3b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22561
pull_request: https://github.com/python/cpython/pull/23692

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:


New changeset b6d98c10fff6f320f8fdf595c3f9a05d8be4e31d by Victor Stinner in 
branch 'master':
bpo-32381: Fix PyRun_SimpleFileExFlags() encoding (GH-23642)
https://github.com/python/cpython/commit/b6d98c10fff6f320f8fdf595c3f9a05d8be4e31d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:

> Isn't it a part of the public API? I can't find it in the docs, but it seems 
> to be declared in the public header.

The Python C API has a strange history...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

> It seems like PyErr_ProgramText() is no longer used in Python.

Isn't it a part of the public API? I can't find it in the docs, but it seems to 
be declared in the public header.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:

> Just so it doesn't get lost: the problem with the contract of 
> PyErr_ProgramText() which I mentioned in my dup 42568 is still there.

It seems like PyErr_ProgramText() is no longer used in Python. 
PyErr_ProgramTextObject() is used and it pass the filename as Python object to 
_Py_fopen_obj().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

Thanks for the patch, Victor, it looks good.

Just so it doesn't get lost: the problem with the contract of 
PyErr_ProgramText() which I mentioned in my dup 42568 is still there.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:

bpo-42568 is marked as a duplicate of this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:

I can reproduce the issue on Python 3.10 with a script called 北京市.py which 
contains: print("hello").

c:\> python 北京市.py
hello

c:\>python __pycache__\北京市.cpython-310.pyc
python: Can't reopen .pyc file

And with my PR 23642 fix, it works as expected:

C:\>python __pycache__\北京市.cpython-310.pyc
hello

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +22510
pull_request: https://github.com/python/cpython/pull/23642

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

Thanks, Eryk, for catching the dup, I missed it somehow.

@ZackerySpytz: do you plan to proceed with your PR? If not, I can pick it up -- 
this issue broke the software I develop after upgrade to 3.8.

I filed issue 42569 to hopefully clarify the status of _Py_fopen() which became 
murky to me.

--
nosy: +izbyshev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[Eryk Sun]

Change by Eryk Sun :


--
versions: +Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:

Hum. In fact, this problem can be fixed differently: modify PyRun_xxx() 
functions to pass the filename as an Unicode string. Maybe pass it as a 
wchar_t* string or even a Python str object.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[Zackery Spytz]

Change by Zackery Spytz :


--
nosy: +ZackerySpytz
versions: +Python 3.8, Python 3.9 -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[Zackery Spytz]

Change by Zackery Spytz :


--
keywords: +patch
pull_requests: +14498
stage: test needed -> patch review
pull_request: https://github.com/python/cpython/pull/14699

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:

> I would make _Py_fopen() more compatible with the PEP 529.

Typo: It* would

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[STINNER Victor]

STINNER Victor  added the comment:

run_file() gets a wchar_t* string which comes from wmain() argv.

run_file() encodes the wchar_t* using PyUnicode_EncodeFSDefault().

Later, PyRun_SimpleFileExFlags() calls indirectly fopen() with the encoded 
filename.

> This could be addressed in _Py_fopen by decoding the path and calling _wfopen 
> instead of fopen. 

I agree that it's the correct fix.

I would make _Py_fopen() more compatible with the PEP 529.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[R. David Murray]

Change by R. David Murray :


--
keywords: +3.6regression

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[Eryk Sun]

Eryk Sun  added the comment:

Workarounds: (1) force 3.6 to use the legacy ANSI filesystem encoding by 
setting the environment variable PYTHONLEGACYWINDOWSFSENCODING. (2) Use 8.3 DOS 
names, if creating them is enabled on the volume. You can check their value in 
CMD via `dir /x`. (3) Create alternative directory symbolic links or junctions 
with ASCII names via CMD's `mklink` command.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[Tianjg]

Tianjg  added the comment:

Thanks a lot. What should I do to  reopen .pyc file with non-ASCII path use
python3.6 in cmd？Could you give me* some **code examples*.Thank you again,
and I look forward to hearing from you

2017-12-20 15:35 GMT+08:00 Eryk Sun :

>
> Eryk Sun  added the comment:
>
> run_file encodes the file path via PyUnicode_EncodeFSDefault, which
> encodes as UTF-8 in Windows, starting with 3.6. PyRun_SimpleFileExFlags
> subsequently tries to open this encoded path via _Py_fopen, which calls
> fopen. The CRT expects an ANSI encoded path, so only the common ASCII
> subset will work. Non-ASCII paths will fail.
>
> This could be addressed in _Py_fopen by decoding the path and calling
> _wfopen instead of fopen.
>
> Executing a .pyc also fails in 3.5 if the wide-character path can't be
> encoded as ANSI, but the 3.5 branch only accepts security fixes.
>
> --
> components: +Interpreter Core, Unicode
> nosy: +eryksun, ezio.melotti, vstinner
> stage:  -> test needed
> title: python3.6 can not reopen .pyc file with Chinese path -> Python 3.6
> cannot reopen .pyc file with non-ASCII path
> type: compile error -> behavior
> versions: +Python 3.7
>
> ___
> Python tracker 
> 
> ___
>

--
nosy: +Tianjg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path

[Eryk Sun]

Eryk Sun  added the comment:

run_file encodes the file path via PyUnicode_EncodeFSDefault, which encodes as 
UTF-8 in Windows, starting with 3.6. PyRun_SimpleFileExFlags subsequently tries 
to open this encoded path via _Py_fopen, which calls fopen. The CRT expects an 
ANSI encoded path, so only the common ASCII subset will work. Non-ASCII paths 
will fail.

This could be addressed in _Py_fopen by decoding the path and calling _wfopen 
instead of fopen. 

Executing a .pyc also fails in 3.5 if the wide-character path can't be encoded 
as ANSI, but the 3.5 branch only accepts security fixes.

--
components: +Interpreter Core, Unicode
nosy: +eryksun, ezio.melotti, vstinner
stage:  -> test needed
title: python3.6 can not reopen .pyc file with Chinese path -> Python 3.6 
cannot reopen .pyc file with non-ASCII path
type: compile error -> behavior
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com