[issue42441] UnicodeEncodeError

2020-11-23 Thread Eryk Sun


Eryk Sun  added the comment:

The default encoding for files in Windows is the process active code page, 
which defaults to the system code page, e.g. code page 1252 if the system 
locale is "en-IN". You need to explicitly use a Unicode encoding such as UTF-8 
when opening the file, e.g. open('some.csv', newline='', encoding='utf-8').

--
nosy: +eryksun
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed
type: crash -> behavior

___
Python tracker 
<https://bugs.python.org/issue42441>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42438] Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding

2020-11-22 Thread Eryk Sun


Change by Eryk Sun :


--
stage: backport needed -> resolved

___
Python tracker 
<https://bugs.python.org/issue42438>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42438] Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding

2020-11-22 Thread Eryk Sun


Eryk Sun  added the comment:

> I reinstalled python from 3.8.2 to 3.9

Going from 3.8 to 3.9 is not an in-place upgrade. You need to create a new 
virtual environment that references the base 3.9 installation.

--
nosy: +eryksun
resolution:  -> not a bug
stage:  -> backport needed
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue42438>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42421] Native open failed to create a file on windows10 when colon exists in the name

2020-11-20 Thread Eryk Sun


Eryk Sun  added the comment:

> 'FocalMix: Semi-Supervised Learning for 3D Medical Image Detection.pdf'

In a filesystem that supports file streams [1], such as NTFS or ReFS, the above 
name refers to a $DATA stream named " Semi-Supervised Learning for 3D Medical 
Image Detection.pdf" in a file name "FocalMix".

Filesystems that do not support file streams may allow colon in filenames, 
regardless of its reserved status in the API. FAT filesystems (e.g. FAT32, 
exFAT) disallow colon in filenames. But the VirtualBox shared-folder filesystem 
(redirector) allows it. Even if colon is allowed by a particular filesystem, I 
strongly advise that you never use it in filenames, considering the problems 
with moving the file to an NTFS drive.

You also need to be vigilant about using DOS device names (e.g. "con", "nul:", 
"com1.txt", etc) in filenames. Creating or opening them may succeed, but 
probably not in the way that you expect. For example, if you open 
r"C:\Temp\nul.txt" for writing, you're actually opening the r"\\.\NUL" device, 
and all of the data written is simply thrown away without error.

---

[1] https://docs.microsoft.com/en-us/windows/win32/fileio/file-streams

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue42421>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42397] lstrip 处理出现bug

2020-11-18 Thread Eryk Sun


Eryk Sun  added the comment:

Python 3.9 added str.removeprefix() and str.removesuffix() [1]:

>>> "data_prj_t_suffix".removeprefix('data_prj_').removesuffix('_suffix')
't'

[1] https://docs.python.org/3/library/stdtypes.html#str.removeprefix

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue42397>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue15263] Guard against invalid file handles in os functions

2020-11-18 Thread Eryk Sun


Eryk Sun  added the comment:

> I don't see any _Py_VerifyFd in the code, is this out of date?

Yes, this issue is out of date. The underlying issue is that the C runtime in 
Windows automatically checks various parameters and calls a registered handler 
for invalid parameters, for which the default handler terminates the process. 
Python 3.5+ uses a new C runtime (ucrt), which supports setting a handler for 
just the current thread, so this issue is addressed more generally nowadays by 
the macros _Py_BEGIN_SUPPRESS_IPH and _Py_END_SUPPRESS_IPH.

--
nosy: +eryksun
resolution:  -> out of date
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue15263>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue10854] Output .pyd name in error message of ImportError when DLL load fails

2020-11-16 Thread Eryk Sun


Eryk Sun  added the comment:

> The patch adding 'name' and 'path' to the ImportError str/repr 
> was not merged yet.

It's up to whatever code raises an ImportError to determine how the name and 
path should be included in the error message. In Windows, the DLL name was 
added to the exception message in 3.8. So I'm closing this issue again, with 
regard to its final scope. 

With regard to its original scope, the reason for closing is "wont fix". There 
are no plans to extend the import error of an extension module in Windows to 
include the missing DLL dependency that caused the import to fail. There are 
tools such as the "DLL Diagnostics" package on PyPI that implement a proper 
trace by parsing loader snaps, and at the very least there are tools such as 
dumpbin.exe (or link.exe /dump) to manually list the direct dependencies of a 
DLL.

--
nosy: +eryksun
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue10854>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue12726] explain that locale.getlocale() does not read system's locales

2020-11-16 Thread Eryk Sun


Eryk Sun  added the comment:

> I tried "import locale; locale.getlocale()" on macOS and 
> windows (3.10) and linux (3.7) and in all cases I got 
> non-None values.  

In Windows, starting with Python 3.8, Python sets the LC_CTYPE locale to the 
user (not system) default locale instead of the CRT's initial "C" locale. (This 
is possibly an unintended consequence of redesigning the interpreter startup 
code, but what's done is done.) The same has been implemented in POSIX going 
back to Python 3.1. It's not a significant change for the core interpreter and 
standard library, which do not use the LC_CTYPE encoding for much in Windows, 
but it might affect third-party code. Embedding applications can use an 
isolated configuration that doesn't modify LC_CTYPE.

locale.getdefaultlocale() is not based on C setlocale() in Windows. It returns 
the language and region of the user locale from WinAPI GetLocaleInfo() paired 
with the process code page from WinAPI GetACP(). The latter is generally the 
same as the system code page, but possibly not in Windows 10 if the application 
manifest sets the process "activeCodePage" to UTF-8. (python.exe as distributed 
doesn't use the "activeCodePage" setting in its manifest, but an embedding 
application might.)

> Given the next paragraph describing 'C' as a non-standard language 
> code, I would have expected ('C',None), but it is as it is.

The documentation is unclear. Locale normalization handles the common cases, 
for better or worse. "C.ASCII" maps to "C", which is parsed as (None, None). 
"C.UTF8" maps to "en_US.UTF-8", and "C.ISO88591" maps to "en_US.ISO8859-1". 
Other encodings combined with the "C" locale have no alias, in which case "C" 
is returned as the language code, even though it's not a valid RFC 1766 code.

--
nosy: +eryksun
status: pending -> open

___
Python tracker 
<https://bugs.python.org/issue12726>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42178] failed pickle hangs python on exit in CMD.exe only

2020-11-16 Thread Eryk Sun


Eryk Sun  added the comment:

Steve or Zach, please review PR 23290, submitted by Teugea Ioan-Teodor. It 
modifies the startup sequence for worker processes by creating the child 
process with a suspended thread and pickling process_obj to a BytesIO instance 
before resuming the thread and writing the pickle data to the pipe. If pickling 
fails, the child process is terminated before the thread is resumed, since 
there's no point in allowing the child to execute. 

This doesn't directly address the issue with interpreter startup that can cause 
Python to hang on a synchronous I/O request, but it does avoid the problem.

Alternatively, when there's a pickling error, the parent could close its end of 
the pipe and wait for the child to exit before propagating the exception, but 
I'd rather skip executing the child completely.

--

___
Python tracker 
<https://bugs.python.org/issue42178>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42178] failed pickle hangs python on exit in CMD.exe only

2020-11-16 Thread Eryk Sun


Change by Eryk Sun :


--
nosy: +davin, pitrou

___
Python tracker 
<https://bugs.python.org/issue42178>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42339] official embedded Python fails to import certain modules

2020-11-14 Thread Eryk Sun


Eryk Sun  added the comment:

> maybe there's a list of updates (or FAQ) to which you 
> want to add KB3063858?

The documentation of the embeddable package [1] could highlight the need for 
KB2533623. Currently it's directly mentioned only in "What's New in Python 3.8" 
[2].

> ImportError: DLL load failed while importing _lzma: 
> De parameter is onjuist.

Mixed-language output is awkward. A common _Py_FormatMessage function could be 
added in 3.10 that tries getting an error message in English before trying to 
get it in the user's default language.

---

[1] https://docs.python.org/3/using/windows.html#the-embeddable-package
[2] https://docs.python.org/3.8/whatsnew/3.8.html#changes-in-the-python-api

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue42339>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue37205] time.perf_counter() is not system-wide on Windows, in disagreement with documentation

2020-11-13 Thread Eryk Sun


Eryk Sun  added the comment:

> QPC typically has a frequency of 1e7, 

Never mind. Apparently with some combinations of chipset, processor, and 
Windows version, the QPC frequency runs far beyond the 1-10 MHz range. I 
thought Windows divided by 1024 instead of letting it run in the GHz range, but 
apparently not from what I see in search results. I misread "*in many cases*, 
QueryPerformanceFrequency returns the TSC frequency divided by 1024".

--

___
Python tracker 
<https://bugs.python.org/issue37205>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue37205] time.perf_counter() is not system-wide on Windows, in disagreement with documentation

2020-11-13 Thread Eryk Sun


Eryk Sun  added the comment:

> suggest to use time.perf_counter_ns() instead of time.perf_counter() 
> for best precision.

QPC typically has a frequency of 1e7, which requires 24 bits for the fraction 
of a second. So a system can be up for years before the 53-bit precision of a 
float is an issue. What am I missing?

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue37205>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42296] Infinite loop uninterruptable on Windows in 3.10

2020-11-13 Thread Eryk Sun


Change by Eryk Sun :


--
priority: release blocker -> 
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue42296>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42296] Infinite loop uninterruptable on Windows in 3.10

2020-11-13 Thread Eryk Sun


Eryk Sun  added the comment:

> That sounds like a micro-optimization which is not worth it.

In the back of my mind I was also thinking to generalize the behavior at 
runtime whenever a signal is tripped by a non-Python thread (e.g. a thread 
created by an extension module or ctypes), instead of special-casing Windows. 

To examine this, I created a C library in Linux that defines a test() function 
that creates two threads via pthread_create(). The first thread sleeps for 10 
seconds, and the second thread sleeps for 5 seconds and then calls 
pthread_kill() to send a SIGINT to the first thread. In 3.8, calling the test() 
function via ctypes followed by executing an infinite loop will interrupt the 
loop with a KeyboardInterrupt as soon as the second thread sends SIGINT. But in 
3.10, the loop never gets interrupted because the C signal handler isn't called 
on the main thread, so eval_breaker never gets set.

--
keywords: +patch -3.9regression
priority:  -> release blocker
resolution: fixed -> 
stage: resolved -> 
status: closed -> open

___
Python tracker 
<https://bugs.python.org/issue42296>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42296] Infinite loop uninterruptable on Windows in 3.10

2020-11-13 Thread Eryk Sun


Eryk Sun  added the comment:

> always interrupt and let the thread decide if it has something to do.

SIGNAL_PENDING_CALLS() is called on a Python thread via signal.raise_signal() 
or _thread.interrupt_main() / PyErr_SetInterrupt(). If you'd rather keep the 
COMPUTE_EVAL_BREAKER() call in that case, the console control-event case can be 
distinguished via PyGILState_GetThisThreadState(). It returns NULL if there's 
no thread state, i.e. WINAPI TlsGetValue returns NULL.

--
stage: patch review -> 

___
Python tracker 
<https://bugs.python.org/issue42296>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42296] Infinite loop uninterruptable on Windows in 3.10

2020-11-12 Thread Eryk Sun


Eryk Sun  added the comment:

Note that this issue only applies to a process that has a single Python thread. 
With multiple Python threads, the signal handler gets called when the main 
thread acquires the GIL. 

As far as Windows console-driven SIGINT and SIGBREAK are concerned, C 
signal_handler() is called on a new thread that the interpreter has never seen 
before and will never see again. But there's also 
signal.raise_signal(signal.SIGINT) to consider, executing on a Python thread. 
Another path to trip_signal() is via _thread.interrupt_main(), i.e. 
PyErr_SetInterrupt().

_Py_ThreadCanHandleSignals() checks whether the current thread is the main 
thread in the main interpreter. It gets called to guarantee this condition in 
multiple places, such as handle_signals() in Python/ceval.c. So the 
_Py_IsMainThread() call is required. In Windows, maybe the main-thread check 
could be bypassed in SIGNAL_PENDING_SIGNALS() for a non-Python thread (i.e. no 
tstate), by directly setting ceval2->eval_breaker instead of calling 
COMPUTE_EVAL_BREAKER() in this case.

--

___
Python tracker 
<https://bugs.python.org/issue42296>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42311] It seems like ctypes code makes truncated pointer values in x64(access violations)

2020-11-10 Thread Eryk Sun


Eryk Sun  added the comment:

A function that returns a pointer needs an explicit `restype` set. A function 
parameter that's a pointer generally requires `argtypes` to be set. For example:

_testdll.GetPointer.restype = ctypes.c_void_p
_testdll.SetPointer.argtypes = (ctypes.c_void_p,)

Unfortunately the ctypes documentation starts with a tutorial that promotes bad 
practices and misuses WinAPI GetModuleHandle multiple times, which gives people 
the wrong idea about pointer return values. However, the tutorial does mention 
that Python integers are passed as C int values by default and that C int is 
the default return type. It also shows how to use the `argtypes` and `restype` 
attributes.

--
nosy: +eryksun
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue42311>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42296] Infinite loop uninterruptable on Windows in 3.10

2020-11-09 Thread Eryk Sun


Eryk Sun  added the comment:

Yes, if I force the return value of _Py_ThreadCanHandleSignals to 1, the loop 
is broken by a KeyboardInterrupt.

--

___
Python tracker 
<https://bugs.python.org/issue42296>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42296] Infinite loop uninterruptable on Windows in 3.10

2020-11-09 Thread Eryk Sun


Eryk Sun  added the comment:

See bpo-40010. COMPUTE_EVAL_BREAKER() in Python/ceval.c -- or 
_Py_ThreadCanHandleSignals in Include/internal/pycore_pystate.h -- needs to 
take into account that SIGNAL_PENDING_SIGNALS() gets called on a completely new 
thread in Windows.

--

___
Python tracker 
<https://bugs.python.org/issue42296>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42296] Infinite loop uninterruptable on Windows in 3.10

2020-11-09 Thread Eryk Sun


Change by Eryk Sun :


--
nosy: +vstinner

___
Python tracker 
<https://bugs.python.org/issue42296>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42296] Infinite loop uninterruptable on Windows in 3.10

2020-11-09 Thread Eryk Sun


Eryk Sun  added the comment:

It also cannot be interrupted in 3.9. But 3.8 and earlier work correctly.

--
nosy: +eryksun
type:  -> behavior
versions: +Python 3.9

___
Python tracker 
<https://bugs.python.org/issue42296>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42293] Installation of pyinstaller in Windows fails with utf8 error

2020-11-08 Thread Eryk Sun


Eryk Sun  added the comment:

This looks to be a bug in the constructor of pip's BuildEnvironment class:

https://github.com/pypa/pip/blob/a4f4bfbf8ba7fd1e60884a439907e3f2a32e117a/src/pip/_internal/build_env.py#L82

Python 3 source files are UTF-8 by default, but text files default to the 
platform's preferred encoding, unless overridden by Python's UTF-8 mode. It is 
thus a mistake to create "sitecustomize.py" without explicitly specifying UTF-8 
as the encoding. 

The preferred encoding in Windows is the process active code page, which is the 
system code page, unless, in Windows 10 only, it's overridden to use UTF-8 by 
the application manifest's "activeCodePage" setting. The system code page is 
usually a legacy code page, such as 1252, unless, in Windows 10 only, it's set 
to UTF-8 or the system language is Unicode only (e.g. Hindi).

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue42293>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42261] Windows legacy I/O mode mistakenly ignores the device encoding

2020-11-04 Thread Eryk Sun


Eryk Sun  added the comment:

> It would be nice to get an unit test for this case.

The process code page from GetACP() is either an ANSI code page or CP_UTF8 
(65001). It should never be a Western OEM code page such as 850. In that case, 
a reliable unit test would check that the configured encoding is a particular 
OEM code page. For example, spawn a new interpreter in a windowless console 
session (i.e. creationflags=CREATE_NO_WINDOW). Set the session's input code 
page to 850 via ctypes.WinDLL('kernel32').SetConsoleCP(850). Set 
os.environ['PYTHONLEGACYWINDOWSSTDIO'] = '1'. Then spawn [sys.executable, '-c', 
'import sys; print(sys.stdin.encoding)'], and verify that the output is 'cp850'.

--

___
Python tracker 
<https://bugs.python.org/issue42261>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42261] Windows legacy I/O mode mistakenly ignores the device encoding

2020-11-04 Thread Eryk Sun


Eryk Sun  added the comment:

> The solution here is to fix config_init_stdio_encoding() to use 
> GetConsoleCP() and GetConsoleOutputCP() to build a "cpXXX" string.

But, as I mentioned, that's only possible by replacing config->stdio_encoding 
with three separate settings: config->stdin_encoding, config->stdout_encoding, 
and config->stderr_encoding.

--

___
Python tracker 
<https://bugs.python.org/issue42261>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42261] Windows legacy I/O mode mistakenly ignores the device encoding

2020-11-04 Thread Eryk Sun


Eryk Sun  added the comment:

There's a related issue that affects opening duplicated file descriptors and 
opening "CON", "CONIN$", and "CONOUT$" in legacy I/O mode, but this case has 
always been broken. For Windows, _Py_device_encoding needs to be generalized to 
use _get_osfhandle and GetNumberOfConsoleInputEvents to detect and 
differentiate console input and output, instead of using isatty() and hard 
coding file descriptors 0-2.

--

___
Python tracker 
<https://bugs.python.org/issue42261>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42261] Windows legacy I/O mode mistakenly ignores the device encoding

2020-11-04 Thread Eryk Sun


New submission from Eryk Sun :

In Python 3.8+, legacy standard I/O mode uses the process code page from GetACP 
instead of the correct device encoding from GetConsoleCP and 
GetConsoleOutputCP. For example:

C:\>chcp 850
Active code page: 850
C:\>set PYTHONLEGACYWINDOWSSTDIO=1

C:\>py -3.7 -c "import sys; print(sys.stdin.encoding)"
cp850
C:\>py -3.8 -c "import sys; print(sys.stdin.encoding)"
cp1252
C:\>py -3.9 -c "import sys; print(sys.stdin.encoding)"
cp1252

This is based on config_init_stdio_encoding() in Python/initconfig.c, which 
sets config->stdio_encoding via config_get_locale_encoding(). Cannot 
config->stdio_encoding be set to NULL for default behavior?

Computing this ahead of time would require separate encodings 
config->stdin_encoding, config->stdout_encoding, and config->stderr_encoding. 
And _Py_device_encoding would have to be duplicated as something like 
config_get_device_encoding(PyConfig *config, int fd, wchar_t **device_encoding).

--
components: IO, Interpreter Core, Windows
messages: 380329
nosy: eryksun, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
stage: needs patch
status: open
title: Windows legacy I/O mode mistakenly ignores the device encoding
type: behavior
versions: Python 3.10, Python 3.8, Python 3.9

___
Python tracker 
<https://bugs.python.org/issue42261>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42252] Embeddable Python indicates that it uses PYTHONPATH

2020-11-03 Thread Eryk Sun


Eryk Sun  added the comment:

The embeddable distribution isn't intended for end users to run Python scripts 
from the command line, so I don't think the CLI help needs to be special cased. 
The documentation you quoted should be clarified as something like "isolated 
from user and system Python settings, including environment variables such as 
PYTHONPATH, registry settings, and installed packages". It would be helpful as 
well if the Windows download links on python.org explained or linked to the 
intended use case for the embeddable distribution. Sometimes people mistakenly 
download it when they really need a normal Python installation.

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue42252>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42196] Python Windows Installation Error 0x800705aa

2020-10-29 Thread Eryk Sun


Eryk Sun  added the comment:

0x8007_05AA is ERROR_NO_SYSTEM_RESOURCES (1450, 0x05AA) returned as an HRESULT 
value. This error is usually due to a system call failing with the status code 
STATUS_INSUFFICIENT_RESOURCES. From "Pyhon39InstallLog.txt", apparently the 
installer failed to create an uninstall registry key for the current user. Can 
you successfully install other programs for just the current user 
(non-elevated)?

For the issue with `python` not doing anything, check `where.exe python`. The 
first match is the one that CMD would try to execute.

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue42196>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42178] failed pickle hangs python on exit in CMD.exe only

2020-10-27 Thread Eryk Sun


Eryk Sun  added the comment:

Due to the exception raised when trying to pickle main..dummy in 
popen_spawn_win32.Popen(), the main process may have already exited by the time 
the spawned worker calls WinAPI OpenProcess in spawn.spawn_main(). I suppose 
the reduction of prep_data and process_obj  in popen_spawn_win32.Popen() could 
be dumped to an io.BytesIO instance before WinAPI CreateProcess is called. That 
way an exception in the pickler would occur before the child process is created.

There's also a race condition that can hang the worker during interpreter 
startup. When sys.stdin is initialized, the buffered reader does a raw tell(). 
If the raw layer is an io.FileIO instance (e.g. console input with 
PYTHONLEGACYWINDOWSSTDIO defined), the raw tell() ultimately calls WinAPI 
SetFilePointerEx (via _lseeki64). This can hang if the kernel file is already 
blocked in a synchronous I/O request, which might be the case if the main 
process has already terminated and its parent (e.g. cmd.exe or powershell.exe) 
has resumed reading from the console. portable_lseek() in Modules/_io/fileio.c 
should detect a non-disk file via WinAPI GetFileType and just set 
self->seekable to 0 in this case.

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue42178>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42175] long lines from interactive stdin are truncated

2020-10-27 Thread Eryk Sun


Eryk Sun  added the comment:

For Windows see issue 41849. The legacy limit (i.e. PYTHONLEGACYWINDOWSSTDIO) 
for reading input via io.FileIO is 8K characters in Windows, but with 
io._WindowsConsoleIO, it's only 512 characters, which is far too small in 
general. The legacy implementation of input() based on C fgets() is capped at 
4K characters in Windows, which is the same as a Linux terminal. The new 
implementation of input() in Windows (i.e. _PyOS_WindowsConsoleReadline) 
increases the limit to 16K characters. I'd like to see both cases increased to 
32K characters, which is the length limit of a command line or file path in 
Windows.

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue42175>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue37609] support "UNC" device paths in ntpath.splitdrive

2020-10-27 Thread Eryk Sun


Eryk Sun  added the comment:

I'm attaching a rewrite of splitdrive() from msg352355. This version uses an 
internal _next() function to get the indices of the next path component, 
ignoring repeated separators. It also flattens the nested structure of the 
previous implementation by adding multiple return statements.

--
Added file: https://bugs.python.org/file49541/splitdrive.py

___
Python tracker 
<https://bugs.python.org/issue37609>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42170] splitdrive fails for UNC path with the "\\?\UNC\" prefix.

2020-10-27 Thread Eryk Sun


Change by Eryk Sun :


--
resolution:  -> duplicate
stage: patch review -> resolved
status: open -> closed
superseder:  -> support "UNC" device paths in ntpath.splitdrive

___
Python tracker 
<https://bugs.python.org/issue42170>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue38233] datetime.datetime.fromtimestamp have different behaviour on windows and mac

2020-10-25 Thread Eryk Sun


Change by Eryk Sun :


--
resolution:  -> duplicate
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue38233>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue11646] 2to3: msvcrt.(get|put)ch -> (get|put)wch

2020-10-22 Thread Eryk Sun


Eryk Sun  added the comment:

The bytes type has supported string methods for a long time now. I don't think 
there's anything else to do here. msvcrt.getch() is a low-level I/O function 
like os.read(). It should not be automatically converted to msvcrt.getwch(). 
Unfortunately the function names weren't changed in Python 3. I'd rather that 
msvcrt.getch() returned str and msvcrt.getchb() returned bytes. Then there 
would have actually been something for 2to3 to convert (e.g. getch -> getchb, 
and getwch -> getch).

--
nosy: +eryksun
resolution:  -> out of date
stage:  -> resolved
status: open -> closed
type:  -> behavior

___
Python tracker 
<https://bugs.python.org/issue11646>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue25655] Python errors related to failures loading DLL's lack information

2020-10-22 Thread Eryk Sun


Eryk Sun  added the comment:

Steve, the PR that you pushed has the wrong error and error message. I told 
Philippe in msg377335 that ctypes raises FileNotFoundError with no error code. 
For example:

>>> try: ctypes.CDLL('spam')
... except OSError as e: err = e
...
>>> err
FileNotFoundError("Could not find module 'spam' (or one of its 
dependencies). Try using the full path with constructor syntax.")
>>> err.winerror is None
True

The advice to use dumpbin is fine and works well in simple cases. I wouldn't 
use it generally since recursiveley parsing through the dependency graph of 
every dependent DLL could be tedious.

--

___
Python tracker 
<https://bugs.python.org/issue25655>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41737] Improper NotADirectoryError when opening a file in a fake directory

2020-10-20 Thread Eryk Sun


Eryk Sun  added the comment:

Regarding documentation, builtin open() and the io and os modules generally do 
not provide information about platform-specific errors. But documenting the 
behavior for the NotADirectoryError exception itself may be useful considering 
it applies to many POSIX functions that access filepaths, such as stat, open, 
mkdir, rmdir, unlink, and rename.

Regarding behavior, I don't see anything reasonable that can or should be done 
for POSIX. In Windows, it should be possible to know whether FileNotFoundError 
is due to a bad path prefix (ERROR_PATH_NOT_FOUND, 3) or a missing file 
(ERROR_FILE_NOT_FOUND, 2), but io.FileIO currently only uses standard C errno, 
which maps both of these cases to ENOENT. So one behavior that *can* be fixed 
in this situation is to get the actual Windows error code from MSVC _doserrno, 
such that the raised FileNotFoundError would have the relevant Windows error 
code in its winerror attribute.

--

___
Python tracker 
<https://bugs.python.org/issue41737>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue38324] [Windows] test_locale and test__locale failures on Windows

2020-10-19 Thread Eryk Sun


Eryk Sun  added the comment:

PR 20529 looks good to me. Thank you, Tiago.

--

___
Python tracker 
<https://bugs.python.org/issue38324>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42041] venv subprocess call to python resolves to wrong interpreter

2020-10-19 Thread Eryk Sun


Eryk Sun  added the comment:

> My biggest concern with the suggested wording ... is that it 
> reintroduces the issue with the redirector.

I thought I addressed that in second paragraph by recommending sys.executable. 
It could be emphasized that running "python[x][.y]" is unreliable, without 
going into detailed examples of where it's unreliable on various platforms 
(i.e. avoid going into detail about issues with naming of binaries, embedding, 
the search path, redirectors, etc). A separate cross-platform note in venv 
could advise scripts to use sys.executable, with a link to the subprocess.Popen 
docs and a reference to the note about the platform-dependent search behavior 
and unreliability of running "python[x][.y]".

If the redirector issue is mentioned anywhere, I think it should be in the venv 
docs. It can include a note about the Windows implementation detail to use a 
redirector for non-symlink virtual environments. Of concern to me here is that 
the process handle and PID returned by CreateProcess is for the redirector. One 
can't use the returned process handle or PID with DuplicateHandle or 
WSADuplicateSocket (i.e. socket.socket.share) to share handles and sockets with 
a script that's running as a child process. It might seem to still be working, 
purely by accident, because the parent script is executing the base "python" 
instead of the redirector, but in that case the child script isn't using the 
virtual environment. There isn't an official way to support running 
sys._base_executable with the __PYVENV_LAUNCHER__ environment variable, as 
multiprocessing implements internally. Maybe the workaround should be 
incorporated implicitly in subprocess.Popen if `executable`, `args`, or args[0] 
is equa
 l to sys.executable and sys._base_executable is different. If the latter is 
implemented, and using sys.executable is strongly advised, it strengthens the 
case to avoid discussing the redirector entirely.

--

___
Python tracker 
<https://bugs.python.org/issue42041>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42041] venv subprocess call to python resolves to wrong interpreter

2020-10-19 Thread Eryk Sun


Eryk Sun  added the comment:

> I'd read it because I'm interested, but that is probably too much detail
> for someone who is trying to get something done quickly.

I did say that I don't know how much should be documented. I tend to dump a lot 
of information in issues so that people can make informed decisions. Personally 
I would keep it limited. I do want more help, and generic help, in the warning, 
but I don't want to spam the already exhausting and MEGO-prone (i.e. my eyes 
glaze over) subprocess docs.

With regard to the use of a platform search, note that subprocess actually 
implements its own search in POSIX. (subprocess supports posix_spawn -- but not 
posix_spawnp -- on macOS and on Linux with glibc 2.24+, but it's limited to a 
narrow subset of Popen arguments, excluding `cwd`, and it requires a qualified 
`executable` path.) A name containing a slash is resolved against `cwd` or the 
process current working directory. An unqualified name is searched for in the 
os.get_exec_path(env) search path, based on PATH from `env` or os.environ, else 
os.defpath. This allows `env` to override the POSIX search path, just as `cwd` 
in POSIX overrides the current working directory when resolving a qualified 
args[0] path and `executable`. This is sort of documented by saying that POSIX 
Popen() works "os.execvp()-like", but to be more correct it should say 
"os.execvpe()-like". The final "e" in "execvpe" is fundamentally important 
since PATH can be sourced from `env`.

In Windows, the search path and the working directory used for resolving paths 
can't be overridden without passing shell=True. subprocess incorrectly 
documents this by claiming in general that "the function looks for `executable` 
(or for the first item in `args`) relative to `cwd` if the executable path is a 
relative path". That wording isn't even correct for POSIX, for which the 
working directory only applies to args[0] if it contains a slash.

In light of this, how about inserting the following warning:

  Resolving the path of *executable* or the first item of *args* is
  platform dependent even with ``shell=False``. For POSIX, see
  :meth:`os.execvpe`, and note that when resolving or searching for the
  executable path, *cwd* overrides the current working directory and *env*
  can override the ``PATH`` environment variable. For Windows, see the
  documentation of the ``lpApplicationName`` and ``lpCommandLine``
  parameters of WinAPI ``CreateProcess``, and note that when resolving or
  searching for the executable path with ``shell=False``, *cwd* does not
  override the current working directory and *env* cannot override the
  ``PATH`` environment variable. Cross-platform usage can improve
  reliability by using a fully-qualified path for the executable, such as
  from :meth:`shutil.which`.

  On all platforms, using :data:`sys.executable` as the path of the
  executable is the recommended and reliable way to run the current Python
  interpreter.

--

___
Python tracker 
<https://bugs.python.org/issue42041>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42041] venv subprocess call to python resolves to wrong interpreter

2020-10-18 Thread Eryk Sun


Eryk Sun  added the comment:

> For platform semantics, I'd prefer a link to the CreateProcessW docs, 
> with advice to read about the lpApplicationName parameter with 
> respect to `executable` and lpCommandLine with respect to `args` 
> and the platform search semantics. 

For example, let's help someone figure out that because "python3.8" has a ".8" 
filename 'extension', ".exe" may or may not be appended in a search.

>>> print(shutil.which('python3.8'))
C:\Users\someone\AppData\Local\Microsoft\WindowsApps\python3.8.EXE

>>> subprocess.call(['python3.8.exe', '-V'])
Python 3.8.6
0

SearchPathW (called internally by CreateProcessW) won't append the ".exe" 
default extension to a name that already has a ".8" extension:

>>> try: subprocess.call(['python3.8', '-V'])
... except OSError as e: print(e)
...
[WinError 2] The system cannot find the file specified

But with shell=True it works because CMD always appends the PATHEXT extensions 
(thankfully there isn't a "python3.8.com" file to get in the way, since .COM is 
usually listed before .EXE in PATHEXT):

>>> subprocess.call('python3.8 -V', shell=True)
Python 3.8.6
0

SearchPathW does append the default ".exe" extension for a qualified path:

>>> 
subprocess.call([r'C:\Users\someone\AppData\Local\Microsoft\WindowsApps\python3.8',
 '-V'])
Python 3.8.6
0

--

___
Python tracker 
<https://bugs.python.org/issue42041>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42041] venv subprocess call to python resolves to wrong interpreter

2020-10-18 Thread Eryk Sun


Eryk Sun  added the comment:

> You can build a cross-platform wrapper on top of native behaviour 
> (witness `shutil.which`) but you can't do the opposite.

Nothing would prevent adding a parameter to use the platform semantics, if that 
had been in the design from the outset. But it's not something we should change 
at this stage. It's too fundamental.

> I'm not convinced that using `shutil.which` is the right 
> answer for anything other than "python".

Actually, I think shutil.which() is the right answer generally (within the 
filesystem scope) except for "python", for which sys.executable is usually what 
people want, i.e. the current interpreter. A PATH search via 
shutil.which('python') doesn't necessarily find the current interpreter, if 
anything at all. So I'm not arguing against advice to use sys.executable, but 
rather that recommmending shutil.which() should be done in a more generic way 
that's separate from the "python" problem. 

For platform semantics, I'd prefer a link to the CreateProcessW docs, with 
advice to read about the lpApplicationName parameter with respect to 
`executable` and lpCommandLine with respect to `args` and the platform search 
semantics. The CreateProcessW docs are rather long and off-putting, so I think 
it helps to narrow it down for people, and explicitly map between Popen() and 
CreateProcessW parameters.

Over the years, I've come across many forum questions in which novice users 
waste days on problems with respect to Windows quirks that can be resolved in a 
minute or so by someone with knowledge and experience. They get lots of advice 
with good intentions from programmers who only have POSIX experience (the 
Python ecosystem online is still heavily dominated by POSIX, despite the number 
of Windows users), and it's mostly wrong advice and a waste of their time (not 
even a learning experience). Documenting platform inconsistencies helps 
experienced programmers to use the docs in order to help novice programmers. 
It's not necessarily about helping novices directly. Often just a nudge in the 
right direction is enough.

--

___
Python tracker 
<https://bugs.python.org/issue42041>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42041] venv subprocess call to python resolves to wrong interpreter

2020-10-18 Thread Eryk Sun


Eryk Sun  added the comment:

> Also, why would we document the Windows rules, but not the POSIX 
> rules? They are arguably just as strange to someone who doesn't 
> know them.

POSIX rules are simply to search PATH for the filename as passed, and if it has 
a slash in it, the path is resolved against the working directory. There is no 
implicit search of the application directory, current directory, and system 
directories. There is no search for "dir/file" in *every* search path directory 
instead of just resolving against the working directory. There is no figuring 
out when ".exe" or PATHEXT extensions will be appended or how to search for a 
filename that has no extension by appending a trailing "." to the name. And 
there is no massive inconsistency between the search semantics of shell=True 
and shell=False. What happens with "platform semantics" in Windows is 
complicated compared to POSIX. I'm more comfortable telling people to search 
via shutil.which() than relying on the platform search. I'd be much more 
comfortable if that's what subprocess.Popen() just did on its own. But we're 
locked into platform semantics, and I don't see that changing.

--

___
Python tracker 
<https://bugs.python.org/issue42041>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42041] venv subprocess call to python resolves to wrong interpreter

2020-10-18 Thread Eryk Sun


Eryk Sun  added the comment:

> I don't think we should document this level of detail

But a lot of it -- most of it -- is also strange behavior that no one would 
expect without reading about it somewhere. Most users of subprocess.Popen() 
will never wade through the documentation of CreateProcessW, SearchPathW, 
ShellExecuteExW, and cmd.exe -- to the extent that the behavior is usefully and 
correctly documented by Microsoft.

One of the reasons I bother writing it out in detail here is to make the case 
for always using shutil.which(), regardless of the value of `shell` (unless 
shell=True is being used beyond the filesystem scope). We have complete control 
of the implementation and documentation of shutil.which(). I don't want that 
advice to be narrowed down to just talking about running "python".

--

___
Python tracker 
<https://bugs.python.org/issue42041>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42041] venv subprocess call to python resolves to wrong interpreter

2020-10-18 Thread Eryk Sun


Eryk Sun  added the comment:

I don't know how much should be documented for subprocess.Popen, but here are 
the details for how searching works with shell=False (default) and shell=True.

For shell=False, the search path used by WinAPI CreateProcessW checks 
%__APPDIR__%; %__CD__% (unless %NoDefaultCurrentDirectoryInExePath% is defined 
and the name to search contains no backslashes); %SystemRoot%\System32; 
%SystemRoot%\System; %SystemRoot%; and then the %PATH% directories. The search 
path is passed to WinAPI SearchPathW, with a default ".exe" extension. 
SearchPathW tries to resolve a relative path (but not a drive-relative or 
rooted-relative path) against every directory in the search path, unless it 
explicitly begins with a "." or ".." component. For the relative case, it 
appends the default ".exe" extension to a name if and only if it has no 
extension (a trailing "." counts as an extension). For the non-relative case, 
it first tries to find the name as given and then with the default extension 
appended, even if the filename already has an extension (no exception is made 
for a trailing dot, i.e. searching for "C:\Temp\spam." will check for "spam." 
and then "spam..exe"
 ).

>From the POSIX perspective, the implicit inclusion of the application 
>directory, working directory, and system directories is strange and, regarding 
>the inclusion of the working directory, troubling. The fact that searching for 
>a relative name with one or more slashes, such as "spam\\python", is not 
>resolved against *only* the working directory is strange and not documented. 
>The rules governing when ".exe" will be appended are complicated and 
>incorrectly documented (e.g. the claim "if the file name contains a path, .exe 
>is not appended").

With shell=True, the CMD shell simply checks %__CD__% (unless 
%NoDefaultCurrentDirectoryInExePath% is defined and the name to search contains 
no slashes) and %PATH%. Support for forward slash in the name to search is 
wonky; it works only for quoted paths. But at least a relative path that 
contains slashes is only resolved against the current working directory instead 
of every directory in the search path. CMD's rules for appending default 
extensions are simpler than SearchPathW in some ways, but also more complicated 
because it's generalized as the PATHEXT list of extensions. In each directory, 
CMD always looks first for the searched name as given and then the name plus 
each extension in PATHEXT, regardless of the filepath type or whether the 
searched name already has an extension. It will not find a name that has no 
extension unless PATHEXT includes a "." entry for the empty extension. (This is 
consistent with the desktop shell API, which supports defining an association 
for the "."
  filetype.)

--

___
Python tracker 
<https://bugs.python.org/issue42041>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42046] Unable to write to file without elevated privileges

2020-10-16 Thread Eryk Sun


Eryk Sun  added the comment:

> processtoken = win32security.OpenProcessToken(process, 
> win32con.MAXIMUM_ALLOWED)
> win32security.GetTokenInformation(processtoken, 
> win32security.TokenMandatoryPolicy)

FYI, starting with Windows 8, the system supports pseudo-handles for the access 
token of the current process -- (HANDLE)-4 -- and the current thread -- 
(HANDLE)-5, which don't have to be opened and closed. In the API, they're 
available as the inlined functions GetCurrentProcessToken() and 
GetCurrentThreadToken(). These pseudo-handles have TOKEN_QUERY and 
TOKEN_QUERY_SOURCE access, so they can be used with token queries, i.e. 
GetTokenInformation(-4, TokenInformationClass).

> As the shell is started with medium integrity level and the file is set to 
> low 
> integrity level the process would get created with low integrity level.

Yes, because the access token of shell, which is a limited medium-integrity 
logon, has a mandatory policy that includes 
TOKEN_MANDATORY_POLICY_NEW_PROCESS_MIN.

> "C:\Program Files", "C:\Users" and "C:\Windows" seem to have their own DACL's.

Those directories have protected DACLs with custom security, so they don't 
inherit the inheritable entries from the root directory. For example:

>>> sd = GetNamedSecurityInfo(r'C:\Program Files', SE_FILE_OBJECT,
... DACL_SECURITY_INFORMATION)
>>> sd.GetSecurityDescriptorControl()[0] & SE_DACL_PROTECTED
4096

That said, Python's installer doesn't set custom security on the installation 
directory, and that's not likely to change. It just relies on inheritance. If 
you install in "C:\Python38-32", and the inheritable security from the root 
directory is problematic, then you need to resolve the problem manually, as you 
have done.

> win32security.GetFileSecurity("C:\\", 
> win32security.SACL_SECURITY_INFORMATION) 
> fails on me even on an elevated prompt.

Querying audit entries in the SACL of an object (SACL_SECURITY_INFORMATION) 
requires ACCESS_SYSTEM_SECURITY access, which requires SeSecurityPrivilege to 
be enabled. Administrators have this privilege, but it's disabled by default. 

Some entries in the SACL can be read with just READ_CONTROL access: the 
mandatory label (LABEL_SECURITY_INFORMATION -- WRITE_OWNER access to set), 
security resource attributes (ATTRIBUTE_SECURITY_INFORMATION -- WRITE_DAC 
access to set), and the central access policy identifier 
(SCOPE_SECURITY_INFORMATION -- ACCESS_SYSTEM_SECURITY access to set).

> "(NW)" is not directly mentioned. I'm assuming "(NR)" and "(NX)" might be the 
> missing integrity policy options for an integrity level entry.

I don't think icacls.exe allows setting no-read-up and no-execute-up access 
control. "NR" and "NX" appear to be ignored. For example:

>>> cmd = r'icacls C:\Temp\spam.txt /setintegritylevel H:(NW)(NR)(NX)'
>>> subprocess.call(cmd)
processed file: C:\Temp\spam.txt
Successfully processed 1 files; Failed processing 0 files
0

>>> sd = GetNamedSecurityInfo(r'C:\Temp\spam.txt', SE_FILE_OBJECT,
... LABEL_SECURITY_INFORMATION)
>>> sacl = sd.GetSecurityDescriptorSacl()
>>> (acetype, aceflags), mask, sid = sacl.GetAce(0)

>>> acetype == SYSTEM_MANDATORY_LABEL_ACE_TYPE
True
>>> aceflags == 0
True
>>> LookupAccountSid(None, sid)
('High Mandatory Level', 'Mandatory Label', 10)

But only the no-write-up access control is set:

>>> mask == SYSTEM_MANDATORY_LABEL_NO_WRITE_UP
True

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue42046>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42046] Unable to write to file without elevated privileges

2020-10-16 Thread Eryk Sun


Eryk Sun  added the comment:

> icacls.exe C:\Python38-32\python.exe lists Mandatory Label\
> Low Mandatory Level:(I)(NW) ** This might be the problem. Removing "L"
> with icacls might work.
>
> **When a user attempts to launch an executable file, the new process is
> created with the minimum of the user integrity level and the file 
> integrity level.**

The token mandatory policy [1] for a standard logon is 
TOKEN_MANDATORY_POLICY_NO_WRITE_UP (1) and 
TOKEN_MANDATORY_POLICY_NEW_PROCESS_MIN (2). The above quote applies to the 
latter. For an elevated logon, the mandatory policy is just 
TOKEN_MANDATORY_POLICY_NO_WRITE_UP, so setting a low-integrity label on 
python.exe has no effect on a new process created from an elevated security 
context. The following queries demonstrate the mandatory policy for both cases:

standard logon:

>>> GetTokenInformation(-4, TokenMandatoryPolicy)
3

elevated logon:

>>> GetTokenInformation(-4, TokenMandatoryPolicy)
1

> >icacls.exe C:\
> C:\ BUILTIN\Administrators:(F)
> BUILTIN\Administrators:(OI)(CI)(IO)(F)
> NT AUTHORITY\SYSTEM:(F)
> NT AUTHORITY\SYSTEM:(OI)(CI)(IO)(F)
> BUILTIN\User:(OI)(CI)(RX)
> NT AUTHORITY\Authenticated Users:(OI)(CI)(IO)(M)
> NT AUTHORITY\Authenticated Users:(AD)
> Mandatory Label\Low Mandatory Level:(OI)(CI)(NW)

Something has modified the security on the root directory of your system drive. 
The low-integrity no-write-up (NW) label that's inheritable by directories (CI) 
and files (OI) is the source of the problem. It's supposed to be a 
high-integrity no-write-up (NW) label that applies to files in the root 
directory (OI)(NP) and not to the root directory itself (IO) or subdirectories 
(no CI):

Mandatory Label\High Mandatory Level:(OI)(NP)(IO)(NW)

> I used to UNIX-syntax as a short-hand for specified permissions relating
> to a specified user. I can see how that could introduce misunderstandings
> for everyone glancing over the text.

I was concerned that you were using a third-party tools such as MSYS2 bash to 
check permissions. POSIX rwx access for a user can be computed in terms of 
effective permissions and generic read, write, and execute access rights. But 
there's no equivalent to POSIX owner and group permissions. Access for a user 
SID has to be computed against all entries in the DACL and the mandatory label.

[1] 
https://docs.microsoft.com/en-us/windows/win32/api/winnt/ns-winnt-token_mandatory_policy

--

___
Python tracker 
<https://bugs.python.org/issue42046>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42041] venv subprocess call to python resolves to wrong interpreter

2020-10-15 Thread Eryk Sun


Eryk Sun  added the comment:

PR 22715 is accurate for the current implementation of subprocess. There has 
been discussion in past issues about implementing a PATH search that's similar 
to what the CMD shell does -- as in calling os.system() or Popen() with 
shell=True -- instead of relying on the search that WinAPI CreateProcessW 
implements. The resolved path would be passed as the `executable` argument, 
which becomes the lpApplicationName argument of CreateProcessW.

--

___
Python tracker 
<https://bugs.python.org/issue42041>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42044] Running Python in unbuffered mode may not write all contents to the console

2020-10-15 Thread Eryk Sun


Eryk Sun  added the comment:

Text mode without a buffer isn't reliable. That said, Python 3.9 no longer 
supports Windows 7, so it can remove the 32 KiB limit on console I/O files.

The size limit in Windows 7 and earlier is due to the LPC-based pseudo-files 
that it uses for I/O. Under the hood, console pseudo-files use a 64 KiB heap 
that's shared between the console host process and client processes. 

In Windows 8+, console files are real files that use the ConDrv device instead 
of LPC, in which case there is practically no limit on the size of buffers that 
can be read and written.

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue42044>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42046] Unable to write to file without elevated privileges

2020-10-15 Thread Eryk Sun


Eryk Sun  added the comment:

> Desktop rwx, 
> Username-directory rwx, 

POSIX permissions are not meaningful in Windows. Please run the following two 
commands in a Python script in order to show the current user and the 
access-control list on the directory:

import subprocess
subprocess.call('whoami.exe')
subprocess.call(r'icacls.exe "C:\Users\Username\Desktop"')

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue42046>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42013] venv on Windows with symlinks is broken if invoked with -I

2020-10-12 Thread Eryk Sun


Eryk Sun  added the comment:

> So I'm +1 on fixing this by calling realpath.

In POSIX, calculate_path() in Modules/getpath.c calls calculate_argv0_path() 
before it calls calculate_read_pyenv(), and calculate_argv0_path() in turn 
calls resolve_symlinks(>argv0_path). Thus "pyvenv.cfg" isn't found 
and isn't used to find the standard library in POSIX when a virtual environment 
uses symlinks. Later on, "pyvenv.cfg" gets read by the site module, if 
importing the latter isn't skipped via -S. The site module sets sys._home to 
the `home` value, but sys._home only appears to be used by sysconfig when 
running from the build directory.

In Windows, calculate_path() in PC/getpathp.c could get the final (real) path 
for argv0_path before calling calculate_pyvenv_file(). Then, just like in 
POSIX, the environment's "pyvenv.cfg" file won't be found and won't be used to 
find the standard library when a virtual environment uses symlinks.

--

___
Python tracker 
<https://bugs.python.org/issue42013>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42020] interpreter hangs forever on invalid input

2020-10-12 Thread Eryk Sun


Eryk Sun  added the comment:

In Windows 10 2004, CSI is currently supported by the console host 
(conhost.exe) if virtual-terminal mode is enabled. Windows Terminal Preview 
supports many more C1 control codes, as will conhost.exe in the next release of 
Windows 10. This should be 'fun' considering the C1 controls block isn't 
reserved by Windows filesystems. CMD and PowerShell depend on the C0 controls 
being reserved and depend on the C1 controls not being implemented. They don't 
implement any escaping of control characters when listing filenames in a 
directory, unlike a tradition POSIX shell. Now that the console and Windows 
Terminal implement C1 controls, maybe PowerShell can be updated to escape them 
in file listings, but I don't see what can be done for CMD, which doesn't 
support string literals with escape sequences.

--

___
Python tracker 
<https://bugs.python.org/issue42020>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42020] interpreter hangs forever on invalid input

2020-10-12 Thread Eryk Sun


Eryk Sun  added the comment:

The terminal you're using apparently implements C1 controls [1]. U+009B is the 
Control Sequence Introducer (CSI) for ANSI escape sequences. U+0090 starts a 
Device Control String (DCS), and it gets terminated by U+009C, a String 
Terminator (ST). For example, in GNOME Terminal in Linux:

>>> print('spam\x9b4Deggs')
eggs

>>> print('spam\x90some DCS string\x9c')
spam

---

[1] https://en.wikipedia.org/wiki/C0_and_C1_control_codes

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue42020>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42013] venv on Windows with symlinks is broken if invoked with -I

2020-10-12 Thread Eryk Sun


Eryk Sun  added the comment:

This issue is partly due to bpo-8901, which changed the behavior of the -E and 
-I command-line options to make them ignore the default PythonPath value in the 
registry key "Software\Python\PythonCore\X.Y\PythonPath". The change itself is 
not wrong. It's just exposing an underlying problem.

The `home` path in pyvenv.cfg is from sys._base_executable. In a launcher-based 
environment that's created from the base installation, sys._base_executable is 
the real base executable. OTOH, in a symlink-based virtual environment, 
sys._base_executable is the same as sys.executable. Consequently, if a virtual 
environment is created from a symlink-based virtual environment, the `home` 
path in pyvenv.cfg refers to the creating environment instead of the base 
installation. In this case, with the current implementation, the standard 
library can only be found by falling back on the default PythonPath in the 
registry.

--

___
Python tracker 
<https://bugs.python.org/issue42013>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42013] venv on Windows with symlinks is broken if invoked with -I

2020-10-12 Thread Eryk Sun


Change by Eryk Sun :


--
Removed message: https://bugs.python.org/msg378489

___
Python tracker 
<https://bugs.python.org/issue42013>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42013] venv on Windows with symlinks is broken if invoked with -I

2020-10-12 Thread Eryk Sun


Eryk Sun  added the comment:

> Well, actually the environment variables /should/ not matter as -I 
> implies -E.

The operative word there is "should". I was grasping for anything that might 
explain why I couldn't reproduce the issue.

> Ok, the missing link is that the python you run into needs to 
> be also a symlink venv

Thank you, that reproduces the problem for me.

--

___
Python tracker 
<https://bugs.python.org/issue42013>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42013] venv on Windows with symlinks is broken if invoked with -I

2020-10-12 Thread Eryk Sun


Eryk Sun  added the comment:

What about the PYTHONHOME and PYTHONPATH environment variables?

>>> 'PYTHONHOME' in os.environ
False
>>> 'PYTHONPATH' in os.environ
False

PYTHONHOME should not be set all. PYTHONPATH needs to be set carefully. It 
should never include standard-library paths.

--

___
Python tracker 
<https://bugs.python.org/issue42013>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue42013] venv on Windows with symlinks is broken if invoked with -I

2020-10-12 Thread Eryk Sun


Eryk Sun  added the comment:

I can't reproduce the issue with the normal 3.9.0 distribution from python.org. 
For example:

>>> venv.EnvBuilder(with_pip=False, symlinks=True).create("venv")
>>> subprocess.check_call(["venv\\Scripts\\python.exe", "-Ic", "import sys; 
print(sys.executable)"])
C:\Temp\env\venv\Scripts\python.exe
0

Which Python version(s) and distribution(s) did you test? Do you have the 
PYTHONHOME and/or PYTHONPATH environment variables set?

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue42013>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue21927] BOM appears in stdin when using Powershell

2020-10-12 Thread Eryk Sun

Eryk Sun  added the comment:

I'm closing this as a third-party issue with older versions of PowerShell. 
Newer versions of PowerShell set the output encoding to UTF-8 without a BOM 
preamble. For example:

PS C:\> $PSVersionTable.PSVersion

Major  Minor  Patch  PreReleaseLabel BuildLabel
-  -  -  --- --
7  0  3

PS C:\> $OutputEncoding.EncodingName
Unicode (UTF-8)

PS C:\> echo ¡¢£¤¥ | py -3 -X utf8 -c "print(ascii(input()))"
'\xa1\xa2\xa3\xa4\xa5'

It's still possible to manually set the output encoding to include a BOM 
preamble. For example:

PS C:\> $OutputEncoding = [System.Text.Encoding]::UTF8
PS C:\> $OutputEncoding.GetPreamble()
239
187
191
PS C:\> echo ¡¢£¤¥ | py -3 -X utf8 -c "print(ascii(input()))"
'\ufeff\xa1\xa2\xa3\xa4\xa5'

I don't know what would be appropriate for Python's I/O stack in terms of 
detecting and handling a UTF-8 preamble on any type of file (console/terminal, 
pipe, disk), i.e. using the "utf-8-sig" encoding instead of "utf-8", as opposed 
to just letting scripts detect and handle an initial BOM character (U+FEFF) 
however they see fit. But that discussion needs a new issue if people are 
interested in supporting new behavior.

--
resolution:  -> third party
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue21927>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue26332] OSError: exception: access violation writing <...> (Windows 10 x64, Python 3.5.1)

2020-10-08 Thread Eryk Sun


Eryk Sun  added the comment:

I see nothing to act on in this issue. A common cause of access violations with 
ctypes is truncated pointer values. There's a lot of sloppy ctypes code from 
the 32-bit era that assumes a C int can store a pointer.

--
nosy: +eryksun
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed
type:  -> behavior

___
Python tracker 
<https://bugs.python.org/issue26332>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41961] Windows install failure "could not set file security"

2020-10-07 Thread Eryk Sun


Eryk Sun  added the comment:

> It might be that Python is the first/only MSI that the user 
> has tried though?

It's likely the first per-user MSI install attempted since the security of the 
"Installer" directory was modified. There's no problem with a per-machine 
install.

> did your change reproduce the "Error: 0"? 

It's the same "Could not set file security for file... Error: 0" dialog, which 
is created by the python-3.9.0.exe child process that executes from a copy in 
%TEMP%. 

In Process Monitor, I see the actual access-denied error due to the installer 
service trying to open the directory with WRITE_DAC and WRITE_OWNER access.

If I also remove the Everyone group that grants read and execute access, the 
installer service fails at an earlier step, which results in an 0x80070643 
fatal installation error.

> It sounds like just resetting the owner isn't enough on its own, 
> but the inherited ACLs should include SYSTEM and not prevent it 
> from working. 

The security descriptor that the installer service sets prevents inheritance of 
discretionary access control entries (i.e. the DACL is protected):

>>> sd = GetFileSecurity('Installer', DACL_SECURITY_INFORMATION)
>>> sd.GetSecurityDescriptorControl()[0] & SE_DACL_PROTECTED
4096

Thus the entries in the ACL for SYSTEM and Administrators groups are set 
explicitly instead of inherited from the parent directory. If something else 
rewrites the security on the directory, it can just as easily protect the DACL 
from inheritance.

In my first experiment, I had left the entry for Everyone (WD) in the DACL, 
but, as mentioned above, it turns out that it's required at an earlier step. So 
a fix has to also add it back:

>icacls "%APPDATA%\Microsoft\Installer" /grant:r *WD:(OI)(CI)(RX)

Also, in my first message, I manually re-added the SYSTEM and Administrators 
entries unnecessarily -- though it doesn't hurt to do so. It turns out that all 
the service needs is for the directory's owner to be set back to the 
Administrators group. Then it implicitly has WRITE_DAC access. It gets 
WRITE_OWNER access as well, even though the file security at the time doesn't 
grant it (the owner of an object does not implicitly have WRITE_OWNER access to 
it), so presumably the service is temporarily enabling SeTakeOwnershipPrivilege 
for the process or thread.

--

___
Python tracker 
<https://bugs.python.org/issue41961>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue36534] tarfile: handling Windows (path) illegal characters in archive member names

2020-10-07 Thread Eryk Sun


Eryk Sun  added the comment:

> extract the sanitizing function into a common module 
> (could be *pathlib*?) to avoid duplicates

I would prefer something common, cross-platform, and function-based such as 
os.path.isreservedname and os.path.sanitizename. In posixpath, it would just 
have to reserve and sanitize slash [/] and null [\0]. The real work would be in 
ntpath.

--

___
Python tracker 
<https://bugs.python.org/issue36534>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41965] distutils.spawn.find_executable() fails to find .py files on Windows

2020-10-07 Thread Eryk Sun


Eryk Sun  added the comment:

The existing behavior of find_executable() is unusual compared to a native 
Windows file search via CreateProcessW or SearchPathW, for which a default 
".exe" extension is appended only if the executable name doesn't already have 
an extension. (CreateProcessW supports searching for a file that has no 
extension by adding a trailing dot to the name, which in Windows is equivalent 
to the name without the trailing dot.) 

However, I don't think it's prudent to change the default behavior of this old 
code, especially since there are plans to deprecate distutils per PEP 632. It's 
recommended to migrate existing code to use shutil.which() and 
subprocess.run(). 

There are open issues to improve the behavior of shutil.which() in Windows. For 
example, currently it's limited to PATHEXT extensions, which is a 
generalization of how find_executable() works with ".exe". shutil.which() won't 
find "rst2man.py" if ".PY" isn't in PATHEXT. The proper behavior, if the CMD 
shell is the standard of correctness, is to include the exact name as the first 
candidate to check before trying the name plus the PATHEXT extensions. To be 
more like the CMD shell, a name without an extension should be excluded, but 
effectively included anyway if PATHEXT contains a "." entry.

--

___
Python tracker 
<https://bugs.python.org/issue41965>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41965] distutils.spawn.find_executable() fails to find .py files on Windows

2020-10-07 Thread Eryk Sun


Eryk Sun  added the comment:

The linked code runs subprocess.check_output([rstbin, path]), which won't work 
with "rst2man.py" in Windows. Reliably running a .py script in Windows requires 
running it explicitly via py[w].exe or python[w].exe, or via sys.executable 
from an existing Python process. 

subprocess.Popen calls WinAPI CreateProcessW, which supports PE executables, 
and also CMD/BAT scripts via %ComSpec%. With shell=True the command is run via 
`cmd.exe /c`, which tries ShellExecuteExW in order to execute the file using 
the filetype's default registered action. However, the default action for .py 
files isn't dependable. It's often configured to open the script in an editor 
or IDE.

Thus script entrypoints in Windows get installed as wrapped launchers, e.g. 
"script.exe" instead of "script.py". The launcher executes the embedded 
entrypoint script via the fully-qualified path of python.exe.

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue41965>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41961] Windows install failure "could not set file security"

2020-10-06 Thread Eryk Sun


Eryk Sun  added the comment:

> Anyone else know of any possible causes for this?

The installer service runs as SYSTEM, with an access token at system integrity 
level that has full privileges and the administrators group enabled. Nothing in 
the file security should prevent the service from opening 
"%APPDATA%\Microsoft\Installer" with WRITE_DAC [1] and WRITE_OWNER [2] access, 
assuming it uses backup semantics with the restore privilege enabled, but 
apparently it doesn't.

I was able to reproduce the error dialog by changing the owner of the 
"Installer" folder to the current user and removing the two DACL entries that 
grant access to Administrators and SYSTEM. I then restored normal operation by 
changing the owner of the folder back to the Administrators group and adding 
the full-control DACL entries for Administrators (BA) and SYSTEM (SY):

takeown /a /f "%APPDATA%\Microsoft\Installer"
icacls "%APPDATA%\Microsoft\Installer" /grant:r *BA:(OI)(CI)(F) 
*SY:(OI)(CI)(F)

---

[1] A security context (access token) is allowed WRITE_DAC access to an object 
(to modify its discretionary ACL and security attributes) if the object's 
mandatory label allows write access to the token's integrity level and either 
the object's discretionary ACL explicitly grants WRITE_DAC access to the 
token's user and enabled groups or implicitly grants this access given the 
object's discretionary ACL does not contain an "OWNER RIGHTS" entry and the 
object's owner is the token's user or in the token's enabled groups. In 
particular for a kernel file object (e.g. opening a filesystem file/directory), 
WRITE_DAC access is always allowed if an open uses backup semantics and 
SeRestorePrivilege is enabled.

[2] A security context (access token) is allowed WRITE_OWNER access to an 
object (to modify its owner, group, and mandatory label) if the token has 
SeTakeOwnershipPrivilege enabled or if the object's mandatory label allows 
write access to the token's integrity level and the object's discretionary ACL 
grants WRITE_OWNER access to the token's user and enabled groups. The object's 
owner can be set to any security principal if the token has SeRestorePrivilege 
enabled, else it's limited to the access token user and any of the enabled 
groups in the access token that is flagged SE_GROUP_OWNER (typically the 
Administrators group). In particular for a kernel file object (e.g. opening a 
filesystem file/directory), WRITE_OWNER access is always allowed if an open 
uses backup semantics and SeRestorePrivilege is enabled.

--

___
Python tracker 
<https://bugs.python.org/issue41961>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue36534] tarfile: handling Windows (path) illegal characters in archive member names

2020-10-06 Thread Eryk Sun


Eryk Sun  added the comment:

> This issue is about tarfile. Maybe create another issue to enhance 
> the pathlib module?

IIRC there's already an open issue for that. But in case anyone were to look to 
pathlib as an example of what should be reserved, I wanted to highlight here 
how its reserved_names list is incomplete and how its is_reserved() method is 
insufficient.

--

___
Python tracker 
<https://bugs.python.org/issue36534>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue36534] tarfile: handling Windows (path) illegal characters in archive member names

2020-10-06 Thread Eryk Sun


Eryk Sun  added the comment:

> The pathlib module has _WindowsFlavour.reserved_names list of 
> Windows reserved names:

pathlib._WindowsFlavour.reserved_names is missing "CONIN$" and "CONOUT$". Prior 
to Windows 8 these two are reserved as relative names. In Windows 8+, they're 
also reserved in directories, just like the other reserved device names.

pathlib._WindowsFlavour.is_reserved() fails to reserve names containing ASCII 
control characters [0-31], vertical bar [|], the file-stream delimiter [:] 
(i.e. "filename:streamname:streamtype"), and the five wildcard characters 
[*?"<>]. (Maybe it should allow the file-stream delimiter, but that requires 
validating that a file stream is proper.) It fails to reserve names that end 
with a dot or space, which includes UNC and device paths except for \\?\ 
verbatim paths. It fails to match all reserved base names, which begin with a 
reserved device name, followed by zero or more spaces, a dot or colon, and zero 
or more characters. If names that contain colon are already reserved, then this 
check only has to be modified to strip trailing spaces before comparing against 
the list of reserved device names.

--

___
Python tracker 
<https://bugs.python.org/issue36534>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41941] Py_Initialize affects the console

2020-10-05 Thread Eryk Sun


Eryk Sun  added the comment:

> config_init_stdio() is not called in an isolated configuration:

config_init_stdio wasn't being called anyway since Py_Initialize uses 
_PyConfig_InitCompatConfig. The issue was primarily due to the LC_CTYPE locale 
being set to the default user locale, as I discussed in msg378024.

Note that for legacy mode, i.e. Py_LegacyWindowsStdioFlag = 1, there's no 
simple way to not modify the standard I/O files. The io stack needs binary 
mode. You'd have to use and modify duped file descriptors.

--

___
Python tracker 
<https://bugs.python.org/issue41941>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41941] Py_Initialize affects the console

2020-10-05 Thread Eryk Sun


Eryk Sun  added the comment:

Having looked at the screenshots, it seems that your issue is in part due to 
non-legacy mode not setting the standard I/O files to binary mode (_O_BINARY) 
from their default ANSI text mode (_O_TEXT). This is not a problem on its own. 

The real issue is that you're not using an isolated configuration, as Victor 
suggested, so the LC_CTYPE locale gets set to the default user locale instead 
of "C". When writing to the console in ANSI text mode with a configured locale 
other than the default "C" locale, the C runtime _write() function does a 
double translation from the locale encoding to the console codepage encoding 
via the internal function write_double_translated_ansi_nolock().

--

___
Python tracker 
<https://bugs.python.org/issue41941>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41941] Py_Initialize affects the console

2020-10-05 Thread Eryk Sun


Change by Eryk Sun :


--
components: +IO, Unicode, Windows
nosy: +ezio.melotti, paul.moore, steve.dower, tim.golden, vstinner, zach.ware

___
Python tracker 
<https://bugs.python.org/issue41941>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41941] Py_Initialize affects the console

2020-10-05 Thread Eryk Sun


Eryk Sun  added the comment:

Python supports Unicode in a Windows console session by using the console API's 
wide-character functions (i.e. ReadConsoleW and WriteConsoleW) with UTF-16 
encoded text. This is implemented in the io stack via io._WindowsConsoleIO, and 
for PyOS_Readline (e.g. builtin `input`) via _PyOS_WindowsConsoleReadline. The 
UTF-16 aspect is an internal detail that's presented externally as UTF-8 by 
automatically converting between the two.

Notwithstanding the behavior of third-party packages, CPython intentionally 
makes no changes to a console session's global settings, including:

* input & output codepages
* input & output modes (except for msvcrt.getwch, etc)
* cursor size and visibility
* screen-buffer size, font, colors, & attributes
* window size
* window title

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue41941>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41925] Lowercase path to python.exe in pip.exe from venv is throwing error

2020-10-04 Thread Eryk Sun


Eryk Sun  added the comment:

> I wouldn't think that changing locales would have an effect on Windows
> paths being case sensitive or not, otherwise folks from other countries
> would be experiencing this error. Perhaps the mixing of locale and system
> language? I had locale set to Japan but system language set to English

It shouldn't make a difference. Filenames are Unicode, the NTFS case table is 
set on disk when the filesystem is formatted, and the launcher uses 
wide-character strings and calls CreateProcessW. I wish I could offer up a 
reasonable explanation, but I think this one stays in the category of strange 
problems fixed by turning it off and on again.

--

___
Python tracker 
<https://bugs.python.org/issue41925>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41929] Detect OEM code page for zip archives in ZipFile based on system locale

2020-10-04 Thread Eryk Sun


Eryk Sun  added the comment:

This is already addressed in bpo-28080, which adds an `encoding` parameter -- 
e.g. `encoding="oem"` ("oem" is only available in Windows). Unfortunately 
bpo-28080 has languished without resolution for four years.

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue41929>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41925] Lowercase path to python.exe in pip.exe from venv is throwing error

2020-10-03 Thread Eryk Sun


Eryk Sun  added the comment:

> "C:\Users\chris\code\project\myenv\Scripts\python.exe"

Have you tried different case combinations for "Users" and "Scripts"?

* C:\users\chris\code\project\myenv\scripts\python.exe
* C:\users\chris\code\project\myenv\Scripts\python.exe
* C:\Users\chris\code\project\myenv\scripts\python.exe
* C:\Users\chris\code\project\myenv\Scripts\python.exe

> fsutil.exe file queryCaseSensitiveInfo C:\

Have you checked "C:\Users\chris\code\project\myenv"?

---

Off Topic

> Open Powershell (via both native Powershell app or the new 
> Windows Terminal) 

FYI, "native PowerShell app" is not the right way to think of this. Both cases 
run the same command-line interface (CLI) shell (i.e. pwsh.exe or 
powershell.exe), which is a console application that either inherits or 
allocates a console session (i.e. an instance of conhost.exe or 
openconsole.exe). When run from a graphical desktop application (e.g. 
explorer.exe), a console application usually has to allocate a new console 
session since there's none to inherit. Currently there's no way to set a 
default terminal for console applications, so a regular console session is 
allocated, which creates its own window. When run from a terminal in Windows 10 
(e.g. Windows Terminal), a console application inherits a headless console 
session (i.e. pseudoconsole mode), and the user interface is provided by the 
terminal.

--

___
Python tracker 
<https://bugs.python.org/issue41925>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41925] Lowercase path to python.exe in pip.exe from venv is throwing error

2020-10-03 Thread Eryk Sun


Change by Eryk Sun :


--
Removed message: https://bugs.python.org/msg377918

___
Python tracker 
<https://bugs.python.org/issue41925>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41925] Lowercase path to python.exe in pip.exe from venv is throwing error

2020-10-03 Thread Eryk Sun


Eryk Sun  added the comment:

> "C:\Users\chris\code\project\myenv\Scripts\pip.exe"

Have you tried different case combinations for "Users" and "Scripts"?

* C:\users\chris\code\project\myenv\scripts\pip.exe
* C:\users\chris\code\project\myenv\Scripts\pip.exe
* C:\Users\chris\code\project\myenv\scripts\pip.exe
* C:\Users\chris\code\project\myenv\Scripts\pip.exe

> fsutil.exe file queryCaseSensitiveInfo C:\

Have you checked "C:\Users\chris\code\project\myenv"?

---

Off Topic

> Open Powershell (via both native Powershell app or the new 
> Windows Terminal) 

FYI, "native PowerShell app" is not the right way to think of this. Both cases 
run the same command-line interface (CLI) shell (i.e. pwsh.exe or 
powershell.exe), which is a console application that either inherits or 
allocates a console session (i.e. an instance of conhost.exe or 
openconsole.exe). When run from a graphical desktop application (e.g. 
explorer.exe), a console application usually has to allocate a new console 
session since there's none to inherit. Currently there's no way to set a 
default terminal for console applications, so a regular console session is 
allocated, which creates its own window. When run from a terminal in Windows 10 
(e.g. Windows Terminal), a console application inherits a headless console 
session (i.e. pseudoconsole mode), and the user interface is provided by the 
terminal.

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue41925>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41908] Make IDLE Start Menu entry more descriptive

2020-10-02 Thread Eryk Sun


Eryk Sun  added the comment:

> maybe we should consider moving the bitness to the folder title 
> (so the folder becomes "Python 3.8 64-bit")?

Splitting up the start-menu entries into separate "Python 3.9 (32-bit)" and 
"Python 3.9 (64-bit)" folders would reduce clutter in the folder when browsing 
the start menu. But the link names have to retain the bitness. If you type 
"idle" in the start menu, all you get is the link name, not the folder context, 
so the link name has to distinguish 32-bit vs 64-bit. Also, link names without 
bitness are already used by the Python store app distribution.

> On Windows, we could go for something, like
> Shell-Editor (3.9 64-bit)

Then searching for "idle" no longer works, which may be a practical trade-off 
considering novices are probably unfamiliar with IDLE.

With or without "IDLE" in the name, this only works well for novices that 
browse the start menu. It's not discoverable if the user just taps the WIN key 
and types "python". That search will return links to python.exe, which uses a 
system console session instead of the IDLE shell. If IDLE should be preferred, 
then maybe the name should be "Python 3.9 Shell+Editor (64-bit)".

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue41908>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue12836] ctypes.cast() creates circular reference in original object

2020-09-29 Thread Eryk Sun


Change by Eryk Sun :


--
stage:  -> needs patch
versions: +Python 3.10, Python 3.8, Python 3.9 -Python 2.7, Python 3.3

___
Python tracker 
<https://bugs.python.org/issue12836>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41883] ctypes pointee goes out of scope, then pointer in struct dangles and crashes

2020-09-29 Thread Eryk Sun


Eryk Sun  added the comment:

> `data_as` method which has the desired behavior: "The returned 
> pointer will keep a reference to the array."

I don't think it's the desired behavior at all. data_as() sets an _arr 
attribute of which ctypes isn't aware. It should cast the address to the given 
type and manually set the array reference in the _objects dict, which ctypes 
will automatically carry forward in all instances of aggregate types (structs 
and arrays) that reference the numpy array. For example:

>>> p = a.ctypes.data_as(ptype)
>>> p._objects['1'] = a

Adding p to an array carries its _objects dict forward:

>>> ptarr = (ptype * 1)(p)
>>> ptarr._objects['0']['1'] is a
True

If the returned pointer is cast() again, then bpo-12836 is an issue. For 
example:

>>> p2 = ctypes.cast(p, ctypes.c_void_p)
>>> p._objects is p2._objects
True
>>> for k in p._objects:
... if p._objects[k] is p:
... print('circular reference')
... 
circular reference

That needs to be fixed. But relying on _arr instead of correctly integrating 
with ctypes isn't a good idea, IMO. Work around the actual bug instead of 
introducing behavior that risks crashing just for the sake of resolving an 
uncommon circular reference problem.

--

___
Python tracker 
<https://bugs.python.org/issue41883>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41883] ctypes pointee goes out of scope, then pointer in struct dangles and crashes

2020-09-28 Thread Eryk Sun


Eryk Sun  added the comment:

I think this is a numpy issue. Its data_as() method doesn't support the ctypes 
_objects protocol to keep the numpy array referenced by subsequently created 
ctypes objects. For example:

import ctypes
import numpy as np

dtype = ctypes.c_double
ptype = ctypes.POINTER(dtype)

class array_struct(ctypes.Structure):
_fields_ = (('dim1', ctypes.c_int),
('dim2', ctypes.c_int),
('ptr', ptype))

n = 12
m = 50
a = np.ones((n, m), dtype=dtype)
p = a.ctypes.data_as(ptype)

data_as() is implemented as a cast() of the ctypes._data pointer. This is a 
c_void_p instance for the base address of the numpy array, but it doesn't 
reference the numpy array object itself. The pointer returned by cast() 
references this _data pointer in its _objects:

>>> p._objects
{139993690976448: c_void_p(42270224)}

data_as() also sets a reference to the numpy array object as a non-ctypes _arr 
attribute:

>>> p._arr is a
True

This _arr attribute keeps the array referenced for the particular instance, but 
ctypes isn't aware of it. When the object returned by data_as() is set in a 
struct, ctypes only carries forward the c_void_p reference from _objects that 
it's aware of:

>>> a_wrap1 = array_struct(n, m, p)
>>> a_wrap1._objects
{'2': {139993690976448: c_void_p(42270224)}}

It would be sufficient to keep the numpy array alive if this c_void_p instance 
referenced the array, but it doesn't. Alternatively, data_as() could update the 
_objects dict to reference the array. For example:

>>> p._objects['1'] = a
>>> a_wrap2 = array_struct(n, m, p)
>>> a_wrap2._objects['2']['1'] is a
True

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue41883>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41862] can not open file in system folder

2020-09-25 Thread Eryk Sun


Eryk Sun  added the comment:

> File "c:\users\chris\appdata\local\programs\python\python38-32
> \lib\runpy.py", line 194, in _run_module_as_main

You have to use 64-bit Python in order to load a 64-bit DLL from the native 
"%SystemRoot%\System32" directory. The fact that the installation directory is 
"python38-32" implies that you're running 32-bit Python. In 64-bit Windows, a 
32-bit program executes as a WOW64 process, for which "%SystemRoot%\System32" 
gets redirected to "%SystemRoot%\SysWOW64".

--
nosy: +eryksun
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue41862>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue24757] Installing Py on Windows: Need to restart or logout for path to be added

2020-09-24 Thread Eryk Sun


Eryk Sun  added the comment:

> We already update the current shell, so if you start a new 
> terminal/Powershell/cmd/etc. instance it should have the 
> updated variable. 

The installer broadcasts an "Environment" message to top-level windows. In 
practice, only Explorer listens for and reloads its environment for this window 
message. For most programs, a new instance has to be run from Explorer in order 
to get the updated environment. 

CMD and PowerShell don't own any windows, so they cannot get this message in 
principle. The console-session host process (conhost or openconsole) may own a 
top-level window if it wasn't allocated with CREATE_NO_WINDOW and wasn't 
created as a ConPTY server (e.g. under Windows Terminal). But the console 
doesn't have a console control event for this case (e.g. something like a 
hypothetical CTRL_ENVIRONMENT_EVENT) that it sends to client processes. It's 
assumed that console applications that need to interact with the desktop 
environment will create at least a hidden window.

Without the "Environment" message, it's still possible for the user to run a 
script that reloads the environment in place in CMD or PowerShell. The 
environment has to be loaded in four passes: system REG_SZ values, system 
REG_EXPAND_SZ values, user REG_SZ values, and user REG_EXPAND_SZ values. User 
values should always be able to depend on system values, and REG_EXPAND_SZ 
values should always be able to depend on REG_SZ values. This is the way WinAPI 
CreateEnvironmentBlock has always worked. When reloading PATH, this is an issue 
if a new machine PATH value depends on new/modified REG_SZ system variables or 
if a new user PATH value depends on new/modified system variables or 
new/modified REG_SZ user variables.

--

___
Python tracker 
<https://bugs.python.org/issue24757>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41849] Support reading long lines with io._WindowsConsoleIO

2020-09-24 Thread Eryk Sun


Eryk Sun  added the comment:

> The biggest risk here is that we have to emulate GNU readline for 
> compatibility, which severely limits the data that can be passed 
> through, and also forces multiple encoding/decoding passes. 

I'm not suggesting to disable the console's line-input and echo-input modes and 
implement our own line editor. If people don't like the built-in line editor 
that the console provides, they can use pyreadline, which directly uses the 
console's low-level API via ctypes.

Here's an example of what I would like to just work by default in Python:

import sys
import win32console

def write_input(h, s):
 records = []
 for c in s:
 b = c.encode('utf-16le')
 for i in range(0, len(b), 2):
 r = win32console.PyINPUT_RECORDType(win32console.KEY_EVENT)
 r.KeyDown = True
 r.RepeatCount = 1
 r.Char = b[i:i+2].decode('utf-16le', 'surrogatepass')
 records.append(r)
 h.WriteConsoleInput(records)

def write_and_read_line(s):
if '\r' not in s:
s += '\r'
h = win32console.GetStdHandle(win32console.STD_INPUT_HANDLE)
mode = h.GetConsoleMode()
h.SetConsoleMode(mode & ~win32console.ENABLE_ECHO_INPUT)
try:
write_input(h, s)
line = sys.stdin.readline()
finally:
h.SetConsoleMode(mode)
return line

>>> src_line = 'a' * 32765 + '\r'
>>> res_line = write_and_read_line(src_line)
>>> assert res_line == src_line.replace('\r', '\n')
Traceback (most recent call last):
  File "", line 1, in 
AssertionError
>>> len(res_line)
511
>>> res_line[:5], res_line[-5:]
('a', '\n')


Currently the ReadConsoleW buffer in read_console_w is capped at 512 (BUFSIZ) 
characters. With the console's processed-input mode enabled, it writes a 
trailing CRLF instead of the raw CR. So the user is limited to typing or 
pasting just 510 characters on a single line. 

I was only thinking to increase the default maximum size up to 32K in 
read_console_w -- removing the fixed BUFSIZ aspect of the implementation in 
favor of capping the buffer used at BUFMAX. In practice this also requires 
similar default increases for the BufferedReader size and TextIOWrapper chunk 
size.

--

___
Python tracker 
<https://bugs.python.org/issue41849>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41849] Support reading long lines with io._WindowsConsoleIO

2020-09-23 Thread Eryk Sun


New submission from Eryk Sun :

io._WindowsConsoleIO reads from the console via ReadConsoleW in line-input 
mode, which limits the line length to the maximum of 256 and the size of the 
client buffer, including the trailing CRLF (or just CR if processed-input mode 
is disabled). Text that's typed or pasted beyond the length limit is ignored. 
The call returns when a carriage return ("\r") is read or the user types the 
enter key anywhere on the line.

Currently the buffer that _WindowsConsoleIO passes to ReadConsoleW is capped at 
512 wide characters, based on the C runtime's BUFSIZ (512) macro. This is too 
small. Legacy mode (i.e. PYTHONLEGACYWINDOWSSTDIO) uses io.FileIO with an 8 KiB 
buffer, which is 8K characters when the console input codepage is a legacy 
single-byte encoding. _WindowsConsoleIO should support at least this much. 

I'd prefer that it allowed up to 32K characters, which is the upper limit for a 
process command line or for a long filepath. By way of comparison, input(), 
which calls _PyOS_WindowsConsoleReadline if stdin is a console file, is 
currently capped at 16K characters.

To be able to read up to 32K characters also requires increasing the 
BufferedReader default buffer size and TextIOWrapper chunk size to 96 KiB (BMP 
codes in the range 0x0800-0x encode as a 3-byte UTF-8 sequence) in order to 
ensure that f.readline() and f.buffer.readline() request the maximum size. This 
would require changing _io__WindowsConsoleIO___init___impl to set 
`self->blksize = 96 * 1024` when `console_type == 'r'`, as well as changing 
_io_open_impl to manually set the _CHUNK_SIZE of the TextIOWrapper to 96 KiB 
for console input (type 'r'). Maybe TextIOWrapper should just query the raw 
_blksize as the initial chunk size. That would remove the need to manually set 
_CHUNK_SIZE.

--
components: IO, Unicode, Windows
messages: 377436
nosy: eryksun, ezio.melotti, paul.moore, steve.dower, tim.golden, vstinner, 
zach.ware
priority: normal
severity: normal
stage: needs patch
status: open
title: Support reading long lines with io._WindowsConsoleIO
type: enhancement
versions: Python 3.10, Python 3.9

___
Python tracker 
<https://bugs.python.org/issue41849>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue35144] TemporaryDirectory clean-up fails with unsearchable directories

2020-09-22 Thread Eryk Sun


Eryk Sun  added the comment:

It seems to me that if `path == name`, then resetperms(path) and possibly a 
recursive call are only needed on the first call. In subsequent calls, if `path 
== name`, then we know that resetperms(path) was already called, so it 
shouldn't handle PermissionError. If resetperms was ineffective (e.g. in 
Windows, a sharing violation or custom discretionary/mandatory permissions), or 
if something else changed the permissions in the mean time, just give up 
instead of risking a RecursionError or stack overflow. For example:


@classmethod
def _rmtree(cls, name, first_call=True):
resetperms_funcs = (_os.unlink, _os.rmdir, _os.scandir, _os.open)

def resetperms(path):
try:
_os.chflags(path, 0)
except AttributeError:
pass
_os.chmod(path, 0o700)

def onerror(func, path, exc_info):
if (issubclass(exc_info[0], PermissionError) and
  func in resetperms_funcs and (first_call or path != name)):
try:
if path != name:
resetperms(_os.path.dirname(path))
resetperms(path)
try:
_os.unlink(path)
# PermissionError is raised on FreeBSD for directories
except (IsADirectoryError, PermissionError):
cls._rmtree(path, first_call=False)
except FileNotFoundError:
pass
elif issubclass(exc_info[0], FileNotFoundError):
pass
else:
raise

_shutil.rmtree(name, onerror=onerror)

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue35144>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue25655] Python errors related to failures loading DLL's lack information

2020-09-22 Thread Eryk Sun


Eryk Sun  added the comment:

> " OSError: [WinError 126] The specified module could not be found" is 
> raised when calling ctypes.CDLL(dll_path) even when this "dll_path" 
> exists... because the error comes from another DLL.

That's the old error. bpo-36085 changed it to FileNotFoundError, with the 
message "Could not find module '%.500S'. Try using the full path with 
constructor syntax." bpo-39393 modified the message to "Could not find module 
'%.500S' (or one of its dependencies). Try using the full path with constructor 
syntax."

IMO, the most direct way to resolve the problem is by enabling "loader snaps" 
for python.exe via gflags and attaching a native debugger to the process. The 
loader outputs debug strings that show the computed DLL search path (from 
LdrpComputeLazyDllPath), each attempt to resolve the dependent DLL to a 
directory in the search path (via LdrpResolveDllName), and the final result 
from loader's work queue (from LdrpProcessWork), which includes the dependent 
DLL that caused loading to fail and the parent module (DLL or EXE) that depends 
on it.

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue25655>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41821] Printing specific Unicode characters causes unwanted beeping in Windows 7 console

2020-09-21 Thread Eryk Sun


Eryk Sun  added the comment:

I've decided to close this issue since extending the "Using Python on Windows" 
section of the docs to recommend using a TrueType font in the console would 
only be generally useful for people using Python 3.8 with Windows 7. Python 
3.9+ requires Windows 8.1+, which defaults to using a TrueType font in console 
windows. In Windows 10, the console doesn't even support OEM "Raster Fonts", 
unless the option is selected to use the legacy console.

--
resolution:  -> third party
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue41821>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41821] Printing specific Unicode characters causes unwanted beeping in Windows 7 console

2020-09-20 Thread Eryk Sun

Eryk Sun  added the comment:

> It is set to use an OEM raster font.

Okay, then the issue can either be closed as third-party or changed to a 
documentation enhancement. It could be documented that Unicode support requires 
selecting a TrueType font in the console properties. "Raster Fonts" uses OEM 
with a lossy best-fit conversion (e.g. "α" -> "a").  

> So I assume that with the UTF-8 flag set the font still has to be 
> set to a TTF-Font so it does not beep?

Yes, the conversion to OEM for "Raster Fonts" occurs in the console itself. 

> I CP850 required to encode those characters via CP850 to the 
> bell character?

Other OEM codepages have a different best-fit mapping. For example, codepage 
437 maps U+2022 to ASCII bell, but not U+2024 or U+2219.

> I was a bit lost on what happens under the hood encoding-wise as 
> sys.stdout.encoding just returns 'utf-8'.

For a console file, the io module uses io._WindowsConsoleIO for the raw layer 
instead of io.FileIO. _WindowsConsoleIO internally uses the console's 
wide-character (UTF-16) API and converts to and from UTF-8. 

Overriding the standard I/O encoding to UTF-8 isn't needed for 
_WindowsConsoleIO. It's needed for regular files and pipes when standard I/O is 
redirected, since those still default to the process active codepage (usually 
system ANSI).

--

___
Python tracker 
<https://bugs.python.org/issue41821>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41821] Printing specific Unicode characters causes unwanted beeping in Windows 7 console

2020-09-20 Thread Eryk Sun


Eryk Sun  added the comment:

If you've selected an OEM raster font in the console properties instead of a 
TrueType font (Consolas, Lucida Console, etc), then the console implements some 
magic to support the OEM character mapping in the raster font. The following 
shows that encoding to codepage 850 (the system OEM codepage in the UK and 
Canada) maps the characters that you've flagged, and only those characters, to 
an ASCII bell (0x07):

>>> [hex(x) for x in range(0x)
...   if codecs.code_page_encode(850, chr(x), 'replace')[0] == b'\x07']
['0x7', '0x2022', '0x2024', '0x2219']

Please confirm whether you're using an OEM raster font. If you are, then this 
issue can be closed.

---

> UnicodeEncodeError: 'charmap' codec can't encode character 
> '\u2023' in position 0: character maps to 

Windows Python defaults to using the process active codepage (WinAPI GetACP) 
for non-console I/O. In Windows 7, the process active codepage is limited to 
the system ANSI encoding, which is limited to a legacy codepage such as 1252. 
You can force standard I/O to use UTF-8 via PYTHONIOENCODING, or force all I/O 
to default to UTF-8 by enabling UTF-8 mode via PYTHONUTF8.

---

For future reference, a console session is hosted by conhost.exe. The cmd.exe 
shell is a console client application, which uses standard I/O just like 
python.exe. If you run python.exe from Explorer, you won't see cmd.exe as a 
parent or in any way involved with the console session.

The confusion stems from how the system pretends that the process that 
allocates a console session owns the console window. This is a convenience to 
allow easily identifying and closing a console session in Task Manager, or via 
taskkill.exe. But it also leads people to confuse the console with the shell 
that they run -- such as CMD or PowerShell.

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue41821>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41602] Python doesn't exit with proper resultcode on SIGINT in runpy (pymain_run_module)

2020-09-16 Thread Eryk Sun


Eryk Sun  added the comment:

> The return code of python on linux when the program is ended with 
> a KeyboardInterrupt should be -2

In general, the exit status for an unhandled KeyboardInterrupt (i.e. 
_Py_UnhandledKeyboardInterrupt) should be the same as the default SIGINT 
handler. This is implemented by exit_sigint in Modules/main.c. In POSIX, it 
uses the actual default SIGINT handler via kill(). In Windows, it uses the exit 
status that the default console control handler would use, 
STATUS_CONTROL_C_EXIT (0xC000_013A).

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue41602>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue14243] tempfile.NamedTemporaryFile not particularly useful on Windows

2020-09-11 Thread Eryk Sun


Eryk Sun  added the comment:

> We'd CreateFile the file and then immediately pass it to 
> _open_osfhandle

Unlike _wopen, _open_osfhandle doesn't truncate Ctrl+Z (0x1A) from the last 
byte when the flags value contains _O_TEXT | _O_RDWR. _wopen implements this to 
allow appending data in text mode. The implementation is based on GetFileType 
(skip pipes and character devices), _lseeki64, _read, and _chsize[_s].

O_TEXT (ANSI text mode) has to be supported for now, but it doesn't properly 
fit in Python 3. The io module opens files using the CRT's binary mode. It 
doesn't implement newline translation for bytes I/O. And io.TextIOWrapper 
doesn't support Ctrl+Z as a logical EOF marker. 

As long as it's supported, O_TEXT should be made the default in os.open (but 
not in msvcrt.open_osfhandle), independent of the CRT default fmode (i.e. 
_get_fmode and _set_fmode). Many callers already assume that's the case. For 
example, tempfile.mkstemp with text=True uses tempfile._text_openflags, which 
doesn't include os.O_TEXT. That assumption is currently wrong if 
_set_fmode(_O_BINARY) is called.

Thankfully, Python has never documented support for the _O_WTEXT, _O_U16TEXT, 
and _O_U8TEXT Unicode text modes in os.open. To my knownledge, there is no 
reasonable way to reimplement these modes. The C runtime doesn't expose a 
public interface to modify a file's internal text and Unicode modes, and 
_open_osfhandle only supports ANSI text mode. If _Py_wopen is implemented, it 
will have to fail the Unicode (UTF-16 or UTF-8) modes with EINVAL. Even without 
_Py_wopen, I'd prefer to modify os.open to fail them because wrapping a 
Unicode-mode fd with io.FileIO doesn't function reliably. FileIO doesn't 
guarantee wchar_t aligned reads and writes, which the CRT requires in Unicode 
mode.

--

___
Python tracker 
<https://bugs.python.org/issue14243>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue14243] tempfile.NamedTemporaryFile not particularly useful on Windows

2020-09-10 Thread Eryk Sun


Eryk Sun  added the comment:

> we'd have to reimplement the UCRT function using the system API. 

Could the implementation drop support for os.O_TEXT? I think Python 3 should 
have removed both os.O_TEXT and os.O_BINARY. 3.x has no need for the C 
runtime's ANSI text mode, with its Ctrl+Z behavior inherited from MS-DOS. I'd 
prefer that os.open always used _O_BINARY and raised a ValueError if passed any 
of the C runtime's text modes, including _O_TEXT, _O_WTEXT, _O_U16TEXT, and 
_O_U8TEXT.

If _O_TEXT is supported, then we have to copy the C runtime's behavior, which 
truncates a Ctrl+Z from the end of the file if it's opened with read-write 
access. If Unicode text modes are supported, then we have to read the BOM, 
which can involve opening the file twice if the caller doesn't request read 
access.

--

___
Python tracker 
<https://bugs.python.org/issue14243>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue14243] tempfile.NamedTemporaryFile not particularly useful on Windows

2020-09-09 Thread Eryk Sun


Eryk Sun  added the comment:

> Nothing preventing someone from contributing the flag on open as well. 

To be clear, supporting delete-access sharing would require re-implementing C 
_wopen in terms of CreateFileW, _open_osfhandle, etc. It could be implemented 
as _Py_wopen in Python/fileutils.c.

--

___
Python tracker 
<https://bugs.python.org/issue14243>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue14243] tempfile.NamedTemporaryFile not particularly useful on Windows

2020-09-09 Thread Eryk Sun


Eryk Sun  added the comment:

> Why do we need to use this O_TEMPORARY flag at all?

Using the O_TEMPORARY flag isn't necessary, but it's usually preferred because 
it ensures the file gets deleted even if the process is terminated abruptly. 
However, it opens the file with delete access, and most Windows programs don't 
share delete access for normal file opens. This can be worked around in Python 
code by using an opener that calls CreateFileW with delete-access sharing. But 
it can't be worked around in general. 

I prefer to provide a way to omit O_TEMPORARY, but still use it by default. 
When it's omitted, I'd also like to be able to close the file within the 
context block without deleting it, for which one use case is to reopen the file 
in a program that doesn't share read or write access. A new delete_on_close 
option would support this case, in addition to providing a way to omit the 
O_TEMPORARY flag. For example:

with tempfile.NamedTemporaryFile(delete_on_close=False) as f:
f.close()
subprocess.run([cmd, f.name])

The file will still be deleted by the context manager, but the tradeoff is that 
it's not as reliable as the default delete-on-close behavior that uses the 
O_TEMPORARY flag.

--

___
Python tracker 
<https://bugs.python.org/issue14243>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41753] subprocess.run on windows does not convert path to string

2020-09-09 Thread Eryk Sun


Eryk Sun  added the comment:

The underlying subprocess.Popen class was updated in 3.8 to support path-like 
objects in `args` (with shell=False) and `executable` on Windows [1]. This 
change was not backported to 3.6 and 3.7.

---

[1] https://docs.python.org/3/library/subprocess.html#subprocess.Popen

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue41753>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue14243] tempfile.NamedTemporaryFile not particularly useful on Windows

2020-09-09 Thread Eryk Sun


Eryk Sun  added the comment:

> For this case, I think the best thing we can probably do is change the 
> default share mode for _all_ opens to include FILE_SHARE_DELETE. 

The C runtime doesn't provide a way to share delete access, except for the 
O_TEMPORARY flag, so Python would have to re-implement open(). Also, this 
doesn't help with re-opening a temporary file in another process since most 
programs do not share file delete access. 

There could be an option to enable a context-manager delete that's independent 
of closing the file. For example, if delete=True and delete_on_close=False, 
then _TemporaryFileCloser.close doesn't delete the file. Instead the file would 
be deleted via os.unlink in _TemporaryFileWrapper.__exit__. The default would 
be delete=True and delete_on_close=True, which would use the O_TEMPORARY flag 
in Windows. Combining delete=False with delete_on_close=True would raise a 
ValueError.

> bringing the default Windows behaviour slightly more in line with 
> how POSIX likes to do things.

In Windows 10, using FILE_SHARE_DELETE is even closer to POSIX behavior when 
the filesystem is NTFS, which supports POSIX delete semantics by renaming the 
file to a hidden system directory ("\$Extend\$Deleted") and setting its delete 
disposition. WinAPI DeleteFileW has been updated to use POSIX semantics if the 
filesystem supports it:

>>> f = tempfile.NamedTemporaryFile()
>>> h = msvcrt.get_osfhandle(f.fileno())
>>> os.unlink(f.name)
>>> info = GetFileInformationByHandleEx(h, FileStandardInfo)
>>> info['DeletePending']
True
>>> GetFinalPathNameByHandle(h, 0)
'?\\C:\\$Extend\\$Deleted\\00180002C4F4301F419F'

--

___
Python tracker 
<https://bugs.python.org/issue14243>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41729] test_winconsoleio fails and hangs on Windows

2020-09-08 Thread Eryk Sun


Eryk Sun  added the comment:

> Is this line needed with a repeat of 1, or should it be removed? 

It's not documented what it means to write a key event with wRepeatCount set to 
0. It happens to work, but I'd leave it set to 1, which means the key was 
pressed once. Ideally, there should also be a subsequent key-up event that's 
the same except bKeyDown is FALSE, but test_winconsoleio doesn't need it.

--

___
Python tracker 
<https://bugs.python.org/issue41729>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41737] Improper NotADirectoryError when opening a file under a fake directory

2020-09-07 Thread Eryk Sun


Eryk Sun  added the comment:

> if NotADirectoryError should be raised, it should mention '/path/to/file' 
> rather then '/path/to/file/somename.txt'.

POSIX specifies that C open() should set errno to ENOTDIR when an existing path 
prefix component is neither a directory nor a symlink to a directory [1]. What 
you propose isn't possible to implement reliably unless the filesystem is 
locked or readonly, so it should be handled by the application instead of by 
the system or standard libraries.

> FileNotFoundError: [Errno 2] No such file or directory: 
> 'C:\\path\\chrome\\to\\file\\somename.txt'

Windows specifies that this case should fail as follows: "[i]f 
Link.File.FileType is not DirectoryFile, the operation MUST be failed with 
STATUS_OBJECT_PATH_NOT_FOUND" [2] (see Phase 6 in the linked pseudocode). 

The Windows API maps this status code to ERROR_PATH_NOT_FOUND (3), which is 
distinct from ERROR_FILE_NOT_FOUND (2). However, the C runtime maps both of 
these system error codes to POSIX ENOENT. All isn't lost, however, because it 
also saves the OS error in _doserrno. io.FileIO could preset _doserrno to 0, 
and if it's non-zero after calling _wopen, use its value with 
PyErr_SetExcFromWindowsErrWithFilenameObject instead of calling 
PyErr_SetFromErrnoWithFilenameObject.

---

[1] https://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html
[2] 
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fsa/8ada5fbe-db4e-49fd-aef6-20d54b748e40

--
nosy: +eryksun

___
Python tracker 
<https://bugs.python.org/issue41737>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41729] test_winconsoleio fails and hangs on Windows

2020-09-06 Thread Eryk Sun

Eryk Sun  added the comment:

> ÄÄ^Z^Z^Z^Z^Z^Z^Z^Z^Z^Z

I don't know why Steve made write_input (PC/_testconsole.c) set wRepeatCount in 
each KeyEvent record to 10. Maybe it was a typo. Previous console 
implementations have ignored the repeat count, so it wasn't an issue.

test_ctrl_z checks (1) that reading a non-ASCII character works when split 
across two reads and (2) that Ctrl+Z (0x1A) as the first character of a read is 
handled as EOF, regardless of where it occurs on the line as long as it's the 
first character in the buffer. The latter has to be tested because 
_WindowsConsoleIO has to manually implement the way that WinAPI ReadFile 
handles Ctrl+Z, a behavior that WinAPI ReadConsoleW does not itself implement.

--

___
Python tracker 
<https://bugs.python.org/issue41729>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue41705] os.makedirs fails on long-path UNC-paths if it is the first sub-folder

2020-09-03 Thread Eryk Sun


Eryk Sun  added the comment:

This behavior is due to issue 37609, i.e. ntpath.splitdrive fails for "UNC" 
device paths.

>>> ntpath.splitdrive('//?/UNC/server/share')
('//?/UNC', '/server/share')

The result should be "//?/UNC/server/share". A path on the "UNC" device 
requires a share component, on which is mounted a local or remote filesystem 
directory. It's functionally part of the 'drive' path. Using just the root path 
or a server path on the "UNC" device is malformed in the context of a normal 
file API open. The former fails as ERROR_INVALID_NAME (123), and the latter 
fails as ERROR_BAD_PATHNAME (161). 

The incorrect splitdrive result in turn makes ntpath.split misbehave:

>>> ntpath.split('//?/UNC/server/share')
('//?/UNC/server', 'share')
>>> ntpath.split('//?/UNC/server')
('//?/UNC/', 'server')

The correct result should be ('//?/UNC/server/share', '').

--
nosy: +eryksun
type: crash -> behavior

___
Python tracker 
<https://bugs.python.org/issue41705>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



  1   2   3   4   5   6   7   8   9   10   >