[issue43749] venv module does not copy the correct python exe
Eryk Sun added the comment: The Windows implementation of symlink_or_copy() actually copies "python.exe" and "pythonw.exe" launchers from "Lib/venv/scripts/nt". One cannot simply copy the "python3.exe" executable because the required DLLs aren't copied. If this seemed to work when testing, it was only because the installation directory was in PATH. The solution that actually works is to copy the launcher as "python3.exe". -- nosy: +eryksun resolution: fixed -> stage: resolved -> status: closed -> open ___ Python tracker <https://bugs.python.org/issue43749> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43749] venv module does not copy the correct python exe
Change by Eryk Sun : -- components: +Windows nosy: +paul.moore, steve.dower, tim.golden, zach.ware versions: +Python 3.11 -Python 3.8 ___ Python tracker <https://bugs.python.org/issue43749> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43749] venv module does not copy the correct python exe
Eryk Sun added the comment:
The code to copy a file could be rewritten to use a regex match. For example:
# Copy src to dst. If src is a base executable, copy a launcher.
dirname, filename = os.path.split(src)
m = re.match(r'(pythonw?)[0-9._]*((?:_d)?(?:\.exe|\.pdb))$',
filename.lower())
if m is not None:
src = os.path.join(os.path.dirname(__file__),
'scripts', 'nt', m.group(1) + m.group(2))
# If the base environment is a Python source build, use
# the launcher in the build directory instead of
# "Lib/venv/scripts/nt".
if sysconfig.is_python_build(True) or not os.path.isfile(src):
basename = ('venvlauncher' if m.group(1) == 'python' else
'venvwlauncher')
src = os.path.join(dirname, basename + m.group(2))
--
___
Python tracker
<https://bugs.python.org/issue43749>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23948] Deprecate os.kill() on Windows
Change by Eryk Sun : -- resolution: -> rejected stage: needs patch -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue23948> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42962] Windows: SystemError during os.kill(..., signal.CTRL_C_EVENT)
Change by Eryk Sun : -- resolution: -> duplicate stage: -> resolved status: open -> closed superseder: -> missing return in win32_kill? ___ Python tracker <https://bugs.python.org/issue42962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14484] missing return in win32_kill?
Eryk Sun added the comment: The details of os.kill() on Windows have been discussed extensively for years in various issues such as bpo-26350, bp-23948, and bp42962. But the problem of the missing return statement is always overwhelmed by discussion of the egregiously bad design of this function and its misuse, which depends on bugs in WinAPI GenerateConsoleCtrlEvent() when passed a PID that is not a process group ID and/or not attached to the console. -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue14484> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14484] missing return in win32_kill?
Change by Eryk Sun : -- Removed message: https://bugs.python.org/msg408336 ___ Python tracker <https://bugs.python.org/issue14484> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14484] missing return in win32_kill?
Eryk Sun added the comment: The details of os.kill() on Windows have been discussed extensively for years in various issues such as bpo-26350, bpo-23948, and bpo-42962. But the problem of the missing return statement is always overwhelmed by discussion of the egregiously bad design of this function and its misuse, which depends on bugs in WinAPI GenerateConsoleCtrlEvent() when passed a PID that is not a process group ID and/or not attached to the console. -- versions: +Python 3.7 -Python 3.10, Python 3.11, Python 3.9 ___ Python tracker <https://bugs.python.org/issue14484> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37701] shutil.copyfile raises SpecialFileError for symlink to fifo
Eryk Sun added the comment: > Raising a SpecialFileError would be OK if `follow_symlinks` was False. I expect it to fail if follow_symlinks is True, which is the default value. I expect it to succeed with follow_symlinks=False, which should create a shallow copy of just the symlink, regardless of its target. Instead, what happens is that it calls shutil._stat(fn) on both src and dst, regardless of follow_symlinks. I think the call should be shutil._stat(fn, follow_symlinks). This requires updating shutil._stat() to pass the value to fn.stat() and os.stat(). -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue37701> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46052] IDLE: make Ctrl, Alt + IME non-ascii letter work on Windows
Eryk Sun added the comment: The alternate keyboard shortcuts for the clipboard work fine with a Russian keyboard layout: copy: Ctrl+Insert cut: Shift+Delete paste: Shift+Insert Some programs also support Alt+Backspace for undo (Ctrl+Z). Watch out with Shift+Delete in GUI shells, since it physically deletes files, bypassing the recycle bin (trash), instead of cutting the selected files to the clipboard. -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46052> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46052] IDLE: make Ctrl, Alt + IME non-ascii letter work on Windows
Eryk Sun added the comment: I think the following wiki article still applies even though it was first discussed in 2003: "KeySyms on platforms other than X11" [1]. In particular, it states the following: On Windows and MacOS X Tk only supports keysyms correctly for a limited number of keys, namely special keys, and the ranges of ASCII and ISO-8859-1 (support of ISO-8859-1 on MacOS X since 8.4.2). I can confirm the result for the Russian keyboard mapping. The value of keysym is "??", even when combined with the control key. x: char: 'ч', ord: 0447, code: 0058, sym: '??', num: 0447. Ctrl+x: char: '\x18', ord: 0018, code: 0058, sym: '??', num: 0447. c: char: 'с', ord: 0441, code: 0043, sym: '??', num: 0441. Ctrl+c: char: '\x03', ord: 0003, code: 0043, sym: '??', num: 0441. v: char: 'м', ord: 043c, code: 0056, sym: '??', num: 043c. Crl+v: char: '\x16', ord: 0016, code: 0056, sym: '??', num: 043c. (I modified the keyevent function from msg408400 to use hexadecimal and repr formatting.) I checked Ubuntu 20.04 with a Russian keyboard layout. It seems in Linux the combination with the control key changes keysym and keysym_num to use the ASCII characters "x", "c", and "v": x: char: 'ч', ord: 0447, code: 0035, sym: 'Cyrillic_che', num: 06de. Ctrl+x: char: '\x18', ord: 0018, code: 0035, sym: 'x', num: 0078. c: char: 'с', ord: 0441, code: 0036, sym: 'Cyrillic_es', num: 06d3. Ctrl+c: char: '\x03', ord: 0003, code: 0036, sym: 'c', num: 0063. v: char: 'м', ord: 043c, code: 0037, sym: 'Cyrillic_em', num: 06cd. Crl+v: char: '\x16', ord: 0016, code: 0037, sym: 'v', num: 0076. --- [1] https://wiki.tcl-lang.org/page/KeySyms+on+platforms+other+than+X11 -- ___ Python tracker <https://bugs.python.org/issue46052> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46056] Cannot use virtual environment on Windows 10 in corporate security settings
Eryk Sun added the comment: Try using symlinks if you're allowed, e.g. `python -m venv --symlinks `. Note that a virtual environment created with symlinks is unreliable in some cases because ShellExecute[Ex]W() eagerly resolves a symlink before calling CreateProcessW(). Also, you won't be able to use EXE script wrappers in an active environment due to the security restrictions in place on your system. You'll need to use `python -m ` alternative commands, such as `python -m pip` instead of `pip`. -- components: -Windows nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46056> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46056] Cannot use virtual environment on Windows 10 in corporate security settings
Eryk Sun added the comment: > symlinks do not work for me Sorry, I forgot that you're using the store app. The store app has to use the copied venv launchers. When a store app is run from the command line, the system executes an appexec link from "%LocalAppData%\Microsoft\WindowsApps". Unfortunately appexec links have to be executed directly in order for CreateProcessW() to find the required app information that's in the link. The API doesn't manually reparse symlinks until it reaches an appexec link. -- ___ Python tracker <https://bugs.python.org/issue46056> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45915] Use fcntl(fd, F_GETFD) to check whether an fd is valid
Eryk Sun added the comment: PR 29821 adds __APPLE__ to the platforms that use fcntl(fd, F_GETFD). Is this okay on macOS, given bpo-30225? Apparently fstat() fails if the other end of a pipe is closed. -- ___ Python tracker <https://bugs.python.org/issue45915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46084] Python 3.9.6 scan_dir returns filenotfound on long paths, but os_walk does not
Eryk Sun added the comment: > Python 3.9.6 scan_dir returns filenotfound on long paths, > but os_walk does not. This would be surprising. os.walk() has been implemented via os.scandir() since Python 3.5. Do you have a concrete example of the directory structure to test? > I see that many people on the internet have said to > change the working directory as a work around. Changing the working directory is a workaround in Unix, not Windows. Without long-path support, the working directory in Windows is limited to 258 (MAX_PATH - 2) characters. Without long-path support, the workaround in Windows is to use an extended path, i.e. a Unicode path that's fully-qualified and normalized -- as returned by os.path.abspath() -- and prefixed by "?\\" or "?\\UNC\\" (e.g. r"\\?\C:\spam" or r"\\?\UNC\server\share\spam"). This allows the native path length limit of about 32760 characters. Some API functions and applications do not support extended paths. In particular setting an extended path as the working directory is unsupported and buggy, even if long-path support is enabled. But extended paths work fine with most file functions in the os and shutil modules, such as os.scandir(), os.stat(), os.open(), shutil.copytree(), and shutil.rmtree(). -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46084] Python 3.9.6 scan_dir returns filenotfound on long paths, but os_walk does not
Eryk Sun added the comment:
It works as expected for me:
>>> len(p)
261
>>> print(p)
C:\Temp\Jim\Documents\jschw_uiowtv3_old\AppData\Local\Google\Chrome\User
Data\Default\Extensions\nenlahapcbofgnanklpelkaejcehkggg\0.1.823.675_0\notifications\pages\Cashback\components\CashBackResolve\components\RewardsActivation\components\CashbackSectionSimple
os.walk() can list the files in directory `p` if the \\?\ prefix is prepended:
>>> next(os.walk('?\\' + p))[-1]
['spam.txt']
Without the prefix, the internal os.scandir() call fails, but by default the
error is ignored:
>>> next(os.walk(p))[-1]
Traceback (most recent call last):
File "", line 1, in
StopIteration
We can print the exception to see that it's the expected ERROR_PATH_NOT_FOUND
(3) error for a path that's too long:
>>> next(os.walk(p, onerror=print))[-1]
[WinError 3] The system cannot find the path specified:
'C:\\Temp\\Jim\\Documents\\jschw_uiowtv3_old\\AppData\\Local\\Google\\Chrome\\User
Data\\Default\\Extensions\\nenlahapcbofgnanklpelkaejcehkggg\\0.1.823.675_0\\notifications\\pages\\Cashback\\components\\CashBackResolve\\components\\RewardsActivation\\components\\CashbackSectionSimple'
Traceback (most recent call last):
File "", line 1, in
StopIteration
--
___
Python tracker
<https://bugs.python.org/issue46084>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46084] Python 3.9.6 scan_dir returns filenotfound on long paths, but os_walk does not
Eryk Sun added the comment: If I had long paths enabled, then next(os.walk(p, onerror=print)) would not have printed the error that I showed in the example and would not have immediately raised StopIteration. Instead it would have returned a (dirpath, dirnames, filenames) result for directory `p`. Did you repeat the simple examples that I showed, exactly as shown, and get a different result? -- ___ Python tracker <https://bugs.python.org/issue46084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46084] Python 3.9.6 scan_dir returns filenotfound on long paths, but os_walk does not
Eryk Sun added the comment: > but errors in DirEntry.is_dir() and DirEntry.is_symlink() > are always ignored In Windows, is_symlink() won't fail due to a long path, since that information comes from the directory listing, but is_dir() might fail for a long path if it's a symlink to a directory. Windows requires that a symlink to a directory is also a directory (i.e. the symlink reparse point is set on an empty directory), but it's not enough to check that it's a directory symlink. is_dir() requires checking that the target exists, which may fail if the path of the link is too long to open and resolve the link target. -- ___ Python tracker <https://bugs.python.org/issue46084> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46143] [docs] IO > Text Encoding info outdated
Eryk Sun added the comment: The rare circumstance in which UTF-8 mode gets enabled automatically is described in the following paragraph [1]: If the PYTHONUTF8 environment variable is not set at all, then the interpreter defaults to using the current locale settings, unless the current locale is identified as a legacy ASCII-based locale (as described for PYTHONCOERCECLOCALE), and locale coercion is either disabled or fails. In such legacy locales, the interpreter will default to enabling UTF-8 mode unless explicitly instructed not to do so. Note that UTF-8 mode is never enabled automatically in Windows. In contrast to POSIX, the locale encoding in Windows is unrelated to the current LC_CTYPE locale. Instead, the locale encoding gets set to the process code page, which is based on the system locale by default and never changes while a process is running. The system locale may be incompatible with the current LC_CTYPE locale, Windows user locale, and preferred UI language (e.g. for text resources such as error messages), so try to explicitly use UTF-8 for text files whenever possible. --- [1] https://docs.python.org/3/library/os.html#utf8-mode -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46143> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41222] POpen bufsize=0 ignored with universal_newlines=True
Eryk Sun added the comment: > Buffering is necessary for implementing the universal_newlines Why is that? I can see that it requires newline state tracking, and the allowance to make two read(fd, &c, 1) system calls for a single read(1) method call, in case a "\n" has to be ignored. testproc-unbuffered.py runs to completion in 3.11 if the following statement that changes the text wrapper's chunk size is added right after creating the Popen() instance: if sys.version_info[0] > 2: process.stdout._CHUNK_SIZE = 1 The initial chunk size for a text wrapper is hard coded as 8192 bytes. For some reason the constructor has no parameter for it. -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue41222> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46153] function fails in exec when locals is given
Eryk Sun added the comment: > If exec gets two separate objects as globals and locals, > the code will be executed as if it were embedded in a > class definition. That's a misleading comparison because a class definition intentionally supports nonlocal closures, which exec() doesn't support and shouldn't support. For example: a = 1 def f(): a = 2 class C: print(a) def g(): a = 2 class C: nonlocal a a = 3 print(a) >>> f() 2 >>> g() 3 exec() executes as module code. Using separate globals and locals mappings doesn't magically change how the code is compiled and executed to make it equivalent to a class definition. To understand the case of separate globals and locals, just remember that assigning to a variable by default makes it a local variable, unless it's declared as a global. Also, class and function definitions are implicitly an assignment, which by default will be local. -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46153> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46153] function fails in exec when locals is given
Eryk Sun added the comment:
> That's taken straight out of the documentation.
Yes, but it's still a misleading comparison.
> Until I understood that exec with two different mapping objects as
> globals and locals behaves as if the code where embedded inside a
> class, I found the reported behaviour totally perplexing.
The basic execution model of Python is that a frame that executes with
non-optimized locals -- in module and class definitions -- can use the same
mapping for globals and locals. Indeed, that's how the interpreter executes
modules. However, exec() is generalized to allow executing module code with
separate globals and locals.
Saying that code will be "executed as if it were embedded in a class
definition" is correct only so far as the fact that globals and locals are
different in this case. But it's also misleading because the code gets compiled
as module-level code, not as class code.
It should be pretty obvious why the following fails:
exec("a = 1\ndef f(): return a\nprint(f())", {}, {})
Assignment is local by default, unless otherwise declared. Function f() has no
access to the local scope where `a` is defined because Python doesn't support
closures over non-optimized locals, particularly because we emphatically do not
want that behavior for class definitions.
It should be equally clear why the following succeeds:
exec("global a\na = 1\ndef f(): return a\nprint(f())", {}, {})
> because a class definition intentionally supports nonlocal closures,
>
>I don't know what you mean by that. Classes are never closures. Only
>functions can be closures.
I didn't say that a class can be a closure. That's never the case because a
class uses non-optimized locals. But a class definition does support free
variables that are bound to an enclosing scope. exec() does not support this,
so the exact same code can execute differently in the context of a class
definition.
> It is equivalent to code executed inside a class scope.
That depends on the code and the context. Please refer to my first example in
comparison to the following:
a = 1
def f():
a = 2
exec('print(a)', globals(), {})
>>> f()
1
It's different behavior for print(a) because both exec() and compile(source,
filename, 'exec') produce module code, not class code. The free variable `a`
gets bound to the global scope for the exec() example, while for the class
definition free variable `a` is bound to the local `a` in the frame of the
function call.
To implement this different behavior, the code object for a class definition
uses bytecode operations such as COPY_FREE_VARS and LOAD_CLASSDEREF, which are
never used for module-level code. For example, from the original example,
here's the class definition code:
>>> dis.dis(f.__code__.co_consts[2])
0 COPY_FREE_VARS 1
2 LOAD_NAME0 (__name__)
4 STORE_NAME 1 (__module__)
6 LOAD_CONST 0 ('f..C')
8 STORE_NAME 2 (__qualname__)
4 10 LOAD_NAME3 (print)
12 LOAD_CLASSDEREF 0 (a)
14 CALL_FUNCTION1
16 POP_TOP
18 LOAD_CONST 1 (None)
20 RETURN_VALUE
--
___
Python tracker
<https://bugs.python.org/issue46153>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46153] function fails in exec when locals is given
Eryk Sun added the comment:
> You seem to be arguing that a description in the docs is "misleading",
> not because it misleads, but because it don't describe a situation
> which has nothing to do with the situation that the docs are describing.
To me it's misleading to say "the code will be executed as if it were embedded
in a class definition" because that is not always the case. The example with
print(a) shows that. One can take it another level to compare function
definitions in a class definition compared to exec(). A function defined in an
exec() is not compiled to bind its free variables to the outer lexical scope in
the context of the exec() call, while a function defined in a class definition
does. For example:
class:
def f():
a = 2
class C:
def g(): print(a)
return C.g
>>> a = 1
>>> g = f()
>>> g()
2
exec():
def f():
a = 2
l = {}
exec('def g(): print(a)', globals(), l)
return l['g']
>>> a = 1
>>> g = f()
>>> g()
1
You asked what I would say in its place, but I don't have a simple answer that
can take the place of the one-liner in the docs. Here's something, but I'm sure
you won't be happy with it:
The code will be executed in a manner that's similar to a class definition with
regard to the use of separate locals and globals scopes. However, there can be
significant differences in certain contexts with regard to how the same code is
compiled for an exec() call compared to a class definition. In particular, code
in a class definition is compiled to bind its free variables to the lexical
scopes of outer function calls in the defining context, which isn't possible
with exec(). Also, the top-level code in a class definition supports `nonlocal`
declarations, which is a syntax error with exec().
--
___
Python tracker
<https://bugs.python.org/issue46153>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46171] venv module produces spurious warning that location has moved
Eryk Sun added the comment: There's no point to making the user worry about short names, symlinks, or non-canonical mount points in the filesystem path of a virtual environment. It's still accessible where the user expects to find it. The problem for bpo-45337 is filesystem redirection in UWP apps, in particular for files and directories created under "%USERPROFILE%\AppData". The warning could be limited to just paths in that tree. -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46171] venv module produces spurious warning that location has moved
Eryk Sun added the comment: There's still the problem of the system using short names in %TEMP%, since by default that's under "%USERPROFILE%\AppData". Either an exception could be made for the temp directory, since it's never redirected (AFAIK), or support could be added for GetLongPathNameW() [1] as nt._getlongpathname(). --- [1] https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getlongpathnamew -- ___ Python tracker <https://bugs.python.org/issue46171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46171] venv module produces spurious warning that location has moved
Eryk Sun added the comment: > There are plenty of other ways to get a venv through a potentially > unexpected path (turns out I've been doing one for years) Examples would be appreciated because I'm drawing a blank here. A junction or directory symlink in the parent path shouldn't be a problem. Otherwise I'd prefer that the solution for bpo-45337 was limited to an app container and paths in "%USERPROFILE%\AppData" (excluding "%TEMP%"). An app container can be detected via the package family name, if any, as returned by GetCurrentPackageFamilyName(). -- ___ Python tracker <https://bugs.python.org/issue46171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46193] Using a dictionary for open files.
Eryk Sun added the comment:
Every time open(r"test.txt", "w") is called, the existing "test.txt" file gets
overwritten in the filesystem as an empty file. For each iteration,
setdefault() returns a reference to the first file object, which advances its
file pointer with each fh.write("\nHello\n") call. For the last write, the file
pointer is at offset 693 (i.e. 7 * 99), and the OS first writes null bytes up
to the file pointer. This is normal behavior, not a bug.
--
nosy: +eryksun
___
Python tracker
<https://bugs.python.org/issue46193>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46171] venv module produces spurious warning that location has moved
Eryk Sun added the comment: > My VHDX mounted in a directory is affected by this I created a VHDX and mounted it in a directory. It's just a regular volume mount point with a junction (IO_REPARSE_TAG_MOUNT_POINT). That won't cause any problems, so I guess your setup must be different in some way. > I suspect (but haven't tested) that some file sharing handlers or > attached devices could also hit it - thinking here of IoT devices > that provide a filesystem-like interface. My takeaway is that because filesystem filter drivers on some systems may arbitrarily redirect some paths in some cases, and redirect differently or not at all in some other context, then we need to always log a warning showing the real path and always embed the real path in pip.exe. The only clear example we have is bpo-45337, but we're erring on the side of caution instead of targeting the warning at the one known case. The generic approach happens to also include any common use of directory symbolic links or junctions, but it's not practical to handle them differently. You're open to special casing short DOS names, however, e.g. by comparing the real path to the long path name from GetLongPathNameW(). Note that if the user continues to use the path with a reparse point or redirection to access the virtual environment (if possible; it's not for bpo-45337), then running `python -m pip` will install scripts or executable script wrappers that refer to the accessed path, not the real path. pip uses distlib for entrypoint scripts. Apparently distlib doesn't resolve the real path of sys.executable when creating the script shebang. For POSIX, including macOS, I don't know the range of possibilities for dependent filesystem redirection. Bind mounts (i.e. mounting an arbitrary path on a directory) can be an issue since they're temporary unless configured in /etc/fstab. However, we can't do anything to warn about them because POSIX realpath() doesn't resolve them. > warning that can be explained in exactly the same way on every platform I guess you mean every edition of Windows. -- ___ Python tracker <https://bugs.python.org/issue46171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34931] os.path.splitext with more dots
Eryk Sun added the comment:
> On Windows the extension of "python.exe" is "exe", not ".exe".
FWIW, a file extension in Windows includes the dot. Trailing dots are stripped
from filenames, so a file can't be named "python." because it's the same as
just "python". (It's possible to prevent stripping trailing dots from a name by
using a \\?\ literal path, but creating such a filename is a bad idea.)
The shell API's file associations include the dot in the file extension. Also,
the PATHEXT environment variable (i.e. the list of extensions that a CLI shell
should try appending in a PATH search) includes the dot in each extension. In
both of the latter cases, an extension that's just "." matches a filename that
has no extension. In other words, the "." file extension can be used to
associate files that have no extension with a ProgID (i.e. a programmatic
identifier, which defines properties and actions for a file type), and adding
"." to PATHEXT includes files that have no extension in a PATH search.
To clarify further, here are some results from PathCchFindExtension() [1]:
import ctypes
path = ctypes.OleDLL('api-ms-win-core-path-l1-1-0')
s = (ctypes.c_wchar * 100)()
ext = ctypes.c_wchar_p()
>>> s.value = 'python.exe'
>>> _ = path.PathCchFindExtension(s, len(s), ctypes.byref(ext))
>>> ext.value
'.exe'
>>> s.value = '...exe'
>>> _ = path.PathCchFindExtension(s, len(s), ctypes.byref(ext))
>>> ext.value
'.exe'
>>> s.value = 'python.'
>>> _ = path.PathCchFindExtension(s, len(s), ctypes.byref(ext))
>>> ext.value
'.'
---
[1]
https://docs.microsoft.com/en-us/windows/win32/api/pathcch/nf-pathcch-pathcchfindextension
--
nosy: +eryksun
___
Python tracker
<https://bugs.python.org/issue34931>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46217] 3.11 build failure on Win10: new _freeze_module changes?
Eryk Sun added the comment: With just PATHCCH_ALLOW_LONG_PATHS, PathCchCombineEx() returns a long path as a normal path (i.e. not a \\?\ extended path) if long-path support is enabled for the current process. Otherwise, if long-path support isn't enabled or available, as is always the case in Windows 8.1, then PATHCCH_ALLOW_LONG_PATHS makes PathCchCombineEx() return a long path as an extended path. In most cases, extended paths work fine. A common exception is that extended paths aren't supported for the working directory. In Windows 10+, PATHCCH_FORCE_ENABLE_LONG_NAME_PROCESS forces PathCchCombineEx() to always return a long path as a normal path, even if long-path support is disabled for the current process. For most cases, this option pretty much guarantees that long paths will cause immediate failures if long-path support is disabled for the current process. In some cases one might want an immediate failure, instead of messing around with extended paths that might fail in some obscure, buggy way. Also, for pure path manipulation one may not care whether the current process can access the path without the \\?\ prefix. -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46217> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46276] ImportError: DLL load failed while importing
Eryk Sun added the comment: "_frida.cp310-win_amd64.pyd" is a bad build or corrupted file. It imports a procedure with no name from a DLL with no name, which causes the loader to search for a file named ".DLL". Even if that were found, it imports another procedure with no name or correct ordinal from "IPHLPAPI.DLL". There are probably more problems with this DLL. -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46276> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46276] ImportError: DLL load failed while importing
Eryk Sun added the comment: > ImportError: DLL load failed while importing _socket: El parámetro no es > correcto. I'm not familiar with the implementation of PyInstaller, which is a third-party tool. The above invalid-parameter error may be indirectly related to the problem with frida, or it may be a separate issue. The problem with frida is simply that "_frida.cp310-win_amd64.pyd" is an invalid or corrupted DLL that the system can't load. I don't know what went wrong with their build and distribution process that caused the problem. It's something to resolve on the frida issue tracker. -- ___ Python tracker <https://bugs.python.org/issue46276> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9937] _winreg.EnumValue causes MemoryError
Eryk Sun added the comment: > Did anyone reproduce on python3.X? In principle, winreg should not have a problem in 3.x because it only uses the UTF-16 wide-character API. -- nosy: +eryksun stage: -> resolved status: pending -> closed ___ Python tracker <https://bugs.python.org/issue9937> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46217] 3.11 build failure on Win10: new _freeze_module changes?
Eryk Sun added the comment: Terry, it's just a simple bug that slipped by. The PATHCCH_OPTIONS enum in the SDK header "PathCch.h" unconditionally defines all of the options. So there's no compiler error when building with a newer version of the SDK, even though we define _WIN32_WINNT to 0x602 (Windows 8). -- nosy: -pablogsal ___ Python tracker <https://bugs.python.org/issue46217> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46297] Python interpreter crashes on bootup with multiple PythonPaths set in registry
Change by Eryk Sun : -- components: +Windows nosy: +paul.moore, tim.golden, zach.ware type: crash -> behavior ___ Python tracker <https://bugs.python.org/issue46297> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46297] Python interpreter crashes on bootup with multiple PythonPaths set in registry
Change by Eryk Sun : -- type: behavior -> crash ___ Python tracker <https://bugs.python.org/issue46297> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46287] UNC path normalisation issues on Windows
Eryk Sun added the comment:
> _Py_abspath/_getfullpathname does not always call GetFullPathNameW on 3.11.
Also, PathCchSkipRoot() doesn't recognize forward slash as a path separator, so
_Py_isabs() is wrong in many cases compared to the same path that uses
backslash as the path separator.
For example, _Py_isabs() wrongly returns false in the following cases, so
GetFullPathNameW() is called, and ironically the misbehavior of _Py_isabs()
leads to the correct result.
>>> os.path.abspath('//spam//eggs. . .')
'spam\\eggs'
>>> os.path.abspath('C:/spam. . .')
'C:\\spam'
>>> os.path.abspath('C:/nul')
'.\\nul'
_Py_isabs() returns true in the following cases, so only normpath() is called:
>>> os.path.abspath(r'\\spam\\eggs. . .')
'spameggs. . .'
>>> os.path.abspath('C:\\spam. . .')
'C:\\spam. . .'
>>> os.path.abspath('C:\\nul')
'C:\\nul'
As the above shows, normpath() doesn't remove trailing dots and spaces from the
last component of a path, and it doesn't special case DOS devices in the last
component of a drive-letter path. The latter is still implemented for the NUL
device in Windows 11 and implemented for all DOS devices in Windows 8.1 and 10
(e.g. CON, CONIN$, CONOUT$, AUX, PRN, COM<1-9>, LPT<1-9>).
I would prefer to remove the _Py_isabs() check from _Py_abspath() in Windows,
unless there's a strong case for startup performance, in which case I'd prefer
to revert the change to nt._getfullpathname() and only use _Py_abspath() during
startup.
--
___
Python tracker
<https://bugs.python.org/issue46287>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46226] User specific paths added to System PATH environment variable
Eryk Sun added the comment: Were you upgrading an existing installation of Python 3.10? Did it actually install in "C:\Program Files\Python310"? Is Python currently installed in per-user "%LocalAppData%\Programs\Python\Python310"? Was it ever installed there? -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46226> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46142] python --help output is too long
Eryk Sun added the comment: > Do you know other projects which dump the full help into stdout > when asking for the "full help"? `ps --help` lists sections "". The "--help all" output is all sections, but truly all help is only available via `man ps`. > For me, the best CLI is "git help", "git help init", etc. "git help > init" opens "man git-init". I don't know its behaviour on platforms > without manpage support, like Windows. With Git for Windows, `git help ` opens an HTML formatted manual page in a web browser. It should use a new tab in an existing browser window. -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46142> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46326] 'virtualenv --clear' should prompt user before nuking entire directory
Eryk Sun added the comment: Ned, the CLI of venv has a `--clear` option, which invokes EnvBuilder [1] with clear=True. It's implemented by EnvBuilder.clear_directory(). Ali wants this method to prompt for confirmation before deleting the directory contents and also proposes the addition of a new `--force` option to skip the prompt. --- [1] https://docs.python.org/3/library/venv.html#api -- nosy: +eryksun status: pending -> open ___ Python tracker <https://bugs.python.org/issue46326> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46326] 'virtualenv --clear' should prompt user before nuking entire directory
Eryk Sun added the comment: The implementation of `--clear` in virtualenv apparently was changed to align with venv. On the linked virtualenv issue, Bernát Gábor suggested creating a bpo issue for venv to determine how the issue should be resolved for virtualenv. -- nosy: +vinay.sajip ___ Python tracker <https://bugs.python.org/issue46326> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46287] UNC path normalisation issues on Windows
Change by Eryk Sun : -- Removed message: https://bugs.python.org/msg410068 ___ Python tracker <https://bugs.python.org/issue46287> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45554] multiprocessing exitcode is insufficiently documented
Eryk Sun added the comment: PR 30142 is sufficient for Unix, but it's missing a bit for Windows. In Windows, the C runtime library maps the console event for Ctrl+Break to SIGBREAK. The default handler for SIGBREAK exits with 0xC13A (i.e. STATUS_CONTROL_C_EXIT). This value is unrelated to the value of signal.SIGBREAK (21) or signal.SIGINT (2). To forcefully terminate a process, Windows taskkill and Task Manager, and pretty much all utilities that can kill a process, call TerminateProcess() with 1 as the exit status. There's no way to know that this is a forced termination as opposed to an unhandled error. The internal terminate() method in Windows uses 0x1 as the real exit status of the process, but the internal wait() method maps this to -signal.SIGTERM (-15). Thus the case of calling Process.terminate(), and only this case, is faked in Windows to look like the process was killed by a signal. -- nosy: +eryksun resolution: fixed -> stage: resolved -> patch review status: closed -> open ___ Python tracker <https://bugs.python.org/issue45554> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45554] multiprocessing exitcode is insufficiently documented
Change by Eryk Sun : -- stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue45554> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46454] '0 -> /dev/null' is lost
Eryk Sun added the comment: > If some one closes fd 0, then he reopens it. it will not be inherited. In Windows, when a console process spawns a child console process without enabling handle inheritance -- e.g. subprocess.Popen(['python.exe']) -- the OS will manually duplicate (not inherit) the standard handles to the child. It doesn't matter in this case that a standard handle isn't inheritable. The latter also doesn't matter when any of the standard handles is overridden by a Popen() call in Windows -- e.g. subprocess.Popen(['python.exe'], stderr=DEVNULL). All of the standard handle values have to be overridden together. Popen() uses this as an opportunity to duplicate an inheritable handle for each that's not overridden, instead of just copying the handle value. On the other hand, if subprocess.Popen() is called in Windows with handle inheritance enabled and without overriding the standard handles -- e.g. subprocess.Popen(['python.exe'], close_fds=False) -- then the standard handles should be inheritable. If any of the standard handles isn't inheritable, then at best the standard handle value in the child will be invalid or used by a handle for a kernel object type other than a file (e.g. process, event). By coincidence, however, a file open in the child could reuse the standard handle value, which can lead to buggy behavior. -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46454> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46490] Add "follow_symlinks=False" support for "os.utime()" on Windows
Eryk Sun added the comment: The Windows API doesn't directly support opening a 'symlink' as Python defines it for the follow_symlinks parameter. That problem should be resolved in a separate issue. Then updating os.utime() would be relatively trivial. -- components: +Windows nosy: +eryksun, paul.moore, steve.dower, tim.golden, zach.ware stage: -> needs patch type: -> enhancement ___ Python tracker <https://bugs.python.org/issue46490> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45382] platform() is not able to detect windows 11
Eryk Sun added the comment:
> It's *very* unlikely you'll ever get output that doesn't fit into MBCS,
When writing to a pipe, wmic.exe hard codes using the process OEM code page
(i.e. CP_OEMCP). If it matters, running wmic.exe with subprocess should use
encoding='oem' instead of text=True.
That said, wmic.exe is deprecated. I suggest using PowerShell instead. For
example:
import os
import json
import subprocess
cmd = 'Get-CimInstance Win32_OperatingSystem | Select Caption, Version |
ConvertTo-Json'
p = subprocess.run(f'powershell.exe -c "{cmd}"', capture_output=True,
encoding=os.device_encoding(1))
result = json.loads(p.stdout)
PowerShell uses the console's output code page (i.e. os.device_encoding(1))
when writing to stdout, even if it's a pipe. (If PowerShell is run without a
console via DETACHED_PROCESS, then it outputs nothing to stdout.) The only way
I know of to make PowerShell write UTF-8 to stdout when it's a pipe is by
temporarily changing the console output code page. Assuming the current process
has a console, you have to first get the current code page with
GetConsoleOutputCP(). Change the code page to UTF-8 via
SetConsoleOutputCP(CP_UTF8). Run the PowerShell command. Finally, restore the
original code page.
Maybe subprocess should provide a context manager to set the console code pages
before a call, and restore the previous console code pages and console modes
after a call completes. That's what CLI shells such as CMD do when running an
external program.
--
___
Python tracker
<https://bugs.python.org/issue45382>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46506] [Windows] wrap CreateFile to support follow_symlinks
New submission from Eryk Sun : Issue 46490 proposes to support follow_symlinks in os.utime() on Windows. Instead of duplicating the os.stat() implementation of follow_symlinks, I suggest factoring out a common _Py_CreateFile() function with two additional parameters: traverse (input) and pFileInfo (output). The `traverse` parameter would indicate whether to traverse or open a name-surrogate reparse point. However, FILE_FLAG_OPEN_REPARSE_POINT should take precedence over `traverse` and open any type of reparse point. The pFileInfo parameter would be an optional pointer to a struct that receives the file type (disk, char, pipe), file attributes, and reparse tag. Querying this information is required when `traverse` is false. There may as well be a way to return it. Since Windows 7 hasn't been supported since 3.8, this is also an opportunity to switch to the newer CreateFile2() function [1], which is required in order to use new file flags added to the API such as FILE_FLAG_OPEN_REQUIRING_OPLOCK. --- [1] https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfile2 -- components: Extension Modules, Windows messages: 411516 nosy: eryksun, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: [Windows] wrap CreateFile to support follow_symlinks type: enhancement ___ Python tracker <https://bugs.python.org/issue46506> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46490] Add "follow_symlinks=False" support for "os.utime()" on Windows
Change by Eryk Sun : -- dependencies: +[Windows] wrap CreateFile to support follow_symlinks ___ Python tracker <https://bugs.python.org/issue46490> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45382] platform() is not able to detect windows 11
Eryk Sun added the comment: > Bit wmic seems nice solution. > Is still working for windows lower than 11? wmic.exe is still included in Windows 10 and 11, but it's officially deprecated [1], which means it's no longer being actively developed, and it might be removed in a future update. PowerShell is the preferred way to use WMI. --- [1] https://docs.microsoft.com/en-us/windows/deployment/planning/windows-10-deprecated-features -- ___ Python tracker <https://bugs.python.org/issue45382> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45382] platform() is not able to detect windows 11
Eryk Sun added the comment: > sys.getwindowsversion() which exposes GetVersionEx() looks fine to me. In Windows 8+, sys.getwindowsversion() returns a version that can vary from 6.2.9200 (Windows 8) up to the actual OS version. It depends on the OS versions supported by the application manifest -- with 6.2.9200 used when there's no application manifest. The platform module uses the shell's VER command because it returns the real OS version. The version number for Windows 11 is 10.0.22000+. The platform module misreports this as Windows 10. In msg403452, I suggested a modification to use build numbers. In msg404451 Steve said we may as well use the WMI result to identify Windows 11 based on the OS caption. If we use WMI, we can choose to rely on wmic.exe (deprecated), PowerShell, or implement our own code in C/C++. > I don't understand why we have to handle XML or JSON and encoding... Using JSON or XML (ElementTree) isn't required, if you'd rather parse human-readable output. AFAIK, the Win32_OperatingSystem caption is always ASCII. In general, wmic.exe writes output text to a pipe that's encoded with the system OEM code page. pwsh.exe, powershell.exe, and cmd.exe write output text to a pipe (from their internal commands/cmdlets) that's encoded using the console's output code page. cmd.exe also supports a /U option to use UTF-16. -- ___ Python tracker <https://bugs.python.org/issue45382> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45382] platform() is not able to detect windows 11
Eryk Sun added the comment: > AFAIK, the Win32_OperatingSystem caption is always ASCII. I think I was wrong here. The "Caption" field is localized, so the wmic.exe OEM encoded output to a pipe isn't reliable. The system OEM code page doesn't necessarily match the display/preferred language of the current user. It could be a lossy encoding with default and best-fit translations (e.g. "?"; "α" -> "a"). If using wmic.exe, it's better to redirect the output to a temp file, which will be UTF-16. -- ___ Python tracker <https://bugs.python.org/issue45382> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46454] '0 -> /dev/null' is lost
Eryk Sun added the comment: > stdin is closed > os.open() is called and creates a new fd=0 > a call to os.dup2(fd, 0) is made but is a noop The dup2() silent noop case is a problem, but not the problem that's reported in msg43. The two examples for this issue are fd.py and fd2.py. In fd.py, "/dev/null" is opened, and then, if the fd isn't 0, the fd is duped to inheritable fd 0. In fd2.py, fd 0 is closed, and then "/dev/null" is opened as fd 0, which isn't inheritable in Python 3. I think the only problems regarding fd2.py are cross-platform inconsistency and/or documentation. If a similar example is tested in Windows it actually works, assuming os.devnull is used and that "sleep.exe" is available (e.g. from MSYS2). The child's stdin will be the same file as the parent's stdin. In regard to the standard files, the subprocess documentation states the following: With the default settings of None, no redirection will occur; the child’s file handles will be inherited from the parent. If close_fds is true, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. The above applies only to POSIX. It's not wrong in that case, but it should be emphasized that any of the standard file descriptors that's not overridden has to be inheritable in POSIX. Also, if overriding to a specific file, the file descriptor must be inheritable in POSIX. By default, file descriptors for opened files are not inheritable, so the subprocess documentation should reference os.set_inheritable(). For Windows it says the following: if close_fds is true then no handles will be inherited by the child process unless explicitly passed in the handle_list element of STARTUPINFO.lpAttributeList, or by standard handle redirection. That's mostly right, except the last part about standard handle redirection is vague. If close_fds is true, none of the standard handles of the current process is ever inherited in Windows. They could be in principle, but subprocess wasn't designed that way. When close_fds is true, and the standard handles aren't overridden, subprocess relies on the OS to duplicate (not inherit) the standard handles to the child when spawning a console application (e.g. "python.exe", but not "pythonw.exe"). If at least one is overridden, subprocess duplicates all three as inheritable handles via _make_inheritable(). (This is a choice. It doesn't have to do this. It could simply ensure that its own pipe/null files are inheritable.) Thus, when close_fds is true it never matters whether the current standard handles are inheritable, or whether a specified override is inheritable. This behavior is in stark contrast to how subprocess works in POSIX. -- ___ Python tracker <https://bugs.python.org/issue46454> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46506] [Windows] wrap CreateFile to support follow_symlinks
Eryk Sun added the comment: > switch to the newer CreateFile2() function Apparently we need to allow FILE_SHARE_READ if CreateFile2() is used to implement os.stat(). It's not a big deal, but this is a design flaw. CreateFile2() always uses the kernel I/O flag FILE_DISALLOW_EXCLUSIVE [1] when it calls NtCreateFile(). The intent of this flag is to avoid exclusive read access on the first open of a file when a user lacks write permission. Thus unprivileged users aren't allowed to prevent read access to critical system files. However, the implementation is flawed because it denies access even if the open doesn't request data access. It's also flawed because it allows exclusive read access when there are previous opens even if the previous opens have no data access. Classically, IoCheckShareAccess() only checks for a sharing violation when an open requests read, write, or delete data access (i.e. FILE_READ_DATA, FILE_EXECUTE, FILE_WRITE_DATA, FILE_APPEND_DATA, DELETE). This is clearly explained in [MS-FSA] [2]. An open that only requests metadata access can never cause a sharing violation and poses no problem with regard to blocking access to a file. --- [1] https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntddk/nf-ntddk-iocreatefileex [2] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fsa/8c0e3f4f-0729-49f4-a14d-7f7add593819 -- ___ Python tracker <https://bugs.python.org/issue46506> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46506] [Windows] wrap CreateFile to support follow_symlinks
Eryk Sun added the comment:
Here's an implementation of _Py_CreateFile2() and win32_xstat_impl():
typedef struct {
DWORD type;
DWORD attributes;
DWORD reparseTag;
} _PY_CREATE_FILE_INFO;
static HANDLE
_Py_CreateFile2(LPCWSTR lpFileName,
DWORD dwDesiredAccess,
DWORD dwShareMode,
DWORD dwCreationDisposition,
LPCREATEFILE2_EXTENDED_PARAMETERS pCreateExParams,
BOOL traverse,
_PY_CREATE_FILE_INFO *pCreateInfo)
{
HANDLE hFile;
DWORD error;
FILE_BASIC_INFO fbi;
FILE_ATTRIBUTE_TAG_INFO fati;
BOOL openReparsePoint = FALSE;
BOOL traverseFailed = FALSE;
_PY_CREATE_FILE_INFO cfi = {0};
CREATEFILE2_EXTENDED_PARAMETERS createExParams = {
sizeof(CREATEFILE2_EXTENDED_PARAMETERS)};
if (!pCreateExParams) {
pCreateExParams = &createExParams;
}
pCreateExParams->dwFileFlags |= FILE_FLAG_BACKUP_SEMANTICS;
if (pCreateExParams->dwFileFlags & FILE_FLAG_OPEN_REPARSE_POINT) {
openReparsePoint = TRUE;
} else if (!traverse) {
pCreateExParams->dwFileFlags |= FILE_FLAG_OPEN_REPARSE_POINT;
}
// Share read access if write access isn't requested because
// CreateFile2 uses the NT I/O flag FILE_DISALLOW_EXCLUSIVE.
if (!(dwDesiredAccess & (GENERIC_ALL | GENERIC_WRITE |
FILE_WRITE_DATA | FILE_APPEND_DATA))) {
dwShareMode |= FILE_SHARE_READ;
}
call_createfile:
hFile = CreateFile2(lpFileName, dwDesiredAccess, dwShareMode,
dwCreationDisposition, pCreateExParams);
error = GetLastError(); // success: 0 or ERROR_ALREADY_EXISTS
if (hFile == INVALID_HANDLE_VALUE) {
// bpo-37834: open an unhandled reparse point if traverse fails.
traverseFailed = (error == ERROR_CANT_ACCESS_FILE);
if (openReparsePoint || !(traverse && traverseFailed)) {
return INVALID_HANDLE_VALUE;
}
pCreateExParams->dwFileFlags |= FILE_FLAG_OPEN_REPARSE_POINT;
hFile = CreateFile2(lpFileName, dwDesiredAccess, dwShareMode,
dwCreationDisposition, pCreateExParams);
if (hFile == INVALID_HANDLE_VALUE) {
SetLastError(error);
return INVALID_HANDLE_VALUE;
}
error = GetLastError(); // 0 or ERROR_ALREADY_EXISTS
}
if (!pCreateInfo && (openReparsePoint || (traverse && !traverseFailed)))
{
return hFile;
}
cfi.type = GetFileType(hFile);
if (cfi.type == FILE_TYPE_UNKNOWN && GetLastError() != 0) {
error = GetLastError();
goto cleanup;
}
if (GetFileInformationByHandleEx(
hFile, FileAttributeTagInfo, &fati, sizeof(fati))) {
cfi.attributes = fati.FileAttributes;
cfi.reparseTag = fati.ReparseTag;
} else if (GetFileInformationByHandleEx(
hFile, FileBasicInfo, &fbi, sizeof(fbi))) {
cfi.attributes = fbi.FileAttributes;
} else {
switch (GetLastError()) {
case ERROR_INVALID_PARAMETER:
case ERROR_INVALID_FUNCTION:
case ERROR_NOT_SUPPORTED:
// The file is not in a filesystem.
break;
default:
error = GetLastError();
}
goto cleanup;
}
if (cfi.attributes & FILE_ATTRIBUTE_REPARSE_POINT) {
if (IsReparseTagNameSurrogate(cfi.reparseTag)) {
if (traverseFailed) {
error = ERROR_CANT_ACCESS_FILE;
goto cleanup;
}
} else if (!openReparsePoint && !traverseFailed) {
// Always try to reparse if it's not a name surrogate.
CloseHandle(hFile);
traverse = TRUE;
pCreateExParams->dwFileFlags &= ~FILE_FLAG_OPEN_REPARSE_POINT;
goto call_createfile;
}
}
cleanup:
if (error && error != ERROR_ALREADY_EXISTS) {
CloseHandle(hFile);
hFile = INVALID_HANDLE_VALUE;
} else if (pCreateInfo) {
*pCreateInfo = cfi;
}
SetLastError(error);
return hFile;
}
static int
win32_xstat_impl(const wchar_t *path, struct _Py_stat_struct *result,
BOOL traverse)
{
DWORD error;
BY_HANDLE_FILE_INFORMATION fileInfo;
_PY_CREATE_FILE_INFO cfi;
int retval = 0;
HANDLE hFile = _Py_CreateFile2(path, FILE_READ_ATTRIBUTES, 0,
OPEN_EXISTING, NULL, traverse, &cfi);
if (hFile == INVALID_HANDLE_VALUE) {
// Either the path doesn't exist, or the caller lacks access.
error = GetLastError();
switch (error) {
case ERROR_INVALID_PARAMETER:
// The "con" DOS device requires read or write access.
hFile = _Py_CreateFile2(path, GENERIC_READ, FILE_SHARE_READ |
FILE_SHARE_WRITE | F
[issue46550] __slots__ updates despite being read-only
Eryk Sun added the comment:
Please read about augmented assignment [1]. In the REPL, use help("+=").
An augmented assignment evaluates the target (which, unlike normal
assignment statements, cannot be an unpacking) and the expression
list, performs the binary operation specific to the type of
assignment on the two operands, and assigns the result to the
original target. The target is only evaluated once.
An augmented assignment expression like x += 1 can be rewritten as
x = x + 1 to achieve a similar, but not exactly equal effect. In
the augmented version, x is only evaluated once. Also, when
possible, the actual operation is performed in-place, meaning that
rather than creating a new object and assigning that to the target,
the old object is modified instead.
Unlike normal assignments, augmented assignments evaluate the left-
hand side before evaluating the right-hand side. For example,
a[i] += f(x) first looks-up a[i], then it evaluates f(x) and
performs the addition, and lastly, it writes the result back to
a[i].
---
[1]
https://docs.python.org/3/reference/simple_stmts.html#augmented-assignment-statements
--
nosy: +eryksun
___
Python tracker
<https://bugs.python.org/issue46550>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46454] '0 -> /dev/null' is lost
Change by Eryk Sun : -- Removed message: https://bugs.python.org/msg411776 ___ Python tracker <https://bugs.python.org/issue46454> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46454] '0 -> /dev/null' is lost
Change by Eryk Sun : -- Removed message: https://bugs.python.org/msg411287 ___ Python tracker <https://bugs.python.org/issue46454> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46550] __slots__ updates despite being read-only
Eryk Sun added the comment:
If the target object of an augmented assignment doesn't support the in-place
binary operation, the normal binary operation is used instead. Thus an
augmented assignment is implemented to always assign the result back to the
target. For an attribute, that's similar to `x.a += 1` -> `x.a = x.a + 1`. For
example:
>>> dis.dis('x.a += 1')
0 RESUME 0
1 2 LOAD_NAME0 (x)
4 DUP_TOP
6 LOAD_ATTR1 (a)
8 LOAD_CONST 0 (1)
10 BINARY_OP 13 (+=)
12 ROT_TWO
14 STORE_ATTR 1 (a)
16 LOAD_CONST 1 (None)
18 RETURN_VALUE
Note the STORE_ATTR instruction in the above bytecode.
As to __slots__, I think that class construction should store it as a tuple,
unless maybe I'm overlooking some use case.
--
___
Python tracker
<https://bugs.python.org/issue46550>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46572] Unicode identifiers not necessarily unique
Eryk Sun added the comment:
Please read "Identifiers and keywords" [1] in the documentation. For example:
>>> import unicodedata as ud
>>> ud.normalize('NFKC', '𝖇𝖆𝖗') == 'bar'
True
>>> c = '\N{CYRILLIC SMALL LETTER A}'
>>> ud.name(ud.normalize('NFKC', c))
'CYRILLIC SMALL LETTER A'
---
[1]
https://docs.python.org/3/reference/lexical_analysis.html?highlight=nfkc#identifiers
--
nosy: +eryksun
resolution: -> not a bug
stage: -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.org/issue46572>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29688] Add support for Path.absolute()
Eryk Sun added the comment:
In Windows, paths that are relative to the current directory on a drive aren't
resolved. The following should be resolved by the current code:
>>> os.chdir('C:/Temp')
>>> pathlib.Path('C:').absolute()
WindowsPath('C:')
But _from_parts() has bugs with drive-relative paths.
Assuming the bugs are fixed, when a path has a drive, Path.absolute() should
resolve against abspath(self.drive) instead of getcwd().
--
___
Python tracker
<https://bugs.python.org/issue29688>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44264] Add descriptive error message when environment variable not detected
Eryk Sun added the comment: This is just a point of clarification. > my del did not change the environment variable os.environ defines the __delitem__ method to call C unsetenv(). Thus `del os.environ[varname]` does unset the environment variable, at least at the level of the C runtime. -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue44264> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46582] Windows builds fail: fatal error RC1116: RC terminating after preprocessor errors
Eryk Sun added the comment: PR 29396 and PR 29501 didn't get backported to 3.8 in November. Probably that was intentional since 3.8 only gets security fixes since 3.8.10, which was released in May. Supporting a newer build environment isn't a security issue. A core developer could make an exception for this case, so I'll leave this open for now. -- components: +Windows versions: -Python 3.10, Python 3.11, Python 3.9 ___ Python tracker <https://bugs.python.org/issue46582> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46585] Should we re-export `PyObj_FromPtr` in `ctypes`?
Eryk Sun added the comment: Alternatively, one can cast the address to py_object and dereference its `value`. For example: >>> obj = bytearray(b'spam') >>> sys.getrefcount(obj) 2 >>> obj2 = ctypes.cast(id(obj), ctypes.py_object).value >>> obj2 is obj True >>> sys.getrefcount(obj) 3 >>> del obj2 >>> sys.getrefcount(obj) 2 -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46585> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46578] cant DEBUG os.spawnv()
Eryk Sun added the comment: > cmd310 = ["/LIBPATH:D:\\python310\\lib\\site-packages\\torch\\lib", The `argv` parameter of os.spawnv() should begin with a command name to ensure that the application parses its command-line correctly. This doesn't necessarily have to be a file path since the `path` argument is what gets executed. For example, in this case I would use `cmd310 = ["link", ...]`. That said, for various reasons, including the correct quoting of command-line arguments that contain spaces, I recommend that you use the subprocess module instead of os.spawnv(). For example, use `p = subprocess.run(cmd310, executable=executable)`. The first item of the argument list still needs to be "link" in this case. Or include the full path of "link.exe" at the start of cmd310, and use `p = subprocess.run(cmd310)`. > But i cant DEBUG os.spawnv() on Pycharm. In Windows, os.spawnv() is a builtin function from the nt extension module, defined in Modules/posixmodule.c. It is not defined in os.py. The C implementation is a minimal wrapper around the C runtime's _wspawnv() function [1]. Probably PyCharm isn't capable of debugging a builtin function. --- [1] https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/spawnv-wspawnv -- components: +Windows -Build nosy: +eryksun, paul.moore, steve.dower, tim.golden, zach.ware ___ Python tracker <https://bugs.python.org/issue46578> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46555] Unicode-mangled names refer inconsistently to constants
Eryk Sun added the comment: Why was it decided to not raise a syntax error when the NFKC normalization of a non-ASCII token matches a keyword? I don't see a use for cases such as `𝕚𝕗 = 1` and `𝕚𝕗 + 1`. It seems the cost in terms of confusion far outweighs any potential benefit. -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46555> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29688] Add support for Path.absolute()
Eryk Sun added the comment: > I'm not seeing what's wrong with your example. "C:" or "C:spam\\eggs" are not absolute paths. They depend on the effective working directory on the drive. An absolute path should never depend on a working directory, which can change at random. WinAPI SetEnvironmentVariableW() allows applications to set environment variables with names that begin with "=". These names are effectively reserved for special use by the OS, at least as documented. In particular, names of the form "=X:", where "X" is a drive letter, are used to store the working directory on a drive. The C runtime _[w]chdir() function sets these per-drive environment variables, as does Python's os.chdir(). As environment variables, they can be inherited by child processes. When then Windows API resolves a file path to access a file, or in GetFullPathNameW(), a drive-relative path such as "X:" or "X:spam\\eggs" is resolved against either the current working directory (if it's on the drive) or the value of the "=X:" environment variable for the drive. If the latter isn't defined, it defaults to the root directory, e.g. "X:\\". If the current working directory is on the drive, the system updates the value of the "=X:" environment variable, if it exists. > on Windows you have to resolve the drive separately from the > working directory and then concatenate them? Yes, if self.drive is defined. This would be handled by os.path.abspath(self.drive), which calls WinAPI GetFullPathNameW(). -- ___ Python tracker <https://bugs.python.org/issue29688> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29688] Add support for Path.absolute()
Change by Eryk Sun : -- Removed message: https://bugs.python.org/msg412220 ___ Python tracker <https://bugs.python.org/issue29688> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29688] Add support for Path.absolute()
Eryk Sun added the comment: > I'm not seeing what's wrong with your example. "C:" or "C:spam\\eggs" are not absolute paths. They depend on the effective working directory on the drive. An absolute path should never depend on a working directory, which can change at random. WinAPI SetEnvironmentVariableW() allows applications to set environment variables with names that begin with "=". These names are effectively reserved for special use by the OS, at least as documented. In particular, names of the form "=X:", where "X" is a drive letter, are used to store the working directory on a drive. The C runtime _[w]chdir() function sets these per-drive environment variables, as does Python's os.chdir(). As environment variables, they can be inherited by child processes. When then Windows API resolves a file path to access a file, or in GetFullPathNameW(), a drive-relative path such as "X:" or "X:spam\\eggs" is resolved against either the current working directory (if it's on the drive) or the value of the "=X:" environment variable for the drive. If the latter isn't defined, it defaults to the root directory, e.g. "X:\\". If the current working directory is on the drive, the system updates the value of the "=X:" environment variable, if it exists. > on Windows you have to resolve the drive separately from the > working directory and then concatenate them? No, if self.drive is defined, then abspath(self.drive) should be called instead of getcwd(). In Windows, ntpath.abspath() calls WinAPI GetFullPathNameW(), which resolves the working directory on the drive. -- ___ Python tracker <https://bugs.python.org/issue29688> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29688] Add support for Path.absolute()
Eryk Sun added the comment:
> I'd imagine that bug is reproducible with `Path('C:\\Temp', 'C:')`
> already, right? If that's the case, should it logged as a
> separate issue?
Yes, it's a separate issue that affects the _from_parts() call in absolute().
How about designing absolute() to create a new instance from an absolute path
that's created by os.path? For example:
join(abspath(self.drive) if self.drive else getcwd(), self)
Of course use accessor functions.
--
___
Python tracker
<https://bugs.python.org/issue29688>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46594] Windows "Edit with IDLE >" only has one selection
Eryk Sun added the comment: Check your settings in the registry. In "HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\FileExts" there should be a ".py" key, but not necessarily. It should have an "OpenWithList" subkey that contains an "MRUList" value (most recently used list). The value should be a sequence of letters, each of which should be a value name in the key. If the launcher was last used to open a ".py" file, the first letter in the list should be a value with the data "py.exe". Ideally there should also be a subkey named "UserChoice" that contains a "ProgId" (programmatic identifier) value with the data "Python.File". This sets the "Python.File" ProgID as the locked-in user choice for ".py" files. In the GUI, you can set this in the open-with dialog by selecting "always use this app to open .py files". The selected app should be "Python", with an icon that contains the Python logo and a rocket (the launcher). If ".py" isn't the locked-in user choice, the shell API will use the most recent user selection in the open-with menu. If there's no user selection, the default association is calculated from "HKCR\.py", which is a merged view of "[HKCU|HKLM]\Software\Classes\.py". The default value of "HKCR\.py" sets the default file association. Ideally it should be "Python.File". "HKCR\Python.File" is a merged view of "[HKCU|HKLM]\Software\Classes\Python.File". For the merged view, if a value name is defined in the same subkey of HKCU and HKLM, the view prefers the HKCU value. There should be a subkey named "shell\editwithidle\shell". It should define one or more subkeys named "edit3*", such as "edit310". Each should contain a "MUIVerb" value that sets the command description in the "open-with" menu. There should also be a "command" subkey that contains the template command as its default value, e.g. ""C:\Program Files\Python310\pythonw.exe" -m idlelib "%L" %*". -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46594> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46594] Windows "Edit with IDLE >" only has one selection
Eryk Sun added the comment:
> the Open With entries may not be being merged.
That would probably be a bug in the Windows shell API. The HKCU and HKLM
subkeys of "Software\Classes\Python.File\Shell\editwithidle\shell" are merged
in the HKCR view. The same key path can exist in both hives, for which the view
contains the union, with precedence for HKCU.
For example, with "Software\Classes\spam":
import winreg
hkm = winreg.CreateKey(winreg.HKEY_LOCAL_MACHINE, r'Software\Classes\spam')
winreg.SetValueEx(hkm, 'eggs', 0, winreg.REG_SZ, 'hklm')
winreg.SetValueEx(hkm, 'baz', 0, winreg.REG_SZ, 'hklm')
hku = winreg.CreateKey(winreg.HKEY_CURRENT_USER, r'Software\Classes\spam')
winreg.SetValueEx(hku, 'eggs', 0, winreg.REG_SZ, 'hkcu')
winreg.SetValueEx(hku, 'bam', 0, winreg.REG_SZ, 'hkcu')
hkr = winreg.OpenKey(winreg.HKEY_CLASSES_ROOT, 'spam')
>>> winreg.EnumValue(hkr, 0)
('bam', 'hkcu', 1)
>>> winreg.EnumValue(hkr, 1)
('baz', 'hklm', 1)
>>> winreg.EnumValue(hkr, 2)
('eggs', 'hkcu', 1)
>>> winreg.EnumValue(hkr, 3)
Traceback (most recent call last):
File "", line 1, in
OSError: [WinError 259] No more data is available
--
___
Python tracker
<https://bugs.python.org/issue46594>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46490] Add "follow_symlinks=False" support for "os.utime()" on Windows
Eryk Sun added the comment:
In case you missed it, I implemented _Py_CreateFile2() in bpo-46506 and rewrote
os.stat() based on it. Check it out in case you're interested in moving forward
with a PR in bpo-46506.
For this issue, follow_symlinks is fairly simple to support with
_Py_CreateFile2(). We may as well add fd support, since that's trivial to add.
For example:
if (path->fd != -1) {
hFile = _Py_get_osfhandle(path->fd);
} else {
Py_BEGIN_ALLOW_THREADS
hFile = _Py_CreateFile2(path->wide, FILE_WRITE_ATTRIBUTES, 0,
OPEN_EXISTING, NULL, follow_symlinks, NULL);
Py_END_ALLOW_THREADS
}
if (hFile == INVALID_HANDLE_VALUE) {
if (path->fd == -1) {
path_error(path);
}
return NULL;
}
One also has to define the following macros to declare follow_symlinks and fd
support: UTIME_HAVE_NOFOLLOW_SYMLINKS and PATH_UTIME_HAVE_FD.
To announce support in os.supports_follow_symlinks and os.supports_fd, it
should be conditioned on MS_WINDOWS, i.e. _add("MS_WINDOWS", "utime"). The os
module is frozen, so changing these two sets requires rebuilding Python.
--
___
Python tracker
<https://bugs.python.org/issue46490>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46631] Implement a "strict" mode for getpass.getuser()
New submission from Eryk Sun :
getpass.getuser() checks the environment variables LOGNAME (login name), USER,
LNAME, and USERNAME, in that order. In Windows, LOGNAME, USER, and LNAME have
no conventional usage. I think there should be a strict mode that restricts
getuser() to check only USERNAME in Windows and only LOGNAME in POSIX [1]. If
the login variable isn't defined, it should fall back on using the system API,
based on the user ID in POSIX and the logon ID in Windows.
For the fallback in Windows, the _winapi module could implement
GetCurrentProcessToken(), GetTokenInformation(), and LsaGetLogonSessionData().
For TokenStatistics, return a dict with just "AuthenticationId". For
LsaGetLogonSessionData(), return a dict with just "UserName".
GetCurrentProcessToken() returns a pseudohandle (-4), which should not be
closed.
For example, assuming _winapi wraps the required functions:
def getuser(strict=False):
"""Get the username from the environment or password database.
First try various environment variables. If strict, check only LOGNAME
in POSIX and only USERNAME in Windows. As a fallback, in POSIX get the
user name from the password database, and in Windows get the user name
from the logon-session data of the current process.
"""
posix = sys.platform != 'win32'
if strict:
names = ('LOGNAME',) if posix else ('USERNAME',)
else:
names = ('LOGNAME', 'USER', 'LNAME', 'USERNAME')
for name in names:
if user := os.environ.get(name):
return user
if posix:
import pwd
return pwd.getpwuid(os.getuid())[0]
import _winapi
logon_id = _winapi.GetTokenInformation(
_winapi.GetCurrentProcessToken(),
_winapi.TokenStatistics)['AuthenticationId']
return _winapi.LsaGetLogonSessionData(logon_id)['UserName']
Like WinAPI GetUserNameW(), the above fallback returns the logon user name
instead of the account name of the token user. As far as I know, the user name
and the account name only differ for the builtin service account logons
"SYSTEM" (999) and "NETWORK SERVICE" (996), for which the user name is the
machine security principal (i.e. the machine's NETBIOS name plus "$"). The user
name of the builtin "LOCAL SERVICE" logon (997), on the other hand, is just the
"LOCAL SERVICE" account name, since this account lacks network access.
Unlike GetUserNameW(), the above code uses the process token instead of the
effective token. This is like POSIX getuid(), whereas what GetUserNameW() does
is like geteuid(). getuser() could implement an `effective` option to return
the effective user name. In Windows this would switch to calling
GetCurrentThreadEffectiveToken() instead of GetCurrentProcessToken().
---
[1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html
--
components: Library (Lib), Windows
messages: 412495
nosy: eryksun, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
stage: needs patch
status: open
title: Implement a "strict" mode for getpass.getuser()
type: enhancement
versions: Python 3.11
___
Python tracker
<https://bugs.python.org/issue46631>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46631] Implement a "strict" mode for getpass.getuser()
Eryk Sun added the comment:
Here's an example for the suggested changes to _winapi.
Include these headers:
#include // LsaGetLogonSessionData
#include // STATUS_SUCCESS
Add these argument-clinic macros to _winapi_functions:
_WINAPI_GETCURRENTPROCESSTOKEN_METHODDEF
_WINAPI_GETTOKENINFORMATION_METHODDEF
_WINAPI_LSAGETLOGONSESSIONDATA_METHODDEF
Add TokenStatistics in winapi_exec():
WINAPI_CONSTANT(F_DWORD, TokenStatistics);
Add minimal implementations that wrap the WinAPI functions:
/*[clinic input]
_winapi.GetCurrentProcessToken -> HANDLE
Return a handle for the access token of the current process.
[clinic start generated code]*/
static HANDLE
_winapi_GetCurrentProcessToken_impl(PyObject *module)
/*[clinic end generated code: output=cf8e8e20dd41dd6e
input=73a282cf3718af9e]*/
{
return GetCurrentProcessToken();
}
/*[clinic input]
_winapi.GetTokenInformation
handle: HANDLE
information_class: unsigned_long
/
Get information from an access token.
[clinic start generated code]*/
static PyObject *
_winapi_GetTokenInformation_impl(PyObject *module, HANDLE handle,
unsigned long information_class)
/*[clinic end generated code: output=caecec0a25658348
input=b277ad2414f1b03e]*/
{
if (information_class != TokenStatistics) {
return PyErr_Format(
PyExc_NotImplementedError,
"Unsupported information class: %d",
information_class);
}
DWORD returned_size;
TOKEN_STATISTICS info;
if (!GetTokenInformation(handle, information_class, &info, sizeof(info),
&returned_size)) {
return PyErr_SetFromWindowsErr(0);
}
PyObject *result = PyDict_New();
if (!result) {
return NULL;
}
PyObject *value = PyLong_FromUnsignedLongLong(
(((uint64_t)info.AuthenticationId.HighPart) << 32) +
info.AuthenticationId.LowPart);
if (!value) {
goto error;
}
if (PyDict_SetItemString(result, "AuthenticationId", value) < 0) {
Py_DECREF(value);
goto error;
}
Py_DECREF(value);
return result;
error:
Py_CLEAR(result);
return NULL;
}
/*[clinic input]
_winapi.LsaGetLogonSessionData
logon_id: unsigned_long_long
/
Get data for the logon session identified by logon_id.
[clinic start generated code]*/
static PyObject *
_winapi_LsaGetLogonSessionData_impl(PyObject *module,
unsigned long long logon_id)
/*[clinic end generated code: output=680ac7725ef34527
input=01ff4216b89d01ef]*/
{
SECURITY_LOGON_SESSION_DATA *pdata;
LUID logon_luid;
logon_luid.HighPart = logon_id >> 32;
logon_luid.LowPart = logon_id & 0x;
NTSTATUS status = LsaGetLogonSessionData(&logon_luid, &pdata);
if (status != STATUS_SUCCESS) {
return PyErr_SetFromWindowsErr(LsaNtStatusToWinError(status));
}
PyObject *result = PyDict_New();
if (!result) {
goto error;
}
PyObject *value = PyUnicode_FromWideChar(pdata->UserName.Buffer,
pdata->UserName.Length / sizeof(WCHAR));
if (!value) {
goto error;
}
if (PyDict_SetItemString(result, "UserName", value) < 0) {
Py_DECREF(value);
goto error;
}
Py_DECREF(value);
LsaFreeReturnBuffer(pdata);
return result;
error:
LsaFreeReturnBuffer(pdata);
Py_CLEAR(result);
return NULL;
}
--
___
Python tracker
<https://bugs.python.org/issue46631>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46654] file_open doesn't handle UNC paths produced by pathlib's resolve() (but can handle UNC paths with additional slashes)
Eryk Sun added the comment:
Builtin open() calls C open(). This C function supports whatever path types are
supported natively. In Windows, C open() calls WinAPI CreateFileW(), which does
not support "file://" URIs. The Windows API handles it as a relative path,
which gets resolved against the current working directory. For example:
>>> os.getcwd()
'C:\\Temp'
>>> nt._getfullpathname('file:host/share/file')
'C:\\Temp\\file:\\host\\share\\file'
>>> nt._getfullpathname('file://host/share/file')
'C:\\Temp\\file:\\host\\share\\file'
As to the resolved path somehow working, that generally will not be the case.
Most filesystems in Windows will reject a path component named "file:" as an
invalid name. The ":" character is usually disallowed in base file and
directory names, since some Windows filesystems use it as a delimiter in file
streams, e.g. "name:stream_name:stream_type". The default data stream in a
regular file has no stream name, and its stream type is "$DATA". Thus for base
name "file", the default data stream can be referenced explicitly as
"file::$DATA". But just "file:", with neither a stream name nor a stream type,
is an invalid name.
--
nosy: +eryksun
___
Python tracker
<https://bugs.python.org/issue46654>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46654] file_open doesn't handle UNC paths produced by pathlib's resolve() (but can handle UNC paths with additional slashes)
Change by Eryk Sun : -- Removed message: https://bugs.python.org/msg412604 ___ Python tracker <https://bugs.python.org/issue46654> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46654] urllib.request.urlopen doesn't handle UNC paths produced by pathlib's resolve() (but can handle UNC paths with additional slashes)
Eryk Sun added the comment: In FileHandler.file_open(), req.host is the host name, which is either None or an empty string for a local drive path such as, respectively, "file:/Z:/test.py" or "file:///Z:/test.py". The value of req.selector never starts with "//", for which file_open() checks, but rather a single slash, such as "/Z:/test.py" or "/share/test.py". This is a bug in file_open(). Due to this bug, it always calls self.open_local_file(req), even if req.host isn't local. The distinction shouldn't matter in Windows, which supports UNC paths, but POSIX has to open a path on the local machine (possibly a mount point for a remote path, but that's irrelevant). In POSIX, if the local machine coincidentally has the req.selector path, then the os.stat() and open() calls will succeed with a bogus result. For "file://host/share/test.py", req.selector is "/share/test.py". In Windows, url2pathname() converts this to r"\share\test.py", which is relative to the drive of the process current working directory. This is a bug in open_local_file() on Windows. For it to work correctly, req.host has to be joined back with req.selector as the UNC path "//host/share/test.py". Of course, this need not be a local file in Windows, so Windows should be exempted from the local file limitation in file_open(). -- ___ Python tracker <https://bugs.python.org/issue46654> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46654] urllib.request.urlopen doesn't handle UNC paths produced by pathlib's resolve() (but can handle UNC paths with additional slashes)
Eryk Sun added the comment: > file://server/host/file.ext on windows, even though > file:server/host/file.ext open just fine. For r"\\host\share\test.py", the two slash conversion "file://host/share/test.py" is correct according to RFC80889 "E.3.1. URI with Authority" [1]. In this case, req.host is "host", and req.selector is "/share/test.py". The four slash version "file:host/share/test.py" is a known variant for a converted UNC path, as noted in RFC8089 "E.3.2. URI with UNC Path". In this case, req.host is an empty string, and req.selector is "//host/share/test.py". There's another variant that uses 5 slashes for a UNC path, but urllib (or url2pathname) doesn't support it. --- [1] https://datatracker.ietf.org/doc/html/rfc8089 -- ___ Python tracker <https://bugs.python.org/issue46654> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46654] urllib.request.urlopen doesn't handle UNC paths produced by pathlib's as_uri() (but can handle UNC paths with additional slashes)
Eryk Sun added the comment:
> The value of req.selector never starts with "//", for which file_open()
> checks, but rather a single slash, such as "/Z:/test.py" or
> "/share/test.py".
To correct myself, actually req.selector will start with "//" for a "file:"
URI, such as "file:host/share/test.py". For this example, req.host is an
empty string, so file_open() still ends up calling open_local_file(), which
will open "//host/share/test.py". In Linux, "//host/share" is the same as
"/host/share". In Cygwin and MSYS2 it's a UNC path. I guess this case should be
allowed, even though the meaning of a "//" root isn't specifically defined in
POSIX.
Unless I'm overlooking something, file_open() only has to check the value of
req.host. In POSIX, it should require opening a 'local' path, i.e. if req.host
isn't None, empty, or a local host, raise URLError.
In Windows, my tests show that the shell API special cases "localhost" (case
insensitive) in "file:" URIs. For example, the following are all equivalent:
"file:/C:/Temp", "file:///C:/Temp", and "file://localhost/C:/Temp". The shell
API does not special case the real local host name or any of its IP addresses,
such as 127.0.0.1. They're all handled as UNC paths.
Here's what I've experimented with thus far, which passes the existing urllib
tests in Linux and Windows:
class FileHandler(BaseHandler):
def file_open(self, req):
if not self._is_local_path(req):
if sys.platform == 'win32':
path = url2pathname(f'//{req.host}{req.selector}')
else:
raise URLError("In POSIX, the file:// scheme is only "
"supported for local file paths.")
else:
path = url2pathname(req.selector)
return self._common_open_file(req, path)
def _is_local_path(self, req):
if req.host:
host, port = _splitport(req.host)
if port:
raise URLError(f"the host cannot have a port: {req.host}")
if host.lower() != 'localhost':
# In Windows, all other host names are UNC.
if sys.platform == 'win32':
return False
# In POSIX, support all names for the local host.
if _safe_gethostbyname(host) not in self.get_names():
return False
return True
# names for the localhost
names = None
def get_names(self):
if FileHandler.names is None:
try:
FileHandler.names = tuple(
socket.gethostbyname_ex('localhost')[2] +
socket.gethostbyname_ex(socket.gethostname())[2])
except socket.gaierror:
FileHandler.names = (socket.gethostbyname('localhost'),)
return FileHandler.names
def open_local_file(self, req):
if not self._is_local_path(req):
raise URLError('file not on local host')
return self._common_open_file(req, url2pathname(req.selector))
def _common_open_file(self, req, path):
import email.utils
import mimetypes
host = req.host
filename = req.selector
try:
if host:
origurl = f'file://{host}{filename}'
else:
origurl = f'file://{filename}'
stats = os.stat(path)
size = stats.st_size
modified = email.utils.formatdate(stats.st_mtime, usegmt=True)
mtype = mimetypes.guess_type(filename)[0] or 'text/plain'
headers = email.message_from_string(
f'Content-type: {mtype}\n'
f'Content-length: {size}\n'
f'Last-modified: {modified}\n')
return addinfourl(open(path, 'rb'), headers, origurl)
except OSError as exp:
raise URLError(exp)
Unfortunately nturl2path.url2pathname() parses some UNC paths incorrectly. For
example, the following path should be an invalid UNC path, since "C:" is an
invalid name, but instead it gets converted into an unrelated local path.
>>> nturl2path.url2pathname('//host/C:/Temp/spam.txt')
'C:\\Temp\\spam.txt'
This goof depends on finding ":" or "|" in the path. It's arguably worse if the
last co
[issue46654] urllib.request.urlopen doesn't handle UNC paths produced by pathlib's as_uri() (but can handle UNC paths with additional slashes)
Change by Eryk Sun : -- assignee: docs@python -> components: -2to3 (2.x to 3.x conversion tool), Argument Clinic, Build, C API, Cross-Build, Demos and Tools, Distutils, Documentation, Extension Modules, FreeBSD, IDLE, IO, Installation, Interpreter Core, Parser, Regular Expressions, SSL, Subinterpreters, Tests, Tkinter, Unicode, Windows, XML, asyncio, ctypes, email, macOS stage: -> needs patch type: performance -> behavior versions: -Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue46654> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46668] encodings: the "mbcs" alias doesn't work
Eryk Sun added the comment: > The Python 3.6 and 3.7 "codecs.register(_alias_mbcs)" doesn't work > because "search_function()" is tested before and it works for "cpXXX" > encodings. Isn't the 3.6-3.10 ordering of search_function() and _alias_mbcs() correct as a fallback? In this case, Python doesn't support a cross-platform encoding for the code page. That's why the old implementation of test_mbcs_alias() mocked _winapi.GetACP() to return 123 and then checked that looking up 'cp123' returned the "mbcs" codec. I'd actually prefer to extend this by implementing _winapi.GetOEMCP() and using "oem" as a fallback for that case. -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46668> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46668] encodings: the "mbcs" alias doesn't work
Eryk Sun added the comment:
> I don't think that this fallback is needed anymore. Which Windows
> code page can be used as ANSI code page which is not already
> implemented as a Python codec?
Python has full coverage of the ANSI and OEM code pages in the standard Windows
locales, but I don't have any experience with custom (i.e. supplemental or
replacement) locales.
https://docs.microsoft.com/en-us/windows/win32/intl/custom-locales
Here's a simple script to check the standard locales.
import codecs
import ctypes
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
LOCALE_ALL = 0
LOCALE_WINDOWS = 1
LOCALE_IDEFAULTANSICODEPAGE = 0x1004
LOCALE_IDEFAULTCODEPAGE = 0x000B # OEM
EnumSystemLocalesEx = kernel32.EnumSystemLocalesEx
GetLocaleInfoEx = kernel32.GetLocaleInfoEx
GetCPInfoExW = kernel32.GetCPInfoExW
EnumLocalesProcEx = ctypes.WINFUNCTYPE(ctypes.c_int,
ctypes.c_wchar_p, ctypes.c_ulong, ctypes.c_void_p)
class CPINFOEXW(ctypes.Structure):
_fields_ = (('MaxCharSize', ctypes.c_uint),
('DefaultChar', ctypes.c_ubyte * 2),
('LeadByte', ctypes.c_ubyte * 12),
('UnicodeDefaultChar', ctypes.c_wchar),
('CodePage', ctypes.c_uint),
('CodePageName', ctypes.c_wchar * 260))
def get_all_locale_code_pages():
result = []
seen = set()
info = (ctypes.c_wchar * 100)()
@EnumLocalesProcEx
def callback(locale, flags, param):
for lctype in (LOCALE_IDEFAULTANSICODEPAGE,
LOCALE_IDEFAULTCODEPAGE):
if (GetLocaleInfoEx(locale, lctype, info, len(info)) and
info.value not in ('0', '1')):
cp = int(info.value)
if cp in seen:
continue
seen.add(cp)
cp_info = CPINFOEXW()
if not GetCPInfoExW(cp, 0, ctypes.byref(cp_info)):
cp_info.CodePage = cp
cp_info.CodePageName = str(cp)
result.append(cp_info)
return True
if not EnumSystemLocalesEx(callback, LOCALE_WINDOWS, None, None):
raise ctypes.WinError(ctypes.get_last_error())
result.sort(key=lambda x: x.CodePage)
return result
supported = []
unsupported = []
for cp_info in get_all_locale_code_pages():
cp = cp_info.CodePage
try:
codecs.lookup(f'cp{cp}')
except LookupError:
unsupported.append(cp_info)
else:
supported.append(cp_info)
if unsupported:
print('Unsupported:\n')
for cp_info in unsupported:
print(cp_info.CodePageName)
print('\nSupported:\n')
else:
print('All Supported:\n')
for cp_info in supported:
print(cp_info.CodePageName)
Output:
All Supported:
437 (OEM - United States)
720 (Arabic - Transparent ASMO)
737 (OEM - Greek 437G)
775 (OEM - Baltic)
850 (OEM - Multilingual Latin I)
852 (OEM - Latin II)
855 (OEM - Cyrillic)
857 (OEM - Turkish)
862 (OEM - Hebrew)
866 (OEM - Russian)
874 (ANSI/OEM - Thai)
932 (ANSI/OEM - Japanese Shift-JIS)
936 (ANSI/OEM - Simplified Chinese GBK)
949 (ANSI/OEM - Korean)
950 (ANSI/OEM - Traditional Chinese Big5)
1250 (ANSI - Central Europe)
1251 (ANSI - Cyrillic)
1252 (ANSI - Latin I)
1253 (ANSI - Greek)
1254 (ANSI - Turkish)
1255 (ANSI - Hebrew)
1256 (ANSI - Arabic)
1257 (ANSI - Baltic)
1258 (ANSI/OEM - Viet Nam)
Some locales are Unicode only (e.g. Hindi-India) or have no OEM code page,
which the above code skips by checking for "0" or "1" as the code page value.
Windows 10+ allows setting the system locale to a Unicode-only locale, for
which it uses UTF-8 (65001) for ANSI and OEM.
The OEM code page matters because the console input and output code pages
default to OEM, e.g. for os.device_encoding(). The console's I/O code pages are
used in Python by low-level os.read() and os.write(). Note that the console
doesn't properly implement using UTF-8 (65001) as the input code page. In this
case, input read from the console via ReadFile() or ReadConsoleA() has a null
byte in place of each non-ASCII character.
--
___
Python tracker
<https://bugs.python.org/issue46668>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46659] Deprecate locale.getdefaultlocale() function
Eryk Sun added the comment:
> getdefaultlocale() falls back to LANG and LANGUAGE.
_Py_SetLocaleFromEnv(LC_CTYPE) (e.g. setlocale(LC_CTYPE, "")) gets called at
startup, except for the isolated configuration [1].
I think calendar.Locale*Calendar should try the LC_CTYPE locale if LC_TIME is
"C", i.e. (None, None). Otherwise, it's introducing new default behavior. For
example, with LC_ALL set to "ru_RU.utf8":
3.8:
>>> locale.getlocale(locale.LC_TIME)
(None, None)
>>> locale.getlocale(locale.LC_CTYPE)
('ru_RU', 'UTF-8')
>>> cal = calendar.LocaleTextCalendar()
>>> cal.formatweekday(0, 15)
' Понедельник '
3.11.0a5+:
>>> locale.getlocale(locale.LC_TIME)
(None, None)
>>> locale.getlocale(locale.LC_CTYPE)
('ru_RU', 'UTF-8')
>>> cal = calendar.LocaleTextCalendar()
>>> cal.formatweekday(0, 15)
' Monday'
>>> locale.setlocale(locale.LC_TIME, '')
'ru_RU.utf8'
>>> cal = calendar.LocaleTextCalendar()
>>> cal.formatweekday(0, 15)
' Понедельник '
---
[1] https://docs.python.org/3/c-api/init_config.html?#isolated-configuration
--
nosy: +eryksun
___
Python tracker
<https://bugs.python.org/issue46659>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46659] Deprecate locale.getdefaultlocale() function
Eryk Sun added the comment: > Oh. Serhiy asked me to use LC_TIME rather than LC_CTYPE. Since Locale*Calendar is documented as not being thread safe, __init__() could get the real default via setlocale(LC_TIME, "") when locale=None and the current LC_TIME is "C". Restore it back to "C" after getting the default. That should usually match the behavior from previous versions that called getdefaultlocale(). In cases where it differs, it's fixing a bug because the default LC_TIME is the correct default. -- ___ Python tracker <https://bugs.python.org/issue46659> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46686] [venv / PC/launcher] issue with a space in the installed python path
Eryk Sun added the comment: run_child() expects `cmdline` to be correctly quoted, and normally it is. I can't reproduce this problem with Python 3.10.2. I created a user account with a space in the account name, logged on, and installed 3.10.2 for the current user, with the option enabled to add Python to PATH. Next I opened a command prompt in the user profile directory and created a virtual environment via `python.exe -m venv .venv`. Running ".venv\Scripts\python.exe -X utf8" worked. The command line used by the venv "python.exe" launcher was properly quoted and thus properly parsed in the original list of command-line arguments, sys.orig_argv. -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46686> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46686] [venv / PC/launcher] issue with a space in the installed python path
Eryk Sun added the comment: I checked the source code in PC/launcher.c process(). It turns out that `executable` is not getting quoted in the venv launcher case. CreateProcessW() tries to get around this. If the command isn't quoted, it has a loop that consumes up to a space (or tab) and checks for an existing file (not a directory). If it finds a file, it rewrites the command line to quote the path of the file. My test happened to work. But it's simple enough to create an example that fails. For example, as an elevated admin, create a file named "C:\Program". Now the venv launcher won't be able to execute a base interpreter that's installed in "C:\Program Files": C:\Temp>echo >C:\Program C:\Temp>"C:\Program Files\Python310\python.exe" -m venv env Error: Command '['C:\\Temp\\env\\Scripts\\python.exe', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 101. -- priority: normal -> critical stage: -> needs patch versions: +Python 3.11 ___ Python tracker <https://bugs.python.org/issue46686> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46686] [venv / PC/launcher] issue with a space in the installed python path
Eryk Sun added the comment:
The venv launcher can quote the executable path either always or only when it
contains spaces. The following is a suggestion for implementing the latter.
Before allocating `executable`, use memchr() (include ) to search the
UTF-8 source for a space. If found, increment the character count by two. After
allocating `executable`, add the initial quote character, and increment the
pointer.
BOOL add_quotes = FALSE;
if (memchr(start, ' ', (size_t)len) != NULL) {
add_quotes = TRUE;
cch += 2;
}
executable = (wchar_t *)malloc(cch * sizeof(wchar_t));
if (executable == NULL) {
error(RC_NO_MEMORY, L"A memory allocation failed");
}
if (add_quotes) {
*executable++ = L'\"';
}
Later, after checking the existence via GetFileAttributesW(), add the trailing
quote and null, and decrement the pointer:
if (GetFileAttributesW(executable) == INVALID_FILE_ATTRIBUTES) {
error(RC_NO_PYTHON, L"No Python at '%ls'", executable);
}
if (add_quotes) {
size_t n = wcslen(executable);
executable[n] = L'\"';
executable[n + 1] = L'\0';
executable--;
}
--
___
Python tracker
<https://bugs.python.org/issue46686>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46697] _ctypes_simple_instance returns inverted logic
Change by Eryk Sun : -- stage: -> patch review versions: -Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue46697> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46703] boolean operation issue (True == False == False)
Eryk Sun added the comment: > True == False == False is really True == False and False == False > wich is False and True which is False Moreover, since the left-hand comparison is `True == False`, which evaluates to False, the right-hand comparison doesn't even get evaluated. In the following example, print(3) doesn't get called because the left-hand comparison, `None != None`, is False: >>> print(1) != print(2) == print(3) 1 2 False -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46703> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46716] regrtest didn't respect the timeout when running test_subprocess on AMD64 Windows11 3.x
Eryk Sun added the comment: > test_call_timeout() or test_timeout() in test_subprocess.py. These tests don't override the standard files, and they only spawn a single child with no descendants. I don't see why this would hang. It shouldn't be a problem with leaked pipe handles (see bpo-43346). It probably will need to be diagnosed by attaching a debugger, or offline with a dump file. > process trees whereas terminating a parent automatically kills the children One can use a job object to manage a child process and all of its descendants, including resource usage and termination. A process can belong to multiple job objects in Windows 8+, which is required by Python 3.9+. For reliability, the child has to be created in a suspended state via CREATE_SUSPENDED. It can be resumed with ResumeThread() after adding it to the job with AssignProcessToJobObject(). You can try to terminate a job cleanly, which is similar in effect to sending SIGTERM to a process group in POSIX. In Windows, this has to be approached differently for console vs graphical processes. To handle console apps, assuming the child inherits the current console, spawn it as a new process group via creationflags=subprocess.CREATE_NEW_PROCESS_GROUP. You can request an exit by sending a Ctrl+Break event to the group via os.kill(p.pid, signal.CTRL_BREAK_EVENT) [1]. The request might be ignored, but typically the default handler is called, which calls ExitProcess(). To handle GUI apps, assuming the child inherits the current desktop (usually "WinSta0\Default"), first enumerate the top-level and message-only windows on the current desktop via EnumWindows() and FindWindowExW(). Use GetWindowThreadProcessId() to filter the list to include only windows that belong to the job. Post WM_CLOSE to each window in the job. A process might ignore a request to close. It could keep the window open or continue running in the background. After an internal timeout, you can call TerminateJobObject() to kill any process in the job that remains alive. This is a forced and abrupt termination, which is similar to sending SIGKILL to a process group in POSIX. --- [1] This usage of os.kill() is what we're stuck with. Rightfully, we should be using os.killpg(p.pid, signal.SIGBREAK) or os.kill(-p.pid, signal.SIGBREAK) (note the negative pid value). -- ___ Python tracker <https://bugs.python.org/issue46716> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46726] Thread spuriously marked dead after interrupting a join call
Eryk Sun added the comment: > The race on := is much smaller than the original race > and I suspect in practice will be very hard to hit. In Windows, the acquire() method of a lock can't be interrupted. Thus, in the main thread, an exception from Ctrl+C gets raised as soon as acquire() returns. This exception definitely will interrupt the assignment. Here's a workaround: global scope: _WINDOWS = _sys.platform == 'win32' in _wait_for_tstate_lock(): acquired = None try: if acquired := lock.acquire(block, timeout): lock.release() self._stop() except: if _WINDOWS and acquired is None: acquired = True if acquired: lock.release() self._stop() raise This doesn't help in POSIX if the STORE_FAST instruction that assigns `acquired` gets interrupted. This can't be distinguished from acquire() itself getting interrupted. But at least the window for this is as small as possible. -- nosy: +eryksun ___ Python tracker <https://bugs.python.org/issue46726> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46726] Thread spuriously marked dead after interrupting a join call
Eryk Sun added the comment: > If the acquire() in fact times out, but the store to the `acquired` > variable is interrupted, `if _WINDOWS and acquired is None` will > succeed, despite that the lock is still locked Yeah, my proposed workaround is no good, so we can't resolve this without a more fundamental solution. Are you looking into a way to prevent the STORE_FAST instruction from getting interrupted by an asynchronous exception? -- ___ Python tracker <https://bugs.python.org/issue46726> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46726] Thread spuriously marked dead after interrupting a join call
Eryk Sun added the comment:
> Wrap everything needed in a custom C function.
Maybe add an `acquire_and_release()` method:
static PyObject *
lock_PyThread_acquire_and_release_lock(
lockobject *self, PyObject *args, PyObject *kwds)
{
_PyTime_t timeout;
if (lock_acquire_parse_args(args, kwds, &timeout) < 0)
return NULL;
PyLockStatus r = acquire_timed(self->lock_lock, timeout);
if (r == PY_LOCK_INTR) {
return NULL;
}
if (r == PY_LOCK_ACQUIRED) {
PyThread_release_lock(self->lock_lock);
self->locked = 0;
}
return PyBool_FromLong(r == PY_LOCK_ACQUIRED);
}
--
___
Python tracker
<https://bugs.python.org/issue46726>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46716] regrtest didn't respect the timeout when running test_subprocess on AMD64 Windows11 3.x
Eryk Sun added the comment: > the fix should be as simple as coercing the timeout values to >= 0. Popen._remaining_time() should return max(endtime - _time(), 0). Popen._wait() should raise OverflowError if the timeout is too large for the implementation. In Windows, the upper limit in milliseconds is `_winapi.INFINITE - 1` (about 49.7 days). It's important to only allow the timeout in milliseconds to be _winapi.INFINITE when `timeout is None`. The DWORD converter in _winapi needs to subclass unsigned_long_converter. The current implementation based on the legacy format unit "k" is too lenient. Negative values and values that are too large should fail. I updated it to use the following definition: class DWORD_converter(unsigned_long_converter): type = 'DWORD' This produces the following improved results: >>> h = _winapi.GetCurrentProcess() >>> _winapi.WaitForSingleObject(h, _winapi.INFINITE + 1) Traceback (most recent call last): File "", line 1, in OverflowError: Python int too large to convert to C unsigned long >>> _winapi.WaitForSingleObject(h, -1) Traceback (most recent call last): File "", line 1, in ValueError: value must be positive -- ___ Python tracker <https://bugs.python.org/issue46716> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46726] Thread spuriously marked dead after interrupting a join call
Eryk Sun added the comment: > is there a bulletproof way to guarantee that `self._stop()` gets > called if the acquire_and_release() succeeds? I don't think it's critical. But we still should deal with the common case in Windows in which a KeyboardInterrupt is raised immediately after the method returns. For example: try: if lock.acquire_and_release(block, timeout): self._stop except: if not lock.locked(): self._stop() The proposed acquire_and_release() method can also be used in threading._shutdown(), when it iterates _shutdown_locks to join non-daemon threads. -- ___ Python tracker <https://bugs.python.org/issue46726> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46726] Thread spuriously marked dead after interrupting a join call
Eryk Sun added the comment: > Anything at the Python level that cares whether the thread is > still alive will call _wait_for_tstate_lock() again It's nice that _maintain_shutdown_locks() gets called in _stop(), but the more important call site is in _set_tstate_lock(). I typed up the example initially with try/finally. Then I changed it to avoid the extra lock.locked() call when there's no exception, but I forgot to add a `raise` statement: try: if lock.acquire_and_release(block, timeout): self._stop except: if not lock.locked(): self._stop() raise -- ___ Python tracker <https://bugs.python.org/issue46726> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46726] Thread spuriously marked dead after interrupting a join call
Eryk Sun added the comment: > I didn't have that in mind at all. I understood what you had in mind, and I don't disagree. I was just highlighting that the only somewhat important work done in _stop() is to clean up the _shutdown_locks set, to keep it from growing too large. But that task is already handled when creating a new thread, and it's arguably more important to handle it there. -- ___ Python tracker <https://bugs.python.org/issue46726> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46716] regrtest didn't respect the timeout when running test_subprocess on AMD64 Windows11 3.x
Eryk Sun added the comment:
> I fear the potential 3rd-party breakage alone should bump
> this to its own issue.
The change to the DWORD converter in _winapi should only be in 3.11+. If this
causes problems for other projects, they're probably depending on undefined
behavior in the standard library. No third-party package or application should
use _winapi directly.
> 1) modify Windows Popen._wait() to raise on out of bounds
> values [< 0 or >= INFINITE]
I think raising ValueError would be best at this level. For example:
if timeout is None:
timeout_millis = _winapi.INFINITE
else:
timeout_millis = int(timeout * 1000)
if timeout_millis >= _winapi.INFINITE:
raise ValueError("'timeout' exceeds the platform limit")
if timeout_millis < 0:
raise ValueError("'timeout' must be non-negative")
--
___
Python tracker
<https://bugs.python.org/issue46716>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46716] regrtest didn't respect the timeout when running test_subprocess on AMD64 Windows11 3.x
Eryk Sun added the comment: > 4) use Job objects to group Windows processes for termination I think a separate issue should be created for this enhancement. _winapi wrappers would be needed for CreateJobObjectW(), SetInformationJobObject(), AssignProcessToJobObject(), TerminatejobObject(), and ResumeThread(), plus the constant CREATE_SUSPENDED. I'd also prefer to make related changes to send_signal(), which would require GetConsoleProcessList() and GenerateConsoleCtrlEvent(). A new Popen option would be needed to configure whether the job allows descendants to break away via the process creation flag CREATE_BREAKAWAY_FROM_JOB. This should be allowed by default. --- send_signal(): SIGKILL, SIGTERM, SIBREAK, SIGINT Support Popen.kill(group=False) and Popen.terminate(group=False) on all platforms as Popen.send_signal(signal.SIGKILL, group=group) and Popen.send_signal(signal.SIGTERM, group=group). The Universal CRT (ucrt) in Windows doesn't define SIGKILL. Even when it's not defined by the platform, signal.SIGKILL should always be defined, preferably as 9 (unused by ucrt), else as an unused value in the range up to NSIG, else as NSIG + 1. The `group` keyword-only option broadens the scope to the process group or job. A process is a group leader if it was created with the flag CREATE_NEW_PROCESS_GROUP (save self._creationflags) and its process ID is in GetConsoleProcessList(). For SIGKILL, always use forced termination. For SIGTERM, use forced termination either if `group` is false or if `group` is true and the process is not a group leader. To force termination, call TerminateJobObject(self._job_handle, 1) if `group` is true, else TerminateProcess(self._handle, 1). For SIGTERM, SIGBREAK, and SIGINT, call GenerateConsoleCtrlEvent() if `group` is true and the process is a group leader. For SIGTERM and SIGBREAK, send CTRL_BREAK_EVENT. For SIGINT, send CTRL_C_EVENT. Behavior to deprecate: When `group` is false and `sig` is CTRL_C_EVENT (0) or CTRL_BREAK_EVENT (1), send_signal() always calls os.kill(). This legacy behavior tries to handle a process as if it's a process group, and it combines POSIX kill() with Windows API constants. subprocess should distance itself from the fundamental problems with os.kill() in Windows. -- ___ Python tracker <https://bugs.python.org/issue46716> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
