[Python-announce] python-build-standalone 3.11 distributions
python-build-standalone (https://github.com/indygreg/python-build-standalone) is a project that produces standalone, highly portable builds of CPython that are designed to run on as many machines as possible with no additional dependencies. I created the project for PyOxidizer (https://github.com/indygreg/PyOxidizer) but it is now being used for various other tools that want to easily "install" a working Python interpreter, such as Bazel's rules_python (https://github.com/bazelbuild/rules_python) and various applications embedding Python. Read more at https://gregoryszorc.com/docs/python-build-standalone/main/ I'm pleased to announce the latest 20230116 release (https://github.com/indygreg/python-build-standalone/releases/tag/20230116) of the project. This release is the first providing Python 3.11 distributions, joining existing support for Python 3.8, 3.9, and 3.10. Distributions are available for Windows, Linux, and macOS covering multiple machine architectures and levels of optimization. ___ Python-announce-list mailing list -- python-announce-list@python.org To unsubscribe send an email to python-announce-list-le...@python.org https://mail.python.org/mailman3/lists/python-announce-list.python.org/ Member address: arch...@mail-archive.com
[Python-announce] PyOxy: single file Python distributions with flexible interpreter control
I'm excited to announce the initial release of PyOxy: a single executable file (C)Python distribution and alternative Python runner that gives you more flexibility and control over how Python interpreters are run. More details at https://gregoryszorc.com/blog/2022/05/10/announcing-the-pyoxy-python-runner/ and https://pyoxidizer.readthedocs.io/en/latest/pyoxy.html. ___ Python-announce-list mailing list -- python-announce-list@python.org To unsubscribe send an email to python-announce-list-le...@python.org https://mail.python.org/mailman3/lists/python-announce-list.python.org/ Member address: arch...@mail-archive.com
[Python-announce] oxidized_importer 0.3 released
oxidized_importer - a pure Rust Python extension module implementing high performance Python module importers - version 0.3 has been released. The documentation is available at https://pyoxidizer.readthedocs.io/en/oxidized-importer-0.3/oxidized_importer.html . Like previous versions of this extension module, version 0.3 supports importing Python modules and resources from memory or from memory mapped files using the OxidizedFinder class. This is the special importer that enables PyOxidizer to generate single file Python applications without relying on run-time extraction of Python modules to a filesystem. New in version 0.3 is the OxidizedZipFinder ( https://pyoxidizer.readthedocs.io/en/oxidized-importer-0.3/oxidized_importer_zip_finder.html) class, which aims to be a drop-in replacement for zipimport.zipimporter. While not quite reaching that goal, this zip file importer is already considerably faster than the one in the Python standard library and can often be leveraged by existing users of zipimport wishing to shave precious milliseconds off application startup overhead. OxidizedZipFinder isn't as fast as OxidizedFinder. But it provides competitive performance while hopefully retaining sufficient compatibility with zipimport. oxidized_importer is developed as part of the PyOxidizer Project and feedback can be submitted via GitHub issues at https://github.com/indygreg/PyOxidizer. ___ Python-announce-list mailing list -- python-announce-list@python.org To unsubscribe send an email to python-announce-list-le...@python.org https://mail.python.org/mailman3/lists/python-announce-list.python.org/ Member address: arch...@mail-archive.com
[issue40413] Py_RunMain() crashes on subsequence call
Gregory Szorc added the comment: I can't remember the circumstances of me reproducing this. The added test demonstrating that the interpreter can be initialized/finalized within a single process seems like a reasonable solution to ensure this is supported. -- ___ Python tracker <https://bugs.python.org/issue40413> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45405] configure fails on macOS with non-Apple clang version 13 which implements --print-multiarch
Gregory Szorc added the comment: > Can you be more specific? This particular issue isn't about cross-compiling. The point I was trying to make is that clang 13 generically implemented --print-multiarch and the triple logic in configure related to MULTIARCH applies to all platforms, not just macOS. I was able to perturb this same error on Linux with an x86_64 Clang 13 with configure options --build=x86_64-unknown-linux-gnu --host=i686-unknown-linux-gnu. https://github.com/indygreg/python-build-standalone/runs/3853832124?check_suite_focus=true should work if you are logged into GitHub. Not the exact same failure, but related since it involves --print-multiarch suddenly working. -- ___ Python tracker <https://bugs.python.org/issue45405> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45405] configure fails on macOS with non-Apple clang version 13 which implements --print-multiarch
Gregory Szorc added the comment: Note that this issue isn't macOS specific: you will get the same failure when cross-compiling targeting Linux. e.g. --build=x86_64-unknown-linux-gnu --host=i686-unknown-linux-gnu. -- nosy: +indygreg ___ Python tracker <https://bugs.python.org/issue45405> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41994] Refcount issues in import
Gregory Szorc added the comment: I didn't want to derail this old issue too much. So I filed issue45307 to track a solution. -- ___ Python tracker <https://bugs.python.org/issue41994> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45307] Removal of _PyImport_FindExtensionObject() in 3.10 limits custom extension module loaders
New submission from Gregory Szorc : https://bugs.python.org/issue41994 / commit 4db8988420e0a122d617df741381b0c385af032c removed _PyImport_FindExtensionObject() from the C API and it is no longer available in CPython 3.10 after alpha 5. At least py2exe and PyOxidizer rely on this API for implementing a custom module loader for extension modules. Essentially, both want to implement a custom Loader.create_module() so they can use a custom mechanism for injecting a shared library into the process. While the details shouldn't be important beyond "they can't use imp.create_dynamic()," both use a similar technique that hooks LoadLibrary() on Windows to enable them to load a DLL from memory (as opposed to a file). While I don't have the extension module loading mechanism fully paged in to my head at the moment, I believe the reason that _PyImport_FindExtensionObject() (now effectively import_find_extension()) is important for py2exe and PyOxidizer is because they need to support at most once initialization, including honoring the multi-phase initialization semantics. Since the state of extension modules is stored in `static PyObject *extensions` and the thread state (which are opaque to the C API), the loss of _PyImport_FindExtensionObject() means there is no way to check for and use an existing extension module module object from the bowels of the importer machinery. And I think this means it isn't possible to implement well-behaved alternate extension module loaders any more. I'm aware the deleted API was "private" and probably shouldn't have been used in the first place. And what py2exe and PyOxidizer are doing here is rather unorthodox. In my mind the simplest path forward is restoring _PyImport_FindExtensionObject(). But a properly designed public API is probably a better solution. Until 3.10 makes equivalent functionality available or another workaround is supported, PyOxidizer won't be able to support loading extension modules from memory on Windows on Python 3.10. This is unfortunate. But probably not a deal breaker and I can probably go several months shipping PyOxidizer with this regression without too many complaints. -- components: C API messages: 402754 nosy: indygreg, petr.viktorin, serhiy.storchaka, vstinner priority: normal severity: normal status: open title: Removal of _PyImport_FindExtensionObject() in 3.10 limits custom extension module loaders type: enhancement versions: Python 3.10 ___ Python tracker <https://bugs.python.org/issue45307> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41994] Refcount issues in import
Gregory Szorc added the comment: While testing 3.10.0rc2, I noticed that commit 4db8988420e0a122d617df741381b0c385af032c removed _PyImport_FindExtensionObject() from libpython. This breaks both PyOxidizer and py2exe, which rely on this function for a custom implementation of Loader.create_module() to enable loading Windows DLLs from memory. It can probably be worked around. But I wanted to note it here in case it was an unwanted regression. -- nosy: +indygreg ___ Python tracker <https://bugs.python.org/issue41994> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45020] Freeze all modules imported during startup.
Gregory Szorc added the comment: I'll throw out that __file__ is strictly optional, per https://docs.python.org/3/reference/datamodel.html. IMO it is more important to define importlib.abc.InspectLoader than __file__ (so tracebacks have source context). Given that __file__ is optional, I'll throw out the idea of shipping modules without __file__/__cached__ and see what issues/pushback occurs. I anticipate this will likely result in having to restore advertising __file__ in frozen modules in 3.11 due to popular demand. But we /could/ use this as an opportunity to start attempting to ween people off __file__ and onto more appropriate APIs, like importlib.resources. (importlib.resources was introduced in 3.7 and I'm not sure that's old enough to force people onto without relying on a shim like https://pypi.org/project/importlib-resources/.) Alternatively, defining a path hook referencing the python executable that knows how to service frozen modules could also work. zipimporter does this (__file__ is /) and it is surprisingly not as brittle as one would think. It might be a good middle ground. -- ___ Python tracker <https://bugs.python.org/issue45020> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems
Gregory Szorc added the comment: I cannot reproduce this on an 11.5 Intel MacBook Pro using an 11.5 SDK targeting x86_64 10.9. However, I can reproduce on a similarly configured M1 using the same OS and SDK but cross-compiling to single arch x86_64. The issue here may reside in how configure handles cross-compiling for Apple. The current logic in configure is a bit opinionated about how things should work and this patch may have tickled things into not working. My opinion is that configure should support various cross-compiling scenarios like this (including building non-fat x86_64 only binaries from an arm64 machine). However, it is clear from the implementation and comments in this issue and elsewhere that the CPython core devs want to limit which exact cross-compiling configurations are supported on Apple. If the core developers say various cross-compiling scenarios aren't supported build configuration any more, I'll accept that and work around the limitation on my end. However, do note there is a significant possibility that Apple stops selling Intel machines for various product models in a few weeks. I suspect various people will want to continue their practice of building x86_64 only binaries for the indefinite future. Regardless of when the ARM only transition occurs, arbitrary restrictions like not supporting Apple arm64 -> x86_64 cross-compiling will disappoint a growing cohort of users over time. I would encourage investing in less opinionated configure logic to support Apple cross-compiling. I could potentially contribute patches in this area, since I've already taught python-build-standalone to cross-compile more robustly (including to targets like iOS). -- ___ Python tracker <https://bugs.python.org/issue44689> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems
Gregory Szorc added the comment: > On Sep 1, 2021, at 22:58, Ned Deily wrote: > > I don't think we have ever claimed to support building on an older system > with a newer SDK, as in building on 10.15 with an 11 SDK. My initial report was from a 10.15 Intel machine in GitHub Actions. My comment a few hours ago was from an 11.3 M1 cross-compiling to x86_64 and targeting 10.9 from a 11.X SDK. -- ___ Python tracker <https://bugs.python.org/issue44689> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems
Gregory Szorc added the comment: I spoke too soon: you can reproduce this with CPython's build system and I think this is a legit regression. I think the function sniffing logic is subtly wrong. Here is the logic as written: #ifdef __APPLE__ #ifdef HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH #define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME \ __builtin_available(macOS 11.0, iOS 14.0, tvOS 14.0, watchOS 7.0, *) #else static bool (*_dyld_shared_cache_contains_path)(const char *path); #define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME \ _dyld_shared_cache_contains_path != NULL #endif #endif The fundamental problem is that HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH comes from configure. Configure is using the target settings to probe for function presence. If I set the deployment target to 10.9, it says `checking for _dyld_shared_cache_contains_path... no`. But if I set the deployment target to 11.0, it says `checking for _dyld_shared_cache_contains_path... yes`. Because we may be targeting a macOS without the function, the function appears as not available to configure. It may exist in the SDK, but it is masked behind availability checks for the test compile. The logic as written assumes that !HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH means the SDK doesn't contain the function at all. This just isn't true for SDKs >= 11.0. If you look at posixmodule.c, the logic for checking for function availability is typically this pattern: #ifdef __APPLE__ #ifdef HAVE_BUILTIN_AVAILABLE #define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME \ __builtin_available(macOS 11.0, iOS 14.0, tvOS 14.0, watchOS 7.0, *) #else #ifdef HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH #define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME (_dyld_shared_cache_contains_path != NULL) #endif #endif #endif This other pattern also reveals another regression with this patch: __builtin_available() may not be available on older compilers. See issue42692. I'm unsure what "older compilers" actually is. But someone is bound to complain about this (if they haven't already). Do these build regressions warrant an unplanned 3.9.8 release? -- ___ Python tracker <https://bugs.python.org/issue44689> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems
Gregory Szorc added the comment: Oh, this might be my custom Modules file not passing the HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH define properly. I suspect I was working around this bug before by disabling the dyld bits. Sorry for the noise! (I should have reproduced with CPython's build system.) -- ___ Python tracker <https://bugs.python.org/issue44689> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems
Gregory Szorc added the comment:
I think patch broke building on macOS < 11 when building with a 11.0+ SDK and
targeting macOS < 11? This build configuration previously used to work with
3.9.6.
clang -Wno-unused-result -Wsign-compare -g -O0 -Wall -arch x86_64
-mmacosx-version-min=10.9 -Wno-nullability-completeness
-Wno-expansion-to-defined -Wno-undef-prefix -isysroot
/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
-fPIC
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/ncursesw
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/uuid
-Werror=unguarded-availability-new -std=c99 -Wextra -Wno-unused-result
-Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes
-Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal
-I. -I./Include -arch x86_64 -mmacosx-version-min=10.9
-Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix
-isysroot
/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/S
DKs/MacOSX11.1.sdk -fPIC
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/ncursesw
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/uuid
-Werror=unguarded-availability-new -DMACOSX -DUSING_MALLOC_CLOSURE_DOT_C=1
-DHAVE_FFI_PREP_CIF_VAR=1 -DHAVE_FFI_PREP_CLOSURE_LOC=1
-DHAVE_FFI_CLOSURE_ALLOC=1 -DPy_BUILD_CORE_BUILTIN -I_ctypes/darwin -c
./Modules/_ctypes/callproc.c -o Modules/callproc.o
python-3.9> ./Modules/_ctypes/callproc.c:1459:15: error: redefinition of
'_dyld_shared_cache_contains_path' as different kind of symbol
cpython-3.9> static bool (*_dyld_shared_cache_contains_path)(const char *path);
cpython-3.9> ^
cpython-3.9>
/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/include/mach-o/dyld.h:121:13:
note: previous definition is here
cpython-3.9> extern bool _dyld_shared_cache_contains_path(const char* path)
__API_AVAILABLE(macos(11.0), ios(14.0), watchos(7.0), tvos(14.0))
DYLD_DRIVERKIT_UNAVAILABLE;
cpython-3.9> ^
cpython-3.9> ./Modules/_ctypes/callproc.c:1464:42: error: non-object type 'bool
(const char *)' is not assignable
cpython-3.9> _dyld_shared_cache_contains_path =
dlsym(libsystem_b_handle, "_dyld_shared_cache_contains_path");
cpython-3.9> ^
cpython-3.9> ./Modules/_ctypes/callproc.c:1464:9: error:
'_dyld_shared_cache_contains_path' is only available on macOS 11.0 or newer
[-Werror,-Wunguarded-availability-new]
cpython-3.9> _dyld_shared_cache_contains_path =
dlsym(libsystem_b_handle, "_dyld_shared_cache_contains_path");
cpython-3.9> ^~~~
cpython-3.9>
/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/include/mach-o/dyld.h:121:13:
note: '_dyld_shared_cache_contains_path' has been marked as being introduced
in macOS 11.0 here, but the deployment target is macOS 10.9.0
cpython-3.9> extern bool _dyld_shared_cache_contains_path(const char* path)
__API_AVAILABLE(macos(11.0), ios(14.0), watchos(7.0), tvos(14.0))
DYLD_DRIVERKIT_UNAVAILABLE;
cpython-3.9> ^
cpython-3.9> ./Modules/_ctypes/callproc.c:1464:9: note: enclose
'_dyld_shared_cache_contains_path' in a __builtin_available check to silence
this warning
cpython-3.9> _dyld_shared_cache_contains_path =
dlsym(libsystem_b_handle, "_dyld_shared_cache_contains_path");
cpython-3.9> ^~~~
cpython-3.9> ./Modules/_ctypes/callproc.c:1482:10: error:
'_dyld_shared_cache_contains_path' is only available on macOS 11.0 or newer
[-Werror,-Wunguarded-availability-new]
cpython-3.9> if (HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME) {
cpython-3.9> ^~~~
cpython-3.9> ./Modules/_ctypes/callproc.c:1474:5: note: expanded from macro
'HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME'
cpython-3.9> _dyld_shared_cache_contains_path != NULL
cpython-3.9> ^~~~
cpython-3.9>
/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/include/mach-o/dyld.h:121:13:
note: '_dyld_shared_cache_contains_path' has been marked as being introduced
in macOS 11.0 here, but the deployment target is macOS 10.9.0
cpython-3.9> extern bool _dyld_shared_cache_contains_path(const char* path)
__API_AVAILABLE(macos(11.0), ios(14.0), watchos(7.0), tvos(14.0
[issue45020] Freeze all modules imported during startup.
Gregory Szorc added the comment: > For stdlib modules it wouldn't be a big problem to set __file__ on > frozen modules. > Would that be enough to solve the problem? What do you set __file__ to? Do you still materialize the .py[c] files on disk and set to that? (This would make the most sense.) If you support setting __file__ on a frozen module, how does this work at the C level? A stdlib module would want a different __file__ from a 3rd party module using the frozen module API. This would seemingly require enhancements to the frozen modules C API or some kind of hackery only available to stdlib frozen modules. >> When I investigated freezing the standard library for PyOxidizer, I ran into >> a rash >> of problems. The frozen importer doesn't behave like PathFinder. It doesn't >> (didn't?) set some common module-level attributes > This is mostly fixable for stdlib modules. Which attributes would > need to be added? Are there other missing behaviors? It was a few years ago and I can't recall specifics. I just remember that after encountering a couple unrelated limitations with the frozen importer I decided it was far too primitive to work as a general importer and decided I'd have to roll my own. > Good point. Supporting more of the FileLoader API on the frozen > loader is something to look into, at least for stdlib modules. >> I agree that we should shore up the frozen importer -- probably in a >> separate PR though. >> (@Eric: do you think this is worth its own bpo issue?) > Yeah. I have some observations about the implications of this. I typed up a long comment but then realized someone would probably complain about me distracting from the technical parts of this issue. Which forum is most appropriate for this less technical commentary? (I want to lay out the case for building out an official importer far more advanced than frozen importer.) -- ___ Python tracker <https://bugs.python.org/issue45020> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45025] Reliance on C bit fields in C API is undefined behavior
Gregory Szorc added the comment: My use case for these low-level APIs is to write tests for low-level string/encoding handling in my custom use of the PyPreConfig and PyConfig structs. I wanted to verify that exact byte sequences were turned into specific representations inside of Python strings. This includes ensuring that certain byte sequences retain their appropriate "character" width in internal storage. I know there are alternative ways of performing this testing. But testing against the actual data structure used internally by CPython seemed the most precise since it isolates problems to the "store in Python" side of the problem and not "what does Python do once the data is stored." -- ___ Python tracker <https://bugs.python.org/issue45025> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45020] Freeze all modules imported during startup.
Gregory Szorc added the comment: Oh, PyOxidizer also ran into more general issues with the frozen importer in that it broke various importlib APIs. e.g. because the frozen importer only supports bytecode, you can't use .__loader__.get_source() to obtain the source of a module. This makes tracebacks more opaque and breaks legitimate API consumers relying on these importlib interfaces. The fundamental limitations with the frozen importer are why I implemented my own meta path importer (implemented in pure Rust), which is more fully featured, like the PathFinder importer that most people rely on today. That importer is available on PyPI (https://pypi.org/project/oxidized-importer/) and has its own API to facilitate PyOxidizer-like functionality (https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer.html) if anyone wants to experiment with it. -- ___ Python tracker <https://bugs.python.org/issue45020> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45020] Freeze all modules imported during startup.
Gregory Szorc added the comment: When I investigated freezing the standard library for PyOxidizer, I ran into a rash of problems. The frozen importer doesn't behave like PathFinder. It doesn't (didn't?) set some common module-level attributes that are documented by the importer "specification" to be set and this failed a handful of tests and lead to runtime issues or breakage in 3rd party packages (such as random packages looking for a __file__ on a common stdlib module). Also, when I last looked at the CPython source, the frozen importer performed a linear scan of its indexed C array performing strcmp() on each entry until it found what it was looking for. So adding hundreds of modules could result in sufficient overhead and justify using a more efficient lookup algorithm. (PyOxidizer uses Rust's HashMap to index modules by name.) I fully support more aggressive usage of frozen modules in the standard library to speed up interpreter startup. However, if you want to ship this as enabled by default, from my experience with PyOxidizer, I highly recommend: * Make sure you run unit tests against the frozen modules. If you don't do this, subtle differences in how the different importers behave will lead to problems. * Shoring up the implementation of the frozen importer to make it better conform with the importer specification. * Be prepared for people to complain about the loss of __file__. e.g. https://github.com/indygreg/PyOxidizer/issues/69 / https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer_behavior_and_compliance.html?highlight=__file__#file-and-cached-module-attributes -- nosy: +indygreg ___ Python tracker <https://bugs.python.org/issue45020> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45025] Reliance on C bit fields in C API is undefined behavior
Change by Gregory Szorc : -- title: Reliance on C bit fields is C API is undefined behavior -> Reliance on C bit fields in C API is undefined behavior ___ Python tracker <https://bugs.python.org/issue45025> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45025] Reliance on C bit fields is C API is undefined behavior
New submission from Gregory Szorc : At least the PyASCIIObject struct in Include/cpython/unicodeobject.h uses bit fields. Various preprocessor macros like PyUnicode_IS_ASCII() and PyUnicode_KIND() access this struct's bit field. This is problematic because according to the C specification, the storage of bit fields is unspecified and may vary from compiler to compiler or architecture to architecture. Theoretically, a build of libpython with compiler A may not have the same storage layout of a bit field as a separate binary built with compiler B. These 2 binaries could be linked/loaded together, resulting in a crash or incorrect behavior at run-time. https://stackoverflow.com/questions/6043483/why-bit-endianness-is-an-issue-in-bitfields/6044223#6044223 To ensure bit field behavior is consistent, the same compiler must be used for all bit field interaction. Since it is effectively impossible to ensure this for programs like Python where multiple compilers are commonly at play (a 3rd party C extension will likely not be built on the same machine that built libpython), bit fields must not be exposed in the C API. If a bit field must exist, the bit field should not be declared in a public .h header and any APIs for accessing the bit field must be implemented as compiled functions such that only a single compiler will define the bit field storage layout. In order to avoid undefined behavior, Python's C API should avoid all use of bit fields. This issue is in response to https://github.com/PyO3/pyo3/issues/1824. -- components: C API messages: 400388 nosy: indygreg priority: normal severity: normal status: open title: Reliance on C bit fields is C API is undefined behavior type: behavior versions: Python 3.10, Python 3.11, Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue45025> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41949] Redefinition of HMAC functions prevents static linking
Gregory Szorc added the comment: I think this was effectively fixed in 3.10 via commit 39258d3595300bc7b952854c915f63ae2d4b9c3e / bpo-43669, which removed code creating the duplicate symbols. It is still a problem in 3.9. But possibly not worth the backport. -- ___ Python tracker <https://bugs.python.org/issue41949> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41100] Support macOS 11 and Apple Silicon Macs
Gregory Szorc added the comment: I ran into a minor issue with the ctypes extension and the _dyld_shared_cache_contains_path dependency that was added as part of this work (commit e8b1c038b14b5fc8120aab62c9bf5fb840274cb6). When building on Intel macOS 10.15 using an 11.x SDK, the configure check for this symbol passes (#define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH 1) and the extension compiles (albeit with a lot of deprecation warnings spewing from the Apple SDK headers). However, at run-time, we blow up due to a missing symbol error: ``` 2021-02-21T21:57:21.4116370Z cpython-3.9> clang -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/_decimal.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/basearith.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/constants.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/context.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/convolute.o build/temp.macosx-11.0-x86_64-3.9/private/var/fol ders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/crt.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/difradix2.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/fnt.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/fourstep.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/io.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/mpalloc.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/mpdecimal .o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/numbertheory.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/sixstep.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/transpose.o -L/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpasb48bzk/tools/pyhost/lib -L/usr/local/lib -lm -o build/lib.macosx-11.0-x86_64-3.9/_decimal.cpython-39-darwin.so 2021-02-21T21:57:27.1635100Z cpython-3.9> *** WARNING: renaming "_ctypes" since importing it failed: dlopen(build/lib.macosx-11.0-x86_64-3.9/_ctypes.cpython-39-darwin.so, 2): Symbol not found: __dyld_shared_cache_contains_path 2021-02-21T21:57:27.1637220Z cpython-3.9> Referenced from: build/lib.macosx-11.0-x86_64-3.9/_ctypes.cpython-39-darwin.so (which was built for Mac OS X 11.0) 2021-02-21T21:57:27.1638560Z cpython-3.9> Expected in: /usr/lib/libSystem.B.dylib 2021-02-21T21:57:27.1639770Z cpython-3.9> in build/lib.macosx-11.0-x86_64-3.9/_ctypes.cpython-39-darwin.so ``` (This build output is from https://github.com/indygreg/python-build-standalone/runs/1947611547?check_suite_focus=true) The missing symbol is provided by libdyld.dylib, which doesn't appear to be linked directly by CPython's build system. I'm only able to reproduce this on a GitHub Actions 10.5 machine using the 11.1 SDK: it seems to work fine on both an 11.1 Intel and M1 device. I reckon the correct fix is to have _ctypes link libdyld is this build configuration. I was also able to work around it in python-build-standalone by unsetting HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH (hacky solution but this feature isn't needed by the build that was failing). And of course linking a non-Apple libffi should also work, which is potentially why pyenv/Homebrew/etc users aren't noticing. python-build-standalone uses its own Clang built from source, so it is entirely possible there's something differing between the Apple Clang and my Clang causing this to fail. However, at least 1 other person ran into this in the wild (https://stackoverflow.com/questions/65853539/symbol-not-found-dyld-shared-cache-contains-path-on-osx-after-installl-from)
[issue42564] "from .__init__ import ..." syntax imports a duplicate module
Gregory Szorc added the comment: Who uses this syntax? https://github.com/search?l=Python=%22from+.__init__+import%22=Code says a lot of random code, surprisingly/sadly. As for python-ideas, thanks for the suggestion: I may very well craft an email! -- ___ Python tracker <https://bugs.python.org/issue42564> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42564] "from .__init__ import ..." syntax imports a duplicate module
Gregory Szorc added the comment: I worked around this in PyOxidizer by stripping a trailing `.__init__` from module names when resolving the indexed resource data. This allows the import to work since it can find the data now, but it also preserves the double module object, which isn't ideal IMO. My preferred solution would be to either ban `__init__` in module name components or strip trailing `.__init__` from the name in `find_spec()`, effectively normalizing it away. Either of these would be backwards incompatible. Could either of these be considered for 3.10? It's worth noting that `__init__` could potentially occur in the interior of the module name. e.g. `foo.__init__.bar`. This would require filenames like `foo/__init__/bar.py`. I wouldn't be surprised if this exists somewhere in the wild. -- ___ Python tracker <https://bugs.python.org/issue42564> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42564] "from .__init__ import ..." syntax imports a duplicate module
New submission from Gregory Szorc : (Rereporting from https://github.com/indygreg/PyOxidizer/issues/317.) $ mkdir foo $ cat > foo/__init__.py < test = True > EOF $ cat > foo/bar.py < from .__init__ import test > EOF $ python3.9 Python 3.9.0 (default, Nov 1 2020, 22:40:00) [GCC 10.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import foo.bar >>> import sys >>> sys.modules['foo'] >>> sys.modules['foo.__init__'] I am surprised that `from .__init__` even works, as `__init__` isn't a valid module name. What appears to be happening is the path based importer doesn't recognize the `__init__` as special and it falls back to its regular file probing technique to locate a module derive from the path. It finds the `__init__.py[c]` file and imports it. A consequence of this is that the explicit `__init__` import/module exists as a separate module object under `sys.modules`. So you can effectively have the same file imported as 2 module objects living under 2 names. This could of course result in subtle software bugs, like module-level variables not updating when you expect them to. (This could also be a feature for code relying on this behavior, of course.) I only attempted to reproduce with 3.9. But this behavior has likely existed for years. -- components: Interpreter Core messages: 382464 nosy: indygreg priority: normal severity: normal status: open title: "from .__init__ import ..." syntax imports a duplicate module type: behavior versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue42564> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41949] Redefinition of HMAC functions prevents static linking
Gregory Szorc added the comment: My workaround was to remove `|| defined(LIBRESSL_VERSION_NUMBER)`. That works with LibreSSL 3.1.4. I'm unsure if the symbols were introduced in a specific version though. But since python-build-standalone pins all dependencies, this is a safe patch for me. -- ___ Python tracker <https://bugs.python.org/issue41949> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41949] Redefinition of HMAC functions prevents static linking
Gregory Szorc added the comment: I was going to work around this in python-build-standalone (https://github.com/indygreg/python-build-standalone) by reverting to OpenSSL for builds. But we must use LibreSSL when linking against musl libc due to some random weirdness getting OpenSSL to statically link against musl. So I'm going to have to work around this via a source patch in python-build-standalone. Not the first time I've had to patch CPython source code to get a working statically linked build working. -- ___ Python tracker <https://bugs.python.org/issue41949> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41949] Redefinition of HMAC functions prevents static linking
Change by Gregory Szorc : -- nosy: +christian.heimes ___ Python tracker <https://bugs.python.org/issue41949> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41949] Redefinition of HMAC functions prevents static linking
Change by Gregory Szorc : -- type: -> compile error ___ Python tracker <https://bugs.python.org/issue41949> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41949] Redefinition of HMAC functions prevents static linking
New submission from Gregory Szorc : Commit 54f2898fe7e4ca1f239e96284af3cc5b34d2ae02 (bpo-40645) introduced the functions HMAC_CTX_new, HMAC_CTX_free, and HMAC_CTX_get_md. These functions share the same names as HMAC functions provided by OpenSSL/LibreSSL. If you attempt to statically link the _hashlib extension as a builtin extension module that is also statically linked against a libcrytpo that provides these functions, the linker may complain about duplicate symbols: cpython-3.9> /tools/host/bin/ld: /tools/deps/lib/libcrypto.a(libcrypto_la-hmac.o): in function `HMAC_CTX_free': cpython-3.9> hmac.c:(.text+0x7d0): multiple definition of `HMAC_CTX_free'; cpython-3.9> Modules/_hashopenssl.o:/build/Python-3.9.0/./Modules/_hashopenssl.c:54: first defined here cpython-3.9> /tools/host/bin/ld: /tools/deps/lib/libcrypto.a(libcrypto_la-hmac.o): in function `HMAC_CTX_get_md': cpython-3.9> hmac.c:(.text+0xa20): multiple definition of `HMAC_CTX_get_md'; Modules/_hashopenssl.o:/build/Python-3.9.0/./Modules/_hashopenssl.c: 63: first defined here cpython-3.9> /tools/host/bin/ld: /tools/deps/lib/libcrypto.a(libcrypto_la-hmac.o): in function `HMAC_CTX_new': cpython-3.9> hmac.c:(.text+0x780): multiple definition of `HMAC_CTX_new'; Modules/_hashopenssl.o:/build/Python-3.9.0/./Modules/_hashopenssl.c:42: first defined here cpython-3.9> clang-10: error: linker command failed with exit code 1 (use -v to see invocation) cpython-3.9> ln: failed to access 'libpython3.9.so.1.0': No such file or directory cpython-3.9> make: *** [libpython3.9.so] Error 1 cpython-3.9> Makefile:656: recipe for target 'libpython3.9.so' failed This log from a build attempting to statically link against LibreSSL 3.1.4. The issue does not reproduce against OpenSSL 1.1.1h for some reason. While statically linking _hashlib as a built-in extension module and statically linking libcrypto isn't in the default configuration, I believe this use case should be supported. Perhaps these 3 functions should be renamed to not conflict with symbols provided by libcrypto? -- components: Extension Modules messages: 378080 nosy: indygreg priority: normal severity: normal status: open title: Redefinition of HMAC functions prevents static linking versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue41949> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41669] Case mismatch between "include" and "Include"
New submission from Gregory Szorc :
On Windows, `sysconfig.get_path('include')` returns `Include` (capital I).
However, the actual installation path is `include` (lowercase i).
I believe the reason for the case mismatch is in this code in PC/layout/main.py:
```
if ns.include_dev:
for dest, src in rglob(ns.source / "Include", "**/*.h"):
yield "include/{}".format(dest), src
src = ns.source / "PC" / "pyconfig.h"
yield "include/pyconfig.h", src
```
The case mismatch is relevant for case sensitive filesystems. In my case, I was
extracting a Windows Python install on Linux and then using the `sysconfig`
value to locate directories within that install. Due to the case mismatch,
Linux says `Include` doesn't exist since `sysconfig` points to "include."
Case only renames can be a bit wonky to perform. I would suggest preserving
what is shipped today and changing `sysconfig` to advertise lowercase
"include." However, this would create a mismatch between the cpython source
repo and built distributions. So maybe attempting the case-only rename is
doable.
Note that Windows will not allow you to have a file/directory varying only in
case. i.e. you can have both an "include" and "Include." I can't recall if you
can perform a case-only rename with a single system call (meaning it is atomic)
or whether you need to go through a temporary directory.
--
components: Windows
messages: 376131
nosy: indygreg, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Case mismatch between "include" and "Include"
type: behavior
versions: Python 3.8
___
Python tracker
<https://bugs.python.org/issue41669>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32030] PEP 432: Rewrite Py_Main()
Change by Gregory Szorc : -- nosy: +indygreg nosy_count: 7.0 -> 8.0 pull_requests: +19218 pull_request: https://github.com/python/cpython/pull/19746 ___ Python tracker <https://bugs.python.org/issue32030> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40415] _asyncio extensions crashes if initialized multiple times in same process
Gregory Szorc added the comment: Oh, I just went to patch this and it is a dupe of 40294, which has already been fixed and backported. -- resolution: -> duplicate stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue40415> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40415] _asyncio extensions crashes if initialized multiple times in same process
Change by Gregory Szorc : -- nosy: +vstinner ___ Python tracker <https://bugs.python.org/issue40415> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40415] _asyncio extensions crashes if initialized multiple times in same process
New submission from Gregory Szorc : Most of CPython's extensions can be initialized and freed multiple times in the same process. However, _asyncio crashes on at least CPython 3.8.2 when this is done. STR: 1. Create a new Python interpreter 2. Have it import _asyncio 3. Finalize that interpreter. 4. Create a new Python interpreter 5. Have it import _asyncio There are probably STR in pure Python by forcing _imp.create_dynamic() to run multiple times after the module is unloaded. The crash occurs due to unchecked NULL access in `Py_INCREF(all_tasks);` in `PyInit__asyncio()`. I think the underlying problem is module_init() is short-circuiting because `module_initialized` is set. And `module_initialized` is set on subsequent module loads because `module_free()` isn't clearing it. -- components: asyncio messages: 367483 nosy: asvetlov, indygreg, yselivanov priority: normal severity: normal status: open title: _asyncio extensions crashes if initialized multiple times in same process versions: Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue40415> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40413] Py_RunMain() crashes on subsequence call
Gregory Szorc added the comment: Actually, I can reproduce the crash without Py_RunMain(). So I don't think the crash is related to the additional cleanup that Py_RunMain() does in addition to Py_FinalizeEx(). But I'd like to keep this issue open to track the original question about expected behavior. -- ___ Python tracker <https://bugs.python.org/issue40413> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40413] Py_RunMain() crashes on subsequence call
New submission from Gregory Szorc : I'm attempting to perform the following actions multiple times in a single process with CPython 3.8.2: 1) Initialize an interpreter using the PEP-587 APIs. 2) Call Py_RunMain() (which finalizes the interpreter). However, I've encountered at least 2 crashes due to use-after-free or unchecked NULL access (due to apparent state getting funky). Are multiple interpreters / Py_RunMain() calls in a single process supported? Should I file bugs for all of the crashes I encounter? -- components: C API messages: 367479 nosy: indygreg, vstinner priority: normal severity: normal status: open title: Py_RunMain() crashes on subsequence call type: crash versions: Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue40413> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40412] inittab_copy not set to NULL after free, can lead to crashes when running multiple interpreters in a single process
Change by Gregory Szorc : -- keywords: +patch pull_requests: +19068 stage: -> patch review pull_request: https://github.com/python/cpython/pull/19746 ___ Python tracker <https://bugs.python.org/issue40412> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40412] inittab_copy not set to NULL after free, can lead to crashes when running multiple interpreters in a single process
New submission from Gregory Szorc : Filing a bug to placate the requirement that pull requests have issues. -- components: C API messages: 367477 nosy: indygreg priority: normal severity: normal status: open title: inittab_copy not set to NULL after free, can lead to crashes when running multiple interpreters in a single process type: crash versions: Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue40412> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40333] Request for multi-phase initialization API to run code after importlib init
Gregory Szorc added the comment: Having approached this with a fresh brain, I was able to port PyOxidizer to use the multi-phase initialization API with minimal regressions! The relevant code exists at https://github.com/indygreg/PyOxidizer/blob/b5aa2b3a96dbd01e9d78857e124f1052f42f86c6/pyembed/src/interpreter.rs#L550. Here's the sequence: 1) Initialize core 2) Import our custom built-in extension module 3) Run some more code to initialize our extension module (update sys.meta_path, etc) 4) Initialize main 5) Remove PathFinder if filesystem imports are disabled #5 isn't ideal: I would prefer an API to disable the registration of that importer completely. But PyOxidizer's importer is first on sys.meta_path and PathFinder shouldn't come into play. So it should be mostly harmless. A super minor paper cut is the lack of a PyConfig_SetBytesString() variant for PyWideStringList_Append(). It was slightly annoying having to convert a POSIX char* path to a wchar_t* since paths on POSIX are bytes. Another potential area for improvement is around error handling before main is initialized. I'd like to represent PyErr raised during initialization as a Rust String, as PyErr isn't appropriate since there isn't a fully initialized Python interpreter. However, I discovered that serializing PyErr to strings is a bit brittle before main is initialized. e.g. https://docs.python.org/3/faq/extending.html#id11 says to use PyErr_Print() and replace sys.stdout. But sys.stdout isn't initialized yet and I'm scared to touch it. It also appears various functions in traceback rely on facilities from an initialized interpreter (such as finding sources). It would be useful if there were some kind of PyErr API that returned a PyString (or PyStatus) and was guaranteed to work before main is initialized. Overall, the new code in PyOxidizer is much, much cleaner! Thanks again for the new API! -- ___ Python tracker <https://bugs.python.org/issue40333> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40333] Request for multi-phase initialization API to run code after importlib init
New submission from Gregory Szorc : I'm porting PyOxidizer to the PEP 587 APIs. So far, it is mostly straightforward. I was looking forward to adopting the multi-phase initialization API because I thought it would enable me to get rid of the very ugly hack PyOxidizer uses to inject its custom meta path importer. This hack is described in gory detail at https://docs.rs/pyembed/0.7.0/pyembed/technotes/index.html#our-importing-mechanism and the tl;dr is we use a modified version of the `importlib._bootstrap_external` module to configure a builtin extension module on sys.meta_path so it can handle all imports during initialization. The good news is the multi-phase initialization API gives us an injection point between "core" and "main." I _think_ I would be able to import the built-in extension here and register it on sys.meta_path. However, the new multi-phase initialization API doesn't give us the total control that we need. Specifically, PyOxidizer's importer is leveraging a few functions in importlib._bootstrap_external as part of its operation. So it needs this module to be available before it can be loaded and installed on sys.meta_path. It also wants total control over sys.meta_path. So we don't want importlib._bootstrap_external to be mucking with sys.meta_path and imports being performed before PyOxidizer has a chance to readjust state. The critical feature that PyOxidizer needs is the ability to muck with sys.meta_path and importlib *after* importlib externals are initialized and *before* any non-builtin, non-frozen import is attempted. In the current state of the initialization code, we need to run custom code between init_importlib_external() and when the first non-builtin, non-frozen import is attempted (currently during _PyUnicode_InitEncodings()). Would it be possible to get a multi-phase initialization API that stops after init_importlib_external()? If not, could we break up PyConfig._install_importlib into 2 pieces to allow disabling of just importlib._bootstrap_external and provide a supported mechanism to initialize the external mechanism between "core" and "main" initialization? (Although I'm not sure if this is possible, since "main" finishes initializing aspects of "sys" before init_importlib_external() and I'm not sure if it is safe to initialize importlib externals before this is done. I'm guessing there is a reason that code runs before importlib is fully initialized.) I suppose I could change PyOxidizer's functionality a bit to work around the lack of an importlib._bootstrap_external module between "core" and "main" initialization. I'm pretty sure I could make this work. But my strong preference is to inject code after importlib external support is fully initialized but before any imports are performed with it. Overall the PEP 587 APIs are terrific and a substantial improvement over what came before. Thank you for all your work on this feature, Victor! -- components: C API messages: 366802 nosy: indygreg, vstinner priority: normal severity: normal status: open title: Request for multi-phase initialization API to run code after importlib init versions: Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue40333> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40333] Request for multi-phase initialization API to run code after importlib init
Change by Gregory Szorc : -- type: -> enhancement ___ Python tracker <https://bugs.python.org/issue40333> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40293] cpython-source-deps project missing release for libffi commits
Gregory Szorc added the comment: I don't like utilizing the dynamic archive links like https://github.com/python/cpython-source-deps/archive/libffi.zip (even if you pin the commit) because GitHub does not guarantee the file content is deterministic over time. I perform SHA-256 validation of all dependencies I download from the Internet. And if I rely on the /archive/ URLs, all it takes is GitHub updating some library that subtly changes the tar/zip structure and my hashes are invalidated. Release artifacts are immutable and don't have this problem. As it stands, I will likely `git clone` and check out the commit I need. Although I would prefer a release artifact. -- ___ Python tracker <https://bugs.python.org/issue40293> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40293] cpython-source-deps project missing release for libffi commits
New submission from Gregory Szorc : The https://github.com/python/cpython-source-deps project is missing a source archive release for commits to libffi needed to support building on Windows. The latest release of libffi is version libffi-3.3.0-rc0-r1, which corresponds to https://github.com/python/cpython-source-deps/commit/8ba2b2c38744420a235e3e7f419cce2591aaf331. There have been a few commits since that release: https://github.com/python/cpython-source-deps/compare/8ba2b2c38744420a235e3e7f419cce2591aaf331...libffi Most notable is https://github.com/python/cpython-source-deps/commit/ed22026f39b37f892ded95d7b30e77dfb5126334 because without this commit, the source is not buildable on Windows using MSVC, at least not with the prepare_ffi.bat script in CPython's Git repository. (Build fails due to issues with FFI_BUILDING_DLL). Would it be possible to cut a new tag and upload a source tarball for libffi? -- components: Build, Windows messages: 366523 nosy: indygreg, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: cpython-source-deps project missing release for libffi commits type: enhancement ___ Python tracker <https://bugs.python.org/issue40293> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36128] ResourceReader for FileLoader inconsistently handles path separators
Gregory Szorc added the comment: I just noticed that there is a parallel discussion ongoing in https://gitlab.com/python-devs/importlib_resources/issues/58. -- ___ Python tracker <https://bugs.python.org/issue36128> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36128] ResourceReader for FileLoader inconsistently handles path separators
Gregory Szorc added the comment: I think disallowing relative paths that are parents of the current anchor point is a reasonable restriction and acceptable backwards incompatible behavior. Disallowing all relative paths with slashes is a larger issue. I would support that if designing/implementing things today as it simplifies things greatly. But since an implementation allowing slashes has shipped in 3.7 and 3.8, I'm not sure if the backwards incompatibility could be stomached, so I'm not going to advocate for it. -- ___ Python tracker <https://bugs.python.org/issue36128> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38594] importlib.metadata documentation deficiencies
New submission from Gregory Szorc : As I was attempting to implement the find_distributions() interface for PyOxidizer, I got confused by importlib.metadata's documentation. The documentation for this module states: ``` What this means in practice is that to support finding distribution package metadata in locations other than the file system, you should derive from ``Distribution`` and implement the ``load_metadata()`` method. Then from your finder, return instances of this derived ``Distribution`` in the ``find_distributions()`` method. ``` The reference to `load_metadata()` is the only occurrence of the string `load_metadata` in the CPython and importlib_metadata code bases. I therefore believe the documentation in both CPython and the importlib_metadata standalone package are wrong because they are referring to a method that is never implemented nor called. Looking at the documentation and source code for importlib.metadata, I'm also a bit confused about how exactly I'm supposed to implement a custom Distribution which isn't based on filesystems. For example, I see that certain APIs return Path-like objects (which I will need to implement). But it isn't clear exactly which attributes are mandated to exist! Am I expected to implement the full pathlib.Path interface or just a subset? Regarding how find_distributions() is called, I also don't understand why the Context is optional and how Context could be used in some situations. For example, the implementation of discover() can construct Context instances with no arguments, which is then fed into find_distributions(). So I guess context=None or context.name=None implies "return Distribution's for every known package?" If so, this behavior is undocumented. I'm also not sure what Context.path is for. I /think/ it is only used for the path-based finder/distribution. But the way it is documented implies it should always exist, which doesn't seem appropriate for cases like PyOxidizer which will retrieve metadata from in-memory without filesystem I/O. I think what I'm trying to say is that the existing documentation for importlib.metadata is not sufficient to robustly implement a custom find_distributions() + Distribution type. I would kindly request that a domain expert revise the documentation such that a 3rd party can implement a custom solution. My preferred solution would be for there to be formal interfaces in importlib.abc like there are for everything else in the importlib realm. (The interfaces for finders and loaders are super useful when implementing a finder/loader from scratch.) FWIW I think I like the new metadata API and I think it is flexible enough to allow tools like PyOxidizer to do crazy things like divorce resources from the filesystem! But it is hard to say for sure since the interfaces aren't clearly defined at present. -- assignee: docs@python components: Documentation messages: 355402 nosy: barry, docs@python, indygreg, jaraco priority: normal severity: normal status: open title: importlib.metadata documentation deficiencies type: enhancement versions: Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue38594> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31601] Availability of utimensat, futimens not checked correctly on macOS
Gregory Szorc added the comment: I ran into this with PyOxidizer. CPython's `configure` needs to be made aware of the current macOS SDK version and filter out symbols not present in the target macOS version. I worked around it by `undef`'ing some entries from the generated `pyconfig.h` file. macOS's clang does emit some warnings when it sees symbols that shouldn't be used with the current target version. CPython's build system should consider adding -Werror=unguarded-availability-new to CFLAGS to turn these warnings into errors so they are caught at compile time and not run time. But there should be proper filtering of affected symbols first, otherwise people won't be able to build CPython when targeting older macOS versions. -- nosy: +indygreg ___ Python tracker <https://bugs.python.org/issue31601> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37459] importlib docs improperly reference get_resource_loader()
Gregory Szorc added the comment: I'm a bit busy with other things this week to submit a PR. -- ___ Python tracker <https://bugs.python.org/issue37459> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37459] importlib docs improperly reference get_resource_loader()
New submission from Gregory Szorc : The documentation in importlib.rst says that a loader should implement `get_resource_loader(fullname)`. This is the only occurrence of `get_resource_loader` in the CPython source tree. It should be changed to `get_resource_reader()`. -- components: Library (Lib) messages: 346951 nosy: indygreg priority: normal severity: normal status: open title: importlib docs improperly reference get_resource_loader() versions: Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue37459> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37326] Python distributions do not contain libffi license
Gregory Szorc added the comment: FWIW LICENSE.txt on Windows contains licenses for additional projects starting on line 258 (at least in CPython 3.7.3), which contains the section header "Additional Conditions for this Windows binary build". But, the Doc/pythonXXX.chm does indeed contain libffi's license text. So we're all good here. I'm not sure why LICENSE.txt on Windows is incomplete. *shrugs* -- ___ Python tracker <https://bugs.python.org/issue37326> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37326] Python distributions do not contain libffi license
New submission from Gregory Szorc : The Modules/_ctypes/libffi_msvc/ffi.h and Modules/_ctypes/libffi_osx/include/ffi.h files in the CPython repository contain copies of libffi. On at least the official Windows distributions, the LICENSE.txt does not contain libffi's license text from these files. This is seemingly in violation of libffi's license agreement, which clearly says "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software." -- components: Build messages: 345944 nosy: indygreg priority: normal severity: normal status: open title: Python distributions do not contain libffi license type: enhancement versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue37326> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37060] import ctypes fails with a statically linked interpreter due to dlopen() failure
New submission from Gregory Szorc : ctypes/__init__.py calls _ctypes.dlopen(None) on Linux as part of code execution during module import. Unfortunately, dlopen() doesn't work if the current executable isn't a dynamic executable. Using a fully statically linked Python executable: $ ldd python3.7 not a dynamic executable $ python3.7 >>> import _ctypes >>> _ctypes.dlopen(None) Traceback (most recent call last): File "", line 1, in OSError: Dynamic loading not supported >>> import ctypes Traceback (most recent call last): File "", line 1, in File "/home/gps/src/python-build-standalone.git/build/python/install/lib/python3.7/ctypes/__init__.py", line 444, in pythonapi = PyDLL(None) File "/home/gps/src/python-build-standalone.git/build/python/install/lib/python3.7/ctypes/__init__.py", line 356, in __init__ self._handle = _dlopen(self._name, mode) OSError: Dynamic loading not supported I think it is a bug that `import ctypes` raises OSError at import time in this environment. I can make a compelling argument that this error should either be suppressed or converted to an ImportError. Producing a fully statically linked Python executable is a bit of work and isn't easily accomplished with the existing build system. My "python-build-standalone" project automates the process. A fully statically linked Python executable is available in the zstd compressed archive at https://github.com/indygreg/python-build-standalone/releases/download/20190505/cpython-3.7.3-linux64-musl-20190526T0219.tar.zst under the python/install/bin/python3.7 path. Simply extract that archive and run that binary to reproduce. -- components: ctypes messages: 343588 nosy: indygreg priority: normal severity: normal status: open title: import ctypes fails with a statically linked interpreter due to dlopen() failure versions: Python 3.7 ___ Python tracker <https://bugs.python.org/issue37060> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36129] io documentation unclear about flush() and close() semantics for wrapped streams
Gregory Szorc added the comment: It's also worth noting that if the wrapped stream close() cascading behavior should be configurable, then many additional types in the standard library need a constructor argument to control this behavior. e.g. io.TextIOWrapper, io.BufferedReader, io.BufferedWriter, codecs.StreamReader, codecs.StreamWriter, etc. In my influenced-by-Rust mind, the problem is similar to "ownership." The question we seem to be dancing around is whether the stream wrapper "owns" the inner stream or just "borrows" it. If it "owns," then close() cascading absolutely makes sense. But if it just "borrows" the inner stream, then close() cascading should not occur. There are viable use cases for both scenarios on pretty much any stream wrapper. Since existing stream wrappers automatically perform close() cascading and __del__ will call close(), I'd be a bit surprised if others weren't complaining about the inability to disable close() cascade on stream wrappers! e.g. it is perfectly viable to want to temporarily wrap a binary stream with an io.TextIOWrapper but the forced close() cascading makes this difficult to do since destruction of the outer stream will close() the inner stream. So you end up needing to keep references to outer streams alive to prevent this. Eww. -- ___ Python tracker <https://bugs.python.org/issue36129> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36129] io documentation unclear about flush() and close() semantics for wrapped streams
Gregory Szorc added the comment: Thank you for the detailed reply, Josh. I generally agree with what you are saying. However, I have some follow-ups. In your answer to #2, you seem to distinguish between an "fd" (implying POSIX file descriptor) and "Python layers" (implying a file object). Is this correct? Should a "closefd" argument only apply when receiving an integer file descriptor? Or should a "closefd" argument also apply when receiving any io.RawIOBase object? If it doesn't apply for any generic file object, is there any recommended way to control whether close() cascades? (My opinion is that "closefd" should apply to any object with a close(), not just integer file descriptors.) lzma.LZMAFile and gzip.GZIPFile do *not* cascade close() when a file object (as opposed to a named file) is passed into the constructor. (Presumably bz2.BZ2File behaves the same way but the documentation doesn't state this.) This is inconsistent with your answer that close() should always cascade. It's worth noting that the docs for GZIPFile explicitly call out reasons why close() does not cascade. None of these compression *File constructors accept a "closefd" argument to control the behavior. If the goal is for close() to cascade by default, then it seems useful for these *File types to support automatic close() cascade. Although changing the default would be a backwards compatibility break and I'm not sure that's feasible. But at least you'd have the ability to opt in to the behavior. It's also worth calling out the behavior of io.BytesIO and io.StringIO. When you close() these streams, you cannot call .getvalue() to get the buffer content. This is consistent with the io.RawIOBase behavior of not allowing I/O after a close(). However, the implication is that if you wrap a BytesIO/StringIO and then a close() on the outer stream cascades into the BytesIO/StringIO, you won't be able to access the written data! In fact, I encountered this problem when writing python-zstandard's unit tests! I had to implement a custom type that allowed access to getvalue() post close() (https://github.com/indygreg/python-zstandard/blob/735771961bc04f8f7de9372297921826a814fd12/tests/common.py#L82) to work around it. Assuming a "closefd" argument applies to all file objects (not just file descriptors) and that all stream wrappers should accept a "closefd" argument to control close() cascade, I think I have a path forward for python-zstandard: I simply add a "closefd" argument to the stream wrapper constructors. But if "closefd" doesn't apply to generic file objects, then I'm still not sure how to proceed, as I don't want to implement behavior that conflicts with the standard library's. -- ___ Python tracker <https://bugs.python.org/issue36129> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36129] io documentation unclear about flush() and close() semantics for wrapped streams
New submission from Gregory Szorc : As part of implementing io.RawIOBase/io.BufferedIOBase compatible stream types for python-zstandard, I became confused about the expected behavior of flush() and close() when a stream is wrapping another stream. The documentation doesn't lay out explicitly when flush() and close() on the outer stream should be proxied to the inner stream, if ever. Here are some specific questions (it would be great to have answers to these in the docs): 1. When flush() is called on the outer stream, should flush() be called on the inner stream as well? Or should we simply write() to the inner stream and not perform a flush() on that inner stream? 2. When close() is called on the outer stream, should close() be called on the inner stream as well? 3. If close() is not automatically called on the inner stream during an outer stream's close(), should the outer stream trigger a flush() or any other special behavior on the inner stream? Or should it just write() any lingering data and then go away? 4. Are any of the answers from 1-3 impacted by whether the stream is a reader or writer? (Obviously reader streams don't have a meaningful flush() implementation.) 5. Are any of the answers from 1-3 impacted by whether the stream is in blocking/non-blocking mode? 6. Do any of the answers from 1-3 vary depending on whether behavior is incurred by the outer stream's __exit__? (This issue was inspired by https://github.com/indygreg/python-zstandard/issues/76.) -- components: Interpreter Core messages: 336715 nosy: indygreg priority: normal severity: normal status: open title: io documentation unclear about flush() and close() semantics for wrapped streams type: enhancement versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue36129> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36128] ResourceReader for FileLoader inconsistently handles path separators
New submission from Gregory Szorc :
The implementation of the ResourceReader API for the FileLoader class in
importlib/_bootstrap_external.py is inconsistent with regards to handling of
path separators.
Specifically, "is_resource()" returns False if "resource" has a path separator.
But "open_resource()" will happily open resources containing a path separator.
I would think the two would agree about whether a path with separators is a
resource or not. The documentation at
https://docs.python.org/3.7/library/importlib.html#importlib.abc.ResourceReader
implies that resources in subdirectories should not be allowed.
One can easily demonstrate this behavior oddity with Mercurial:
(Pdb) p
sys.modules['mercurial'].__spec__.loader.get_resource_reader('mercurial').open_resource('help/config.txt')
<_io.FileIO name='/home/gps/src/hg/mercurial/help/config.txt' mode='rb'
closefd=True>
(Pdb) p
sys.modules['mercurial'].__spec__.loader.get_resource_reader('mercurial').is_resource('help/config.txt')
False
The behavior has been present since the functionality was added
(https://github.com/python/cpython/pull/5168).
--
components: Library (Lib)
messages: 336712
nosy: barry, indygreg
priority: normal
severity: normal
status: open
title: ResourceReader for FileLoader inconsistently handles path separators
type: behavior
versions: Python 3.7, Python 3.8
___
Python tracker
<https://bugs.python.org/issue36128>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35642] _asynciomodule.c compiled in both pythoncore.vcxproj and _asyncio.vcxproj
New submission from Gregory Szorc : The _asynciomodule.c source file is compiled as part of both pythoncore.vcxproj (providing pythonXY.dll) and _asyncio.vcxproj (providing _asyncio.pyd). PC\config.c doesn't reference PyInit__asyncio. I'm inclined to believe that _asynciomodule.c being built as part of pythoncore.vcxproj is a mistake. If all goes according to plan, I will contribute my first CPython patch with a fix shortly... -- components: Build messages: 332887 nosy: Gregory.Szorc priority: normal severity: normal status: open title: _asynciomodule.c compiled in both pythoncore.vcxproj and _asyncio.vcxproj type: enhancement versions: Python 3.5, Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue35642> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31148] Can we get an MSI installer for something past 3.4.4?
Gregory Szorc <gregory.sz...@gmail.com> added the comment: I was going to update some CI that tests all supported versions of Python and pins the Python 3 versions. That CI was previously using the MSI installers for Python 3.4. To my surprise, the MSIs stopped being generated after the 3.4.4 release! https://www.python.org/ftp/python/3.4.4/ has MSIs (and MacOS .pkg files). But all subsequent releases (e.g. https://www.python.org/ftp/python/3.4.7/) do not. I'm OK with the decision to abandon MSIs and move to the .exe installer for newer Python releases (like 3.5 and 3.6). However, ceasing to produce the MSIs (and MacOS .pkg files for that matter) midway through 3.4's support cycle feels wrong to me. 3.4.4 was the last release with binary distributions for Windows and MacOS. 3.4.5 and newer releases have fixes to security issues. That means people using the binary distributions of 3.4 on these platforms are stuck on 3.4.4 and are thus vulnerable to known security issues. That's... not great. I think the MSI (and .pkg) distributions should be restored for 3.4. Furthermore, I would encourage Python to adopt the practice that distribution mechanisms aren't modified during a Python version's support cycle. Someone will inevitably rely on any distribution format that is published. And taking it away in a support cycle will orphan those users on an old, buggy release. -- nosy: +Gregory.Szorc ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31148> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30625] Documentation is unclear how "y*" and "y#" format units vary
Gregory Szorc added the comment: IMO I don't find the 3.2 docs more useful. Specifically, the behavior for memoryview is still unclear. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30625> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30625] Documentation is unclear how "y*" and "y#" formit units vary
New submission from Gregory Szorc: https://github.com/indygreg/python-zstandard/issues/26 is a bug report against python-zstandard that a C API using the "y#" format unit is unable to accept a memoryview instance constructed from a bytes literal. It fails with "TypeError: argument 1 must be read-only bytes-like object, not memoryview". I understand why the "y*" format unit and the buffer protocol are a superior API. In hindsight, I should have used the buffer protocol from the beginning in python-zstandard. However, this decision was primarily influenced because the docs aren't clear about the apparent limitations of "y#" compared to "y*" and I believed "y#" would get me what I wanted (pointer to read-only memory) without having to be burdened by the complexity of the buffer protocol in my C code. So, docs issue #1 is that the limitations of "y#" compared to "y*" aren't immediately obvious when reading their respective sections in https://docs.python.org/3/c-api/arg.html#strings-and-buffers. To their credit, the docs for "y*" do have a bold recommendation to use it. But what's missing is the critical "why." There is also a paragraph above the format unit list explaining "bytes-like object" but this is detached from the definitions for "y#" and "y*" so it is easy to pass over. Issue #2 (which may be an implementation bug) is why "y#" isn't accepting a memoryview constructed from bytes. The docs for "y#" say it accepts a "read-only bytes-like object," which is defined at https://docs.python.org/3/glossary.html#term-bytes-like-object. And the paragraphs there explicitly state that a memoryview of an immutable bytes is in fact a "read-only bytes-like object." So I have no clue why "y#" is refusing such an object. I'll gladly author a documentation fix. However, I'm not sure what to say because I don't understand why "y#" isn't working in this memoryview(bytes) case and whether that issue applies to other types. -- assignee: docs@python components: Documentation messages: 295649 nosy: Gregory.Szorc, docs@python priority: normal severity: normal status: open title: Documentation is unclear how "y*" and "y#" formit units vary type: enhancement versions: Python 3.5 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30625> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23051] multiprocessing.pool methods imap()[_unordered()] deadlock
Gregory Szorc added the comment: For posterity, I think we ran into a similar problem in https://bugzilla.mozilla.org/show_bug.cgi?id=1191877, where our code uses apply_async(): 11:09:47 INFO - Exception in thread Thread-2: 11:09:47 INFO - Traceback (most recent call last): 11:09:47 INFO -File /tools/python/lib/python2.7/threading.py, line 551, in __bootstrap_inner 11:09:47 INFO - self.run() 11:09:47 INFO -File /tools/python/lib/python2.7/threading.py, line 504, in run 11:09:47 INFO - self.__target(*self.__args, **self.__kwargs) 11:09:47 INFO -File /tools/python/lib/python2.7/multiprocessing/pool.py, line 319, in _handle_tasks 11:09:47 INFO - put(task) 11:09:47 INFO - RuntimeError: dictionary changed size during iteration This resulted in deadlock, just like this issue. The added try..except around the iteration of taskseq likely fixes this as well. -- nosy: +Gregory.Szorc ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23051 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23246] distutils fails to locate vcvarsall with Visual C++ Compiler for Python
Gregory Szorc added the comment: Thanks, Steve. The package I was trying to build has its setup.py importing setup from distutils, not setuptools. I'll see about porting them to the future. For posterity, https://bitbucket.org/pypa/setuptools/issue/258, which was first released in setuptools 6.0. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23246 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23246] distutils fails to locate vcvarsall with Visual C++ Compiler for Python
Gregory Szorc added the comment: The first sentence in my original report is ambiguous. I tested with distutils on Python 2.7.9. But considering the code in question hasn't changed since 2011, I think it's safe to say this isn't a regression in CPython. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23246 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23246] distutils fails to locate vcvarsall with Visual C++ Compiler for Python
New submission from Gregory Szorc: distutils as of Python 2.7.9 is unable to locate vcvarsall.bat if Visual C++ Compiler for Python is the only Visual Studio distribution installed. STR: 1) Do a fresh install of Windows + all updates 2) Install Microsoft Visual C++ Compiler for Python from http://www.microsoft.com/en-us/download/details.aspx?id=44266 3) Enter a Visual C++ 2008 command prompt via start menu 4) Attempt to run any |python.exe setup.py| that contains C extensions 5) Unable to find vcvarsall.bat Examining the behavior of MSVC for Python's bat scripts and filesystem layout, it is different enough from full distributions that it confused distutils. First, MSVC for Python doesn't appear to set any meaningful registry entries. So, the registry checking in msvc9compiler fails to find anything. Second, the command environment for MSVC for Python doesn't export VS90COMNTOOLS, so msvc9compiler has no clue where to go looking for files. Third, even if VS90COMNTOOLS is set, msvc9compiler isn't able to find vcvarsall.bat because it is installed in %installdir%/vcvarsall.bat and not %installdir%/VC/vcvarsall.bat, unlike every other Visual Studio AFAICT. Another concern is that distutils.msvc9compiler.find_vcvarsall() first attempts to read from the registry, not the environment. If you are in a MSVC for Python command shell and you also have Visual Studio 2008 installed, distutils will use vcvarsall.bat from the full Visual Studio installation instead of the Python one. I think this is wrong. The MSVC for Python command prompt does have an environment variable that can be used: VCINSTALLDIR. It is set to %installdir%\VC\, which is equivalent to ~/AppData/Local/Programs/Common/Microsoft/Visual C++ for Python/9.0/VC/ if you launch the installer with default options. distutils could be patched to find vcvarsall.bat in %VCINSTALLDIR%\..\vcvarsall.bat Fortunately, a workaround is available: 1) Enter MSVC for Python command prompt 2) SET DISTUTILS_USE_SDK=1 3) SET MSSdk=1 4) python.exe setup.py ... -- components: Distutils messages: 234110 nosy: Gregory.Szorc, dstufft, eric.araujo priority: normal severity: normal status: open title: distutils fails to locate vcvarsall with Visual C++ Compiler for Python versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23246 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
