[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems
Tobias Bergkvist added the comment: I made a typo (forgetting the -D): clang -Wno-unused-result -Wsign-compare -g -O0 -Wall -arch x86_64 -mmacosx-version-min=10.9 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -fPIC -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/ncursesw -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/uuid -Werror=unguarded-availability-new -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I. -I./Include -arch x86_64 -mmacosx-version-min=10.9 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/S DKs/MacOSX11.1.sdk -fPIC -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/ncursesw -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/uuid -Werror=unguarded-availability-new -DMACOSX -DUSING_MALLOC_CLOSURE_DOT_C=1 -DHAVE_FFI_PREP_CIF_VAR=1 -DHAVE_FFI_PREP_CLOSURE_LOC=1 -DHAVE_FFI_CLOSURE_ALLOC=1 -DHAVE_DYLD_SHARED_CACHE_CONTAINS_PATH=1 -DPy_BUILD_CORE_BUILTIN -I_ctypes/darwin -c ./Modules/_ctypes/callproc.c -o Modules/callproc.o -- ___ Python tracker <https://bugs.python.org/issue44689> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems
Tobias Bergkvist added the comment: It seems like _dyld_shared_cache_contains_path exists, while the define flag HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH has not been set. Essentially, the define-flags being used does not seem to agree with the SDK version you are using. Could you try adding HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH like this and see if it builds correctly? clang -Wno-unused-result -Wsign-compare -g -O0 -Wall -arch x86_64 -mmacosx-version-min=10.9 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -fPIC -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/ncursesw -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/uuid -Werror=unguarded-availability-new -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I. -I./Include -arch x86_64 -mmacosx-version-min=10.9 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/S DKs/MacOSX11.1.sdk -fPIC -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/ncursesw -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/uuid -Werror=unguarded-availability-new -DMACOSX -DUSING_MALLOC_CLOSURE_DOT_C=1 -DHAVE_FFI_PREP_CIF_VAR=1 -DHAVE_FFI_PREP_CLOSURE_LOC=1 -DHAVE_FFI_CLOSURE_ALLOC=1 -HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH=1 -DPy_BUILD_CORE_BUILTIN -I_ctypes/darwin -c ./Modules/_ctypes/callproc.c -o Modules/callproc.o -- ___ Python tracker <https://bugs.python.org/issue44689> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur
Tobias Bergkvist added the comment: I realised that I needed to define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME in the source file myself - as this is not defined after running the configure-script. I've updated the PR and its description to only focus on the legacy/deprecated approach on building on Catalina and running on Big Sur. Now a dynamic loading version of _dyld_shared_cache_contains_path is only used when compiling on MacOS < 11 (Catalina or older). And the weak-linking approach is used when HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH is defined (MacOS >= 11). -- ___ Python tracker <https://bugs.python.org/issue44689> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur
Tobias Bergkvist added the comment:
This makes a lot of sense now.
Thank you so much for the thorough explanation Ned - and for highlighting where
my assumptions were wrong!
I didn’t realise I needed to specify MACOSX_DEPLOYMENT_TARGET to enable
backwards compatibility. Is there a reason for not setting this to as old a
version as possible by default when running ./configure? (Bigger binaries? Or
something else?)
If I understand correctly, the following two statements are now equivalent (in
Python >= 3.9, but not in Python == 3.8), after the added Python-support for
weak linking you mention:
```
if (__builtin_available(macOS 11.0, iOS 14.0, tvOS 14.0, watchOS 7.0, *)) { //
...
If (HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME) { // ...
```
From:
https://github.com/python/cpython/blob/3d315c311676888201f4a3576e4ee3698684a3a2/Modules/_ctypes/callproc.c#L1452
And in order to support the deprecated case of building on an older
MacOS-version, I would want to do something like the following:
```
#ifdef __APPLE__
#ifdef HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH
// leave current implementation as is
#else
// dynamic loading implementation
#endif
#endif
```
That way, dynamic loading will only be used when building on old MacOS, and we
keep the current code path using weak linking when possible.
This means we will still get useful compiler errors if for example Apple
decides to suddenly remove the _dyld_shared_cache_contains_path symbol again in
the future, or change its signature - rather than having to wait for the test
suite to fail. It makes sense to prioritise good error reporting for the latest
version of MacOS, since this will reduce the debugging time for CPython
developers whenever Apple introduces new breaking changes to MacOS.
--
___
Python tracker
<https://bugs.python.org/issue44689>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur
Tobias Bergkvist added the comment:
Okay, I decided to look into how I could do dynamic loading as you suggested.
Here is a POC (executable) for using _dyld_shared_cache_contains_path when
available:
```
#include
#include
void* libsystemb_handle;
typedef bool (*_dyld_shared_cache_contains_path_f)(const char* path);
_dyld_shared_cache_contains_path_f _dyld_shared_cache_contains_path;
bool _dyld_shared_cache_contains_path_fallback(const char* name) {
return false;
}
__attribute__((constructor)) void load_libsystemb(void) {
if (
(libsystemb_handle = dlopen("/usr/lib/libSystem.B.dylib", RTLD_LAZY))
== NULL ||
(_dyld_shared_cache_contains_path = dlsym(libsystemb_handle,
"_dyld_shared_cache_contains_path")) == NULL
) {
_dyld_shared_cache_contains_path =
_dyld_shared_cache_contains_path_fallback;
}
}
__attribute__((destructor)) void unload_libsystemb(void) {
if (libsystemb_handle != NULL) {
dlclose(libsystemb_handle);
}
}
int main(int argc, char ** argv) {
printf("Library exists in cache: %d\n",
_dyld_shared_cache_contains_path(argv[1]));
}
```
A fallback function is used when _dyld_shared_cache_contains_path cannot be
loaded, which always returns false. If there is no cache - the (nonexistent)
cache also does not contain whatever path you pass it.
The constructor function is called when the Python extension is loaded -
ensuring that _dyld_shared_cache_contains_path is defined and callable. I've
read that C extension modules cannot be autoreloaded
(https://ipython.org/ipython-doc/3/config/extensions/autoreload.html) - so this
might mean there is no need for a deconstructor? Instead the OS would handle
cleanup once the process exits?
This could be compiled on either MacOS Catalina or Big Sur, and run without
problems on the other MacOS version.
Regarding the "explicit weak linking" when building on MacOS Big Sur and later;
wouldn't this mean that a Big Sur build wouldn't work on Catalina?
Would you be positive towards a PR with the approach I demonstrated here?
--
___
Python tracker
<https://bugs.python.org/issue44689>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur
Tobias Bergkvist added the comment:
You are absolutely right! From the manpage of dlopen(3) on MacOS Big Sur:
> dlopen() examines the mach-o file specified by path. If the file is
> compatible with the current process and has not already been loaded into the
> current process, it is loaded and linked. After being linked, if it contains
> any initializer functions, they are called, before dlopen() returns. dlopen()
> can load dynamic libraries and bundles. It returns a handle that can be used
> with dlsym() and dlclose(). A second call to dlopen() with the same path will
> return the same handle, but the internal reference count for the handle will
> be incremented. Therefore all dlopen() calls should be balanced with a
> dlclose() call.
Essentially, if the shared library contains initializer functions that have
some kind of side-effects, dlopen will also trigger these side-effects.
Basic example:
```
// mylib.c
#include
void myinit(void) {
printf("Hello from mylib\n");
}
__attribute__((section("__DATA,__mod_init_func"))) typeof(myinit) *__init =
myinit;
```
---
```
// main.c
#include
#include
int main(void) {
void* handle = dlopen("./mylib.dyld", RTLD_LAZY);
if (handle == NULL) printf("Failed to load");
}
```
$ clang mylib.c -shared -o mylib.dyld
$ clang main.c -o main
$ ./main
Hello from mylib
--
___
Python tracker
<https://bugs.python.org/issue44689>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur
Tobias Bergkvist added the comment: An alternative to using _dyld_shared_cache_contains_path is to use dlopen to check for library existence (which is what Apple recommends in their change notes: https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11_0_1-release-notes). > New in macOS Big Sur 11.0.1, the system ships with a built-in dynamic linker > cache of all system-provided libraries. As part of this change, copies of > dynamic libraries are no longer present on the filesystem. Code that attempts > to check for dynamic library presence by looking for a file at a path or > enumerating a directory will fail. Instead, check for library presence by > attempting to dlopen() the path, which will correctly check for the library > in the cache. (62986286) I have created a PR which modifies the current find_library from using _dyld_shared_cache_contains_path to dlopen. It passes all of the existing find_library-tests: https://github.com/python/cpython/pull/27251 There might be downsides to using dlopen (performance?) or something else I haven't considered. The huge upside however, is that the function is basically available on all Unix-systems. -- ___ Python tracker <https://bugs.python.org/issue44689> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur
New submission from Tobias Bergkvist :
Python-binaries compiled on either Big Sur or Catalina - and moved to the other
MacOS-version will not work as expected when code depends on
ctypes.util.find_library.
Example symptom of this issue:
https://github.com/jupyterlab/jupyterlab/issues/9863
I have personally faced this when using Python from nixpkgs - since nixpkgs
Python has been built on Catalina - and I'm using Big Sur.
Scenario 1: Compile on Catalina, copy binaries to BigSur, and call
ctypes.util.find_library('c')
Python 3.11.0a0 (heads/main:635bfe8162, Jul 19 2021, 08:09:05) [Clang 12.0.0
(clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library; print(find_library('c'))
None
Scenario 2: Compile on Big Sur, copy binaries to Catalina, and call
ctypes.util.find_library('c'):
Python 3.11.0a0 (heads/main:635bfe8162, Jul 19 2021, 08:28:48) [Clang 12.0.5
(clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library; print(find_library('c'))
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.11/ctypes/__init__.py", line 8, in
from _ctypes import Union, Structure, Array
^^^
ImportError:
dlopen(/usr/local/lib/python3.11/lib-dynload/_ctypes.cpython-311-darwin.so, 2):
Symbol not found: __dyld_shared_cache_contains_path
Referenced from:
/usr/local/lib/python3.11/lib-dynload/_ctypes.cpython-311-darwin.so (which was
built for Mac OS X 11.4)
Expected in: /usr/lib/libSystem.B.dylib
in /usr/local/lib/python3.11/lib-dynload/_ctypes.cpython-311-darwin.so
--
components: ctypes
messages: 397916
nosy: bergkvist
priority: normal
pull_requests: 25816
severity: normal
status: open
title: MacOS: Python binaries not portable between Catalina and Big Sur
type: behavior
versions: Python 3.10, Python 3.11, Python 3.8, Python 3.9
___
Python tracker
<https://bugs.python.org/issue44689>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
