[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

[miss-islington]

Change by miss-islington :


--
pull_requests: +26498
pull_request: https://github.com/python/cpython/pull/28054

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

[miss-islington]

Change by miss-islington :


--
pull_requests: +26497
pull_request: https://github.com/python/cpython/pull/28053

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

[miss-islington]

Change by miss-islington :


--
keywords: +patch
nosy: +miss-islington
nosy_count: 3.0 -> 4.0
pull_requests: +26496
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/28052

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

[Ned Deily]

Ned Deily  added the comment:


New changeset 71853a73024a98aa38a3c0444fe364dbd9709134 by Tobias Bergkvist in 
branch 'main':
bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when 
built on older macOS systems (#27251)
https://github.com/python/cpython/commit/71853a73024a98aa38a3c0444fe364dbd9709134


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

[Tobias Bergkvist]

Tobias Bergkvist  added the comment:

I realised that I needed to define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME 
in the source file myself - as this is not defined after running the 
configure-script. I've updated the PR and its description to only focus on the 
legacy/deprecated approach on building on Catalina and running on Big Sur.

Now a dynamic loading version of _dyld_shared_cache_contains_path is only used 
when compiling on MacOS < 11 (Catalina or older). And the weak-linking approach 
is used when HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH is defined (MacOS >= 11).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

[Tobias Bergkvist]

Tobias Bergkvist  added the comment:

This makes a lot of sense now.

Thank you so much for the thorough explanation Ned - and for highlighting where 
my assumptions were wrong!

I didn’t realise I needed to specify MACOSX_DEPLOYMENT_TARGET to enable 
backwards compatibility. Is there a reason for not setting this to as old a 
version as possible by default when running ./configure? (Bigger binaries? Or 
something else?)

If I understand correctly, the following two statements are now equivalent (in 
Python >= 3.9, but not in Python == 3.8), after the added Python-support for 
weak linking you mention:

```
if (__builtin_available(macOS 11.0, iOS 14.0, tvOS 14.0, watchOS 7.0, *)) { // 
...
If (HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME) { // ...
```
From: 
https://github.com/python/cpython/blob/3d315c311676888201f4a3576e4ee3698684a3a2/Modules/_ctypes/callproc.c#L1452


And in order to support the deprecated case of building on an older 
MacOS-version, I would want to do something like the following:

```
#ifdef __APPLE__
#ifdef HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH
// leave current implementation as is
#else
// dynamic loading implementation
#endif
#endif
```

That way, dynamic loading will only be used when building on old MacOS, and we 
keep the current code path using weak linking when possible.

This means we will still get useful compiler errors if for example Apple 
decides to suddenly remove the _dyld_shared_cache_contains_path symbol again in 
the future, or change its signature - rather than having to wait for the test 
suite to fail. It makes sense to prioritise good error reporting for the latest 
version of MacOS, since this will reduce the debugging time for CPython 
developers whenever Apple introduces new breaking changes to MacOS.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

[Ned Deily]

Ned Deily  added the comment:

> Regarding the "explicit weak linking" when building on MacOS Big Sur and 
> later; wouldn't this mean that a Big Sur build wouldn't work on Catalina?

No, if it is done correctly.  I think you are trying to solve the wrong problem 
here.  As Ronald noted earlier, we now fully support building Python on a newer 
version of macOS to run correctly on that version and on older versions (for 
current python.org-provided macOS binary installers, we support one build that 
runs on macOS 10.9 through 11 Big Sur). The key to this is weak-linking with 
the help of Apple-provided availability macros and by properly setting the 
MACOSX_DEPLOYMENT_TARGET variable when running ./configure to the oldest 
desired supported macOS release.

Because this area is neither well-understood nor well-documented, let me try to 
get it written down here at the risk of covering some familiar ground.

To support more than one version of macOS when building Python, there are 
basically two approaches: either build on the oldest targeted system and trust 
that Apple will continue to provide compatibility when running older binaries 
on new systems; or, build on the newest targeted system and dynamically test at 
runtime whether specific newer OS features are available and gracefully handle 
cases where they are not available (which we call "weak-linking" here for 
short).

Prior to Python 3.9.1, we did not support the latter approach, i.e. 
weak-linking for many APIs / features added in recent macOS releases.  So our 
practice and recommendation was to always build on the oldest macOS release to 
be supported.  That's the approach we took for many years, for example, with 
the macOS 64-bit Intel installer variant for 10.9+ systems.  Because Apple has 
had a very good track record of providing compatibility on newer systems (at 
least for the mostly C-based APIs CPython uses), that approached worked 
reasonably well. The main drawback was that certain new features added to 
Python, primarily in the os module, were not available when using the 
python.org installer binaries on newer (post-10.9) systems. That was not ideal 
but, for the most part, the missing features weren't commonly used yet and this 
was essentially only an issue if you were using the python.org-supplied 
binaries; you could always use or build a Python targeted for the system in use.

However, things changed with macOS 11 Big Sur and the removal of system library 
files which broke ctype's find_library() when searching for system files, the 
subject of this issue. There were a number of other changes needed in CPython 
to fully support Big Sur, as documented in Issue41100 and others. As part of 
that work, Ronald and Lawrence D'Anna bit the bullet and went through the 
interpreter and the standard library to finally properly support weak-linking 
for multiple macOS versions. That means, as of 3.9.1 with the introduction of 
Big Sur support, it is finally possible to build on newer systems but still 
work properly on older ones. For 3.9.1, we introduced a new python.org 
installer variant, the "universal2" variant, that provides Intel and Apple 
Silicon fat binaries that should work on all Macs that can run macOS 10.9 
through at least 11 with newer features conditionally tested at runtime.

So our recommendation has changed as of 3.9.1 to now use the second approach 
above (which previously could cause Python segfaults when running on older 
systems) and to deprecate and phase out the use of the first approach (which 
still works as before - i.e. missing some features - with the notable exception 
of find_library() with system libraries on Big Sur).  Note that the 
find_library() issue is only one reason for that change in recommendation.

How does this work? Here's a quick demo using current head of Python 3.10 
(although you should see similar results with Python 3.9.x as of 3.9.1), the 
latest versions of macOS 11, 10.15, and 10.9.  We'll build on 11 in all cases, 
then deploy and run test_ctypes and test_posix on 11, 10.15, and 10.9.


1. Default case, no MACOSX_DEPLOYMENT_TARGET specified.  If the deployment 
target is not specified, configure normally uses the operating system version 
the build is running on, so in this case, 11 Big Sur.

$ sw_vers
ProductName:macOS
ProductVersion: 11.5.1
BuildVersion:   20G80
$ ./configure --prefix=/tmp/py && make -j3 && make install
[...]
checking which MACOSX_DEPLOYMENT_TARGET to use... 11.5
[...]

# run on 11, works as expected
$ /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug  4 2021, 16:46:59) [Clang 12.0.5 
(clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 1.86 Run tests sequentially
0:00:00 load avg: 1.86 [1/1] test_ctypes

== Tests result: SUCCESS ==

1 test OK.

Total duration: 762 ms
Tests 

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

[Tobias Bergkvist]

Tobias Bergkvist  added the comment:

Okay, I decided to look into how I could do dynamic loading as you suggested.

Here is a POC (executable) for using _dyld_shared_cache_contains_path when 
available:

```
#include 
#include 

void* libsystemb_handle;
typedef bool (*_dyld_shared_cache_contains_path_f)(const char* path);
_dyld_shared_cache_contains_path_f _dyld_shared_cache_contains_path;

bool _dyld_shared_cache_contains_path_fallback(const char* name) {
return false;
}

__attribute__((constructor)) void load_libsystemb(void) {
if (
(libsystemb_handle = dlopen("/usr/lib/libSystem.B.dylib", RTLD_LAZY)) 
== NULL ||
(_dyld_shared_cache_contains_path = dlsym(libsystemb_handle, 
"_dyld_shared_cache_contains_path")) == NULL
) {
_dyld_shared_cache_contains_path = 
_dyld_shared_cache_contains_path_fallback;
}
}

__attribute__((destructor)) void unload_libsystemb(void) {
if (libsystemb_handle != NULL) {
dlclose(libsystemb_handle);
}
}

int main(int argc, char ** argv) {
printf("Library exists in cache: %d\n", 
_dyld_shared_cache_contains_path(argv[1]));
}
```

A fallback function is used when _dyld_shared_cache_contains_path cannot be 
loaded, which always returns false. If there is no cache - the (nonexistent) 
cache also does not contain whatever path you pass it.

The constructor function is called when the Python extension is loaded - 
ensuring that _dyld_shared_cache_contains_path is defined and callable. I've 
read that C extension modules cannot be autoreloaded 
(https://ipython.org/ipython-doc/3/config/extensions/autoreload.html) - so this 
might mean there is no need for a deconstructor? Instead the OS would handle 
cleanup once the process exits?

This could be compiled on either MacOS Catalina or Big Sur, and run without 
problems on the other MacOS version.

Regarding the "explicit weak linking" when building on MacOS Big Sur and later; 
wouldn't this mean that a Big Sur build wouldn't work on Catalina?

Would you be positive towards a PR with the approach I demonstrated here?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

[Tobias Bergkvist]

Tobias Bergkvist  added the comment:

You are absolutely right! From the manpage of dlopen(3) on MacOS Big Sur:

> dlopen() examines the mach-o file specified by path. If the file is 
> compatible with the current process and has not already been loaded into the 
> current process, it is loaded and linked.  After being linked, if it contains 
> any initializer functions, they are called, before dlopen() returns. dlopen() 
> can load dynamic libraries and bundles.  It returns a handle that can be used 
> with dlsym() and dlclose(). A second call to dlopen() with the same path will 
> return the same handle, but the internal reference count for the handle will 
> be incremented. Therefore all dlopen() calls should be balanced with a 
> dlclose() call.

Essentially, if the shared library contains initializer functions that have 
some kind of side-effects, dlopen will also trigger these side-effects.

Basic example:
```
// mylib.c
#include 
void myinit(void) {
printf("Hello from mylib\n");
}
__attribute__((section("__DATA,__mod_init_func"))) typeof(myinit) *__init = 
myinit;
```

---
```
// main.c
#include 
#include 
int main(void) {
void* handle = dlopen("./mylib.dyld", RTLD_LAZY);
if (handle == NULL) printf("Failed to load"); 
}
```

$ clang mylib.c -shared -o mylib.dyld
$ clang main.c -o main
$ ./main
Hello from mylib

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

[Ronald Oussoren]

Ronald Oussoren  added the comment:

The disadvantage of using dlopen is that this function has side effects, and 
those can affect program behaviour.  Because of this I'm against switching to 
using dlopen to probe for libraries.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

[Tobias Bergkvist]

Tobias Bergkvist  added the comment:

An alternative to using _dyld_shared_cache_contains_path is to use dlopen to 
check for library existence (which is what Apple recommends in their change 
notes: 
https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11_0_1-release-notes).

> New in macOS Big Sur 11.0.1, the system ships with a built-in dynamic linker 
> cache of all system-provided libraries. As part of this change, copies of 
> dynamic libraries are no longer present on the filesystem. Code that attempts 
> to check for dynamic library presence by looking for a file at a path or 
> enumerating a directory will fail. Instead, check for library presence by 
> attempting to dlopen() the path, which will correctly check for the library 
> in the cache. (62986286)

I have created a PR which modifies the current find_library from using 
_dyld_shared_cache_contains_path to dlopen. It passes all of the existing 
find_library-tests:
https://github.com/python/cpython/pull/27251

There might be downsides to using dlopen (performance?) or something else I 
haven't considered. The huge upside however, is that the function is basically 
available on all Unix-systems.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

[Ronald Oussoren]

Ronald Oussoren  added the comment:

Anyways, the solution for "build on older macOS version, deploy to Big Sur" is 
to dynamically look for the relevant API (``_dyld_shared_cache_contains_path``) 
and use it when available. But only in that scenario, the current code path 
using explicit weak linking should be kept for those building using a recent 
SDK (cleaner code, better error reporting).

This should be a fairly easy patch, but I don't know when I'll get around to 
looking into this further.

Alternatively we could require that Python is build using the macOS 11 SDK (or 
later) when targeting Big Sur.

I'm dropping 3.8 from the list of versions because it is in "bug fix only" mode 
and won't receive a patch for this.  IIRC 3.8 also doesn't support Big Sur in 
the first place, we've only back ported Big Sur support to 3.9.

--
stage:  -> needs patch
versions:  -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

[Ronald Oussoren]

Ronald Oussoren  added the comment:

The problem with moving from Catalina to Big Sur is a known issue, AFAIK 
there's an open issue for this.

The problem is that Big Sur moved system libraries into a big blob (which Apple 
calls the shared library cache). Ctypes uses an API that's new in macOS 11 to 
check if a library is in that cache, but only when compiled with the the macOS 
11 SDK or later as the API is not available in earlier SDKs.

Moving from Big Sur to earlier version should work fine, but only if you set 
the deployment target correctly during the build. This is how the "universal2" 
installers on python.org are build.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

[Zachary Ware]

Change by Zachary Ware :


--
components: +macOS
nosy: +ned.deily, ronaldoussoren

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] MacOS: Python binaries not portable between Catalina and Big Sur

[Tobias Bergkvist]

New submission from Tobias Bergkvist :

Python-binaries compiled on either Big Sur or Catalina - and moved to the other 
MacOS-version will not work as expected when code depends on 
ctypes.util.find_library.

Example symptom of this issue: 
https://github.com/jupyterlab/jupyterlab/issues/9863
I have personally faced this when using Python from nixpkgs - since nixpkgs 
Python has been built on Catalina - and I'm using Big Sur.


Scenario 1: Compile on Catalina, copy binaries to BigSur, and call 
ctypes.util.find_library('c')
Python 3.11.0a0 (heads/main:635bfe8162, Jul 19 2021, 08:09:05) [Clang 12.0.0 
(clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library; print(find_library('c'))
None

Scenario 2: Compile on Big Sur, copy binaries to Catalina, and call 
ctypes.util.find_library('c'):
Python 3.11.0a0 (heads/main:635bfe8162, Jul 19 2021, 08:28:48) [Clang 12.0.5 
(clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library; print(find_library('c'))
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python3.11/ctypes/__init__.py", line 8, in 
from _ctypes import Union, Structure, Array
^^^
ImportError: 
dlopen(/usr/local/lib/python3.11/lib-dynload/_ctypes.cpython-311-darwin.so, 2): 
Symbol not found: __dyld_shared_cache_contains_path
  Referenced from: 
/usr/local/lib/python3.11/lib-dynload/_ctypes.cpython-311-darwin.so (which was 
built for Mac OS X 11.4)
  Expected in: /usr/lib/libSystem.B.dylib
 in /usr/local/lib/python3.11/lib-dynload/_ctypes.cpython-311-darwin.so

--
components: ctypes
messages: 397916
nosy: bergkvist
priority: normal
pull_requests: 25816
severity: normal
status: open
title: MacOS: Python binaries not portable between Catalina and Big Sur
type: behavior
versions: Python 3.10, Python 3.11, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com