Ned Deily <n...@python.org> added the comment:
> Regarding the "explicit weak linking" when building on MacOS Big Sur and
> later; wouldn't this mean that a Big Sur build wouldn't work on Catalina?
No, if it is done correctly. I think you are trying to solve the wrong problem
here. As Ronald noted earlier, we now fully support building Python on a newer
version of macOS to run correctly on that version and on older versions (for
current python.org-provided macOS binary installers, we support one build that
runs on macOS 10.9 through 11 Big Sur). The key to this is weak-linking with
the help of Apple-provided availability macros and by properly setting the
MACOSX_DEPLOYMENT_TARGET variable when running ./configure to the oldest
desired supported macOS release.
Because this area is neither well-understood nor well-documented, let me try to
get it written down here at the risk of covering some familiar ground.
To support more than one version of macOS when building Python, there are
basically two approaches: either build on the oldest targeted system and trust
that Apple will continue to provide compatibility when running older binaries
on new systems; or, build on the newest targeted system and dynamically test at
runtime whether specific newer OS features are available and gracefully handle
cases where they are not available (which we call "weak-linking" here for
short).
Prior to Python 3.9.1, we did not support the latter approach, i.e.
weak-linking for many APIs / features added in recent macOS releases. So our
practice and recommendation was to always build on the oldest macOS release to
be supported. That's the approach we took for many years, for example, with
the macOS 64-bit Intel installer variant for 10.9+ systems. Because Apple has
had a very good track record of providing compatibility on newer systems (at
least for the mostly C-based APIs CPython uses), that approached worked
reasonably well. The main drawback was that certain new features added to
Python, primarily in the os module, were not available when using the
python.org installer binaries on newer (post-10.9) systems. That was not ideal
but, for the most part, the missing features weren't commonly used yet and this
was essentially only an issue if you were using the python.org-supplied
binaries; you could always use or build a Python targeted for the system in use.
However, things changed with macOS 11 Big Sur and the removal of system library
files which broke ctype's find_library() when searching for system files, the
subject of this issue. There were a number of other changes needed in CPython
to fully support Big Sur, as documented in Issue41100 and others. As part of
that work, Ronald and Lawrence D'Anna bit the bullet and went through the
interpreter and the standard library to finally properly support weak-linking
for multiple macOS versions. That means, as of 3.9.1 with the introduction of
Big Sur support, it is finally possible to build on newer systems but still
work properly on older ones. For 3.9.1, we introduced a new python.org
installer variant, the "universal2" variant, that provides Intel and Apple
Silicon fat binaries that should work on all Macs that can run macOS 10.9
through at least 11 with newer features conditionally tested at runtime.
So our recommendation has changed as of 3.9.1 to now use the second approach
above (which previously could cause Python segfaults when running on older
systems) and to deprecate and phase out the use of the first approach (which
still works as before - i.e. missing some features - with the notable exception
of find_library() with system libraries on Big Sur). Note that the
find_library() issue is only one reason for that change in recommendation.
How does this work? Here's a quick demo using current head of Python 3.10
(although you should see similar results with Python 3.9.x as of 3.9.1), the
latest versions of macOS 11, 10.15, and 10.9. We'll build on 11 in all cases,
then deploy and run test_ctypes and test_posix on 11, 10.15, and 10.9.
--------------------------------------------
1. Default case, no MACOSX_DEPLOYMENT_TARGET specified. If the deployment
target is not specified, configure normally uses the operating system version
the build is running on, so in this case, 11 Big Sur.
$ sw_vers
ProductName: macOS
ProductVersion: 11.5.1
BuildVersion: 20G80
$ ./configure --prefix=/tmp/py && make -j3 && make install
[...]
checking which MACOSX_DEPLOYMENT_TARGET to use... 11.5
[...]
# run on 11, works as expected
$ /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug 4 2021, 16:46:59) [Clang 12.0.5
(clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 1.86 Run tests sequentially
0:00:00 load avg: 1.86 [1/1] test_ctypes
== Tests result: SUCCESS ==
1 test OK.
Total duration: 762 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 1.87 Run tests sequentially
0:00:00 load avg: 1.87 [1/1] test_posix
== Tests result: SUCCESS ==
1 test OK.
Total duration: 758 ms
Tests result: SUCCESS
# same build, copied to 10.15 system: find_library test fails and test_posix
fails
% sw_vers
ProductName: Mac OS X
ProductVersion: 10.15.7
BuildVersion: 19H1323
sysadmin@pyb15 ~ % /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug 4 2021, 16:46:59) [Clang 12.0.5
(clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 0.97 Run tests sequentially
0:00:00 load avg: 0.97 [1/1] test_ctypes
test_ctypes skipped --
dlopen(/tmp/py/lib/python3.10/lib-dynload/_ctypes.cpython-310-darwin.so, 2):
Symbol not found: __dyld_shared_cache_contains_path
Referenced from:
/tmp/py/lib/python3.10/lib-dynload/_ctypes.cpython-310-darwin.so (which was
built for Mac OS X 11.5)
Expected in: /usr/lib/libSystem.B.dylib
in /tmp/py/lib/python3.10/lib-dynload/_ctypes.cpython-310-darwin.so
test_ctypes skipped
== Tests result: SUCCESS ==
1 test skipped:
test_ctypes
Total duration: 72 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 2.04 Run tests sequentially
0:00:00 load avg: 2.04 [1/1] test_posix
dyld: lazy symbol binding failed: Symbol not found: _preadv
Referenced from: /tmp/py/bin/python3.10 (which was built for Mac OS X 11.5)
Expected in: /usr/lib/libSystem.B.dylib
dyld: Symbol not found: _preadv
Referenced from: /tmp/py/bin/python3.10 (which was built for Mac OS X 11.5)
Expected in: /usr/lib/libSystem.B.dylib
Fatal Python error: Aborted
Current thread 0x000000011373bdc0 (most recent call first):
File "/tmp/py/lib/python3.10/test/test_posix.py", line 302 in test_preadv
File "/tmp/py/lib/python3.10/unittest/case.py", line 549 in _callTestMethod
File "/tmp/py/lib/python3.10/unittest/case.py", line 592 in run
File "/tmp/py/lib/python3.10/unittest/case.py", line 652 in __call__
File "/tmp/py/lib/python3.10/unittest/suite.py", line 122 in run
File "/tmp/py/lib/python3.10/unittest/suite.py", line 84 in __call__
File "/tmp/py/lib/python3.10/unittest/suite.py", line 122 in run
File "/tmp/py/lib/python3.10/unittest/suite.py", line 84 in __call__
File "/tmp/py/lib/python3.10/test/support/testresult.py", line 169 in run
File "/tmp/py/lib/python3.10/test/support/__init__.py", line 971 in _run_suite
File "/tmp/py/lib/python3.10/test/support/__init__.py", line 1096 in
run_unittest
File "/tmp/py/lib/python3.10/test/test_posix.py", line 2176 in test_main
File "/tmp/py/lib/python3.10/test/libregrtest/runtest.py", line 297 in
_runtest_inner2
File "/tmp/py/lib/python3.10/test/libregrtest/runtest.py", line 335 in
_runtest_inner
File "/tmp/py/lib/python3.10/test/libregrtest/runtest.py", line 215 in
_runtest
File "/tmp/py/lib/python3.10/test/libregrtest/runtest.py", line 245 in runtest
File "/tmp/py/lib/python3.10/test/libregrtest/main.py", line 437 in
run_tests_sequential
File "/tmp/py/lib/python3.10/test/libregrtest/main.py", line 535 in run_tests
File "/tmp/py/lib/python3.10/test/libregrtest/main.py", line 708 in _main
File "/tmp/py/lib/python3.10/test/libregrtest/main.py", line 655 in main
File "/tmp/py/lib/python3.10/test/libregrtest/main.py", line 733 in main
File "/tmp/py/lib/python3.10/test/__main__.py", line 2 in <module>
File "/tmp/py/lib/python3.10/runpy.py", line 86 in _run_code
File "/tmp/py/lib/python3.10/runpy.py", line 196 in _run_module_as_main
Extension modules: _testcapi (total: 1)
# same build, copied to 10.9 system: interpreter fails to launch
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.9.5
BuildVersion: 13F1911
$ /tmp/py/bin/python3.10
dyld: lazy symbol binding failed: Symbol not found: _getentropy
Referenced from: /tmp/py/bin/python3.10
Expected in: /usr/lib/libSystem.B.dylib
dyld: Symbol not found: _getentropy
Referenced from: /tmp/py/bin/python3.10
Expected in: /usr/lib/libSystem.B.dylib
Trace/BPT trap: 5
--------------------------------------------
2. Build on Big Sur 11 with MACOSX_DEPLOYMENT_TARGET=10.15
$ ./configure --prefix=/tmp/py MACOSX_DEPLOYMENT_TARGET=10.15
[...]
checking which MACOSX_DEPLOYMENT_TARGET to use... 10.15
[...]
# run on 11, still works as expected
$ /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug 4 2021, 17:00:45) [Clang 12.0.5
(clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 2.99 Run tests sequentially
0:00:00 load avg: 2.99 [1/1] test_ctypes
== Tests result: SUCCESS ==
1 test OK.
Total duration: 736 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 2.99 Run tests sequentially
0:00:00 load avg: 2.99 [1/1] test_posix
== Tests result: SUCCESS ==
1 test OK.
# same build, copied to 10.15 system: test_ctypes and test_posix both now pass
% /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug 4 2021, 17:00:45) [Clang 12.0.5
(clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
% /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 1.29 Run tests sequentially
0:00:00 load avg: 1.29 [1/1] test_ctypes
== Tests result: SUCCESS ==
1 test OK.
Total duration: 908 ms
Tests result: SUCCESS
% /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 1.09 Run tests sequentially
0:00:00 load avg: 1.09 [1/1] test_posix
== Tests result: SUCCESS ==
1 test OK.
Total duration: 812 ms
Tests result: SUCCESS
# same build, copied to 10.9: still fails to launch
$ /tmp/py/bin/python3.10
dyld: lazy symbol binding failed: Symbol not found: _getentropy
Referenced from: /tmp/py/bin/python3.10
Expected in: /usr/lib/libSystem.B.dylib
dyld: Symbol not found: _getentropy
Referenced from: /tmp/py/bin/python3.10
Expected in: /usr/lib/libSystem.B.dylib
Trace/BPT trap: 5
--------------------------------------------
3. Build on Big Sur 11 with MACOSX_DEPLOYMENT_TARGET=10.9
$ ./configure --prefix=/tmp/py MACOSX_DEPLOYMENT_TARGET=10.9
[...]
checking which MACOSX_DEPLOYMENT_TARGET to use... 10.9
[...]
# run on 11, still works as expected
$ /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug 4 2021, 17:07:23) [Clang 12.0.5
(clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 2.31 Run tests sequentially
0:00:00 load avg: 2.31 [1/1] test_ctypes
== Tests result: SUCCESS ==
1 test OK.
Total duration: 646 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 2.12 Run tests sequentially
0:00:00 load avg: 2.12 [1/1] test_posix
== Tests result: SUCCESS ==
1 test OK.
Total duration: 767 ms
Tests result: SUCCESS
# same build run on 10.15, still works as expected
% /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug 4 2021, 17:07:23) [Clang 12.0.5
(clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 1.65 Run tests sequentially
0:00:00 load avg: 1.65 [1/1] test_ctypes
== Tests result: SUCCESS ==
1 test OK.
Total duration: 964 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 1.68 Run tests sequentially
0:00:00 load avg: 1.68 [1/1] test_posix
== Tests result: SUCCESS ==
1 test OK.
Total duration: 795 ms
Tests result: SUCCESS
# same build run on 10.9, now also works as expected
$ /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug 4 2021, 17:07:23) [Clang 12.0.5
(clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 0.68 Run tests sequentially
0:00:00 load avg: 0.68 [1/1] test_ctypes
== Tests result: SUCCESS ==
1 test OK.
Total duration: 398 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 0.62 Run tests sequentially
0:00:00 load avg: 0.62 [1/1] test_posix
== Tests result: SUCCESS ==
1 test OK.
Total duration: 543 ms
Tests result: SUCCESS
--------------------------------------------
To summarize, we believe that building on current systems and targeting older
systems by using MACOS_DEPLOYMENT_TARGET should work properly and, going
forward from 3.9.1, is the recommended and supported method for current
multi-version macOS builds for 3.9.x+. (For the record, note that, while
3.8.10, the final maintenance release of the 3.8 series, did gain support for
running natively on macOS 11 when built on macOS 11, the much more invasive
weak-linking support was not backported. 3.8.x is now in the security-fix-only
phase of its life cycle.)
So, what are the remaining issues? I believe them to be and in rough priority
order:
1. Better document how to build Python on macOS for multiple releases.
Mac/README.txt was updated to include a section on weak linking support but it
should be expanded and should give more concrete examples along the lines of
above. We should do that for the next releases: 3.9.7 and 3.10.0. We should
also re-iterate that building Python for multiple macOS versions and/or
architectures is only supported when building on macOS using the Apple-supplied
build tools (I will make sure this gets done.)
2. Make the universal2 10.9+ python.org installer variant the default download
for 3.9.7 and mark the Intel-only 10.9+ variant as deprecated for 3.9.7 and
drop in a future 3.9.x maintenance release. (I will also make sure this gets
done.) With 3.10.0 pre-releases, we are already only providing the 10.9+
universal2 variant.
3. Consider accepting a PR to support find_library() on macOS 11+ in the legacy
and deprecated case of building on older versions of macOS to also run on newer
versions. The current revision of PR 27251 is still problematic; as Ronald
noted before, "the current code path using explicit weak linking should be kept
for those building using a recent SDK (cleaner code, better error reporting)."
And the description is misleading as demonstrated above.
Opinions?
BTW, thanks again for all your work so far on this issue, Tobias. Because of
it, I think we will end up with some major improvements for both builders and
users of Python.
https://stackoverflow.com/questions/25352389/what-is-the-difference-between-macosx-deployment-target-and-mmacosx-version-min
----------
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue44689>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
