[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems

[Ned Deily]

Ned Deily  added the comment:

Thanks for the updates!

>If the core developers say various cross-compiling scenarios aren't supported 
>build configuration any more, I'll accept that and work around the limitation 
>on my end. However, do note there is a significant possibility that Apple 
>stops selling Intel machines for various product models in a few weeks. I 
>suspect various people will want to continue their practice of building x86_64 
>only binaries for the indefinite future.

I think you make very good points. I also think that we all agree that we do 
need to work towards better supporting the coming primarily non-Intel Mac 
world. I would rather put it that there are cross-compiling scenarios that were 
never supported before (but also not necessarily fully documented as such) but 
may have worked coincidentally some of the time; moving forward, we need to 
identify the important scenarios and make sure we fully support them. We're not 
there yet but I have some ideas on how to do that which I will try to get 
written down soon.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems

[Gregory Szorc]

Gregory Szorc  added the comment:

I cannot reproduce this on an 11.5 Intel MacBook Pro using an 11.5 SDK 
targeting x86_64 10.9.

However, I can reproduce on a similarly configured M1 using the same OS and SDK 
but cross-compiling to single arch x86_64.

The issue here may reside in how configure handles cross-compiling for Apple. 
The current logic in configure is a bit opinionated about how things should 
work and this patch may have tickled things into not working.

My opinion is that configure should support various cross-compiling scenarios 
like this (including building non-fat x86_64 only binaries from an arm64 
machine). However, it is clear from the implementation and comments in this 
issue and elsewhere that the CPython core devs want to limit which exact 
cross-compiling configurations are supported on Apple.

If the core developers say various cross-compiling scenarios aren't supported 
build configuration any more, I'll accept that and work around the limitation 
on my end. However, do note there is a significant possibility that Apple stops 
selling Intel machines for various product models in a few weeks. I suspect 
various people will want to continue their practice of building x86_64 only 
binaries for the indefinite future. Regardless of when the ARM only transition 
occurs, arbitrary restrictions like not supporting Apple arm64 -> x86_64 
cross-compiling will disappoint a growing cohort of users over time. I would 
encourage investing in less opinionated configure logic to support Apple 
cross-compiling. I could potentially contribute patches in this area, since 
I've already taught python-build-standalone to cross-compile more robustly 
(including to targets like iOS).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems

[Gregory Szorc]

Gregory Szorc  added the comment:

> On Sep 1, 2021, at 22:58, Ned Deily  wrote:
> 
> I don't think we have ever claimed to support building on an older system 
> with a newer SDK, as in building on 10.15 with an 11 SDK.

My initial report was from a 10.15 Intel machine in GitHub Actions. My comment 
a few hours ago was from an 11.3 M1 cross-compiling to x86_64 and targeting 
10.9 from a 11.X SDK.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems

[Ned Deily]

Ned Deily  added the comment:

I don't think we have ever claimed to support building on an older system with 
a newer SDK, as in building on 10.15 with an 11 SDK. I am sure there have been 
problems with trying to do this in the past for some releases.  It *may* work 
but there are no guarantees. If you want to build on an older system you should 
use the SDK for that system even if a newer version of Xcode / Command Line 
Tools is released that provides the newer SDK.  That said, I have no objection 
to trying to fix this issue but I think we should avoid claiming to support 
such configurations and I don't see this issue as needing to make a new 
release. I am willing to be convinced otherwise :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems

[Gregory Szorc]

Gregory Szorc  added the comment:

I spoke too soon: you can reproduce this with CPython's build system and I 
think this is a legit regression.

I think the function sniffing logic is subtly wrong.

Here is the logic as written:

#ifdef __APPLE__
  #ifdef HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH
#define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME \
  __builtin_available(macOS 11.0, iOS 14.0, tvOS 14.0, watchOS 7.0, *)
  #else
static bool (*_dyld_shared_cache_contains_path)(const char *path);

  #define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME \
  _dyld_shared_cache_contains_path != NULL
  #endif
#endif

The fundamental problem is that HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH comes from 
configure. Configure is using the target settings to probe for function 
presence. If I set the deployment target to 10.9, it says `checking for 
_dyld_shared_cache_contains_path... no`. But if I set the deployment target to 
11.0, it says `checking for _dyld_shared_cache_contains_path... yes`. Because 
we may be targeting a macOS without the function, the function appears as not 
available to configure. It may exist in the SDK, but it is masked behind 
availability checks for the test compile.

The logic as written assumes that !HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH means 
the SDK doesn't contain the function at all. This just isn't true for SDKs >= 
11.0.

If you look at posixmodule.c, the logic for checking for function availability 
is typically this pattern:

#ifdef __APPLE__
  #ifdef HAVE_BUILTIN_AVAILABLE
#define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME \
  __builtin_available(macOS 11.0, iOS 14.0, tvOS 14.0, watchOS 7.0, *)
  #else
#ifdef HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH 
  #define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME 
(_dyld_shared_cache_contains_path != NULL)
#endif
  #endif
#endif

This other pattern also reveals another regression with this patch: 
__builtin_available() may not be available on older compilers. See issue42692. 
I'm unsure what "older compilers" actually is. But someone is bound to complain 
about this (if they haven't already).

Do these build regressions warrant an unplanned 3.9.8 release?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems

[Gregory Szorc]

Gregory Szorc  added the comment:

Oh, this might be my custom Modules file not passing the 
HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH define properly. I suspect I was working 
around this bug before by disabling the dyld bits. Sorry for the noise! (I 
should have reproduced with CPython's build system.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems

[Tobias Bergkvist]

Tobias Bergkvist  added the comment:

I made a typo (forgetting the -D):

clang -Wno-unused-result -Wsign-compare -g -O0 -Wall -arch x86_64 
-mmacosx-version-min=10.9 -Wno-nullability-completeness 
-Wno-expansion-to-defined -Wno-undef-prefix -isysroot 
/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
 -fPIC 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include
 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/ncursesw
 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/uuid
 -Werror=unguarded-availability-new   -std=c99 -Wextra -Wno-unused-result 
-Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes 
-Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal 
 -I. -I./Include -arch x86_64 -mmacosx-version-min=10.9 
-Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix 
-isysroot 
/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/S
 DKs/MacOSX11.1.sdk -fPIC 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include
 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/ncursesw
 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/uuid
 -Werror=unguarded-availability-new   -DMACOSX -DUSING_MALLOC_CLOSURE_DOT_C=1 
-DHAVE_FFI_PREP_CIF_VAR=1 -DHAVE_FFI_PREP_CLOSURE_LOC=1 
-DHAVE_FFI_CLOSURE_ALLOC=1 -DHAVE_DYLD_SHARED_CACHE_CONTAINS_PATH=1 
-DPy_BUILD_CORE_BUILTIN  -I_ctypes/darwin -c ./Modules/_ctypes/callproc.c -o 
Modules/callproc.o

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems

[Tobias Bergkvist]

Tobias Bergkvist  added the comment:

It seems like _dyld_shared_cache_contains_path exists, while the define flag 
HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH has not been set.

Essentially, the define-flags being used does not seem to agree with the SDK 
version you are using.

Could you try adding HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH like this and see if 
it builds correctly?

clang -Wno-unused-result -Wsign-compare -g -O0 -Wall -arch x86_64 
-mmacosx-version-min=10.9 -Wno-nullability-completeness 
-Wno-expansion-to-defined -Wno-undef-prefix -isysroot 
/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
 -fPIC 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include
 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/ncursesw
 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/uuid
 -Werror=unguarded-availability-new   -std=c99 -Wextra -Wno-unused-result 
-Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes 
-Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal 
 -I. -I./Include -arch x86_64 -mmacosx-version-min=10.9 
-Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix 
-isysroot 
/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/S
 DKs/MacOSX11.1.sdk -fPIC 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include
 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/ncursesw
 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/uuid
 -Werror=unguarded-availability-new   -DMACOSX -DUSING_MALLOC_CLOSURE_DOT_C=1 
-DHAVE_FFI_PREP_CIF_VAR=1 -DHAVE_FFI_PREP_CLOSURE_LOC=1 
-DHAVE_FFI_CLOSURE_ALLOC=1 -HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH=1 
-DPy_BUILD_CORE_BUILTIN  -I_ctypes/darwin -c ./Modules/_ctypes/callproc.c -o 
Modules/callproc.o

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems

[Gregory Szorc]

Gregory Szorc  added the comment:

I think patch broke building on macOS < 11 when building with a 11.0+ SDK and 
targeting macOS < 11? This build configuration previously used to work with 
3.9.6.

clang -Wno-unused-result -Wsign-compare -g -O0 -Wall -arch x86_64 
-mmacosx-version-min=10.9 -Wno-nullability-completeness 
-Wno-expansion-to-defined -Wno-undef-prefix -isysroot 
/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
 -fPIC 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include
 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/ncursesw
 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/uuid
 -Werror=unguarded-availability-new   -std=c99 -Wextra -Wno-unused-result 
-Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes 
-Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal 
 -I. -I./Include -arch x86_64 -mmacosx-version-min=10.9 
-Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix 
-isysroot 
/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/S
 DKs/MacOSX11.1.sdk -fPIC 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include
 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/ncursesw
 
-I/var/folders/24/8k48jl6d249_n_qfxwsl6xvmgn/T/tmpkfji88v7/tools/deps/include/uuid
 -Werror=unguarded-availability-new   -DMACOSX -DUSING_MALLOC_CLOSURE_DOT_C=1 
-DHAVE_FFI_PREP_CIF_VAR=1 -DHAVE_FFI_PREP_CLOSURE_LOC=1 
-DHAVE_FFI_CLOSURE_ALLOC=1 -DPy_BUILD_CORE_BUILTIN  -I_ctypes/darwin -c 
./Modules/_ctypes/callproc.c -o Modules/callproc.o
python-3.9> ./Modules/_ctypes/callproc.c:1459:15: error: redefinition of 
'_dyld_shared_cache_contains_path' as different kind of symbol
cpython-3.9> static bool (*_dyld_shared_cache_contains_path)(const char *path);
cpython-3.9>   ^
cpython-3.9> 
/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/include/mach-o/dyld.h:121:13:
 note: previous definition is here
cpython-3.9> extern bool _dyld_shared_cache_contains_path(const char* path) 
  __API_AVAILABLE(macos(11.0), ios(14.0), watchos(7.0), tvos(14.0)) 
DYLD_DRIVERKIT_UNAVAILABLE;
cpython-3.9> ^
cpython-3.9> ./Modules/_ctypes/callproc.c:1464:42: error: non-object type 'bool 
(const char *)' is not assignable
cpython-3.9> _dyld_shared_cache_contains_path = 
dlsym(libsystem_b_handle, "_dyld_shared_cache_contains_path");
cpython-3.9>  ^
cpython-3.9> ./Modules/_ctypes/callproc.c:1464:9: error: 
'_dyld_shared_cache_contains_path' is only available on macOS 11.0 or newer 
[-Werror,-Wunguarded-availability-new]
cpython-3.9> _dyld_shared_cache_contains_path = 
dlsym(libsystem_b_handle, "_dyld_shared_cache_contains_path");
cpython-3.9> ^~~~
cpython-3.9> 
/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/include/mach-o/dyld.h:121:13:
 note: '_dyld_shared_cache_contains_path' has been marked as being introduced 
in macOS 11.0 here, but the deployment target is macOS 10.9.0
cpython-3.9> extern bool _dyld_shared_cache_contains_path(const char* path) 
  __API_AVAILABLE(macos(11.0), ios(14.0), watchos(7.0), tvos(14.0)) 
DYLD_DRIVERKIT_UNAVAILABLE;
cpython-3.9> ^
cpython-3.9> ./Modules/_ctypes/callproc.c:1464:9: note: enclose 
'_dyld_shared_cache_contains_path' in a __builtin_available check to silence 
this warning
cpython-3.9> _dyld_shared_cache_contains_path = 
dlsym(libsystem_b_handle, "_dyld_shared_cache_contains_path");
cpython-3.9> ^~~~
cpython-3.9> ./Modules/_ctypes/callproc.c:1482:10: error: 
'_dyld_shared_cache_contains_path' is only available on macOS 11.0 or newer 
[-Werror,-Wunguarded-availability-new]
cpython-3.9>  if (HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME) {
cpython-3.9>  ^~~~
cpython-3.9> ./Modules/_ctypes/callproc.c:1474:5: note: expanded from macro 
'HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME'
cpython-3.9> _dyld_shared_cache_contains_path != NULL
cpython-3.9> ^~~~
cpython-3.9> 
/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/include/mach-o/dyld.h:121:13:
 note: '_dyld_shared_cache_contains_path' has been marked as being introduced 
in macOS 11.0 here, but the deployment target is macOS 10.9.0
cpython-3.9> extern bool _dyld_shared_cache_contains_path(const char* path) 
  __API_AVAILABLE(macos(11.0), ios(14.0), watchos(7.0), tvos(14.0)) 
DYLD_DRIVERKIT_UNAVAILABLE;
cpython-3.9> ^
cpython-3.9> ./Modules/_ctypes/callproc.c:1482:10: note: 

[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems

[miss-islington]

miss-islington  added the comment:


New changeset 41c87c41761739e90867a43de26e0f6755c80ff7 by Miss Islington (bot) 
in branch '3.10':
bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when 
built on older macOS systems (GH-27251)
https://github.com/python/cpython/commit/41c87c41761739e90867a43de26e0f6755c80ff7


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset 4b55837e7c747e0f3bd2df1b5c8996ce86c6f60a by Miss Islington (bot) 
in branch '3.9':
bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when 
built on older macOS systems (GH-27251) (GH-28053)
https://github.com/python/cpython/commit/4b55837e7c747e0f3bd2df1b5c8996ce86c6f60a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset 7234e67642bc8e75f44a2e1f2ce06687b5ac72ec by Miss Islington (bot) 
in branch '3.8':
bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when 
built on older macOS systems (GH-27251) (GH-28054)
https://github.com/python/cpython/commit/7234e67642bc8e75f44a2e1f2ce06687b5ac72ec


--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44689] ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems

[Ned Deily]

Change by Ned Deily :


--
assignee:  -> ned.deily
stage: patch review -> commit review
title: MacOS: Python binaries not portable between Catalina and Big Sur -> 
ctypes.util.find_library() does not find macOS 11+ system libraries when built 
on older macOS systems
versions: +Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com