Re: Reason for rpath usage on arm64

[Ken Cunningham]

Iain has added what MacPorts needed to the gcc branches he maintains, and
plans to push them through. This option will let us add the needed rpath to
where MacPorts puts the gcc libraries.

--with-darwin-add-path=


Also, re cmake PG, changing the cmake option for in-build library paths
should have no affect on the final installed files.

Best,

Ken


On Sun, May 29, 2022 at 23:56 Ryan Schmidt  wrote:

> On May 30, 2022, at 01:01, Ken Cunningham wrote:
>
> > ah yes, your question was arm64-specific so that led to the gcc
> assumption.
> >
> > the way the cmake PG configures cmake breaks testing on all archs and
> systems.
>
> It worked for me on Catalina x86_64 but not on Big Sur arm64, and I was
> trying to understand why this difference exists.
>
>
> > this comes up regularly, eg
> >
> >
> https://lists.macports.org/pipermail/macports-dev/2021-September/043754.html
> >
> > Perhaps it might beneficially be changed.
>
> I'm assuming that just changing the default in the cmake 1.1 portgroup
> could cause problems (or at least nonreproducibility) for ports already
> using the portgroup, so perhaps this change, plus any others we want to
> make to the cmake portgroup, should be in a new cmake 1.2 portgroup.
>
>

Re: Reason for rpath usage on arm64

[Ryan Schmidt]

On May 30, 2022, at 01:01, Ken Cunningham wrote:

> ah yes, your question was arm64-specific so that led to the gcc assumption.
> 
> the way the cmake PG configures cmake breaks testing on all archs and systems.

It worked for me on Catalina x86_64 but not on Big Sur arm64, and I was trying 
to understand why this difference exists.


> this comes up regularly, eg
> 
> https://lists.macports.org/pipermail/macports-dev/2021-September/043754.html
> 
> Perhaps it might beneficially be changed.

I'm assuming that just changing the default in the cmake 1.1 portgroup could 
cause problems (or at least nonreproducibility) for ports already using the 
portgroup, so perhaps this change, plus any others we want to make to the cmake 
portgroup, should be in a new cmake 1.2 portgroup.

Re: Reason for rpath usage on arm64

[Ken Cunningham]

ah yes, your question was arm64-specific so that led to the gcc assumption.

the way the cmake PG configures cmake breaks testing on all archs and
systems.

this comes up regularly, eg

https://lists.macports.org/pipermail/macports-dev/2021-September/043754.html

Perhaps it might beneficially be changed.

K

On Sun, May 29, 2022 at 22:17 Ryan Schmidt  wrote:

> On May 24, 2022, at 10:22, Ken Cunningham wrote:
> >
> >> Does anyone know why things seem to be using rpath on arm64 builds
> >
> > Basically, it comes down to the fact that Apple has disallowed
> DYLD_LIBRARY_PATH to work on newer systems.
>
> > So gcc is installing it’s libraries with the @rpath linkage
>
> So you're talking about how SIP doesn't pass DYLD_LIBRARY_PATH to child
> processes on El Capitan and later, which breaks the gcc test suite,
> therefore gcc now uses @rpath at build time to fix the problem, and does
> not rewrite the install_names at install time, therefore things with @rpath
> get installed.
>
> That's not the issue that prompted me to start this thread.
>
> When I enabled the test suite for the portmidi port (which does not use
> gcc)...
>
>
> https://github.com/macports/macports-ports/commit/fd447d210e1814b1303c5013d088efa673470d77
>
> ...I set "DYLD_LIBRARY_PATH=." in test.env so that the executable it runs
> can find the library it just built. This worked fine on Catalina x86_64 but
> on Big Sur arm64 it failed with the error message:
>
> dyld: Library not loaded: @rpath/libportmidi.2.dylib
>   Referenced from: /path/to/portmidi/work/build/./qtest
>   Reason: image not found
>
> Seeing "@rpath" in this error message, I added
> "-DCMAKE_BUILD_WITH_INSTALL_RPATH=OFF" to configure.args (to counteract
> "-DCMAKE_BUILD_WITH_INSTALL_RPATH=ON" set by the cmake portgroup) and the
> problem went away.
>
> Perhaps placing the blame on @rpath was the wrong conclusion, since I now
> realize that the install_names also use @rpath on Catalina x86_64 yet the
> test worked there. I just don't understand then why disabling @rpath on Big
> Sur arm64 made the tests work.
>
> It seemed to me like I was seeing a lot of @rpath-related issues
> especially on arm64 lately, and this latest instance with portmidi prompted
> me to ask about it, but it could be that most or all of the other instances
> were related to the gcc situation you already discussed. I know we are
> using a different branch of gcc for arm64 so maybe that explains why the
> @rpath-related changes you mentioned only appear on arm64 so far.
>
>

Re: Reason for rpath usage on arm64

[Ryan Schmidt]

On May 24, 2022, at 10:22, Ken Cunningham wrote:
> 
>> Does anyone know why things seem to be using rpath on arm64 builds
> 
> Basically, it comes down to the fact that Apple has disallowed 
> DYLD_LIBRARY_PATH to work on newer systems.

> So gcc is installing it’s libraries with the @rpath linkage

So you're talking about how SIP doesn't pass DYLD_LIBRARY_PATH to child 
processes on El Capitan and later, which breaks the gcc test suite, therefore 
gcc now uses @rpath at build time to fix the problem, and does not rewrite the 
install_names at install time, therefore things with @rpath get installed.

That's not the issue that prompted me to start this thread.

When I enabled the test suite for the portmidi port (which does not use gcc)...

https://github.com/macports/macports-ports/commit/fd447d210e1814b1303c5013d088efa673470d77

...I set "DYLD_LIBRARY_PATH=." in test.env so that the executable it runs can 
find the library it just built. This worked fine on Catalina x86_64 but on Big 
Sur arm64 it failed with the error message:

dyld: Library not loaded: @rpath/libportmidi.2.dylib
  Referenced from: /path/to/portmidi/work/build/./qtest
  Reason: image not found

Seeing "@rpath" in this error message, I added 
"-DCMAKE_BUILD_WITH_INSTALL_RPATH=OFF" to configure.args (to counteract 
"-DCMAKE_BUILD_WITH_INSTALL_RPATH=ON" set by the cmake portgroup) and the 
problem went away.

Perhaps placing the blame on @rpath was the wrong conclusion, since I now 
realize that the install_names also use @rpath on Catalina x86_64 yet the test 
worked there. I just don't understand then why disabling @rpath on Big Sur 
arm64 made the tests work.

It seemed to me like I was seeing a lot of @rpath-related issues especially on 
arm64 lately, and this latest instance with portmidi prompted me to ask about 
it, but it could be that most or all of the other instances were related to the 
gcc situation you already discussed. I know we are using a different branch of 
gcc for arm64 so maybe that explains why the @rpath-related changes you 
mentioned only appear on arm64 so far.

Re: Reason for rpath usage on arm64

[Joshua Root]

On 2022-5-25 03:01 , Ken Cunningham wrote:

That comes right from Iain - here's the relevant gcc ticket I believe:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88590 



I'm sure Iain would be delighted to find a way around this.


OK, so it's only when setting the var via /bin/sh and friends, which we 
already knew about. I'm sure it's a pain if your build system does that 
a lot.


- Josh

Re: Reason for rpath usage on arm64

[Clemens Lang]

Hi,

On Tue, May 24, 2022 at 10:01:09AM -0700, Ken Cunningham wrote:
> That comes right from Iain - here's the relevant gcc ticket I believe:
> 
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88590
> 
> I'm sure Iain would be delighted to find a way around this.

MacPorts works around this by making a copy of the binary to be
executed, stripping the SIP flag, and transparently running this copy.
See [1].

Re-using this solution in GCC would additionally require interposing the
execve(2) and posix_spawn(2) syscalls. We have code to do that available
in MacPorts at [2].

I'm not sure whether Iain would want to port this solution, though. On
the other hand, it has been working reasonably well since 10.11 (except
for a fix recently required to restore it from abysmal performance
because Apple broke their API [3] and a few resource leak fixes [4]).

HTH,
Clemens

 [1]: 
https://github.com/macports/macports-base/blob/master/src/pextlib1.0/sip_copy_proc.c#L473-L479
 [2]: 
https://github.com/macports/macports-base/blob/master/src/darwintracelib1.0/proc.c#L247-L287
 [3]: 
https://github.com/macports/macports-base/commit/3b949ccfe7a309bce1cc7041cdf56871890b784d
 [4]: 
https://github.com/macports/macports-base/commit/f1391cfaea525f57e82b5cbe3dce6b0a782325bb

Re: Reason for rpath usage on arm64

[Chris Jones]

> On 24 May 2022, at 4:22 pm, Ken Cunningham  
> wrote:
> 
> 
>> 
>> Does anyone know why things seem to be using rpath on arm64 builds
> 
> Basically, it comes down to the fact that Apple has disallowed 
> DYLD_LIBRARY_PATH to work on newer systems.
> 
> gcc for Darwin has one developer working on it, practically. Iain. He has 
> been maintaining it for 10+ years, wrote the objective C implementation it 
> uses, and is pretty much the only reason gcc works on Darwin at all. Perhaps 
> someone would fill his shoes if he stopped doing it, hard to say.
> 
> He maintains hardware machines from 10.5 to the current, and he tries to keep 
> as many versions of gcc as possible running on as many systems as he can.
> 
> Once Apple disallowed DYLD_LIBRARY_PATH, he has had a lot of trouble running 
> the test suites on the gcc libraries, as he can’t force the libraries under 
> test to be the ones used. Instead the libraries installed are the ones used, 
> which means he has to actually install the gcc before testing it.
> 
> Multiply that by perhaps 6 active gcc versions on each system, and 50+ test 
> runs of gcc weekly, and you can see the problem.
> 
> Using @rpath gets around that problem very nicely.
> 
> Now, LLVM and many other software projects have the same issue. They also use 
> @rpath to do it. But on INSTALLATION these @rpaths can be rewritten 
> automatically by cmake (or meson) to be full pathnames. So that is what 
> MacPorts does.
> 
> gcc’s build system does not currently offer that installation option. I don’t 
> know how hard it would be to add it (any volunteers?) but for now at least, 
> Iain is not keen on adding that @rpath rewrite-on-install business to gcc.

It probably would not be too to do to be honest. I do similar things in other 
places where I need to alter binaries post destroot, using install-name-tool 
etc. Recursively searching the destroot for all binaries and running some fix 
up on them ? What, *precisely* would that fix need to be ? To ensure they have 
an rpath entry for the place the binaries get installed, or something else ?

Chris
> 
> So gcc is installing it’s libraries with the @rpath linkages — it’s not just 
> arm, by the way. It’s just arm right now because that is the tip of the 
> spear. Very soon ALL gcc installs will use @rpaths, as that is (for now, 
> barring a better option) the only way Iain can maintain gcc at all on Darwin.
> 
> Ken
> 
> 

Re: Reason for rpath usage on arm64

[Ken Cunningham]

Oh, nothing whatsoever to do with arm64 specifically. This is going to
affect all gcc builds on all darwin-supported architectures once Iain
pushes it through to all the branches he plans to keep supporting.

On Tue, May 24, 2022 at 9:50 AM Joshua Root  wrote:

> On 2022-5-25 01:22 , Ken Cunningham wrote:
> >> Does anyone know why things seem to be using rpath on arm64 builds
> >
> > Basically, it comes down to the fact that Apple has disallowed
> > DYLD_LIBRARY_PATH to work on newer systems.
>
> This is the first I've heard of it. Is this supposed to be the case only
> on arm64? I know DYLD_* variables are ignored when running SIP-flagged
> executables, but that wouldn't apply to something that has just been built.
>
> We use DYLD_LIBRARY_PATH in our own build system:
> 
>
> - Josh
>

Re: Reason for rpath usage on arm64

[Ken Cunningham]

That comes right from Iain - here's the relevant gcc ticket I believe:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88590

I'm sure Iain would be delighted to find a way around this.



On Tue, May 24, 2022 at 9:50 AM Joshua Root  wrote:

> On 2022-5-25 01:22 , Ken Cunningham wrote:
> >> Does anyone know why things seem to be using rpath on arm64 builds
> >
> > Basically, it comes down to the fact that Apple has disallowed
> > DYLD_LIBRARY_PATH to work on newer systems.
>
> This is the first I've heard of it. Is this supposed to be the case only
> on arm64? I know DYLD_* variables are ignored when running SIP-flagged
> executables, but that wouldn't apply to something that has just been built.
>
> We use DYLD_LIBRARY_PATH in our own build system:
> 
>
> - Josh
>

Re: Reason for rpath usage on arm64

[Joshua Root]

On 2022-5-25 01:22 , Ken Cunningham wrote:

Does anyone know why things seem to be using rpath on arm64 builds


Basically, it comes down to the fact that Apple has disallowed 
DYLD_LIBRARY_PATH to work on newer systems.


This is the first I've heard of it. Is this supposed to be the case only 
on arm64? I know DYLD_* variables are ignored when running SIP-flagged 
executables, but that wouldn't apply to something that has just been built.


We use DYLD_LIBRARY_PATH in our own build system: 



- Josh

Re: Reason for rpath usage on arm64

[Ken Cunningham]

> Does anyone know why things seem to be using rpath on arm64 builds

Basically, it comes down to the fact that Apple has disallowed 
DYLD_LIBRARY_PATH to work on newer systems.

gcc for Darwin has one developer working on it, practically. Iain. He has been 
maintaining it for 10+ years, wrote the objective C implementation it uses, and 
is pretty much the only reason gcc works on Darwin at all. Perhaps someone 
would fill his shoes if he stopped doing it, hard to say.

He maintains hardware machines from 10.5 to the current, and he tries to keep 
as many versions of gcc as possible running on as many systems as he can.

Once Apple disallowed DYLD_LIBRARY_PATH, he has had a lot of trouble running 
the test suites on the gcc libraries, as he can’t force the libraries under 
test to be the ones used. Instead the libraries installed are the ones used, 
which means he has to actually install the gcc before testing it.

Multiply that by perhaps 6 active gcc versions on each system, and 50+ test 
runs of gcc weekly, and you can see the problem.

Using @rpath gets around that problem very nicely.

Now, LLVM and many other software projects have the same issue. They also use 
@rpath to do it. But on INSTALLATION these @rpaths can be rewritten 
automatically by cmake (or meson) to be full pathnames. So that is what 
MacPorts does.

gcc’s build system does not currently offer that installation option. I don’t 
know how hard it would be to add it (any volunteers?) but for now at least, 
Iain is not keen on adding that @rpath rewrite-on-install business to gcc.

So gcc is installing it’s libraries with the @rpath linkages — it’s not just 
arm, by the way. It’s just arm right now because that is the tip of the spear. 
Very soon ALL gcc installs will use @rpaths, as that is (for now, barring a 
better option) the only way Iain can maintain gcc at all on Darwin.

Ken