Re: A macOS static linking mystery
One peculiarity with the ordering is that linkers only search static archives for existing undefined references. If the reference to `Cffi` actually comes first, then nothing should be required from it yet and *it shouldn't be linked* (absent options like GNU ld's `--whole-archive`). This said, ghc also passes a bunch of explicitly undefined symbols (`-Wl,-u…`) to ld; if those include symbols from `-lCffi` then it will in fact be linked in. On Tue, Aug 9, 2022 at 9:59 PM Ryan Scott wrote: > > I believe I've figured out what is happening here. This will take a bit of > explanation, so bear with me. > > First, it's important to note that this behavior depends on whether your GHC > was configured with the `--with-system-libffi` flag or not. If it was > configured with this flag, then GHC will always link against your system's > version of libffi, and the behavior seen in this thread would not occur. I am > using a bindist provided by ghcup which was _not_ configured with > `--with-system-libffi`, so GHC bundles its own copy of libffi. Interestingly, > there appear to be at least two copies of libffi that are bundled: one > dynamic library, simply named libffi.dylib, and one static library, which is > named libCffi.a [1]. Remember the name libCffi.a, since it will come up again > later. > > Next, let's take a look at how GHC performs the final linking step when > compiling an executable. GHC will call the C compiler to perform this final > linking step, which will have roughly the following shape on macOS: > > gcc ... -L/rts -lCffi > -Wl,-dead_strip_dylibs -o Main > > I'm omitting quite a few details here, but the important bits are that (1) > -lCffi comes before any of the C libraries listed in an `extra-libraries` > stanza in a .cabal file, and (2) GHC on macOS will pass > -Wl,-dead_strip_dylibs, which causes the linker to remove any references to > shared libraries that go unused. > > In the particular case of libffi.cabal, we have `extra-libraries: ffi`, so > the C compiler is given `-lCffi -lffi` as arguments, in that order. Recall > that -libCffi.a (i.e., -lCffi) is a statically linked copy of libffi, whereas > libffi.dylib (i.e., -lffi) is a dynamically linked version. The linker will > resolve all of its references to libffi functionality from libCffi.a because > it appears first, and since the linker never needs to resolve anything from > libffi.dylib, the -Wl,-dead_strip_dylibs will remove the reference to > libffi.dylib entirely. The result is that the final executable will > statically link against libffi. This is why this phenomenon only occurs with > libffi and not with other C libraries, since only libffi is given special > treatment within GHC like this. > > Why, then, does this only happen on macOS and not on, say, Linux? On Linux, > the final linking step looks slightly different. Instead of passing > -Wl,-dead_strip_dylibs (which is a macOS-specific option), it will pass > -Wl,--no-as-needed. Note that -Wl,--no-as-needed has the entirely _opposite_ > effect of -Wl,-dead_strip_dylibs: even if a shared library never gets > referenced, the linker will still declare a dynamic dependency on it. That is > exactly what happens here, as the -lCffi argument will still take precedence > over the -lffi argument on Linux, but there will be a useless dynamic > dependency on a dynamic libffi library due to -Wl,--no-as-needed. > > At this point, I realized that one consequence of all this is that if you > want to statically link against against libffi, and you are using a GHC > configured without ` --with-system-libffi`, then it suffices to simply > compile your program without any `extra-libraries` at all. That is because > the final link step will pass `-lCffi` no matter what, so the resulting > executable will always statically link against libffi by way of libCffi.a. > Moreover, libCffi is provided on all platforms, not just macOS, so this trick > is reasonably portable. > > - > > Having explained all this, I am forced to wonder if this is intended behavior > of GHC, or if it is a series of coincidences. The fact that macOS and Linux > pass -Wl,-dead_strip_dylibs and -Wl,--no-as-needed, two flags with completely > opposite behavior, is certainly a little unusual. But these choices appear to > be deliberate, as they are explained in Notes in the GHC source code here [2] > (for macOS) and here [3] (for Linux). The other possibly unusual thing is > that GHC always passes -lCffi before any of the `extra-libraries`. That could > very well be just a coincidence—I'm not sure if there's a particular reason > for -lCffi to come first. > > The reason that I'm looking so deeply into this matter is that I'm trying to > see what it would take to reliably link against libffi statically in the > Haskell bindings [4]. I've discovered a trick above to make it work for most > versions of GHC: just rely on `-lCffi`. Does this trick seem like it would >
Re: A macOS static linking mystery
I believe I've figured out what is happening here. This will take a bit of explanation, so bear with me. First, it's important to note that this behavior depends on whether your GHC was configured with the `--with-system-libffi` flag or not. If it was configured with this flag, then GHC will always link against your system's version of libffi, and the behavior seen in this thread would not occur. I am using a bindist provided by ghcup which was _not_ configured with `--with-system-libffi`, so GHC bundles its own copy of libffi. Interestingly, there appear to be at least two copies of libffi that are bundled: one dynamic library, simply named libffi.dylib, and one static library, which is named libCffi.a [1]. Remember the name libCffi.a, since it will come up again later. Next, let's take a look at how GHC performs the final linking step when compiling an executable. GHC will call the C compiler to perform this final linking step, which will have roughly the following shape on macOS: gcc ... -L/rts -lCffi -Wl,-dead_strip_dylibs -o Main I'm omitting quite a few details here, but the important bits are that (1) -lCffi comes before any of the C libraries listed in an `extra-libraries` stanza in a .cabal file, and (2) GHC on macOS will pass -Wl,-dead_strip_dylibs, which causes the linker to remove any references to shared libraries that go unused. In the particular case of libffi.cabal, we have `extra-libraries: ffi`, so the C compiler is given `-lCffi -lffi` as arguments, in that order. Recall that -libCffi.a (i.e., -lCffi) is a statically linked copy of libffi, whereas libffi.dylib (i.e., -lffi) is a dynamically linked version. The linker will resolve all of its references to libffi functionality from libCffi.a because it appears first, and since the linker never needs to resolve anything from libffi.dylib, the -Wl,-dead_strip_dylibs will remove the reference to libffi.dylib entirely. The result is that the final executable will statically link against libffi. This is why this phenomenon only occurs with libffi and not with other C libraries, since only libffi is given special treatment within GHC like this. Why, then, does this only happen on macOS and not on, say, Linux? On Linux, the final linking step looks slightly different. Instead of passing -Wl,-dead_strip_dylibs (which is a macOS-specific option), it will pass -Wl,--no-as-needed. Note that -Wl,--no-as-needed has the entirely _opposite_ effect of -Wl,-dead_strip_dylibs: even if a shared library never gets referenced, the linker will still declare a dynamic dependency on it. That is exactly what happens here, as the -lCffi argument will still take precedence over the -lffi argument on Linux, but there will be a useless dynamic dependency on a dynamic libffi library due to -Wl,--no-as-needed. At this point, I realized that one consequence of all this is that if you want to statically link against against libffi, and you are using a GHC configured without ` --with-system-libffi`, then it suffices to simply compile your program without any `extra-libraries` at all. That is because the final link step will pass `-lCffi` no matter what, so the resulting executable will always statically link against libffi by way of libCffi.a. Moreover, libCffi is provided on all platforms, not just macOS, so this trick is reasonably portable. - Having explained all this, I am forced to wonder if this is intended behavior of GHC, or if it is a series of coincidences. The fact that macOS and Linux pass -Wl,-dead_strip_dylibs and -Wl,--no-as-needed, two flags with completely opposite behavior, is certainly a little unusual. But these choices appear to be deliberate, as they are explained in Notes in the GHC source code here [2] (for macOS) and here [3] (for Linux). The other possibly unusual thing is that GHC always passes -lCffi before any of the `extra-libraries`. That could very well be just a coincidence—I'm not sure if there's a particular reason for -lCffi to come first. The reason that I'm looking so deeply into this matter is that I'm trying to see what it would take to reliably link against libffi statically in the Haskell bindings [4]. I've discovered a trick above to make it work for most versions of GHC: just rely on `-lCffi`. Does this trick seem like it would continue to work for the foreseeable future, or is this approach liable to break in a future release? Ryan - [1] See https://gitlab.haskell.org/ghc/ghc/-/issues/20395 for more on this unusual amount of library duplication. [2] https://gitlab.haskell.org/ghc/ghc/-/blob/d71a20514546e0befe6e238d0658cbaad5a13996/compiler/GHC/Driver/Pipeline.hs#L293-340 [3] https://gitlab.haskell.org/ghc/ghc/-/blob/d71a20514546e0befe6e238d0658cbaad5a13996/compiler/GHC/SysTools/Info.hs#L57-69 [4] https://github.com/remiturk/libffi/issues/6 On Tue, Aug 9, 2022 at 9:58 AM Ryan Scott wrote: > No, I'm using macOS 10.15.7 (BuildVersion 19H2) on an x86_64 machine. > > Ryan >
Re: [ANNOUNCE] GHC 9.4.1 is now available
Ben Gamari writes: > The GHC developers are very pleased to announce the availability of GHC > 9.4.1. Binary distributions, source distributions, and documentation are > available at downloads.haskell.org: > > https://downloads.haskell.org/ghc/9.4.1 > Hi all, Due to an unfortunate packaging issue, the macOS binary distributions for 9.4.1 are not usable as uploaded. The problem is described in #21974, which also includes a small patch to mitigate the breakage. We will be releasing a 9.4.2 within the week fixing the issue. Cheers, - Ben signature.asc Description: PGP signature ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: ghc doesn't work after installing 9.4.1 on my Mac
Known issue George. See e.g.
https://gitlab.haskell.org/ghc/ghc/-/issues/21985
On Tue, Aug 9, 2022, 4:33 PM George Colpitts
wrote:
> Unfortunately ghc doesn't work after installing 9.4.1 on my Mac. Does it
> work for others?
>
> After the install finishes when I do the following:
>
> $ ghc --version
> bash: /usr/local/bin/ghc: Permission denied
> $ sudo chmod +x /usr/local/bin/ghc
> $ghc --version
> /usr/local/bin/ghc: line 1: exec: : not found
> $ cat /usr/local/bin/ghc
> exec "$executablename" -B"$libdir" ${1+"$@"}
> $ cat /usr/local/bin/ghc-9.4.1
> #!/bin/sh
> exedir="/usr/local/lib/ghc-9.4.1/bin"
> exeprog="ghc-9.4.1"
> executablename="/usr/local/lib/ghc-9.4.1/bin/ghc-9.4.1"
> bindir="/usr/local/bin"
> libdir="/usr/local/lib/ghc-9.4.1/lib"
> docdir="/usr/local/share/doc/ghc-9.4.1"
> includedir="/usr/local/include"
>
> exec "$executablename" -B"$libdir" ${1+"$@"}
>
>
> Thanks
> George
>
>
>
> On Sun, Aug 7, 2022 at 6:30 PM Ben Gamari wrote:
>
>> The GHC developers are very pleased to announce the availability of GHC
>> 9.4.1. Binary distributions, source distributions, and documentation are
>> available at downloads.haskell.org:
>>
>> https://downloads.haskell.org/ghc/9.4.1
>>
>> This release includes:
>>
>> - A new profiling mode, `-fprof-late`, which adds automatic cost-center
>>annotations to all top-level functions *after* Core optimisation has
>>run. This provides informative profiles while interfering
>>significantly less with GHC's aggressive optimisations, making it
>>easier to understand the performance of programs which depend upon
>>simplification..
>>
>> - A variety of plugin improvements including the introduction of a new
>>plugin type, *defaulting plugins*, and the ability for typechecking
>>plugins to rewrite type-families.
>>
>> - An improved constructed product result analysis, allowing unboxing of
>>nested structures, and a new boxity analysis, leading to less reboxing.
>>
>> - Introduction of a tag-check elision optimisation, bringing
>>significant performance improvements in strict programs.
>>
>> - Generalisation of a variety of primitive types to be levity
>>polymorphic. Consequently, the `ArrayArray#` type can at long last be
>>retired, replaced by standard `Array#`.
>>
>> - Introduction of the `\cases` syntax from [GHC proposal 0302].
>>
>> - A complete overhaul of GHC's Windows support. This includes a
>>migration to a fully Clang-based C toolchain, a deep refactoring of
>>the linker, and many fixes in WinIO.
>>
>> - Support for multiple home packages, significantly improving support
>>in IDEs and other tools for multi-package projects.
>>
>> - A refactoring of GHC's error message infrastructure, allowing GHC to
>>provide diagnostic information to downstream consumers as structured
>>data, greatly easing IDE support.
>>
>> - Significant compile-time improvements to runtime and memory
>> consumption.
>>
>> - On overhaul of our packaging infrastructure, allowing full
>>traceability of release artifacts and more reliable binary
>>distributions.
>>
>> - Reintroduction of deep subsumption (which was previously dropped with
>> the
>>*simplified subsumption* change) as a language extension.
>>
>> - ... and much more. See the [release notes] for a full accounting.
>>
>> Note that, as 9.4.1 is the first release for which the released artifacts
>> will
>> all be generated by our Hadrian build system, it is possible that there
>> will be
>> packaging issues. If you enounter trouble while using a binary
>> distribution,
>> please open a [ticket]. Likewise, if you are a downstream packager, do
>> consider
>> migrating to [Hadrian] to run your build; the Hadrian build system can be
>> built
>> using `cabal-install`, `stack`, or the in-tree [bootstrap script]. See
>> the accompanying
>> [blog post] for details on migrating packaging to Hadrian.
>>
>> We would like to thank Microsoft Azure, GitHub, IOG, the Zw3rk stake pool,
>> Well-Typed, Tweag I/O, Serokell, Equinix, SimSpace, Haskell Foundation,
>> and
>> other anonymous contributors whose on-going financial and in-kind support
>> has
>> facilitated GHC maintenance and release management over the years.
>> Finally,
>> this release would not have been possible without the hundreds of
>> open-source
>> contributors whose work comprise this release.
>>
>> As always, do give this release a try and open a [ticket] if you see
>> anything amiss.
>>
>> Happy testing,
>>
>> - Ben
>>
>>
>> [GHC proposal 0302]:
>> https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0302-cases.rst
>> [ticket]: https://gitlab.haskell.org/ghc/ghc/-/issues/new
>> [bootstrap script]:
>> https://gitlab.haskell.org/ghc/ghc/-/blob/e2520df3fffa0cf22fb19c5fb872832d11c07d35/hadrian/bootstrap/README.md
>> [Hadrian]:
>> https://gitlab.haskell.org/ghc/ghc/-/blob/e2520df3fffa0cf22fb19c5fb872832d11c07d35/hadrian
>> [release notes]:
>>
ghc doesn't work after installing 9.4.1 on my Mac
Unfortunately ghc doesn't work after installing 9.4.1 on my Mac. Does it
work for others?
After the install finishes when I do the following:
$ ghc --version
bash: /usr/local/bin/ghc: Permission denied
$ sudo chmod +x /usr/local/bin/ghc
$ghc --version
/usr/local/bin/ghc: line 1: exec: : not found
$ cat /usr/local/bin/ghc
exec "$executablename" -B"$libdir" ${1+"$@"}
$ cat /usr/local/bin/ghc-9.4.1
#!/bin/sh
exedir="/usr/local/lib/ghc-9.4.1/bin"
exeprog="ghc-9.4.1"
executablename="/usr/local/lib/ghc-9.4.1/bin/ghc-9.4.1"
bindir="/usr/local/bin"
libdir="/usr/local/lib/ghc-9.4.1/lib"
docdir="/usr/local/share/doc/ghc-9.4.1"
includedir="/usr/local/include"
exec "$executablename" -B"$libdir" ${1+"$@"}
Thanks
George
On Sun, Aug 7, 2022 at 6:30 PM Ben Gamari wrote:
> The GHC developers are very pleased to announce the availability of GHC
> 9.4.1. Binary distributions, source distributions, and documentation are
> available at downloads.haskell.org:
>
> https://downloads.haskell.org/ghc/9.4.1
>
> This release includes:
>
> - A new profiling mode, `-fprof-late`, which adds automatic cost-center
>annotations to all top-level functions *after* Core optimisation has
>run. This provides informative profiles while interfering
>significantly less with GHC's aggressive optimisations, making it
>easier to understand the performance of programs which depend upon
>simplification..
>
> - A variety of plugin improvements including the introduction of a new
>plugin type, *defaulting plugins*, and the ability for typechecking
>plugins to rewrite type-families.
>
> - An improved constructed product result analysis, allowing unboxing of
>nested structures, and a new boxity analysis, leading to less reboxing.
>
> - Introduction of a tag-check elision optimisation, bringing
>significant performance improvements in strict programs.
>
> - Generalisation of a variety of primitive types to be levity
>polymorphic. Consequently, the `ArrayArray#` type can at long last be
>retired, replaced by standard `Array#`.
>
> - Introduction of the `\cases` syntax from [GHC proposal 0302].
>
> - A complete overhaul of GHC's Windows support. This includes a
>migration to a fully Clang-based C toolchain, a deep refactoring of
>the linker, and many fixes in WinIO.
>
> - Support for multiple home packages, significantly improving support
>in IDEs and other tools for multi-package projects.
>
> - A refactoring of GHC's error message infrastructure, allowing GHC to
>provide diagnostic information to downstream consumers as structured
>data, greatly easing IDE support.
>
> - Significant compile-time improvements to runtime and memory consumption.
>
> - On overhaul of our packaging infrastructure, allowing full
>traceability of release artifacts and more reliable binary
>distributions.
>
> - Reintroduction of deep subsumption (which was previously dropped with
> the
>*simplified subsumption* change) as a language extension.
>
> - ... and much more. See the [release notes] for a full accounting.
>
> Note that, as 9.4.1 is the first release for which the released artifacts
> will
> all be generated by our Hadrian build system, it is possible that there
> will be
> packaging issues. If you enounter trouble while using a binary
> distribution,
> please open a [ticket]. Likewise, if you are a downstream packager, do
> consider
> migrating to [Hadrian] to run your build; the Hadrian build system can be
> built
> using `cabal-install`, `stack`, or the in-tree [bootstrap script]. See the
> accompanying
> [blog post] for details on migrating packaging to Hadrian.
>
> We would like to thank Microsoft Azure, GitHub, IOG, the Zw3rk stake pool,
> Well-Typed, Tweag I/O, Serokell, Equinix, SimSpace, Haskell Foundation, and
> other anonymous contributors whose on-going financial and in-kind support
> has
> facilitated GHC maintenance and release management over the years. Finally,
> this release would not have been possible without the hundreds of
> open-source
> contributors whose work comprise this release.
>
> As always, do give this release a try and open a [ticket] if you see
> anything amiss.
>
> Happy testing,
>
> - Ben
>
>
> [GHC proposal 0302]:
> https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0302-cases.rst
> [ticket]: https://gitlab.haskell.org/ghc/ghc/-/issues/new
> [bootstrap script]:
> https://gitlab.haskell.org/ghc/ghc/-/blob/e2520df3fffa0cf22fb19c5fb872832d11c07d35/hadrian/bootstrap/README.md
> [Hadrian]:
> https://gitlab.haskell.org/ghc/ghc/-/blob/e2520df3fffa0cf22fb19c5fb872832d11c07d35/hadrian
> [release notes]:
> https://downloads.haskell.org/~ghc/9.4.1/docs/users_guide/9.4.1-notes.html
> [blog post]:
> https://www.haskell.org/ghc/blog/20220805-make-to-hadrian.html
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
Re: A macOS static linking mystery
No, I'm using macOS 10.15.7 (BuildVersion 19H2) on an x86_64 machine. Ryan ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: A macOS static linking mystery
i think brandon is also referring to the Rosetta stuff on Arm Macs, are you using one of those? On Mon, Aug 8, 2022 at 8:00 PM Brandon Allbery wrote: > https://developer.apple.com/forums/thread/692383 is what I'm thinking > of, but in this case there would still be library references shown by > "otool -L", so I guess that's not what you're seeing. > > On Mon, Aug 8, 2022 at 7:49 PM Ryan Scott wrote: > > > > Possibly? I'm not familiar with the system cache you're referring to, > but it's a more solid lead than what I have currently. Do you know how one > would confirm if this is the case or not? > > > > Ryan > > > > -- > brandon s allbery kf8nh > allber...@gmail.com > ___ > ghc-devs mailing list > ghc-devs@haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
