Re: [Nix-dev] [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment

[Joseph Myers]

On Sun, 16 Oct 2016, Shea Levy wrote:

> options) and clearly have the semantics we want. Ideally we would be
> able to specify something on the level of abstraction of "this directory
> should be treated like you would normally treat /usr" and get
> e.g. /include, /lib, frameworks on OS X, etc. handled properly.

What that suggests to me is options for having multiple sysroots (which 
are treated like / not like /usr, but that's the existing concept for a 
directory containing both header and library subdirectories, and you could 
combine this with a Hurd-style configuration of the expected sysroot 
subdirectories, i.e. no /usr inside the sysroot).  That would however be 
rather complex; both GCC and ld presume there is a single global sysroot 
(modulo SYSROOT_SUFFIX_SPEC / SYSROOT_HEADERS_SUFFIX_SPEC that append to 
it), as do the interfaces for other specs that pass down sysroot 
information to cc1 etc. - and ld then interprets absolute paths in linker 
scripts such as libc.so in a sysroot relative to that sysroot (so would 
need to track which sysroot a particular linker script was found in to 
know how to interpret absolute paths in it), and options such as -I=dir 
for a sysroot-relative include don't have a clear meaning with multiple 
sysroots.

I'm wary of adding environment variables as they tend to make debugging 
hard when the same compiler behaves differently for different people for 
no obvious reason.

You should not need to exclude linker options (as opposed to linker input 
files) from the command line when not linking, or compiler options when 
linking.

-- 
Joseph S. Myers
jos...@codesourcery.com
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment

[Ludovic Courtès]

Hi!

Shea Levy  skribis:

> Your patches look good! My biggest concern is how the ld wrapper behaves
> in the presence of response files. Have you tested that?

It surely doesn’t (yet?).

However, GCC does not pass “@file” arguments when it invokes ‘ld’, and
the bug report you mentioned¹ talks about GHC invoking ‘gcc’, not ‘ld’,
so I guess it’s fine to ignore response files in the ld wrapper.

Ludo’.

¹ 
https://github.com/NixOS/nixpkgs/commit/a421e7bd4a28c69bded8b17888325e31554f61a1
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment

[Shea Levy]

Hi Ludo’,

Your patches look good! My biggest concern is how the ld wrapper behaves
in the presence of response files. Have you tested that?

Thanks,
Shea

Ludovic Courtès  writes:

> Hi Shea,
>
> Shea Levy skribis:
>
>> Unlike the traditional approach of installing system libraries into one
>> central location like /usr/{lib,include}, the nix package manager [1]
>> installs each package into it's own prefix
>> (e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
>> each package is built in its own environment determined from its
>> explicitly listed dependencies, regardless of what else is installed on
>> the system. Because not all package build scripts properly respect
>> CFLAGS etc., we currently wrap the compiler [2] to respect custom
>> environment variables like NIX_CFLAGS_COMPILE, so during the build of a
>> package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
>> to "-isystem 
>> /nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include 
>> -isystem /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".
>>
>> Unfortunately, as you can see if you click through the link or look
>> through the git history, the wrapper is quite complex (frankly, hacky)
>
>> [2]: 
>> https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh
>
> Guix avoids the compiler wrapper altogether like this:
>
>   • We use C_INCLUDE_PATH, LIBRARY_PATH, and friends:
> 
> .
>
>   • We have a simple linker wrapper aimed at adding -Wl,-rpath flags:
> 
> .
> The comment in that file explains why the other options considered
> were unsuitable.
>
>   • We modify the built-in “lib” spec of GCC to add the necessary -L and
> -rpath flags:
> 
> .
>
>   • Likewise, we tell Clang where to find libc and friends:
> 
> 
> 
> .
>
> This is not too intrusive and more robust than wrapping everything.
>
> I suppose GCC and Clang could facilitate this by providing configure
> options to augment the “lib” spec, specify the location of libc alone,
> or something along these lines.
>
> Thoughts?
>
> Ludo’.


signature.asc
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment

[Shea Levy]

Hey Ludo’,

Amazing, more than a decade of close working with these tools and I
never knew about C_INCLUDE_PATH et al! It looks like those will solve a
huge portion of the problem.

Will look at your gcc and clang patches as well, thank you!

~Shea

Ludovic Courtès  writes:

> Hi Shea,
>
> Shea Levy skribis:
>
>> Unlike the traditional approach of installing system libraries into one
>> central location like /usr/{lib,include}, the nix package manager [1]
>> installs each package into it's own prefix
>> (e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
>> each package is built in its own environment determined from its
>> explicitly listed dependencies, regardless of what else is installed on
>> the system. Because not all package build scripts properly respect
>> CFLAGS etc., we currently wrap the compiler [2] to respect custom
>> environment variables like NIX_CFLAGS_COMPILE, so during the build of a
>> package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
>> to "-isystem 
>> /nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include 
>> -isystem /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".
>>
>> Unfortunately, as you can see if you click through the link or look
>> through the git history, the wrapper is quite complex (frankly, hacky)
>
>> [2]: 
>> https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh
>
> Guix avoids the compiler wrapper altogether like this:
>
>   • We use C_INCLUDE_PATH, LIBRARY_PATH, and friends:
> 
> .
>
>   • We have a simple linker wrapper aimed at adding -Wl,-rpath flags:
> 
> .
> The comment in that file explains why the other options considered
> were unsuitable.
>
>   • We modify the built-in “lib” spec of GCC to add the necessary -L and
> -rpath flags:
> 
> .
>
>   • Likewise, we tell Clang where to find libc and friends:
> 
> 
> 
> .
>
> This is not too intrusive and more robust than wrapping everything.
>
> I suppose GCC and Clang could facilitate this by providing configure
> options to augment the “lib” spec, specify the location of libc alone,
> or something along these lines.
>
> Thoughts?
>
> Ludo’.


signature.asc
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment

[Ludovic Courtès]

Hi Shea,

Shea Levy skribis:

> Unlike the traditional approach of installing system libraries into one
> central location like /usr/{lib,include}, the nix package manager [1]
> installs each package into it's own prefix
> (e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
> each package is built in its own environment determined from its
> explicitly listed dependencies, regardless of what else is installed on
> the system. Because not all package build scripts properly respect
> CFLAGS etc., we currently wrap the compiler [2] to respect custom
> environment variables like NIX_CFLAGS_COMPILE, so during the build of a
> package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
> to "-isystem 
> /nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include -isystem 
> /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".
>
> Unfortunately, as you can see if you click through the link or look
> through the git history, the wrapper is quite complex (frankly, hacky)

> [2]: 
> https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh

Guix avoids the compiler wrapper altogether like this:

  • We use C_INCLUDE_PATH, LIBRARY_PATH, and friends:
.

  • We have a simple linker wrapper aimed at adding -Wl,-rpath flags:

.
The comment in that file explains why the other options considered
were unsuitable.

  • We modify the built-in “lib” spec of GCC to add the necessary -L and
-rpath flags:
.

  • Likewise, we tell Clang where to find libc and friends:


.

This is not too intrusive and more robust than wrapping everything.

I suppose GCC and Clang could facilitate this by providing configure
options to augment the “lib” spec, specify the location of libc alone,
or something along these lines.

Thoughts?

Ludo’.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] [RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment

[Shea Levy]

Hello GCC and Clang devs!

Unlike the traditional approach of installing system libraries into one
central location like /usr/{lib,include}, the nix package manager [1]
installs each package into it's own prefix
(e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
each package is built in its own environment determined from its
explicitly listed dependencies, regardless of what else is installed on
the system. Because not all package build scripts properly respect
CFLAGS etc., we currently wrap the compiler [2] to respect custom
environment variables like NIX_CFLAGS_COMPILE, so during the build of a
package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
to "-isystem 
/nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include -isystem 
/nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".

Unfortunately, as you can see if you click through the link or look
through the git history, the wrapper is quite complex (frankly, hacky)
and has evolved mostly through trial and error. Moreover, it's known to
break things like response files [3] and is generally speaking a source
of frustration. I believe the situation would be much improved if gcc
and clang supported some form of "environment-specific" configuration,
either through environment variables or, if absolutely necessary,
command line flags that can be passed unconditionally (note, for
example, that we currently parse the cc command line to see if we're
going to do any linking before deciding to pass in linking-specific
options) and clearly have the semantics we want. Ideally we would be
able to specify something on the level of abstraction of "this directory
should be treated like you would normally treat /usr" and get
e.g. /include, /lib, frameworks on OS X, etc. handled properly.

Would patches aimed at achieving this be considered for inclusion
upstream? My current thought for a first step would be an environment
variable specifying a file with command line flags that are ignored by
the compiler driver in contexts where they are inapplicable or
overridden by other command line flags, but I'm definitely open to
guidance on how this should best be achieved from your perspective. I'm
happy to do the work needed to get this in place if there is interest,
please let me know!

Thanks,
Shea Levy

[1]: https://nixos.org/nix
[2]: 
https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh
[3]: 
https://github.com/NixOS/nixpkgs/commit/a421e7bd4a28c69bded8b17888325e31554f61a1


signature.asc
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev