Re: #55884: mandoc @1.14.3: not using the right compiler

[Ryan Schmidt]

On Mar 3, 2018, at 15:14, Jan Stary wrote:

> On Feb 28 08:03:31, Ken Cunningham wrote:
>> The basic principle is that a build of a port on any darwin_N system
>> anywhere will be identical to every other build on that darwin version
>> on all other machines.
> 
> Does that mean that MP will set the same configure.cc for mandoc
> on e.g. every Darwin 17.4.0?

It will set it to /usr/bin/clang.

> What of I have two of those, with different Xcode on each?

Then the version of clang may differ slightly. If a port requires a specific 
version of clang, it can so indicate using the compiler_blacklist_versions 
portgroup.

> Can you please indly point me to the place in MP base
> where the value of configure.cc get decided?

portconfigure.tcl

>> To do that, but build environment (and the runtime environment)
>> need to be fully pre-defined.
> 
> As long as it's possible to have different Xcode versions on one darwin_N,
> in what sense can the build environment be "fully pre-defined"? For example,
> https://trac.macports.org/wiki/UsingTheRightCompiler says:
> 
>For example, with Xcode 4.2 and later, CC is set to
>"/usr/bin/clang" so that there is no ambiguity.
> 
> "/usr/bin/clang" will be a different compiler with Xcode 4.2
> and with Xcode 4.N

That's true. We currently do not completely lock down the compiler version. If 
we want 100% reproducible builds, we may have to change our strategy for this 
and many other things. But for now, this is what we do. Reproducible builds are 
a long-term goal that we will work towards over the coming years.

In Xcode 3 and earlier, gcc was used. When clang was introduced as an optional 
compiler in Xcode 3.2, and made the default in Xcode 4.2 and later, we had 
*many* ports that would not build with clang. It was necessary for the port to 
be able to blacklist clang, so that MacPorts base could pick the next best 
compiler to use. For this to work, the build system needs to use the compiler 
MacPorts tells it to.

>> See here:  for more info,
>> and why this is what we need. I hope you can come to accept
>> some of these MacPorts policies
> 
> This makes sense, unlike what's being said at
> https://trac.macports.org/wiki/UsingTheRightCompiler
> 
> Are these policies clearly described somewhere?

Apologies if the wording of that wiki page is unclear. I wrote it a long time 
ago, before reproducible builds were a goal. I remember struggling to write 
that page, and it getting much longer than I had originally intended, as I 
tried to communicate the problem clearly, and I guess I still didn't succeed. 
Improvements to wiki pages are welcome.

> On Feb 28 10:20:56, Ryan Schmidt wrote:
 I had my MacPorts configured to error out if `cc` is used
 using the method described at  UsingTheRightCompiler#testing.
>>> 
>>> Why would you do that?
>> 
>> To detect ports that do not use a specific compiler.
>> MacPorts policy is that all ports shall use a specific compiler.
> 
> Is this policy described somewhere?

I don't know. We've just described it to you...

>>> BTW, this is how configure.cc is described in the guide:
>>> 
>>> configure.cc
>>>   Set CC compiler flags for selecting a C compiler.
>>>   Default: ???
>>>   Example:
>>> 
>>> "compiler flags for selecting a compiler"?
>> 
>> Rainer has now adjusted these descriptions in the guide.
> 
> https://guide.macports.org/chunked/reference.phases.html still says
> "Set CC compiler flags for selecting a C compiler."

There is currently no automated process to deploy guide updates. I need to add 
those tasks to the buildbot. Until I do, Rainer or Clemens must manually run a 
task on the server to update the guide.


> On Feb 28 17:36:38, Rainer Mueller wrote:
>> On 2018-02-28 08:52, Jan Stary wrote:
>>> The ticket is closed, so I will continue here.
>>> 
 Actually it's not specific to the universal variant. Rather, the problem
 is that the port is not UsingTheRightCompiler.
>>> 
>>> mandoc's ./configure sets CC to whatever make(1) thinks CC is:
>>> CC=`printf "all:\\n\\t@echo \\\$(CC)\\n" | env -i make -sf -`
>>> Why exactly is it "not the right compiler"?
>> 
>> As env -i clears the environment,
>> how is this supposed to work at all?
>> This does not respect CC in the environment, which is what MacPorts
>> usually does.
>> 
>> $ export CC=foo
>> $ printf "all:\\n\\t@echo \\\$(CC)\\n" | env -i make -sf -
>> cc
> 
> That's on purpose: compile mandoc with
> whatever the local make(1) thinks $(CC) is.
> 
>  # You can manually override the compiler to be used.
>  # But that's rarely useful because ./configure asks your make(1)
>  # which compiler to use, and that answer will hardly be wrong.
> 
> mandoc's ./configure does not take CC from the environment.
> It determines it as above. You can override it of course,
> by setting CC=... in configure.local
> 
> That seems to be the issue here: MP _wants_to_ tell mandoc what compiler
> to use, but setting CC in the 

Re: #55884: mandoc @1.14.3: not using the right compiler

[Ken Cunningham]

On 2018-03-03, at 1:14 PM, Jan Stary wrote:
> 
> What of I have two of those, with different Xcode on each?

That is undesirable for reproducible builds. 

Most users have the default Xcode version for a give system, as Apple flushes 
all the updates that way.

If users report broken builds and we recognize non-standard configurations, we 
encourage them to use the default version of Xcode for their system, and 
generally report that the default toolchain installation is the only supported 
configuration. 

Otherwise, chaos. 

> 
> Can you please indly point me to the place in MP base
> where the value of configure.cc get decided?
> 

/opt/local/libexec/macports/lib/port1.0/portconfigure.tcl

Re: #55884: mandoc @1.14.3: not using the right compiler

[Jan Stary]

> > > mandoc's ./configure sets CC to whatever make(1) thinks CC is:
> > > CC=`printf "all:\\n\\t@echo \\\$(CC)\\n" | env -i make -sf -`
> > > Why exactly is it "not the right compiler"?
> > 
> > As env -i clears the environment,
> > how is this supposed to work at all?
> > This does not respect CC in the environment, which is what MacPorts
> > usually does.
> > 
> > $ export CC=foo
> > $ printf "all:\\n\\t@echo \\\$(CC)\\n" | env -i make -sf -
> > cc
> 
> That's on purpose: compile mandoc with
> whatever the local make(1) thinks $(CC) is.
> 
>   # You can manually override the compiler to be used.
>   # But that's rarely useful because ./configure asks your make(1)
>   # which compiler to use, and that answer will hardly be wrong.
> 
> mandoc's ./configure does not take CC from the environment.

In fact, configure says

  # --- default settings -
  # Initialize all variables here,
  # such that nothing can leak in from the environment.

> It determines it as above. You can override it of course,
> by setting CC=... in configure.local

Re: #55884: mandoc @1.14.3: not using the right compiler

[Jan Stary]

On Feb 28 08:03:31, ken.cunningham.web...@gmail.com wrote:
> The basic principle is that a build of a port on any darwin_N system
> anywhere will be identical to every other build on that darwin version
> on all other machines.

Does that mean that MP will set the same configure.cc for mandoc
on e.g. every Darwin 17.4.0? What of I have two of those,
with different Xcode on each?

Can you please indly point me to the place in MP base
where the value of configure.cc get decided?

> To do that, but build environment (and the runtime environment)
> need to be fully pre-defined.

As long as it's possible to have different Xcode versions on one darwin_N,
in what sense can the build environment be "fully pre-defined"? For example,
https://trac.macports.org/wiki/UsingTheRightCompiler says:

 For example, with Xcode 4.2 and later, CC is set to
 "/usr/bin/clang" so that there is no ambiguity.

"/usr/bin/clang" will be a different compiler with Xcode 4.2
and with Xcode 4.N


> See here:  for more info,
> and why this is what we need. I hope you can come to accept
> some of these MacPorts policies

This makes sense, unlike what's being said at
https://trac.macports.org/wiki/UsingTheRightCompiler

Are these policies clearly described somewhere?


On Feb 28 10:20:56, ryandes...@macports.org wrote:
> >> I had my MacPorts configured to error out if `cc` is used
> >> using the method described at  UsingTheRightCompiler#testing.
> > 
> > Why would you do that?
> 
> To detect ports that do not use a specific compiler.
> MacPorts policy is that all ports shall use a specific compiler.

Is this policy described somewhere?

> > BTW, this is how configure.cc is described in the guide:
> > 
> >  configure.cc
> >Set CC compiler flags for selecting a C compiler.
> >Default: ???
> >Example:
> > 
> > "compiler flags for selecting a compiler"?
> 
> Rainer has now adjusted these descriptions in the guide.

https://guide.macports.org/chunked/reference.phases.html still says
"Set CC compiler flags for selecting a C compiler."


On Feb 28 17:36:38, rai...@macports.org wrote:
> On 2018-02-28 08:52, Jan Stary wrote:
> > The ticket is closed, so I will continue here.
> > 
> >>  Actually it's not specific to the universal variant. Rather, the problem
> >>  is that the port is not UsingTheRightCompiler.
> > 
> > mandoc's ./configure sets CC to whatever make(1) thinks CC is:
> > CC=`printf "all:\\n\\t@echo \\\$(CC)\\n" | env -i make -sf -`
> > Why exactly is it "not the right compiler"?
> 
> As env -i clears the environment,
> how is this supposed to work at all?
> This does not respect CC in the environment, which is what MacPorts
> usually does.
> 
> $ export CC=foo
> $ printf "all:\\n\\t@echo \\\$(CC)\\n" | env -i make -sf -
> cc

That's on purpose: compile mandoc with
whatever the local make(1) thinks $(CC) is.

  # You can manually override the compiler to be used.
  # But that's rarely useful because ./configure asks your make(1)
  # which compiler to use, and that answer will hardly be wrong.

mandoc's ./configure does not take CC from the environment.
It determines it as above. You can override it of course,
by setting CC=... in configure.local

That seems to be the issue here: MP _wants_to_ tell mandoc what compiler
to use, but setting CC in the environment does not do it. Hence the CC
dance that we have in the Portfile now. Is this understanding correct?


> > It's true that /usr/bin/cc is a symlink to some suitable default compiler.
> > So call that, thus calling a suitable default compiler, _by_default_.
> 
> You are already assuming that 'cc' is /usr/bin/cc. If anyone has a
> /opt/local/bin/cc, it might be a different compiler.

If it is actually a C compiler, it will compile mandoc just fine.
If it is instead a shell script that errors out with "you must not run
this as your compiler", then it will not of course.

> In general, we do not want to use what the user selected as their
> preferred compiler, but what MacPorts selected to compile this port.

Is this strict policy clearly described somewhere, including the
rationale? Not in https://trac.macports.org/wiki/UsingTheRightCompiler

> > Yes, there are ports that do need to call some particular compiler,
> > but it's those ports' job then to find and use that.

> >> I had my MacPorts configured to error out if `cc` is used
> >> using the method described at  UsingTheRightCompiler#testing.
> > 
> > Why would you do that?
> >
> > Mandoc is extremely portable. I have yet to see a UNIX system where
> > it does not build. Except here: of course it "breaks" if you explicitly
> > make it impossible to run 'cc', which is the right compiler.
> 
> This setup is a good idea for developers to catch ports that are not
> respecting the environment variables for compiler selection.

This seems to confirm that this is what's "wrong" with mandoc in this
respect - it does not listen to MP's CC setting and has to be told
explicitly.

> 

Re: #55884: mandoc @1.14.3: not using the right compiler

[Rainer Müller]

On 2018-02-28 08:52, Jan Stary wrote:
> The ticket is closed, so I will continue here.
> 
>>  Actually it's not specific to the universal variant. Rather, the problem
>>  is that the port is not UsingTheRightCompiler.
> 
> mandoc's ./configure sets CC to whatever make(1) thinks CC is:
> CC=`printf "all:\\n\\t@echo \\\$(CC)\\n" | env -i make -sf -`
> Why exactly is it "not the right compiler"?

As env -i clears the environment, how is this supposed to work at all?
This does not respect CC in the environment, which is what MacPorts
usually does.

$ export CC=foo
$ printf "all:\\n\\t@echo \\\$(CC)\\n" | env -i make -sf -
cc

> Maybe I don't get the whole premise:
> 
>   If not instructed otherwise, most software builds C code using "cc" or
>   "gcc". But "/usr/bin/cc" and "/usr/bin/gcc" are not specific
>   compilers; they're symlinks to some suitable default compiler, but it
>   varies based on Xcode version. Users might also have created
>   a "/opt/local/bin/gcc" symlink, pointing to any installed compiler, by
>   using "port select --set gcc". So we cannot rely on "cc" oc "gcc"
>   being any particular compiler. 
> 
> Yes, we cannot, and that's OK because we should not, _by_default_.
> This is a UNIX system compiling C code. Call 'cc'. Duh.
> 
> It's true that /usr/bin/cc is a symlink to some suitable default compiler.
> So call that, thus calling a suitable default compiler, _by_default_.

You are already assuming that 'cc' is /usr/bin/cc. If anyone has a
/opt/local/bin/cc, it might be a different compiler.

In general, we do not want to use what the user selected as their
preferred compiler, but what MacPorts selected to compile this port.
This is more important for binary names such as cpp or c++, which are
actually modified by 'port select gcc'.

> Yes, there are ports that do need to call some particular compiler,
> but it's those ports' job then to find and use that.

Unfortunately, it is not that easy. Ports need to respect what is set in
the environment. Using the right compiler is especially important when
compiling C++, as that also controls which standard library will be
linked. Historically, Apple used to ship gcc, llvm-gcc, and clang and on
some macOS version.

>> I had my MacPorts configured to error out if `cc` is used
>> using the method described at  UsingTheRightCompiler#testing.
> 
> Why would you do that?
>
> Mandoc is extremely portable. I have yet to see a UNIX system where
> it does not build. Except here: of course it "breaks" if you explicitly
> make it impossible to run 'cc', which is the right compiler.

This setup is a good idea for developers to catch ports that are not
respecting the environment variables for compiler selection.

> BTW, this is how configure.cc is described in the guide:
> 
>   configure.cc
> Set CC compiler flags for selecting a C compiler.
> Default: ???
> Example:
> 
> "compiler flags for selecting a compiler"?

Looks like an old copy & paste mistake to me. I committed a fix. [1]
Ryan was even faster to notice than me typing this mail. :-)

>> Perhaps the portfile code that set `CC`
>>  that was removed in [14f7de9e9ad521cce6389678675706e8593b9834/macports-
>>  ports] was needed after all.
> 
> It is only "needed" if you make it artificially impossible
> to call 'cc' (or whatever the port ./configures to be CC).
> 
> So: because ports must not call 'cc' on your system,
> let's complicate the mandoc Portfile for everyone.

Build systems are supposed to follow CC in the environment. The whole
point of options like configure.cc is that they are respected when
building the port.

As an alternative to pass compiler in environment variables would be
generating wrapper scripts that use the right compiler as selected by
MacPorts. This is the way other projects like Fink do it. Then the usual
names cc/gcc/clang/... will always end up at the intended binary by
putting them to the front of PATH. However, this is a long-standing
issue and I am not aware anyone was working on this. Therefore, build
systems still need to be patched to honor CC from the environment.

Rainer

[1]
https://github.com/macports/macports-guide/commit/5a6f3a7f7ca016f22f8ac7aa0f497860680302de

Re: #55884: mandoc @1.14.3: not using the right compiler

[Ryan Schmidt]

On Feb 28, 2018, at 01:52, Jan Stary wrote:

>> I had my MacPorts configured to error out if `cc` is used
>> using the method described at  UsingTheRightCompiler#testing.
> 
> Why would you do that?

To detect ports that do not use a specific compiler. MacPorts policy is that 
all ports shall use a specific compiler.


> BTW, this is how configure.cc is described in the guide:
> 
>  configure.cc
>Set CC compiler flags for selecting a C compiler.
>Default: ???
>Example:
> 
> "compiler flags for selecting a compiler"?

Rainer has now adjusted these descriptions in the guide.