Re: FYI: what it takes for RAM+swap to build devel/llvm40 with 4 processors or cores and WITH__DEBUG= (powerpc64 example)

[Dimitry Andric]

On 27 Mar 2017, at 23:11, Mark Millard  wrote:
> 
> On 2017-Mar-27, at 5:53 AM, Dimitry Andric  wrote:
>> On 27 Mar 2017, at 12:25, Mark Millard  wrote:
>>> 
>>> On 2017-Mar-27, at 2:41 AM, Dimitry Andric  wrote:
 On 26 Mar 2017, at 23:36, Mark Millard  wrote:
>> ...
> Installed packages to be REMOVED:
>   llvm40-4.0.0.r4
> 
> Number of packages to be removed: 1
> 
> The operation will free 49 GiB.
 
 Yes, this is big.  But there is no real need to build the llvm ports
 with debug information, unless you want to hack on llvm itself.  And
 in that case, you are better served by a Subversion checkout or Git
 clone from upstream instead.
>> ...
>>> Historically unless something extreme like this ends up
>>> involved I build everything using WITH_DEBUG=  or explicit
>>> -g's in order to have better information on any failure.
>> 
>> The problem with the ports implementation of WITH_DEBUG is that it
>> always disables all optimizations, without a possibility to override.
>> Which bloats the resulting object files, libraries and executables, and
>> especially so for large C++ projects such as LLVM.
>> 
>> I can recommend the following workaround.  If you want to build a port
>> with optimizations disabled, you can always pass -O0 in CFLAGS.
>> 
>> -Dimitry
>> 
>> Index: Mk/bsd.port.mk
>> ===
>> --- Mk/bsd.port.mk   (revision 436685)
>> +++ Mk/bsd.port.mk   (working copy)
>> @@ -1646,7 +1646,7 @@ MAKE_ENV+= DONTSTRIP=yes
>> STRIP_CMD=   ${TRUE}
>> .endif
>> DEBUG_FLAGS?=-g
>> -CFLAGS:=${CFLAGS:N-O*:N-fno-strict*} ${DEBUG_FLAGS}
>> +CFLAGS:=${CFLAGS} ${DEBUG_FLAGS}
>> .if defined(INSTALL_TARGET)
>> INSTALL_TARGET:= ${INSTALL_TARGET:S/^install-strip$/install/g}
>> .endif
> 
> Interesting. WITH_DEBUG's description in the file does not
> mention that stripping of optimization flags:
> 
> # WITH_DEBUG- If set, debugging flags are added to CFLAGS and the
> # binaries don't get stripped by INSTALL_PROGRAM or
> # INSTALL_LIB. Besides, individual ports might
> # add their specific to produce binaries for debugging
> # purposes. You can override the debug flags that are
> # passed to the compiler by setting DEBUG_FLAGS. It is
> # set to "-g" at default.
> 
> I'll probably give myself an override that I can specify in
> /etc/make.conf , such as:
> 
> # svnlite diff /usr/ports/Mk/bsd.port.mk
> Index: /usr/ports/Mk/bsd.port.mk
> ===
> --- /usr/ports/Mk/bsd.port.mk (revision 436747)
> +++ /usr/ports/Mk/bsd.port.mk (working copy)
> @@ -1646,7 +1646,11 @@
> STRIP_CMD=${TRUE}
> .endif
> DEBUG_FLAGS?= -g
> +.if defined(ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG)
> +CFLAGS:= ${CFLAGS} ${DEBUG_FLAGS}
> +.else
> CFLAGS:=  ${CFLAGS:N-O*:N-fno-strict*} ${DEBUG_FLAGS}
> +.endif
> .if defined(INSTALL_TARGET)
> INSTALL_TARGET:=  ${INSTALL_TARGET:S/^install-strip$/install/g}
> .endif

Effectively, this gives some sort of support for three of CMake's build
types, e.g:
* Debug, which disables optimization, and obviously adds debuginfo
* Release, which optimizes for speed, and does not add debuginfo
* RelWithDebInfo, similar to Release but does add debuginfo

It would be nice if there was some way of directly utilizing these.  The
RelWithDebInfo target is very useful with the LLVM projects.

-Dimitry



signature.asc
Description: Message signed with OpenPGP

Re: FYI: what it takes for RAM+swap to build devel/llvm40 with 4 processors or cores and WITH__DEBUG= (powerpc64 example)

[Mark Millard]

On 2017-Mar-27, at 5:53 AM, Dimitry Andric  wrote:

> On 27 Mar 2017, at 12:25, Mark Millard  wrote:
>> 
>> On 2017-Mar-27, at 2:41 AM, Dimitry Andric  wrote:
>>> On 26 Mar 2017, at 23:36, Mark Millard  wrote:
> ...
 Installed packages to be REMOVED:
llvm40-4.0.0.r4
 
 Number of packages to be removed: 1
 
 The operation will free 49 GiB.
>>> 
>>> Yes, this is big.  But there is no real need to build the llvm ports
>>> with debug information, unless you want to hack on llvm itself.  And
>>> in that case, you are better served by a Subversion checkout or Git
>>> clone from upstream instead.
> ...
>> Historically unless something extreme like this ends up
>> involved I build everything using WITH_DEBUG=  or explicit
>> -g's in order to have better information on any failure.
> 
> The problem with the ports implementation of WITH_DEBUG is that it
> always disables all optimizations, without a possibility to override.
> Which bloats the resulting object files, libraries and executables, and
> especially so for large C++ projects such as LLVM.
> 
> I can recommend the following workaround.  If you want to build a port
> with optimizations disabled, you can always pass -O0 in CFLAGS.
> 
> -Dimitry
> 
> Index: Mk/bsd.port.mk
> ===
> --- Mk/bsd.port.mk(revision 436685)
> +++ Mk/bsd.port.mk(working copy)
> @@ -1646,7 +1646,7 @@ MAKE_ENV+=  DONTSTRIP=yes
> STRIP_CMD=${TRUE}
> .endif
> DEBUG_FLAGS?= -g
> -CFLAGS:= ${CFLAGS:N-O*:N-fno-strict*} ${DEBUG_FLAGS}
> +CFLAGS:= ${CFLAGS} ${DEBUG_FLAGS}
> .if defined(INSTALL_TARGET)
> INSTALL_TARGET:=  ${INSTALL_TARGET:S/^install-strip$/install/g}
> .endif

Interesting. WITH_DEBUG's description in the file does not
mention that stripping of optimization flags:

# WITH_DEBUG- If set, debugging flags are added to CFLAGS and the
# binaries don't get stripped by INSTALL_PROGRAM or
# INSTALL_LIB. Besides, individual ports might
# add their specific to produce binaries for debugging
# purposes. You can override the debug flags that are
# passed to the compiler by setting DEBUG_FLAGS. It is
# set to "-g" at default.

I'll probably give myself an override that I can specify in
/etc/make.conf , such as:

# svnlite diff /usr/ports/Mk/bsd.port.mk
Index: /usr/ports/Mk/bsd.port.mk
===
--- /usr/ports/Mk/bsd.port.mk   (revision 436747)
+++ /usr/ports/Mk/bsd.port.mk   (working copy)
@@ -1646,7 +1646,11 @@
 STRIP_CMD= ${TRUE}
 .endif
 DEBUG_FLAGS?=  -g
+.if defined(ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG)
+CFLAGS:=   ${CFLAGS} ${DEBUG_FLAGS}
+.else
 CFLAGS:=   ${CFLAGS:N-O*:N-fno-strict*} ${DEBUG_FLAGS}
+.endif
 .if defined(INSTALL_TARGET)
 INSTALL_TARGET:=   ${INSTALL_TARGET:S/^install-strip$/install/g}
 .endif


===
Mark Millard
markmi at dsl-only.net

___
freebsd-toolchain@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: FYI: what it takes for RAM+swap to build devel/llvm40 with 4 processors or cores and WITH__DEBUG= (powerpc64 example)

[Dimitry Andric]

On 27 Mar 2017, at 12:25, Mark Millard  wrote:
> 
> On 2017-Mar-27, at 2:41 AM, Dimitry Andric  wrote:
>> On 26 Mar 2017, at 23:36, Mark Millard  wrote:
...
>>> Installed packages to be REMOVED:
>>> llvm40-4.0.0.r4
>>> 
>>> Number of packages to be removed: 1
>>> 
>>> The operation will free 49 GiB.
>> 
>> Yes, this is big.  But there is no real need to build the llvm ports
>> with debug information, unless you want to hack on llvm itself.  And
>> in that case, you are better served by a Subversion checkout or Git
>> clone from upstream instead.
...
> Historically unless something extreme like this ends up
> involved I build everything using WITH_DEBUG=  or explicit
> -g's in order to have better information on any failure.

The problem with the ports implementation of WITH_DEBUG is that it
always disables all optimizations, without a possibility to override.
Which bloats the resulting object files, libraries and executables, and
especially so for large C++ projects such as LLVM.

I can recommend the following workaround.  If you want to build a port
with optimizations disabled, you can always pass -O0 in CFLAGS.

-Dimitry

Index: Mk/bsd.port.mk
===
--- Mk/bsd.port.mk  (revision 436685)
+++ Mk/bsd.port.mk  (working copy)
@@ -1646,7 +1646,7 @@ MAKE_ENV+=DONTSTRIP=yes
 STRIP_CMD= ${TRUE}
 .endif
 DEBUG_FLAGS?=  -g
-CFLAGS:=   ${CFLAGS:N-O*:N-fno-strict*} ${DEBUG_FLAGS}
+CFLAGS:=   ${CFLAGS} ${DEBUG_FLAGS}
 .if defined(INSTALL_TARGET)
 INSTALL_TARGET:=   ${INSTALL_TARGET:S/^install-strip$/install/g}
 .endif



signature.asc
Description: Message signed with OpenPGP

Re: FYI: what it takes for RAM+swap to build devel/llvm40 with 4 processors or cores and WITH__DEBUG= (powerpc64 example)

[Mark Millard]

On 2017-Mar-27, at 3:25 AM, Mark Millard  wrote:

> On 2017-Mar-27, at 2:41 AM, Dimitry Andric  wrote:
> 
>> On 26 Mar 2017, at 23:36, Mark Millard  wrote:
>>> 
>>> I upgraded from llvm40 r4 to final. An interesting result was
>>> its creation of a backup package for llvm40-4.0.0.r4:
>>> 
>>> about 13 cpu-core-hours running pkg create
>>> 
>>> (Remember: I've been building with WITH_DEBUG= ) Its
>>> single-threaded status stands out via elapsed time
>>> approximately matching.
>>> 
>>> I'll note that it was somewhat under 6 elapsed hours for
>>> staging to have been populated (-j4 with 4 cores present
>>> helps for this part).
>>> 
>>> (Of course these elapsed-time figures are rather system
>>> dependent, although the ratio might be more stable.)
>>> 
>>> 
>>> 
>>> Also interesting was:
>>> 
>>> Installed packages to be REMOVED:
>>> llvm40-4.0.0.r4
>>> 
>>> Number of packages to be removed: 1
>>> 
>>> The operation will free 49 GiB.
>> 
>> Yes, this is big.  But there is no real need to build the llvm ports
>> with debug information, unless you want to hack on llvm itself.  And
>> in that case, you are better served by a Subversion checkout or Git
>> clone from upstream instead.
>> 
>> -Dimitry
> 
> FYI:
> 
> Historically unless something extreme like this ends up
> involved I build everything using WITH_DEBUG=  or explicit
> -g's in order to have better information on any failure.
> 
> This is extreme enough that next time I synchronize
> /usr/ports and it has a devel/llvm40 update I'll
> likely rebuild devel/llvm40 without using WITH_DEBUG= .
> I'm more concerned with the time it takes than with
> the file system space involved.

[Bad example of a incomplete thought. . .]

That last presumes a hardware environment with lots
of RAM (such as 16 GiBytes) --which I usually do not
have access to. Having such is why the report was
from a powerpc64 context: that is where I've access
to that much RAM for FreeBSD.

===
Mark Millard
markmi at dsl-only.net

___
freebsd-toolchain@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"

Re: FYI: what it takes for RAM+swap to build devel/llvm40 with 4 processors or cores and WITH__DEBUG= (powerpc64 example)

[Mark Millard]

On 2017-Mar-27, at 2:41 AM, Dimitry Andric  wrote:

> On 26 Mar 2017, at 23:36, Mark Millard  wrote:
>> 
>> I upgraded from llvm40 r4 to final. An interesting result was
>> its creation of a backup package for llvm40-4.0.0.r4:
>> 
>> about 13 cpu-core-hours running pkg create
>> 
>> (Remember: I've been building with WITH_DEBUG= ) Its
>> single-threaded status stands out via elapsed time
>> approximately matching.
>> 
>> I'll note that it was somewhat under 6 elapsed hours for
>> staging to have been populated (-j4 with 4 cores present
>> helps for this part).
>> 
>> (Of course these elapsed-time figures are rather system
>> dependent, although the ratio might be more stable.)
>> 
>> 
>> 
>> Also interesting was:
>> 
>> Installed packages to be REMOVED:
>>  llvm40-4.0.0.r4
>> 
>> Number of packages to be removed: 1
>> 
>> The operation will free 49 GiB.
> 
> Yes, this is big.  But there is no real need to build the llvm ports
> with debug information, unless you want to hack on llvm itself.  And
> in that case, you are better served by a Subversion checkout or Git
> clone from upstream instead.
> 
> -Dimitry

FYI:

Historically unless something extreme like this ends up
involved I build everything using WITH_DEBUG=  or explicit
-g's in order to have better information on any failure.

This is extreme enough that next time I synchronize
/usr/ports and it has a devel/llvm40 update I'll
likely rebuild devel/llvm40 without using WITH_DEBUG= .
I'm more concerned with the time it takes than with
the file system space involved.

===
Mark Millard
markmi at dsl-only.net

___
freebsd-toolchain@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"