Re: svn commit: r367577 - in head: share/mk sys/conf tools/build/options

2020-11-10 Thread Brooks Davis 
On Tue, Nov 10, 2020 at 02:44:45PM -0500, Shawn Webb wrote:
> On Tue, Nov 10, 2020 at 07:17:29PM +, Brooks Davis wrote:
> > On Tue, Nov 10, 2020 at 07:15:14PM +, Brooks Davis wrote:
> > > Author: brooks
> > > Date: Tue Nov 10 19:15:13 2020
> > > New Revision: 367577
> > > URL: https://svnweb.freebsd.org/changeset/base/367577
> > > 
> > > Log:
> > >   Support initializing stack variables on function entry
> > >   
> > >   There are two options:
> > >- WITH_INIT_ALL_ZERO: Zero all variables on the stack.
> > >- WITH_INIT_ALL_PATTERN: Initialize variables with well-defined 
> > > patterns.
> > >   
> > >   The exact pattern are a compiler implementation detail and vary by type.
> > >   They are somewhat documented in the LLVM commit message:
> > >   https://reviews.llvm.org/rL349442
> > >   I've used WITH_INIT_ALL_* to match Microsoft's InitAll feature rather
> > >   than naming them after the LLVM specific compiler flags.
> > >   
> > >   In a range of consumer products, options like these are used in
> > >   both debug and production builds with debugs builds using patterns
> > >   (intended to provoke crashes on use of uninitialized values) and
> > >   production using zeros (deemed more likely to lead to harmless
> > >   misbehavior or NULL-pointer dereferences).
> > 
> > We've tested this extensively in CheriBSD on RISC-V, in the wild it's
> > probably most tested on Arm64 and x86.
> > 
> > Despite the silly compiler flag you'll spot in the code, the zeroing
> > option isn't going away in practice as Apple, Google, and Microsoft all
> > ship with this feature in some of their products.
> 
> HardenedBSD's testing of this last year on amd64 have (privately)
> shown the feature to really hinder performance on more complex
> applications (like when applied to clang/lld). A build of base
> without init all zero applied to clang/lld would take around 1.5
> hours on my system. A build with it applied to clang/lld took around
> four hours, if my memory serves correctly. I would probably advise
> against applying it system-wide. But YMMV.

I agree a more nuanced approach is likely useful in practice, but this
does work and is part of the configuration we shipped for DARPA's FETT bug
bounty.  Hopefully this provides a starting point for further
exploration.

-- Brooks


signature.asc
Description: PGP signature

Re: svn commit: r367577 - in head: share/mk sys/conf tools/build/options

2020-11-10 Thread Brooks Davis 
Sorry about that.  I've fixed it in r367579.

-- Brooks

On Tue, Nov 10, 2020 at 12:46:45PM -0800, Matthew Macy wrote:
> These flags aren't defined by default when building external kernel modules:
> 
> gmake[2]: Entering directory '/usr/home/matt/devel/ZoF/module'
> env -u MAKEFLAGS make -C /home/matt/devel/ZoF/module -f Makefile.bsd -w
> make[3]: Entering directory `/home/matt/devel/ZoF/module'
> make[3]: "/usr/home/matt/devel/freebsd/sys/conf/kern.mk" line 233:
> Malformed conditional (${MK_INIT_ALL_ZERO} == "yes")
> make[3]: Fatal errors encountered -- cannot continue
> make[3]: stopped in /home/matt/devel/ZoF/module
> gmake[2]: *** [Makefile:53: modules-FreeBSD] Error 1
> gmake[2]: Leaving directory '/usr/home/matt/devel/ZoF/module'
> 
> On Tue, Nov 10, 2020 at 11:15 AM Brooks Davis  wrote:
> >
> > Author: brooks
> > Date: Tue Nov 10 19:15:13 2020
> > New Revision: 367577
> > URL: https://svnweb.freebsd.org/changeset/base/367577
> >
> > Log:
> >   Support initializing stack variables on function entry
> >
> >   There are two options:
> >- WITH_INIT_ALL_ZERO: Zero all variables on the stack.
> >- WITH_INIT_ALL_PATTERN: Initialize variables with well-defined patterns.
> >
> >   The exact pattern are a compiler implementation detail and vary by type.
> >   They are somewhat documented in the LLVM commit message:
> >   https://reviews.llvm.org/rL349442
> >   I've used WITH_INIT_ALL_* to match Microsoft's InitAll feature rather
> >   than naming them after the LLVM specific compiler flags.
> >
> >   In a range of consumer products, options like these are used in
> >   both debug and production builds with debugs builds using patterns
> >   (intended to provoke crashes on use of uninitialized values) and
> >   production using zeros (deemed more likely to lead to harmless
> >   misbehavior or NULL-pointer dereferences).
> >
> >   Reviewed by:  emaste
> >   Obtained from:CheriBSD
> >   Sponsored by: DARPA
> >   Differential Revision:https://reviews.freebsd.org/D27131
> >
> > Added:
> >   head/tools/build/options/WITH_INIT_ALL_PATTERN   (contents, props changed)
> >   head/tools/build/options/WITH_INIT_ALL_ZERO   (contents, props changed)
> > Modified:
> >   head/share/mk/bsd.compiler.mk
> >   head/share/mk/bsd.lib.mk
> >   head/share/mk/bsd.opts.mk
> >   head/share/mk/bsd.prog.mk
> >   head/sys/conf/kern.mk
> >
> > Modified: head/share/mk/bsd.compiler.mk
> > ==
> > --- head/share/mk/bsd.compiler.mk   Tue Nov 10 19:09:35 2020
> > (r367576)
> > +++ head/share/mk/bsd.compiler.mk   Tue Nov 10 19:15:13 2020
> > (r367577)
> > @@ -24,6 +24,7 @@
> >  # - c++11: supports full (or nearly full) C++11 programming 
> > environment.
> >  # - retpoline: supports the retpoline speculative execution vulnerability
> >  #  mitigation.
> > +# - init-all:  supports stack variable initialization.
> >  #
> >  # These variables with an X_ prefix will also be provided if XCC is set.
> >  #
> > @@ -214,7 +215,7 @@ ${X_}COMPILER_FEATURES= c++11 c++14
> >  ${X_}COMPILER_FEATURES+=   c++17
> >  .endif
> >  .if ${${X_}COMPILER_TYPE} == "clang"
> > -${X_}COMPILER_FEATURES+=   retpoline
> > +${X_}COMPILER_FEATURES+=   retpoline init-all
> >  .endif
> >
> >  .else
> >
> > Modified: head/share/mk/bsd.lib.mk
> > ==
> > --- head/share/mk/bsd.lib.mkTue Nov 10 19:09:35 2020(r367576)
> > +++ head/share/mk/bsd.lib.mkTue Nov 10 19:15:13 2020(r367577)
> > @@ -85,6 +85,25 @@ LDFLAGS+= -Wl,-zretpolineplt
> >  .endif
> >  .endif
> >
> > +# Initialize stack variables on function entry
> > +.if ${MK_INIT_ALL_ZERO} == "yes"
> > +.if ${COMPILER_FEATURES:Minit-all}
> > +CFLAGS+= -ftrivial-auto-var-init=zero \
> > +
> > -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang
> > +CXXFLAGS+= -ftrivial-auto-var-init=zero \
> > +
> > -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang
> > +.else
> > +.warning InitAll (zeros) requested but not support by compiler
> > +.endif
> > +.elif ${MK_INIT_ALL_PATTERN} == "yes"
> > +.if ${COMPILER_FEATURES:Minit-all}
> > +CFLAGS+= -ftrivial-auto-var-init=pattern
> > +CXXFLAGS+= -ftrivial-auto-var-init=pattern
> > +.else
> > +.warning InitAll (pattern) requested but not support by compiler
> > +.endif
> > +.endif
> > +
> >  .if ${MK_DEBUG_FILES} != "no" && empty(DEBUG_FLAGS:M-g) && \
> >  empty(DEBUG_FLAGS:M-gdwarf*)
> >  CFLAGS+= ${DEBUG_FILES_CFLAGS}
> >
> > Modified: head/share/mk/bsd.opts.mk
> > ==
> > --- head/share/mk/bsd.opts.mk   Tue Nov 10 19:09:35 2020(r367576)
> > +++ head/share/mk/bsd.opts.mk   Tue Nov 10 19:15:13 2020(r367577)
> > @@ -71,6 +71,8 @@ __DEFAULT_NO_OPTIONS = \
> >  BIND_NOW \
> >

Re: svn commit: r367577 - in head: share/mk sys/conf tools/build/options

2020-11-10 Thread Matthew Macy 
These flags aren't defined by default when building external kernel modules:

gmake[2]: Entering directory '/usr/home/matt/devel/ZoF/module'
env -u MAKEFLAGS make -C /home/matt/devel/ZoF/module -f Makefile.bsd -w
make[3]: Entering directory `/home/matt/devel/ZoF/module'
make[3]: "/usr/home/matt/devel/freebsd/sys/conf/kern.mk" line 233:
Malformed conditional (${MK_INIT_ALL_ZERO} == "yes")
make[3]: Fatal errors encountered -- cannot continue
make[3]: stopped in /home/matt/devel/ZoF/module
gmake[2]: *** [Makefile:53: modules-FreeBSD] Error 1
gmake[2]: Leaving directory '/usr/home/matt/devel/ZoF/module'

On Tue, Nov 10, 2020 at 11:15 AM Brooks Davis  wrote:
>
> Author: brooks
> Date: Tue Nov 10 19:15:13 2020
> New Revision: 367577
> URL: https://svnweb.freebsd.org/changeset/base/367577
>
> Log:
>   Support initializing stack variables on function entry
>
>   There are two options:
>- WITH_INIT_ALL_ZERO: Zero all variables on the stack.
>- WITH_INIT_ALL_PATTERN: Initialize variables with well-defined patterns.
>
>   The exact pattern are a compiler implementation detail and vary by type.
>   They are somewhat documented in the LLVM commit message:
>   https://reviews.llvm.org/rL349442
>   I've used WITH_INIT_ALL_* to match Microsoft's InitAll feature rather
>   than naming them after the LLVM specific compiler flags.
>
>   In a range of consumer products, options like these are used in
>   both debug and production builds with debugs builds using patterns
>   (intended to provoke crashes on use of uninitialized values) and
>   production using zeros (deemed more likely to lead to harmless
>   misbehavior or NULL-pointer dereferences).
>
>   Reviewed by:  emaste
>   Obtained from:CheriBSD
>   Sponsored by: DARPA
>   Differential Revision:https://reviews.freebsd.org/D27131
>
> Added:
>   head/tools/build/options/WITH_INIT_ALL_PATTERN   (contents, props changed)
>   head/tools/build/options/WITH_INIT_ALL_ZERO   (contents, props changed)
> Modified:
>   head/share/mk/bsd.compiler.mk
>   head/share/mk/bsd.lib.mk
>   head/share/mk/bsd.opts.mk
>   head/share/mk/bsd.prog.mk
>   head/sys/conf/kern.mk
>
> Modified: head/share/mk/bsd.compiler.mk
> ==
> --- head/share/mk/bsd.compiler.mk   Tue Nov 10 19:09:35 2020
> (r367576)
> +++ head/share/mk/bsd.compiler.mk   Tue Nov 10 19:15:13 2020
> (r367577)
> @@ -24,6 +24,7 @@
>  # - c++11: supports full (or nearly full) C++11 programming environment.
>  # - retpoline: supports the retpoline speculative execution vulnerability
>  #  mitigation.
> +# - init-all:  supports stack variable initialization.
>  #
>  # These variables with an X_ prefix will also be provided if XCC is set.
>  #
> @@ -214,7 +215,7 @@ ${X_}COMPILER_FEATURES= c++11 c++14
>  ${X_}COMPILER_FEATURES+=   c++17
>  .endif
>  .if ${${X_}COMPILER_TYPE} == "clang"
> -${X_}COMPILER_FEATURES+=   retpoline
> +${X_}COMPILER_FEATURES+=   retpoline init-all
>  .endif
>
>  .else
>
> Modified: head/share/mk/bsd.lib.mk
> ==
> --- head/share/mk/bsd.lib.mkTue Nov 10 19:09:35 2020(r367576)
> +++ head/share/mk/bsd.lib.mkTue Nov 10 19:15:13 2020(r367577)
> @@ -85,6 +85,25 @@ LDFLAGS+= -Wl,-zretpolineplt
>  .endif
>  .endif
>
> +# Initialize stack variables on function entry
> +.if ${MK_INIT_ALL_ZERO} == "yes"
> +.if ${COMPILER_FEATURES:Minit-all}
> +CFLAGS+= -ftrivial-auto-var-init=zero \
> +-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang
> +CXXFLAGS+= -ftrivial-auto-var-init=zero \
> +-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang
> +.else
> +.warning InitAll (zeros) requested but not support by compiler
> +.endif
> +.elif ${MK_INIT_ALL_PATTERN} == "yes"
> +.if ${COMPILER_FEATURES:Minit-all}
> +CFLAGS+= -ftrivial-auto-var-init=pattern
> +CXXFLAGS+= -ftrivial-auto-var-init=pattern
> +.else
> +.warning InitAll (pattern) requested but not support by compiler
> +.endif
> +.endif
> +
>  .if ${MK_DEBUG_FILES} != "no" && empty(DEBUG_FLAGS:M-g) && \
>  empty(DEBUG_FLAGS:M-gdwarf*)
>  CFLAGS+= ${DEBUG_FILES_CFLAGS}
>
> Modified: head/share/mk/bsd.opts.mk
> ==
> --- head/share/mk/bsd.opts.mk   Tue Nov 10 19:09:35 2020(r367576)
> +++ head/share/mk/bsd.opts.mk   Tue Nov 10 19:15:13 2020(r367577)
> @@ -71,6 +71,8 @@ __DEFAULT_NO_OPTIONS = \
>  BIND_NOW \
>  CCACHE_BUILD \
>  CTF \
> +INIT_ALL_PATTERN \
> +INIT_ALL_ZERO \
>  INSTALL_AS_USER \
>  PIE \
>  RETPOLINE \
> @@ -84,6 +86,10 @@ __DEFAULT_DEPENDENT_OPTIONS = \
>
>
>  .include 
> +
> +.if ${MK_INIT_ALL_PATTERN} == "yes" && ${MK_INIT_ALL_ZERO} == "yes"
> +.error WITH_INIT_ALL_PATTERN and WITH_INIT_ALL_ZERO are mutually exclusive.
> +.endif
>
>  #

Re: svn commit: r367577 - in head: share/mk sys/conf tools/build/options

2020-11-10 Thread Shawn Webb 
On Tue, Nov 10, 2020 at 07:17:29PM +, Brooks Davis wrote:
> On Tue, Nov 10, 2020 at 07:15:14PM +, Brooks Davis wrote:
> > Author: brooks
> > Date: Tue Nov 10 19:15:13 2020
> > New Revision: 367577
> > URL: https://svnweb.freebsd.org/changeset/base/367577
> > 
> > Log:
> >   Support initializing stack variables on function entry
> >   
> >   There are two options:
> >- WITH_INIT_ALL_ZERO: Zero all variables on the stack.
> >- WITH_INIT_ALL_PATTERN: Initialize variables with well-defined patterns.
> >   
> >   The exact pattern are a compiler implementation detail and vary by type.
> >   They are somewhat documented in the LLVM commit message:
> >   https://reviews.llvm.org/rL349442
> >   I've used WITH_INIT_ALL_* to match Microsoft's InitAll feature rather
> >   than naming them after the LLVM specific compiler flags.
> >   
> >   In a range of consumer products, options like these are used in
> >   both debug and production builds with debugs builds using patterns
> >   (intended to provoke crashes on use of uninitialized values) and
> >   production using zeros (deemed more likely to lead to harmless
> >   misbehavior or NULL-pointer dereferences).
> 
> We've tested this extensively in CheriBSD on RISC-V, in the wild it's
> probably most tested on Arm64 and x86.
> 
> Despite the silly compiler flag you'll spot in the code, the zeroing
> option isn't going away in practice as Apple, Google, and Microsoft all
> ship with this feature in some of their products.

HardenedBSD's testing of this last year on amd64 have (privately)
shown the feature to really hinder performance on more complex
applications (like when applied to clang/lld). A build of base
without init all zero applied to clang/lld would take around 1.5
hours on my system. A build with it applied to clang/lld took around
four hours, if my memory serves correctly. I would probably advise
against applying it system-wide. But YMMV.

Thanks,

-- 
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

GPG Key ID:  0xFF2E67A277F8E1FA
GPG Key Fingerprint: D206 BB45 15E0 9C49 0CF9  3633 C85B 0AF8 AB23 0FB2
https://git-01.md.hardenedbsd.org/HardenedBSD/pubkeys/src/branch/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc


signature.asc
Description: PGP signature

Re: svn commit: r367577 - in head: share/mk sys/conf tools/build/options

2020-11-10 Thread Brooks Davis 
On Tue, Nov 10, 2020 at 07:15:14PM +, Brooks Davis wrote:
> Author: brooks
> Date: Tue Nov 10 19:15:13 2020
> New Revision: 367577
> URL: https://svnweb.freebsd.org/changeset/base/367577
> 
> Log:
>   Support initializing stack variables on function entry
>   
>   There are two options:
>- WITH_INIT_ALL_ZERO: Zero all variables on the stack.
>- WITH_INIT_ALL_PATTERN: Initialize variables with well-defined patterns.
>   
>   The exact pattern are a compiler implementation detail and vary by type.
>   They are somewhat documented in the LLVM commit message:
>   https://reviews.llvm.org/rL349442
>   I've used WITH_INIT_ALL_* to match Microsoft's InitAll feature rather
>   than naming them after the LLVM specific compiler flags.
>   
>   In a range of consumer products, options like these are used in
>   both debug and production builds with debugs builds using patterns
>   (intended to provoke crashes on use of uninitialized values) and
>   production using zeros (deemed more likely to lead to harmless
>   misbehavior or NULL-pointer dereferences).

We've tested this extensively in CheriBSD on RISC-V, in the wild it's
probably most tested on Arm64 and x86.

Despite the silly compiler flag you'll spot in the code, the zeroing
option isn't going away in practice as Apple, Google, and Microsoft all
ship with this feature in some of their products.

-- Brooks


signature.asc
Description: PGP signature

svn commit: r367577 - in head: share/mk sys/conf tools/build/options

2020-11-10 Thread Brooks Davis 
Author: brooks
Date: Tue Nov 10 19:15:13 2020
New Revision: 367577
URL: https://svnweb.freebsd.org/changeset/base/367577

Log:
  Support initializing stack variables on function entry
  
  There are two options:
   - WITH_INIT_ALL_ZERO: Zero all variables on the stack.
   - WITH_INIT_ALL_PATTERN: Initialize variables with well-defined patterns.
  
  The exact pattern are a compiler implementation detail and vary by type.
  They are somewhat documented in the LLVM commit message:
  https://reviews.llvm.org/rL349442
  I've used WITH_INIT_ALL_* to match Microsoft's InitAll feature rather
  than naming them after the LLVM specific compiler flags.
  
  In a range of consumer products, options like these are used in
  both debug and production builds with debugs builds using patterns
  (intended to provoke crashes on use of uninitialized values) and
  production using zeros (deemed more likely to lead to harmless
  misbehavior or NULL-pointer dereferences).
  
  Reviewed by:  emaste
  Obtained from:CheriBSD
  Sponsored by: DARPA
  Differential Revision:https://reviews.freebsd.org/D27131

Added:
  head/tools/build/options/WITH_INIT_ALL_PATTERN   (contents, props changed)
  head/tools/build/options/WITH_INIT_ALL_ZERO   (contents, props changed)
Modified:
  head/share/mk/bsd.compiler.mk
  head/share/mk/bsd.lib.mk
  head/share/mk/bsd.opts.mk
  head/share/mk/bsd.prog.mk
  head/sys/conf/kern.mk

Modified: head/share/mk/bsd.compiler.mk
==
--- head/share/mk/bsd.compiler.mk   Tue Nov 10 19:09:35 2020
(r367576)
+++ head/share/mk/bsd.compiler.mk   Tue Nov 10 19:15:13 2020
(r367577)
@@ -24,6 +24,7 @@
 # - c++11: supports full (or nearly full) C++11 programming environment.
 # - retpoline: supports the retpoline speculative execution vulnerability
 #  mitigation.
+# - init-all:  supports stack variable initialization.
 #
 # These variables with an X_ prefix will also be provided if XCC is set.
 #
@@ -214,7 +215,7 @@ ${X_}COMPILER_FEATURES= c++11 c++14
 ${X_}COMPILER_FEATURES+=   c++17
 .endif
 .if ${${X_}COMPILER_TYPE} == "clang"
-${X_}COMPILER_FEATURES+=   retpoline
+${X_}COMPILER_FEATURES+=   retpoline init-all
 .endif
 
 .else

Modified: head/share/mk/bsd.lib.mk
==
--- head/share/mk/bsd.lib.mkTue Nov 10 19:09:35 2020(r367576)
+++ head/share/mk/bsd.lib.mkTue Nov 10 19:15:13 2020(r367577)
@@ -85,6 +85,25 @@ LDFLAGS+= -Wl,-zretpolineplt
 .endif
 .endif
 
+# Initialize stack variables on function entry
+.if ${MK_INIT_ALL_ZERO} == "yes"
+.if ${COMPILER_FEATURES:Minit-all}
+CFLAGS+= -ftrivial-auto-var-init=zero \
+-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang
+CXXFLAGS+= -ftrivial-auto-var-init=zero \
+-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang
+.else
+.warning InitAll (zeros) requested but not support by compiler
+.endif
+.elif ${MK_INIT_ALL_PATTERN} == "yes"
+.if ${COMPILER_FEATURES:Minit-all}
+CFLAGS+= -ftrivial-auto-var-init=pattern
+CXXFLAGS+= -ftrivial-auto-var-init=pattern
+.else
+.warning InitAll (pattern) requested but not support by compiler
+.endif
+.endif
+
 .if ${MK_DEBUG_FILES} != "no" && empty(DEBUG_FLAGS:M-g) && \
 empty(DEBUG_FLAGS:M-gdwarf*)
 CFLAGS+= ${DEBUG_FILES_CFLAGS}

Modified: head/share/mk/bsd.opts.mk
==
--- head/share/mk/bsd.opts.mk   Tue Nov 10 19:09:35 2020(r367576)
+++ head/share/mk/bsd.opts.mk   Tue Nov 10 19:15:13 2020(r367577)
@@ -71,6 +71,8 @@ __DEFAULT_NO_OPTIONS = \
 BIND_NOW \
 CCACHE_BUILD \
 CTF \
+INIT_ALL_PATTERN \
+INIT_ALL_ZERO \
 INSTALL_AS_USER \
 PIE \
 RETPOLINE \
@@ -84,6 +86,10 @@ __DEFAULT_DEPENDENT_OPTIONS = \
 
 
 .include 
+
+.if ${MK_INIT_ALL_PATTERN} == "yes" && ${MK_INIT_ALL_ZERO} == "yes"
+.error WITH_INIT_ALL_PATTERN and WITH_INIT_ALL_ZERO are mutually exclusive.
+.endif
 
 #
 # Supported NO_* options (if defined, MK_* will be forced to "no",

Modified: head/share/mk/bsd.prog.mk
==
--- head/share/mk/bsd.prog.mk   Tue Nov 10 19:09:35 2020(r367576)
+++ head/share/mk/bsd.prog.mk   Tue Nov 10 19:15:13 2020(r367577)
@@ -60,6 +60,25 @@ LDFLAGS+= -Wl,-zretpolineplt
 .endif
 .endif
 
+# Initialize stack variables on function entry
+.if ${MK_INIT_ALL_ZERO} == "yes"
+.if ${COMPILER_FEATURES:Minit-all}
+CFLAGS+= -ftrivial-auto-var-init=zero \
+-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang
+CXXFLAGS+= -ftrivial-auto-var-init=zero \
+-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang
+.else
+.warning InitAll (zeros) requested but not support by compiler
+.endif
+.elif ${MK_INIT_ALL_PATTERN} ==