The branch, master has been updated via f8947538b5e lib/fuzzing: Add oss-fuzz info to README.md via cc128c78856 lib/fuzzing/oss-fuzz: copy required libraries to the build target via 4946811eb67 lib/fuzzing/oss-fuzz: Install chrpath as we use it in the build.sh script to set -rpath via fbb2377d51f lib/fuzzing/oss-fuzz: Add build_image.sh using Samba's bootstrap tools via ec4f6f8fd36 lib/fuzzing/oss-fuzz: add stub build.sh that will not change often via f57c0238e91 lib/fuzzing/oss-fuzz: Add build_samba.sh for oss-fuzz via 92ee6478583 lib/fuzzing: Use --fuzz-target-ldflags if specified via cb240510977 build: Set fuzzer=True on fuzzer binaries from 9e51bb77f82 s4:rpc_server: Return the status code from dcesrv_transport_session_key()
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit f8947538b5e445ab9c1931d1f7826771ee582385 Author: Andrew Bartlett <abart...@samba.org> Date: Thu Nov 21 10:21:54 2019 +1300 lib/fuzzing: Add oss-fuzz info to README.md Note that Samba has not been accepted yet, but will be soon once some requirements are addressed per: https://github.com/google/oss-fuzz/pull/2993 Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Uri Simchoni <u...@samba.org> Autobuild-User(master): Douglas Bagnall <dbagn...@samba.org> Autobuild-Date(master): Thu Nov 21 00:45:33 UTC 2019 on sn-devel-184 commit cc128c788562d6d86c24cbcb784e1e8fca1f06a1 Author: Andrew Bartlett <abart...@samba.org> Date: Thu Nov 7 14:22:07 2019 +1300 lib/fuzzing/oss-fuzz: copy required libraries to the build target This is an alternative to static linking as we do not have static source libraries for all the things we depend on. Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Uri Simchoni <u...@samba.org> commit 4946811eb67c6e4e6abdfd64a552e7a2686feeec Author: Andrew Bartlett <abart...@samba.org> Date: Thu Nov 7 17:17:40 2019 +1300 lib/fuzzing/oss-fuzz: Install chrpath as we use it in the build.sh script to set -rpath Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Uri Simchoni <u...@samba.org> commit fbb2377d51f00617e9065a8a3daecc8f5c340ca0 Author: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Date: Thu Nov 21 09:59:24 2019 +1300 lib/fuzzing/oss-fuzz: Add build_image.sh using Samba's bootstrap tools Google's oss-fuzz environment is Ubuntu 16.04 based so we can just use the maintained bootstrap system rather than a manual package list here that will get out of date. Signed-off-by: Andrew Bartlett <abart...@samba.org> Signed-off-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Uri Simchoni <u...@samba.org> Pair-programmed-by: Andrew Bartlett <abart...@samba.org> commit ec4f6f8fd36edfda8b949dcf7451c153f6599427 Author: Andrew Bartlett <abart...@samba.org> Date: Thu Nov 21 10:29:57 2019 +1300 lib/fuzzing/oss-fuzz: add stub build.sh that will not change often This makes local development of build_samba.sh easier as it will remain in the source tree. Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Uri Simchoni <u...@samba.org> commit f57c0238e91061698787e4a4b18fb549cc2c86f2 Author: Andrew Bartlett <abart...@samba.org> Date: Wed Nov 20 14:03:13 2019 +1300 lib/fuzzing/oss-fuzz: Add build_samba.sh for oss-fuzz We work hard to put the primary logic for oss-fuzz here, and where possible into waf, so that only a tiny stub needs to be maintained in the Google oss-fuzz repo. This will be called by build.sh (not copied directly because it is too easy to forget to copy in an updated version when doing development in the docker image). Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Uri Simchoni <u...@samba.org> commit 92ee647858331db012db06314d98e08fe1e5eb5b Author: Andrew Bartlett <abart...@samba.org> Date: Wed Nov 6 12:24:18 2019 +1300 lib/fuzzing: Use --fuzz-target-ldflags if specified This makes integration with oss-fuzz possible. Only the fuzzer binaries should be linked with libFuzzer, not things like asn1_compile, so this can not be done via the global ADDITIONAL_LDFLAGS. Signed-off-by: Andrew Bartlett <abart...@samba.org> Reviewed-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Uri Simchoni <u...@samba.org> commit cb240510977e60fd46ef0f35b3da77f00733829a Author: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Date: Fri Nov 1 17:18:24 2019 +1300 build: Set fuzzer=True on fuzzer binaries This ensures that the binaries are the only binaries built when configured for fuzzing. Signed-off-by: Douglas Bagnall <douglas.bagn...@catalyst.net.nz> Reviewed-by: Uri Simchoni <u...@samba.org> Reviewed-by: Andrew Bartlett <abart...@samba.org> ----------------------------------------------------------------------- Summary of changes: buildtools/wafsamba/wafsamba.py | 15 +++++++++++- buildtools/wafsamba/wscript | 10 +++++++- lib/fuzzing/README.md | 26 +++++++++++++++++++++ lib/fuzzing/oss-fuzz/build.sh | 6 +++++ lib/fuzzing/oss-fuzz/build_image.sh | 11 +++++++++ lib/fuzzing/oss-fuzz/build_samba.sh | 46 +++++++++++++++++++++++++++++++++++++ lib/fuzzing/wscript_build | 29 +++++++---------------- 7 files changed, 120 insertions(+), 23 deletions(-) create mode 100644 lib/fuzzing/oss-fuzz/build.sh create mode 100755 lib/fuzzing/oss-fuzz/build_image.sh create mode 100755 lib/fuzzing/oss-fuzz/build_samba.sh Changeset truncated at 500 lines: diff --git a/buildtools/wafsamba/wafsamba.py b/buildtools/wafsamba/wafsamba.py index 7081f382eaa..7460fbae020 100644 --- a/buildtools/wafsamba/wafsamba.py +++ b/buildtools/wafsamba/wafsamba.py @@ -360,13 +360,26 @@ def SAMBA_BINARY(bld, binname, source, subdir=None, install=True, install_path=None, - enabled=True): + enabled=True, + fuzzer=False): '''define a Samba binary''' if not enabled: SET_TARGET_TYPE(bld, binname, 'DISABLED') return + # Fuzzing builds do not build normal binaries + # however we must build asn1compile etc + + if not use_hostcc and bld.env.enable_libfuzzer != fuzzer: + SET_TARGET_TYPE(bld, binname, 'DISABLED') + return + + if fuzzer: + install = False + if ldflags is None: + ldflags = bld.env['FUZZ_TARGET_LDFLAGS'] + if not SET_TARGET_TYPE(bld, binname, 'BINARY'): return diff --git a/buildtools/wafsamba/wscript b/buildtools/wafsamba/wscript index b601dd2596d..9987c6e4fcc 100644 --- a/buildtools/wafsamba/wscript +++ b/buildtools/wafsamba/wscript @@ -130,9 +130,16 @@ def options(opt): dest='undefined_sanitizer', default=False) gr.add_option('--enable-libfuzzer', - help=("Build fuzzing binaries (requires compiler options for libFuzzer or compiler wrapper such as honggfuzz/hfuzz-cc)"), + help=("Build fuzzing binaries (use ADDITIONAL_CFLAGS to specify compiler options for libFuzzer or use a compiler wrapper such as honggfuzz/hfuzz-cc)"), action="store_true", dest='enable_libfuzzer', default=False) + # Fuzz targets may need additional LDFLAGS that we can't use on + # internal binaries like asn1_compile + + gr.add_option('--fuzz-target-ldflags', + help=("Linker flags to be used when building fuzz targets"), + action="store", dest='FUZZ_TARGET_LDFLAGS', default='') + gr.add_option('--abi-check', help=("Check ABI signatures for libraries"), action='store_true', dest='ABI_CHECK', default=False) @@ -596,6 +603,7 @@ struct foo bar = { .y = 'X', .x = 1 }; conf.env.enable_libfuzzer = Options.options.enable_libfuzzer if conf.env.enable_libfuzzer: conf.DEFINE('ENABLE_LIBFUZZER', 1) + conf.env.FUZZ_TARGET_LDFLAGS = Options.options.FUZZ_TARGET_LDFLAGS conf.load('clang_compilation_database') diff --git a/lib/fuzzing/README.md b/lib/fuzzing/README.md index 3848838ba02..97b49ed0fb1 100644 --- a/lib/fuzzing/README.md +++ b/lib/fuzzing/README.md @@ -32,4 +32,30 @@ buildtools/bin/waf --targets=fuzz_tiniparser build && \ --rlimit_rss 100 -f .../tiniparser-corpus -- bin/fuzz_tiniparser ``` +# oss-fuzz + +Samba can be fuzzed by Google's oss-fuzz system. Assuming you have an +oss-fuzz checkout from https://github.com/google/oss-fuzz with Samba's +metadata in projects/samba, the following guides will help: + +## Testing locally + +https://google.github.io/oss-fuzz/getting-started/new-project-guide/#testing-locally + +## Debugging oss-fuzz + +See https://google.github.io/oss-fuzz/advanced-topics/debugging/ + +## Samba-specific hints + +A typical debugging workflow is: + +oss-fuzz$ python infra/helper.py shell samba +git fetch $REMOTE $BRANCH +git checkout FETCH_HEAD +lib/fuzzing/oss-fuzz/build_image.sh +compile + +This will pull in any new Samba deps and build Samba's fuzzers. + # vim: set sw=8 sts=8 ts=8 tw=79 : diff --git a/lib/fuzzing/oss-fuzz/build.sh b/lib/fuzzing/oss-fuzz/build.sh new file mode 100644 index 00000000000..14ba6ee83b2 --- /dev/null +++ b/lib/fuzzing/oss-fuzz/build.sh @@ -0,0 +1,6 @@ +#!/bin/sh -e + +# This will be copied into $SRC, being the root of the source tree by +# build_image.sh + +exec lib/fuzzing/oss-fuzz/build_samba.sh diff --git a/lib/fuzzing/oss-fuzz/build_image.sh b/lib/fuzzing/oss-fuzz/build_image.sh new file mode 100755 index 00000000000..fa899effc37 --- /dev/null +++ b/lib/fuzzing/oss-fuzz/build_image.sh @@ -0,0 +1,11 @@ +#!/bin/sh -e + +DIST=ubuntu1604 +SCRIPT_DIR=`dirname $0` + +$SCRIPT_DIR/../../../bootstrap/generated-dists/$DIST/bootstrap.sh +$SCRIPT_DIR/../../../bootstrap/generated-dists/$DIST/locale.sh + +apt-get install chrpath + +cp $SCRIPT_DIR/build.sh $SRC/ diff --git a/lib/fuzzing/oss-fuzz/build_samba.sh b/lib/fuzzing/oss-fuzz/build_samba.sh new file mode 100755 index 00000000000..aa8e223b0a1 --- /dev/null +++ b/lib/fuzzing/oss-fuzz/build_samba.sh @@ -0,0 +1,46 @@ +#!/bin/sh -e +# +# This is not a general-purpose build script, but instead one specific to the Google oss-fuzz compile environment. +# +# https://google.github.io/oss-fuzz/getting-started/new-project-guide/#Requirements +# +# https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/README.md#provided-environment-variables +# +# We have to push to oss-fuzz CFLAGS into the waf ADDITIONAL_CFLAGS +# as otherwise waf's configure fails linking the first test binary +# +# CFLAGS are supplied by the caller, eg the oss-fuzz compile command +# +ADDITIONAL_CFLAGS="$CFLAGS" +export ADDITIONAL_CFLAGS +CFLAGS="" +export CFLAGS +LD="$CXX" +export LD + +# $LIB_FUZZING_ENGINE is provided by the oss-fuzz "compile" command +# + +./configure -C --without-gettext --enable-debug --enable-developer \ + --address-sanitizer --enable-libfuzzer \ + --disable-warnings-as-errors \ + --abi-check-disable \ + --fuzz-target-ldflags="$LIB_FUZZING_ENGINE" \ + --nonshared-binary=ALL LINK_CC="$CXX" + +make -j + +# Make a directory for the system shared libraries to be copied into +mkdir -p $OUT/lib + +# We can't static link to all the system libs with waf, so copy them +# to $OUT/lib and set the rpath to point there. This is similar to how +# firefox handles this. + +for x in bin/fuzz_* +do + cp $x $OUT/ + bin=`basename $x` + ldd $OUT/$bin | cut -f 2 -d '>' | cut -f 1 -d \( | cut -f 2 -d ' ' | xargs -i cp \{\} $OUT/lib/ + chrpath -r '$ORIGIN/lib' $OUT/$bin +done diff --git a/lib/fuzzing/wscript_build b/lib/fuzzing/wscript_build index 7305ce41262..386145c43b2 100644 --- a/lib/fuzzing/wscript_build +++ b/lib/fuzzing/wscript_build @@ -3,53 +3,40 @@ bld.SAMBA_SUBSYSTEM('fuzzing', source='fuzzing.c', deps='talloc', - enabled=bld.env.enable_libfuzzer, + enabled=bld.env.enable_libfuzzer ) bld.SAMBA_BINARY('fuzz_tiniparser', source='fuzz_tiniparser.c', deps='fuzzing tiniparser talloc', - install=False, - enabled=bld.env.enable_libfuzzer) + fuzzer=True) bld.SAMBA_BINARY('fuzz_oLschema2ldif', source='fuzz_oLschema2ldif.c', deps='fuzzing oLschema2ldif-lib', - install=False, - enabled=bld.env.enable_libfuzzer, - ) + fuzzer=True) bld.SAMBA_BINARY('fuzz_reg_parse', source='fuzz_reg_parse.c', deps='fuzzing samba3-util smbconf REGFIO', - install=False, - enabled=bld.env.enable_libfuzzer, - ) + fuzzer=True) bld.SAMBA_BINARY('fuzz_regfio', source='fuzz_regfio.c', deps='fuzzing samba3-util smbconf REGFIO', - install=False, - enabled=bld.env.enable_libfuzzer, - ) + fuzzer=True) bld.SAMBA_BINARY('fuzz_lzxpress', source='fuzz_lzxpress.c', deps='fuzzing LZXPRESS', - install=False, - enabled=bld.env.enable_libfuzzer, - ) + fuzzer=True) bld.SAMBA_BINARY('fuzz_ldap_decode', source='fuzz_ldap_decode.c', deps='fuzzing cli-ldap', - install=False, - enabled=bld.env.enable_libfuzzer, - ) + fuzzer=True) bld.SAMBA_BINARY('fuzz_ldb_parse_tree', source='fuzz_ldb_parse_tree.c', deps='fuzzing ldb', - install=False, - enabled=bld.env.enable_libfuzzer - ) + fuzzer=True) -- Samba Shared Repository