On Thu, Aug 10, 2017 at 3:13 PM, Kostya Serebryany <k...@google.com> wrote:
> > > On Thu, Aug 10, 2017 at 12:01 PM, Nico Weber <tha...@chromium.org> wrote: > >> On Thu, Aug 10, 2017 at 2:04 PM, Kostya Serebryany <k...@google.com> >> wrote: >> >>> >>> >>> On Thu, Aug 10, 2017 at 10:56 AM, Nico Weber via cfe-commits < >>> cfe-commits@lists.llvm.org> wrote: >>> >>>> I really believe this has way too many deps to live in the clang repro, >>>> as said on the review already. >>>> >>> >>> I don't have a very strong opinion here and would be happy to move if I >>> see more support for Nico's opinion >>> (I haven't seen it on the review, and you didn't object further, so we >>> proceeded). >>> Again, my rational is that the simpler it is to use the more likely >>> other researchers will extend this work. >>> >>> BTW, I am going to commit a Dockerfile that will make experimenting with >>> this trivial. >>> My current (dirty) version looks like this. Not too much trouble. >>> >>> FROM ubuntu:16.04 >>> RUN apt-get update -y && apt-get install -y autoconf automake libtool >>> curl make g++ unzip >>> RUN apt-get install -y wget >>> RUN apt-get install -y git binutils liblzma-dev libz-dev >>> RUN apt-get install -y python-all >>> RUN apt-get install -y cmake ninja-build >>> RUN apt-get install -y subversion >>> >>> WORKDIR /root >>> RUN wget -qO- https://github.com/google/prot >>> obuf/releases/download/v3.3.0/protobuf-cpp-3.3.0.tar.gz | tar zxf - >>> RUN cd protobuf-3.3.0 && ./autogen.sh && ./configure && make -j $(nproc) >>> && make check -j $(nproc) && make install && ldconfig >>> RUN apt-get install -y pkg-config >>> RUN svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm >>> RUN cd llvm/tools && svn co http://llvm.org/svn/llvm-project/cfe/trunk >>> clang -r $(cd ../ && svn info | grep Revision | awk '{print $2}') >>> RUN cd llvm/projects && svn co http://llvm.org/svn/llvm-proje >>> ct/compiler-rt/trunk clang -r $(cd ../ && svn info | grep Revision | >>> awk '{print $2}') >>> RUN mkdir build0 && cd build0 && cmake -GNinja >>> -DCMAKE_BUILD_TYPE=Release ../llvm && ninja >>> RUN mkdir build1 && cd build1 && cmake -GNinja >>> -DCMAKE_BUILD_TYPE=Release ../llvm -DLLVM_ENABLE_ASSERTIONS=ON >>> -DCMAKE_C_COMPILER=`pwd`/../build0/bin/clang >>> -DCMAKE_CXX_COMPILER=`pwd`/../build0/bin/clang++ >>> -DLLVM_USE_SANITIZE_COVERAGE=YES -DLLVM_USE_SANITIZER=Address >>> -DCLANG_ENABLE_PROTO_FUZZER=ON >>> RUN cd build1 && ninja clang-fuzzer >>> RUN cd build1 && ninja clang-proto-fuzzer >>> #RUN cd build1 && ninja clang-proto-to-cxx >>> >>> >>> >>>> Maybe this could live in clang-extra instead? >>>> >>> >>> clang-extra? >>> >> >> clang-tools-extra, sorry. >> >> >>> That's a separate repo, right? >>> >> >> Yes. >> >> >>> It may require more cmake trickery, and we'll also have to share the >>> clang-fuzzer-specific code between two repos. >>> >> >> We could move the whole thing. I'd imagine that at most 3% of people who >> use clang will use this fuzzer, so having it elsewhere seems reasonable. >> (I'd imagine many more people to use clang-tidy for example, and that's in >> the other repro.) >> > > The clang-tidy argument doesn't work for me. > clang-tidy is a separate tool. > clang*fuzzer are ways to test clang, and so they have more reasons to stay > closer to clang (for the same reason that the clang tests stay with clang). > Then think of the "[cfe-dev] Proposal for an ABI testsuite for clang" thread instead, which was about testing clang. We ended up putting that into a completely separate repo. > > --kcc > > >> Also see the "Contributing Extensions to Clang" section on >> http://clang.llvm.org/get_involved.html >> >> >>> I do want the original clang-fuzzer to remain where it was, and both >>> (clang-fuzzer and clang-proto-fuzzer) share the code. >>> >>> >>> >>> >>>> >>>> On Aug 8, 2017 4:15 PM, "Matt Morehouse via cfe-commits" < >>>> cfe-commits@lists.llvm.org> wrote: >>>> >>>>> Author: morehouse >>>>> Date: Tue Aug 8 13:15:04 2017 >>>>> New Revision: 310408 >>>>> >>>>> URL: http://llvm.org/viewvc/llvm-project?rev=310408&view=rev >>>>> Log: >>>>> Integrate Kostya's clang-proto-fuzzer with LLVM. >>>>> >>>>> Summary: >>>>> The clang-proto-fuzzer models a subset of C++ as a protobuf and >>>>> uses libprotobuf-mutator to generate interesting mutations of C++ >>>>> programs. Clang-proto-fuzzer has already found several bugs in >>>>> Clang (e.g., https://bugs.llvm.org/show_bug.cgi?id=33747, >>>>> https://bugs.llvm.org/show_bug.cgi?id=33749). >>>>> >>>>> As with clang-fuzzer, clang-proto-fuzzer requires the following >>>>> cmake flags: >>>>> - CMAKE_C_COMPILER=clang >>>>> - CMAKE_CXX_COMPILER=clang++ >>>>> - LLVM_USE_SANITIZE_COVERAGE=YES // needed for libFuzzer >>>>> - LLVM_USE_SANITIZER=Address // needed for libFuzzer >>>>> >>>>> In addition, clang-proto-fuzzer requires: >>>>> - CLANG_ENABLE_PROTO_FUZZER=ON >>>>> >>>>> clang-proto-fuzzer also requires the following dependencies: >>>>> - binutils // needed for libprotobuf-mutator >>>>> - liblzma-dev // needed for libprotobuf-mutator >>>>> - libz-dev // needed for libprotobuf-mutator >>>>> - docbook2x // needed for libprotobuf-mutator >>>>> - Recent version of protobuf [3.3.0 is known to work] >>>>> >>>>> A working version of libprotobuf-mutator will automatically be >>>>> downloaded and built as an external project. >>>>> >>>>> Implementation of clang-proto-fuzzer provided by Kostya >>>>> Serebryany. >>>>> >>>>> https://bugs.llvm.org/show_bug.cgi?id=33829 >>>>> >>>>> Reviewers: kcc, vitalybuka, bogner >>>>> >>>>> Reviewed By: kcc, vitalybuka >>>>> >>>>> Subscribers: thakis, mgorny, cfe-commits >>>>> >>>>> Differential Revision: https://reviews.llvm.org/D36324 >>>>> >>>>> Added: >>>>> cfe/trunk/cmake/modules/ProtobufMutator.cmake >>>>> cfe/trunk/tools/clang-fuzzer/ExampleClangProtoFuzzer.cpp >>>>> cfe/trunk/tools/clang-fuzzer/README.txt >>>>> cfe/trunk/tools/clang-fuzzer/cxx_proto.proto >>>>> cfe/trunk/tools/clang-fuzzer/handle-cxx/ >>>>> cfe/trunk/tools/clang-fuzzer/handle-cxx/CMakeLists.txt >>>>> cfe/trunk/tools/clang-fuzzer/handle-cxx/handle_cxx.cpp >>>>> cfe/trunk/tools/clang-fuzzer/handle-cxx/handle_cxx.h >>>>> cfe/trunk/tools/clang-fuzzer/proto-to-cxx/ >>>>> cfe/trunk/tools/clang-fuzzer/proto-to-cxx/CMakeLists.txt >>>>> cfe/trunk/tools/clang-fuzzer/proto-to-cxx/proto_to_cxx.cpp >>>>> cfe/trunk/tools/clang-fuzzer/proto-to-cxx/proto_to_cxx.h >>>>> cfe/trunk/tools/clang-fuzzer/proto-to-cxx/proto_to_cxx_main.cpp >>>>> Modified: >>>>> cfe/trunk/CMakeLists.txt >>>>> cfe/trunk/tools/clang-fuzzer/CMakeLists.txt >>>>> cfe/trunk/tools/clang-fuzzer/ClangFuzzer.cpp >>>>> >>>>> Modified: cfe/trunk/CMakeLists.txt >>>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/CMakeLists.txt >>>>> ?rev=310408&r1=310407&r2=310408&view=diff >>>>> ============================================================ >>>>> ================== >>>>> --- cfe/trunk/CMakeLists.txt (original) >>>>> +++ cfe/trunk/CMakeLists.txt Tue Aug 8 13:15:04 2017 >>>>> @@ -377,6 +377,8 @@ option(CLANG_ENABLE_STATIC_ANALYZER "Bui >>>>> option(CLANG_ANALYZER_BUILD_Z3 >>>>> "Build the static analyzer with the Z3 constraint manager." OFF) >>>>> >>>>> +option(CLANG_ENABLE_PROTO_FUZZER "Build Clang protobuf fuzzer." OFF) >>>>> + >>>>> if(NOT CLANG_ENABLE_STATIC_ANALYZER AND (CLANG_ENABLE_ARCMT OR >>>>> CLANG_ANALYZER_BUILD_Z3)) >>>>> message(FATAL_ERROR "Cannot disable static analyzer while enabling >>>>> ARCMT or Z3") >>>>> endif() >>>>> >>>>> Added: cfe/trunk/cmake/modules/ProtobufMutator.cmake >>>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/cmake/modules/ >>>>> ProtobufMutator.cmake?rev=310408&view=auto >>>>> ============================================================ >>>>> ================== >>>>> --- cfe/trunk/cmake/modules/ProtobufMutator.cmake (added) >>>>> +++ cfe/trunk/cmake/modules/ProtobufMutator.cmake Tue Aug 8 13:15:04 >>>>> 2017 >>>>> @@ -0,0 +1,24 @@ >>>>> +set(PBM_PREFIX protobuf_mutator) >>>>> +set(PBM_PATH ${CMAKE_CURRENT_BINARY_DIR}/${ >>>>> PBM_PREFIX}/src/${PBM_PREFIX}) >>>>> +set(PBM_LIB_PATH ${PBM_PATH}/src/libprotobuf-mutator.a) >>>>> +set(PBM_FUZZ_LIB_PATH ${PBM_PATH}/src/libfuzzer/libp >>>>> rotobuf-mutator-libfuzzer.a) >>>>> + >>>>> +ExternalProject_Add(${PBM_PREFIX} >>>>> + PREFIX ${PBM_PREFIX} >>>>> + GIT_REPOSITORY https://github.com/google/libprotobuf-mutator.git >>>>> + GIT_TAG 34287f8 >>>>> + CONFIGURE_COMMAND ${CMAKE_COMMAND} -G${CMAKE_GENERATOR} >>>>> + -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER} >>>>> + -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER} >>>>> + -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} >>>>> + BUILD_COMMAND ${CMAKE_MAKE_PROGRAM} >>>>> + BUILD_BYPRODUCTS ${PBM_LIB_PATH} ${PBM_FUZZ_LIB_PATH} >>>>> + BUILD_IN_SOURCE 1 >>>>> + INSTALL_COMMAND "" >>>>> + LOG_DOWNLOAD 1 >>>>> + LOG_CONFIGURE 1 >>>>> + LOG_BUILD 1 >>>>> + ) >>>>> + >>>>> +set(ProtobufMutator_INCLUDE_DIRS ${PBM_PATH}) >>>>> +set(ProtobufMutator_LIBRARIES ${PBM_FUZZ_LIB_PATH} ${PBM_LIB_PATH}) >>>>> >>>>> Modified: cfe/trunk/tools/clang-fuzzer/CMakeLists.txt >>>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-fu >>>>> zzer/CMakeLists.txt?rev=310408&r1=310407&r2=310408&view=diff >>>>> ============================================================ >>>>> ================== >>>>> --- cfe/trunk/tools/clang-fuzzer/CMakeLists.txt (original) >>>>> +++ cfe/trunk/tools/clang-fuzzer/CMakeLists.txt Tue Aug 8 13:15:04 >>>>> 2017 >>>>> @@ -1,21 +1,60 @@ >>>>> if( LLVM_USE_SANITIZE_COVERAGE ) >>>>> set(LLVM_LINK_COMPONENTS ${LLVM_TARGETS_TO_BUILD}) >>>>> >>>>> + if(CLANG_ENABLE_PROTO_FUZZER) >>>>> + # Create protobuf .h and .cc files, and put them in a library for >>>>> use by >>>>> + # clang-proto-fuzzer components. >>>>> + find_package(Protobuf REQUIRED) >>>>> + add_definitions(-DGOOGLE_PROTOBUF_NO_RTTI) >>>>> + include_directories(${PROTOBUF_INCLUDE_DIRS}) >>>>> + include_directories(${CMAKE_CURRENT_BINARY_DIR}) >>>>> + protobuf_generate_cpp(PROTO_SRCS PROTO_HDRS cxx_proto.proto) >>>>> + # Hack to bypass LLVM's cmake sources check and allow multiple >>>>> libraries and >>>>> + # executables from this directory. >>>>> + set(LLVM_OPTIONAL_SOURCES >>>>> + ClangFuzzer.cpp >>>>> + ExampleClangProtoFuzzer.cpp >>>>> + ${PROTO_SRCS} >>>>> + ) >>>>> + add_clang_library(clangCXXProto >>>>> + ${PROTO_SRCS} >>>>> + ${PROTO_HDRS} >>>>> + >>>>> + LINK_LIBS >>>>> + ${PROTOBUF_LIBRARIES} >>>>> + ) >>>>> + >>>>> + # Build and include libprotobuf-mutator >>>>> + include(ProtobufMutator) >>>>> + include_directories(${ProtobufMutator_INCLUDE_DIRS}) >>>>> + >>>>> + # Build the protobuf->C++ translation library and driver. >>>>> + add_clang_subdirectory(proto-to-cxx) >>>>> + >>>>> + # Build the protobuf fuzzer >>>>> + add_clang_executable(clang-proto-fuzzer >>>>> ExampleClangProtoFuzzer.cpp) >>>>> + target_link_libraries(clang-proto-fuzzer >>>>> + ${ProtobufMutator_LIBRARIES} >>>>> + clangCXXProto >>>>> + clangHandleCXX >>>>> + clangProtoToCXX >>>>> + LLVMFuzzer >>>>> + ) >>>>> + else() >>>>> + # Hack to bypass LLVM's cmake sources check and allow multiple >>>>> libraries and >>>>> + # executables from this directory. >>>>> + set(LLVM_OPTIONAL_SOURCES ClangFuzzer.cpp >>>>> ExampleClangProtoFuzzer.cpp) >>>>> + endif() >>>>> + >>>>> + add_clang_subdirectory(handle-cxx) >>>>> + >>>>> add_clang_executable(clang-fuzzer >>>>> EXCLUDE_FROM_ALL >>>>> ClangFuzzer.cpp >>>>> ) >>>>> >>>>> target_link_libraries(clang-fuzzer >>>>> - ${CLANG_FORMAT_LIB_DEPS} >>>>> - clangAST >>>>> - clangBasic >>>>> - clangCodeGen >>>>> - clangDriver >>>>> - clangFrontend >>>>> - clangRewriteFrontend >>>>> - clangStaticAnalyzerFrontend >>>>> - clangTooling >>>>> + clangHandleCXX >>>>> LLVMFuzzer >>>>> ) >>>>> endif() >>>>> >>>>> Modified: cfe/trunk/tools/clang-fuzzer/ClangFuzzer.cpp >>>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-fu >>>>> zzer/ClangFuzzer.cpp?rev=310408&r1=310407&r2=310408&view=diff >>>>> ============================================================ >>>>> ================== >>>>> --- cfe/trunk/tools/clang-fuzzer/ClangFuzzer.cpp (original) >>>>> +++ cfe/trunk/tools/clang-fuzzer/ClangFuzzer.cpp Tue Aug 8 13:15:04 >>>>> 2017 >>>>> @@ -13,43 +13,12 @@ >>>>> /// >>>>> //===------------------------------------------------------ >>>>> ----------------===// >>>>> >>>>> -#include "clang/Tooling/Tooling.h" >>>>> -#include "clang/CodeGen/CodeGenAction.h" >>>>> -#include "clang/Frontend/CompilerInstance.h" >>>>> -#include "clang/Lex/PreprocessorOptions.h" >>>>> -#include "llvm/Option/Option.h" >>>>> -#include "llvm/Support/TargetSelect.h" >>>>> +#include "handle-cxx/handle_cxx.h" >>>>> >>>>> -using namespace clang; >>>>> +using namespace clang_fuzzer; >>>>> >>>>> extern "C" int LLVMFuzzerTestOneInput(uint8_t *data, size_t size) { >>>>> std::string s((const char *)data, size); >>>>> - llvm::InitializeAllTargets(); >>>>> - llvm::InitializeAllTargetMCs(); >>>>> - llvm::InitializeAllAsmPrinters(); >>>>> - llvm::InitializeAllAsmParsers(); >>>>> - >>>>> - llvm::opt::ArgStringList CC1Args; >>>>> - CC1Args.push_back("-cc1"); >>>>> - CC1Args.push_back("./test.cc"); >>>>> - CC1Args.push_back("-O2"); >>>>> - llvm::IntrusiveRefCntPtr<FileManager> Files( >>>>> - new FileManager(FileSystemOptions())); >>>>> - IgnoringDiagConsumer Diags; >>>>> - IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts = new >>>>> DiagnosticOptions(); >>>>> - DiagnosticsEngine Diagnostics( >>>>> - IntrusiveRefCntPtr<clang::DiagnosticIDs>(new DiagnosticIDs()), >>>>> &*DiagOpts, >>>>> - &Diags, false); >>>>> - std::unique_ptr<clang::CompilerInvocation> Invocation( >>>>> - tooling::newInvocation(&Diagnostics, CC1Args)); >>>>> - std::unique_ptr<llvm::MemoryBuffer> Input = >>>>> - llvm::MemoryBuffer::getMemBuffer(s); >>>>> - Invocation->getPreprocessorOpts().addRemappedFile("./test.cc", >>>>> Input.release()); >>>>> - std::unique_ptr<tooling::ToolAction> action( >>>>> - tooling::newFrontendActionFactory<clang::EmitObjAction>()); >>>>> - std::shared_ptr<PCHContainerOperations> PCHContainerOps = >>>>> - std::make_shared<PCHContainerOperations>(); >>>>> - action->runInvocation(std::move(Invocation), Files.get(), >>>>> PCHContainerOps, >>>>> - &Diags); >>>>> + HandleCXX(s, {"-O2"}); >>>>> return 0; >>>>> } >>>>> >>>>> Added: cfe/trunk/tools/clang-fuzzer/ExampleClangProtoFuzzer.cpp >>>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-fu >>>>> zzer/ExampleClangProtoFuzzer.cpp?rev=310408&view=auto >>>>> ============================================================ >>>>> ================== >>>>> --- cfe/trunk/tools/clang-fuzzer/ExampleClangProtoFuzzer.cpp (added) >>>>> +++ cfe/trunk/tools/clang-fuzzer/ExampleClangProtoFuzzer.cpp Tue Aug >>>>> 8 13:15:04 2017 >>>>> @@ -0,0 +1,28 @@ >>>>> +//===-- ExampleClangProtoFuzzer.cpp - Fuzz Clang >>>>> --------------------------===// >>>>> +// >>>>> +// The LLVM Compiler Infrastructure >>>>> +// >>>>> +// This file is distributed under the University of Illinois Open >>>>> Source >>>>> +// License. See LICENSE.TXT for details. >>>>> +// >>>>> +//===------------------------------------------------------ >>>>> ----------------===// >>>>> +/// >>>>> +/// \file >>>>> +/// \brief This file implements a function that runs Clang on a single >>>>> +/// input and uses libprotobuf-mutator to find new inputs. This >>>>> function is >>>>> +/// then linked into the Fuzzer library. >>>>> +/// >>>>> +//===------------------------------------------------------ >>>>> ----------------===// >>>>> + >>>>> +#include "cxx_proto.pb.h" >>>>> +#include "handle-cxx/handle_cxx.h" >>>>> +#include "proto-to-cxx/proto_to_cxx.h" >>>>> + >>>>> +#include "src/libfuzzer/libfuzzer_macro.h" >>>>> + >>>>> +using namespace clang_fuzzer; >>>>> + >>>>> +DEFINE_BINARY_PROTO_FUZZER(const Function& input) { >>>>> + auto S = FunctionToString(input); >>>>> + HandleCXX(S, {"-O2"}); >>>>> +} >>>>> >>>>> Added: cfe/trunk/tools/clang-fuzzer/README.txt >>>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-fu >>>>> zzer/README.txt?rev=310408&view=auto >>>>> ============================================================ >>>>> ================== >>>>> --- cfe/trunk/tools/clang-fuzzer/README.txt (added) >>>>> +++ cfe/trunk/tools/clang-fuzzer/README.txt Tue Aug 8 13:15:04 2017 >>>>> @@ -0,0 +1,73 @@ >>>>> +This directory contains two utilities for fuzzing Clang: clang-fuzzer >>>>> and >>>>> +clang-proto-fuzzer. Both use libFuzzer to generate inputs to clang >>>>> via >>>>> +coverage-guided mutation. >>>>> + >>>>> +The two utilities differ, however, in how they structure inputs to >>>>> Clang. >>>>> +clang-fuzzer makes no attempt to generate valid C++ programs and is >>>>> therefore >>>>> +primarily useful for stressing the surface layers of Clang (i.e. >>>>> lexer, parser). >>>>> +clang-proto-fuzzer uses a protobuf class to describe a subset of the >>>>> C++ >>>>> +language and then uses libprotobuf-mutator to mutate instantiations >>>>> of that >>>>> +class, producing valid C++ programs in the process. As a result, >>>>> +clang-proto-fuzzer is better at stressing deeper layers of Clang and >>>>> LLVM. >>>>> + >>>>> +=================================== >>>>> + Building clang-fuzzer >>>>> +=================================== >>>>> +Within your LLVM build directory, run CMake with the following >>>>> variable >>>>> +definitions: >>>>> +- CMAKE_C_COMPILER=clang >>>>> +- CMAKE_CXX_COMPILER=clang++ >>>>> +- LLVM_USE_SANITIZE_COVERAGE=YES >>>>> +- LLVM_USE_SANITIZER=Address >>>>> + >>>>> +Then build the clang-fuzzer target. >>>>> + >>>>> +Example: >>>>> + cd $LLVM_SOURCE_DIR >>>>> + mkdir build && cd build >>>>> + cmake .. -GNinja -DCMAKE_C_COMPILER=clang >>>>> -DCMAKE_CXX_COMPILER=clang++ \ >>>>> + -DLLVM_USE_SANITIZE_COVERAGE=YES -DLLVM_USE_SANITIZER=Address >>>>> + ninja clang-fuzzer >>>>> + >>>>> + >>>>> +======================================================= >>>>> + Building clang-proto-fuzzer (Linux-only instructions) >>>>> +======================================================= >>>>> +Install the necessary dependencies: >>>>> +- binutils // needed for libprotobuf-mutator >>>>> +- liblzma-dev // needed for libprotobuf-mutator >>>>> +- libz-dev // needed for libprotobuf-mutator >>>>> +- docbook2x // needed for libprotobuf-mutator >>>>> +- Recent version of protobuf [3.3.0 is known to work] >>>>> + >>>>> +Within your LLVM build directory, run CMake with the following >>>>> variable >>>>> +definitions: >>>>> +- CMAKE_C_COMPILER=clang >>>>> +- CMAKE_CXX_COMPILER=clang++ >>>>> +- LLVM_USE_SANITIZE_COVERAGE=YES >>>>> +- LLVM_USE_SANITIZER=Address >>>>> +- CLANG_ENABLE_PROTO_FUZZER=ON >>>>> + >>>>> +Then build the clang-proto-fuzzer and clang-proto-to-cxx targets. >>>>> Optionally, >>>>> +you may also build clang-fuzzer with this setup. >>>>> + >>>>> +Example: >>>>> + cd $LLVM_SOURCE_DIR >>>>> + mkdir build && cd build >>>>> + cmake .. -GNinja -DCMAKE_C_COMPILER=clang >>>>> -DCMAKE_CXX_COMPILER=clang++ \ >>>>> + -DLLVM_USE_SANITIZE_COVERAGE=YES -DLLVM_USE_SANITIZER=Address \ >>>>> + -DCLANG_ENABLE_PROTO_FUZZER=ON >>>>> + ninja clang-proto-fuzzer clang-proto-to-cxx >>>>> + >>>>> + >>>>> +===================== >>>>> + Running the fuzzers >>>>> +===================== >>>>> +clang-fuzzer: >>>>> + bin/clang-fuzzer CORPUS_DIR >>>>> + >>>>> +clang-proto-fuzzer: >>>>> + bin/clang-proto-fuzzer CORPUS_DIR >>>>> + >>>>> +Translating a clang-proto-fuzzer corpus output to C++: >>>>> + bin/clang-proto-to-cxx CORPUS_OUTPUT_FILE >>>>> >>>>> Added: cfe/trunk/tools/clang-fuzzer/cxx_proto.proto >>>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-fu >>>>> zzer/cxx_proto.proto?rev=310408&view=auto >>>>> ============================================================ >>>>> ================== >>>>> --- cfe/trunk/tools/clang-fuzzer/cxx_proto.proto (added) >>>>> +++ cfe/trunk/tools/clang-fuzzer/cxx_proto.proto Tue Aug 8 13:15:04 >>>>> 2017 >>>>> @@ -0,0 +1,93 @@ >>>>> +//===-- cxx_proto.proto - Protobuf description of C++ >>>>> ---------------------===// >>>>> +// >>>>> +// The LLVM Compiler Infrastructure >>>>> +// >>>>> +// This file is distributed under the University of Illinois Open >>>>> Source >>>>> +// License. See LICENSE.TXT for details. >>>>> +// >>>>> +//===------------------------------------------------------ >>>>> ----------------===// >>>>> +/// >>>>> +/// \file >>>>> +/// \brief This file describes a subset of C++ as a protobuf. It is >>>>> used to >>>>> +/// more easily find interesting inputs for fuzzing Clang. >>>>> +/// >>>>> +//===------------------------------------------------------ >>>>> ----------------===// >>>>> + >>>>> +syntax = "proto2"; >>>>> + >>>>> +message VarRef { >>>>> + required int32 varnum = 1; >>>>> +} >>>>> + >>>>> +message Lvalue { >>>>> + required VarRef varref = 1; >>>>> +} >>>>> + >>>>> +message Const { >>>>> + required int32 val = 1; >>>>> +} >>>>> + >>>>> +message BinaryOp { >>>>> + enum Op { >>>>> + PLUS = 0; >>>>> + MINUS = 1; >>>>> + MUL = 2; >>>>> + DIV = 3; >>>>> + MOD = 4; >>>>> + XOR = 5; >>>>> + AND = 6; >>>>> + OR = 7; >>>>> + EQ = 8; >>>>> + NE = 9; >>>>> + LE = 10; >>>>> + GE = 11; >>>>> + LT = 12; >>>>> + GT = 13; >>>>> + }; >>>>> + required Op op = 1; >>>>> + required Rvalue left = 2; >>>>> + required Rvalue right = 3; >>>>> +} >>>>> + >>>>> +message Rvalue { >>>>> + oneof rvalue_oneof { >>>>> + VarRef varref = 1; >>>>> + Const cons = 2; >>>>> + BinaryOp binop = 3; >>>>> + } >>>>> +} >>>>> + >>>>> +message AssignmentStatement { >>>>> + required Lvalue lvalue = 1; >>>>> + required Rvalue rvalue = 2; >>>>> +} >>>>> + >>>>> + >>>>> +message IfElse { >>>>> + required Rvalue cond = 1; >>>>> + required StatementSeq if_body = 2; >>>>> + required StatementSeq else_body = 3; >>>>> +} >>>>> + >>>>> +message While { >>>>> + required Rvalue cond = 1; >>>>> + required StatementSeq body = 2; >>>>> +} >>>>> + >>>>> +message Statement { >>>>> + oneof stmt_oneof { >>>>> + AssignmentStatement assignment = 1; >>>>> + IfElse ifelse = 2; >>>>> + While while_loop = 3; >>>>> + } >>>>> +} >>>>> + >>>>> +message StatementSeq { >>>>> + repeated Statement statements = 1; >>>>> +} >>>>> + >>>>> +message Function { >>>>> + required StatementSeq statements = 1; >>>>> +} >>>>> + >>>>> +package clang_fuzzer; >>>>> >>>>> Added: cfe/trunk/tools/clang-fuzzer/handle-cxx/CMakeLists.txt >>>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-fu >>>>> zzer/handle-cxx/CMakeLists.txt?rev=310408&view=auto >>>>> ============================================================ >>>>> ================== >>>>> --- cfe/trunk/tools/clang-fuzzer/handle-cxx/CMakeLists.txt (added) >>>>> +++ cfe/trunk/tools/clang-fuzzer/handle-cxx/CMakeLists.txt Tue Aug 8 >>>>> 13:15:04 2017 >>>>> @@ -0,0 +1,11 @@ >>>>> +set(LLVM_LINK_COMPONENTS ${LLVM_TARGETS_TO_BUILD}) >>>>> + >>>>> +add_clang_library(clangHandleCXX >>>>> + handle_cxx.cpp >>>>> + >>>>> + LINK_LIBS >>>>> + clangCodeGen >>>>> + clangFrontend >>>>> + clangLex >>>>> + clangTooling >>>>> + ) >>>>> >>>>> Added: cfe/trunk/tools/clang-fuzzer/handle-cxx/handle_cxx.cpp >>>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-fu >>>>> zzer/handle-cxx/handle_cxx.cpp?rev=310408&view=auto >>>>> ============================================================ >>>>> ================== >>>>> --- cfe/trunk/tools/clang-fuzzer/handle-cxx/handle_cxx.cpp (added) >>>>> +++ cfe/trunk/tools/clang-fuzzer/handle-cxx/handle_cxx.cpp Tue Aug 8 >>>>> 13:15:04 2017 >>>>> @@ -0,0 +1,58 @@ >>>>> +//==-- handle_cxx.cpp - Helper function for Clang fuzzers >>>>> ------------------==// >>>>> +// >>>>> +// The LLVM Compiler Infrastructure >>>>> +// >>>>> +// This file is distributed under the University of Illinois Open >>>>> Source >>>>> +// License. See LICENSE.TXT for details. >>>>> +// >>>>> +//===------------------------------------------------------ >>>>> ----------------===// >>>>> +// >>>>> +// Implements HandleCXX for use by the Clang fuzzers. >>>>> +// >>>>> +//===------------------------------------------------------ >>>>> ----------------===// >>>>> + >>>>> +#include "handle_cxx.h" >>>>> + >>>>> +#include "clang/CodeGen/CodeGenAction.h" >>>>> +#include "clang/Frontend/CompilerInstance.h" >>>>> +#include "clang/Lex/PreprocessorOptions.h" >>>>> +#include "clang/Tooling/Tooling.h" >>>>> +#include "llvm/Option/Option.h" >>>>> +#include "llvm/Support/TargetSelect.h" >>>>> + >>>>> +using namespace clang; >>>>> + >>>>> +void clang_fuzzer::HandleCXX(const std::string &S, >>>>> + const std::vector<const char *> >>>>> &ExtraArgs) { >>>>> + llvm::InitializeAllTargets(); >>>>> + llvm::InitializeAllTargetMCs(); >>>>> + llvm::InitializeAllAsmPrinters(); >>>>> + llvm::InitializeAllAsmParsers(); >>>>> + >>>>> + llvm::opt::ArgStringList CC1Args; >>>>> + CC1Args.push_back("-cc1"); >>>>> + for (auto &A : ExtraArgs) >>>>> + CC1Args.push_back(A); >>>>> + CC1Args.push_back("./test.cc"); >>>>> + >>>>> + llvm::IntrusiveRefCntPtr<FileManager> Files( >>>>> + new FileManager(FileSystemOptions())); >>>>> + IgnoringDiagConsumer Diags; >>>>> + IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts = new >>>>> DiagnosticOptions(); >>>>> + DiagnosticsEngine Diagnostics( >>>>> + IntrusiveRefCntPtr<clang::DiagnosticIDs>(new DiagnosticIDs()), >>>>> &*DiagOpts, >>>>> + &Diags, false); >>>>> + std::unique_ptr<clang::CompilerInvocation> Invocation( >>>>> + tooling::newInvocation(&Diagnostics, CC1Args)); >>>>> + std::unique_ptr<llvm::MemoryBuffer> Input = >>>>> + llvm::MemoryBuffer::getMemBuffer(S); >>>>> + Invocation->getPreprocessorOpts().addRemappedFile("./test.cc", >>>>> + Input.release()); >>>>> + std::unique_ptr<tooling::ToolAction> action( >>>>> + tooling::newFrontendActionFactory<clang::EmitObjAction>()); >>>>> + std::shared_ptr<PCHContainerOperations> PCHContainerOps = >>>>> + std::make_shared<PCHContainerOperations>(); >>>>> + action->runInvocation(std::move(Invocation), Files.get(), >>>>> PCHContainerOps, >>>>> + &Diags); >>>>> +} >>>>> + >>>>> >>>>> Added: cfe/trunk/tools/clang-fuzzer/handle-cxx/handle_cxx.h >>>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-fu >>>>> zzer/handle-cxx/handle_cxx.h?rev=310408&view=auto >>>>> ============================================================ >>>>> ================== >>>>> --- cfe/trunk/tools/clang-fuzzer/handle-cxx/handle_cxx.h (added) >>>>> +++ cfe/trunk/tools/clang-fuzzer/handle-cxx/handle_cxx.h Tue Aug 8 >>>>> 13:15:04 2017 >>>>> @@ -0,0 +1,25 @@ >>>>> +//==-- handle_cxx.h - Helper function for Clang fuzzers >>>>> --------------------==// >>>>> +// >>>>> +// The LLVM Compiler Infrastructure >>>>> +// >>>>> +// This file is distributed under the University of Illinois Open >>>>> Source >>>>> +// License. See LICENSE.TXT for details. >>>>> +// >>>>> +//===------------------------------------------------------ >>>>> ----------------===// >>>>> +// >>>>> +// Defines HandleCXX for use by the Clang fuzzers. >>>>> +// >>>>> +//===------------------------------------------------------ >>>>> ----------------===// >>>>> + >>>>> +#ifndef LLVM_CLANG_TOOLS_CLANG_FUZZER_HANDLE_CXX_HANDLECXX_H >>>>> +#define LLVM_CLANG_TOOLS_CLANG_FUZZER_HANDLE_CXX_HANDLECXX_H >>>>> + >>>>> +#include <string> >>>>> +#include <vector> >>>>> + >>>>> +namespace clang_fuzzer { >>>>> +void HandleCXX(const std::string &S, >>>>> + const std::vector<const char *> &ExtraArgs); >>>>> +} // namespace clang_fuzzer >>>>> + >>>>> +#endif >>>>> >>>>> Added: cfe/trunk/tools/clang-fuzzer/proto-to-cxx/CMakeLists.txt >>>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-fu >>>>> zzer/proto-to-cxx/CMakeLists.txt?rev=310408&view=auto >>>>> ============================================================ >>>>> ================== >>>>> --- cfe/trunk/tools/clang-fuzzer/proto-to-cxx/CMakeLists.txt (added) >>>>> +++ cfe/trunk/tools/clang-fuzzer/proto-to-cxx/CMakeLists.txt Tue Aug >>>>> 8 13:15:04 2017 >>>>> @@ -0,0 +1,10 @@ >>>>> +set(LLVM_LINK_COMPONENTS ${LLVM_TARGETS_TO_BUILD}) >>>>> + >>>>> +# Hack to bypass LLVM's CMake source checks so we can have both a >>>>> library and >>>>> +# an executable built from this directory. >>>>> +set(LLVM_OPTIONAL_SOURCES proto_to_cxx.cpp proto_to_cxx_main.cpp) >>>>> + >>>>> +add_clang_library(clangProtoToCXX proto_to_cxx.cpp LINK_LIBS >>>>> clangCXXProto) >>>>> + >>>>> +add_clang_executable(clang-proto-to-cxx proto_to_cxx_main.cpp) >>>>> +target_link_libraries(clang-proto-to-cxx clangProtoToCXX) >>>>> >>>>> Added: cfe/trunk/tools/clang-fuzzer/proto-to-cxx/proto_to_cxx.cpp >>>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-fu >>>>> zzer/proto-to-cxx/proto_to_cxx.cpp?rev=310408&view=auto >>>>> ============================================================ >>>>> ================== >>>>> --- cfe/trunk/tools/clang-fuzzer/proto-to-cxx/proto_to_cxx.cpp (added) >>>>> +++ cfe/trunk/tools/clang-fuzzer/proto-to-cxx/proto_to_cxx.cpp Tue >>>>> Aug 8 13:15:04 2017 >>>>> @@ -0,0 +1,102 @@ >>>>> +//==-- proto_to_cxx.cpp - Protobuf-C++ conversion >>>>> --------------------------==// >>>>> +// >>>>> +// The LLVM Compiler Infrastructure >>>>> +// >>>>> +// This file is distributed under the University of Illinois Open >>>>> Source >>>>> +// License. See LICENSE.TXT for details. >>>>> +// >>>>> +//===------------------------------------------------------ >>>>> ----------------===// >>>>> +// >>>>> +// Implements functions for converting between protobufs and C++. >>>>> +// >>>>> +//===------------------------------------------------------ >>>>> ----------------===// >>>>> + >>>>> +#include "proto_to_cxx.h" >>>>> +#include "cxx_proto.pb.h" >>>>> + >>>>> +#include <ostream> >>>>> +#include <sstream> >>>>> + >>>>> +namespace clang_fuzzer { >>>>> + >>>>> +// Forward decls. >>>>> +std::ostream &operator<<(std::ostream &os, const BinaryOp &x); >>>>> +std::ostream &operator<<(std::ostream &os, const StatementSeq &x); >>>>> + >>>>> +// Proto to C++. >>>>> +std::ostream &operator<<(std::ostream &os, const Const &x) { >>>>> + return os << "(" << x.val() << ")"; >>>>> +} >>>>> +std::ostream &operator<<(std::ostream &os, const VarRef &x) { >>>>> + return os << "a[" << (static_cast<uint32_t>(x.varnum()) % 100) << >>>>> "]"; >>>>> +} >>>>> +std::ostream &operator<<(std::ostream &os, const Lvalue &x) { >>>>> + return os << x.varref(); >>>>> +} >>>>> +std::ostream &operator<<(std::ostream &os, const Rvalue &x) { >>>>> + if (x.has_varref()) return os << x.varref(); >>>>> + if (x.has_cons()) return os << x.cons(); >>>>> + if (x.has_binop()) return os << x.binop(); >>>>> + return os << "1"; >>>>> +} >>>>> +std::ostream &operator<<(std::ostream &os, const BinaryOp &x) { >>>>> + os << "(" << x.left(); >>>>> + switch (x.op()) { >>>>> + case BinaryOp::PLUS: os << "+"; break; >>>>> + case BinaryOp::MINUS: os << "-"; break; >>>>> + case BinaryOp::MUL: os << "*"; break; >>>>> + case BinaryOp::DIV: os << "/"; break; >>>>> + case BinaryOp::MOD: os << "%"; break; >>>>> + case BinaryOp::XOR: os << "^"; break; >>>>> + case BinaryOp::AND: os << "&"; break; >>>>> + case BinaryOp::OR: os << "|"; break; >>>>> + case BinaryOp::EQ: os << "=="; break; >>>>> + case BinaryOp::NE: os << "!="; break; >>>>> + case BinaryOp::LE: os << "<="; break; >>>>> + case BinaryOp::GE: os << ">="; break; >>>>> + case BinaryOp::LT: os << "<"; break; >>>>> + case BinaryOp::GT: os << ">"; break; >>>>> + } >>>>> + return os << x.right() << ")"; >>>>> +} >>>>> +std::ostream &operator<<(std::ostream &os, const AssignmentStatement >>>>> &x) { >>>>> + return os << x.lvalue() << "=" << x.rvalue() << ";\n"; >>>>> +} >>>>> +std::ostream &operator<<(std::ostream &os, const IfElse &x) { >>>>> + return os << "if (" << x.cond() << "){\n" >>>>> + << x.if_body() << "} else { \n" >>>>> + << x.else_body() << "}\n"; >>>>> +} >>>>> +std::ostream &operator<<(std::ostream &os, const While &x) { >>>>> + return os << "while (" << x.cond() << "){\n" << x.body() << "}\n"; >>>>> +} >>>>> +std::ostream &operator<<(std::ostream &os, const Statement &x) { >>>>> + if (x.has_assignment()) return os << x.assignment(); >>>>> + if (x.has_ifelse()) return os << x.ifelse(); >>>>> + if (x.has_while_loop()) return os << x.while_loop(); >>>>> + return os << "(void)0;\n"; >>>>> +} >>>>> +std::ostream &operator<<(std::ostream &os, const StatementSeq &x) { >>>>> + for (auto &st : x.statements()) os << st; >>>>> + return os; >>>>> +} >>>>> +std::ostream &operator<<(std::ostream &os, const Function &x) { >>>>> + return os << "void foo(int *a) {\n" << x.statements() << "}\n"; >>>>> +} >>>>> + >>>>> +// --------------------------------- >>>>> + >>>>> +std::string FunctionToString(const Function &input) { >>>>> + std::ostringstream os; >>>>> + os << input; >>>>> + return os.str(); >>>>> + >>>>> +} >>>>> +std::string ProtoToCxx(const uint8_t *data, size_t size) { >>>>> + Function message; >>>>> + if (!message.ParseFromArray(data, size)) >>>>> + return "#error invalid proto\n"; >>>>> + return FunctionToString(message); >>>>> +} >>>>> + >>>>> +} // namespace clang_fuzzer >>>>> >>>>> Added: cfe/trunk/tools/clang-fuzzer/proto-to-cxx/proto_to_cxx.h >>>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-fu >>>>> zzer/proto-to-cxx/proto_to_cxx.h?rev=310408&view=auto >>>>> ============================================================ >>>>> ================== >>>>> --- cfe/trunk/tools/clang-fuzzer/proto-to-cxx/proto_to_cxx.h (added) >>>>> +++ cfe/trunk/tools/clang-fuzzer/proto-to-cxx/proto_to_cxx.h Tue Aug >>>>> 8 13:15:04 2017 >>>>> @@ -0,0 +1,22 @@ >>>>> +//==-- proto_to_cxx.h - Protobuf-C++ conversion >>>>> ----------------------------==// >>>>> +// >>>>> +// The LLVM Compiler Infrastructure >>>>> +// >>>>> +// This file is distributed under the University of Illinois Open >>>>> Source >>>>> +// License. See LICENSE.TXT for details. >>>>> +// >>>>> +//===------------------------------------------------------ >>>>> ----------------===// >>>>> +// >>>>> +// Defines functions for converting between protobufs and C++. >>>>> +// >>>>> +//===------------------------------------------------------ >>>>> ----------------===// >>>>> + >>>>> +#include <cstdint> >>>>> +#include <cstddef> >>>>> +#include <string> >>>>> + >>>>> +namespace clang_fuzzer { >>>>> +class Function; >>>>> +std::string FunctionToString(const Function &input); >>>>> +std::string ProtoToCxx(const uint8_t *data, size_t size); >>>>> +} >>>>> >>>>> Added: cfe/trunk/tools/clang-fuzzer/proto-to-cxx/proto_to_cxx_main.cpp >>>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-fu >>>>> zzer/proto-to-cxx/proto_to_cxx_main.cpp?rev=310408&view=auto >>>>> ============================================================ >>>>> ================== >>>>> --- cfe/trunk/tools/clang-fuzzer/proto-to-cxx/proto_to_cxx_main.cpp >>>>> (added) >>>>> +++ cfe/trunk/tools/clang-fuzzer/proto-to-cxx/proto_to_cxx_main.cpp >>>>> Tue Aug 8 13:15:04 2017 >>>>> @@ -0,0 +1,30 @@ >>>>> +//==-- proto_to_cxx_main.cpp - Driver for protobuf-C++ conversion >>>>> ----------==// >>>>> +// >>>>> +// The LLVM Compiler Infrastructure >>>>> +// >>>>> +// This file is distributed under the University of Illinois Open >>>>> Source >>>>> +// License. See LICENSE.TXT for details. >>>>> +// >>>>> +//===------------------------------------------------------ >>>>> ----------------===// >>>>> +// >>>>> +// Implements a simple driver to print a C++ program from a protobuf. >>>>> +// >>>>> +//===------------------------------------------------------ >>>>> ----------------===// >>>>> +#include <fstream> >>>>> +#include <iostream> >>>>> +#include <streambuf> >>>>> +#include <string> >>>>> + >>>>> +#include "proto_to_cxx.h" >>>>> + >>>>> +int main(int argc, char **argv) { >>>>> + for (int i = 1; i < argc; i++) { >>>>> + std::fstream in(argv[i]); >>>>> + std::string str((std::istreambuf_iterator<char>(in)), >>>>> + std::istreambuf_iterator<char>()); >>>>> + std::cout << "// " << argv[i] << std::endl; >>>>> + std::cout << clang_fuzzer::ProtoToCxx( >>>>> + reinterpret_cast<const uint8_t *>(str.data()), str.size()); >>>>> + } >>>>> +} >>>>> + >>>>> >>>>> >>>>> _______________________________________________ >>>>> cfe-commits mailing list >>>>> cfe-commits@lists.llvm.org >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits >>>>> >>>> >>>> _______________________________________________ >>>> cfe-commits mailing list >>>> cfe-commits@lists.llvm.org >>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits >>>> >>>> >>> >> >
_______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits