My aspiration for k2k21 was to get ASan working on OpenBSD. The
following is a dump of the current state and questions about future
directions.
There are roughly two parts to ASan, the compiler instrumentation and
the runtime library. The instrumentation is likely much less system
dependent than the runtime. There was an unfinished porting attempt in
LLVM tree and a cleanroom libasan runtime by ori@.
Ori had simple programs correctly detect the problems using his
library. Building such programs required a somewhat careful linker
invocation. My first task was to reduce friction in building such
programs. With a lot of help from people in the hackroom we prepared an
LLVM patch[0] with a variation landed[1] in src.
This got me to the point of at least partially reproducing Ori's results
on trivial programs with a simpler clang command line (-fsanitizer is
all you need once the library is in place).
Unfortunately, I couldn't get more complicated programs, like those
using getaddrinfo(3) to work. They crash with a free(): bogus pointer.
I suspected more instrumentation was required and further realized that
there is a major amount of work that went into the original LLVM ASan
runtime. In particular, ASan reports are very usable and actionable.
This does come with extra complexity and more code. Yet, since we looked
last at the tradeoff of developing a cleanroom implementation, some
things have changed. Namely, OpenBSD tree now carries ASan source. We
might as well try to use it.
This brings me to the second part of this hackathon. Building LLVM ASan
runtime required some sleuthing. In the end, this is enough to get the
builds going after adding some tweaks[2]:
mkdir build
cd build
cmake -G "Unix Makefiles" -DLLVM_ENABLE_PROJECTS='compiler-rt;clang' ../llvm
gmake asan
for i in lib/clang/11.1.0/lib/openbsd/libclang_rt.asan*-x86_64.a; do
p=$(basename $i)
doas install $i /usr/lib/clang/11.1.0/lib/${p%-x86_64.a}.a
done
A simple program can now be built with these libraries. Sadly, they hang
before hitting main. The reason is mmap interposing is not currently
done correctly. The program is spinning in
#0 mmap ()
at
llvm/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:7181
#1 0x000000694511c368 in MmapNamed ()
at llvm/compiler-rt/lib/sanitizer_common/sanitizer_posix.cpp:389
#2 MmapOrDie () at
llvm/compiler-rt/lib/sanitizer_common/sanitizer_posix.cpp:46
#3 0x00000069451113cb in Allocate ()
at llvm/compiler-rt/lib/sanitizer_common/sanitizer_allocator.cpp:218
#4 0x0000006945118594 in __sanitizer::FlagParser::FlagParser() ()
at llvm/compiler-rt/lib/sanitizer_common/sanitizer_flag_parser.cpp:188
This brings me to the "what's next" part. Obviously the mmap
interception needs to be fixed to get anywhere. MmapNamed invokes
internal_mmap which is defined in sanitizer_openbsd.cpp as:
uptr internal_mmap(void *addr, size_t length, int prot, int flags, int fd,
u64 offset) {
return (uptr)mmap(addr, length, prot, flags, fd, offset);
}
This sadly can't work because mmap in sanitizer_common_interceptors.inc
calls internal_mmap if !asan_inited and hence the endless
recursion. Other systems have alternative ways of reaching the original
libc mmap. E.g. directly calling mmap syscall in sanitizer_linux.cpp or
calling __mmap in sanitizer_netbsd.cpp. Neither of these seems to be
available on OpenBSD. So, what can we do to interpose mmap?
Thanks
Greg
[0] https://reviews.llvm.org/D109051
[1]
https://github.com/openbsd/src/commit/0b99cc4d5d8311a90145e8f4c6ae23275c275c52
[2]
https://github.com/blackgnezdo/llvm-project/commit/cd697c9c442380c4ed01c21f52b3fea7671ed644
