Re: Undefined reference with clang16 and address sanitizer
Noah Watkins writes: > Making it through > further in the compilation process, though, we start to hit more > undefined references with the sanitizers turned on with clang16: > > ``` > clang -fsanitize=address -fsanitize=leak -g -O2 -ggdb3 -Wall -W > -Wno-sign-compare -Wmissing-prototypes -Wmissing-declarations > -Wstrict-prototypes -Wpointer-arith -Wbad-function-cast > -Wnested-externs -L.. -fsanitize=address -fsanitize=leak sexp-conv.o > input.o output.o parse.o misc.o ../getopt.o ../getopt1.o -lnettle > -lgmp -o sexp-conv > /usr/bin/ld: ../libnettle.so: undefined reference to > `_nettle_aes192_encrypt_aesni' This, and the other missing symbols, appear to all be for symbols that are expected to be defined in assembly files. E.g, in my x86_64 build, this symbol should be defined in the object file produced by these build rules: /usr/bin/m4 /home/nisse/hack/nettle/m4-utils.m4 /home/nisse/hack/nettle/asm.m4 config.m4 machine.m4 aes192-encrypt-2.asm >aes192-encrypt-2.s gcc -I. -DHAVE_CONFIG_H -g -O2 -ggdb3 -Wall -W -Wno-sign-compare -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wpointer-arith -Wbad-function-cast -Wnested-externs -fpic -MT aes192-encrypt-2.o -MD -MP -MF aes192-encrypt-2.o.d -c aes192-encrypt-2.s As can be verified with $ nm aes192-encrypt-2.o T _nettle_aes192_encrypt_aesni So you need to investigate what corresponding steps produce in your build. (Maybe it would make sense to change the Makefile rules to not use $(CC) for processing assembly files, but traditionally, unix C compilers do the right thing in that case). Also double check that you do a make distclean or use a fresh build directory when changing configure options, since you can get all sorts of weird errors if you have inconsstent object files lying around. Regards, /Niels -- Niels Möller. PGP key CB4962D070D77D7FCB8BA36271D8F1FF368C6677. Internet email is subject to wholesale government surveillance. ___ nettle-bugs mailing list -- nettle-bugs@lists.lysator.liu.se To unsubscribe send an email to nettle-bugs-le...@lists.lysator.liu.se
Re: Undefined reference with clang16 and address sanitizer
Niels, That seems to have fixed the memory leak, thanks. Making it through further in the compilation process, though, we start to hit more undefined references with the sanitizers turned on with clang16: ``` clang -fsanitize=address -fsanitize=leak -g -O2 -ggdb3 -Wall -W -Wno-sign-compare -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wpointer-arith -Wbad-function-cast -Wnested-externs -L.. -fsanitize=address -fsanitize=leak sexp-conv.o input.o output.o parse.o misc.o ../getopt.o ../getopt1.o -lnettle -lgmp -o sexp-conv /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_aes192_encrypt_aesni' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_poly1305_set_key' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_salsa20_2core' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_poly1305_blocks' /usr/bin/ld: ../libnettle.so: undefined reference to `nettle_serpent_decrypt' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_memxor_sse2' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_umac_nh_n' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_camellia_crypt' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_aes128_encrypt_aesni' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_ghash_update_pclmul' /usr/bin/ld: ../libnettle.so: undefined reference to `nettle_memxor3' /usr/bin/ld: ../libnettle.so: undefined reference to `nettle_md5_compress' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_sha256_compress_n_x86_64' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_chacha_core' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_aes256_encrypt_aesni' /usr/bin/ld: ../libnettle.so: undefined reference to `nettle_serpent_encrypt' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_sha256_compress_n_sha_ni' /usr/bin/ld: ../libnettle.so: undefined reference to `nettle_sha3_permute' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_poly1305_block' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_sha1_compress_sha_ni' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_cbc_aes192_encrypt_aesni' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_aes192_decrypt_aesni' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_sha1_compress_x86_64' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_poly1305_digest' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_ghash_set_key_pclmul' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_aes128_decrypt_aesni' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_aes_encrypt' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_aes256_decrypt_aesni' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_aes_decrypt' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_umac_nh' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_cbc_aes256_encrypt_aesni' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_cbc_aes128_encrypt_aesni' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_cpuid' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_memxor_x86_64' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_sha512_compress' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_ghash_update_table' /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_salsa20_core' ``` I'm happy to provide a docker container to reproduce this if that would be helpful. Clang16 is now standard on Fedora 38. - Noah On Mon, May 15, 2023 at 11:05 AM Niels Möller wrote: > > Noah Watkins writes: > > > Any thoughts on this? Since this runs as part of the build process, it > > is blocking one of our pipelines. > > Looks like a missing mpz_clear. I think it should be fixed with > https://git.lysator.liu.se/nettle/nettle/-/commit/966da449232766ad41b9be4f263fcccd4500bd22 > > Thanks for the report, > /Niels > > -- > Niels Möller. PGP key CB4962D070D77D7FCB8BA36271D8F1FF368C6677. > Internet email is subject to wholesale government surveillance. ___ nettle-bugs mailing list -- nettle-bugs@lists.lysator.liu.se To unsubscribe send an email to nettle-bugs-le...@lists.lysator.liu.se
Re: Undefined reference with clang16 and address sanitizer
Noah Watkins writes: > Any thoughts on this? Since this runs as part of the build process, it > is blocking one of our pipelines. Looks like a missing mpz_clear. I think it should be fixed with https://git.lysator.liu.se/nettle/nettle/-/commit/966da449232766ad41b9be4f263fcccd4500bd22 Thanks for the report, /Niels -- Niels Möller. PGP key CB4962D070D77D7FCB8BA36271D8F1FF368C6677. Internet email is subject to wholesale government surveillance. ___ nettle-bugs mailing list -- nettle-bugs@lists.lysator.liu.se To unsubscribe send an email to nettle-bugs-le...@lists.lysator.liu.se
Re: Undefined reference with clang16 and address sanitizer
Niels, Any thoughts on this? Since this runs as part of the build process, it is blocking one of our pipelines. On Thu, May 11, 2023 at 9:36 PM Noah Watkins wrote: > > Here is the leak report > > ``` > make[2]: Leaving directory > '/home/nwatkins/src/redpanda-llvm16/vbuild/debug/clang/v_deps_build/nettle-prefix/src/nettle-build' > echo stamp > eccdata.stamp > ./eccdata curve25519 11 6 64 > ecc-curve25519.hT && mv > ecc-curve25519.hT ecc-curve25519.h > Table size: 256 entries > > = > ==1667484==ERROR: LeakSanitizer: detected memory leaks > > Direct leak of 64 byte(s) in 1 object(s) allocated from: > #0 0x565361b74bfe in __interceptor_malloc > /home/nwatkins/src/redpanda-llvm16/vbuild/llvm/src/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3 > #1 0x565361baf79e in gmp_default_alloc eccdata.c > #2 0x565361bb7fbc in gmp_xalloc_limbs eccdata.c > #3 0x565361bb8a1b in mpz_realloc eccdata.c > #4 0x565361bb92ea in mpz_set > (/home/nwatkins/src/redpanda-llvm16/vbuild/debug/clang/v_deps_build/nettle-prefix/src/nettle-build/eccdata+0x11c2ea) > #5 0x565361bb96b5 in mpz_init_set > (/home/nwatkins/src/redpanda-llvm16/vbuild/debug/clang/v_deps_build/nettle-prefix/src/nettle-build/eccdata+0x11c6b5) > #6 0x565361bc377e in mpz_div_qr eccdata.c > #7 0x565361bc426a in mpz_mod > (/home/nwatkins/src/redpanda-llvm16/vbuild/debug/clang/v_deps_build/nettle-prefix/src/nettle-build/eccdata+0x12726a) > #8 0x565361bf1412 in output_bignum_redc eccdata.c > #9 0x565361be65b7 in output_curve eccdata.c > #10 0x565361bde8e6 in main > (/home/nwatkins/src/redpanda-llvm16/vbuild/debug/clang/v_deps_build/nettle-prefix/src/nettle-build/eccdata+0x1418e6) > #11 0x7f965423bb49 in __libc_start_call_main > (/lib64/libc.so.6+0x27b49) (BuildId: > 245240a31888ad5c11bbc55b18e02d87388f59a9) > #12 0x7f965423bc0a in __libc_start_main@GLIBC_2.2.5 > (/lib64/libc.so.6+0x27c0a) (BuildId: > 245240a31888ad5c11bbc55b18e02d87388f59a9) > #13 0x565361adb364 in _start > (/home/nwatkins/src/redpanda-llvm16/vbuild/debug/clang/v_deps_build/nettle-prefix/src/nettle-build/eccdata+0x3e364) > > SUMMARY: AddressSanitizer: 64 byte(s) leaked in 1 allocation(s). > ``` > > On Sun, May 7, 2023 at 7:38 AM Niels Möller wrote: > > > > Noah Watkins writes: > > > > > (fwiw sanitizer does report a memory leak when eccdata is > > > running at the end of make). > > > > If it looks like the sanitizer could be right, can you share the error > > report? > > > > Regards, > > /Niels > > > > -- > > Niels Möller. PGP key CB4962D070D77D7FCB8BA36271D8F1FF368C6677. > > Internet email is subject to wholesale government surveillance. ___ nettle-bugs mailing list -- nettle-bugs@lists.lysator.liu.se To unsubscribe send an email to nettle-bugs-le...@lists.lysator.liu.se
Re: Undefined reference with clang16 and address sanitizer
Here is the leak report ``` make[2]: Leaving directory '/home/nwatkins/src/redpanda-llvm16/vbuild/debug/clang/v_deps_build/nettle-prefix/src/nettle-build' echo stamp > eccdata.stamp ./eccdata curve25519 11 6 64 > ecc-curve25519.hT && mv ecc-curve25519.hT ecc-curve25519.h Table size: 256 entries = ==1667484==ERROR: LeakSanitizer: detected memory leaks Direct leak of 64 byte(s) in 1 object(s) allocated from: #0 0x565361b74bfe in __interceptor_malloc /home/nwatkins/src/redpanda-llvm16/vbuild/llvm/src/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3 #1 0x565361baf79e in gmp_default_alloc eccdata.c #2 0x565361bb7fbc in gmp_xalloc_limbs eccdata.c #3 0x565361bb8a1b in mpz_realloc eccdata.c #4 0x565361bb92ea in mpz_set (/home/nwatkins/src/redpanda-llvm16/vbuild/debug/clang/v_deps_build/nettle-prefix/src/nettle-build/eccdata+0x11c2ea) #5 0x565361bb96b5 in mpz_init_set (/home/nwatkins/src/redpanda-llvm16/vbuild/debug/clang/v_deps_build/nettle-prefix/src/nettle-build/eccdata+0x11c6b5) #6 0x565361bc377e in mpz_div_qr eccdata.c #7 0x565361bc426a in mpz_mod (/home/nwatkins/src/redpanda-llvm16/vbuild/debug/clang/v_deps_build/nettle-prefix/src/nettle-build/eccdata+0x12726a) #8 0x565361bf1412 in output_bignum_redc eccdata.c #9 0x565361be65b7 in output_curve eccdata.c #10 0x565361bde8e6 in main (/home/nwatkins/src/redpanda-llvm16/vbuild/debug/clang/v_deps_build/nettle-prefix/src/nettle-build/eccdata+0x1418e6) #11 0x7f965423bb49 in __libc_start_call_main (/lib64/libc.so.6+0x27b49) (BuildId: 245240a31888ad5c11bbc55b18e02d87388f59a9) #12 0x7f965423bc0a in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x27c0a) (BuildId: 245240a31888ad5c11bbc55b18e02d87388f59a9) #13 0x565361adb364 in _start (/home/nwatkins/src/redpanda-llvm16/vbuild/debug/clang/v_deps_build/nettle-prefix/src/nettle-build/eccdata+0x3e364) SUMMARY: AddressSanitizer: 64 byte(s) leaked in 1 allocation(s). ``` On Sun, May 7, 2023 at 7:38 AM Niels Möller wrote: > > Noah Watkins writes: > > > (fwiw sanitizer does report a memory leak when eccdata is > > running at the end of make). > > If it looks like the sanitizer could be right, can you share the error > report? > > Regards, > /Niels > > -- > Niels Möller. PGP key CB4962D070D77D7FCB8BA36271D8F1FF368C6677. > Internet email is subject to wholesale government surveillance. ___ nettle-bugs mailing list -- nettle-bugs@lists.lysator.liu.se To unsubscribe send an email to nettle-bugs-le...@lists.lysator.liu.se
Re: Undefined reference with clang16 and address sanitizer
Noah Watkins writes: > (fwiw sanitizer does report a memory leak when eccdata is > running at the end of make). If it looks like the sanitizer could be right, can you share the error report? Regards, /Niels -- Niels Möller. PGP key CB4962D070D77D7FCB8BA36271D8F1FF368C6677. Internet email is subject to wholesale government surveillance. ___ nettle-bugs mailing list -- nettle-bugs@lists.lysator.liu.se To unsubscribe send an email to nettle-bugs-le...@lists.lysator.liu.se
Re: Undefined reference with clang16 and address sanitizer
Niels, Thank you very much for the quick response. > Also, Nettle's configure modifies CFLAGS (which is somewhat debatable), > so it may be more reliable with > > CC="clang -fsanitize=address" This indeed did the trick. It's still a bit mysterious why older versions of clang didn't have this problem, and just generally mysterious why the sanitizer would cause the issue. But, it does seem to be ok now (fwiw sanitizer does report a memory leak when eccdata is running at the end of make). On Tue, Apr 25, 2023 at 10:35 PM Niels Möller wrote: > > Noah Watkins writes: > > > Hi nettle-bugs, > > > > We just updated to clang-16 and are encountering an issue compiling > > nettle with -fsanitize=address. > > > > Configured with > > > > CC=clang CXX=clang++ CFLAGS="-fsanitize=address" > > LDFLAGS="-fsanitize=address" ./configure --disable-documentation > > --enable-x86-aesni > > Note that --enable-x86-aesni has no effect in a fat build (which is the > default). If you really want to unconditionally use aesni instructions, you > need --disable-fat --enable-x86-aesni. Thanks a lot. I'm not entirely sure of the lineage of the flags we are using, but indeed the intention was to unconditionally enable so we'll apply this. > > To be clear, you mean the nettle-3.8.1 release? I was testing with a fresh clone of nettle from git as well as 3.8.1 release. - Noah ___ nettle-bugs mailing list -- nettle-bugs@lists.lysator.liu.se To unsubscribe send an email to nettle-bugs-le...@lists.lysator.liu.se
Re: Undefined reference with clang16 and address sanitizer
Noah Watkins writes: > Hi nettle-bugs, > > We just updated to clang-16 and are encountering an issue compiling > nettle with -fsanitize=address. > > Configured with > > CC=clang CXX=clang++ CFLAGS="-fsanitize=address" > LDFLAGS="-fsanitize=address" ./configure --disable-documentation > --enable-x86-aesni Note that --enable-x86-aesni has no effect in a fat build (which is the default). If you really want to unconditionally use aesni instructions, you need --disable-fat --enable-x86-aesni. Also, Nettle's configure modifies CFLAGS (which is somewhat debatable), so it may be more reliable with CC="clang -fsanitize=address" > I did just test with upstream nettle and the issue appears to be present > there. To be clear, you mean the nettle-3.8.1 release? > /usr/bin/ld: ../libnettle.so: undefined reference to > `_nettle_aes192_encrypt_aesni' > /usr/bin/ld: ../libnettle.so: undefined reference to > `_nettle_poly1305_set_key' > /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_salsa20_2core' > /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_poly1305_blocks' > /usr/bin/ld: ../libnettle.so: undefined reference to `nettle_serpent_decrypt' > /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_memxor_sse2' > /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_umac_nh_n' > /usr/bin/ld: ../libnettle.so: undefined reference to `_nettle_camellia_crypt' > /usr/bin/ld: ../libnettle.so: undefined reference to > `_nettle_aes128_encrypt_aesni' > /usr/bin/ld: ../libnettle.so: undefined reference to > `_nettle_ghash_update_pclmul' > ``` It seems you get link errors on all the assembly files that should go into libnettle.so. The way it's supposed to work, configure should symlink various .asm files from the x86_64/ subdirectories into the build directory, where they are preprocessed with m4 into .s files, and passed to the compiler (unix C compiler frontends traditionally recognize .s files as assembly, and invoke the appropriate assembler). To troubleshoot, I would suggest (i) double check that you start with a clean tree, make distclean, (ii) comparing configure output and build steps between your clang-14 and clang-16 builds, (iii) examine the contents of libnettle.so, so try to figure out if the assembly files are missing completely, or if they're there but with some other symbol names (those are tweaked a bit by fat build logic: names with suffixes like _aesni and _pclmul are typical for fat builds, and setup by the wrapper files in x86_64/fat/). Regards, /Niels -- Niels Möller. PGP key CB4962D070D77D7FCB8BA36271D8F1FF368C6677. Internet email is subject to wholesale government surveillance. ___ nettle-bugs mailing list -- nettle-bugs@lists.lysator.liu.se To unsubscribe send an email to nettle-bugs-le...@lists.lysator.liu.se
