Hi, I got this problem on my CI, used AI to find the cause, here is its
report and a way to reproduce the bug every time (I tested it myself,
everything's checks out).
AI report (it seems I can't add more than 1 attachment):
Reproducer and root cause, on Ubuntu 26.04 amd64 with dotnet-runtime-10.0
10.0.12-0ubuntu1~26.04.1 and libunwind8 1.8.3-0ubuntu1.
The attached repro.sh requires Bash, a working Docker or Podman
installation, and network access. It starts a disposable Ubuntu 26.04
linux/amd64 container and installs dotnet-sdk-10.0 inside it; no host
.NET installation is needed. It builds a 30-line console app with no
external NuGet package dependencies in which 32 threads throw their
first exceptions at the same moment, then runs the app repeatedly and
counts aborts with "Fatal error. Internal CLR error. (0x80131506)".
Usage (the optional arguments are runs per case and threads):
```sh
./repro.sh
./repro.sh 200 32
CPUSET=0-3 ./repro.sh 200
CONTAINER_ENGINE=podman ./repro.sh
```
Docker or Podman is selected automatically; CONTAINER_ENGINE overrides
that choice. CPUSET sets the container's CPU affinity through `--cpuset-
cpus`; CPU numbers must be available to the container engine. The script
installs current Ubuntu packages and reports their versions, so results
may change as fixes are published. Native amd64 is preferred for
reproducing this timing-sensitive race; ARM hosts require an engine
configured for amd64 emulation, which may change the observed behavior.
Previously collected results in a stock ubuntu:26.04 container after
`apt install dotnet-sdk-10.0`, using the package versions above:
- 32 CPUs: 33/100 runs abort
- 4 CPUs (CPU affinity restricted to CPUs 0-3): 43/200
- 2 CPUs (CPU affinity restricted to CPUs 0-1): 7/200
- Same runs, but one exception thrown on a single thread first ("warm" in the
script): 0 aborts in every case
- Microsoft's build of the same runtime 10.0.12 (the
builds.dotnet.microsoft.com tarball) and Red Hat's build, same program: 0/600
and 0/400
The real-world trigger is `dotnet build` / `dotnet test` on a solution
with several projects. MSBuild connects to its worker nodes from a
Parallel.For (NodeProviderOutOfProcBase), and every refused connect
throws a SocketException, so the process's first exceptions are thrown
concurrently. We hit it in about one CI run in three, always within the
first second of `dotnet test`.
Stack, symbolized with dotnet-runtime-10.0-dbgsym: IL_Throw ->
SoftwareExceptionFrame::InitAndLink -> SoftwareExceptionFrame::Init ->
EEPolicy::HandleFatalError(0x80131506). Init aborts because
PAL_VirtualUnwind failed. An LD_PRELOAD wrapper around _ULx86_64_step
shows unw_step returning -1, with errno EBADF, while stepping out of
IL_Throw.
Root cause: libunwind 1.8.3, src/mi/Gaddress_validator.c.
_write_validate() creates its address-validation pipe lazily, behind
atomic_flag_test_and_set(). Threads that lose that race read() from fd
-1, get EBADF, and call _open_pipe(), which closes the shared
descriptors while other threads are writing to them. The write fails,
the stack address is reported unreadable, and unw_step fails.
- Introduced upstream by commit ec03043244 ("Simplified address validation"):
https://github.com/libunwind/libunwind/issues/715
- Fixed upstream by https://github.com/libunwind/libunwind/pull/993 (merged
2026-04-24, one pipe per thread). It is not in any release yet; 1.8.3 is the
latest.
- .NET's bundled libunwind reverts ec03043244
(src/native/external/libunwind-version.txt in dotnet/runtime), which is why
builds using the bundled copy are not affected. dotnet/runtime tracks this as
https://github.com/dotnet/runtime/issues/130577.
- dotnet10 sets DOTNET_USE_SYSTEM_LIBUNWIND = true for .NET 9 and later
(debian/eng/dotnet-pkg-info.mk), and false only on arm64, s390x and ppc64el, so
amd64 is the affected architecture.
Either change would fix it:
(a) build dotnet10 with the bundled libunwind on amd64 as well, as already done
on the other three architectures; or
(b) backport libunwind PR #993 into libunwind 1.8.3, which would also fix other
multithreaded programs that unwind with it.
repro.sh is the test case either way: it should show 0 aborts with the
fix. Bug 2143357 (25.10, libunwind 1.8.1, expired as not reproducible)
is the same issue.
** Bug watch added: github.com/libunwind/libunwind/issues #715
https://github.com/libunwind/libunwind/issues/715
** Bug watch added: github.com/dotnet/runtime/issues #130577
https://github.com/dotnet/runtime/issues/130577
** Attachment added: "repro.sh"
https://bugs.launchpad.net/ubuntu/+source/dotnet10/+bug/2163034/+attachment/6007374/+files/repro.sh
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2163034
Title:
dotnet crashed with SIGABRT — fatal CLR error 0x80131506 in
SoftwareExceptionFrame::Init (PAL_VirtualUnwind failure) during dotnet
test
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/dotnet10/+bug/2163034/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
