On Tue, Nov 15, 2022 at 06:50:50PM +0100, Stefan Sperling wrote:
> On Tue, Nov 15, 2022 at 03:07:05PM +0100, Leah Neukirchen wrote:
> >
> > I hit the same issue on a 7.2-RELEASE system, which was idle and had
> > roughly 3 weeks of uptime.
> >
> > Stopped at rt_ifa_del+0x39: movb 0x1b6(%rax),%bl
> > Same backtrace as in parent message.
> >
> > The system is virtualized on QEMU/KVM 7.0 on Linux x86_64, has networking
> > over a bridge where radvd 2.19 announces a prefix. The same setup has
> > been running for years with older OpenBSD versions, without issues.
KVM seems to be the crucial point here.
I could not reproduce this issue on real amd64, arm64 and sparc64
hardware within a week.
Using shared VPS amd64 KVM instances with varying CPU configurations
(all at least two cores), I saw this panic exactly twice across a total
of 14 VMs over the course of one week.
The first occured on 7.2-release, like these reports, but got lost to a
reboot as I'm too stupid to use this provider's web console.
The second triggered on a recent snapshot, but didn't provide more than
what is already known.
Thanks to graphical-only VGA console access in semi-broken browser based
VNC applications, I was not able to obtain enough btrace logs from the
croll back buffer (that would scroll up but not down).
For real test machines, I spun up rad(8) to hand out different prefixes
with varying life times and produced traffic, randomly flashed the NDP
cache, deleted addresses, toggled AUTOCONF6, etc.
For VMs, the provider hands out a public /64 via SLAAC by default using
the following /etc/hostname.vio file:
inet6 autoconf -temporary -soii
There I've been using this script for tracing/reproducing on otherwise
completely idle default installations:
btrace -e 'tracepoint:refcnt:ifaddr {
printf("%s %x %u %+d%s", probe, arg0, arg1, arg2, kstack)
}' >/dev/console &
while sleep 3 ; do
# disable SLAAC, keep link-local to avoid churn
ifconfig vio0 inet6 -autoconf
# enable SLAAC, avoid temporary to avoid churn
ifconfig vio0 inet6 autoconf -temporary
done &
One can disable/avoid IPv4 to further reduce ref-count churn in btrace
output and/or play with toggling link-local/temporary addresses as well.
(In my case, all at the cost of potentially losing relevant traces to
stupid web VGA console scroll back buffers.)
Maybe others can reproduce it more easily in their setup, hopefully with
usable tooling that provides copy/paste access to textual serial console
and other modern luxuries.
I'll keep two of the VMs running for a bit longer, but will otherwise
not do more reproducing; maybe I'll find a bug or two these days while
going through our little sys/netinet6/ mess.
> FWIW, I have found that disabling IPv6 autoconf reliably avoids this.
Makes sense, since without SLAAC there is nothing that removes and adds
addresses automatically.
>
> I have also seen a related crash when running the command below. Which
> means that it's not just the nd6 expiry task affected by this issue.
>
> It is not yet known where the actual race is. Help appreciated.
>
> # ifconfig vio0 -inet6 autoconf
>
> login: kernel: protection fault trap, code=0
> Stopped at rt_ifa_del+0x39: movb 0x1b6(%rax),%bl
> ddb{2}> bt
> rt_ifa_del(ffff8000008a0d00,800100,dead0009deadbeef,0) at rt_ifa_del+0x39
> in6_unlink_ifa(ffff8000008a0d00,ffff8000004d72a8) at in6_unlink_ifa+0xae
> in6_purgeaddr(ffff8000008a0d00) at in6_purgeaddr+0x127
> in6_ifdetach(ffff8000004d72a8) at in6_ifdetach+0x19e
> ifioctl(fffffd8782bf95b8,801169ac,ffff800022edac90,ffff800022e24fc8) at
> ifioctl
> +0xdcc
> soo_ioctl(fffffd877fc2ef00,801169ac,ffff800022edac90,ffff800022e24fc8) at
> soo_i
> octl+0x171
> sys_ioctl(ffff800022e24fc8,ffff800022edada0,ffff800022edae00) at
> sys_ioctl+0x2c
> 4
> syscall(ffff800022edae70) at syscall+0x384
> Xsyscall() at Xsyscall+0x128
> end of kernel
> end trace frame: 0x7f7ffffe1900, count: -9
> ddb{2}> ps
> PID TID PPID UID S FLAGS WAIT COMMAND
> *88890 7233 11006 0 7 0x3 ifconfig
>
