Hoi,
I've found that issuing a new link-local from Linux will emit a Netlink
address remove followed by an address add with the new address.
Linux CP will see this removal and disable linklocal processing on the
interface in src/plugins/linux-cp/lcp_router.c:686:
if (ip6_address_is_link_local_unicast (&ip_addr_v6 (&nh)))
if (is_del)
ip6_link_disable (sw_if_index);
else
{
ip6_link_enable (sw_if_index, NULL);
ip6_link_set_local_address (sw_if_index, &ip_addr_v6
(&nh));
}
The subsequent add will re-enable and set the link-local address. There
is no ip6_link_remove_local_address() so the code as written makes
sense, but there's a lot of book-keeping that occurs on link
disable/enable, including address listeners, MLD, VRRP, FIB changes and
a few other bits and pieces.
I propose that we disable the link iff there are no addresses left,
rather than unconditionally on ip6ll changes from netlink.
https://gerrit.fd.io/r/c/vpp/+/46038 has this change, but I'm not sure
as the original crash is related to having no ilt_fibs entry for the
sw_if_index, which is done by ip6_link_disable(), but should be
re-enabled with the subsequent add from ip6_link_enable+set_address.
Considering these calls are done under barrier lock, the only
explanation I could come up with, is that the address delete comes in on
batch #1, leaving ilt_fibs[sw_if_index] = ~0, while the address add come
in on batch #2, and the window of time between #1 and #2 is where an
incoming NS can cause the crash.
Matt, what do you think?
On 10.06.2026 21:27, Pim van Pelt via lists.fd.io wrote:
Hoi,
I am running VPP on a few aarch64 machines and observed regular
crashes with a stacktrace that suggests IPv6 FIB lookup issue -
Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: #0 0x0000fc1b8ac808f8
Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: from linux-vdso.so.1
Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: #1 0x0000fc1b8998ac3c
ip6_fib_table_lookup_exact_match + 0x3c
Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: from
/lib/aarch64-linux-gnu/libvnet.so.26.06
Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: #2 0x0000fc1b89a21d2c
proxy_arp_intfc_walk + 0x4030
Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: from
/lib/aarch64-linux-gnu/libvnet.so.26.06
Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: #3 0x0000fc1b89101658
vlib_exit_with_status + 0x808
Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: from
/lib/aarch64-linux-gnu/libvlib.so.26.06
Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: #4 0x0000fc1b89103eb0
vlib_exit_with_status + 0x3060
Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: from
/lib/aarch64-linux-gnu/libvlib.so.26.06
Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: #5 0x0000fc1b8912d88c
vlib_worker_thread_bootstrap_fn + 0x6c
Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: from
/lib/aarch64-linux-gnu/libvlib.so.26.06
Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: #6 0x0000fc1b88de595c
pthread_condattr_setpshared + 0x5bc
Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: from
/lib/aarch64-linux-gnu/libc.so.6
Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: #7 0x0000fc1b88e4bb4c
__clone + 0x2cc
Jun 04 19:42:29 dpu0-ddln0 vpp[1179016]: from
/lib/aarch64-linux-gnu/libc.so.6
I was lucky enough to get two coredumps out of it, which I fed to
Claude and it came back with an analysis that pointed to me having
changed the link-local address on an interface:
The interface did have IPv6/ND enabled — otherwise the
icmp6_neighbor_solicitation node would never have run. What it
transiently lacked was its per-interface link-local FIB table
(ilt_fibs[sw_if_index]), which is a structure distinct from "IPv6
is enabled." That table is created lazily when a link-local
address is added (ip6_ll_fib_create) and freed, with the slot
reset to ~0, when the last FIB_SOURCE_IP6_ND entry is removed
(ip6_ll_table.c:150-154). Crucially, ip6_link_set_local_address()
(ip6_link.c:359-362) changes a link-local address by doing
delete-then-update: it removes the old LL prefix (which frees the
FIB and sets ilt_fibs = ~0), then re-adds the new one (which
recreates it). The ND node stays enabled across this whole
sequence. So the NS did not arrive anywhere unexpected — it hit a
normally-configured interface during the brief window in which its
LL FIB had been torn down and not yet rebuilt. A worker thread
forwarding an ordinary link-local NS in that window read ilt_fibs
== ~0 and segfaulted; two crashes 31 minutes apart is consistent
with two separate LL-address-change events, matching your note
that you'd been changing link-local addresses.
The debugging session made my head spin a little bit as I'm not very
good with gdb, but what the cores do prove: an unguarded ~0 from
ip6_ll_fib_get() in the link-local NS branch causes an out-of-bounds
pool_elt_at_index; the trigger is an NS for a link-local target
arriving while ilt_fibs[sw_if_index] == ~0.
The fix is a simple check of ip6_ll_fib_get() before doing the FIB
lookup, just as happens a few lines further down in the same
vnet/ip6-nd/ip6_nd.c file.
A candidate fix is in https://gerrit.fd.io/r/c/vpp/+/46038 and I have
not observed crashes after applying it, although I'm not certain if
returning FIB_NODE_INDEX_INVALID and dropping the packet is the right
call in this case, or if we have to do something more?
I have the coredumps and symbols here if somebody wants to take a
closer look.
groet,
Pim
--
Pim van Pelt<[email protected]>
PBVP1-RIPEhttps://ipng.ch/
--
Pim van Pelt
PBVP1-RIPE -https://ipng.ch/
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#27070): https://lists.fd.io/g/vpp-dev/message/27070
Mute This Topic: https://lists.fd.io/mt/119745721/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
