Hello I had a LACP crash. The following is a terse description of what we think happened written by Claude.
We hit a total LACP outage on a VPP 26.06 host after 57 days uptime and traced it to per-thread clock drift introduced in 26.06. Symptom - Both LACP bond members (2x E810 100G, DPDK, fast rate, Juniper peers) stopped sending LACPDUs at the same second. Links stayed up, VPP kept receiving the peer's LACPDUs (RX-state CURRENT), main thread healthy, vppctl responsive. Peers went Defaulted/Detached; bonds 0 active members. - "show lacp details": last LACP PDU sent 1700+ s ago, periodic timer stuck at ~27 s remaining, last LACP PDU received "-27 seconds ago" (negative). - "show clock": every worker reports "Thread N offset 7.47 error -27.4". Worker time is ~27 s ahead of main. A second 26.06 host with 56 days uptime shows -21 s. Reading the 26.02 code, the worker offset was reset to main's time at every barrier, so we expect no such skew there, but we have not measured a 26.02 host. Why LACP dies - lacp_update_ntt() (rx_machine.c) runs on the worker that received the PDU and does lacp_start_periodic_timer(vm, mif, 0), i.e. periodic_timer = worker's vlib_time_now = main_now + skew. - lacp_periodic() runs on main and compares against main's clock, so the timer is skew seconds in the future. With a fast-rate peer, every incoming (mismatching) PDU re-arms it one second later, so it never expires and VPP never transmits again. Any single partner-info mismatch locks the bundle once skew exceeds ~1 s. Why the clocks drift - Change 36770 "vppinfra: fix time handling for cpu migration and monotonicity" (merged 2026-03-19, first in 26.06) added clib_max(total_cpu_time, recomputed) in clib_time_verify_frequency() and clamped the worker offset in vlib_worker_thread_barrier_check() so time can never decrease below the pre-barrier value. - Both clamps turn estimation jitter into a forward-only ratchet. Each thread's clock becomes a running maximum, workers can never be pulled back to main, and the skew grows without bound. Before 36770 the worker offset was reset to main's time at every barrier. - Still present in master and v26.10-rc0. Suggestions - Revert or rework the two clamps so workers converge back to main. - Independently, lacp should arm its timers with main-thread time (vlib_get_first_main()), since they are checked on main. - Other worker-armed / main-checked timers (bfd, nat, neighbor aging) are likely exposed too. Workaround for now: restart VPP periodically. Happy to provide full "show clock verbose" / "show lacp details" output. Regards, Baldur
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#27196): https://lists.fd.io/g/vpp-dev/message/27196 Mute This Topic: https://lists.fd.io/mt/121360743/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
