The issue seems to be that the main thread wants to reply to an ARP, but it
timeouts on the worker barrier:
1) why the worker does not yield to the main thread in a timely manner:
workers should always complete processing in less than 1s. You can try to use
elog to identify which nodes takes too long:
https://s3-docs.fd.io/vpp/23.02/developer/corefeatures/eventviewer.html
2) why does the main thread is receiving ARP to begin with? Can you share the
output of 'show int rx'?
Best
ben
> -----Original Message-----
> From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Mechthild
> Buescher via lists.fd.io
> Sent: Thursday, December 22, 2022 12:25
> To: vpp-dev@lists.fd.io
> Subject: Re: [vpp-dev] VPP crashes with
> vlib_worker_thread_barrier_syn_int:thread deadlock
>
> Hi Piotr,
>
>
>
> Thanks for your hint. We could nail down the problem a bit and would like
> to ask for your suggestion on how to solve it.
>
>
>
> The problem occurs when there is traffic received on a DPDK-interface and
> shall be forwarded to a host interface. Here are the snippets of the
> configuration which we think is relevant (note, it’s a different setup
> compared to previous email, here we have one socket and less cpus):
>
>
>
> We configure CPU’s 2,17,18 to be isolated:
>
> # cat /proc/cmdline
>
> BOOT_IMAGE=/vmlinuz-5.3.18-150300.59.76-default root=UUID=439b3b24-9c1d-
> 4b6f-b024-539b50cb7480 rootflags=subvol=@ intel_iommu=on iommu=pt
> intel_idle.max_cstate=0 processor.max_cstate=0 idle=poll
> intel_pstate=disable isolcpus=2,3,4,5,6,17,18,19,20,21,22 nohz=on
> nohz_full=2,3,4,5,6,17,18,19,20,21,22
> rcu_nocbs=2,3,4,5,6,17,18,19,20,21,22 rcu_nocb_poll
> irqaffinity=0,1,7,8,9,10,11,12,13,14,15,16,23,24,25,26,27,28,29,30,31
> hugepagesz=2M hugepages=2048 hugepagesz=1G hugepages=4
> default_hugepagesz=2M transparent_hugepage=never nosoftlookup
> nmi_watchdog=0 tsc=reliable hpet=disable clocksource=tsc skew_tick=1
> mce=ignore_ce splash console=ttyS0,115200 psi=1 audit=1 apparmor=1
> security=apparmor
>
>
>
> And we use those isolated CPU’s for the workers and one non-isolated CPU
> for the main-thread:
>
> cpu {
>
> main-core 1
>
> corelist-workers 2,17,18
>
> }
>
>
>
> The relevant DPDK-interface is Radio-0:
>
> dpdk {
>
> dev default {
>
> num-rx-queues 3
>
> }
>
>
>
> uio-driver vfio-pci
>
>
>
> dev 0000:17:00.1 {
>
> name Radio-0
>
> }
>
> :
>
> }
>
>
>
> And then we have the following configuration:
>
> set interface state Radio-0 up
>
> create host-interface name Vpp2Host
>
> set interface state host-Vpp2Host up
>
> set interface rx-placement host-Vpp2Host main
>
> create sub-interfaces Radio-0 3092
>
> set interface state Radio-0.3092 up
>
> create sub-interfaces host-Vpp2Host 3092
>
> set interface state host-Vpp2Host.3092 up
>
> set interface l2 bridge Radio-0.3092 3092
>
> set interface l2 bridge host-Vpp2Host.3092 3092
>
>
>
> This means, we receive traffic on a DPDK interface and try to forward it
> via a L2 bridge to the host. The DPDK interface is on an isolated CPU
> while the host interface is on a non-isolated CPU. My suspicion is that
> this is the problem – do you agree? Do you have any idea how we can solve
> this? The Radio-0 interface is used for OAM via vlan 3092 (this is what
> you see in the above configuration) as well as for traffic (untagged),
> that’s why we want to have it on an isolated CPU.
>
>
>
> Thank you for your support,
>
>
>
> BR/Mechthild
>
>
>
> From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> on behalf of Bronowski,
> PiotrX via lists.fd.io <piotrx.bronowski=intel....@lists.fd.io>
> Date: Wednesday, 21. December 2022 at 17:24
> To: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io>
> Subject: Re: [vpp-dev] VPP crashes with
> vlib_worker_thread_barrier_syn_int:thread deadlock
>
> Hi Mechthild,
>
>
>
> Your issue is caused by the main thread waiting too long for a worker to
> finish. You may examine these lines in src/vlib/threads.h:
>
>
>
> 171 /* Check for a barrier sync request every 30ms */
>
> 172 #define BARRIER_SYNC_DELAY (0.030000)
>
> 173
>
> 174 #if CLIB_DEBUG > 0
>
> 175 /* long barrier timeout, for gdb... */
>
> 176 #define BARRIER_SYNC_TIMEOUT (600.1)
>
> 177 #else
>
> 178 #define BARRIER_SYNC_TIMEOUT (1.0)
>
> 179 #endif
>
>
>
> Your restart is caused by the timeout defined in these lines. You may
> increase it to investigate your issue (of course it is not a fix). After
> increasing timeout and being in interactive mode you can issue command
> “show run” it will tell you in which node you are spending most of your
> time and potentially identify source of your problem. Alternatively, you
> may use perf tool for that task.
>
> BR,
>
> Piotr
>
>
>
> From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Mechthild
> Buescher via lists.fd.io
> Sent: Wednesday, December 21, 2022 3:24 PM
> To: vpp-dev@lists.fd.io
> Subject: [vpp-dev] VPP crashes with
> vlib_worker_thread_barrier_syn_int:thread deadlock
>
>
>
> Hi,
>
>
>
> We have a severe problem with VPP – it’s cyclic restarting due to the
> following error:
>
>
> Dec 21 09:46:32 843V0N3 vpp[3846]: vlib_worker_thread_barrier_sync_int:
> worker thread deadlock
>
>
>
> This happens on both servers of the setup and it cannot recover. Can you
> please help us to debug this issue?
>
>
>
>
>
> VPP version:
>
> # vppctl show version
>
> vpp v22.02.0-1~g0d1b46707-dirty built by suse on SUSE at 2022-05-
> 02T09:46:05
>
>
>
> which is a built of version 22.02.-1 on SLES 15 SP3 including the dpdk-
> patch 0001-add-patch-to-disable-source-pruning-in-i40e-driver.patch
>
>
>
> The startup.conf:
>
> unix {
>
> nodaemon
>
> log /var/log/vpp/vpp.log
>
> full-coredump
>
> coredump-size unlimited
>
> cli-listen /run/vpp/cli.sock
>
> gid vpp
>
> startup-config /etc/vpp/vpp-static-config.txt
>
> }
>
>
>
> api-trace {
>
> on
>
> }
>
>
>
> api-segment {
>
> gid vpp
>
> }
>
>
>
> socksvr {
>
> socket-name /var/run/vpp/ic-api.sock
>
> }
>
>
>
> memory {
>
> main-heap-page-size 1G
>
> }
>
>
>
> cpu {
>
> main-core 2
>
> corelist-workers 4,6,42,44,46
>
> }
>
>
>
> buffers {
>
> buffers-per-numa 128000
>
> }
>
>
>
> dpdk {
>
> dev default {
>
> num-rx-queues 5
>
> }
>
>
>
> uio-driver vfio-pci
>
>
>
> dev 0000:3b:00.0 {
>
> name Radio-0
>
> }
>
> dev 0000:3b:00.1 {
>
> name Ext-0
>
> }
>
> dev 0000:5e:02.1 {
>
> name NCIC-1-v1
>
> }
>
> }
>
>
>
> plugins {
>
> plugin default { disable }
>
> plugin dpdk_plugin.so { enable }
>
> plugin ioam_plugin.so { enable }
>
> plugin perfmon_plugin.so { enable }
>
> plugin tracedump_plugin.so { enable }
>
> plugin l3xc_plugin.so { enable }
>
> plugin ping_plugin.so { enable }
>
> plugin avf_plugin.so { enable }
>
> plugin acl_plugin.so { enable }
>
> plugin svs_plugin.so { enable }
>
> plugin vrrp_plugin.so { enable }
>
> plugin dhcp_plugin.so { enable }
>
> plugin nat_plugin.so { enable }
>
> plugin abf_plugin.so { enable }
>
> plugin lacp_plugin.so { enable }
>
> plugin flowprobe_plugin.so { enable }
>
> }
>
>
>
> The log gives:
>
> Dec 21 10:33:27 hostname systemd[1]: Starting Vector Packet Processing
> Process...
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: +
> dpdk_devbind=/usr/local/bin/dpdk-devbind.py
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + VPP_CONF=/etc/vpp/ic-
> startup.conf
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64760]: ++ grep -v '#'
> /etc/vpp/ic-startup.conf
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64761]: ++ grep dev
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64762]: ++ grep -v default
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64763]: ++ sed 's/.*dev //'
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64764]: ++ cut '-d ' -f1
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + DEVICES='0000:3b:00.0
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: 0000:3b:00.1
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: 0000:5e:02.1'
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64766]: ++ grep uio-driver
> /etc/vpp/ic-startup.conf
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64767]: ++ sed 's/.*uio-driver
> //'
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64768]: ++ cut '-d ' -f1
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + DPDK_DRV=vfio-pci
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + '[' --start == --stop
> ']'
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + for dev in $DEVICES
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64770]: ++ /usr/local/bin/dpdk-
> devbind.py -s
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64771]: ++ grep 0000:3b:00.0
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + cdev='0000:3b:00.0
> '\''Ethernet Controller XXV710 for 25GbE SFP28 158b'\'' if=Radio-0
> drv=i40e unused=vfio-pci '
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64952]: ++ get_drv
> '0000:3b:00.0 '\''Ethernet Controller XXV710 for 25GbE SFP28 158b'\''
> if=Radio-0 drv=i40e unused=vfio-pci '
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64952]: ++ [[ -z 0000:3b:00.0
> 'Ethernet Controller XXV710 for 25GbE SFP28 158b' if=Radio-0 drv=i40e
> unused=vfio-pci ]]
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64952]: ++ echo i40e
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + drv=i40e
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + check_intf --start
> 0000:3b:00.0 ''\''Ethernet' Controller XXV710 for 25GbE SFP28 '158b'\'''
> if=Radio-0 drv=i40e unused=vfio-pci
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + action=--start
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + shift
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64954]: ++ echo 0000:3b:00.0
> ''\''Ethernet' Controller XXV710 for 25GbE SFP28 '158b'\''' if=Radio-0
> drv=i40e unused=vfio-pci
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64955]: ++ sed 's/.* if=//'
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64956]: ++ cut '-d ' -f1
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + name=Radio-0
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + [[ Radio-0 == *\-\v*
> ]]
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + '[' --start == --stop
> ']'
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + ip l s Radio-0 down
>
> Dec 21 10:33:27 hostname ic-vpp-service.sh[64758]: + /usr/local/bin/dpdk-
> devbind.py -b vfio-pci 0000:3b:00.0
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + for dev in $DEVICES
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[66019]: ++ grep 0000:3b:00.1
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[66018]: ++ /usr/local/bin/dpdk-
> devbind.py -s
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + cdev='0000:3b:00.1
> '\''Ethernet Controller XXV710 for 25GbE SFP28 158b'\'' if=Ext-0 drv=i40e
> unused=vfio-pci '
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[66366]: ++ get_drv
> '0000:3b:00.1 '\''Ethernet Controller XXV710 for 25GbE SFP28 158b'\''
> if=Ext-0 drv=i40e unused=vfio-pci '
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[66366]: ++ [[ -z 0000:3b:00.1
> 'Ethernet Controller XXV710 for 25GbE SFP28 158b' if=Ext-0 drv=i40e
> unused=vfio-pci ]]
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[66366]: ++ echo i40e
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + drv=i40e
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + check_intf --start
> 0000:3b:00.1 ''\''Ethernet' Controller XXV710 for 25GbE SFP28 '158b'\'''
> if=Ext-0 drv=i40e unused=vfio-pci
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + action=--start
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + shift
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[66368]: ++ echo 0000:3b:00.1
> ''\''Ethernet' Controller XXV710 for 25GbE SFP28 '158b'\''' if=Ext-0
> drv=i40e unused=vfio-pci
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[66369]: ++ sed 's/.* if=//'
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[66370]: ++ cut '-d ' -f1
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + name=Ext-0
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + [[ Ext-0 == *\-\v* ]]
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + '[' --start == --stop
> ']'
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + ip l s Ext-0 down
>
> Dec 21 10:33:30 hostname ic-vpp-service.sh[64758]: + /usr/local/bin/dpdk-
> devbind.py -b vfio-pci 0000:3b:00.1
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + for dev in $DEVICES
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[66986]: ++ /usr/local/bin/dpdk-
> devbind.py -s
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[66987]: ++ grep 0000:5e:02.1
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + cdev='0000:5e:02.1
> '\''Ethernet Virtual Function 700 Series 154c'\'' if=NCIC-1-v1 drv=iavf
> unused=vfio-pci '
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[67199]: ++ get_drv
> '0000:5e:02.1 '\''Ethernet Virtual Function 700 Series 154c'\'' if=NCIC-1-
> v1 drv=iavf unused=vfio-pci '
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[67199]: ++ [[ -z '' ]]
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[67199]: ++ echo iavf
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + drv=iavf
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + check_intf --start
> 0000:5e:02.1 ''\''Ethernet' Virtual Function 700 Series '154c'\'''
> if=NCIC-1-v1 drv=iavf unused=vfio-pci
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + action=--start
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + shift
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[67201]: ++ echo 0000:5e:02.1
> ''\''Ethernet' Virtual Function 700 Series '154c'\''' if=NCIC-1-v1
> drv=iavf unused=vfio-pci
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[67202]: ++ sed 's/.* if=//'
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[67203]: ++ cut '-d ' -f1
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + name=NCIC-1-v1
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + [[ NCIC-1-v1 == *\-
> \v* ]]
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + parent=NCIC-1
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + vf=1
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + '[' --start == --stop
> ']'
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + ip l s NCIC-1 vf 1
> spoof off state enable trust on
>
> Dec 21 10:33:33 hostname ic-vpp-service.sh[64758]: + /usr/local/bin/dpdk-
> devbind.py -b vfio-pci 0000:5e:02.1
>
> Dec 21 10:33:34 hostname systemd[1]: Started Vector Packet Processing
> Process.
>
> Dec 21 10:33:37 hostname vnet[67475]: dpdk/cryptodev: dpdk_cryptodev_init:
> Not enough cryptodev resources
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> adl plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> arping plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> builtinurl plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> cdp plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> ct6 plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> dns plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> geneve plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> gtpu plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> http_static plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> ikev2 plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> l2tp plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> lb plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load:
> vat_plugin_register_gpe: lisp_gpe plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> lldp plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> mactime plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> mdata plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> memif plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> nsh plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> nsim plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> oddbuf plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> pppoe plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> rdma plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> stn plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> tls_openssl plugin not loaded...
>
> Dec 21 10:33:37 hostname vnet[67475]: vat-plug/load: vat_plugin_register:
> vmxnet3 plugin not loaded...
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition:282: VR [0]
> sw_if_index 2 VR ID 11 IPv4 transitioning to Backup
>
> Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition_vmac:120:
> Deleting virtual MAC address 00:00:5e:00:01:0b on hardware interface 2
>
> Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_start_stop:768: 1 VRs
> configured, 1 VRs running
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition:282: VR [1]
> sw_if_index 24 VR ID 207 IPv4 transitioning to Backup
>
> Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition_vmac:120:
> Deleting virtual MAC address 00:00:5e:00:01:cf on hardware interface 2
>
> Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_start_stop:768: 2 VRs
> configured, 2 VRs running
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition:282: VR [2]
> sw_if_index 25 VR ID 202 IPv4 transitioning to Backup
>
> Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition_vmac:120:
> Deleting virtual MAC address 00:00:5e:00:01:ca on hardware interface 2
>
> Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_start_stop:768: 3 VRs
> configured, 3 VRs running
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: interface: hw_add_del_mac_address:
> vnet_hw_interface_add_del_mac_address: Secondary MAC Addresses not
> supported for interface index 0
>
> Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition:282: VR [3]
> sw_if_index 26 VR ID 201 IPv4 transitioning to Backup
>
> Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_transition_vmac:120:
> Deleting virtual MAC address 00:00:5e:00:01:c9 on hardware interface 2
>
> Dec 21 10:33:40 hostname vnet[67475]: vrrp_vr_start_stop:768: 4 VRs
> configured, 4 VRs running
>
> Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition:282: VR [0]
> sw_if_index 2 VR ID 11 IPv4 transitioning to Master
>
> Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_addrs:237: Adding
> VR addresses on sw_if_index 2
>
> Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Adding
> virtual MAC address 00:00:5e:00:01:0b on hardware interface 2
>
> Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition:282: VR [1]
> sw_if_index 24 VR ID 207 IPv4 transitioning to Master
>
> Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_addrs:237: Adding
> VR addresses on sw_if_index 24
>
> Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Adding
> virtual MAC address 00:00:5e:00:01:cf on hardware interface 2
>
> Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition:282: VR [2]
> sw_if_index 25 VR ID 202 IPv4 transitioning to Master
>
> Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_addrs:237: Adding
> VR addresses on sw_if_index 25
>
> Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Adding
> virtual MAC address 00:00:5e:00:01:ca on hardware interface 2
>
> Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition:282: VR [3]
> sw_if_index 26 VR ID 201 IPv4 transitioning to Master
>
> Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_addrs:237: Adding
> VR addresses on sw_if_index 26
>
> Dec 21 10:33:44 hostname vnet[67475]: vrrp_vr_transition_vmac:120: Adding
> virtual MAC address 00:00:5e:00:01:c9 on hardware interface 2
>
> Dec 21 10:34:07 hostname vnet[67475]: vrrp_vr_start_stop:732: Attempting
> to start already started VR ([0] sw_if_index 2 VR ID 11 IPv4)
>
> Dec 21 10:34:07 hostname vnet[67475]: vrrp_vr_start_stop:732: Attempting
> to start already started VR ([1] sw_if_index 24 VR ID 207 IPv4)
>
> Dec 21 10:34:07 hostname vnet[67475]: vrrp_vr_start_stop:732: Attempting
> to start already started VR ([2] sw_if_index 25 VR ID 202 IPv4)
>
> Dec 21 10:34:07 hostname vnet[67475]: vrrp_vr_start_stop:732: Attempting
> to start already started VR ([3] sw_if_index 26 VR ID 201 IPv4)
>
> Dec 21 10:34:36 hostname vpp[67475]: vlib_worker_thread_barrier_sync_int:
> worker thread deadlock
>
> Dec 21 10:34:36 hostname vnet[67475]: received signal SIGABRT, PC
> 0x7f7207c5acdb
>
> Dec 21 10:34:36 hostname vnet[67475]: #0 0x00007f7208ea3234
> unix_signal_handler + 0x124
>
> Dec 21 10:34:36 hostname vnet[67475]: #1 0x00007f72087e68c0
> 0x7f72087e68c0
>
> Dec 21 10:34:36 hostname vnet[67475]: #2 0x00007f7207c5acdb gsignal +
> 0x10d
>
> Dec 21 10:34:36 hostname vnet[67475]: #3 0x00007f7207c5c375 abort + 0x177
>
> Dec 21 10:34:36 hostname vnet[67475]: #4 0x0000000000407643 0x407643
>
> Dec 21 10:34:36 hostname vnet[67475]: #5 0x00007f7208e89541
> vlib_worker_thread_barrier_sync_int + 0x451
>
> Dec 21 10:34:36 hostname vnet[67475]: #6 0x00007f720a5aa3af
> vl_api_rpc_call_main_thread + 0x3cf
>
> Dec 21 10:34:36 hostname vnet[67475]: #7 0x00007f7209a95c1b arp_reply +
> 0x7eb
>
> Dec 21 10:34:36 hostname vnet[67475]: #8 0x00007f7208e50b65
> dispatch_pending_node + 0x135
>
> Dec 21 10:34:36 hostname vnet[67475]: #9 0x00007f7208e55214 vlib_main +
> 0xd34
>
> Dec 21 10:34:36 hostname vnet[67475]: #10 0x00007f7208ea2288 thread0 +
> 0x28
>
> Dec 21 10:34:36 hostname vnet[67475]: #11 0x00007f72083b600c
> 0x7f72083b600c
>
> Dec 21 10:34:49 hostname systemd[1]: vpp.service: Main process exited,
> code=dumped, status=6/ABRT
>
>
>
> We have our own controller which uses the API to do some dynamic
> configuration of VPP – apart from different routes in different tables, it
> configures VRRP as follows:
>
> # vppctl show vrrp vr
>
> [0] sw_if_index 2 VR ID 11 IPv4
>
> state Master flags: preempt yes accept yes unicast no
>
> priority: configured 200 adjusted 200
>
> timers: adv interval 100 master adv 100 skew 21 master down 321
>
> virtual MAC 00:00:5e:00:01:0b
>
> addresses 10.95.255.2
>
> peer addresses
>
> tracked interfaces
>
> [1] sw_if_index 24 VR ID 207 IPv4
>
> state Master flags: preempt yes accept yes unicast no
>
> priority: configured 200 adjusted 200
>
> timers: adv interval 100 master adv 100 skew 21 master down 321
>
> virtual MAC 00:00:5e:00:01:cf
>
> addresses 10.94.131.34
>
> peer addresses
>
> tracked interfaces
>
> [2] sw_if_index 25 VR ID 202 IPv4
>
> state Master flags: preempt yes accept yes unicast no
>
> priority: configured 200 adjusted 200
>
> timers: adv interval 100 master adv 100 skew 21 master down 321
>
> virtual MAC 00:00:5e:00:01:ca
>
> addresses 10.94.29.90
>
> peer addresses
>
> tracked interfaces
>
> [3] sw_if_index 26 VR ID 201 IPv4
>
> state Master flags: preempt yes accept yes unicast no
>
> priority: configured 200 adjusted 200
>
> timers: adv interval 100 master adv 100 skew 21 master down 321
>
> virtual MAC 00:00:5e:00:01:c9
>
> addresses 10.95.250.3
>
> peer addresses
>
> tracked interfaces
>
>
>
> The IP addresses of the VRRP master are:
>
>
>
> # vppctl show int addr
>
> Ext-0 (up):
>
> L3 10.95.255.3/29 ip4 table-id 1 fib-idx 4
>
> L3 10.95.255.2/29 ip4 table-id 1 fib-idx 4
>
> Ext-0.507 (up):
>
> L3 10.94.131.35/29 ip4 table-id 2 fib-idx 6
>
> L3 10.94.131.34/29 ip4 table-id 2 fib-idx 6
>
> Ext-0.506 (up):
>
> L3 10.94.29.91/29 ip4 table-id 3 fib-idx 7
>
> L3 10.94.29.90/29 ip4 table-id 3 fib-idx 7
>
> Ext-0.501 (up):
>
> L3 10.95.250.4/29 ip4 table-id 4 fib-idx 8
>
> L3 10.95.250.3/29 ip4 table-id 4 fib-idx 8
>
>
>
> I don’t have gdb on the affected system. On the build server, I only see:
>
> Core was generated by `/usr/bin/vpp -c /etc/vpp/ic-startup.conf'.
>
> Program terminated with signal SIGABRT, Aborted.
>
> #0 0x00007f18c5b23cdb in ?? ()
>
> [Current thread is 1 (LWP 14555)]
>
> (gdb) list
>
> 91 vat_plugin_hash_create ();
>
> 92 #endif
>
> 93
>
> 94 if (!vlib_plugin_path)
>
> 95 vpp_find_plugin_path ();
>
> 96 }
>
> 97
>
> 98 /*
>
> 99 * Default path for runtime data
>
> 100 */
>
> (gdb) bt
>
> #0 0x00007f18c5b23cdb in ?? ()
>
> Backtrace stopped: Cannot access memory at address 0x7f18c116cac0
>
>
>
> Any ideas on what to check or change to get rid of the cyclic vpp restart?
>
>
>
> Any help is appreciated,
>
>
>
> Thank you,
>
>
>
> BR/Mechthild
>
> --------------------------------------------------------------
> Intel Research and Development Ireland Limited
> Registered in Ireland
> Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
> Registered Number: 308263
>
> This e-mail and any attachments may contain confidential material for the
> sole use of the intended recipient(s). Any review or distribution by
> others is strictly prohibited. If you are not the intended recipient,
> please contact the sender and delete all copies.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22406): https://lists.fd.io/g/vpp-dev/message/22406
Mute This Topic: https://lists.fd.io/mt/95806883/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
