NP, sorry for the issues, code simply not tested multi-core. BTW we just merged a refactor patch which converts the dns resolver into a plugin. Later this afternoon, I’ll do some multi-core testing. It may take a bit of work to repro and fix the problem you’ve reported.
Dave From: Carlito Nueno <carlitonu...@gmail.com> Sent: Thursday, August 22, 2019 10:55 AM To: Dave Barach (dbarach) <dbar...@cisco.com> Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Crash when using dns_name_server Thanks Dave! Let me know if you need me do more tests or gather more info. On Thu, Aug 22, 2019 at 4:48 AM Dave Barach (dbarach) <dbar...@cisco.com<mailto:dbar...@cisco.com>> wrote: Ack. The DNS server has had zero multi-core testing, aside from what you’ve done. I’ll look at it when I can. From: Carlito Nueno <carlitonu...@gmail.com<mailto:carlitonu...@gmail.com>> Sent: Wednesday, August 21, 2019 10:03 PM To: Dave Barach (dbarach) <dbar...@cisco.com<mailto:dbar...@cisco.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Crash when using dns_name_server Hi Dave, Sorry about the late reply. I used below configs to eliminate most of the complexity. I did not see binary-api being truncated. Steps: 1. I used basic vpp.conf (see below) without the dns_name_server commands 2. gdb run -c /etc/vpp/startup.conf (see below) 3. sudo vppctl 4. Entered dns_name_server commands manually 5. ping google.com<http://google.com> 6. vpp crash Outputs collected: gbd run, gdb backtrace, syslog Step 4: DNS cache output vpp# nat44 add identity mapping external TenGigabitEthernet8/0/0 udp 53053 vpp# bin dns_name_server_add_del 8.8.8.8 vpp# bin dns_enable_disable vpp# sh dns cache verbose DNS cache contains 15 entries bserver-1.kbfs.keybaseapi.com<http://bserver-1.kbfs.keybaseapi.com> -> bserver-1.kbfs.keybaseapi.com<http://bserver-1.kbfs.keybaseapi.com>: 34.235.251.175 [59] 52.54.47.119 [59] TTL left 593.7 mdserver-0.kbfs.keybaseapi.com<http://mdserver-0.kbfs.keybaseapi.com> -> mdserver-0.kbfs.keybaseapi.com<http://mdserver-0.kbfs.keybaseapi.com>: 34.225.12.137 [45] 34.197.228.196 [45] TTL left 593.7 vortex.data.microsoft.com<http://vortex.data.microsoft.com> -> vortex.data.microsoft.com<http://vortex.data.microsoft.com>: 64.4.54.254 [263] TTL left 593.9 api.keybase.io<http://api.keybase.io> -> api.keybase.io<http://api.keybase.io>: 35.153.89.209 [34] 52.4.215.1 [34] TTL left 594.0 push.apple.com<http://push.apple.com> -> push.apple.com<http://push.apple.com>: TTL left 594.4 api.dropboxapi.com<http://api.dropboxapi.com> -> api.dropboxapi.com<http://api.dropboxapi.com>: 162.125.7.7 [59] TTL left 595.0 people-pa.clients6.google.com<http://people-pa.clients6.google.com> -> people-pa.clients6.google.com<http://people-pa.clients6.google.com>: 172.217.6.42 [240] TTL left 595.0 bolt.dropbox.com<http://bolt.dropbox.com> -> bolt.dropbox.com<http://bolt.dropbox.com>: 162.125.34.129 [59] TTL left 595.1 www.google.com<http://www.google.com> -> www.google.com<http://www.google.com>: 172.217.0.36 [263] TTL left 595.1 play.google.com<http://play.google.com> -> play.google.com<http://play.google.com>: 172.217.5.110 [27] TTL left 595.4 mail.google.com<http://mail.google.com> -> mail.google.com<http://mail.google.com>: 172.217.6.37 [299] TTL left 595.5 gateway-carry.icloud.com<http://gateway-carry.icloud.com> -> gateway-carry.icloud.com<http://gateway-carry.icloud.com>: 17.248.128.151 [59] 17.248.128.168 [59] 17.248.128.169 [59] 17.248.128.171 [59] 17.248.128.178 [59] 17.248.128.232 [59] 17.248.128.172 [59] 17.248.128.142 [59] TTL left 595.5 push.services.mozilla.com<http://push.services.mozilla.com> -> push.services.mozilla.com<http://push.services.mozilla.com>: 35.164.35.9 [56] TTL left 599.5 0.client-channel.google.com<http://0.client-channel.google.com> -> 0.client-channel.google.com<http://0.client-channel.google.com>: 74.125.28.189 [239] TTL left 599.6 airtable.com<http://airtable.com> -> airtable.com<http://airtable.com>: 3.221.153.172 [35] 34.193.210.213 [35] 52.22.150.146 [35] TTL left 599.6 Step 2: gdb run (gdb) run -c /etc/vpp/startup.conf Starting program: /usr/bin/vpp -c /etc/vpp/startup.conf [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". vlib_plugin_early_init:361: plugin path /usr/lib/x86_64-linux-gnu/vpp_plugins:/usr/lib/vpp_plugins load_one_plugin:189: Loaded plugin: abf_plugin.so (Access Control List (ACL) Based Forwarding) load_one_plugin:189: Loaded plugin: acl_plugin.so (Access Control Lists (ACL)) load_one_plugin:189: Loaded plugin: avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device Driver) load_one_plugin:189: Loaded plugin: cdp_plugin.so (Cisco Discovery Protocol (CDP)) load_one_plugin:189: Loaded plugin: crypto_ia32_plugin.so (Intel IA32 Software Crypto Engine) load_one_plugin:189: Loaded plugin: crypto_ipsecmb_plugin.so (Intel IPSEC Multi-buffer Crypto Engine) load_one_plugin:189: Loaded plugin: crypto_openssl_plugin.so (OpenSSL Crypto Engine) load_one_plugin:189: Loaded plugin: ct6_plugin.so (IPv6 Connection Tracker) load_one_plugin:189: Loaded plugin: dpdk_plugin.so (Data Plane Development Kit (DPDK)) load_one_plugin:189: Loaded plugin: flowprobe_plugin.so (Flow per Packet) load_one_plugin:189: Loaded plugin: gbp_plugin.so (Group Based Policy (GBP)) load_one_plugin:189: Loaded plugin: gtpu_plugin.so (GPRS Tunnelling Protocol, User Data (GTPv1-U)) load_one_plugin:189: Loaded plugin: hs_apps_plugin.so (Host Stack Applications) load_one_plugin:189: Loaded plugin: http_static_plugin.so (HTTP Static Server) load_one_plugin:189: Loaded plugin: igmp_plugin.so (Internet Group Management Protocol (IGMP)) load_one_plugin:189: Loaded plugin: ikev2_plugin.so (Internet Key Exchange (IKEv2) Protocol) load_one_plugin:189: Loaded plugin: ila_plugin.so (Identifier Locator Addressing (ILA) for IPv6) load_one_plugin:189: Loaded plugin: ioam_plugin.so (Inbound Operations, Administration, and Maintenance (OAM)) load_one_plugin:117: Plugin disabled (default): ixge_plugin.so load_one_plugin:189: Loaded plugin: l2e_plugin.so (Layer 2 (L2) Emulation) load_one_plugin:189: Loaded plugin: l3xc_plugin.so (L3 Cross-Connect (L3XC)) load_one_plugin:189: Loaded plugin: lacp_plugin.so (Link Aggregation Control Protocol (LACP)) load_one_plugin:189: Loaded plugin: lb_plugin.so (Load Balancer (LB)) load_one_plugin:189: Loaded plugin: mactime_plugin.so (Time-based MAC Source Address Filter) load_one_plugin:189: Loaded plugin: map_plugin.so (Mapping of Address and Port (MAP)) load_one_plugin:189: Loaded plugin: memif_plugin.so (Packet Memory Interface (memif) -- Experimental) load_one_plugin:189: Loaded plugin: nat_plugin.so (Network Address Translation (NAT)) load_one_plugin:189: Loaded plugin: nsh_plugin.so (Network Service Header (NSH)) load_one_plugin:189: Loaded plugin: nsim_plugin.so (Network Delay Simulator) load_one_plugin:189: Loaded plugin: perfmon_plugin.so (Performance Monitor) load_one_plugin:189: Loaded plugin: pppoe_plugin.so (PPP over Ethernet (PPPoE)) load_one_plugin:189: Loaded plugin: quic_plugin.so (Quic transport protocol) load_one_plugin:189: Loaded plugin: rdma_plugin.so (RDMA IBverbs Device Driver) load_one_plugin:117: Plugin disabled (default): sctp_plugin.so load_one_plugin:189: Loaded plugin: srv6ad_plugin.so (Dynamic Segment Routing for IPv6 (SRv6) Proxy) load_one_plugin:189: Loaded plugin: srv6am_plugin.so (Masquerading Segment Routing for IPv6 (SRv6) Proxy) load_one_plugin:189: Loaded plugin: srv6as_plugin.so (Static Segment Routing for IPv6 (SRv6) Proxy) load_one_plugin:189: Loaded plugin: stn_plugin.so (VPP Steals the NIC (STN) for Container Integration) load_one_plugin:189: Loaded plugin: svs_plugin.so (Source Virtual Routing and Fowarding (VRF) Select) load_one_plugin:189: Loaded plugin: tlsmbedtls_plugin.so (Transport Layer Security (TLS) Engine, Mbedtls Based) load_one_plugin:189: Loaded plugin: tlsopenssl_plugin.so (Transport Layer Security (TLS) Engine, OpenSSL Based) load_one_plugin:117: Plugin disabled (default): unittest_plugin.so load_one_plugin:189: Loaded plugin: vmxnet3_plugin.so (VMWare Vmxnet3 Device Driver) /usr/bin/vpp[9185]: clib_elf_parse_file: open `linux-vdso.so.1': No such file or directory /usr/bin/vpp[9185]: vnet_feature_init:143: WARNING: arp arc: last node is error-drop, but expected arp-disabled! /usr/bin/vpp[9185]: perfmon: No table for cpuid 506f1 /usr/bin/vpp[9185]: perfmon: model 5f, stepping 1 /usr/bin/vpp[9185]: dpdk: Unsupported PCI device 0x8086:0x19e2 found at PCI address 0000:01:00.0 /usr/bin/vpp[9185]: vlib_pci_bind_to_uio: Skipping PCI device 0000:05:00.0 as host interface enp5s0 is up /usr/bin/vpp[9185]: dpdk: EAL init args: -c c -n 4 --in-memory --file-prefix vpp -b 0000:05:00.0 --master-lcore 2 [New Thread 0x7fff90cff700 (LWP 9203)] [New Thread 0x7fff8bfff700 (LWP 9204)] loop0 tap0 vlib_worker_thread_barrier_sync_int: worker thread deadlock Thread 1 "vpp_main" received signal SIGABRT, Aborted. __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. ----------------------- Step 6: gdb backtrace (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #1 0x00007ffff61ca801 in __GI_abort () at abort.c:79 #2 0x000055555555bdf3 in os_panic () at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vpp/vnet/main.c:355 #3 0x00007ffff6ab621b in vlib_worker_thread_barrier_sync_int ( vm=vm@entry=0x7ffff6ce6a00 <vlib_global_main>, func_name=func_name@entry=0x7ffff6ad2ab0 <__FUNCTION__.43806> "vlib_cli_dispatch_sub_commands") at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/threads.c:1476 #4 0x00007ffff6a6c937 in vlib_cli_dispatch_sub_commands ( vm=vm@entry=0x7ffff6ce6a00 <vlib_global_main>, cm=cm@entry=0x7ffff6ce6c08 <vlib_global_main+520>, input=input@entry=0x7fffb6a36f60, parent_command_index=<optimized out>) at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/cli.c:643 #5 0x00007ffff6a6c718 in vlib_cli_dispatch_sub_commands ( vm=vm@entry=0x7ffff6ce6a00 <vlib_global_main>, cm=cm@entry=0x7ffff6ce6c08 <vlib_global_main+520>, input=input@entry=0x7fffb6a36f60, parent_command_index=<optimized out>) at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/cli.c:606 #6 0x00007ffff6a6c718 in vlib_cli_dispatch_sub_commands ( vm=vm@entry=0x7ffff6ce6a00 <vlib_global_main>, cm=cm@entry=0x7ffff6ce6c08 <vlib_global_main+520>, input=input@entry=0x7fffb6a36f60, parent_command_index=parent_command_index@entry=0) at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/cli.c:606 #7 0x00007ffff6a6cab8 in vlib_cli_input () at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/cli.c:746 #8 0x00007ffff6ac5be2 in unix_cli_process_input (cm=0x7ffff6ce74c0 <unix_cli_main>, cli_file_index=0) at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/unix/cli.c:2525 ---Type <return> to continue, or q <return> to quit--- #9 unix_cli_process () at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/unix/cli.c:2641 #10 0x00007ffff6a881a6 in vlib_process_bootstrap (_a=<optimized out>) at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/main.c:1468 #11 0x00007ffff65a3834 in clib_calljmp () from /usr/lib/x86_64-linux-gnu/libvppinfra.so.19.08 #12 0x00007fffb5836b50 in ?? () #13 0x00007ffff6a8bf59 in vlib_process_startup (f=0x0, p=0x7fffb6a26000, vm=0x7ffff6ce6a00 <vlib_global_main>) at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vppinfra/types.h:133 #14 dispatch_process (vm=0x7ffff6ce6a00 <vlib_global_main>, p=0x7fffb6a26000, last_time_stamp=0, f=0x0) at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/main.c:1535 #15 0x0000000000000000 in ?? () /var/log/syslog Aug 21 17:39:37 test1 vpp: buffer: vlib_physmem_shared_map_create: pmalloc_map_pages: Unable to fulfill huge page allocation request: No such file or directory Aug 21 17:39:37 test1 vpp: buffer: falling back to non-hugepage backed buffer pool Aug 21 17:39:37 test1 vpp: perfmon: No table for cpuid 506f1 Aug 21 17:39:37 test1 vpp: perfmon: model 5f, stepping 1 Info and configs used ++++++++++++++++++++++ Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-45-generic x86_64) ----------------------- vpp# sh version verbose Version: v19.08-release Compiled by: root Compile host: 365637461ad3 Compile date: Wed Aug 21 18:20:49 UTC 2019 Compile location: /w/workspace/vpp-beta-merge-1908-ubuntu1804 Compiler: GCC 8.3.0 Current PID: 9617 ------------------------ startup.conf unix { nodaemon log /var/log/vpp/vpp.log full-coredump cli-listen /run/vpp/cli.sock gid vpp startup-config /home/test1/vpp.conf } api-trace { on } api-segment { gid vpp } socksvr { default } statseg { default } cpu { main-core 2 corelist-workers 3 } nat { translation hash buckets 16000 max translations per user 500 endpoint-dependent } ------------------------- vpp.conf set int state TenGigabitEthernet8/0/0 up set int state GigabitEthernet2/0/0 up set int state GigabitEthernet3/0/0 up set int ip address TenGigabitEthernet8/0/0 72.77.156.155/29<http://72.77.156.155/29> ip route add 0.0.0.0/0<http://0.0.0.0/0> via 72.77.156.158 loopback create set int l2 bridge loop0 1 bvi set int ip address loop0 10.155.6.1/24<http://10.155.6.1/24> set int state loop0 up set int l2 bridge GigabitEthernet2/0/0 1 set int l2 bridge GigabitEthernet3/0/0 1 comment { dhcp server listens on this interface } create tap id 0 host-ip4-addr 10.155.6.2/24<http://10.155.6.2/24> host-if-name novlan set int l2 bridge tap0 1 set int state tap0 up nat44 add interface address TenGigabitEthernet8/0/0 set interface nat44 in loop0 out TenGigabitEthernet8/0/0 ------------------------- dnsmasq.conf # Gateway dhcp-option=novlan,3,10.155.6.1 # DNS server dhcp-option=novlan,6,10.155.6.1 -------------------------- Really appreciate your help! On Sat, Aug 17, 2019 at 1:37 PM Dave Barach via Lists.Fd.Io<http://Lists.Fd.Io> <dbarach=cisco....@lists.fd.io<mailto:cisco....@lists.fd.io>> wrote: I am trying to use DNS server and on "ping google.com<http://google.com>" VPP is crashing Color me unsurprised. Please fix your configuration, to the point where first-order obvious errors have been eliminated. It looks to me like the “binary-api dns_name_server_add_del 8.8.8.8” command has been truncated: Aug 13 21:31:10 test1-vpp vnet[853]: unknown input `add_del 8.8.8.8 Aug 13 21:31:28 test1-vpp vnet[853]: dns cache: add / del / clear required.. Aug 13 21:31:36 test1-vpp vnet[853]: vl_api_dns_resolve_name_reply_t_handler:2556: ip4 address 23.75.7.244 Aug 13 21:32:24 test1-vpp vnet[853]: dns cache: add / del / clear required.. Aug 13 21:34:43 test1-vpp vnet[853]: resolve_event:247: name server 8.8.8.8 backfire Here is the config that I use on my home gateway. uncomment { nat44 add identity mapping external GigabitEthernet3/0/0 udp 53053 } uncomment { bin dns_name_server_add_del 8.8.8.8 } uncomment { bin dns_enable_disable } $ ping google.com<http://google.com> # on my desktop, behind the vpp gateway PING google.com<http://google.com> (172.217.7.174) 56(84) bytes of data. 64 bytes from 172.217.7.174 (172.217.7.174): icmp_seq=1 ttl=54 time=26.3 ms 64 bytes from 172.217.7.174 (172.217.7.174): icmp_seq=2 ttl=54 time=21.5 ms vpp# sh dns cache weatherlink.com<http://weatherlink.com> -> weatherlink.com<http://weatherlink.com>: 54.172.164.1 [21338] TTL left 489.9 xxl.trane.com<http://xxl.trane.com> -> xxl.trane.com<http://xxl.trane.com>: 168.65.229.203 [781] EXPIRED google.com<http://google.com> -> google.com<http://google.com>: 172.217.7.174 [189] TTL left 552.1 www.google.com<http://www.google.com> -> www.google.com<http://www.google.com>: 172.217.164.164 [189] TTL left 569.6 dns.msftncsi.com<http://dns.msftncsi.com> -> dns.msftncsi.com<http://dns.msftncsi.com>: 131.107.255.255 [15] TTL left 569.9 kv801-prod.do.dsp.mp.microsoft.com<http://kv801-prod.do.dsp.mp.microsoft.com> -> kv801-prod.do.dsp.mp.microsoft.com<http://kv801-prod.do.dsp.mp.microsoft.com>: 23.67.120.218 [19] TTL left 571.7 geo-prod.do.dsp.mp.microsoft.com<http://geo-prod.do.dsp.mp.microsoft.com> -> geo-prod.do.dsp.mp.microsoft.com<http://geo-prod.do.dsp.mp.microsoft.com>: 40.69.221.239 [121] TTL left 571.1 outlook.office365.com<http://outlook.office365.com> -> outlook.office365.com<http://outlook.office365.com>: 52.96.62.226 [38] TTL left 571.5 174.7.217.172.in-addr.arpa -> 174.7.217.172.in-addr.arpa: TTL left 573.6 client.wns.windows.com<http://client.wns.windows.com> -> client.wns.windows.com<http://client.wns.windows.com>: 52.242.211.89 [299] TTL left 574.1 www.nr-extensions.tk<http://www.nr-extensions.tk> -> www.nr-extensions.tk<http://www.nr-extensions.tk>: 165.22.1.78 [190] TTL left 589.7 ip-api.com<http://ip-api.com> -> ip-api.com<http://ip-api.com>: 147.135.15.186 [99] TTL left 589.9 www.instagram.com<http://www.instagram.com> -> www.instagram.com<http://www.instagram.com>: 31.13.65.174 [59] TTL left 590.2 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13779): https://lists.fd.io/g/vpp-dev/message/13779 Mute This Topic: https://lists.fd.io/mt/32881233/675621 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev%2bow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [carlitonu...@gmail.com<mailto:carlitonu...@gmail.com>] -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13817): https://lists.fd.io/g/vpp-dev/message/13817 Mute This Topic: https://lists.fd.io/mt/32881233/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
