Thanks Dave! Let me know if you need me do more tests or gather more info.
On Thu, Aug 22, 2019 at 4:48 AM Dave Barach (dbarach) <dbar...@cisco.com>
wrote:
> Ack. The DNS server has had *zero* multi-core testing, aside from what
> you’ve done. I’ll look at it when I can.
>
>
>
> *From:* Carlito Nueno <carlitonu...@gmail.com>
> *Sent:* Wednesday, August 21, 2019 10:03 PM
> *To:* Dave Barach (dbarach) <dbar...@cisco.com>
> *Cc:* vpp-dev@lists.fd.io
> *Subject:* Re: [vpp-dev] Crash when using dns_name_server
>
>
>
> Hi Dave,
>
> Sorry about the late reply.
>
> I used below configs to eliminate most of the complexity. I did not see
> binary-api being truncated.
>
> Steps:
> 1. I used basic vpp.conf (see below) without the dns_name_server commands
> 2. gdb run -c /etc/vpp/startup.conf (see below)
> 3. sudo vppctl
> 4. Entered dns_name_server commands manually
> 5. ping google.com
> 6. vpp crash
>
> Outputs collected: gbd run, gdb backtrace, syslog
>
> *Step 4: DNS cache output*
>
> vpp# nat44 add identity mapping external TenGigabitEthernet8/0/0 udp 53053
> vpp# bin dns_name_server_add_del 8.8.8.8
> vpp# bin dns_enable_disable
> vpp# sh dns cache verbose
> DNS cache contains 15 entries
> bserver-1.kbfs.keybaseapi.com -> bserver-1.kbfs.keybaseapi.com:
> 34.235.251.175 [59] 52.54.47.119 [59] TTL left 593.7
> mdserver-0.kbfs.keybaseapi.com -> mdserver-0.kbfs.keybaseapi.com:
> 34.225.12.137 [45] 34.197.228.196 [45] TTL left 593.7
> vortex.data.microsoft.com -> vortex.data.microsoft.com: 64.4.54.254
> [263] TTL left 593.9
> api.keybase.io -> api.keybase.io: 35.153.89.209 [34] 52.4.215.1 [34]
> TTL left 594.0
> push.apple.com -> push.apple.com: TTL left 594.4
> api.dropboxapi.com -> api.dropboxapi.com: 162.125.7.7 [59] TTL left
> 595.0
> people-pa.clients6.google.com -> people-pa.clients6.google.com:
> 172.217.6.42 [240] TTL left 595.0
> bolt.dropbox.com -> bolt.dropbox.com: 162.125.34.129 [59] TTL left
> 595.1
> www.google.com -> www.google.com: 172.217.0.36 [263] TTL left 595.1
> play.google.com -> play.google.com: 172.217.5.110 [27] TTL left
> 595.4
> mail.google.com -> mail.google.com: 172.217.6.37 [299] TTL left
> 595.5
> gateway-carry.icloud.com -> gateway-carry.icloud.com: 17.248.128.151
> [59] 17.248.128.168 [59] 17.248.128.169 [59] 17.248.128.171 [59]
> 17.248.128.178 [59] 17.248.128.232 [59] 17.248.128.172 [59] 17.248.128.142
> [59] TTL left 595.5
> push.services.mozilla.com -> push.services.mozilla.com: 35.164.35.9
> [56] TTL left 599.5
> 0.client-channel.google.com -> 0.client-channel.google.com:
> 74.125.28.189 [239] TTL left 599.6
> airtable.com -> airtable.com: 3.221.153.172 [35] 34.193.210.213 [35]
> 52.22.150.146 [35] TTL left 599.6
>
>
> *Step 2: gdb run*
>
> (gdb) run -c /etc/vpp/startup.conf
> Starting program: /usr/bin/vpp -c /etc/vpp/startup.conf
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> vlib_plugin_early_init:361: plugin path
> /usr/lib/x86_64-linux-gnu/vpp_plugins:/usr/lib/vpp_plugins
> load_one_plugin:189: Loaded plugin: abf_plugin.so (Access Control List
> (ACL) Based Forwarding)
> load_one_plugin:189: Loaded plugin: acl_plugin.so (Access Control Lists
> (ACL))
> load_one_plugin:189: Loaded plugin: avf_plugin.so (Intel Adaptive Virtual
> Function (AVF) Device Driver)
> load_one_plugin:189: Loaded plugin: cdp_plugin.so (Cisco Discovery
> Protocol (CDP))
> load_one_plugin:189: Loaded plugin: crypto_ia32_plugin.so (Intel IA32
> Software Crypto Engine)
> load_one_plugin:189: Loaded plugin: crypto_ipsecmb_plugin.so (Intel IPSEC
> Multi-buffer Crypto Engine)
> load_one_plugin:189: Loaded plugin: crypto_openssl_plugin.so (OpenSSL
> Crypto Engine)
> load_one_plugin:189: Loaded plugin: ct6_plugin.so (IPv6 Connection Tracker)
> load_one_plugin:189: Loaded plugin: dpdk_plugin.so (Data Plane Development
> Kit (DPDK))
> load_one_plugin:189: Loaded plugin: flowprobe_plugin.so (Flow per Packet)
> load_one_plugin:189: Loaded plugin: gbp_plugin.so (Group Based Policy
> (GBP))
> load_one_plugin:189: Loaded plugin: gtpu_plugin.so (GPRS Tunnelling
> Protocol, User Data (GTPv1-U))
> load_one_plugin:189: Loaded plugin: hs_apps_plugin.so (Host Stack
> Applications)
> load_one_plugin:189: Loaded plugin: http_static_plugin.so (HTTP Static
> Server)
> load_one_plugin:189: Loaded plugin: igmp_plugin.so (Internet Group
> Management Protocol (IGMP))
> load_one_plugin:189: Loaded plugin: ikev2_plugin.so (Internet Key Exchange
> (IKEv2) Protocol)
> load_one_plugin:189: Loaded plugin: ila_plugin.so (Identifier Locator
> Addressing (ILA) for IPv6)
> load_one_plugin:189: Loaded plugin: ioam_plugin.so (Inbound Operations,
> Administration, and Maintenance (OAM))
> load_one_plugin:117: Plugin disabled (default): ixge_plugin.so
> load_one_plugin:189: Loaded plugin: l2e_plugin.so (Layer 2 (L2) Emulation)
> load_one_plugin:189: Loaded plugin: l3xc_plugin.so (L3 Cross-Connect
> (L3XC))
> load_one_plugin:189: Loaded plugin: lacp_plugin.so (Link Aggregation
> Control Protocol (LACP))
> load_one_plugin:189: Loaded plugin: lb_plugin.so (Load Balancer (LB))
> load_one_plugin:189: Loaded plugin: mactime_plugin.so (Time-based MAC
> Source Address Filter)
> load_one_plugin:189: Loaded plugin: map_plugin.so (Mapping of Address and
> Port (MAP))
> load_one_plugin:189: Loaded plugin: memif_plugin.so (Packet Memory
> Interface (memif) -- Experimental)
> load_one_plugin:189: Loaded plugin: nat_plugin.so (Network Address
> Translation (NAT))
> load_one_plugin:189: Loaded plugin: nsh_plugin.so (Network Service Header
> (NSH))
> load_one_plugin:189: Loaded plugin: nsim_plugin.so (Network Delay
> Simulator)
> load_one_plugin:189: Loaded plugin: perfmon_plugin.so (Performance Monitor)
> load_one_plugin:189: Loaded plugin: pppoe_plugin.so (PPP over Ethernet
> (PPPoE))
> load_one_plugin:189: Loaded plugin: quic_plugin.so (Quic transport
> protocol)
> load_one_plugin:189: Loaded plugin: rdma_plugin.so (RDMA IBverbs Device
> Driver)
> load_one_plugin:117: Plugin disabled (default): sctp_plugin.so
> load_one_plugin:189: Loaded plugin: srv6ad_plugin.so (Dynamic Segment
> Routing for IPv6 (SRv6) Proxy)
> load_one_plugin:189: Loaded plugin: srv6am_plugin.so (Masquerading Segment
> Routing for IPv6 (SRv6) Proxy)
> load_one_plugin:189: Loaded plugin: srv6as_plugin.so (Static Segment
> Routing for IPv6 (SRv6) Proxy)
> load_one_plugin:189: Loaded plugin: stn_plugin.so (VPP Steals the NIC
> (STN) for Container Integration)
> load_one_plugin:189: Loaded plugin: svs_plugin.so (Source Virtual Routing
> and Fowarding (VRF) Select)
> load_one_plugin:189: Loaded plugin: tlsmbedtls_plugin.so (Transport Layer
> Security (TLS) Engine, Mbedtls Based)
> load_one_plugin:189: Loaded plugin: tlsopenssl_plugin.so (Transport Layer
> Security (TLS) Engine, OpenSSL Based)
> load_one_plugin:117: Plugin disabled (default): unittest_plugin.so
> load_one_plugin:189: Loaded plugin: vmxnet3_plugin.so (VMWare Vmxnet3
> Device Driver)
> /usr/bin/vpp[9185]: clib_elf_parse_file: open `linux-vdso.so.1': No such
> file or directory
> /usr/bin/vpp[9185]: vnet_feature_init:143: WARNING: arp arc: last node is
> error-drop, but expected arp-disabled!
> /usr/bin/vpp[9185]: perfmon: No table for cpuid 506f1
> /usr/bin/vpp[9185]: perfmon: model 5f, stepping 1
> /usr/bin/vpp[9185]: dpdk: Unsupported PCI device 0x8086:0x19e2 found at
> PCI address 0000:01:00.0
> /usr/bin/vpp[9185]: vlib_pci_bind_to_uio: Skipping PCI device 0000:05:00.0
> as host interface enp5s0 is up
> /usr/bin/vpp[9185]: dpdk: EAL init args: -c c -n 4 --in-memory
> --file-prefix vpp -b 0000:05:00.0 --master-lcore 2
> [New Thread 0x7fff90cff700 (LWP 9203)]
> [New Thread 0x7fff8bfff700 (LWP 9204)]
> loop0
> tap0
> vlib_worker_thread_barrier_sync_int: worker thread deadlock
>
> Thread 1 "vpp_main" received signal SIGABRT, Aborted.
> __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
> 51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
>
> -----------------------
>
> *Step 6: gdb backtrace*
>
> (gdb) bt
> #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
> #1 0x00007ffff61ca801 in __GI_abort () at abort.c:79
> #2 0x000055555555bdf3 in os_panic ()
> at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vpp/vnet/main.c:355
> #3 0x00007ffff6ab621b in vlib_worker_thread_barrier_sync_int (
> vm=vm@entry=0x7ffff6ce6a00 <vlib_global_main>,
> func_name=func_name@entry=0x7ffff6ad2ab0 <__FUNCTION__.43806>
> "vlib_cli_dispatch_sub_commands")
> at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/threads.c:1476
> #4 0x00007ffff6a6c937 in vlib_cli_dispatch_sub_commands (
> vm=vm@entry=0x7ffff6ce6a00 <vlib_global_main>, cm=cm@entry=0x7ffff6ce6c08
> <vlib_global_main+520>,
> input=input@entry=0x7fffb6a36f60, parent_command_index=<optimized
> out>)
> at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/cli.c:643
> #5 0x00007ffff6a6c718 in vlib_cli_dispatch_sub_commands (
> vm=vm@entry=0x7ffff6ce6a00 <vlib_global_main>, cm=cm@entry=0x7ffff6ce6c08
> <vlib_global_main+520>,
> input=input@entry=0x7fffb6a36f60, parent_command_index=<optimized
> out>)
> at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/cli.c:606
> #6 0x00007ffff6a6c718 in vlib_cli_dispatch_sub_commands (
> vm=vm@entry=0x7ffff6ce6a00 <vlib_global_main>, cm=cm@entry=0x7ffff6ce6c08
> <vlib_global_main+520>,
> input=input@entry=0x7fffb6a36f60,
> parent_command_index=parent_command_index@entry=0)
> at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/cli.c:606
> #7 0x00007ffff6a6cab8 in vlib_cli_input ()
> at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/cli.c:746
> #8 0x00007ffff6ac5be2 in unix_cli_process_input (cm=0x7ffff6ce74c0
> <unix_cli_main>, cli_file_index=0)
> at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/unix/cli.c:2525
> ---Type <return> to continue, or q <return> to quit---
> #9 unix_cli_process () at
> /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/unix/cli.c:2641
> #10 0x00007ffff6a881a6 in vlib_process_bootstrap (_a=<optimized out>)
> at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/main.c:1468
> #11 0x00007ffff65a3834 in clib_calljmp () from
> /usr/lib/x86_64-linux-gnu/libvppinfra.so.19.08
> #12 0x00007fffb5836b50 in ?? ()
> #13 0x00007ffff6a8bf59 in vlib_process_startup (f=0x0, p=0x7fffb6a26000,
> vm=0x7ffff6ce6a00 <vlib_global_main>)
> at /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vppinfra/types.h:133
> #14 dispatch_process (vm=0x7ffff6ce6a00 <vlib_global_main>,
> p=0x7fffb6a26000, last_time_stamp=0,
> f=0x0) at
> /w/workspace/vpp-beta-merge-1908-ubuntu1804/src/vlib/main.c:1535
> #15 0x0000000000000000 in ?? ()
>
>
> */var/log/syslog*
>
> Aug 21 17:39:37 test1 vpp: buffer: vlib_physmem_shared_map_create:
> pmalloc_map_pages: Unable to fulfill huge page allocation request: No such
> file or directory
> Aug 21 17:39:37 test1 vpp: buffer: falling back to non-hugepage backed
> buffer pool
> Aug 21 17:39:37 test1 vpp: perfmon: No table for cpuid 506f1
> Aug 21 17:39:37 test1 vpp: perfmon: model 5f, stepping 1
>
>
> *Info and configs used*
> ++++++++++++++++++++++
>
> Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-45-generic x86_64)
> -----------------------
> vpp# sh version verbose
> Version: v19.08-release
> Compiled by: root
> Compile host: 365637461ad3
> Compile date: Wed Aug 21 18:20:49 UTC 2019
> Compile location: /w/workspace/vpp-beta-merge-1908-ubuntu1804
> Compiler: GCC 8.3.0
> Current PID: 9617
> ------------------------
>
> *startup.conf*
>
> unix {
> nodaemon
> log /var/log/vpp/vpp.log
> full-coredump
> cli-listen /run/vpp/cli.sock
> gid vpp
> startup-config /home/test1/vpp.conf
> }
>
> api-trace {
> on
> }
>
> api-segment {
> gid vpp
> }
>
> socksvr {
> default
> }
>
> statseg {
> default
> }
>
> cpu {
> main-core 2
> corelist-workers 3
> }
>
> nat {
> translation hash buckets 16000
> max translations per user 500
> endpoint-dependent
> }
>
> -------------------------
>
> *vpp.conf*
>
> set int state TenGigabitEthernet8/0/0 up
> set int state GigabitEthernet2/0/0 up
> set int state GigabitEthernet3/0/0 up
>
> set int ip address TenGigabitEthernet8/0/0 72.77.156.155/29
> ip route add 0.0.0.0/0 via 72.77.156.158
>
> loopback create
> set int l2 bridge loop0 1 bvi
> set int ip address loop0 10.155.6.1/24
> set int state loop0 up
>
> set int l2 bridge GigabitEthernet2/0/0 1
> set int l2 bridge GigabitEthernet3/0/0 1
>
> comment { dhcp server listens on this interface }
> create tap id 0 host-ip4-addr 10.155.6.2/24 host-if-name novlan
> set int l2 bridge tap0 1
> set int state tap0 up
>
> nat44 add interface address TenGigabitEthernet8/0/0
> set interface nat44 in loop0 out TenGigabitEthernet8/0/0
>
> -------------------------
> *dnsmasq.conf*
>
> # Gateway
> dhcp-option=novlan,3,10.155.6.1
>
> # DNS server
> dhcp-option=novlan,6,10.155.6.1
>
> --------------------------
>
>
>
> Really appreciate your help!
>
>
>
>
>
> On Sat, Aug 17, 2019 at 1:37 PM Dave Barach via Lists.Fd.Io <dbarach=
> cisco....@lists.fd.io> wrote:
>
> I am trying to use DNS server and on "ping google.com" VPP is crashing
>
>
>
> Color me unsurprised. Please fix your configuration, to the point where
> first-order obvious errors have been eliminated. It looks to me like the
> “binary-api dns_name_server_add_del 8.8.8.8” command has been truncated:
>
>
>
>
>
> Aug 13 21:31:10 test1-vpp vnet[853]: unknown input `add_del 8.8.8.8
>
> Aug 13 21:31:28 test1-vpp vnet[853]: dns cache: add / del / clear
> required..
>
> Aug 13 21:31:36 test1-vpp vnet[853]:
>
> vl_api_dns_resolve_name_reply_t_handler:2556: ip4 address 23.75.7.244
>
> Aug 13 21:32:24 test1-vpp vnet[853]: dns cache: add / del / clear
> required..
>
> Aug 13 21:34:43 test1-vpp vnet[853]: resolve_event:247: name server
> 8.8.8.8 backfire
>
>
>
> Here is the config that I use on my home gateway.
>
>
>
> uncomment { nat44 add identity mapping external GigabitEthernet3/0/0 udp
> 53053 }
>
> uncomment { bin dns_name_server_add_del 8.8.8.8 }
>
> uncomment { bin dns_enable_disable }
>
>
>
> $ ping google.com # on my desktop, behind the vpp gateway
>
> PING google.com (172.217.7.174) 56(84) bytes of data.
>
> 64 bytes from 172.217.7.174 (172.217.7.174): icmp_seq=1 ttl=54 time=26.3 ms
>
> 64 bytes from 172.217.7.174 (172.217.7.174): icmp_seq=2 ttl=54 time=21.5 ms
>
>
>
> vpp# sh dns cache
>
> weatherlink.com -> weatherlink.com: 54.172.164.1 [21338] TTL left
> 489.9
>
> xxl.trane.com -> xxl.trane.com: 168.65.229.203 [781] EXPIRED
>
> google.com -> google.com: 172.217.7.174 [189] TTL left 552.1
>
> www.google.com -> www.google.com: 172.217.164.164 [189] TTL left
> 569.6
>
> dns.msftncsi.com -> dns.msftncsi.com: 131.107.255.255 [15] TTL left
> 569.9
>
> kv801-prod.do.dsp.mp.microsoft.com ->
> kv801-prod.do.dsp.mp.microsoft.com: 23.67.120.218 [19] TTL left 571.7
>
> geo-prod.do.dsp.mp.microsoft.com -> geo-prod.do.dsp.mp.microsoft.com:
> 40.69.221.239 [121] TTL left 571.1
>
> outlook.office365.com -> outlook.office365.com: 52.96.62.226 [38]
> TTL left 571.5
>
> 174.7.217.172.in-addr.arpa -> 174.7.217.172.in-addr.arpa: TTL left
> 573.6
>
> client.wns.windows.com -> client.wns.windows.com: 52.242.211.89
> [299] TTL left 574.1
>
> www.nr-extensions.tk -> www.nr-extensions.tk: 165.22.1.78 [190] TTL
> left 589.7
>
> ip-api.com -> ip-api.com: 147.135.15.186 [99] TTL left 589.9
>
> www.instagram.com -> www.instagram.com: 31.13.65.174 [59] TTL left
> 590.2
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
>
> View/Reply Online (#13779): https://lists.fd.io/g/vpp-dev/message/13779
> Mute This Topic: https://lists.fd.io/mt/32881233/675621
> Group Owner: vpp-dev+ow...@lists.fd.io
> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [carlitonu...@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#13814): https://lists.fd.io/g/vpp-dev/message/13814
Mute This Topic: https://lists.fd.io/mt/32881233/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
