Re: [vpp-dev] Crash when using dns_name_server
Please try master/latest: https://gerrit.fd.io/r/c/vpp/+/21468 fixed a source of trivial spin-lock bugs. HTH... Dave From: Dave Barach (dbarach) Sent: Thursday, August 22, 2019 12:11 PM To: Carlito Nueno Cc: vpp-dev@lists.fd.io Subject: RE: [vpp-dev] Crash when using dns_name_server NP, sorry for the issues, code simply not tested multi-core. BTW we just merged a refactor patch which converts the dns resolver into a plugin. Later this afternoon, I’ll do some multi-core testing. It may take a bit of work to repro and fix the problem you’ve reported. Dave From: Carlito Nueno mailto:carlitonu...@gmail.com>> Sent: Thursday, August 22, 2019 10:55 AM To: Dave Barach (dbarach) mailto:dbar...@cisco.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Crash when using dns_name_server Thanks Dave! Let me know if you need me do more tests or gather more info. On Thu, Aug 22, 2019 at 4:48 AM Dave Barach (dbarach) mailto:dbar...@cisco.com>> wrote: Ack. The DNS server has had zero multi-core testing, aside from what you’ve done. I’ll look at it when I can. From: Carlito Nueno mailto:carlitonu...@gmail.com>> Sent: Wednesday, August 21, 2019 10:03 PM To: Dave Barach (dbarach) mailto:dbar...@cisco.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Crash when using dns_name_server Hi Dave, Sorry about the late reply. I used below configs to eliminate most of the complexity. I did not see binary-api being truncated. Steps: 1. I used basic vpp.conf (see below) without the dns_name_server commands 2. gdb run -c /etc/vpp/startup.conf (see below) 3. sudo vppctl 4. Entered dns_name_server commands manually 5. ping google.com<http://google.com> 6. vpp crash Outputs collected: gbd run, gdb backtrace, syslog Step 4: DNS cache output vpp# nat44 add identity mapping external TenGigabitEthernet8/0/0 udp 53053 vpp# bin dns_name_server_add_del 8.8.8.8 vpp# bin dns_enable_disable vpp# sh dns cache verbose DNS cache contains 15 entries bserver-1.kbfs.keybaseapi.com<http://bserver-1.kbfs.keybaseapi.com> -> bserver-1.kbfs.keybaseapi.com<http://bserver-1.kbfs.keybaseapi.com>: 34.235.251.175 [59] 52.54.47.119 [59] TTL left 593.7 mdserver-0.kbfs.keybaseapi.com<http://mdserver-0.kbfs.keybaseapi.com> -> mdserver-0.kbfs.keybaseapi.com<http://mdserver-0.kbfs.keybaseapi.com>: 34.225.12.137 [45] 34.197.228.196 [45] TTL left 593.7 vortex.data.microsoft.com<http://vortex.data.microsoft.com> -> vortex.data.microsoft.com<http://vortex.data.microsoft.com>: 64.4.54.254 [263] TTL left 593.9 api.keybase.io<http://api.keybase.io> -> api.keybase.io<http://api.keybase.io>: 35.153.89.209 [34] 52.4.215.1 [34] TTL left 594.0 push.apple.com<http://push.apple.com> -> push.apple.com<http://push.apple.com>: TTL left 594.4 api.dropboxapi.com<http://api.dropboxapi.com> -> api.dropboxapi.com<http://api.dropboxapi.com>: 162.125.7.7 [59] TTL left 595.0 people-pa.clients6.google.com<http://people-pa.clients6.google.com> -> people-pa.clients6.google.com<http://people-pa.clients6.google.com>: 172.217.6.42 [240] TTL left 595.0 bolt.dropbox.com<http://bolt.dropbox.com> -> bolt.dropbox.com<http://bolt.dropbox.com>: 162.125.34.129 [59] TTL left 595.1 www.google.com<http://www.google.com> -> www.google.com<http://www.google.com>: 172.217.0.36 [263] TTL left 595.1 play.google.com<http://play.google.com> -> play.google.com<http://play.google.com>: 172.217.5.110 [27] TTL left 595.4 mail.google.com<http://mail.google.com> -> mail.google.com<http://mail.google.com>: 172.217.6.37 [299] TTL left 595.5 gateway-carry.icloud.com<http://gateway-carry.icloud.com> -> gateway-carry.icloud.com<http://gateway-carry.icloud.com>: 17.248.128.151 [59] 17.248.128.168 [59] 17.248.128.169 [59] 17.248.128.171 [59] 17.248.128.178 [59] 17.248.128.232 [59] 17.248.128.172 [59] 17.248.128.142 [59] TTL left 595.5 push.services.mozilla.com<http://push.services.mozilla.com> -> push.services.mozilla.com<http://push.services.mozilla.com>: 35.164.35.9 [56] TTL left 599.5 0.client-channel.google.com<http://0.client-channel.google.com> -> 0.client-channel.google.com<http://0.client-channel.google.com>: 74.125.28.189 [239] TTL left 599.6 airtable.com<http://airtable.com> -> airtable.com<http://airtable.com>: 3.221.153.172 [35] 34.193.210.213 [35] 52.22.150.146 [35] TTL left 599.6 Step 2: gdb run (gdb) run -c /etc/vpp/startup.conf Starting program: /usr/bin/vpp -c /etc/vpp/startup.conf [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". vlib_plugin_early_init:361: plugin path
Re: [vpp-dev] Crash when using dns_name_server
Got it. I'll look at the refactor patch and, also try to apply the patch a user posted on the old thread and test. Thanks! On Thu, Aug 22, 2019 at 9:11 AM Dave Barach (dbarach) wrote: > NP, sorry for the issues, code simply not tested multi-core. > > > > BTW we just merged a refactor patch which converts the dns resolver into a > plugin. Later this afternoon, I’ll do some multi-core testing. It may take > a bit of work to repro and fix the problem you’ve reported. > > > > Dave > > > > *From:* Carlito Nueno > *Sent:* Thursday, August 22, 2019 10:55 AM > *To:* Dave Barach (dbarach) > *Cc:* vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] Crash when using dns_name_server > > > > Thanks Dave! Let me know if you need me do more tests or gather more info. > > > > On Thu, Aug 22, 2019 at 4:48 AM Dave Barach (dbarach) > wrote: > > Ack. The DNS server has had *zero* multi-core testing, aside from what > you’ve done. I’ll look at it when I can. > > > > *From:* Carlito Nueno > *Sent:* Wednesday, August 21, 2019 10:03 PM > *To:* Dave Barach (dbarach) > *Cc:* vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] Crash when using dns_name_server > > > > Hi Dave, > > Sorry about the late reply. > > I used below configs to eliminate most of the complexity. I did not see > binary-api being truncated. > > Steps: > 1. I used basic vpp.conf (see below) without the dns_name_server commands > 2. gdb run -c /etc/vpp/startup.conf (see below) > 3. sudo vppctl > 4. Entered dns_name_server commands manually > 5. ping google.com > 6. vpp crash > > Outputs collected: gbd run, gdb backtrace, syslog > > *Step 4: DNS cache output* > > vpp# nat44 add identity mapping external TenGigabitEthernet8/0/0 udp 53053 > vpp# bin dns_name_server_add_del 8.8.8.8 > vpp# bin dns_enable_disable > vpp# sh dns cache verbose > DNS cache contains 15 entries > bserver-1.kbfs.keybaseapi.com -> bserver-1.kbfs.keybaseapi.com: > 34.235.251.175 [59] 52.54.47.119 [59] TTL left 593.7 > mdserver-0.kbfs.keybaseapi.com -> mdserver-0.kbfs.keybaseapi.com: > 34.225.12.137 [45] 34.197.228.196 [45] TTL left 593.7 > vortex.data.microsoft.com -> vortex.data.microsoft.com: 64.4.54.254 > [263] TTL left 593.9 > api.keybase.io -> api.keybase.io: 35.153.89.209 [34] 52.4.215.1 [34] > TTL left 594.0 > push.apple.com -> push.apple.com: TTL left 594.4 > api.dropboxapi.com -> api.dropboxapi.com: 162.125.7.7 [59] TTL left > 595.0 > people-pa.clients6.google.com -> people-pa.clients6.google.com: > 172.217.6.42 [240] TTL left 595.0 > bolt.dropbox.com -> bolt.dropbox.com: 162.125.34.129 [59] TTL left > 595.1 > www.google.com -> www.google.com: 172.217.0.36 [263] TTL left 595.1 > play.google.com -> play.google.com: 172.217.5.110 [27] TTL left > 595.4 > mail.google.com -> mail.google.com: 172.217.6.37 [299] TTL left > 595.5 > gateway-carry.icloud.com -> gateway-carry.icloud.com: 17.248.128.151 > [59] 17.248.128.168 [59] 17.248.128.169 [59] 17.248.128.171 [59] > 17.248.128.178 [59] 17.248.128.232 [59] 17.248.128.172 [59] 17.248.128.142 > [59] TTL left 595.5 > push.services.mozilla.com -> push.services.mozilla.com: 35.164.35.9 > [56] TTL left 599.5 > 0.client-channel.google.com -> 0.client-channel.google.com: > 74.125.28.189 [239] TTL left 599.6 > airtable.com -> airtable.com: 3.221.153.172 [35] 34.193.210.213 [35] > 52.22.150.146 [35] TTL left 599.6 > > > *Step 2: gdb run* > > (gdb) run -c /etc/vpp/startup.conf > Starting program: /usr/bin/vpp -c /etc/vpp/startup.conf > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > vlib_plugin_early_init:361: plugin path > /usr/lib/x86_64-linux-gnu/vpp_plugins:/usr/lib/vpp_plugins > load_one_plugin:189: Loaded plugin: abf_plugin.so (Access Control List > (ACL) Based Forwarding) > load_one_plugin:189: Loaded plugin: acl_plugin.so (Access Control Lists > (ACL)) > load_one_plugin:189: Loaded plugin: avf_plugin.so (Intel Adaptive Virtual > Function (AVF) Device Driver) > load_one_plugin:189: Loaded plugin: cdp_plugin.so (Cisco Discovery > Protocol (CDP)) > load_one_plugin:189: Loaded plugin: crypto_ia32_plugin.so (Intel IA32 > Software Crypto Engine) > load_one_plugin:189: Loaded plugin: crypto_ipsecmb_plugin.so (Intel IPSEC > Multi-buffer Crypto Engine) > load_one_plugin:189: Loaded plugin: crypto_openssl_plugin.so (OpenSSL > Crypto Engine) > load_one_plugin:189: Loaded plugin: ct6_plugin.so (IPv6 Connection Tracker) > load_one_plugin:189: Loaded plugin: dpdk_plugin.
Re: [vpp-dev] Crash when using dns_name_server
NP, sorry for the issues, code simply not tested multi-core. BTW we just merged a refactor patch which converts the dns resolver into a plugin. Later this afternoon, I’ll do some multi-core testing. It may take a bit of work to repro and fix the problem you’ve reported. Dave From: Carlito Nueno Sent: Thursday, August 22, 2019 10:55 AM To: Dave Barach (dbarach) Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Crash when using dns_name_server Thanks Dave! Let me know if you need me do more tests or gather more info. On Thu, Aug 22, 2019 at 4:48 AM Dave Barach (dbarach) mailto:dbar...@cisco.com>> wrote: Ack. The DNS server has had zero multi-core testing, aside from what you’ve done. I’ll look at it when I can. From: Carlito Nueno mailto:carlitonu...@gmail.com>> Sent: Wednesday, August 21, 2019 10:03 PM To: Dave Barach (dbarach) mailto:dbar...@cisco.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Crash when using dns_name_server Hi Dave, Sorry about the late reply. I used below configs to eliminate most of the complexity. I did not see binary-api being truncated. Steps: 1. I used basic vpp.conf (see below) without the dns_name_server commands 2. gdb run -c /etc/vpp/startup.conf (see below) 3. sudo vppctl 4. Entered dns_name_server commands manually 5. ping google.com<http://google.com> 6. vpp crash Outputs collected: gbd run, gdb backtrace, syslog Step 4: DNS cache output vpp# nat44 add identity mapping external TenGigabitEthernet8/0/0 udp 53053 vpp# bin dns_name_server_add_del 8.8.8.8 vpp# bin dns_enable_disable vpp# sh dns cache verbose DNS cache contains 15 entries bserver-1.kbfs.keybaseapi.com<http://bserver-1.kbfs.keybaseapi.com> -> bserver-1.kbfs.keybaseapi.com<http://bserver-1.kbfs.keybaseapi.com>: 34.235.251.175 [59] 52.54.47.119 [59] TTL left 593.7 mdserver-0.kbfs.keybaseapi.com<http://mdserver-0.kbfs.keybaseapi.com> -> mdserver-0.kbfs.keybaseapi.com<http://mdserver-0.kbfs.keybaseapi.com>: 34.225.12.137 [45] 34.197.228.196 [45] TTL left 593.7 vortex.data.microsoft.com<http://vortex.data.microsoft.com> -> vortex.data.microsoft.com<http://vortex.data.microsoft.com>: 64.4.54.254 [263] TTL left 593.9 api.keybase.io<http://api.keybase.io> -> api.keybase.io<http://api.keybase.io>: 35.153.89.209 [34] 52.4.215.1 [34] TTL left 594.0 push.apple.com<http://push.apple.com> -> push.apple.com<http://push.apple.com>: TTL left 594.4 api.dropboxapi.com<http://api.dropboxapi.com> -> api.dropboxapi.com<http://api.dropboxapi.com>: 162.125.7.7 [59] TTL left 595.0 people-pa.clients6.google.com<http://people-pa.clients6.google.com> -> people-pa.clients6.google.com<http://people-pa.clients6.google.com>: 172.217.6.42 [240] TTL left 595.0 bolt.dropbox.com<http://bolt.dropbox.com> -> bolt.dropbox.com<http://bolt.dropbox.com>: 162.125.34.129 [59] TTL left 595.1 www.google.com<http://www.google.com> -> www.google.com<http://www.google.com>: 172.217.0.36 [263] TTL left 595.1 play.google.com<http://play.google.com> -> play.google.com<http://play.google.com>: 172.217.5.110 [27] TTL left 595.4 mail.google.com<http://mail.google.com> -> mail.google.com<http://mail.google.com>: 172.217.6.37 [299] TTL left 595.5 gateway-carry.icloud.com<http://gateway-carry.icloud.com> -> gateway-carry.icloud.com<http://gateway-carry.icloud.com>: 17.248.128.151 [59] 17.248.128.168 [59] 17.248.128.169 [59] 17.248.128.171 [59] 17.248.128.178 [59] 17.248.128.232 [59] 17.248.128.172 [59] 17.248.128.142 [59] TTL left 595.5 push.services.mozilla.com<http://push.services.mozilla.com> -> push.services.mozilla.com<http://push.services.mozilla.com>: 35.164.35.9 [56] TTL left 599.5 0.client-channel.google.com<http://0.client-channel.google.com> -> 0.client-channel.google.com<http://0.client-channel.google.com>: 74.125.28.189 [239] TTL left 599.6 airtable.com<http://airtable.com> -> airtable.com<http://airtable.com>: 3.221.153.172 [35] 34.193.210.213 [35] 52.22.150.146 [35] TTL left 599.6 Step 2: gdb run (gdb) run -c /etc/vpp/startup.conf Starting program: /usr/bin/vpp -c /etc/vpp/startup.conf [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". vlib_plugin_early_init:361: plugin path /usr/lib/x86_64-linux-gnu/vpp_plugins:/usr/lib/vpp_plugins load_one_plugin:189: Loaded plugin: abf_plugin.so (Access Control List (ACL) Based Forwarding) load_one_plugin:189: Loaded plugin: acl_plugin.so (Access Control Lists (ACL)) load_one_plugin:189: Loaded plugin: avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device Driver) load_one_plugin:189: Loaded plugin: cdp_plugin.so (
Re: [vpp-dev] Crash when using dns_name_server
Thanks Dave! Let me know if you need me do more tests or gather more info. On Thu, Aug 22, 2019 at 4:48 AM Dave Barach (dbarach) wrote: > Ack. The DNS server has had *zero* multi-core testing, aside from what > you’ve done. I’ll look at it when I can. > > > > *From:* Carlito Nueno > *Sent:* Wednesday, August 21, 2019 10:03 PM > *To:* Dave Barach (dbarach) > *Cc:* vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] Crash when using dns_name_server > > > > Hi Dave, > > Sorry about the late reply. > > I used below configs to eliminate most of the complexity. I did not see > binary-api being truncated. > > Steps: > 1. I used basic vpp.conf (see below) without the dns_name_server commands > 2. gdb run -c /etc/vpp/startup.conf (see below) > 3. sudo vppctl > 4. Entered dns_name_server commands manually > 5. ping google.com > 6. vpp crash > > Outputs collected: gbd run, gdb backtrace, syslog > > *Step 4: DNS cache output* > > vpp# nat44 add identity mapping external TenGigabitEthernet8/0/0 udp 53053 > vpp# bin dns_name_server_add_del 8.8.8.8 > vpp# bin dns_enable_disable > vpp# sh dns cache verbose > DNS cache contains 15 entries > bserver-1.kbfs.keybaseapi.com -> bserver-1.kbfs.keybaseapi.com: > 34.235.251.175 [59] 52.54.47.119 [59] TTL left 593.7 > mdserver-0.kbfs.keybaseapi.com -> mdserver-0.kbfs.keybaseapi.com: > 34.225.12.137 [45] 34.197.228.196 [45] TTL left 593.7 > vortex.data.microsoft.com -> vortex.data.microsoft.com: 64.4.54.254 > [263] TTL left 593.9 > api.keybase.io -> api.keybase.io: 35.153.89.209 [34] 52.4.215.1 [34] > TTL left 594.0 > push.apple.com -> push.apple.com: TTL left 594.4 > api.dropboxapi.com -> api.dropboxapi.com: 162.125.7.7 [59] TTL left > 595.0 > people-pa.clients6.google.com -> people-pa.clients6.google.com: > 172.217.6.42 [240] TTL left 595.0 > bolt.dropbox.com -> bolt.dropbox.com: 162.125.34.129 [59] TTL left > 595.1 > www.google.com -> www.google.com: 172.217.0.36 [263] TTL left 595.1 > play.google.com -> play.google.com: 172.217.5.110 [27] TTL left > 595.4 > mail.google.com -> mail.google.com: 172.217.6.37 [299] TTL left > 595.5 > gateway-carry.icloud.com -> gateway-carry.icloud.com: 17.248.128.151 > [59] 17.248.128.168 [59] 17.248.128.169 [59] 17.248.128.171 [59] > 17.248.128.178 [59] 17.248.128.232 [59] 17.248.128.172 [59] 17.248.128.142 > [59] TTL left 595.5 > push.services.mozilla.com -> push.services.mozilla.com: 35.164.35.9 > [56] TTL left 599.5 > 0.client-channel.google.com -> 0.client-channel.google.com: > 74.125.28.189 [239] TTL left 599.6 > airtable.com -> airtable.com: 3.221.153.172 [35] 34.193.210.213 [35] > 52.22.150.146 [35] TTL left 599.6 > > > *Step 2: gdb run* > > (gdb) run -c /etc/vpp/startup.conf > Starting program: /usr/bin/vpp -c /etc/vpp/startup.conf > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > vlib_plugin_early_init:361: plugin path > /usr/lib/x86_64-linux-gnu/vpp_plugins:/usr/lib/vpp_plugins > load_one_plugin:189: Loaded plugin: abf_plugin.so (Access Control List > (ACL) Based Forwarding) > load_one_plugin:189: Loaded plugin: acl_plugin.so (Access Control Lists > (ACL)) > load_one_plugin:189: Loaded plugin: avf_plugin.so (Intel Adaptive Virtual > Function (AVF) Device Driver) > load_one_plugin:189: Loaded plugin: cdp_plugin.so (Cisco Discovery > Protocol (CDP)) > load_one_plugin:189: Loaded plugin: crypto_ia32_plugin.so (Intel IA32 > Software Crypto Engine) > load_one_plugin:189: Loaded plugin: crypto_ipsecmb_plugin.so (Intel IPSEC > Multi-buffer Crypto Engine) > load_one_plugin:189: Loaded plugin: crypto_openssl_plugin.so (OpenSSL > Crypto Engine) > load_one_plugin:189: Loaded plugin: ct6_plugin.so (IPv6 Connection Tracker) > load_one_plugin:189: Loaded plugin: dpdk_plugin.so (Data Plane Development > Kit (DPDK)) > load_one_plugin:189: Loaded plugin: flowprobe_plugin.so (Flow per Packet) > load_one_plugin:189: Loaded plugin: gbp_plugin.so (Group Based Policy > (GBP)) > load_one_plugin:189: Loaded plugin: gtpu_plugin.so (GPRS Tunnelling > Protocol, User Data (GTPv1-U)) > load_one_plugin:189: Loaded plugin: hs_apps_plugin.so (Host Stack > Applications) > load_one_plugin:189: Loaded plugin: http_static_plugin.so (HTTP Static > Server) > load_one_plugin:189: Loaded plugin: igmp_plugin.so (Internet Group > Management Protocol (IGMP)) > load_one_plugin:189: Loaded plugin: ikev2_plugin.so (Internet Key Exchange > (IKEv2) Protocol) > load_one_plugin:189: Loaded plugin: ila_plugin.so (Identifier Locator > Addressing (
Re: [vpp-dev] Crash when using dns_name_server
Ack. The DNS server has had zero multi-core testing, aside from what you’ve done. I’ll look at it when I can. From: Carlito Nueno Sent: Wednesday, August 21, 2019 10:03 PM To: Dave Barach (dbarach) Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Crash when using dns_name_server Hi Dave, Sorry about the late reply. I used below configs to eliminate most of the complexity. I did not see binary-api being truncated. Steps: 1. I used basic vpp.conf (see below) without the dns_name_server commands 2. gdb run -c /etc/vpp/startup.conf (see below) 3. sudo vppctl 4. Entered dns_name_server commands manually 5. ping google.com<http://google.com> 6. vpp crash Outputs collected: gbd run, gdb backtrace, syslog Step 4: DNS cache output vpp# nat44 add identity mapping external TenGigabitEthernet8/0/0 udp 53053 vpp# bin dns_name_server_add_del 8.8.8.8 vpp# bin dns_enable_disable vpp# sh dns cache verbose DNS cache contains 15 entries bserver-1.kbfs.keybaseapi.com<http://bserver-1.kbfs.keybaseapi.com> -> bserver-1.kbfs.keybaseapi.com<http://bserver-1.kbfs.keybaseapi.com>: 34.235.251.175 [59] 52.54.47.119 [59] TTL left 593.7 mdserver-0.kbfs.keybaseapi.com<http://mdserver-0.kbfs.keybaseapi.com> -> mdserver-0.kbfs.keybaseapi.com<http://mdserver-0.kbfs.keybaseapi.com>: 34.225.12.137 [45] 34.197.228.196 [45] TTL left 593.7 vortex.data.microsoft.com<http://vortex.data.microsoft.com> -> vortex.data.microsoft.com<http://vortex.data.microsoft.com>: 64.4.54.254 [263] TTL left 593.9 api.keybase.io<http://api.keybase.io> -> api.keybase.io<http://api.keybase.io>: 35.153.89.209 [34] 52.4.215.1 [34] TTL left 594.0 push.apple.com<http://push.apple.com> -> push.apple.com<http://push.apple.com>: TTL left 594.4 api.dropboxapi.com<http://api.dropboxapi.com> -> api.dropboxapi.com<http://api.dropboxapi.com>: 162.125.7.7 [59] TTL left 595.0 people-pa.clients6.google.com<http://people-pa.clients6.google.com> -> people-pa.clients6.google.com<http://people-pa.clients6.google.com>: 172.217.6.42 [240] TTL left 595.0 bolt.dropbox.com<http://bolt.dropbox.com> -> bolt.dropbox.com<http://bolt.dropbox.com>: 162.125.34.129 [59] TTL left 595.1 www.google.com<http://www.google.com> -> www.google.com<http://www.google.com>: 172.217.0.36 [263] TTL left 595.1 play.google.com<http://play.google.com> -> play.google.com<http://play.google.com>: 172.217.5.110 [27] TTL left 595.4 mail.google.com<http://mail.google.com> -> mail.google.com<http://mail.google.com>: 172.217.6.37 [299] TTL left 595.5 gateway-carry.icloud.com<http://gateway-carry.icloud.com> -> gateway-carry.icloud.com<http://gateway-carry.icloud.com>: 17.248.128.151 [59] 17.248.128.168 [59] 17.248.128.169 [59] 17.248.128.171 [59] 17.248.128.178 [59] 17.248.128.232 [59] 17.248.128.172 [59] 17.248.128.142 [59] TTL left 595.5 push.services.mozilla.com<http://push.services.mozilla.com> -> push.services.mozilla.com<http://push.services.mozilla.com>: 35.164.35.9 [56] TTL left 599.5 0.client-channel.google.com<http://0.client-channel.google.com> -> 0.client-channel.google.com<http://0.client-channel.google.com>: 74.125.28.189 [239] TTL left 599.6 airtable.com<http://airtable.com> -> airtable.com<http://airtable.com>: 3.221.153.172 [35] 34.193.210.213 [35] 52.22.150.146 [35] TTL left 599.6 Step 2: gdb run (gdb) run -c /etc/vpp/startup.conf Starting program: /usr/bin/vpp -c /etc/vpp/startup.conf [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". vlib_plugin_early_init:361: plugin path /usr/lib/x86_64-linux-gnu/vpp_plugins:/usr/lib/vpp_plugins load_one_plugin:189: Loaded plugin: abf_plugin.so (Access Control List (ACL) Based Forwarding) load_one_plugin:189: Loaded plugin: acl_plugin.so (Access Control Lists (ACL)) load_one_plugin:189: Loaded plugin: avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device Driver) load_one_plugin:189: Loaded plugin: cdp_plugin.so (Cisco Discovery Protocol (CDP)) load_one_plugin:189: Loaded plugin: crypto_ia32_plugin.so (Intel IA32 Software Crypto Engine) load_one_plugin:189: Loaded plugin: crypto_ipsecmb_plugin.so (Intel IPSEC Multi-buffer Crypto Engine) load_one_plugin:189: Loaded plugin: crypto_openssl_plugin.so (OpenSSL Crypto Engine) load_one_plugin:189: Loaded plugin: ct6_plugin.so (IPv6 Connection Tracker) load_one_plugin:189: Loaded plugin: dpdk_plugin.so (Data Plane Development Kit (DPDK)) load_one_plugin:189: Loaded plugin: flowprobe_plugin.so (Flow per Packet) load_one_plugin:189: Loaded plugin: gbp_plugin.so (Group Based Policy (GBP)) load_one_plugin:189: Loaded plugin: gtpu_plugin.so (GPRS Tunnelling Protocol, User Data (GTPv1
Re: [vpp-dev] Crash when using dns_name_server
Hi Dave, Sorry about the late reply. I used below configs to eliminate most of the complexity. I did not see binary-api being truncated. Steps: 1. I used basic vpp.conf (see below) without the dns_name_server commands 2. gdb run -c /etc/vpp/startup.conf (see below) 3. sudo vppctl 4. Entered dns_name_server commands manually 5. ping google.com 6. vpp crash Outputs collected: gbd run, gdb backtrace, syslog *Step 4: DNS cache output* vpp# nat44 add identity mapping external TenGigabitEthernet8/0/0 udp 53053 vpp# bin dns_name_server_add_del 8.8.8.8 vpp# bin dns_enable_disable vpp# sh dns cache verbose DNS cache contains 15 entries bserver-1.kbfs.keybaseapi.com -> bserver-1.kbfs.keybaseapi.com: 34.235.251.175 [59] 52.54.47.119 [59] TTL left 593.7 mdserver-0.kbfs.keybaseapi.com -> mdserver-0.kbfs.keybaseapi.com: 34.225.12.137 [45] 34.197.228.196 [45] TTL left 593.7 vortex.data.microsoft.com -> vortex.data.microsoft.com: 64.4.54.254 [263] TTL left 593.9 api.keybase.io -> api.keybase.io: 35.153.89.209 [34] 52.4.215.1 [34] TTL left 594.0 push.apple.com -> push.apple.com: TTL left 594.4 api.dropboxapi.com -> api.dropboxapi.com: 162.125.7.7 [59] TTL left 595.0 people-pa.clients6.google.com -> people-pa.clients6.google.com: 172.217.6.42 [240] TTL left 595.0 bolt.dropbox.com -> bolt.dropbox.com: 162.125.34.129 [59] TTL left 595.1 www.google.com -> www.google.com: 172.217.0.36 [263] TTL left 595.1 play.google.com -> play.google.com: 172.217.5.110 [27] TTL left 595.4 mail.google.com -> mail.google.com: 172.217.6.37 [299] TTL left 595.5 gateway-carry.icloud.com -> gateway-carry.icloud.com: 17.248.128.151 [59] 17.248.128.168 [59] 17.248.128.169 [59] 17.248.128.171 [59] 17.248.128.178 [59] 17.248.128.232 [59] 17.248.128.172 [59] 17.248.128.142 [59] TTL left 595.5 push.services.mozilla.com -> push.services.mozilla.com: 35.164.35.9 [56] TTL left 599.5 0.client-channel.google.com -> 0.client-channel.google.com: 74.125.28.189 [239] TTL left 599.6 airtable.com -> airtable.com: 3.221.153.172 [35] 34.193.210.213 [35] 52.22.150.146 [35] TTL left 599.6 *Step 2: gdb run* (gdb) run -c /etc/vpp/startup.conf Starting program: /usr/bin/vpp -c /etc/vpp/startup.conf [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". vlib_plugin_early_init:361: plugin path /usr/lib/x86_64-linux-gnu/vpp_plugins:/usr/lib/vpp_plugins load_one_plugin:189: Loaded plugin: abf_plugin.so (Access Control List (ACL) Based Forwarding) load_one_plugin:189: Loaded plugin: acl_plugin.so (Access Control Lists (ACL)) load_one_plugin:189: Loaded plugin: avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device Driver) load_one_plugin:189: Loaded plugin: cdp_plugin.so (Cisco Discovery Protocol (CDP)) load_one_plugin:189: Loaded plugin: crypto_ia32_plugin.so (Intel IA32 Software Crypto Engine) load_one_plugin:189: Loaded plugin: crypto_ipsecmb_plugin.so (Intel IPSEC Multi-buffer Crypto Engine) load_one_plugin:189: Loaded plugin: crypto_openssl_plugin.so (OpenSSL Crypto Engine) load_one_plugin:189: Loaded plugin: ct6_plugin.so (IPv6 Connection Tracker) load_one_plugin:189: Loaded plugin: dpdk_plugin.so (Data Plane Development Kit (DPDK)) load_one_plugin:189: Loaded plugin: flowprobe_plugin.so (Flow per Packet) load_one_plugin:189: Loaded plugin: gbp_plugin.so (Group Based Policy (GBP)) load_one_plugin:189: Loaded plugin: gtpu_plugin.so (GPRS Tunnelling Protocol, User Data (GTPv1-U)) load_one_plugin:189: Loaded plugin: hs_apps_plugin.so (Host Stack Applications) load_one_plugin:189: Loaded plugin: http_static_plugin.so (HTTP Static Server) load_one_plugin:189: Loaded plugin: igmp_plugin.so (Internet Group Management Protocol (IGMP)) load_one_plugin:189: Loaded plugin: ikev2_plugin.so (Internet Key Exchange (IKEv2) Protocol) load_one_plugin:189: Loaded plugin: ila_plugin.so (Identifier Locator Addressing (ILA) for IPv6) load_one_plugin:189: Loaded plugin: ioam_plugin.so (Inbound Operations, Administration, and Maintenance (OAM)) load_one_plugin:117: Plugin disabled (default): ixge_plugin.so load_one_plugin:189: Loaded plugin: l2e_plugin.so (Layer 2 (L2) Emulation) load_one_plugin:189: Loaded plugin: l3xc_plugin.so (L3 Cross-Connect (L3XC)) load_one_plugin:189: Loaded plugin: lacp_plugin.so (Link Aggregation Control Protocol (LACP)) load_one_plugin:189: Loaded plugin: lb_plugin.so (Load Balancer (LB)) load_one_plugin:189: Loaded plugin: mactime_plugin.so (Time-based MAC Source Address Filter) load_one_plugin:189: Loaded plugin: map_plugin.so (Mapping of Address and Port (MAP)) load_one_plugin:189: Loaded plugin: memif_plugin.so (Packet Memory Interface (memif) -- Experimental) load_one_plugin:189: Loaded plugin: nat_plugin.so (Network Address Translation (NAT)) load_one_plugin:189: Loaded plugin: nsh_plugin.so (Network Service Header (NSH)) load_one_plugin:189: Loaded plugin: nsim_plugin.so (Network Delay
Re: [vpp-dev] Crash when using dns_name_server
I am trying to use DNS server and on "ping google.com" VPP is crashing
Color me unsurprised. Please fix your configuration, to the point where
first-order obvious errors have been eliminated. It looks to me like the
"binary-api dns_name_server_add_del 8.8.8.8" command has been truncated:
Aug 13 21:31:10 test1-vpp vnet[853]: unknown input `add_del 8.8.8.8
Aug 13 21:31:28 test1-vpp vnet[853]: dns cache: add / del / clear required..
Aug 13 21:31:36 test1-vpp vnet[853]:
vl_api_dns_resolve_name_reply_t_handler:2556: ip4 address 23.75.7.244
Aug 13 21:32:24 test1-vpp vnet[853]: dns cache: add / del / clear required..
Aug 13 21:34:43 test1-vpp vnet[853]: resolve_event:247: name server 8.8.8.8
backfire
Here is the config that I use on my home gateway.
uncomment { nat44 add identity mapping external GigabitEthernet3/0/0 udp 53053 }
uncomment { bin dns_name_server_add_del 8.8.8.8 }
uncomment { bin dns_enable_disable }
$ ping google.com # on my desktop, behind the vpp gateway
PING google.com (172.217.7.174) 56(84) bytes of data.
64 bytes from 172.217.7.174 (172.217.7.174): icmp_seq=1 ttl=54 time=26.3 ms
64 bytes from 172.217.7.174 (172.217.7.174): icmp_seq=2 ttl=54 time=21.5 ms
vpp# sh dns cache
weatherlink.com -> weatherlink.com: 54.172.164.1 [21338] TTL left 489.9
xxl.trane.com -> xxl.trane.com: 168.65.229.203 [781] EXPIRED
google.com -> google.com: 172.217.7.174 [189] TTL left 552.1
www.google.com -> www.google.com: 172.217.164.164 [189] TTL left 569.6
dns.msftncsi.com -> dns.msftncsi.com: 131.107.255.255 [15] TTL left 569.9
kv801-prod.do.dsp.mp.microsoft.com -> kv801-prod.do.dsp.mp.microsoft.com:
23.67.120.218 [19] TTL left 571.7
geo-prod.do.dsp.mp.microsoft.com -> geo-prod.do.dsp.mp.microsoft.com:
40.69.221.239 [121] TTL left 571.1
outlook.office365.com -> outlook.office365.com: 52.96.62.226 [38] TTL
left 571.5
174.7.217.172.in-addr.arpa -> 174.7.217.172.in-addr.arpa: TTL left 573.6
client.wns.windows.com -> client.wns.windows.com: 52.242.211.89 [299] TTL
left 574.1
www.nr-extensions.tk -> www.nr-extensions.tk: 165.22.1.78 [190] TTL left
589.7
ip-api.com -> ip-api.com: 147.135.15.186 [99] TTL left 589.9
www.instagram.com -> www.instagram.com: 31.13.65.174 [59] TTL left 590.2
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#13779): https://lists.fd.io/g/vpp-dev/message/13779
Mute This Topic: https://lists.fd.io/mt/32881233/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Crash when using dns_name_server
Hi Carlito, The standard bug reporting practice is documented on the wiki [0]. Alternatively, run your debug version of VPP in gdb and report the backtrace when vpp crashes. Thanks, -daw- [0] https://wiki.fd.io/view/VPP/BugReports On 8/16/2019 11:12 PM, carlito nueno wrote: Hi Dave, Thanks for the patch. I merged your edits and compiled a debug version using stable/1908 as base. Every time a make a ping request from a LAN device, VPP is restarting. Sometimes vppctl just hangs, but when I do get into vppctl, if I run a command (ex: sh nat44 address), VPP again restarts. I know this information is not that helpful. Please let me know what information you need and, I can also run more tests. Thanks! On Thu, Aug 15, 2019 at 12:20 PM Dave Barach via Lists.Fd.Io wrote: See https://jira.fd.io/browse/VPP-1746, and https://gerrit.fd.io/r/c/vpp/+/21338 which fixes gross non-operation of the name resolver. Process created on demand, with node index in the main_t. Needed to remove the static vlib_node_registration_t and use dm->resolver_process_node_index vs. unused_mumble_registration.node_index. Passing 0 when signaling name resolution events couldn’t possibly work. D. From: vpp-dev@lists.fd.io On Behalf Of Dave Barach via Lists.Fd.Io Sent: Thursday, August 15, 2019 2:54 PM To: anoopnairh...@gmail.com; vpp-dev@lists.fd.io Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Crash when using dns_name_server Folks, I’ll look at these issues. It would be helpful if people would contribute patches, or at least write Jira tickets. If we don’t know it’s broken, it won’t get fixed... To level-set: the DNS name resolver has been lightly used. Nothing would surprise me at this point. D. From: vpp-dev@lists.fd.io On Behalf Of anoopnairh...@gmail.com Sent: Thursday, August 15, 2019 1:07 PM To: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Crash when using dns_name_server Hi Carlio, I had faced a similar crash with DNS module while resolving names. The dns_cache_lock is in locked state after initialization. Because of this the first worker thread which attempts to take this lock will find it in "locked" state and spin forever. So the main thread panics when it tries for barrier sync. Attached the patch which solved my problem I could find couple of other issues in the DNS module and the patch has the fix for them as well. - DNS lock is not released while processing dns request -> causes deadlock - resolve a name from VAT when there is no server configured -> crash due to a NULL pointer deference - delete_random_entry() is invoked while holding DNS lock -> a potential deadlock Please check if it helps you. thanks Anoop -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13754): https://lists.fd.io/g/vpp-dev/message/13754 Mute This Topic: https://lists.fd.io/mt/32881233/675621 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [carlitonu...@gmail.com] -=-=-=-=-=-=-=-=-=-=-=- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13771): https://lists.fd.io/g/vpp-dev/message/13771 Mute This Topic: https://lists.fd.io/mt/32881233/675079 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [dwallac...@gmail.com] -=-=-=-=-=-=-=-=-=-=-=- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13777): https://lists.fd.io/g/vpp-dev/message/13777 Mute This Topic: https://lists.fd.io/mt/32881233/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Crash when using dns_name_server
Hi Dave, Thanks for the patch. I merged your edits and compiled a debug version using stable/1908 as base. Every time a make a ping request from a LAN device, VPP is restarting. Sometimes vppctl just hangs, but when I do get into vppctl, if I run a command (ex: sh nat44 address), VPP again restarts. I know this information is not that helpful. Please let me know what information you need and, I can also run more tests. Thanks! On Thu, Aug 15, 2019 at 12:20 PM Dave Barach via Lists.Fd.Io wrote: > > See https://jira.fd.io/browse/VPP-1746, and > https://gerrit.fd.io/r/c/vpp/+/21338 which fixes gross non-operation of the > name resolver. > > > > Process created on demand, with node index in the main_t. Needed to remove > the static vlib_node_registration_t and use dm->resolver_process_node_index > vs. unused_mumble_registration.node_index. > > > > Passing 0 when signaling name resolution events couldn’t possibly work. > > > > D. > > > > From: vpp-dev@lists.fd.io On Behalf Of Dave Barach via > Lists.Fd.Io > Sent: Thursday, August 15, 2019 2:54 PM > To: anoopnairh...@gmail.com; vpp-dev@lists.fd.io > Cc: vpp-dev@lists.fd.io > Subject: Re: [vpp-dev] Crash when using dns_name_server > > > > Folks, > > > > I’ll look at these issues. It would be helpful if people would contribute > patches, or at least write Jira tickets. If we don’t know it’s broken, it > won’t get fixed... > > > > To level-set: the DNS name resolver has been lightly used. Nothing would > surprise me at this point. > > > > D. > > > > From: vpp-dev@lists.fd.io On Behalf Of > anoopnairh...@gmail.com > Sent: Thursday, August 15, 2019 1:07 PM > To: vpp-dev@lists.fd.io > Subject: Re: [vpp-dev] Crash when using dns_name_server > > > > Hi Carlio, > I had faced a similar crash with DNS module while resolving names. > > The dns_cache_lock is in locked state after initialization. Because of this > the first worker thread which attempts to take this lock will find it in > "locked" state and spin forever. So the main thread panics when it tries for > barrier sync. Attached the patch which solved my problem > > I could find couple of other issues in the DNS module and the patch has the > fix for them as well. > > - DNS lock is not released while processing dns request -> causes deadlock > > - resolve a name from VAT when there is no server configured -> crash > due to a NULL pointer deference > > - delete_random_entry() is invoked while holding DNS lock -> a potential > deadlock > > Please check if it helps you. > > thanks > Anoop > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#13754): https://lists.fd.io/g/vpp-dev/message/13754 > Mute This Topic: https://lists.fd.io/mt/32881233/675621 > Group Owner: vpp-dev+ow...@lists.fd.io > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [carlitonu...@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13771): https://lists.fd.io/g/vpp-dev/message/13771 Mute This Topic: https://lists.fd.io/mt/32881233/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Crash when using dns_name_server
See https://jira.fd.io/browse/VPP-1746, and https://gerrit.fd.io/r/c/vpp/+/21338 which fixes gross non-operation of the name resolver. Process created on demand, with node index in the main_t. Needed to remove the static vlib_node_registration_t and use dm->resolver_process_node_index vs. unused_mumble_registration.node_index. Passing 0 when signaling name resolution events couldn’t possibly work. D. From: vpp-dev@lists.fd.io On Behalf Of Dave Barach via Lists.Fd.Io Sent: Thursday, August 15, 2019 2:54 PM To: anoopnairh...@gmail.com; vpp-dev@lists.fd.io Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Crash when using dns_name_server Folks, I’ll look at these issues. It would be helpful if people would contribute patches, or at least write Jira tickets. If we don’t know it’s broken, it won’t get fixed... To level-set: the DNS name resolver has been lightly used. Nothing would surprise me at this point. D. From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of anoopnairh...@gmail.com<mailto:anoopnairh...@gmail.com> Sent: Thursday, August 15, 2019 1:07 PM To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Crash when using dns_name_server Hi Carlio, I had faced a similar crash with DNS module while resolving names. The dns_cache_lock is in locked state after initialization. Because of this the first worker thread which attempts to take this lock will find it in "locked" state and spin forever. So the main thread panics when it tries for barrier sync. Attached the patch which solved my problem I could find couple of other issues in the DNS module and the patch has the fix for them as well. - DNS lock is not released while processing dns request -> causes deadlock - resolve a name from VAT when there is no server configured -> crash due to a NULL pointer deference - delete_random_entry() is invoked while holding DNS lock -> a potential deadlock Please check if it helps you. thanks Anoop -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13754): https://lists.fd.io/g/vpp-dev/message/13754 Mute This Topic: https://lists.fd.io/mt/32881233/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Crash when using dns_name_server
Folks, I’ll look at these issues. It would be helpful if people would contribute patches, or at least write Jira tickets. If we don’t know it’s broken, it won’t get fixed... To level-set: the DNS name resolver has been lightly used. Nothing would surprise me at this point. D. From: vpp-dev@lists.fd.io On Behalf Of anoopnairh...@gmail.com Sent: Thursday, August 15, 2019 1:07 PM To: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Crash when using dns_name_server Hi Carlio, I had faced a similar crash with DNS module while resolving names. The dns_cache_lock is in locked state after initialization. Because of this the first worker thread which attempts to take this lock will find it in "locked" state and spin forever. So the main thread panics when it tries for barrier sync. Attached the patch which solved my problem I could find couple of other issues in the DNS module and the patch has the fix for them as well. - DNS lock is not released while processing dns request -> causes deadlock - resolve a name from VAT when there is no server configured -> crash due to a NULL pointer deference - delete_random_entry() is invoked while holding DNS lock -> a potential deadlock Please check if it helps you. thanks Anoop -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13753): https://lists.fd.io/g/vpp-dev/message/13753 Mute This Topic: https://lists.fd.io/mt/32881233/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Crash when using dns_name_server
Hi Carlio, I had faced a similar crash with DNS module while resolving names. The dns_cache_lock is in locked state after initialization. Because of this the first worker thread which attempts to take this lock will find it in "locked" state and spin forever. So the main thread panics when it tries for barrier sync. Attached the patch which solved my problem I could find couple of other issues in the DNS module and the patch has the fix for them as well. - DNS lock is not released while processing dns request -> causes deadlock - resolve a name from VAT when there is no server configured -> crash due to a NULL pointer deference - delete_random_entry() is invoked while holding DNS lock -> a potential deadlock Please check if it helps you. thanks Anoop vpp-19.01-dns-deadlock-fix.patch Description: Binary data -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13752): https://lists.fd.io/g/vpp-dev/message/13752 Mute This Topic: https://lists.fd.io/mt/32881233/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Crash when using dns_name_server
Hi Dave, Yep. When I made the packet trace, I had the dns config bits. VPP is caching DNS queries [P] DNS query: id 18 no-recur recur-des no-trunc non-auth 2 queries, 0 answers, 0 name-servers, 0 add'l recs Queries: Name: www.apple.com: type A Name: www.apple.com: type But LAN (inside network) device is not able to resolve any url LAN device is at 10.155.6.202 dig @10.155.6.1 www.apple.com ; <<>> DiG 9.10.6 <<>> @10.155.6.1 www.apple.com ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached Here is the config I was using: set int state wan0 up set int state lan0 up set int state lan1 up loopback create set int l2 bridge loop0 1 bvi set int ip address loop0 10.155.1.1/24 set int state loop0 up create sub lan0 1 set int state lan0.1 up set int l2 bridge lan0.1 1 set int l2 tag-rewrite lan0.1 pop 1 create sub lan1 1 set int state lan1.1 up set int l2 bridge lan1.1 1 set int l2 tag-rewrite lan1.1 pop 1 create tap id 0 host-ip4-addr 10.155.1.2/24 host-if-name mgmt set int l2 bridge tap0 1 set int state tap0 up loopback create set int l2 bridge loop1 2 bvi set int ip address loop1 10.155.2.1/24 set int state loop1 up create sub lan0 2 set int state lan0.2 up set int l2 bridge lan0.2 2 set int l2 tag-rewrite lan0.2 pop 1 create sub lan1 2 set int state lan1.2 up set int l2 bridge lan1.2 2 set int l2 tag-rewrite lan1.2 pop 1 create tap id 1 host-ip4-addr 10.155.2.2/24 host-if-name private set int l2 bridge tap1 2 set int state tap1 up loopback create set int l2 bridge loop2 3 bvi set int ip address loop2 10.155.6.1/24 set int state loop2 up set int l2 bridge lan0 3 set int l2 bridge lan1 3 create tap id 2 host-ip4-addr 10.155.6.2/24 host-if-name novlan set int l2 bridge tap2 3 set int state tap2 up nat44 add interface address wan0 set interface nat44 in loop0 in loop1 in loop2 set interface nat44 out wan0 nat44 add identity mapping external wan0 udp 53053 bin dns_name_server_add_del 8.8.8.8 bin dns_enable_disable DHCP server settings OPTION: 6 ( 4) DNS server10.155.6.1 OPTION: 3 ( 4) Routers 10.155.6.1 Thanks! On Thu, Aug 15, 2019 at 5:02 AM Dave Barach (dbarach) wrote: > > Four bits of config required: > > nat44 add identity mapping external GigabitEthernet3/0/0 udp 53053 > binary-api dns_name_server_add_del 8.8.8.8 > binary-api dns_enable_disable > > Inside network DHCP server needs to set option 6 (DNS name server) to the vpp > gateway address. > > D. > > -Original Message- > From: vpp-dev@lists.fd.io On Behalf Of carlito nueno > Sent: Wednesday, August 14, 2019 11:46 PM > To: Carlito Nueno > Cc: vpp-dev@lists.fd.io > Subject: Re: [vpp-dev] Crash when using dns_name_server > > Did a packet trace and I noticed two things: > > dns4-request: DNS pkts pending upstream name resolution > nat44-out2in: no translation > > > Packet 8 > > 00:28:11:659028: dpdk-input > lan1 rx queue 0 > buffer 0x8aeef: current data 0, length 89, buffer-pool 0, ref-count 1, > totlen-nifb 0, trace 0x5 > ext-hdr-valid > l4-cksum-computed l4-cksum-correct > PKT MBUF: port 2, nb_segs 1, pkt_len 89 > buf_len 2176, data_len 89, ol_flags 0x180, data_off 128, phys_addr > 0xe64bbc40 > packet_type 0x211 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 > rss 0x0 fdir.hi 0x0 fdir.lo 0x0 > Packet Offload Flags > PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid > PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid > Packet Types > RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet > RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension headers > RTE_PTYPE_L4_UDP (0x0200) UDP packet > IP4: a0:36:9f:3b:a2:b2 -> de:ad:00:00:00:05 > UDP: 10.155.6.203 -> 10.155.6.1 > tos 0x00, ttl 64, length 75, checksum 0x55ce > fragment id 0xc2d2, flags DONT_FRAGMENT > UDP: 33177 -> 53 > length 55, checksum 0x96d9 > 00:28:11:659031: ethernet-input > frame: flags 0x3, hw-if-index 3, sw-if-index 3 > IP4: a0:36:9f:3b:a2:b2 -> de:ad:00:00:00:05 > 00:28:11:659032: l2-input > l2-input: sw_if_index 3 dst de:ad:00:00:00:05 src a0:36:9f:3b:a2:b2 > 00:28:11:659033: l2-learn > l2-learn: sw_if_index 3 dst de:ad:00:00:00:05 src a0:36:9f:3b:a2:b2 > bd_index 6 > 00:28:11:659034: l2-fwd > l2-fwd: sw_if_index 3 dst de:ad:00:00:00:05 src a0:36:9f:3b:a2:b2 > bd_index 6 result [0x70025, 37] static age-not bvi > 00:28:11:659036: ip4-input > UDP: 10.155.6.203 -> 10.155.6.1 > tos 0x00, ttl 64, length 75, checksum 0x55ce > fragment id 0xc2d2, flags DONT_FRAGMENT > UDP: 33177 -> 53 > length 55, checksum 0x96d9 > 00:28:11:659037: nat44-in2out > NAT44_IN2OUT_FAST_PATH: sw_if_i
Re: [vpp-dev] Crash when using dns_name_server
Four bits of config required: nat44 add identity mapping external GigabitEthernet3/0/0 udp 53053 binary-api dns_name_server_add_del 8.8.8.8 binary-api dns_enable_disable Inside network DHCP server needs to set option 6 (DNS name server) to the vpp gateway address. D. -Original Message- From: vpp-dev@lists.fd.io On Behalf Of carlito nueno Sent: Wednesday, August 14, 2019 11:46 PM To: Carlito Nueno Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Crash when using dns_name_server Did a packet trace and I noticed two things: dns4-request: DNS pkts pending upstream name resolution nat44-out2in: no translation Packet 8 00:28:11:659028: dpdk-input lan1 rx queue 0 buffer 0x8aeef: current data 0, length 89, buffer-pool 0, ref-count 1, totlen-nifb 0, trace 0x5 ext-hdr-valid l4-cksum-computed l4-cksum-correct PKT MBUF: port 2, nb_segs 1, pkt_len 89 buf_len 2176, data_len 89, ol_flags 0x180, data_off 128, phys_addr 0xe64bbc40 packet_type 0x211 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 rss 0x0 fdir.hi 0x0 fdir.lo 0x0 Packet Offload Flags PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid Packet Types RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension headers RTE_PTYPE_L4_UDP (0x0200) UDP packet IP4: a0:36:9f:3b:a2:b2 -> de:ad:00:00:00:05 UDP: 10.155.6.203 -> 10.155.6.1 tos 0x00, ttl 64, length 75, checksum 0x55ce fragment id 0xc2d2, flags DONT_FRAGMENT UDP: 33177 -> 53 length 55, checksum 0x96d9 00:28:11:659031: ethernet-input frame: flags 0x3, hw-if-index 3, sw-if-index 3 IP4: a0:36:9f:3b:a2:b2 -> de:ad:00:00:00:05 00:28:11:659032: l2-input l2-input: sw_if_index 3 dst de:ad:00:00:00:05 src a0:36:9f:3b:a2:b2 00:28:11:659033: l2-learn l2-learn: sw_if_index 3 dst de:ad:00:00:00:05 src a0:36:9f:3b:a2:b2 bd_index 6 00:28:11:659034: l2-fwd l2-fwd: sw_if_index 3 dst de:ad:00:00:00:05 src a0:36:9f:3b:a2:b2 bd_index 6 result [0x70025, 37] static age-not bvi 00:28:11:659036: ip4-input UDP: 10.155.6.203 -> 10.155.6.1 tos 0x00, ttl 64, length 75, checksum 0x55ce fragment id 0xc2d2, flags DONT_FRAGMENT UDP: 33177 -> 53 length 55, checksum 0x96d9 00:28:11:659037: nat44-in2out NAT44_IN2OUT_FAST_PATH: sw_if_index 37, next index 3, session -1 00:28:11:659037: nat44-in2out-slowpath NAT44_IN2OUT_SLOW_PATH: sw_if_index 37, next index 0, session -1 00:28:11:659038: ip4-lookup fib 0 dpo-idx 10 flow hash: 0x UDP: 10.155.6.203 -> 10.155.6.1 tos 0x00, ttl 64, length 75, checksum 0x55ce fragment id 0xc2d2, flags DONT_FRAGMENT UDP: 33177 -> 53 length 55, checksum 0x96d9 00:28:11:659040: ip4-local UDP: 10.155.6.203 -> 10.155.6.1 tos 0x00, ttl 64, length 75, checksum 0x55ce fragment id 0xc2d2, flags DONT_FRAGMENT UDP: 33177 -> 53 length 55, checksum 0x96d9 00:28:11:659041: ip4-local-end-of-arc UDP: 10.155.6.203 -> 10.155.6.1 tos 0x00, ttl 64, length 75, checksum 0x55ce fragment id 0xc2d2, flags DONT_FRAGMENT UDP: 33177 -> 53 length 55, checksum 0x96d9 00:28:11:659041: ip4-udp-lookup UDP: src-port 33177 dst-port 53 00:28:11:659042: dns4-request DNS46_REPLY: pool index -1, disposition 6 00:28:11:659044: error-drop rx:loop5 00:28:11:659044: drop dns4-request: DNS pkts pending upstream name resolution Packet 9 00:28:13:589187: dpdk-input wan0 rx queue 0 buffer 0x504bc: current data 0, length 113, buffer-pool 0, ref-count 1, totlen-nifb 0, trace 0x8 ext-hdr-valid l4-cksum-computed l4-cksum-correct PKT MBUF: port 5, nb_segs 1, pkt_len 113 buf_len 2176, data_len 113, ol_flags 0x180, data_off 128, phys_addr 0xe5a12f80 packet_type 0x211 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 rss 0x0 fdir.hi 0x0 fdir.lo 0x0 Packet Offload Flags PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid Packet Types RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension headers RTE_PTYPE_L4_UDP (0x0200) UDP packet IP4: 4a:1d:70:63:fc:d4 -> 06:35:31:eb:33:22 UDP: 8.8.8.8 -> 72.33.156.100 tos 0x20, ttl 122, length 99, checksum 0x94c7 fragment id 0x9ea9 UDP: 53 -> 53053 length 79, checksum 0x9c5e 00:28:13:589189: ethernet-input frame: flags 0x3, hw-if-index 6, sw-if-index 6 IP4: 4a:1d:70:63:fc:d4 -> 06:35:31:eb:33:22 00:28:13:589190: ip4-input-no-checksum UDP: 8.8.8.8 -> 72.33.156.100 tos 0x20, ttl 122, length 99, checksum 0x94c7 fragment id 0x9ea9 UDP: 53 -> 53053 length 79, checksum 0x9c5e 00:28:13:589191: nat44-out2in NAT44_OUT2IN: sw_if_index 6, next index 0, session index -1 00:28:13:589192: err
Re: [vpp-dev] Crash when using dns_name_server
Did a packet trace and I noticed two things: dns4-request: DNS pkts pending upstream name resolution nat44-out2in: no translation Packet 8 00:28:11:659028: dpdk-input lan1 rx queue 0 buffer 0x8aeef: current data 0, length 89, buffer-pool 0, ref-count 1, totlen-nifb 0, trace 0x5 ext-hdr-valid l4-cksum-computed l4-cksum-correct PKT MBUF: port 2, nb_segs 1, pkt_len 89 buf_len 2176, data_len 89, ol_flags 0x180, data_off 128, phys_addr 0xe64bbc40 packet_type 0x211 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 rss 0x0 fdir.hi 0x0 fdir.lo 0x0 Packet Offload Flags PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid Packet Types RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension headers RTE_PTYPE_L4_UDP (0x0200) UDP packet IP4: a0:36:9f:3b:a2:b2 -> de:ad:00:00:00:05 UDP: 10.155.6.203 -> 10.155.6.1 tos 0x00, ttl 64, length 75, checksum 0x55ce fragment id 0xc2d2, flags DONT_FRAGMENT UDP: 33177 -> 53 length 55, checksum 0x96d9 00:28:11:659031: ethernet-input frame: flags 0x3, hw-if-index 3, sw-if-index 3 IP4: a0:36:9f:3b:a2:b2 -> de:ad:00:00:00:05 00:28:11:659032: l2-input l2-input: sw_if_index 3 dst de:ad:00:00:00:05 src a0:36:9f:3b:a2:b2 00:28:11:659033: l2-learn l2-learn: sw_if_index 3 dst de:ad:00:00:00:05 src a0:36:9f:3b:a2:b2 bd_index 6 00:28:11:659034: l2-fwd l2-fwd: sw_if_index 3 dst de:ad:00:00:00:05 src a0:36:9f:3b:a2:b2 bd_index 6 result [0x70025, 37] static age-not bvi 00:28:11:659036: ip4-input UDP: 10.155.6.203 -> 10.155.6.1 tos 0x00, ttl 64, length 75, checksum 0x55ce fragment id 0xc2d2, flags DONT_FRAGMENT UDP: 33177 -> 53 length 55, checksum 0x96d9 00:28:11:659037: nat44-in2out NAT44_IN2OUT_FAST_PATH: sw_if_index 37, next index 3, session -1 00:28:11:659037: nat44-in2out-slowpath NAT44_IN2OUT_SLOW_PATH: sw_if_index 37, next index 0, session -1 00:28:11:659038: ip4-lookup fib 0 dpo-idx 10 flow hash: 0x UDP: 10.155.6.203 -> 10.155.6.1 tos 0x00, ttl 64, length 75, checksum 0x55ce fragment id 0xc2d2, flags DONT_FRAGMENT UDP: 33177 -> 53 length 55, checksum 0x96d9 00:28:11:659040: ip4-local UDP: 10.155.6.203 -> 10.155.6.1 tos 0x00, ttl 64, length 75, checksum 0x55ce fragment id 0xc2d2, flags DONT_FRAGMENT UDP: 33177 -> 53 length 55, checksum 0x96d9 00:28:11:659041: ip4-local-end-of-arc UDP: 10.155.6.203 -> 10.155.6.1 tos 0x00, ttl 64, length 75, checksum 0x55ce fragment id 0xc2d2, flags DONT_FRAGMENT UDP: 33177 -> 53 length 55, checksum 0x96d9 00:28:11:659041: ip4-udp-lookup UDP: src-port 33177 dst-port 53 00:28:11:659042: dns4-request DNS46_REPLY: pool index -1, disposition 6 00:28:11:659044: error-drop rx:loop5 00:28:11:659044: drop dns4-request: DNS pkts pending upstream name resolution Packet 9 00:28:13:589187: dpdk-input wan0 rx queue 0 buffer 0x504bc: current data 0, length 113, buffer-pool 0, ref-count 1, totlen-nifb 0, trace 0x8 ext-hdr-valid l4-cksum-computed l4-cksum-correct PKT MBUF: port 5, nb_segs 1, pkt_len 113 buf_len 2176, data_len 113, ol_flags 0x180, data_off 128, phys_addr 0xe5a12f80 packet_type 0x211 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 rss 0x0 fdir.hi 0x0 fdir.lo 0x0 Packet Offload Flags PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid Packet Types RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension headers RTE_PTYPE_L4_UDP (0x0200) UDP packet IP4: 4a:1d:70:63:fc:d4 -> 06:35:31:eb:33:22 UDP: 8.8.8.8 -> 72.33.156.100 tos 0x20, ttl 122, length 99, checksum 0x94c7 fragment id 0x9ea9 UDP: 53 -> 53053 length 79, checksum 0x9c5e 00:28:13:589189: ethernet-input frame: flags 0x3, hw-if-index 6, sw-if-index 6 IP4: 4a:1d:70:63:fc:d4 -> 06:35:31:eb:33:22 00:28:13:589190: ip4-input-no-checksum UDP: 8.8.8.8 -> 72.33.156.100 tos 0x20, ttl 122, length 99, checksum 0x94c7 fragment id 0x9ea9 UDP: 53 -> 53053 length 79, checksum 0x9c5e 00:28:13:589191: nat44-out2in NAT44_OUT2IN: sw_if_index 6, next index 0, session index -1 00:28:13:589192: error-drop rx:wan0 00:28:13:589192: drop nat44-out2in: no translation Packet 10 00:28:13:590291: dpdk-input wan0 rx queue 0 buffer 0x416b6: current data 0, length 236, buffer-pool 0, ref-count 1, totlen-nifb 0, trace 0x9 ext-hdr-valid l4-cksum-computed l4-cksum-correct PKT MBUF: port 5, nb_segs 1, pkt_len 236 buf_len 2176, data_len 236, ol_flags 0x180, data_off 128, phys_addr 0xe565ae00 packet_type 0x211 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 rss 0x0 fdir.hi 0x0 fdir.lo 0x0 Packet Offload
Re: [vpp-dev] Crash when using dns_name_server
VPP is not crashing anymore. I didn't change anything. VPP is caching DNS queries [P] DNS query: id 18 no-recur recur-des no-trunc non-auth 2 queries, 0 answers, 0 name-servers, 0 add'l recs Queries: Name: www.apple.com: type A Name: www.apple.com: type But LAN device is not able to resolve any url LAN device is at 10.155.6.202 dig @10.155.6.1 www.apple.com ; <<>> DiG 9.10.6 <<>> @10.155.6.1 www.apple.com ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached On Wed, Aug 14, 2019 at 4:41 PM carlito nueno via Lists.Fd.Io wrote: > > Hi all, > > I am trying to use DNS server and on "ping google.com" VPP is crashing > > Aug 13 21:31:10 test1-vpp vnet[853]: unknown input `add_del 8.8.8.8 > Aug 13 21:31:28 test1-vpp vnet[853]: dns cache: add / del / clear required.. > Aug 13 21:31:36 test1-vpp vnet[853]: > vl_api_dns_resolve_name_reply_t_handler:2556: ip4 address 23.75.7.244 > Aug 13 21:32:24 test1-vpp vnet[853]: dns cache: add / del / clear required.. > Aug 13 21:34:43 test1-vpp vnet[853]: resolve_event:247: name server > 8.8.8.8 backfire > > When I try to restart it, it just hangs > > Aug 13 21:35:16 test1-vpp vnet[853]: unix_signal_handler:170: received > signal SIGCONT, PC 0x7f9bf5ff7e20 > Aug 13 21:35:16 test1-vpp vnet[853]: received SIGTERM, exiting... > Aug 13 21:35:16 test1-vpp systemd[1]: Stopping vector packet > processing engine... > Aug 13 21:35:16 test1-vpp vnet[853]: unix_signal_handler:170: received > signal SIGCONT, PC 0x7f9bf5ff7e20 > > vpp.conf > > set int state wan0 up > set dhcp client intfc wan0 hostname vpp > > loopback create > set int l2 bridge loop5 6 bvi > set int ip address loop5 10.155.6.1/24 > set int state loop5 up > > set int l2 bridge lan0 6 > set int state lan0 up > > create tap id 5 host-ip4-addr 10.155.6.2/24 host-if-name lstack > host-ip4-gw 10.155.6.1 > set int l2 bridge tap5 6 > set int state tap5 up > > nat44 add interface address wan0 > set interface nat44 in loop5 in out wan0 > > nat44 add identity mapping external wan0 udp 53053 > bin dns_name_server_add_del 8.8.8.8 > bin dns_name_server_add_del 8.8.8.4 > bin dns_enable_disable > > DHCP server settings > OPTION: 6 ( 4) DNS server 10.155.6.1 > OPTION: 3 ( 4) Routers 10.155.6.1 > > Thanks! > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#13739): https://lists.fd.io/g/vpp-dev/message/13739 > Mute This Topic: https://lists.fd.io/mt/32881233/675621 > Group Owner: vpp-dev+ow...@lists.fd.io > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [carlitonu...@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13740): https://lists.fd.io/g/vpp-dev/message/13740 Mute This Topic: https://lists.fd.io/mt/32881233/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
