Re: Troubleshooting slow DNS lookup
In message aanlktims2mfbib5lpdpqyarc8ds1gg4db7b2tea=b...@mail.gmail.com, Rian to Wahyudi writes: Hi Mark, Thanks for your quick response ! Standards Track. RFC 2671 Extension Mechanisms for DNS (EDNS0) RFC 3226 DNSSEC and IPv6 A6 aware server/resolver message size requiremen= ts Unfortunately RFC is not considered as good enough ... unless if we can find an actual proof that can be replicated :( I also done some dnssec trace demonstration, and it still not a good enough reason : ie : dig www.anyhostname.com +trace +dnssec . This test always fail and it produce FWSM log entry similar to: : %FWSM-2-106007: Deny inbound UDP from 198.142.0.51/53 to 10.0.0.1/64788 due to DNS Response I also suggest that you ask your firewall people to talk to the CISCO TAC about how to properly configure the firewall for a nameserver that supports EDNS. The defaults are not setup for a nameserver that supports EDNS. If they don't want to do that read what CISCO recommends here: https://supportforums.cisco.com/message/3221565#3221565 Informational. RFC 4294 IPv6 Node Requirements http://labs.ripe.net/Members/anandb/content-testing-your-resolver-dns-rep= ly-size-issues How about the root servers? - Any example of dns record that send packet larger than 512 ? The root servers. =A0 =A0 =A0 =A0dig +dnssec dnskey . This for some reason works without any problem : Well if you ask the root servers dig +dnssec dnskey . @a.root-servers.net With just dig +dnssec dnskey . you are talking to your own server so are not going through the firewall. You will also notice it took 1/2 a second to get that answer so named did several different attempts in that 1/2 second. ;; Query time: 547 msec -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Troubleshooting slow DNS lookup
Our network team are quite reluctant to make any changes on the FWSM in regards to DNS inspection. So it seems that we are stuck with maximum UDP packet of 512 byte. Unfortunately, I do not have much evidence (ie user complains) to escalate this issue much further except from few number of users who *intermittently* unable to access www.paypal.com. The term intermittently is the main keyword, and because of that the finger are now point back the the DNS server. I believe that Increasing the maximum limit or disable inspection will fix the issue , but I will need to gather sufficient case and compelling report. - Does any one have a good example of prominent website that have DNSEC setup properly other than paypal? - Any example of dns record that send packet larger than 512 ? - Any other information I can use to help create the report ? As a work around I can possibly set EDNS UDP size to match the firewall limit, but I think this is my last option. Any help is greatly appreciated! Regards, Rianto Wahyudi ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Troubleshooting slow DNS lookup
In message aanlkti=t5tj29_gmngbtpug8cfyrqpgadr=-yvfwj...@mail.gmail.com, Rian to Wahyudi writes: Our network team are quite reluctant to make any changes on the FWSM in regards to DNS inspection. So it seems that we are stuck with maximum UDP packet of 512 byte. Unfortunately, I do not have much evidence (ie user complains) to escalate this issue much further except from few number of users who *intermittently* unable to access www.paypal.com. The term intermittently is the main keyword, and because of that the finger are now point back the the DNS server. It's intermittent because it takes named time to workout what will work with your firewall and the clients timeout in the meantime. This will only get worse over time. I believe that Increasing the maximum limit or disable inspection will fix the issue , but I will need to gather sufficient case and compelling report. Standards Track. RFC 2671 Extension Mechanisms for DNS (EDNS0) RFC 3226 DNSSEC and IPv6 A6 aware server/resolver message size requirements Informational. RFC 4294 IPv6 Node Requirements http://labs.ripe.net/Members/anandb/content-testing-your-resolver-dns-reply-size-issues - Does any one have a good example of prominent website that have DNSEC setup properly other than paypal? How about the root servers? - Any example of dns record that send packet larger than 512 ? The root servers. dig +dnssec dnskey . - Any other information I can use to help create the report ? As a work around I can possibly set EDNS UDP size to match the firewall limit, but I think this is my last option. Any help is greatly appreciated! Regards, Rianto Wahyudi ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Troubleshooting slow DNS lookup
Hi Mark, Thanks for your quick response ! Standards Track. RFC 2671 Extension Mechanisms for DNS (EDNS0) RFC 3226 DNSSEC and IPv6 A6 aware server/resolver message size requirements Unfortunately RFC is not considered as good enough ... unless if we can find an actual proof that can be replicated :( I also done some dnssec trace demonstration, and it still not a good enough reason : ie : dig www.anyhostname.com +trace +dnssec . This test always fail and it produce FWSM log entry similar to: : %FWSM-2-106007: Deny inbound UDP from 198.142.0.51/53 to 10.0.0.1/64788 due to DNS Response Informational. RFC 4294 IPv6 Node Requirements http://labs.ripe.net/Members/anandb/content-testing-your-resolver-dns-reply-size-issues How about the root servers? - Any example of dns record that send packet larger than 512 ? The root servers. dig +dnssec dnskey . This for some reason works without any problem : ; DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 +dnssec dnskey . ;; global options: printcmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 64905 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 13, ADDITIONAL: 14 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 512 ;; QUESTION SECTION: ;. IN DNSKEY ;; ANSWER SECTION: . 86400 IN DNSKEY 256 3 8 AwEAAcAPhPM4CQHqg6hZ49y2P3IdKZuF44QNCc50vjATD7W+je4va6dj Y5JpnNP0pIohKNYiCFap/b4Y9jjJGSOkOfkfBR8neI7X5LisMEGUjwRc rG8J9UYP1S1unTNqRcWyDYFH2q3KnIO08zImh5DiFt8yfCdKoqZUN1du p5hy0UWz . 86400 IN DNSKEY 257 3 8 AwEAAagAIKlVZrpC6Ia7gEzahOR+9W29euxhJhVVLOyQbSEW0O8gcCjF FVQUTf6v58fLjwBd0YI0EzrAcQqBGCzh/RStIoO8g0NfnfL2MTJRkxoX bfDaUeVPQuYEhg37NZWAJQ9VnMVDxP/VHL496M/QZxkjf5/Efucp2gaD X6RS6CXpoY68LsvPVjR0ZSwzz1apAzvN9dlzEheX7ICJBBtuA6G3LQpz W5hOA2hzCTMjJPJ8LbqF6dsV6DoBQzgul0sGIcGOYl7OyQdXfZ57relS Qageu+ipAdTTJ25AsRTAoub8ONGcLmqrAmRLKBP1dfwhYB4N7knNnulq QxA+Uk1ihz0= ;; AUTHORITY SECTION: . 518400 IN NS c.root-servers.net. . 518400 IN NS d.root-servers.net. . 518400 IN NS e.root-servers.net. . 518400 IN NS f.root-servers.net. . 518400 IN NS g.root-servers.net. . 518400 IN NS h.root-servers.net. . 518400 IN NS i.root-servers.net. . 518400 IN NS j.root-servers.net. . 518400 IN NS k.root-servers.net. . 518400 IN NS l.root-servers.net. . 518400 IN NS m.root-servers.net. . 518400 IN NS a.root-servers.net. . 518400 IN NS b.root-servers.net. ;; ADDITIONAL SECTION: a.root-servers.net. 2592000 IN A 198.41.0.4 b.root-servers.net. 2592000 IN A 192.228.79.201 c.root-servers.net. 2592000 IN A 192.33.4.12 d.root-servers.net. 2592000 IN A 128.8.10.90 e.root-servers.net. 2592000 IN A 192.203.230.10 f.root-servers.net. 2592000 IN A 192.5.5.241 g.root-servers.net. 2592000 IN A 192.112.36.4 h.root-servers.net. 2592000 IN A 128.63.2.53 i.root-servers.net. 2592000 IN A 192.36.148.17 k.root-servers.net. 2592000 IN A 193.0.14.129 a.root-servers.net. 2592000 IN 2001:503:ba3e::2:30 f.root-servers.net. 2592000 IN 2001:500:2f::f h.root-servers.net. 2592000 IN 2001:500:1::803f:235 ;; Query time: 547 msec ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Troubleshooting slow DNS lookup
On 26/11/10 5:58 AM, Mark Andrews ma...@isc.org wrote: In message aanlktimzmc4pgne7n72hnb7gnjuat9r2oktigaazv...@mail.gmail.com, Rian to Wahyudi writes: Hi Mark, Thanks for the pointers , your are spot on! Doing dig +trace +dnssec www.paypal.com always fail. After some investigation with the network guys, it appear that our upstream firewall are dropping DNS UDP packet larger than 512. Cisco FWSM have this configuration enabled by default : http://www.cisco.com/en/US/docs/security/fwsm/fwsm31/command/reference/i2.htm l#wp1565355 So the default is inspect dns maximum-length 512 if I read that page correctly. inspect dns or as a minimum inspect dns maximum-length 4096 will allow reply traffic through for named. I thought I had heard that Cisco had code which looked for the EDNS UDP size option and adjusted the maximum length based on that on a per transaction basis and enforced 512 if there wasn't a EDNS option. Yes, but I think its a recent addition to their code. The Cisco ASA supports: message-length maximum client auto This will use the OPT value as the maximum. I know its supported on version 8.3 of ASA software. It might not be supported by the switch modules of the OP. Mark -- Kal Feher ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Troubleshooting slow DNS lookup
Hi all, Im trying to troubleshoot and find out the reason why some of our DNS lookup take a long time : ns-dev ~ # rndc flushname www.paypal.com ; dig www.paypal.com @localhost ; DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 www.paypal.com @localhost ;; global options: printcmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 29297 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 3, ADDITIONAL: 2 ;; QUESTION SECTION: ;www.paypal.com.IN A ;; ANSWER SECTION: www.paypal.com. 300 IN A 64.4.241.33 www.paypal.com. 300 IN A 64.4.241.49 www.paypal.com. 300 IN A 66.211.169.2 www.paypal.com. 300 IN A 66.211.169.65 ;; AUTHORITY SECTION: paypal.com. 252 IN NS ns2.isc-sns.com. paypal.com. 252 IN NS ns3.isc-sns.info. paypal.com. 252 IN NS ns1.isc-sns.net. ;; ADDITIONAL SECTION: ns3.isc-sns.info. 3559IN A 63.243.194.1 ns3.isc-sns.info. 86352 IN 2001:5a0:10::1 ;; Query time: 5119 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Fri Nov 26 12:05:49 2010 ;; MSG SIZE rcvd: 225 Doing trace : ; DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 www.paypal.com @localhost +trace ;; global options: printcmd . 516870 IN NS i.root-servers.net. . 516870 IN NS j.root-servers.net. . 516870 IN NS k.root-servers.net. . 516870 IN NS l.root-servers.net. . 516870 IN NS m.root-servers.net. . 516870 IN NS a.root-servers.net. . 516870 IN NS b.root-servers.net. . 516870 IN NS c.root-servers.net. . 516870 IN NS d.root-servers.net. . 516870 IN NS e.root-servers.net. . 516870 IN NS f.root-servers.net. . 516870 IN NS g.root-servers.net. . 516870 IN NS h.root-servers.net. ;; Received 492 bytes from 127.0.0.1#53(127.0.0.1) in 2 ms com.172800 IN NS b.gtld-servers.net. com.172800 IN NS h.gtld-servers.net. com.172800 IN NS f.gtld-servers.net. com.172800 IN NS m.gtld-servers.net. com.172800 IN NS c.gtld-servers.net. com.172800 IN NS e.gtld-servers.net. com.172800 IN NS d.gtld-servers.net. com.172800 IN NS a.gtld-servers.net. com.172800 IN NS g.gtld-servers.net. com.172800 IN NS l.gtld-servers.net. com.172800 IN NS j.gtld-servers.net. com.172800 IN NS i.gtld-servers.net. com.172800 IN NS k.gtld-servers.net. ;; Received 504 bytes from 192.36.148.17#53(i.root-servers.net) in 57 ms paypal.com. 172800 IN NS ns1.isc-sns.net. paypal.com. 172800 IN NS ns2.isc-sns.com. paypal.com. 172800 IN NS ns3.isc-sns.info. ;; Received 177 bytes from 192.33.14.30#53(b.gtld-servers.net) in 5498 ms www.paypal.com. 300 IN A 66.211.169.65 www.paypal.com. 300 IN A 64.4.241.33 www.paypal.com. 300 IN A 64.4.241.49 www.paypal.com. 300 IN A 66.211.169.2 paypal.com. 300 IN NS ns3.isc-sns.info. paypal.com. 300 IN NS ns1.isc-sns.net. paypal.com. 300 IN NS ns2.isc-sns.com. ;; Received 285 bytes from 72.52.71.1#53(ns1.isc-sns.net) in 174 ms Version of bind installed : bind-9.3.6-4.P1.el5_4.2 IPv6 has been disabled on the host and firewall turned off during the test. Any toughts ? Regards, Rianto ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Troubleshooting slow DNS lookup
In message aanlktikwrke2mtopsuj-rh28wnknhw5mqhbc5mqms...@mail.gmail.com, Rian to Wahyudi writes: Hi all, Im trying to troubleshoot and find out the reason why some of our DNS lookup take a long time : ns-dev ~ # rndc flushname www.paypal.com ; dig www.paypal.com @localhost ; DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 www.paypal.com @localhost ;; global options: printcmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 29297 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 3, ADDITIONAL: 2 ;; QUESTION SECTION: ;www.paypal.com.IN A ;; ANSWER SECTION: www.paypal.com. 300 IN A 64.4.241.33 www.paypal.com. 300 IN A 64.4.241.49 www.paypal.com. 300 IN A 66.211.169.2 www.paypal.com. 300 IN A 66.211.169.65 ;; AUTHORITY SECTION: paypal.com. 252 IN NS ns2.isc-sns.com. paypal.com. 252 IN NS ns3.isc-sns.info. paypal.com. 252 IN NS ns1.isc-sns.net. ;; ADDITIONAL SECTION: ns3.isc-sns.info. 3559IN A 63.243.194.1 ns3.isc-sns.info. 86352 IN 2001:5a0:10::1 ;; Query time: 5119 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Fri Nov 26 12:05:49 2010 ;; MSG SIZE rcvd: 225 Doing trace : You need to mimic the nameserver more closely and turn on +dnssec. dig +trace +dnssec www.paypal.com I suspect you have a firewall that is blocking the larger replies +dnssec produces. Named will work around this by adjustting the queries it makes but that requires timouts and hence the longer resolution time. Mark ; DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 www.paypal.com @localhost +trace ;; global options: printcmd . 516870 IN NS i.root-servers.net. . 516870 IN NS j.root-servers.net. . 516870 IN NS k.root-servers.net. . 516870 IN NS l.root-servers.net. . 516870 IN NS m.root-servers.net. . 516870 IN NS a.root-servers.net. . 516870 IN NS b.root-servers.net. . 516870 IN NS c.root-servers.net. . 516870 IN NS d.root-servers.net. . 516870 IN NS e.root-servers.net. . 516870 IN NS f.root-servers.net. . 516870 IN NS g.root-servers.net. . 516870 IN NS h.root-servers.net. ;; Received 492 bytes from 127.0.0.1#53(127.0.0.1) in 2 ms com.172800 IN NS b.gtld-servers.net. com.172800 IN NS h.gtld-servers.net. com.172800 IN NS f.gtld-servers.net. com.172800 IN NS m.gtld-servers.net. com.172800 IN NS c.gtld-servers.net. com.172800 IN NS e.gtld-servers.net. com.172800 IN NS d.gtld-servers.net. com.172800 IN NS a.gtld-servers.net. com.172800 IN NS g.gtld-servers.net. com.172800 IN NS l.gtld-servers.net. com.172800 IN NS j.gtld-servers.net. com.172800 IN NS i.gtld-servers.net. com.172800 IN NS k.gtld-servers.net. ;; Received 504 bytes from 192.36.148.17#53(i.root-servers.net) in 57 ms paypal.com. 172800 IN NS ns1.isc-sns.net. paypal.com. 172800 IN NS ns2.isc-sns.com. paypal.com. 172800 IN NS ns3.isc-sns.info. ;; Received 177 bytes from 192.33.14.30#53(b.gtld-servers.net) in 5498 ms www.paypal.com. 300 IN A 66.211.169.65 www.paypal.com. 300 IN A 64.4.241.33 www.paypal.com. 300 IN A 64.4.241.49 www.paypal.com. 300 IN A 66.211.169.2 paypal.com. 300 IN NS ns3.isc-sns.info. paypal.com. 300 IN NS ns1.isc-sns.net. paypal.com. 300 IN NS ns2.isc-sns.com. ;; Received 285 bytes from 72.52.71.1#53(ns1.isc-sns.net) in 174 ms Version of bind installed : bind-9.3.6-4.P1.el5_4.2 IPv6 has been disabled on the host and firewall turned off during the test. Any toughts ? Regards, Rianto --00163646c12e7eca910495eaeb22 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi all,=A0divbr/divdivIm trying to troubleshoot and find out the re= ason why some of our DNS lookup take =A0a long time :/divdivbr/div=
Re: Troubleshooting slow DNS lookup
Hi Mark, Thanks for the pointers , your are spot on! Doing dig +trace +dnssec www.paypal.com always fail. After some investigation with the network guys, it appear that our upstream firewall are dropping DNS UDP packet larger than 512. Cisco FWSM have this configuration enabled by default : http://www.cisco.com/en/US/docs/security/fwsm/fwsm31/command/reference/i2.html#wp1565355 Once again thanks for the help! Regards, Rianto Wahyudi You need to mimic the nameserver more closely and turn on +dnssec. dig +trace +dnssec www.paypal.com I suspect you have a firewall that is blocking the larger replies +dnssec produces. Named will work around this by adjustting the queries it makes but that requires timouts and hence the longer resolution time. Mark --===2929699010037471745==-- -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Troubleshooting slow DNS lookup
In message aanlktimzmc4pgne7n72hnb7gnjuat9r2oktigaazv...@mail.gmail.com, Rian to Wahyudi writes: Hi Mark, Thanks for the pointers , your are spot on! Doing dig +trace +dnssec www.paypal.com always fail. After some investigation with the network guys, it appear that our upstream firewall are dropping DNS UDP packet larger than 512. Cisco FWSM have this configuration enabled by default : http://www.cisco.com/en/US/docs/security/fwsm/fwsm31/command/reference/i2.htm l#wp1565355 So the default is inspect dns maximum-length 512 if I read that page correctly. inspect dns or as a minimum inspect dns maximum-length 4096 will allow reply traffic through for named. I thought I had heard that Cisco had code which looked for the EDNS UDP size option and adjusted the maximum length based on that on a per transaction basis and enforced 512 if there wasn't a EDNS option. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
