Re: [Freeipa-users] DNS forwarding configuration randomly breaks and stops working
On 6.10.2015 18:57, nat...@nathanpeters.com wrote: >> Your expectation #1 is correct, but there can be multiple reasons why it >> fails. >> >> Did you try to set forward policy = only as I advised you in the previous >> e-mail? Forward policy 'first' does not make sense when split-DNS is >> involved >> because you can end up with mixture of records from different views in one >> cache, which obviously results in a mess. > > Yes, we ended up having to use the forward only policy to get this > working. That is unfortunate, because if our forwarding server ever goes > down or gets rebooted, that essentially disconnects us from being able to > resolve external internet domain names. It would be nice to have > recursion as a fallback, but it seems to go into that mode too often to be > useful in our split DNS situation. > >>> 2. We did some more network packet capture, and noticed that in forward >>> first mode, the FreeIPA server, always sent out both a forward request >>> to >>> the forwarding server, and an additional simultaneous request to the >>> root >>> name servers (recursive mode). It got back responses to both the >>> forwarded and recursive queries it had performed. The recursive query >>> failed due to split DNS and the forwarded query succeeded due to it >>> going >>> to an internal server which had the correct records. Strangely >>> enough... >>> the IPA server ignored the successful forwarded answer, and sent back >>> the >>> 'failed' answer it had gotten through recursion back to the requesting >>> client. What is the behavior supposed to be in this situation and why >>> is >>> the server always sending out the recursive request, even when it gets a >>> valid answer from the forwarded request? >> >> This is weird, but again - it can have multiple reasons. Do you see >> something >> in BIND logs? Does it e.g. complain about DNSSEC validation failures? >> >> Petr^2 Spacek >> > > Yes, we actually were getting DNSSEC validation failures. We had to > disable DNSSEC to get the forward only policy to work. With DNSSEC turned > on, forward only would not work because DNSSEC still tried to directly > contact root servers. It is very likely that this was caused by some misconfiguration in your DNS views. Could you share error messages from BIND logs? We could use them to improve detection logic so we can warn users early instead of tedious debugging. BTW what version if IPA do you use? We were adding checks to catch common misconfigurations to version 4.2. -- Petr Spacek @ Red Hat -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] DNS forwarding configuration randomly breaks and stops working
> Your expectation #1 is correct, but there can be multiple reasons why it > fails. > > Did you try to set forward policy = only as I advised you in the previous > e-mail? Forward policy 'first' does not make sense when split-DNS is > involved > because you can end up with mixture of records from different views in one > cache, which obviously results in a mess. Yes, we ended up having to use the forward only policy to get this working. That is unfortunate, because if our forwarding server ever goes down or gets rebooted, that essentially disconnects us from being able to resolve external internet domain names. It would be nice to have recursion as a fallback, but it seems to go into that mode too often to be useful in our split DNS situation. >> 2. We did some more network packet capture, and noticed that in forward >> first mode, the FreeIPA server, always sent out both a forward request >> to >> the forwarding server, and an additional simultaneous request to the >> root >> name servers (recursive mode). It got back responses to both the >> forwarded and recursive queries it had performed. The recursive query >> failed due to split DNS and the forwarded query succeeded due to it >> going >> to an internal server which had the correct records. Strangely >> enough... >> the IPA server ignored the successful forwarded answer, and sent back >> the >> 'failed' answer it had gotten through recursion back to the requesting >> client. What is the behavior supposed to be in this situation and why >> is >> the server always sending out the recursive request, even when it gets a >> valid answer from the forwarded request? > > This is weird, but again - it can have multiple reasons. Do you see > something > in BIND logs? Does it e.g. complain about DNSSEC validation failures? > > Petr^2 Spacek > Yes, we actually were getting DNSSEC validation failures. We had to disable DNSSEC to get the forward only policy to work. With DNSSEC turned on, forward only would not work because DNSSEC still tried to directly contact root servers. -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] DNS forwarding configuration randomly breaks and stops working
On 5.10.2015 21:57, nat...@nathanpeters.com wrote: Looking at the log entries, it appears that there may have been a network connectivity 'blip' (maybe a switch or router was restarted) at some point and even after connectivity was restored, the global forwarding was failing because the "we can't contact our forwarder" status seemed to get stuck in memory. >> >> Most likely. >> [root@dc1 ~]# ipa dnsconfig-show Global forwarders: 10.21.0.14 Allow PTR sync: TRUE >> >> This means that you are using the default forward policy which is 'first'. >> I.e. BIND daemon on the IPA server is trying to use the forwarder first >> and >> when it fails it fallbacks to asking server on the public Internet. >> >> I speculate that public servers know nothing about the name you were >> asking >> for and this negative answer got cached. This is default behavior in BIND >> and >> IPA did not change it. >> >> Workaround for network problems could be >> $ ipa dnsconfig-mod --forward-policy=only >> which will prevent BIND from falling back to public servers. >> >> Anyway, you should solve network connectivity problems, too :-) >> >> I hope this helps. >> >> -- >> Petr^2 Spacek >> >> -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> Go to http://freeipa.org for more info on the project >> > > Ok, we managed to figure out what was happening here, but I still think > there is a bug somewhere in the FreeIPA DNS components that is > exacerbating the issue. > > We have split DNS in our company. We have a public copy of our DNS > records, which contain only A records. We also have an internal copy of > our DNS records, which contains a bunch of CNAME records. > > When we use nslookup to query the IPA server for stash.externaldomain.net > NSLOOKUP returns that stash.externaldomain.net is a CNAME and it returns > both the CNAME and the associated A address. > > When we query FreeIPA though a DNS client, FreeIPA returns that stash is a > CNAME and does not return the associated A address. It seems like at that > point, FreeIPA decides that instead of sticking in 'forward' mode and > forwarding the request for the CNAME record to the forwarding server, it > decides instead to switch into recursive mode and begin the lookup by > querying root servers for externaldomain.net at which point since it is > going out to the general internet, it gets back answers from our public > facing DNS servers (remember we use split DNS). > > The answer it gets is (quite rightly) that there is no such CNAME record > on the public DNS server. > > So I have a couple questions about how the forward first policy is > supposed to work... Here I would say that IPA does not do anything special - IPA just configures BIND and all the rest is standard BIND behavior. I.e. when in doubt, consult respective BIND docs. > 1. In forward first mode, if the forward server returns a CNAME, is > FreeIPA supposed to ask the same forwarding server for the A record > associated with the CNAME, or is it supposed to then flip into recursive > mode and go to the root servers to find the CNAME? I would expect #1, but > it seems like #2 is always happening no matter what. I would only expect > it to attempt recursion if the initial query actually failed, not just > returned something other than an A record. Your expectation #1 is correct, but there can be multiple reasons why it fails. Did you try to set forward policy = only as I advised you in the previous e-mail? Forward policy 'first' does not make sense when split-DNS is involved because you can end up with mixture of records from different views in one cache, which obviously results in a mess. > 2. We did some more network packet capture, and noticed that in forward > first mode, the FreeIPA server, always sent out both a forward request to > the forwarding server, and an additional simultaneous request to the root > name servers (recursive mode). It got back responses to both the > forwarded and recursive queries it had performed. The recursive query > failed due to split DNS and the forwarded query succeeded due to it going > to an internal server which had the correct records. Strangely enough... > the IPA server ignored the successful forwarded answer, and sent back the > 'failed' answer it had gotten through recursion back to the requesting > client. What is the behavior supposed to be in this situation and why is > the server always sending out the recursive request, even when it gets a > valid answer from the forwarded request? This is weird, but again - it can have multiple reasons. Do you see something in BIND logs? Does it e.g. complain about DNSSEC validation failures? Petr^2 Spacek -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] DNS forwarding configuration randomly breaks and stops working
>>> Looking at the log entries, it appears that there may have been a >>> network >>> connectivity 'blip' (maybe a switch or router was restarted) at some >>> point >>> and even after connectivity was restored, the global forwarding was >>> failing because the "we can't contact our forwarder" status seemed to >>> get >>> stuck in memory. > > Most likely. > >>> [root@dc1 ~]# ipa dnsconfig-show >>> Global forwarders: 10.21.0.14 >>> Allow PTR sync: TRUE > > This means that you are using the default forward policy which is 'first'. > I.e. BIND daemon on the IPA server is trying to use the forwarder first > and > when it fails it fallbacks to asking server on the public Internet. > > I speculate that public servers know nothing about the name you were > asking > for and this negative answer got cached. This is default behavior in BIND > and > IPA did not change it. > > Workaround for network problems could be > $ ipa dnsconfig-mod --forward-policy=only > which will prevent BIND from falling back to public servers. > > Anyway, you should solve network connectivity problems, too :-) > > I hope this helps. > > -- > Petr^2 Spacek > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project > Ok, we managed to figure out what was happening here, but I still think there is a bug somewhere in the FreeIPA DNS components that is exacerbating the issue. We have split DNS in our company. We have a public copy of our DNS records, which contain only A records. We also have an internal copy of our DNS records, which contains a bunch of CNAME records. When we use nslookup to query the IPA server for stash.externaldomain.net NSLOOKUP returns that stash.externaldomain.net is a CNAME and it returns both the CNAME and the associated A address. When we query FreeIPA though a DNS client, FreeIPA returns that stash is a CNAME and does not return the associated A address. It seems like at that point, FreeIPA decides that instead of sticking in 'forward' mode and forwarding the request for the CNAME record to the forwarding server, it decides instead to switch into recursive mode and begin the lookup by querying root servers for externaldomain.net at which point since it is going out to the general internet, it gets back answers from our public facing DNS servers (remember we use split DNS). The answer it gets is (quite rightly) that there is no such CNAME record on the public DNS server. So I have a couple questions about how the forward first policy is supposed to work... 1. In forward first mode, if the forward server returns a CNAME, is FreeIPA supposed to ask the same forwarding server for the A record associated with the CNAME, or is it supposed to then flip into recursive mode and go to the root servers to find the CNAME? I would expect #1, but it seems like #2 is always happening no matter what. I would only expect it to attempt recursion if the initial query actually failed, not just returned something other than an A record. 2. We did some more network packet capture, and noticed that in forward first mode, the FreeIPA server, always sent out both a forward request to the forwarding server, and an additional simultaneous request to the root name servers (recursive mode). It got back responses to both the forwarded and recursive queries it had performed. The recursive query failed due to split DNS and the forwarded query succeeded due to it going to an internal server which had the correct records. Strangely enough... the IPA server ignored the successful forwarded answer, and sent back the 'failed' answer it had gotten through recursion back to the requesting client. What is the behavior supposed to be in this situation and why is the server always sending out the recursive request, even when it gets a valid answer from the forwarded request? -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] DNS forwarding configuration randomly breaks and stops working
>>> Looking at the log entries, it appears that there may have been a >>> network >>> connectivity 'blip' (maybe a switch or router was restarted) at some >>> point >>> and even after connectivity was restored, the global forwarding was >>> failing because the "we can't contact our forwarder" status seemed to >>> get >>> stuck in memory. > > Most likely. > >>> [root@dc1 ~]# ipa dnsconfig-show >>> Global forwarders: 10.21.0.14 >>> Allow PTR sync: TRUE > > This means that you are using the default forward policy which is 'first'. > I.e. BIND daemon on the IPA server is trying to use the forwarder first > and > when it fails it fallbacks to asking server on the public Internet. > > I speculate that public servers know nothing about the name you were > asking > for and this negative answer got cached. This is default behavior in BIND > and > IPA did not change it. > > Workaround for network problems could be > $ ipa dnsconfig-mod --forward-policy=only > which will prevent BIND from falling back to public servers. > > Anyway, you should solve network connectivity problems, too :-) > > I hope this helps. > > -- > Petr^2 Spacek > > -- > Manage your subscription for the Freeipa-users mailing list: > https://www.redhat.com/mailman/listinfo/freeipa-users > Go to http://freeipa.org for more info on the project > Ok, we managed to figure out what was happening here, but I still think there is a bug somewhere in the FreeIPA DNS components that is exacerbating the issue. We have split DNS in our company. We have a public copy of our DNS records, which contain only A records. We also have an internal copy of our DNS records, which contains a bunch of CNAME records. When we use nslookup to query the IPA server for stash.externaldomain.net NSLOOKUP returns that stash.externaldomain.net is a CNAME and it returns the associated A address. When we query FreeIPA though a DNS client, FreeIPA returns that stash is a cname and does not return the associated A address. It seems like at that point, FreeIPA decides that instead of sticking in 'forward' mode and forwarding the request for the CNAME -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] DNS forwarding configuration randomly breaks and stops working
On 3.10.2015 01:47, nat...@nathanpeters.com wrote: > This issue has occured again and I am once again trying to troubleshoot it. > > show forwarder > -- > -bash-4.2$ ipa dnsconfig-show > Global forwarders: 10.21.0.14 > Allow PTR sync: TRUE > > attempt ping > > -bash-4.2$ ping stash.externaldomain.net > ping: unknown host stash.externaldomain.net > > -attempt nslookup > - > -bash-4.2$ nslookup >> stash.externaldomain.net > Server: 127.0.0.1 > Address:127.0.0.1#53 > > ** server can't find stash.externaldomain.net: NXDOMAIN > > *comment* : strange it doesn't work against localhost. Lets make sure > that localhost lookups work at all : > > -bash-4.2$ nslookup >> google.com > Server: 127.0.0.1 > Address:127.0.0.1#53 > > Non-authoritative answer: > Name: google.com > Address: 216.58.216.142 > > *comment* : yup, I can resolve google.com when talking to localhost... > > now lets try to talk to the forwarder configured in the global settings > --- >> server 10.21.0.14 > Default server: 10.21.0.14 > Address: 10.21.0.14#53 >> stash.externaldomain.net > Server: 10.21.0.14 > Address:10.21.0.14#53 > > Non-authoritative answer: > stash.externaldomain.net canonical name = git1.externaldomain.net. > Name: git1.externaldomain.net > Address: 10.20.10.30 > > more troubleshooting > > I ran wireshark to see what the freeipa server was sending back to the > client : > > 3 0.00039310.178.0.99 10.178.21.2 DNS 163 > Standard query response 0x12ca > No such name CNAME git1.externaldomain.net > > I've never seen a 'no such CNAME' response before. Lets look at the > contents of the packet: > > > Frame 4: 163 bytes on wire (1304 bits), 163 bytes captured (1304 bits) > Ethernet II, Src: Vmware_b7:09:c6 (00:50:56:b7:09:c6), Dst: > HewlettP_3c:f9:48 (2c:59:e5:3c:f9:48) > Internet Protocol Version 4, Src: 10.178.0.99 (10.178.0.99), Dst: > 10.178.21.2 (10.178.21.2) > User Datagram Protocol, Src Port: 53 (53), Dst Port: 57374 (57374) > Domain Name System (response) > [Request In: 1] > [Time: 0.000414000 seconds] > Transaction ID: 0x12ca > Flags: 0x8183 Standard query response, No such name > 1... = Response: Message is a response > .000 0... = Opcode: Standard query (0) > .0.. = Authoritative: Server is not an authority > for domain > ..0. = Truncated: Message is not truncated > ...1 = Recursion desired: Do query recursively > 1... = Recursion available: Server can do recursive > queries > .0.. = Z: reserved (0) > ..0. = Answer authenticated: Answer/authority > portion was not authenticated by the server > ...0 = Non-authenticated data: Unacceptable > 0011 = Reply code: No such name (3) > Questions: 1 > Answer RRs: 1 > Authority RRs: 1 > Additional RRs: 0 > Queries > stash.externaldomain.net: type A, class IN > Name: stash.externaldomain.net > [Name Length: 21] > [Label Count: 3] > Type: A (Host Address) (1) > Class: IN (0x0001) > Answers > stash.externaldomain.net: type CNAME, class IN, cname > git1.externaldomain.net > Name: stash.externaldomain.net > Type: CNAME (Canonical NAME for an alias) (5) > Class: IN (0x0001) > Time to live: 22483 > Data length: 20 > CNAME: git1.externaldomain.net > Authoritative nameservers > externaldomain.net: type SOA, class IN, mname > van-dns1.externaldomain.net > Name: externaldomain.net > Type: SOA (Start Of a zone of Authority) (6) > Class: IN (0x0001) > Time to live: 518 > Data length: 38 > Primary name server: van-dns1.externaldomain.net > Responsible authority's mailbox: tech.externaldomain.net > Serial Number: 2015092101 > Refresh Interval: 10800 (3 hours) > Retry Interval: 900 (15 minutes) > Expire limit: 604800 (7 days) > Minimum TTL: 86400 (1 day) > > >> We have a FreeIPA domain running IPA server 4.1.4 on CentOS 7. >> >> We have no per zone forwarding enabled, only a single global forwarder. >> This seems to work fine, but then after a while (several weeks I think) >> will randomly stop working. >> >> We had this issue several weeks ago on a different IPA domain (identical >> setup) in our production network but it was ignored because a server >> restart fixed it. >> >> This issue then re-surfaced in our development domain today (different >> network, different physical hardware, same OS and IPA versions).
Re: [Freeipa-users] DNS forwarding configuration randomly breaks and stops working
This issue has occured again and I am once again trying to troubleshoot it. show forwarder -- -bash-4.2$ ipa dnsconfig-show Global forwarders: 10.21.0.14 Allow PTR sync: TRUE attempt ping -bash-4.2$ ping stash.externaldomain.net ping: unknown host stash.externaldomain.net -attempt nslookup - -bash-4.2$ nslookup > stash.externaldomain.net Server: 127.0.0.1 Address:127.0.0.1#53 ** server can't find stash.externaldomain.net: NXDOMAIN *comment* : strange it doesn't work against localhost. Lets make sure that localhost lookups work at all : -bash-4.2$ nslookup > google.com Server: 127.0.0.1 Address:127.0.0.1#53 Non-authoritative answer: Name: google.com Address: 216.58.216.142 *comment* : yup, I can resolve google.com when talking to localhost... now lets try to talk to the forwarder configured in the global settings --- > server 10.21.0.14 Default server: 10.21.0.14 Address: 10.21.0.14#53 > stash.externaldomain.net Server: 10.21.0.14 Address:10.21.0.14#53 Non-authoritative answer: stash.externaldomain.net canonical name = git1.externaldomain.net. Name: git1.externaldomain.net Address: 10.20.10.30 more troubleshooting I ran wireshark to see what the freeipa server was sending back to the client : 3 0.00039310.178.0.99 10.178.21.2 DNS 163 Standard query response 0x12ca No such name CNAME git1.externaldomain.net I've never seen a 'no such CNAME' response before. Lets look at the contents of the packet: Frame 4: 163 bytes on wire (1304 bits), 163 bytes captured (1304 bits) Ethernet II, Src: Vmware_b7:09:c6 (00:50:56:b7:09:c6), Dst: HewlettP_3c:f9:48 (2c:59:e5:3c:f9:48) Internet Protocol Version 4, Src: 10.178.0.99 (10.178.0.99), Dst: 10.178.21.2 (10.178.21.2) User Datagram Protocol, Src Port: 53 (53), Dst Port: 57374 (57374) Domain Name System (response) [Request In: 1] [Time: 0.000414000 seconds] Transaction ID: 0x12ca Flags: 0x8183 Standard query response, No such name 1... = Response: Message is a response .000 0... = Opcode: Standard query (0) .0.. = Authoritative: Server is not an authority for domain ..0. = Truncated: Message is not truncated ...1 = Recursion desired: Do query recursively 1... = Recursion available: Server can do recursive queries .0.. = Z: reserved (0) ..0. = Answer authenticated: Answer/authority portion was not authenticated by the server ...0 = Non-authenticated data: Unacceptable 0011 = Reply code: No such name (3) Questions: 1 Answer RRs: 1 Authority RRs: 1 Additional RRs: 0 Queries stash.externaldomain.net: type A, class IN Name: stash.externaldomain.net [Name Length: 21] [Label Count: 3] Type: A (Host Address) (1) Class: IN (0x0001) Answers stash.externaldomain.net: type CNAME, class IN, cname git1.externaldomain.net Name: stash.externaldomain.net Type: CNAME (Canonical NAME for an alias) (5) Class: IN (0x0001) Time to live: 22483 Data length: 20 CNAME: git1.externaldomain.net Authoritative nameservers externaldomain.net: type SOA, class IN, mname van-dns1.externaldomain.net Name: externaldomain.net Type: SOA (Start Of a zone of Authority) (6) Class: IN (0x0001) Time to live: 518 Data length: 38 Primary name server: van-dns1.externaldomain.net Responsible authority's mailbox: tech.externaldomain.net Serial Number: 2015092101 Refresh Interval: 10800 (3 hours) Retry Interval: 900 (15 minutes) Expire limit: 604800 (7 days) Minimum TTL: 86400 (1 day) > We have a FreeIPA domain running IPA server 4.1.4 on CentOS 7. > > We have no per zone forwarding enabled, only a single global forwarder. > This seems to work fine, but then after a while (several weeks I think) > will randomly stop working. > > We had this issue several weeks ago on a different IPA domain (identical > setup) in our production network but it was ignored because a server > restart fixed it. > > This issue then re-surfaced in our development domain today (different > network, different physical hardware, same OS and IPA versions). > > I received a report today from a developer that he could not ping a > machine in another domain so I verified network connectivity and > everything was fine. When I tried to resolve the name from the IPA dc > using ping it would fail, but nslookup directly to the forward server > worked fine. > > ip
