This issue has occured again and I am once again trying to troubleshoot it.

show forwarder
--------------
-bash-4.2$ ipa dnsconfig-show
  Global forwarders: 10.21.0.14
  Allow PTR sync: TRUE

attempt ping
------------
  -bash-4.2$ ping stash.externaldomain.net
ping: unknown host stash.externaldomain.net

-attempt nslookup
-----------------
-bash-4.2$ nslookup
> stash.externaldomain.net
Server:         127.0.0.1
Address:        127.0.0.1#53

** server can't find stash.externaldomain.net: NXDOMAIN

*comment* : strange it doesn't work against localhost.  Lets make sure
that localhost lookups work at all :

-bash-4.2$ nslookup
> google.com
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
Name:   google.com
Address: 216.58.216.142

*comment* : yup, I can resolve google.com when talking to localhost...

now lets try to talk to the forwarder configured in the global settings
-----------------------------------------------------------------------
> server 10.21.0.14
Default server: 10.21.0.14
Address: 10.21.0.14#53
> stash.externaldomain.net
Server:         10.21.0.14
Address:        10.21.0.14#53

Non-authoritative answer:
stash.externaldomain.net   canonical name = git1.externaldomain.net.
Name:   git1.externaldomain.net
Address: 10.20.10.30

more troubleshooting
--------------------
I ran wireshark to see what the freeipa server was sending back to the
client :

3       0.000393        10.178.0.99     10.178.21.2     DNS     163     
Standard query response 0x12ca
No such name CNAME git1.externaldomain.net

I've never seen a 'no such CNAME' response before.  Lets look at the
contents of the packet:


Frame 4: 163 bytes on wire (1304 bits), 163 bytes captured (1304 bits)
Ethernet II, Src: Vmware_b7:09:c6 (00:50:56:b7:09:c6), Dst:
HewlettP_3c:f9:48 (2c:59:e5:3c:f9:48)
Internet Protocol Version 4, Src: 10.178.0.99 (10.178.0.99), Dst:
10.178.21.2 (10.178.21.2)
User Datagram Protocol, Src Port: 53 (53), Dst Port: 57374 (57374)
Domain Name System (response)
    [Request In: 1]
    [Time: 0.000414000 seconds]
    Transaction ID: 0x12ca
    Flags: 0x8183 Standard query response, No such name
        1... .... .... .... = Response: Message is a response
        .000 0... .... .... = Opcode: Standard query (0)
        .... .0.. .... .... = Authoritative: Server is not an authority
for domain
        .... ..0. .... .... = Truncated: Message is not truncated
        .... ...1 .... .... = Recursion desired: Do query recursively
        .... .... 1... .... = Recursion available: Server can do recursive
queries
        .... .... .0.. .... = Z: reserved (0)
        .... .... ..0. .... = Answer authenticated: Answer/authority
portion was not authenticated by the server
        .... .... ...0 .... = Non-authenticated data: Unacceptable
        .... .... .... 0011 = Reply code: No such name (3)
    Questions: 1
    Answer RRs: 1
    Authority RRs: 1
    Additional RRs: 0
    Queries
        stash.externaldomain.net: type A, class IN
            Name: stash.externaldomain.net
            [Name Length: 21]
            [Label Count: 3]
            Type: A (Host Address) (1)
            Class: IN (0x0001)
    Answers
        stash.externaldomain.net: type CNAME, class IN, cname
git1.externaldomain.net
            Name: stash.externaldomain.net
            Type: CNAME (Canonical NAME for an alias) (5)
            Class: IN (0x0001)
            Time to live: 22483
            Data length: 20
            CNAME: git1.externaldomain.net
    Authoritative nameservers
        externaldomain.net: type SOA, class IN, mname
van-dns1.externaldomain.net
            Name: externaldomain.net
            Type: SOA (Start Of a zone of Authority) (6)
            Class: IN (0x0001)
            Time to live: 518
            Data length: 38
            Primary name server: van-dns1.externaldomain.net
            Responsible authority's mailbox: tech.externaldomain.net
            Serial Number: 2015092101
            Refresh Interval: 10800 (3 hours)
            Retry Interval: 900 (15 minutes)
            Expire limit: 604800 (7 days)
            Minimum TTL: 86400 (1 day)


> We have a FreeIPA domain running IPA server 4.1.4 on CentOS 7.
>
> We have no per zone forwarding enabled, only a single global forwarder.
> This seems to work fine, but then after a while (several weeks I think)
> will randomly stop working.
>
> We had this issue several weeks ago on a different IPA domain (identical
> setup) in our production network but it was ignored because a server
> restart fixed it.
>
> This issue then re-surfaced in our development domain today (different
> network, different physical hardware, same OS and IPA versions).
>
> I received a report today from a developer that he could not ping a
> machine in another domain so I verified network connectivity and
> everything was fine.  When I tried to resolve the name from the IPA dc
> using ping it would fail, but nslookup directly to the forward server
> worked fine.
>
> ipactl showed no issues, and only after I restarted the server did the
> lookups start working again.
>
> Console log below :
>
> Using username "myipausername".
> Last login: Thu Oct  1 16:36:51 2015 from 10.5.5.57
> [myipausername@dc1 ~]$ sudo su -
> Last login: Tue Sep 29 19:03:39 UTC 2015 on pts/3
>
> ATTEMPT FIRST PING TO UNRESOLVABLE HOST
> =======================================
> [root@dc1 ~]# ping artifactory.externaldomain.net
> ping: unknown host artifactory.externaldomain.net
>
> CHECK IPA STATUS
> ================
> [root@dc1 ~]# ipactl status
> Directory Service: RUNNING
> krb5kdc Service: RUNNING
> kadmin Service: RUNNING
> named Service: RUNNING
> ipa_memcached Service: RUNNING
> httpd Service: RUNNING
> pki-tomcatd Service: RUNNING
> smb Service: RUNNING
> winbind Service: RUNNING
> ipa-otpd Service: RUNNING
> ipa-dnskeysyncd Service: RUNNING
> ipa: INFO: The ipactl command was successful
>
> ATTEMPT PING OF GLOBAL FORWARDER
> ================================
> [root@dc1 ~]# ping 10.21.0.14
> PING 10.21.0.14 (10.21.0.14) 56(84) bytes of data.
> 64 bytes from 10.21.0.14: icmp_seq=1 ttl=64 time=0.275 ms
> 64 bytes from 10.21.0.14: icmp_seq=2 ttl=64 time=0.327 ms
> ^C
> --- 10.21.0.14 ping statistics ---
> 2 packets transmitted, 2 received, 0% packet loss, time 1000ms
> rtt min/avg/max/mdev = 0.275/0.301/0.327/0.026 ms
>
> MANUAL NSLOOKUP OF DOMAIN ON GLOBAL FORWARDER FROM IPA DC
> =========================================================
> [root@dc1 ~]# nslookup
>> server 10.21.0.14
> Default server: 10.21.0.14
> Address: 10.21.0.14#53
>> artifactory.externaldomain.net
> Server:         10.21.0.14
> Address:        10.21.0.14#53
>
> Non-authoritative answer:
> artifactory.externaldomain.net     canonical name =
> van-artifactory1.externaldomain.net.
> Name:   van-artifactory1.externaldomain.net
> Address: 10.20.10.14
>
> RE-ATTEMPT PING SINCE WE KNOW THAT NAME RESOLUTION (at least via nslookup
> IS WORKING FROM THIS MACHINE
> ======================================================================================================
>> ^C[root@dc1 ~]# ping artifactory.externaldomain.net
> ping: unknown host artifactory.externaldomain.net
> [root@dc1 ~]# ping van-artifactory1.externaldomain.net
> ping: unknown host van-artifactory1.externaldomain.net
>
> RESTART IPA SERVICES
> ====================
> [root@dc1 ~]# ipactl restart
> Restarting Directory Service
> Restarting krb5kdc Service
> Restarting kadmin Service
> Restarting named Service
> Restarting ipa_memcached Service
> Restarting httpd Service
> Restarting pki-tomcatd Service
> Restarting smb Service
> Restarting winbind Service
> Restarting ipa-otpd Service
> Restarting ipa-dnskeysyncd Service
> ipa: INFO: The ipactl command was successful
> [root@dc1 ~]# ipa dnsconfig-show
> ipa: ERROR: did not receive Kerberos credentials
> [root@dc1 ~]# kinit myipausername
> Password for myipausern...@ipadomain.net:
>
> OUTPUT GLOBAL FORWARDER CONFIG FOR TROUBLESHOOTING
> ==================================================
> [root@dc1 ~]# ipa dnsconfig-show
>   Global forwarders: 10.21.0.14
>   Allow PTR sync: TRUE
>
> PING NOW WORKS BECAUSE IPA SERVICES WERE RESTARTED
> ==================================================
> [root@dc1 ~]# ping artifactory.externaldomain.net
> PING van-artifactory1.externaldomain.net (10.20.10.14) 56(84) bytes of
> data.
> 64 bytes from 10.20.10.14: icmp_seq=1 ttl=60 time=3.00 ms
> 64 bytes from 10.20.10.14: icmp_seq=2 ttl=60 time=1.42 ms
> 64 bytes from 10.20.10.14: icmp_seq=3 ttl=60 time=2.39 ms
> ^C
> --- van-artifactory1.externaldomain.net ping statistics ---
> 3 packets transmitted, 3 received, 0% packet loss, time 2004ms
> rtt min/avg/max/mdev = 1.420/2.274/3.004/0.653 ms
> [root@dc1 ~]#
>
> Here are some strange enties from my /var/log/messages relating to errors
> from today :
>
> Oct  1 20:39:31 dc1 named-pkcs11[15066]: checkhints: unable to get root NS
> rrset from cache: not found
> Oct  1 20:39:17 dc1 named-pkcs11[15066]: error (network unreachable)
> resolving 'pmdb1.ipadomain.net/A/IN': 2001:500:2f::f#53
> Oct  1 20:39:17 dc1 named-pkcs11[15066]: error (network unreachable)
> resolving 'pmdb1.ipadomain.net/AAAA/IN': 2001:500:2f::f#53
>
> Looking at the log entries, it appears that there may have been a network
> connectivity 'blip' (maybe a switch or router was restarted) at some point
> and even after connectivity was restored, the global forwarding was
> failing because the "we can't contact our forwarder" status seemed to get
> stuck in memory.
>
>
> --
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project
>



-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Reply via email to