That would make sense of a few things. I was afraid to restart the daemon when I didn't see anything in the db.X.X files because I thought it was currently running on some kind of cached or memory resident results, and that if I restarted it would break everything after rereading an empty zone file. So you are saying if I restarted the daemon again a second time after makedns (which I think forces a restart during it doesn't it?) that the zone files would be populated?
The only real snag in me testing this is that this is in a heavily utilized production environment that runs cluster jobs 24x7 except for scheduled maintenance outages, and I have seen the DNS break on one occasion and the result was not pretty on the cluster. I don't actually have a choice for now, but this is just to explain why I was afraid to restart named without knowing what it would do when I saw nothing in the zone files. Once I get some Service Nodes up hopefully they can provide me a redundant DNS to the cluster so I can more easily take risks like restarting the named daemon multiple times. :-) -Josh On Thu, Jan 16, 2014 at 12:50 PM, Russell Jones <russell-l...@jonesmail.me>wrote: > Just a tip and something that bit me before - bind updates its data in > the journal and won't dump new data to the zone files immediately. You can > force it do dump the data so that you can see it by restarting bind. > > You can also test to make sure DNS is being added by digging for the > record. IE, dig @localhost dayhoff.morgan.haib.org. If you get an A > record back, your DNS is working fine. > > > On 1/16/2014 12:39 PM, Josh Nielsen wrote: > > Is there any way to debug what exactly the makedns command does to > named/bind, as in what command it issues? Or does it directly edit the > named files? I could run the named daemon with "-d 9" for debugging level 9 > but I've done that before and the output is daunting and nearly > incomprehensible for the non-veteran in DNS. As I said, running "makedns" > by it self works fine, but using -d or -n causes problems. I have no way of > knowing what xCAT does in the background with those options. > > I checked and /etc/hosts is correct, and /etc/resolv.conf is as well. The > site table has all five possible dns attributes (domain, dnshandler, > forwarders, master, and nameservers) set correctly, and the networks are > defined for every subnet we use. There is nothing that I can see that would > cause it to parse incorrectly, especially since it echoed to > /var/log/messages that it was going through /etc/hosts and creating entries > (but really it didn't) for each host. I did have to add one entry manually > into /etc/hosts though for a SAN storage system. Would seeing an > unrecognized node cause problems? And if so why is there nothing in the > logs about it? I saw it say that it was adding an entry for that node (even > though it didn't - like every other entry): > > "Jan 16 10:32:53 x3650-head01 named[8249]: client 10.20.0.1#56079: > updating zone 'morgan.haib.org/IN': adding an RR at ' > dayhoff.morgan.haib.org' A" > > If I can find out what xCAT is doing to make the changes perhaps I can > reproduce it manually in a test environment and see what is causing it to > wipe out the entries. > > -Josh > > > > > On Thu, Jan 16, 2014 at 12:01 PM, Lissa Valletta <lis...@us.ibm.com>wrote: > >> You will see in the document that makedns is very dependent on what you >> do to setup you Management node. Your /etc/hosts /etc/resolv.conf, etc >> Setting that must be added to the database for your cluster such as in >> the site table, defining your network in the networks table. You can use >> that document to check your database configuration. >> >> Lissa K. Valletta >> 8-3/B10 >> Poughkeepsie, NY 12601 >> (tie 293) 433-3102 >> >> >> >> [image: Inactive hide details for Josh Nielsen ---01/16/2014 12:44:27 >> PM---Okay, I will look through that document. This is the first a]Josh >> Nielsen ---01/16/2014 12:44:27 PM---Okay, I will look through that >> document. This is the first and only time I've used XCATBYPASS with t >> >> From: Josh Nielsen <jniel...@hudsonalpha.org> >> To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>, >> Date: 01/16/2014 12:44 PM >> Subject: Re: [xcat-user] Makedns wiping out DNS records >> ------------------------------ >> >> >> >> Okay, I will look through that document. This is the first and only time >> I've used XCATBYPASS with that command (to get more clarity into what was >> going on in the background). Makedns has been presenting us with the >> problem(s) I mentioned for well over a year now, with previous versions of >> xCAT as well. I'm guessing that if this doesn't happen for other people >> that it must have been some setting when xCAT was originally installed >> which was misconfigured, or maybe something funny in one of the xCAT db >> tables like 'site'. I just wanted to see if anyone has had problems with >> using the "-n" or "-d" options deleting more than they are supposed to. >> >> -Josh >> >> >> On Thu, Jan 16, 2014 at 11:23 AM, Lissa Valletta >> <*lis...@us.ibm.com*<lis...@us.ibm.com>> >> wrote: >> >> We have an entire doc on name resolution also I would not make it a >> common practice to use XCATBYPASS mode. Running with the xcatd is much >> more secure and some things do not work well in XCATBYPASS mode. >> >> >> *https://sourceforge.net/apps/mediawiki/xcat/index.php?title=Cluster_Name_Resolution*<https://sourceforge.net/apps/mediawiki/xcat/index.php?title=Cluster_Name_Resolution> >> >> Lissa K. Valletta >> 8-3/B10 >> Poughkeepsie, NY 12601 >> (tie 293) 433-3102 >> >> >> >> [image: Inactive hide details for Josh Nielsen ---01/16/2014 12:03:28 >> PM---Regardless of the version of xCAT we have used in the past,]Josh >> Nielsen ---01/16/2014 12:03:28 PM---Regardless of the version of xCAT we >> have used in the past, my coworkers and I have noticed that run >> >> From: Josh Nielsen <*jniel...@hudsonalpha.org*<jniel...@hudsonalpha.org> >> > >> To: xCAT Users Mailing list >> <*xcat-user@lists.sourceforge.net*<xcat-user@lists.sourceforge.net>>, >> >> Date: 01/16/2014 12:03 PM >> Subject: [xcat-user] Makedns wiping out DNS records >> ------------------------------ >> >> >> >> Regardless of the version of xCAT we have used in the past, my >> coworkers and I have noticed that running "makedns" with any following >> options can be dangerous for us, because something happens where it will >> wipe out all the records in DNS. If we run "makedns" with no options >> generally any new nodes that have been added to /etc/hosts will be parsed >> and added appropriately. Maybe I am misunderstanding what happens when the >> command is run. >> >> Here is a recent example. My entry for a new node "xcat-serv1" for >> some reason didn't have the reverse lookup from the IP in the DNS file for >> that subnet (db.10.20), and so I decided to try to remove just the DNS >> entry for that node and then re-add it. Here is what I ran: >> >> ---- >> # makedns -d xcat-serv1 >> Handling xcat-serv1 in /etc/hosts. >> Getting reverse zones, this may take several minutes for a large >> cluster. >> Completed getting reverse zones. >> Updating zones. >> Completed updating zones. >> Updating DNS records, this may take several minutes for a large >> cluster. >> Completed updating DNS records. >> ---- >> >> Then I ran it with the "-n" parameter to re-add that node (with >> XCATBYPASS=1): >> >> ---- >> # export XCATBYPASS=1 >> [root@x3650-head01 centos]# makedns -n xcat-serv1 >> Unrecognized escape \h passed through at >> /opt/xcat/lib/perl/xCAT/zvmUtils.pm line 2408. >> Subroutine handled_commands redefined at >> /opt/xcat/lib/perl/xCAT_plugin/*hosts.pm* <http://hosts.pm/> line 28. >> Subroutine delnode redefined at /opt/xcat/lib/perl/xCAT_plugin/ >> *hosts.pm* <http://hosts.pm/> line 33. >> Subroutine addnode redefined at /opt/xcat/lib/perl/xCAT_plugin/ >> *hosts.pm* <http://hosts.pm/> line 58. >> Subroutine build_line redefined at /opt/xcat/lib/perl/xCAT_plugin/ >> *hosts.pm* <http://hosts.pm/> line 117. >> Subroutine addotherinterfaces redefined at >> /opt/xcat/lib/perl/xCAT_plugin/*hosts.pm* <http://hosts.pm/> line 182. >> Subroutine delotherinterfaces redefined at >> /opt/xcat/lib/perl/xCAT_plugin/*hosts.pm* <http://hosts.pm/> line 209. >> Subroutine add_hosts_content redefined at >> /opt/xcat/lib/perl/xCAT_plugin/*hosts.pm* <http://hosts.pm/> line 234. >> Subroutine process_request redefined at >> /opt/xcat/lib/perl/xCAT_plugin/*hosts.pm* <http://hosts.pm/> line 305. >> Subroutine writeout redefined at /opt/xcat/lib/perl/xCAT_plugin/ >> *hosts.pm* <http://hosts.pm/> line 511. >> Subroutine donics redefined at /opt/xcat/lib/perl/xCAT_plugin/ >> *hosts.pm* <http://hosts.pm/> line 545. >> Subroutine getIPdomain redefined at /opt/xcat/lib/perl/xCAT_plugin/ >> *hosts.pm* <http://hosts.pm/> line 801. >> Handling xcat-serv1 in /etc/hosts. >> Getting reverse zones, this may take several minutes for a large >> cluster. >> Completed getting reverse zones. >> Stopping named: . [ OK ] >> Updating zones. >> Completed updating zones. >> Restarting named >> Restarting named complete >> Updating DNS records, this may take several minutes for a large >> cluster. >> Completed updating DNS records. >> DNS setup is completed >> .... >> .... >> ---- >> >> When I did an 'ls' in /var/named many records were missing and the >> ones still left we virtually empty except one entry for the headnode: >> >> BEFORE: >> >> # ls /var/named >> data db.10.29 db.10.40 db.127.0.0 db.172.26.41 >> db.172.29 db.192.168.100 db.haig db.morgan.haib.org.jnl >> db.10.29.jnl db.10.60 db.172.20 db.172.26.42 db.172.30 >> db.cache db.morgan slaves >> db.10.20 db.10.30 db.10.60.jnl db.172.26.40 db.172.26.43 >> db.172.40 db.haib >> *db.morgan.haib.org*<http://db.morgan.haib.org/> >> >> AFTER: >> # ls /var/named >> data db.10.20.jnl db.10.29.jnl db.10.30.jnl db.10.40.jnl >> db.10.60.jnl db.172.26.41 db.172.26.43 db.morgan.haib.org.jnl >> db.10.20 db.10.29 db.10.30 db.10.40 db.10.60 >> db.172.26.40 db.172.26.42 >> *db.morgan.haib.org*<http://db.morgan.haib.org/> >> slaves >> >> >> As you can see several files are missing, Luckily I made a full >> backup yesterday and was able to restore the old files. >> >> >From /var/log/messages you can see the initial deletion message for >> "xcat-serv1": >> >> ---- >> Jan 16 10:32:55 x3650-head01 named[8249]: client 10.20.0.1#56080: >> updating zone '40.10.IN-ADDR.ARPA/IN': adding an RR at >> '11.102.40.10.IN-ADDR.ARPA' PTR >> Jan 16 10:32:55 x3650-head01 named[8249]: client 10.20.0.1#56080: >> updating zone '40.10.IN-ADDR.ARPA/IN': adding an RR at >> '7.101.40.10.IN-ADDR.ARPA' PTR >> Jan 16 10:32:55 x3650-head01 named[8249]: client 10.20.0.1#56080: >> updating zone '40.10.IN-ADDR.ARPA/IN': adding an RR at >> '10.102.40.10.IN-ADDR.ARPA' PTR >> Jan 16 10:32:55 x3650-head01 named[8249]: client 10.20.0.1#56080: >> updating zone '40.10.IN-ADDR.ARPA/IN': adding an RR at >> '27.102.40.10.IN-ADDR.ARPA' PTR >> Jan 16 10:32:55 x3650-head01 named[8249]: client 10.20.0.1#56080: >> updating zone '40.10.IN-ADDR.ARPA/IN': adding an RR at >> '33.101.40.10.IN-ADDR.ARPA' PTR >> Jan 16 10:32:55 x3650-head01 named[8249]: client 10.20.0.1#56080: >> updating zone '40.10.IN-ADDR.ARPA/IN': adding an RR at >> '1.102.40.10.IN-ADDR.ARPA' PTR >> Jan 16 10:32:55 x3650-head01 named[8249]: client 10.20.0.1#56080: >> updating zone >> >> Then for the makedens -a xcat-serv you actually see it update EVERY >> ZONE: >> >> Jan 16 10:32:42 x3650-head01 named[8249]: client 10.20.0.1#43788: >> updating zone '30.10.IN-ADDR.ARPA/IN': adding an RR at >> '1.50.30.10.IN-ADDR.ARPA' PTR >> Jan 16 10:32:42 x3650-head01 named[8249]: client 10.20.0.1#43788: >> updating zone '30.10.IN-ADDR.ARPA/IN': adding an RR at >> '4.50.30.10.IN-ADDR.ARPA' PTR >> Jan 16 10:32:42 x3650-head01 named[8249]: client 10.20.0.1#43788: >> updating zone '30.10.IN-ADDR.ARPA/IN': adding an RR at >> '2.2.30.10.IN-ADDR.ARPA' PTR >> Jan 16 10:32:42 x3650-head01 named[8249]: client 10.20.0.1#43788: >> updating zone '30.10.IN-ADDR.ARPA/IN': adding an RR at >> '3.50.30.10.IN-ADDR.ARPA' PTR >> Jan 16 10:32:42 x3650-head01 named[8249]: client 10.20.0.1#43788: >> updating zone '30.10.IN-ADDR.ARPA/IN': adding an RR at >> '1.2.30.10.IN-ADDR.ARPA' PTR >> Jan 16 10:32:42 x3650-head01 named[8249]: client 10.20.0.1#43788: >> updating zone '30.10.IN-ADDR.ARPA/IN': adding an RR at >> '2.50.30.10.IN-ADDR.ARPA' PTR >> .... >> [long list] >> ---- >> >> After this is when I listed and saw files missing and most of the >> records empty. My guess is the "-d" wiped out all the records somehow and >> not just "xcat-serv1". Either that or the makedns -a deleted all the >> records in order to recreate them. But why would it affect ALL the records >> when I specifically typed "xcat-serv1"? I tried a plain "makedns" with no >> options after this and it also says "adding an RR at" (is that even the >> expected message?) for all the IPs and zones but nothing actually gets >> added. What is up with that? If it just something with our installation or >> does makedns do this foe other people as well? >> >> Regards, >> Josh >> >> >> ------------------------------------------------------------------------------ >> CenturyLink Cloud: The Leader in Enterprise Cloud Services. >> Learn Why More Businesses Are Choosing CenturyLink Cloud For >> Critical Workloads, Development Environments & Everything In Between. >> Get a Quote or Start a Free Trial Today. >> >> >> *http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk*<http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk> >> _______________________________________________ >> xCAT-user mailing list >> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net> >> *https://lists.sourceforge.net/lists/listinfo/xcat-user*<https://lists.sourceforge.net/lists/listinfo/xcat-user> >> >> >> >> >> ------------------------------------------------------------------------------ >> CenturyLink Cloud: The Leader in Enterprise Cloud Services. >> Learn Why More Businesses Are Choosing CenturyLink Cloud For >> Critical Workloads, Development Environments & Everything In Between. >> Get a Quote or Start a Free Trial Today. >> >> >> *http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk*<http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk> >> _______________________________________________ >> xCAT-user mailing list >> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net> >> *https://lists.sourceforge.net/lists/listinfo/xcat-user*<https://lists.sourceforge.net/lists/listinfo/xcat-user> >> >> >> ------------------------------------------------------------------------------ >> CenturyLink Cloud: The Leader in Enterprise Cloud Services. >> Learn Why More Businesses Are Choosing CenturyLink Cloud For >> Critical Workloads, Development Environments & Everything In Between. >> Get a Quote or Start a Free Trial Today. >> >> http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk >> _______________________________________________ >> xCAT-user mailing list >> xCAT-user@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/xcat-user >> >> >> >> ------------------------------------------------------------------------------ >> CenturyLink Cloud: The Leader in Enterprise Cloud Services. >> Learn Why More Businesses Are Choosing CenturyLink Cloud For >> Critical Workloads, Development Environments & Everything In Between. >> Get a Quote or Start a Free Trial Today. >> >> http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk >> _______________________________________________ >> xCAT-user mailing list >> xCAT-user@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/xcat-user >> >> > > > ------------------------------------------------------------------------------ > CenturyLink Cloud: The Leader in Enterprise Cloud Services. > Learn Why More Businesses Are Choosing CenturyLink Cloud For > Critical Workloads, Development Environments & Everything In Between. > Get a Quote or Start a Free Trial Today. > http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk > > > > _______________________________________________ > xCAT-user mailing > listxCAT-user@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/xcat-user > > > > > ------------------------------------------------------------------------------ > CenturyLink Cloud: The Leader in Enterprise Cloud Services. > Learn Why More Businesses Are Choosing CenturyLink Cloud For > Critical Workloads, Development Environments & Everything In Between. > Get a Quote or Start a Free Trial Today. > > http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk > _______________________________________________ > xCAT-user mailing list > xCAT-user@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xcat-user > >
<<image/gif>>
------------------------------------------------------------------------------ CenturyLink Cloud: The Leader in Enterprise Cloud Services. Learn Why More Businesses Are Choosing CenturyLink Cloud For Critical Workloads, Development Environments & Everything In Between. Get a Quote or Start a Free Trial Today. http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________ xCAT-user mailing list xCAT-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xcat-user
