Sorry for the lapse in my communications. I wear a lot of hats around my office and sometimes things get shoved by the way-side.
I would have posted to the list sooner but the Perl script is working too blasted well and my attention was directed away from this issue for a while. I apologize if my lack of communication has caused anyone any issues. I do, however, have some developments that I can share. I believe that the cause of the PFDNS crashing could be related to the system clock as was suggested before by another user. I will be looking into a way to better track my system clock to verify this, but the trouble is that the crashes seem to happen so randomly except for one thing. The last three crashes I had happened on Nov 17 3:34am, Dec 1 3:26am, Dec 8 3:39am. All the crashes have been ~3:30am, it may be that is when PF is doing its log rotations, compressions, etc. and the CPU may be getting taxed causing the system clock to drift outside of some critical zone causing the DNSSec portion of PFDNS to crash. I dont have any HARD evidence of this yet, but it seems likely. Regardless, by checking the status of the PFDNS process one every 60 seconds and restarting it if it fails I have not had a single outage noticed by my users since I wrote the script. >> I pretty much put my pilot on hold for this issue. Over all the PF product has been very stable and I would encourage you to continue on with your pilot. Just keep an eye on your PF processes. I can share my script if anyone is interested. It is probably horrendously bad so if you are a more experienced Perl programmer feel free to make improvements, if you do please contribute them back. I have some pressing matters to attend to at the moment or I would do a more through write-up but I will endeavour to assist anyone I can if you have any specific questions. Jake Sallee Godfather of Bandwidth System Engineer University of Mary Hardin-Baylor 900 College St. Belton, Texas 76513 Fone: 254-295-4658 Phax: 254-295-4221 ________________________________________ From: Stephen Wittstruck [[email protected]] Sent: Friday, December 06, 2013 6:23 PM To: Packetfence Users Digest Subject: Re: [PacketFence-users] PFDNS The saga continues Hi Jake, Just curious is you know of any news from Inverse regarding the DNS abend issue you found? I pretty much put my pilot on hold for this issue. Steve CSM On Oct 9, 2013, at 2:40 PM, Sallee, Stephen (Jake) <[email protected]> wrote: >>> I'm sorry, I wasn't running 4.0.6-2, only 4.0.6-1 (not sure how that >>> happened.) > > NP, thanks for the info. > > I would still like to find the root cause of my PFDNS service crashing, but > so far it has been pretty stable. > > Right now I have no idea why it dies since it seems to fail completely > silently. > > So what do I do? I wrote a Perl script that monitors the PFNDS service and > pulls all the PF logs and the syslog from the server if it fails, I also have > a rolling pcap running that I can use to reconstruct all the DNS traffic > from the last 10 min. If the service stops the script gathers all the logs > and the pcaps and tars it up for me, the it tries to restart the service. If > it is successful it just goes back to watching and waiting, if not it bombs > out. > > Hopefully I will find something in the tarball when I have another incident. > > Jake Sallee > Godfather of Bandwidth > System Engineer > University of Mary Hardin-Baylor > 900 College St. > Belton TX. 76513 > Fone: 254-295-4658 > Phax: 254-295-4221 > HTTP://WWW.UMHB.EDU > > -----Original Message----- > From: Stephen Wittstruck [mailto:[email protected]] > Sent: Wednesday, October 09, 2013 10:55 AM > To: [email protected] > Subject: Re: [PacketFence-users] PFDNS The saga continues > > Hi again, Jake, Et al: > > I'm sorry, I wasn't running 4.0.6-2, only 4.0.6-1 (not sure how that > happened.) > > Turns out the 4.0.6-2 GUI does stop the individual PF processes (at least the > 3 or 4 I tried.) All processes would restart too except for PFDNS, at least > according to the GUI and pfcmd; I had to reboot the server to recover PFDNS. > I'm not a linux admin so don't know any tricks to confirm this except for the > ps command, which I didn't try. > > My apology for the bad info earlier. > > Steve > CSM > > > On Sep 30, 2013, at 10:49 AM, Stephen Wittstruck <[email protected]> wrote: > >> Hi Jake, >> >> I'm running the exact same platform, i.e. OS and PF, though not in >> production yet. >> >> I couldn't get PFDNS to stop through the GUI. Curiously I tried the others, >> only PFDHCPLISTENER would stop by using the GUI; it would restart also. >> >> Still curious I tried the command line "./pfcmd service pfdns stop" which >> didn't work. Restart stopped it, but it looks like a server reboot is >> needed to restart it as nothing else is working (I haven't done this yet), >> including the GUI. Below is the terminal session text of these tests. >> >> Maybe "./pfcmd service pfdns watch" is what you need? >> >> ============================================ >> [swittstr@nac-dev bin]$ ./pfcmd service pfdns stop >> service|command >> pfdns|stop >> >> [swittstr@nac-dev bin]$ ./pfcmd service pfdns status >> service|shouldBeStarted|pid >> pfdns|1|1573 >> >> [swittstr@nac-dev bin]$ ./pfcmd service pfdns >> Usage: pfcmd service <service> [start|stop|restart|status|watch] >> >> stop/stop/restart specified service >> status returns PID of specified PF daemon or 0 if not running watch >> acts as a service watcher which can send email/restart the services >> >> Services managed by PacketFence: >> dhcpd | dhcpd daemon >> httpd.webservices| Apache Webservices >> httpd.admin | Apache Web admin >> httpd.portal | Apache Captive Portal >> pfdns | DNS daemon >> pf | all services that should be running based on your config >> pfdetect | PF snort alert parser >> pfdhcplistener | PF DHCP monitoring daemon >> pfmon | PF ARP monitoring daemon >> pfsetvlan | PF VLAN isolation daemon >> radiusd | FreeRADIUS daemon >> snmptrapd | SNMP trap receiver daemon >> snort | Sourcefire Snort IDS >> suricata | Suricata IDS >> >> watch >> Watch performs services checks to make sure that everything is fine. >> It's behavior is controlled by servicewatch configuration parameters. >> watch is typically best called from cron with something like: >> */5 * * * * /usr/local/pf/bin/pfcmd service pf watch >> >> [swittstr@nac-dev bin]$ ./pfcmd service pfdns watch >> >> [swittstr@nac-dev bin]$ ./pfcmd service pfdns restart >> service|command >> config files|restart >> iptables|restart >> pfdns|restart >> >> [swittstr@nac-dev bin]$ ./pfcmd service pfdns status >> service|shouldBeStarted|pid >> pfdns|1|0 >> >> [swittstr@nac-dev bin]$ ./pfcmd service pfdns restart >> service|command >> config files|restart >> iptables|restart >> pfdns|restart >> >> [swittstr@nac-dev bin]$ ./pfcmd service pfdns status >> service|shouldBeStarted|pid >> pfdns|1|0 >> >> [swittstr@nac-dev bin]$ ./pfcmd service pfdns start >> httpd.admin|already running Checking configuration sanity... >> service|command >> config files|start >> iptables|start >> pfdns|start >> >> [swittstr@nac-dev bin]$ ./pfcmd service pfdns status >> service|shouldBeStarted|pid >> pfdns|1|0 >> >> [swittstr@nac-dev bin]$ ./pfcmd service pfdns status (after waiting 10 or >> 15 minutes) >> [sudo] password for swittstr: >> service|shouldBeStarted|pid >> pfdns|1|0 >> ============================================= >> >> Steve >> CSM >> >> >> On Sep 30, 2013, at 9:57 AM, "Sallee, Stephen (Jake)" >> <[email protected]> >> wrote: >> >>> Hello PacketFence Family! >>> >>> I am running PF 4.0.6-2 on CentOS 6.4 fully updated. >>> >>> I am still seeing an issue with PFNDS seemingly randomly crashing. I would >>> like to get some more information of the problem but since I cannot stare >>> at a single terminal all day to see exactly what is happening I am looking >>> for some kind of monitoring solution. >>> >>> Ideally I would like to monitor the PFDNS process and take some actions if >>> I see it fail, namely starting the bloody thing back up again as well as >>> pulling all the logs for further dissection. >>> >>> I can do this with some srcipt-fu but I was wondering of anyone out there >>> already has something like this or knows of it, that way I can avoid >>> reinventing the proverbial wheel. >>> >>> Also, I have noticed that the issue I reported some time ago where >>> some PF services cannot be started from the webgui is still around >>> for me. Can anyone verify this? Specifically, if PFDNS is stopped >>> try starting it again using the butting in the services menu in the >>> webgui. For me it does not work, but I do not get an error banner as >>> normal. The service still says stopped though.] >>> >>> As always, thank you for your time and consideration. >>> >>> Jake Sallee >>> Godfather of Bandwidth >>> System Engineer >>> University of Mary Hardin-Baylor >>> >>> 900 College St. >>> Belton, Texas >>> 76513 >>> >>> Fone: 254-295-4658 >>> Phax: 254-295-4221 >>> >>> --------------------------------------------------------------------- >>> --------- October Webinars: Code for Performance Free Intel webinars >>> can help you accelerate application performance. >>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the >>> most from the latest Intel processors and coprocessors. See abstracts >>> and register > >>> http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg. >>> clktrk _______________________________________________ >>> PacketFence-users mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/packetfence-users >> > > > ------------------------------------------------------------------------------ > October Webinars: Code for Performance > Free Intel webinars can help you accelerate application performance. > Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from > the latest Intel processors and coprocessors. See abstracts and register > > http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk > _______________________________________________ > PacketFence-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/packetfence-users ------------------------------------------------------------------------------ Sponsored by Intel(R) XDK Develop, test and display web and hybrid apps with a single code base. Download it for free now! http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk _______________________________________________ PacketFence-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/packetfence-users
