Thanks Jake. No problem at all.
I've put my PF tests on hold several times for months at a time for the same multi hat reasons you have, so postponing our pilot wasn't anything new for us ;-}. I missed the system clock suggestion. Thanks again. I appreciate your regular posts. Steve, CSM On Dec 9, 2013, at 8:20 AM, Sallee, Stephen (Jake) <[email protected]> wrote: > Sorry for the lapse in my communications. I wear a lot of hats around my > office and sometimes things get shoved by the way-side. > > I would have posted to the list sooner but the Perl script is working too > blasted well and my attention was directed away from this issue for a while. > > I apologize if my lack of communication has caused anyone any issues. > > I do, however, have some developments that I can share. > > I believe that the cause of the PFDNS crashing could be related to the system > clock as was suggested before by another user. I will be looking into a way > to better track my system clock to verify this, but the trouble is that the > crashes seem to happen so randomly except for one thing. The last three > crashes I had happened on Nov 17 3:34am, Dec 1 3:26am, Dec 8 3:39am. > > All the crashes have been ~3:30am, it may be that is when PF is doing its log > rotations, compressions, etc. and the CPU may be getting taxed causing the > system clock to drift outside of some critical zone causing the DNSSec > portion of PFDNS to crash. > > I dont have any HARD evidence of this yet, but it seems likely. > > Regardless, by checking the status of the PFDNS process one every 60 seconds > and restarting it if it fails I have not had a single outage noticed by my > users since I wrote the script. > >>> I pretty much put my pilot on hold for this issue. > > Over all the PF product has been very stable and I would encourage you to > continue on with your pilot. Just keep an eye on your PF processes. > > I can share my script if anyone is interested. It is probably horrendously > bad so if you are a more experienced Perl programmer feel free to make > improvements, if you do please contribute them back. > > I have some pressing matters to attend to at the moment or I would do a more > through write-up but I will endeavour to assist anyone I can if you have any > specific questions. > > > Jake Sallee > Godfather of Bandwidth > System Engineer > University of Mary Hardin-Baylor > > 900 College St. > Belton, Texas > 76513 > > Fone: 254-295-4658 > Phax: 254-295-4221 > > ________________________________________ > From: Stephen Wittstruck [[email protected]] > Sent: Friday, December 06, 2013 6:23 PM > To: Packetfence Users Digest > Subject: Re: [PacketFence-users] PFDNS The saga continues > > Hi Jake, > > Just curious is you know of any news from Inverse regarding the DNS abend > issue you found? I pretty much put my pilot on hold for this issue. > > Steve > CSM > > > On Oct 9, 2013, at 2:40 PM, Sallee, Stephen (Jake) <[email protected]> > wrote: > >>>> I'm sorry, I wasn't running 4.0.6-2, only 4.0.6-1 (not sure how that >>>> happened.) >> >> NP, thanks for the info. >> >> I would still like to find the root cause of my PFDNS service crashing, but >> so far it has been pretty stable. >> >> Right now I have no idea why it dies since it seems to fail completely >> silently. >> >> So what do I do? I wrote a Perl script that monitors the PFNDS service and >> pulls all the PF logs and the syslog from the server if it fails, I also >> have a rolling pcap running that I can use to reconstruct all the DNS >> traffic from the last 10 min. If the service stops the script gathers all >> the logs and the pcaps and tars it up for me, the it tries to restart the >> service. If it is successful it just goes back to watching and waiting, if >> not it bombs out. >> >> Hopefully I will find something in the tarball when I have another incident. >> >> Jake Sallee >> Godfather of Bandwidth >> System Engineer >> University of Mary Hardin-Baylor >> 900 College St. >> Belton TX. 76513 >> Fone: 254-295-4658 >> Phax: 254-295-4221 >> HTTP://WWW.UMHB.EDU >> >> -----Original Message----- >> From: Stephen Wittstruck [mailto:[email protected]] >> Sent: Wednesday, October 09, 2013 10:55 AM >> To: [email protected] >> Subject: Re: [PacketFence-users] PFDNS The saga continues >> >> Hi again, Jake, Et al: >> >> I'm sorry, I wasn't running 4.0.6-2, only 4.0.6-1 (not sure how that >> happened.) >> >> Turns out the 4.0.6-2 GUI does stop the individual PF processes (at least >> the 3 or 4 I tried.) All processes would restart too except for PFDNS, at >> least according to the GUI and pfcmd; I had to reboot the server to recover >> PFDNS. I'm not a linux admin so don't know any tricks to confirm this >> except for the ps command, which I didn't try. >> >> My apology for the bad info earlier. >> >> Steve >> CSM >> >> >> On Sep 30, 2013, at 10:49 AM, Stephen Wittstruck <[email protected]> wrote: >> >>> Hi Jake, >>> >>> I'm running the exact same platform, i.e. OS and PF, though not in >>> production yet. >>> >>> I couldn't get PFDNS to stop through the GUI. Curiously I tried the >>> others, only PFDHCPLISTENER would stop by using the GUI; it would restart >>> also. >>> >>> Still curious I tried the command line "./pfcmd service pfdns stop" which >>> didn't work. Restart stopped it, but it looks like a server reboot is >>> needed to restart it as nothing else is working (I haven't done this yet), >>> including the GUI. Below is the terminal session text of these tests. >>> >>> Maybe "./pfcmd service pfdns watch" is what you need? >>> >>> ============================================ >>> [swittstr@nac-dev bin]$ ./pfcmd service pfdns stop >>> service|command >>> pfdns|stop >>> >>> [swittstr@nac-dev bin]$ ./pfcmd service pfdns status >>> service|shouldBeStarted|pid >>> pfdns|1|1573 >>> >>> [swittstr@nac-dev bin]$ ./pfcmd service pfdns >>> Usage: pfcmd service <service> [start|stop|restart|status|watch] >>> >>> stop/stop/restart specified service >>> status returns PID of specified PF daemon or 0 if not running watch >>> acts as a service watcher which can send email/restart the services >>> >>> Services managed by PacketFence: >>> dhcpd | dhcpd daemon >>> httpd.webservices| Apache Webservices >>> httpd.admin | Apache Web admin >>> httpd.portal | Apache Captive Portal >>> pfdns | DNS daemon >>> pf | all services that should be running based on your config >>> pfdetect | PF snort alert parser >>> pfdhcplistener | PF DHCP monitoring daemon >>> pfmon | PF ARP monitoring daemon >>> pfsetvlan | PF VLAN isolation daemon >>> radiusd | FreeRADIUS daemon >>> snmptrapd | SNMP trap receiver daemon >>> snort | Sourcefire Snort IDS >>> suricata | Suricata IDS >>> >>> watch >>> Watch performs services checks to make sure that everything is fine. >>> It's behavior is controlled by servicewatch configuration parameters. >>> watch is typically best called from cron with something like: >>> */5 * * * * /usr/local/pf/bin/pfcmd service pf watch >>> >>> [swittstr@nac-dev bin]$ ./pfcmd service pfdns watch >>> >>> [swittstr@nac-dev bin]$ ./pfcmd service pfdns restart >>> service|command >>> config files|restart >>> iptables|restart >>> pfdns|restart >>> >>> [swittstr@nac-dev bin]$ ./pfcmd service pfdns status >>> service|shouldBeStarted|pid >>> pfdns|1|0 >>> >>> [swittstr@nac-dev bin]$ ./pfcmd service pfdns restart >>> service|command >>> config files|restart >>> iptables|restart >>> pfdns|restart >>> >>> [swittstr@nac-dev bin]$ ./pfcmd service pfdns status >>> service|shouldBeStarted|pid >>> pfdns|1|0 >>> >>> [swittstr@nac-dev bin]$ ./pfcmd service pfdns start >>> httpd.admin|already running Checking configuration sanity... >>> service|command >>> config files|start >>> iptables|start >>> pfdns|start >>> >>> [swittstr@nac-dev bin]$ ./pfcmd service pfdns status >>> service|shouldBeStarted|pid >>> pfdns|1|0 >>> >>> [swittstr@nac-dev bin]$ ./pfcmd service pfdns status (after waiting 10 or >>> 15 minutes) >>> [sudo] password for swittstr: >>> service|shouldBeStarted|pid >>> pfdns|1|0 >>> ============================================= >>> >>> Steve >>> CSM >>> >>> >>> On Sep 30, 2013, at 9:57 AM, "Sallee, Stephen (Jake)" >>> <[email protected]> >>> wrote: >>> >>>> Hello PacketFence Family! >>>> >>>> I am running PF 4.0.6-2 on CentOS 6.4 fully updated. >>>> >>>> I am still seeing an issue with PFNDS seemingly randomly crashing. I >>>> would like to get some more information of the problem but since I cannot >>>> stare at a single terminal all day to see exactly what is happening I am >>>> looking for some kind of monitoring solution. >>>> >>>> Ideally I would like to monitor the PFDNS process and take some actions if >>>> I see it fail, namely starting the bloody thing back up again as well as >>>> pulling all the logs for further dissection. >>>> >>>> I can do this with some srcipt-fu but I was wondering of anyone out there >>>> already has something like this or knows of it, that way I can avoid >>>> reinventing the proverbial wheel. >>>> >>>> Also, I have noticed that the issue I reported some time ago where >>>> some PF services cannot be started from the webgui is still around >>>> for me. Can anyone verify this? Specifically, if PFDNS is stopped >>>> try starting it again using the butting in the services menu in the >>>> webgui. For me it does not work, but I do not get an error banner as >>>> normal. The service still says stopped though.] >>>> >>>> As always, thank you for your time and consideration. >>>> >>>> Jake Sallee >>>> Godfather of Bandwidth >>>> System Engineer >>>> University of Mary Hardin-Baylor >>>> >>>> 900 College St. >>>> Belton, Texas >>>> 76513 >>>> >>>> Fone: 254-295-4658 >>>> Phax: 254-295-4221 >>>> >>>> --------------------------------------------------------------------- >>>> --------- October Webinars: Code for Performance Free Intel webinars >>>> can help you accelerate application performance. >>>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the >>>> most from the latest Intel processors and coprocessors. See abstracts >>>> and register > >>>> http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg. >>>> clktrk _______________________________________________ >>>> PacketFence-users mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/packetfence-users >>> >> >> >> ------------------------------------------------------------------------------ >> October Webinars: Code for Performance >> Free Intel webinars can help you accelerate application performance. >> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from >> the latest Intel processors and coprocessors. See abstracts and register > >> http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk >> _______________________________________________ >> PacketFence-users mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/packetfence-users > > > ------------------------------------------------------------------------------ > Sponsored by Intel(R) XDK > Develop, test and display web and hybrid apps with a single code base. > Download it for free now! > http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk > _______________________________________________ > PacketFence-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/packetfence-users
smime.p7s
Description: S/MIME cryptographic signature
------------------------------------------------------------------------------ Rapidly troubleshoot problems before they affect your business. Most IT organizations don't have a clear picture of how application performance affects their revenue. With AppDynamics, you get 100% visibility into your Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro! http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________ PacketFence-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/packetfence-users
