Well that is a puzzle, sort of like the crypto outside of the CIA. Just reboot and upgrade if you feel a need.
BUT, if I were you, I would figure it out and hit 1,000, but if it is a production box, then just reboot, time is money..... I see quite a few people with the same issue while googling, old kernels and various distros. One fix was to stop NetworkManager. I don't know about your particular server, but did you set it up? If not, maybe "service NetworkManager stop" may do the trick. Thanks, Steve Totaro On Wed, Nov 19, 2008 at 8:58 PM, Jeff LaCoursiere <[EMAIL PROTECTED]> wrote: > > No... isn't that a GUI? This is a colo'ed server running a prepaid > calling card app. > > Cheers, > > j > > On Wed, 19 Nov 2008, Steve Totaro wrote: > >> Are you using NetworkManager? >> >> On Wed, Nov 19, 2008 at 8:29 PM, Jeff LaCoursiere <[EMAIL PROTECTED]> wrote: >>> >>> Happy for all suggestions, of course! No offense intended with my reply. >>> Not sure what you are trying to get at with init.d, but here you go: >>> >>> ast% ls /etc/init.d >>> /etc/init.d@ >>> >>> Am guessing you expected a bit more than that, so allow me to assume what >>> you are looking for (and sum up the state of the discussion): >>> >>> ast% ls -ltr /etc/init.d/iptables >>> -rwxr-xr-x 1 root root 7135 Nov 11 2004 /etc/init.d/iptables* >>> ast% >>> >>> This is of course the script that loads and unloads the kernel modules >>> associated with iptables, and was run as "iptables stop" on November 3rd >>> which caused the problem in discussion. One of the lines in this script >>> does a "modprobe -r ipt_state" which hung. Apparently the actual module >>> which hung while unloading is ip_conntrack: >>> >>> ast% cat /proc/modules | head >>> ip_conntrack 45573 0 - Unloading 0xf8945000 >>> [snip] >>> >>> Some bug in this module must be in an infinite loop in kernel space, as >>> the process eats 100% of the CPU even when reniced to +19. Because of >>> this the modprobe process cannot be killed and won't respond to >>> interrupts. >>> >>> An attempt at getting a kernel stack trace failed, which is extremely >>> unfortunate :( Cool to learn about /proc/sysrq-trigger, though! >>> >>> Did I miss anything? >>> >>> Cheers, >>> >>> j >>> >>> On Wed, 19 Nov 2008, Steve Totaro wrote: >>> >>>> I was not implying that you upgrade anything but iptables. >>>> >>>> What is the output of "ls /etc/init.d/" >>>> >>>> On Wed, Nov 19, 2008 at 8:02 PM, Jeff LaCoursiere <[EMAIL PROTECTED]> >>>> wrote: >>>>> >>>>> Hi Steve, >>>>> >>>>> [EMAIL PROTECTED] ~]# ls -ltr /etc/init.d >>>>> lrwxrwxrwx 1 root root 11 Nov 29 2007 /etc/init.d -> rc.d/init.d >>>>> [EMAIL PROTECTED] ~]# >>>>> >>>>> Although I agree that updating the kernel et all would be a good idea, the >>>>> whole point is to keep the machine running for 19 more days without the >>>>> rogue process interfering with my voice quality. If I cannot unload the >>>>> module or otherwise interrupt the process which is currently spinning in >>>>> kernel space, no upgrade will be possible. I am quite sure that rebooting >>>>> will fix this problem, but the "puzzle" was to fix it without doing so... >>>>> >>>>> Cheers, >>>>> >>>>> j >>>>> >>>>> On Wed, 19 Nov 2008, Steve Totaro wrote: >>>>> >>>>>> Well then use whatever package manager you have. Apt-get I assume. >>>>>> Maybe that might help. >>>>>> >>>>>> What do you get with "#ls -ltr /etc/init.d"? >>>>>> -- >>>>>> Thanks, >>>>>> Steve Totaro >>>>>> +18887771888 (Toll Free) >>>>>> +12409381212 (Cell) >>>>>> +12024369784 (Skype) >>>>>> >>>>>> On Wed, Nov 19, 2008 at 7:19 PM, Jeff LaCoursiere <[EMAIL PROTECTED]> >>>>>> wrote: >>>>>>> >>>>>>> Its not Centos - there is no 'yum'. >>>>>>> "service iptables stop" is what >>>>>>> produced the hanging process in the first place - I think my big problem >>>>>>> here is that a kernel module is broken, and there is no way to stop it, >>>>>>> and there seems to be no way to unload it (in fact it is hung trying to >>>>>>> do >>>>>>> just that). >>>>>>> >>>>>>> Thanks for the suggestions, though! >>>>>>> >>>>>>> j >>>>>>> >>>>>>> On Wed, 19 Nov 2008, Steve Totaro wrote: >>>>>>> >>>>>>>> YUM update? "service iptables stop" "service iptables start"? >>>>>>>> >>>>>>>> On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere <[EMAIL PROTECTED]> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hmm, I am more of a BSD guy I guess. I would expect a pipe to show a >>>>>>>>> 'p' >>>>>>>>> in a long ls. This is interesting though: >>>>>>>>> >>>>>>>>> [EMAIL PROTECTED] init.d]# cat /proc/modules | head >>>>>>>>> ip_conntrack 45573 0 - Unloading 0xf8945000 >>>>>>>>> [EMAIL PROTECTED] init.d]# rmmod -f ip_conntrack >>>>>>>>> ERROR: Removing 'ip_conntrack': Device or resource busy >>>>>>>>> >>>>>>>>> (sigh) >>>>>>>>> >>>>>>>>> I am pretty sure ip_conntrack is part of the iptables stuff... >>>>>>>>> >>>>>>>>> j >>>>>>>>> >>>>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote: >>>>>>>>> >>>>>>>>>> /proc/modules is a pipe >>>>>>>>>> You can see what is in there by type cat /proc/modules|more >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: [EMAIL PROTECTED] >>>>>>>>>> [mailto:[EMAIL PROTECTED] On Behalf Of Jeff >>>>>>>>>> LaCoursiere >>>>>>>>>> Sent: Wednesday, November 19, 2008 2:47 PM >>>>>>>>>> To: Asterisk Users Mailing List - Non-Commercial Discussion >>>>>>>>>> Subject: Re: [asterisk-users] puzzle >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> A good idea! The modprobe command is actually in the ps below - it >>>>>>>>>> is >>>>>>>>>> part of the /etc/init.d/iptables script, and apparently was trying to >>>>>>>>>> remove the ipt_state module. The result, however: >>>>>>>>>> >>>>>>>>>> [EMAIL PROTECTED] init.d]# rmmod ipt_state >>>>>>>>>> ERROR: Module ipt_state does not exist in /proc/modules >>>>>>>>>> >>>>>>>>>> (sigh). In fact /proc/modules is empty. >>>>>>>>>> >>>>>>>>>> [EMAIL PROTECTED] init.d]# ls -ltr /proc/modules >>>>>>>>>> -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules >>>>>>>>>> >>>>>>>>>> j >>>>>>>>>> >>>>>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote: >>>>>>>>>> >>>>>>>>>>> Your could try this >>>>>>>>>>> History|grep modprobe >>>>>>>>>>> Rmmod XXX where xxx is the parameter from the history|grep modprobe. >>>>>>>>>>> This of course assumes that the command is in your last 1000 >>>>>>>>>>> commands. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: [EMAIL PROTECTED] >>>>>>>>>>> [mailto:[EMAIL PROTECTED] On Behalf Of Jeff >>>>>>>>>>> LaCoursiere >>>>>>>>>>> Sent: Wednesday, November 19, 2008 2:20 PM >>>>>>>>>>> To: Asterisk Users Mailing List - Non-Commercial Discussion >>>>>>>>>>> Subject: Re: [asterisk-users] puzzle >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Yes, the second 'ps' below showed the parent to be '1' (init), >>>>>>>>>>> which means >>>>>>>>>>> its real parent died already. >>>>>>>>>>> >>>>>>>>>>> Any attempt to flush the iptables hangs :( >>>>>>>>>>> >>>>>>>>>>> j >>>>>>>>>>> >>>>>>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote: >>>>>>>>>>> >>>>>>>>>>>> Have you done a ps -elf to see if the process has a parent that is >>>>>>>>>>>> re-launching or preserving it? >>>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: [EMAIL PROTECTED] >>>>>>>>>>>> [mailto:[EMAIL PROTECTED] On Behalf Of Jeff >>>>>>>>>>>> LaCoursiere >>>>>>>>>>>> Sent: Wednesday, November 19, 2008 1:58 PM >>>>>>>>>>>> To: asterisk-users@lists.digium.com >>>>>>>>>>>> Subject: [asterisk-users] puzzle >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Sorry again for the only marginal relation to asterisk, but the >>>>>>>>>>>> issue >>>>>>>>>> does >>>>>>>>>>>> affect the voice performance I am experiencing, so I am soothing >>>>>>>>>>>> my guilt >>>>>>>>>>>> with that. >>>>>>>>>>>> >>>>>>>>>>>> Bet you don't see this every day: >>>>>>>>>>>> >>>>>>>>>>>> ast% uptime >>>>>>>>>>>> 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, >>>>>>>>>>>> 1.01 >>>>>>>>>>>> ast% >>>>>>>>>>>> >>>>>>>>>>>> I *REALLY* want this machine to see 1000 days uptime, if for >>>>>>>>>>>> nothing >>>>>>>>>> other >>>>>>>>>>>> than bragging rights. Its been through mysql and asterisk >>>>>>>>>>>> upgrades, a >>>>>>>>>>>> horrible hacking nightmare that very nearly made me reboot, and >>>>>>>>>>>> several >>>>>>>>>>>> power outages where the batteries lasted JUST long enough to keep >>>>>>>>>>>> her up. >>>>>>>>>>>> >>>>>>>>>>>> After all of this, I find I may have to reboot after all. Because >>>>>>>>>>>> there >>>>>>>>>>>> is a [EMAIL PROTECTED] process running, consuming 100% CPU (note >>>>>>>>>>>> the load average), >>>>>>>>>>>> and I cannot seem to kill it: >>>>>>>>>>>> >>>>>>>>>>>> ast% ps auxw | grep modprobe >>>>>>>>>>>> root 17744 99.9 0.0 2688 412 ? RN Nov03 23223:01 >>>>>>>>>>>> modprobe >>>>>>>>>>>> -r ipt_state >>>>>>>>>>>> ast% ps ealx | grep modprobe | grep -v grep >>>>>>>>>>>> 4 0 17744 1 39 19 2688 412 - RN ? >>>>>>>>>>>> 23223:38 >>>>>>>>>>>> modprobe -r ipt_state >>>>>>>>>>>> ast% sudo kill 17744 >>>>>>>>>>>> ast% sudo kill 17744 >>>>>>>>>>>> ast% sudo kill -9 17744 >>>>>>>>>>>> ast% sudo kill -9 17744 >>>>>>>>>>>> ast% !ps >>>>>>>>>>>> ps ealx | grep modprobe | grep -v grep >>>>>>>>>>>> 4 0 17744 1 39 19 2688 412 - RN ? >>>>>>>>>>>> 23224:41 >>>>>>>>>>>> modprobe -r ipt_state >>>>>>>>>>>> ast% >>>>>>>>>>>> >>>>>>>>>>>> You may also notice that I tried "renice" to bump it all the way >>>>>>>>>>>> to +19 >>>>>>>>>>>> and still it consumes 100% of the CPU. The result for asterisk is >>>>>>>>>>>> that I >>>>>>>>>>>> hear bits of robot noise during conversations, which is annoying >>>>>>>>>>>> as hell >>>>>>>>>>>> but not neccessarily show stopping. But for another 19 days?? >>>>>>>>>>>> Argg! >>>>>>>>>>>> >>>>>>>>>>>> I assume that because it is 'modprobe' it has tickled some kernel >>>>>>>>>>>> bug >>>>>>>>>> that >>>>>>>>>>>> is merrily spinning away and won't respond to interrupts. I even >>>>>>>>>>>> tried >>>>>>>>>> to >>>>>>>>>>>> stop it with gdb and strace, both of which also hung and had to be >>>>>>>>>>>> killed >>>>>>>>>>>> with -9. >>>>>>>>>>>> >>>>>>>>>>>> It seems to be related to me screwing with the iptables a few >>>>>>>>>>>> weeks ago. >>>>>>>>>>>> >>>>>>>>>>>> Any ideas other than rebooting? >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> >>>>>>>>>>>> j >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> asterisk-users mailing list >>>>>>>>>>>> To UNSUBSCRIBE or update options visit: >>>>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> asterisk-users mailing list >>>>>>>>>>>> To UNSUBSCRIBE or update options visit: >>>>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> asterisk-users mailing list >>>>>>>>>>> To UNSUBSCRIBE or update options visit: >>>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> asterisk-users mailing list >>>>>>>>>>> To UNSUBSCRIBE or update options visit: >>>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>>>>>>> >>>>>>>>>> asterisk-users mailing list >>>>>>>>>> To UNSUBSCRIBE or update options visit: >>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>>>>>>> >>>>>>>>>> asterisk-users mailing list >>>>>>>>>> To UNSUBSCRIBE or update options visit: >>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>>>>>> >>>>>>>>> asterisk-users mailing list >>>>>>>>> To UNSUBSCRIBE or update options visit: >>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Thanks, >>>>>>>> Steve Totaro >>>>>>>> +18887771888 (Toll Free) >>>>>>>> +12409381212 (Cell) >>>>>>>> +12024369784 (Skype) >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>>>>> >>>>>>>> asterisk-users mailing list >>>>>>>> To UNSUBSCRIBE or update options visit: >>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>>>> >>>>>>> asterisk-users mailing list >>>>>>> To UNSUBSCRIBE or update options visit: >>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>>> >>>>>> asterisk-users mailing list >>>>>> To UNSUBSCRIBE or update options visit: >>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>> >>>>> >>>>> _______________________________________________ >>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>> >>>>> asterisk-users mailing list >>>>> To UNSUBSCRIBE or update options visit: >>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>> >>>> >>>> >>>> >>>> -- >>>> Thanks, >>>> Steve Totaro >>>> +18887771888 (Toll Free) >>>> +12409381212 (Cell) >>>> +12024369784 (Skype) >>>> >>>> _______________________________________________ >>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>> >>>> asterisk-users mailing list >>>> To UNSUBSCRIBE or update options visit: >>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>> >>> >>> _______________________________________________ >>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>> >>> asterisk-users mailing list >>> To UNSUBSCRIBE or update options visit: >>> http://lists.digium.com/mailman/listinfo/asterisk-users >>> >> >> >> >> -- >> Thanks, >> Steve Totaro >> +18887771888 (Toll Free) >> +12409381212 (Cell) >> +12024369784 (Skype) >> >> _______________________________________________ >> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >> >> asterisk-users mailing list >> To UNSUBSCRIBE or update options visit: >> http://lists.digium.com/mailman/listinfo/asterisk-users >> > > _______________________________________________ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users > -- Thanks, Steve Totaro +18887771888 (Toll Free) +12409381212 (Cell) +12024369784 (Skype) _______________________________________________ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
