Are you using NetworkManager? On Wed, Nov 19, 2008 at 8:29 PM, Jeff LaCoursiere <[EMAIL PROTECTED]> wrote: > > Happy for all suggestions, of course! No offense intended with my reply. > Not sure what you are trying to get at with init.d, but here you go: > > ast% ls /etc/init.d > /etc/init.d@ > > Am guessing you expected a bit more than that, so allow me to assume what > you are looking for (and sum up the state of the discussion): > > ast% ls -ltr /etc/init.d/iptables > -rwxr-xr-x 1 root root 7135 Nov 11 2004 /etc/init.d/iptables* > ast% > > This is of course the script that loads and unloads the kernel modules > associated with iptables, and was run as "iptables stop" on November 3rd > which caused the problem in discussion. One of the lines in this script > does a "modprobe -r ipt_state" which hung. Apparently the actual module > which hung while unloading is ip_conntrack: > > ast% cat /proc/modules | head > ip_conntrack 45573 0 - Unloading 0xf8945000 > [snip] > > Some bug in this module must be in an infinite loop in kernel space, as > the process eats 100% of the CPU even when reniced to +19. Because of > this the modprobe process cannot be killed and won't respond to > interrupts. > > An attempt at getting a kernel stack trace failed, which is extremely > unfortunate :( Cool to learn about /proc/sysrq-trigger, though! > > Did I miss anything? > > Cheers, > > j > > On Wed, 19 Nov 2008, Steve Totaro wrote: > >> I was not implying that you upgrade anything but iptables. >> >> What is the output of "ls /etc/init.d/" >> >> On Wed, Nov 19, 2008 at 8:02 PM, Jeff LaCoursiere <[EMAIL PROTECTED]> wrote: >>> >>> Hi Steve, >>> >>> [EMAIL PROTECTED] ~]# ls -ltr /etc/init.d >>> lrwxrwxrwx 1 root root 11 Nov 29 2007 /etc/init.d -> rc.d/init.d >>> [EMAIL PROTECTED] ~]# >>> >>> Although I agree that updating the kernel et all would be a good idea, the >>> whole point is to keep the machine running for 19 more days without the >>> rogue process interfering with my voice quality. If I cannot unload the >>> module or otherwise interrupt the process which is currently spinning in >>> kernel space, no upgrade will be possible. I am quite sure that rebooting >>> will fix this problem, but the "puzzle" was to fix it without doing so... >>> >>> Cheers, >>> >>> j >>> >>> On Wed, 19 Nov 2008, Steve Totaro wrote: >>> >>>> Well then use whatever package manager you have. Apt-get I assume. >>>> Maybe that might help. >>>> >>>> What do you get with "#ls -ltr /etc/init.d"? >>>> -- >>>> Thanks, >>>> Steve Totaro >>>> +18887771888 (Toll Free) >>>> +12409381212 (Cell) >>>> +12024369784 (Skype) >>>> >>>> On Wed, Nov 19, 2008 at 7:19 PM, Jeff LaCoursiere <[EMAIL PROTECTED]> >>>> wrote: >>>>> >>>>> Its not Centos - there is no 'yum'. >>>>> "service iptables stop" is what >>>>> produced the hanging process in the first place - I think my big problem >>>>> here is that a kernel module is broken, and there is no way to stop it, >>>>> and there seems to be no way to unload it (in fact it is hung trying to do >>>>> just that). >>>>> >>>>> Thanks for the suggestions, though! >>>>> >>>>> j >>>>> >>>>> On Wed, 19 Nov 2008, Steve Totaro wrote: >>>>> >>>>>> YUM update? "service iptables stop" "service iptables start"? >>>>>> >>>>>> On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere <[EMAIL PROTECTED]> >>>>>> wrote: >>>>>>> >>>>>>> Hmm, I am more of a BSD guy I guess. I would expect a pipe to show a >>>>>>> 'p' >>>>>>> in a long ls. This is interesting though: >>>>>>> >>>>>>> [EMAIL PROTECTED] init.d]# cat /proc/modules | head >>>>>>> ip_conntrack 45573 0 - Unloading 0xf8945000 >>>>>>> [EMAIL PROTECTED] init.d]# rmmod -f ip_conntrack >>>>>>> ERROR: Removing 'ip_conntrack': Device or resource busy >>>>>>> >>>>>>> (sigh) >>>>>>> >>>>>>> I am pretty sure ip_conntrack is part of the iptables stuff... >>>>>>> >>>>>>> j >>>>>>> >>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote: >>>>>>> >>>>>>>> /proc/modules is a pipe >>>>>>>> You can see what is in there by type cat /proc/modules|more >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: [EMAIL PROTECTED] >>>>>>>> [mailto:[EMAIL PROTECTED] On Behalf Of Jeff >>>>>>>> LaCoursiere >>>>>>>> Sent: Wednesday, November 19, 2008 2:47 PM >>>>>>>> To: Asterisk Users Mailing List - Non-Commercial Discussion >>>>>>>> Subject: Re: [asterisk-users] puzzle >>>>>>>> >>>>>>>> >>>>>>>> A good idea! The modprobe command is actually in the ps below - it is >>>>>>>> part of the /etc/init.d/iptables script, and apparently was trying to >>>>>>>> remove the ipt_state module. The result, however: >>>>>>>> >>>>>>>> [EMAIL PROTECTED] init.d]# rmmod ipt_state >>>>>>>> ERROR: Module ipt_state does not exist in /proc/modules >>>>>>>> >>>>>>>> (sigh). In fact /proc/modules is empty. >>>>>>>> >>>>>>>> [EMAIL PROTECTED] init.d]# ls -ltr /proc/modules >>>>>>>> -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules >>>>>>>> >>>>>>>> j >>>>>>>> >>>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote: >>>>>>>> >>>>>>>>> Your could try this >>>>>>>>> History|grep modprobe >>>>>>>>> Rmmod XXX where xxx is the parameter from the history|grep modprobe. >>>>>>>>> This of course assumes that the command is in your last 1000 commands. >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: [EMAIL PROTECTED] >>>>>>>>> [mailto:[EMAIL PROTECTED] On Behalf Of Jeff >>>>>>>>> LaCoursiere >>>>>>>>> Sent: Wednesday, November 19, 2008 2:20 PM >>>>>>>>> To: Asterisk Users Mailing List - Non-Commercial Discussion >>>>>>>>> Subject: Re: [asterisk-users] puzzle >>>>>>>>> >>>>>>>>> >>>>>>>>> Yes, the second 'ps' below showed the parent to be '1' (init), which >>>>>>>>> means >>>>>>>>> its real parent died already. >>>>>>>>> >>>>>>>>> Any attempt to flush the iptables hangs :( >>>>>>>>> >>>>>>>>> j >>>>>>>>> >>>>>>>>> On Wed, 19 Nov 2008, Danny Nicholas wrote: >>>>>>>>> >>>>>>>>>> Have you done a ps -elf to see if the process has a parent that is >>>>>>>>>> re-launching or preserving it? >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: [EMAIL PROTECTED] >>>>>>>>>> [mailto:[EMAIL PROTECTED] On Behalf Of Jeff >>>>>>>>>> LaCoursiere >>>>>>>>>> Sent: Wednesday, November 19, 2008 1:58 PM >>>>>>>>>> To: [email protected] >>>>>>>>>> Subject: [asterisk-users] puzzle >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Sorry again for the only marginal relation to asterisk, but the issue >>>>>>>> does >>>>>>>>>> affect the voice performance I am experiencing, so I am soothing my >>>>>>>>>> guilt >>>>>>>>>> with that. >>>>>>>>>> >>>>>>>>>> Bet you don't see this every day: >>>>>>>>>> >>>>>>>>>> ast% uptime >>>>>>>>>> 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, >>>>>>>>>> 1.01 >>>>>>>>>> ast% >>>>>>>>>> >>>>>>>>>> I *REALLY* want this machine to see 1000 days uptime, if for nothing >>>>>>>> other >>>>>>>>>> than bragging rights. Its been through mysql and asterisk upgrades, >>>>>>>>>> a >>>>>>>>>> horrible hacking nightmare that very nearly made me reboot, and >>>>>>>>>> several >>>>>>>>>> power outages where the batteries lasted JUST long enough to keep >>>>>>>>>> her up. >>>>>>>>>> >>>>>>>>>> After all of this, I find I may have to reboot after all. Because >>>>>>>>>> there >>>>>>>>>> is a [EMAIL PROTECTED] process running, consuming 100% CPU (note the >>>>>>>>>> load average), >>>>>>>>>> and I cannot seem to kill it: >>>>>>>>>> >>>>>>>>>> ast% ps auxw | grep modprobe >>>>>>>>>> root 17744 99.9 0.0 2688 412 ? RN Nov03 23223:01 >>>>>>>>>> modprobe >>>>>>>>>> -r ipt_state >>>>>>>>>> ast% ps ealx | grep modprobe | grep -v grep >>>>>>>>>> 4 0 17744 1 39 19 2688 412 - RN ? 23223:38 >>>>>>>>>> modprobe -r ipt_state >>>>>>>>>> ast% sudo kill 17744 >>>>>>>>>> ast% sudo kill 17744 >>>>>>>>>> ast% sudo kill -9 17744 >>>>>>>>>> ast% sudo kill -9 17744 >>>>>>>>>> ast% !ps >>>>>>>>>> ps ealx | grep modprobe | grep -v grep >>>>>>>>>> 4 0 17744 1 39 19 2688 412 - RN ? 23224:41 >>>>>>>>>> modprobe -r ipt_state >>>>>>>>>> ast% >>>>>>>>>> >>>>>>>>>> You may also notice that I tried "renice" to bump it all the way to >>>>>>>>>> +19 >>>>>>>>>> and still it consumes 100% of the CPU. The result for asterisk is >>>>>>>>>> that I >>>>>>>>>> hear bits of robot noise during conversations, which is annoying as >>>>>>>>>> hell >>>>>>>>>> but not neccessarily show stopping. But for another 19 days?? Argg! >>>>>>>>>> >>>>>>>>>> I assume that because it is 'modprobe' it has tickled some kernel bug >>>>>>>> that >>>>>>>>>> is merrily spinning away and won't respond to interrupts. I even >>>>>>>>>> tried >>>>>>>> to >>>>>>>>>> stop it with gdb and strace, both of which also hung and had to be >>>>>>>>>> killed >>>>>>>>>> with -9. >>>>>>>>>> >>>>>>>>>> It seems to be related to me screwing with the iptables a few weeks >>>>>>>>>> ago. >>>>>>>>>> >>>>>>>>>> Any ideas other than rebooting? >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> >>>>>>>>>> j >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>>>>>>> >>>>>>>>>> asterisk-users mailing list >>>>>>>>>> To UNSUBSCRIBE or update options visit: >>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>>>>>>> >>>>>>>>>> asterisk-users mailing list >>>>>>>>>> To UNSUBSCRIBE or update options visit: >>>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>>>>>> >>>>>>>>> asterisk-users mailing list >>>>>>>>> To UNSUBSCRIBE or update options visit: >>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>>>>>> >>>>>>>>> asterisk-users mailing list >>>>>>>>> To UNSUBSCRIBE or update options visit: >>>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>>>>> >>>>>>>> asterisk-users mailing list >>>>>>>> To UNSUBSCRIBE or update options visit: >>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>>>>> >>>>>>>> asterisk-users mailing list >>>>>>>> To UNSUBSCRIBE or update options visit: >>>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>>>> >>>>>>> asterisk-users mailing list >>>>>>> To UNSUBSCRIBE or update options visit: >>>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks, >>>>>> Steve Totaro >>>>>> +18887771888 (Toll Free) >>>>>> +12409381212 (Cell) >>>>>> +12024369784 (Skype) >>>>>> >>>>>> _______________________________________________ >>>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>>> >>>>>> asterisk-users mailing list >>>>>> To UNSUBSCRIBE or update options visit: >>>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>>> >>>>> >>>>> _______________________________________________ >>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>> >>>>> asterisk-users mailing list >>>>> To UNSUBSCRIBE or update options visit: >>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>> >>>> >>>> _______________________________________________ >>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>> >>>> asterisk-users mailing list >>>> To UNSUBSCRIBE or update options visit: >>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>> >>> >>> _______________________________________________ >>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>> >>> asterisk-users mailing list >>> To UNSUBSCRIBE or update options visit: >>> http://lists.digium.com/mailman/listinfo/asterisk-users >>> >> >> >> >> -- >> Thanks, >> Steve Totaro >> +18887771888 (Toll Free) >> +12409381212 (Cell) >> +12024369784 (Skype) >> >> _______________________________________________ >> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >> >> asterisk-users mailing list >> To UNSUBSCRIBE or update options visit: >> http://lists.digium.com/mailman/listinfo/asterisk-users >> > > _______________________________________________ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users >
-- Thanks, Steve Totaro +18887771888 (Toll Free) +12409381212 (Cell) +12024369784 (Skype) _______________________________________________ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
