Re: [asterisk-users] puzzle
Is this a competition about how many levels of quotes the list can handle or something? SCNR. ;-) Steve Totaro schrieb: On Wed, Nov 19, 2008 at 8:58 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: On Wed, 19 Nov 2008, Steve Totaro wrote: On Wed, Nov 19, 2008 at 8:29 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: On Wed, 19 Nov 2008, Steve Totaro wrote: On Wed, Nov 19, 2008 at 8:02 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: On Wed, 19 Nov 2008, Steve Totaro wrote: On Wed, Nov 19, 2008 at 7:19 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: On Wed, 19 Nov 2008, Steve Totaro wrote: On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: On Wed, 19 Nov 2008, Danny Nicholas wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere On Wed, 19 Nov 2008, Danny Nicholas wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere On Wed, 19 Nov 2008, Danny Nicholas wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ Philipp Kempgen -- http://www.das-asterisk-buch.de - http://www.the-asterisk-book.com Amooma GmbH - Bachstr. 126 - 56566 Neuwied - http://www.amooma.de Geschäftsführer: Stefan Wintermeyer, Handelsregister: Neuwied B14998 -- ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] puzzle
Always a self appoited list Nazi. If it bothers you, then don't bother reading. On Thu, Nov 20, 2008 at 7:23 AM, Philipp Kempgen [EMAIL PROTECTED] wrote: Is this a competition about how many levels of quotes the list can handle or something? SCNR. ;-) Steve Totaro schrieb: On Wed, Nov 19, 2008 at 8:58 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: On Wed, 19 Nov 2008, Steve Totaro wrote: On Wed, Nov 19, 2008 at 8:29 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: On Wed, 19 Nov 2008, Steve Totaro wrote: On Wed, Nov 19, 2008 at 8:02 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: On Wed, 19 Nov 2008, Steve Totaro wrote: On Wed, Nov 19, 2008 at 7:19 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: On Wed, 19 Nov 2008, Steve Totaro wrote: On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: On Wed, 19 Nov 2008, Danny Nicholas wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere On Wed, 19 Nov 2008, Danny Nicholas wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere On Wed, 19 Nov 2008, Danny Nicholas wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ Philipp Kempgen -- http://www.das-asterisk-buch.de - http://www.the-asterisk-book.com Amooma GmbH - Bachstr. 126 - 56566 Neuwied - http://www.amooma.de Geschäftsführer: Stefan Wintermeyer, Handelsregister: Neuwied B14998 -- ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users -- Thanks, Steve Totaro +18887771888 (Toll Free) +12409381212 (Cell) +12024369784 (Skype) ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] puzzle
Try flushing all of your iptables and see if that helps. See if there's anything in your dmesg that might indicate what's up. Jeff LaCoursiere wrote: Sorry again for the only marginal relation to asterisk, but the issue does affect the voice performance I am experiencing, so I am soothing my guilt with that. Bet you don't see this every day: ast% uptime 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01 ast% I *REALLY* want this machine to see 1000 days uptime, if for nothing other than bragging rights. Its been through mysql and asterisk upgrades, a horrible hacking nightmare that very nearly made me reboot, and several power outages where the batteries lasted JUST long enough to keep her up. After all of this, I find I may have to reboot after all. Because there is a [EMAIL PROTECTED] process running, consuming 100% CPU (note the load average), and I cannot seem to kill it: ast% ps auxw | grep modprobe root 17744 99.9 0.0 2688 412 ?RN Nov03 23223:01 modprobe -r ipt_state ast% ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23223:38 modprobe -r ipt_state ast% sudo kill 17744 ast% sudo kill 17744 ast% sudo kill -9 17744 ast% sudo kill -9 17744 ast% !ps ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23224:41 modprobe -r ipt_state ast% You may also notice that I tried renice to bump it all the way to +19 and still it consumes 100% of the CPU. The result for asterisk is that I hear bits of robot noise during conversations, which is annoying as hell but not neccessarily show stopping. But for another 19 days?? Argg! I assume that because it is 'modprobe' it has tickled some kernel bug that is merrily spinning away and won't respond to interrupts. I even tried to stop it with gdb and strace, both of which also hung and had to be killed with -9. It seems to be related to me screwing with the iptables a few weeks ago. Any ideas other than rebooting? Cheers, j ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] puzzle
Have you done a ps -elf to see if the process has a parent that is re-launching or preserving it? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 1:58 PM To: asterisk-users@lists.digium.com Subject: [asterisk-users] puzzle Sorry again for the only marginal relation to asterisk, but the issue does affect the voice performance I am experiencing, so I am soothing my guilt with that. Bet you don't see this every day: ast% uptime 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01 ast% I *REALLY* want this machine to see 1000 days uptime, if for nothing other than bragging rights. Its been through mysql and asterisk upgrades, a horrible hacking nightmare that very nearly made me reboot, and several power outages where the batteries lasted JUST long enough to keep her up. After all of this, I find I may have to reboot after all. Because there is a [EMAIL PROTECTED] process running, consuming 100% CPU (note the load average), and I cannot seem to kill it: ast% ps auxw | grep modprobe root 17744 99.9 0.0 2688 412 ?RN Nov03 23223:01 modprobe -r ipt_state ast% ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23223:38 modprobe -r ipt_state ast% sudo kill 17744 ast% sudo kill 17744 ast% sudo kill -9 17744 ast% sudo kill -9 17744 ast% !ps ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23224:41 modprobe -r ipt_state ast% You may also notice that I tried renice to bump it all the way to +19 and still it consumes 100% of the CPU. The result for asterisk is that I hear bits of robot noise during conversations, which is annoying as hell but not neccessarily show stopping. But for another 19 days?? Argg! I assume that because it is 'modprobe' it has tickled some kernel bug that is merrily spinning away and won't respond to interrupts. I even tried to stop it with gdb and strace, both of which also hung and had to be killed with -9. It seems to be related to me screwing with the iptables a few weeks ago. Any ideas other than rebooting? Cheers, j ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] puzzle
Yes, the second 'ps' below showed the parent to be '1' (init), which means its real parent died already. Any attempt to flush the iptables hangs :( j On Wed, 19 Nov 2008, Danny Nicholas wrote: Have you done a ps -elf to see if the process has a parent that is re-launching or preserving it? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 1:58 PM To: asterisk-users@lists.digium.com Subject: [asterisk-users] puzzle Sorry again for the only marginal relation to asterisk, but the issue does affect the voice performance I am experiencing, so I am soothing my guilt with that. Bet you don't see this every day: ast% uptime 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01 ast% I *REALLY* want this machine to see 1000 days uptime, if for nothing other than bragging rights. Its been through mysql and asterisk upgrades, a horrible hacking nightmare that very nearly made me reboot, and several power outages where the batteries lasted JUST long enough to keep her up. After all of this, I find I may have to reboot after all. Because there is a [EMAIL PROTECTED] process running, consuming 100% CPU (note the load average), and I cannot seem to kill it: ast% ps auxw | grep modprobe root 17744 99.9 0.0 2688 412 ?RN Nov03 23223:01 modprobe -r ipt_state ast% ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23223:38 modprobe -r ipt_state ast% sudo kill 17744 ast% sudo kill 17744 ast% sudo kill -9 17744 ast% sudo kill -9 17744 ast% !ps ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23224:41 modprobe -r ipt_state ast% You may also notice that I tried renice to bump it all the way to +19 and still it consumes 100% of the CPU. The result for asterisk is that I hear bits of robot noise during conversations, which is annoying as hell but not neccessarily show stopping. But for another 19 days?? Argg! I assume that because it is 'modprobe' it has tickled some kernel bug that is merrily spinning away and won't respond to interrupts. I even tried to stop it with gdb and strace, both of which also hung and had to be killed with -9. It seems to be related to me screwing with the iptables a few weeks ago. Any ideas other than rebooting? Cheers, j ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] puzzle
Your could try this History|grep modprobe Rmmod XXX where xxx is the parameter from the history|grep modprobe. This of course assumes that the command is in your last 1000 commands. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:20 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle Yes, the second 'ps' below showed the parent to be '1' (init), which means its real parent died already. Any attempt to flush the iptables hangs :( j On Wed, 19 Nov 2008, Danny Nicholas wrote: Have you done a ps -elf to see if the process has a parent that is re-launching or preserving it? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 1:58 PM To: asterisk-users@lists.digium.com Subject: [asterisk-users] puzzle Sorry again for the only marginal relation to asterisk, but the issue does affect the voice performance I am experiencing, so I am soothing my guilt with that. Bet you don't see this every day: ast% uptime 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01 ast% I *REALLY* want this machine to see 1000 days uptime, if for nothing other than bragging rights. Its been through mysql and asterisk upgrades, a horrible hacking nightmare that very nearly made me reboot, and several power outages where the batteries lasted JUST long enough to keep her up. After all of this, I find I may have to reboot after all. Because there is a [EMAIL PROTECTED] process running, consuming 100% CPU (note the load average), and I cannot seem to kill it: ast% ps auxw | grep modprobe root 17744 99.9 0.0 2688 412 ?RN Nov03 23223:01 modprobe -r ipt_state ast% ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23223:38 modprobe -r ipt_state ast% sudo kill 17744 ast% sudo kill 17744 ast% sudo kill -9 17744 ast% sudo kill -9 17744 ast% !ps ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23224:41 modprobe -r ipt_state ast% You may also notice that I tried renice to bump it all the way to +19 and still it consumes 100% of the CPU. The result for asterisk is that I hear bits of robot noise during conversations, which is annoying as hell but not neccessarily show stopping. But for another 19 days?? Argg! I assume that because it is 'modprobe' it has tickled some kernel bug that is merrily spinning away and won't respond to interrupts. I even tried to stop it with gdb and strace, both of which also hung and had to be killed with -9. It seems to be related to me screwing with the iptables a few weeks ago. Any ideas other than rebooting? Cheers, j ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] puzzle
On Wed, Nov 19, 2008 at 07:57:33PM +, Jeff LaCoursiere wrote: Sorry again for the only marginal relation to asterisk, but the issue does affect the voice performance I am experiencing, so I am soothing my guilt with that. Bet you don't see this every day: ast% uptime 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01 ast% I *REALLY* want this machine to see 1000 days uptime, if for nothing other than bragging rights. Its been through mysql and asterisk upgrades, a horrible hacking nightmare that very nearly made me reboot, and several power outages where the batteries lasted JUST long enough to keep her up. After all of this, I find I may have to reboot after all. Because there is a [EMAIL PROTECTED] process running, consuming 100% CPU (note the load average), and I cannot seem to kill it: ast% ps auxw | grep modprobe root 17744 99.9 0.0 2688 412 ?RN Nov03 23223:01 modprobe -r ipt_state modprobe -r is basically rmmod . rmmod and insmod and nowdays mostly wrappers to kernel code. So while an strace of that process might give some more information about it, I believe that the kernel-level backtrace would be more interesting. For that, try either the 'p' or 't' sysrq commands. 'p' gives a stack trace of the current process. 't': of all the processes. You can give a sysrq command either through the console (on x86: alt-sysrq-key) or: echo key /proc/sysrq-trigger The output goes to the kernel logs, e.g. in dmesg . ast% ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23223:38 modprobe -r ipt_state ast% sudo kill 17744 ast% sudo kill 17744 ast% sudo kill -9 17744 ast% sudo kill -9 17744 This will probably apply when the process will leave whatever busy context it is in. ast% !ps ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23224:41 modprobe -r ipt_state ast% You may also notice that I tried renice to bump it all the way to +19 and still it consumes 100% of the CPU. The result for asterisk is that I hear bits of robot noise during conversations, which is annoying as hell but not neccessarily show stopping. But for another 19 days?? Argg! I assume that because it is 'modprobe' it has tickled some kernel bug that is merrily spinning away and won't respond to interrupts. I even tried to stop it with gdb and strace, both of which also hung and had to be killed with -9. It seems to be related to me screwing with the iptables a few weeks ago. Any ideas other than rebooting? BTW: what kernel? What ditsribution? -- Tzafrir Cohen icq#16849755 jabber:[EMAIL PROTECTED] +972-50-7952406 mailto:[EMAIL PROTECTED] http://www.xorcom.com iax:[EMAIL PROTECTED]/tzafrir ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] puzzle
A good idea! The modprobe command is actually in the ps below - it is part of the /etc/init.d/iptables script, and apparently was trying to remove the ipt_state module. The result, however: [EMAIL PROTECTED] init.d]# rmmod ipt_state ERROR: Module ipt_state does not exist in /proc/modules (sigh). In fact /proc/modules is empty. [EMAIL PROTECTED] init.d]# ls -ltr /proc/modules -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules j On Wed, 19 Nov 2008, Danny Nicholas wrote: Your could try this History|grep modprobe Rmmod XXX where xxx is the parameter from the history|grep modprobe. This of course assumes that the command is in your last 1000 commands. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:20 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle Yes, the second 'ps' below showed the parent to be '1' (init), which means its real parent died already. Any attempt to flush the iptables hangs :( j On Wed, 19 Nov 2008, Danny Nicholas wrote: Have you done a ps -elf to see if the process has a parent that is re-launching or preserving it? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 1:58 PM To: asterisk-users@lists.digium.com Subject: [asterisk-users] puzzle Sorry again for the only marginal relation to asterisk, but the issue does affect the voice performance I am experiencing, so I am soothing my guilt with that. Bet you don't see this every day: ast% uptime 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01 ast% I *REALLY* want this machine to see 1000 days uptime, if for nothing other than bragging rights. Its been through mysql and asterisk upgrades, a horrible hacking nightmare that very nearly made me reboot, and several power outages where the batteries lasted JUST long enough to keep her up. After all of this, I find I may have to reboot after all. Because there is a [EMAIL PROTECTED] process running, consuming 100% CPU (note the load average), and I cannot seem to kill it: ast% ps auxw | grep modprobe root 17744 99.9 0.0 2688 412 ?RN Nov03 23223:01 modprobe -r ipt_state ast% ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23223:38 modprobe -r ipt_state ast% sudo kill 17744 ast% sudo kill 17744 ast% sudo kill -9 17744 ast% sudo kill -9 17744 ast% !ps ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23224:41 modprobe -r ipt_state ast% You may also notice that I tried renice to bump it all the way to +19 and still it consumes 100% of the CPU. The result for asterisk is that I hear bits of robot noise during conversations, which is annoying as hell but not neccessarily show stopping. But for another 19 days?? Argg! I assume that because it is 'modprobe' it has tickled some kernel bug that is merrily spinning away and won't respond to interrupts. I even tried to stop it with gdb and strace, both of which also hung and had to be killed with -9. It seems to be related to me screwing with the iptables a few weeks ago. Any ideas other than rebooting? Cheers, j ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] puzzle
On Wed, 19 Nov 2008, Tzafrir Cohen wrote: On Wed, Nov 19, 2008 at 07:57:33PM +, Jeff LaCoursiere wrote: ast% ps auxw | grep modprobe root 17744 99.9 0.0 2688 412 ?RN Nov03 23223:01 modprobe -r ipt_state modprobe -r is basically rmmod . rmmod and insmod and nowdays mostly wrappers to kernel code. So while an strace of that process might give some more information about it, I believe that the kernel-level backtrace would be more interesting. For that, try either the 'p' or 't' sysrq commands. 'p' gives a stack trace of the current process. 't': of all the processes. You can give a sysrq command either through the console (on x86: alt-sysrq-key) or: echo key /proc/sysrq-trigger No access to the console, sadly, so I tried the trigger method: [EMAIL PROTECTED] init.d]# echo p /proc/sysrq-trigger which resulted in a single line in /var/log/messages: Nov 19 14:51:10 ast kernel: SysRq : Show Regs I waited a few minutes, then tried the 't': [EMAIL PROTECTED] init.d]# echo t /proc/sysrq-trigger which seemed to hang, so I killed it about thirty seconds later, and now my /var/log/messages has 20,000 extra lines :):) I grepped for the PID and found this: Nov 19 14:52:40 ast kernel: modprobe R running 2988 17744 1 31140 28078 (NOTLB) The next line started with 'sshd', so I guess there was no trace with this? BTW: what kernel? What ditsribution? Keep in mind it has been running almost 1000 days ;) [EMAIL PROTECTED] init.d]# uname -a Linux ast.jbtelenet.com 2.6.9-22.0.2.ELsmp #1 SMP Thu Jan 5 17:13:01 EST 2006 i686 i686 i386 GNU/Linux I believe it is Redhat 9. Its a colo... Thanks for the interesting debug pointers! j ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] puzzle
On Wed, Nov 19, 2008 at 09:06:47PM +, Jeff LaCoursiere wrote: I grepped for the PID and found this: Nov 19 14:52:40 ast kernel: modprobe R running 2988 17744 1 31140 28078 (NOTLB) The next line started with 'sshd', so I guess there was no trace with this? Right :-( BTW: what kernel? What ditsribution? Keep in mind it has been running almost 1000 days ;) [EMAIL PROTECTED] init.d]# uname -a Linux ast.jbtelenet.com 2.6.9-22.0.2.ELsmp #1 SMP Thu Jan 5 17:13:01 EST 2006 i686 i686 i386 GNU/Linux I believe it is Redhat 9. Its a colo... RHEL 4.2 (or compatible, e.g. Centos 4.2) -- Tzafrir Cohen icq#16849755 jabber:[EMAIL PROTECTED] +972-50-7952406 mailto:[EMAIL PROTECTED] http://www.xorcom.com iax:[EMAIL PROTECTED]/tzafrir ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] puzzle
/proc/modules is a pipe You can see what is in there by type cat /proc/modules|more -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:47 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle A good idea! The modprobe command is actually in the ps below - it is part of the /etc/init.d/iptables script, and apparently was trying to remove the ipt_state module. The result, however: [EMAIL PROTECTED] init.d]# rmmod ipt_state ERROR: Module ipt_state does not exist in /proc/modules (sigh). In fact /proc/modules is empty. [EMAIL PROTECTED] init.d]# ls -ltr /proc/modules -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules j On Wed, 19 Nov 2008, Danny Nicholas wrote: Your could try this History|grep modprobe Rmmod XXX where xxx is the parameter from the history|grep modprobe. This of course assumes that the command is in your last 1000 commands. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:20 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle Yes, the second 'ps' below showed the parent to be '1' (init), which means its real parent died already. Any attempt to flush the iptables hangs :( j On Wed, 19 Nov 2008, Danny Nicholas wrote: Have you done a ps -elf to see if the process has a parent that is re-launching or preserving it? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 1:58 PM To: asterisk-users@lists.digium.com Subject: [asterisk-users] puzzle Sorry again for the only marginal relation to asterisk, but the issue does affect the voice performance I am experiencing, so I am soothing my guilt with that. Bet you don't see this every day: ast% uptime 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01 ast% I *REALLY* want this machine to see 1000 days uptime, if for nothing other than bragging rights. Its been through mysql and asterisk upgrades, a horrible hacking nightmare that very nearly made me reboot, and several power outages where the batteries lasted JUST long enough to keep her up. After all of this, I find I may have to reboot after all. Because there is a [EMAIL PROTECTED] process running, consuming 100% CPU (note the load average), and I cannot seem to kill it: ast% ps auxw | grep modprobe root 17744 99.9 0.0 2688 412 ?RN Nov03 23223:01 modprobe -r ipt_state ast% ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23223:38 modprobe -r ipt_state ast% sudo kill 17744 ast% sudo kill 17744 ast% sudo kill -9 17744 ast% sudo kill -9 17744 ast% !ps ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23224:41 modprobe -r ipt_state ast% You may also notice that I tried renice to bump it all the way to +19 and still it consumes 100% of the CPU. The result for asterisk is that I hear bits of robot noise during conversations, which is annoying as hell but not neccessarily show stopping. But for another 19 days?? Argg! I assume that because it is 'modprobe' it has tickled some kernel bug that is merrily spinning away and won't respond to interrupts. I even tried to stop it with gdb and strace, both of which also hung and had to be killed with -9. It seems to be related to me screwing with the iptables a few weeks ago. Any ideas other than rebooting? Cheers, j ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] puzzle
Hmm, I am more of a BSD guy I guess. I would expect a pipe to show a 'p' in a long ls. This is interesting though: [EMAIL PROTECTED] init.d]# cat /proc/modules | head ip_conntrack 45573 0 - Unloading 0xf8945000 [EMAIL PROTECTED] init.d]# rmmod -f ip_conntrack ERROR: Removing 'ip_conntrack': Device or resource busy (sigh) I am pretty sure ip_conntrack is part of the iptables stuff... j On Wed, 19 Nov 2008, Danny Nicholas wrote: /proc/modules is a pipe You can see what is in there by type cat /proc/modules|more -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:47 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle A good idea! The modprobe command is actually in the ps below - it is part of the /etc/init.d/iptables script, and apparently was trying to remove the ipt_state module. The result, however: [EMAIL PROTECTED] init.d]# rmmod ipt_state ERROR: Module ipt_state does not exist in /proc/modules (sigh). In fact /proc/modules is empty. [EMAIL PROTECTED] init.d]# ls -ltr /proc/modules -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules j On Wed, 19 Nov 2008, Danny Nicholas wrote: Your could try this History|grep modprobe Rmmod XXX where xxx is the parameter from the history|grep modprobe. This of course assumes that the command is in your last 1000 commands. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:20 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle Yes, the second 'ps' below showed the parent to be '1' (init), which means its real parent died already. Any attempt to flush the iptables hangs :( j On Wed, 19 Nov 2008, Danny Nicholas wrote: Have you done a ps -elf to see if the process has a parent that is re-launching or preserving it? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 1:58 PM To: asterisk-users@lists.digium.com Subject: [asterisk-users] puzzle Sorry again for the only marginal relation to asterisk, but the issue does affect the voice performance I am experiencing, so I am soothing my guilt with that. Bet you don't see this every day: ast% uptime 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01 ast% I *REALLY* want this machine to see 1000 days uptime, if for nothing other than bragging rights. Its been through mysql and asterisk upgrades, a horrible hacking nightmare that very nearly made me reboot, and several power outages where the batteries lasted JUST long enough to keep her up. After all of this, I find I may have to reboot after all. Because there is a [EMAIL PROTECTED] process running, consuming 100% CPU (note the load average), and I cannot seem to kill it: ast% ps auxw | grep modprobe root 17744 99.9 0.0 2688 412 ?RN Nov03 23223:01 modprobe -r ipt_state ast% ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23223:38 modprobe -r ipt_state ast% sudo kill 17744 ast% sudo kill 17744 ast% sudo kill -9 17744 ast% sudo kill -9 17744 ast% !ps ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23224:41 modprobe -r ipt_state ast% You may also notice that I tried renice to bump it all the way to +19 and still it consumes 100% of the CPU. The result for asterisk is that I hear bits of robot noise during conversations, which is annoying as hell but not neccessarily show stopping. But for another 19 days?? Argg! I assume that because it is 'modprobe' it has tickled some kernel bug that is merrily spinning away and won't respond to interrupts. I even tried to stop it with gdb and strace, both of which also hung and had to be killed with -9. It seems to be related to me screwing with the iptables a few weeks ago. Any ideas other than rebooting? Cheers, j ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] puzzle
YUM update? service iptables stop service iptables start? On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Hmm, I am more of a BSD guy I guess. I would expect a pipe to show a 'p' in a long ls. This is interesting though: [EMAIL PROTECTED] init.d]# cat /proc/modules | head ip_conntrack 45573 0 - Unloading 0xf8945000 [EMAIL PROTECTED] init.d]# rmmod -f ip_conntrack ERROR: Removing 'ip_conntrack': Device or resource busy (sigh) I am pretty sure ip_conntrack is part of the iptables stuff... j On Wed, 19 Nov 2008, Danny Nicholas wrote: /proc/modules is a pipe You can see what is in there by type cat /proc/modules|more -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:47 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle A good idea! The modprobe command is actually in the ps below - it is part of the /etc/init.d/iptables script, and apparently was trying to remove the ipt_state module. The result, however: [EMAIL PROTECTED] init.d]# rmmod ipt_state ERROR: Module ipt_state does not exist in /proc/modules (sigh). In fact /proc/modules is empty. [EMAIL PROTECTED] init.d]# ls -ltr /proc/modules -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules j On Wed, 19 Nov 2008, Danny Nicholas wrote: Your could try this History|grep modprobe Rmmod XXX where xxx is the parameter from the history|grep modprobe. This of course assumes that the command is in your last 1000 commands. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:20 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle Yes, the second 'ps' below showed the parent to be '1' (init), which means its real parent died already. Any attempt to flush the iptables hangs :( j On Wed, 19 Nov 2008, Danny Nicholas wrote: Have you done a ps -elf to see if the process has a parent that is re-launching or preserving it? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 1:58 PM To: asterisk-users@lists.digium.com Subject: [asterisk-users] puzzle Sorry again for the only marginal relation to asterisk, but the issue does affect the voice performance I am experiencing, so I am soothing my guilt with that. Bet you don't see this every day: ast% uptime 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01 ast% I *REALLY* want this machine to see 1000 days uptime, if for nothing other than bragging rights. Its been through mysql and asterisk upgrades, a horrible hacking nightmare that very nearly made me reboot, and several power outages where the batteries lasted JUST long enough to keep her up. After all of this, I find I may have to reboot after all. Because there is a [EMAIL PROTECTED] process running, consuming 100% CPU (note the load average), and I cannot seem to kill it: ast% ps auxw | grep modprobe root 17744 99.9 0.0 2688 412 ?RN Nov03 23223:01 modprobe -r ipt_state ast% ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23223:38 modprobe -r ipt_state ast% sudo kill 17744 ast% sudo kill 17744 ast% sudo kill -9 17744 ast% sudo kill -9 17744 ast% !ps ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23224:41 modprobe -r ipt_state ast% You may also notice that I tried renice to bump it all the way to +19 and still it consumes 100% of the CPU. The result for asterisk is that I hear bits of robot noise during conversations, which is annoying as hell but not neccessarily show stopping. But for another 19 days?? Argg! I assume that because it is 'modprobe' it has tickled some kernel bug that is merrily spinning away and won't respond to interrupts. I even tried to stop it with gdb and strace, both of which also hung and had to be killed with -9. It seems to be related to me screwing with the iptables a few weeks ago. Any ideas other than rebooting? Cheers, j ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list
Re: [asterisk-users] puzzle
Its not Centos - there is no 'yum'. service iptables stop is what produced the hanging process in the first place - I think my big problem here is that a kernel module is broken, and there is no way to stop it, and there seems to be no way to unload it (in fact it is hung trying to do just that). Thanks for the suggestions, though! j On Wed, 19 Nov 2008, Steve Totaro wrote: YUM update? service iptables stop service iptables start? On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Hmm, I am more of a BSD guy I guess. I would expect a pipe to show a 'p' in a long ls. This is interesting though: [EMAIL PROTECTED] init.d]# cat /proc/modules | head ip_conntrack 45573 0 - Unloading 0xf8945000 [EMAIL PROTECTED] init.d]# rmmod -f ip_conntrack ERROR: Removing 'ip_conntrack': Device or resource busy (sigh) I am pretty sure ip_conntrack is part of the iptables stuff... j On Wed, 19 Nov 2008, Danny Nicholas wrote: /proc/modules is a pipe You can see what is in there by type cat /proc/modules|more -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:47 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle A good idea! The modprobe command is actually in the ps below - it is part of the /etc/init.d/iptables script, and apparently was trying to remove the ipt_state module. The result, however: [EMAIL PROTECTED] init.d]# rmmod ipt_state ERROR: Module ipt_state does not exist in /proc/modules (sigh). In fact /proc/modules is empty. [EMAIL PROTECTED] init.d]# ls -ltr /proc/modules -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules j On Wed, 19 Nov 2008, Danny Nicholas wrote: Your could try this History|grep modprobe Rmmod XXX where xxx is the parameter from the history|grep modprobe. This of course assumes that the command is in your last 1000 commands. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:20 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle Yes, the second 'ps' below showed the parent to be '1' (init), which means its real parent died already. Any attempt to flush the iptables hangs :( j On Wed, 19 Nov 2008, Danny Nicholas wrote: Have you done a ps -elf to see if the process has a parent that is re-launching or preserving it? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 1:58 PM To: asterisk-users@lists.digium.com Subject: [asterisk-users] puzzle Sorry again for the only marginal relation to asterisk, but the issue does affect the voice performance I am experiencing, so I am soothing my guilt with that. Bet you don't see this every day: ast% uptime 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01 ast% I *REALLY* want this machine to see 1000 days uptime, if for nothing other than bragging rights. Its been through mysql and asterisk upgrades, a horrible hacking nightmare that very nearly made me reboot, and several power outages where the batteries lasted JUST long enough to keep her up. After all of this, I find I may have to reboot after all. Because there is a [EMAIL PROTECTED] process running, consuming 100% CPU (note the load average), and I cannot seem to kill it: ast% ps auxw | grep modprobe root 17744 99.9 0.0 2688 412 ?RN Nov03 23223:01 modprobe -r ipt_state ast% ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23223:38 modprobe -r ipt_state ast% sudo kill 17744 ast% sudo kill 17744 ast% sudo kill -9 17744 ast% sudo kill -9 17744 ast% !ps ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23224:41 modprobe -r ipt_state ast% You may also notice that I tried renice to bump it all the way to +19 and still it consumes 100% of the CPU. The result for asterisk is that I hear bits of robot noise during conversations, which is annoying as hell but not neccessarily show stopping. But for another 19 days?? Argg! I assume that because it is 'modprobe' it has tickled some kernel bug that is merrily spinning away and won't respond to interrupts. I even tried to stop it with gdb and strace, both of which also hung and had to be killed with -9. It seems to be related to me screwing with the iptables a few weeks ago. Any ideas other than rebooting? Cheers, j ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] puzzle
Well then use whatever package manager you have. Apt-get I assume. Maybe that might help. What do you get with #ls -ltr /etc/init.d? -- Thanks, Steve Totaro +18887771888 (Toll Free) +12409381212 (Cell) +12024369784 (Skype) On Wed, Nov 19, 2008 at 7:19 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Its not Centos - there is no 'yum'. service iptables stop is what produced the hanging process in the first place - I think my big problem here is that a kernel module is broken, and there is no way to stop it, and there seems to be no way to unload it (in fact it is hung trying to do just that). Thanks for the suggestions, though! j On Wed, 19 Nov 2008, Steve Totaro wrote: YUM update? service iptables stop service iptables start? On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Hmm, I am more of a BSD guy I guess. I would expect a pipe to show a 'p' in a long ls. This is interesting though: [EMAIL PROTECTED] init.d]# cat /proc/modules | head ip_conntrack 45573 0 - Unloading 0xf8945000 [EMAIL PROTECTED] init.d]# rmmod -f ip_conntrack ERROR: Removing 'ip_conntrack': Device or resource busy (sigh) I am pretty sure ip_conntrack is part of the iptables stuff... j On Wed, 19 Nov 2008, Danny Nicholas wrote: /proc/modules is a pipe You can see what is in there by type cat /proc/modules|more -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:47 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle A good idea! The modprobe command is actually in the ps below - it is part of the /etc/init.d/iptables script, and apparently was trying to remove the ipt_state module. The result, however: [EMAIL PROTECTED] init.d]# rmmod ipt_state ERROR: Module ipt_state does not exist in /proc/modules (sigh). In fact /proc/modules is empty. [EMAIL PROTECTED] init.d]# ls -ltr /proc/modules -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules j On Wed, 19 Nov 2008, Danny Nicholas wrote: Your could try this History|grep modprobe Rmmod XXX where xxx is the parameter from the history|grep modprobe. This of course assumes that the command is in your last 1000 commands. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:20 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle Yes, the second 'ps' below showed the parent to be '1' (init), which means its real parent died already. Any attempt to flush the iptables hangs :( j On Wed, 19 Nov 2008, Danny Nicholas wrote: Have you done a ps -elf to see if the process has a parent that is re-launching or preserving it? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 1:58 PM To: asterisk-users@lists.digium.com Subject: [asterisk-users] puzzle Sorry again for the only marginal relation to asterisk, but the issue does affect the voice performance I am experiencing, so I am soothing my guilt with that. Bet you don't see this every day: ast% uptime 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01 ast% I *REALLY* want this machine to see 1000 days uptime, if for nothing other than bragging rights. Its been through mysql and asterisk upgrades, a horrible hacking nightmare that very nearly made me reboot, and several power outages where the batteries lasted JUST long enough to keep her up. After all of this, I find I may have to reboot after all. Because there is a [EMAIL PROTECTED] process running, consuming 100% CPU (note the load average), and I cannot seem to kill it: ast% ps auxw | grep modprobe root 17744 99.9 0.0 2688 412 ?RN Nov03 23223:01 modprobe -r ipt_state ast% ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23223:38 modprobe -r ipt_state ast% sudo kill 17744 ast% sudo kill 17744 ast% sudo kill -9 17744 ast% sudo kill -9 17744 ast% !ps ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23224:41 modprobe -r ipt_state ast% You may also notice that I tried renice to bump it all the way to +19 and still it consumes 100% of the CPU. The result for asterisk is that I hear bits of robot noise during conversations, which is annoying as hell but not neccessarily show stopping. But for another 19 days?? Argg! I assume that because it is 'modprobe' it has tickled some kernel bug that is merrily spinning away and won't respond to interrupts. I even tried to stop it with gdb and strace, both of which also hung and had to be killed with -9. It seems to be related to me screwing with the iptables a few weeks ago. Any ideas
Re: [asterisk-users] puzzle
Hi Steve, [EMAIL PROTECTED] ~]# ls -ltr /etc/init.d lrwxrwxrwx 1 root root 11 Nov 29 2007 /etc/init.d - rc.d/init.d [EMAIL PROTECTED] ~]# Although I agree that updating the kernel et all would be a good idea, the whole point is to keep the machine running for 19 more days without the rogue process interfering with my voice quality. If I cannot unload the module or otherwise interrupt the process which is currently spinning in kernel space, no upgrade will be possible. I am quite sure that rebooting will fix this problem, but the puzzle was to fix it without doing so... Cheers, j On Wed, 19 Nov 2008, Steve Totaro wrote: Well then use whatever package manager you have. Apt-get I assume. Maybe that might help. What do you get with #ls -ltr /etc/init.d? -- Thanks, Steve Totaro +18887771888 (Toll Free) +12409381212 (Cell) +12024369784 (Skype) On Wed, Nov 19, 2008 at 7:19 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Its not Centos - there is no 'yum'. service iptables stop is what produced the hanging process in the first place - I think my big problem here is that a kernel module is broken, and there is no way to stop it, and there seems to be no way to unload it (in fact it is hung trying to do just that). Thanks for the suggestions, though! j On Wed, 19 Nov 2008, Steve Totaro wrote: YUM update? service iptables stop service iptables start? On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Hmm, I am more of a BSD guy I guess. I would expect a pipe to show a 'p' in a long ls. This is interesting though: [EMAIL PROTECTED] init.d]# cat /proc/modules | head ip_conntrack 45573 0 - Unloading 0xf8945000 [EMAIL PROTECTED] init.d]# rmmod -f ip_conntrack ERROR: Removing 'ip_conntrack': Device or resource busy (sigh) I am pretty sure ip_conntrack is part of the iptables stuff... j On Wed, 19 Nov 2008, Danny Nicholas wrote: /proc/modules is a pipe You can see what is in there by type cat /proc/modules|more -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:47 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle A good idea! The modprobe command is actually in the ps below - it is part of the /etc/init.d/iptables script, and apparently was trying to remove the ipt_state module. The result, however: [EMAIL PROTECTED] init.d]# rmmod ipt_state ERROR: Module ipt_state does not exist in /proc/modules (sigh). In fact /proc/modules is empty. [EMAIL PROTECTED] init.d]# ls -ltr /proc/modules -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules j On Wed, 19 Nov 2008, Danny Nicholas wrote: Your could try this History|grep modprobe Rmmod XXX where xxx is the parameter from the history|grep modprobe. This of course assumes that the command is in your last 1000 commands. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:20 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle Yes, the second 'ps' below showed the parent to be '1' (init), which means its real parent died already. Any attempt to flush the iptables hangs :( j On Wed, 19 Nov 2008, Danny Nicholas wrote: Have you done a ps -elf to see if the process has a parent that is re-launching or preserving it? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 1:58 PM To: asterisk-users@lists.digium.com Subject: [asterisk-users] puzzle Sorry again for the only marginal relation to asterisk, but the issue does affect the voice performance I am experiencing, so I am soothing my guilt with that. Bet you don't see this every day: ast% uptime 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01 ast% I *REALLY* want this machine to see 1000 days uptime, if for nothing other than bragging rights. Its been through mysql and asterisk upgrades, a horrible hacking nightmare that very nearly made me reboot, and several power outages where the batteries lasted JUST long enough to keep her up. After all of this, I find I may have to reboot after all. Because there is a [EMAIL PROTECTED] process running, consuming 100% CPU (note the load average), and I cannot seem to kill it: ast% ps auxw | grep modprobe root 17744 99.9 0.0 2688 412 ?RN Nov03 23223:01 modprobe -r ipt_state ast% ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23223:38 modprobe -r ipt_state ast% sudo kill 17744 ast% sudo kill 17744 ast% sudo kill -9 17744 ast% sudo kill -9 17744 ast% !ps ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN
Re: [asterisk-users] puzzle
No. You can't restart the iptables scripts of any distro and expect them to unstick a conntrack module, even if they explicitly reload those modules from the script (as the user himself tried to do and failed) rather than simply installing iptables rules and expecting them to be loaded on demand. Steve Totaro wrote: Well then use whatever package manager you have. Apt-get I assume. Maybe that might help. What do you get with #ls -ltr /etc/init.d? -- Alex Balashov Evariste Systems Web: http://www.evaristesys.com/ Tel: (+1) (678) 954-0670 Direct : (+1) (678) 954-0671 Mobile : (+1) (706) 338-8599 ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] puzzle
I was not implying that you upgrade anything but iptables. What is the output of ls /etc/init.d/ On Wed, Nov 19, 2008 at 8:02 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Hi Steve, [EMAIL PROTECTED] ~]# ls -ltr /etc/init.d lrwxrwxrwx 1 root root 11 Nov 29 2007 /etc/init.d - rc.d/init.d [EMAIL PROTECTED] ~]# Although I agree that updating the kernel et all would be a good idea, the whole point is to keep the machine running for 19 more days without the rogue process interfering with my voice quality. If I cannot unload the module or otherwise interrupt the process which is currently spinning in kernel space, no upgrade will be possible. I am quite sure that rebooting will fix this problem, but the puzzle was to fix it without doing so... Cheers, j On Wed, 19 Nov 2008, Steve Totaro wrote: Well then use whatever package manager you have. Apt-get I assume. Maybe that might help. What do you get with #ls -ltr /etc/init.d? -- Thanks, Steve Totaro +18887771888 (Toll Free) +12409381212 (Cell) +12024369784 (Skype) On Wed, Nov 19, 2008 at 7:19 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Its not Centos - there is no 'yum'. service iptables stop is what produced the hanging process in the first place - I think my big problem here is that a kernel module is broken, and there is no way to stop it, and there seems to be no way to unload it (in fact it is hung trying to do just that). Thanks for the suggestions, though! j On Wed, 19 Nov 2008, Steve Totaro wrote: YUM update? service iptables stop service iptables start? On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Hmm, I am more of a BSD guy I guess. I would expect a pipe to show a 'p' in a long ls. This is interesting though: [EMAIL PROTECTED] init.d]# cat /proc/modules | head ip_conntrack 45573 0 - Unloading 0xf8945000 [EMAIL PROTECTED] init.d]# rmmod -f ip_conntrack ERROR: Removing 'ip_conntrack': Device or resource busy (sigh) I am pretty sure ip_conntrack is part of the iptables stuff... j On Wed, 19 Nov 2008, Danny Nicholas wrote: /proc/modules is a pipe You can see what is in there by type cat /proc/modules|more -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:47 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle A good idea! The modprobe command is actually in the ps below - it is part of the /etc/init.d/iptables script, and apparently was trying to remove the ipt_state module. The result, however: [EMAIL PROTECTED] init.d]# rmmod ipt_state ERROR: Module ipt_state does not exist in /proc/modules (sigh). In fact /proc/modules is empty. [EMAIL PROTECTED] init.d]# ls -ltr /proc/modules -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules j On Wed, 19 Nov 2008, Danny Nicholas wrote: Your could try this History|grep modprobe Rmmod XXX where xxx is the parameter from the history|grep modprobe. This of course assumes that the command is in your last 1000 commands. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:20 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle Yes, the second 'ps' below showed the parent to be '1' (init), which means its real parent died already. Any attempt to flush the iptables hangs :( j On Wed, 19 Nov 2008, Danny Nicholas wrote: Have you done a ps -elf to see if the process has a parent that is re-launching or preserving it? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 1:58 PM To: asterisk-users@lists.digium.com Subject: [asterisk-users] puzzle Sorry again for the only marginal relation to asterisk, but the issue does affect the voice performance I am experiencing, so I am soothing my guilt with that. Bet you don't see this every day: ast% uptime 13:48:08 up 981 days, 18:29, 1 user, load average: 1.08, 1.02, 1.01 ast% I *REALLY* want this machine to see 1000 days uptime, if for nothing other than bragging rights. Its been through mysql and asterisk upgrades, a horrible hacking nightmare that very nearly made me reboot, and several power outages where the batteries lasted JUST long enough to keep her up. After all of this, I find I may have to reboot after all. Because there is a [EMAIL PROTECTED] process running, consuming 100% CPU (note the load average), and I cannot seem to kill it: ast% ps auxw | grep modprobe root 17744 99.9 0.0 2688 412 ?RN Nov03 23223:01 modprobe -r ipt_state ast% ps ealx | grep modprobe | grep -v grep 4 0 17744 1 39 19 2688 412 - RN ?23223:38 modprobe -r ipt_state ast% sudo kill
Re: [asterisk-users] puzzle
Happy for all suggestions, of course! No offense intended with my reply. Not sure what you are trying to get at with init.d, but here you go: ast% ls /etc/init.d /etc/init.d@ Am guessing you expected a bit more than that, so allow me to assume what you are looking for (and sum up the state of the discussion): ast% ls -ltr /etc/init.d/iptables -rwxr-xr-x 1 root root 7135 Nov 11 2004 /etc/init.d/iptables* ast% This is of course the script that loads and unloads the kernel modules associated with iptables, and was run as iptables stop on November 3rd which caused the problem in discussion. One of the lines in this script does a modprobe -r ipt_state which hung. Apparently the actual module which hung while unloading is ip_conntrack: ast% cat /proc/modules | head ip_conntrack 45573 0 - Unloading 0xf8945000 [snip] Some bug in this module must be in an infinite loop in kernel space, as the process eats 100% of the CPU even when reniced to +19. Because of this the modprobe process cannot be killed and won't respond to interrupts. An attempt at getting a kernel stack trace failed, which is extremely unfortunate :( Cool to learn about /proc/sysrq-trigger, though! Did I miss anything? Cheers, j On Wed, 19 Nov 2008, Steve Totaro wrote: I was not implying that you upgrade anything but iptables. What is the output of ls /etc/init.d/ On Wed, Nov 19, 2008 at 8:02 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Hi Steve, [EMAIL PROTECTED] ~]# ls -ltr /etc/init.d lrwxrwxrwx 1 root root 11 Nov 29 2007 /etc/init.d - rc.d/init.d [EMAIL PROTECTED] ~]# Although I agree that updating the kernel et all would be a good idea, the whole point is to keep the machine running for 19 more days without the rogue process interfering with my voice quality. If I cannot unload the module or otherwise interrupt the process which is currently spinning in kernel space, no upgrade will be possible. I am quite sure that rebooting will fix this problem, but the puzzle was to fix it without doing so... Cheers, j On Wed, 19 Nov 2008, Steve Totaro wrote: Well then use whatever package manager you have. Apt-get I assume. Maybe that might help. What do you get with #ls -ltr /etc/init.d? -- Thanks, Steve Totaro +18887771888 (Toll Free) +12409381212 (Cell) +12024369784 (Skype) On Wed, Nov 19, 2008 at 7:19 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Its not Centos - there is no 'yum'. service iptables stop is what produced the hanging process in the first place - I think my big problem here is that a kernel module is broken, and there is no way to stop it, and there seems to be no way to unload it (in fact it is hung trying to do just that). Thanks for the suggestions, though! j On Wed, 19 Nov 2008, Steve Totaro wrote: YUM update? service iptables stop service iptables start? On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Hmm, I am more of a BSD guy I guess. I would expect a pipe to show a 'p' in a long ls. This is interesting though: [EMAIL PROTECTED] init.d]# cat /proc/modules | head ip_conntrack 45573 0 - Unloading 0xf8945000 [EMAIL PROTECTED] init.d]# rmmod -f ip_conntrack ERROR: Removing 'ip_conntrack': Device or resource busy (sigh) I am pretty sure ip_conntrack is part of the iptables stuff... j On Wed, 19 Nov 2008, Danny Nicholas wrote: /proc/modules is a pipe You can see what is in there by type cat /proc/modules|more -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:47 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle A good idea! The modprobe command is actually in the ps below - it is part of the /etc/init.d/iptables script, and apparently was trying to remove the ipt_state module. The result, however: [EMAIL PROTECTED] init.d]# rmmod ipt_state ERROR: Module ipt_state does not exist in /proc/modules (sigh). In fact /proc/modules is empty. [EMAIL PROTECTED] init.d]# ls -ltr /proc/modules -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules j On Wed, 19 Nov 2008, Danny Nicholas wrote: Your could try this History|grep modprobe Rmmod XXX where xxx is the parameter from the history|grep modprobe. This of course assumes that the command is in your last 1000 commands. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:20 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle Yes, the second 'ps' below showed the parent to be '1' (init), which means its real parent died already. Any attempt to flush the iptables hangs :( j On Wed, 19 Nov 2008, Danny Nicholas wrote: Have you done a ps -elf to see if the process has a parent that is re-launching or preserving it? -Original
Re: [asterisk-users] puzzle
Are you using NetworkManager? On Wed, Nov 19, 2008 at 8:29 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Happy for all suggestions, of course! No offense intended with my reply. Not sure what you are trying to get at with init.d, but here you go: ast% ls /etc/init.d /etc/init.d@ Am guessing you expected a bit more than that, so allow me to assume what you are looking for (and sum up the state of the discussion): ast% ls -ltr /etc/init.d/iptables -rwxr-xr-x 1 root root 7135 Nov 11 2004 /etc/init.d/iptables* ast% This is of course the script that loads and unloads the kernel modules associated with iptables, and was run as iptables stop on November 3rd which caused the problem in discussion. One of the lines in this script does a modprobe -r ipt_state which hung. Apparently the actual module which hung while unloading is ip_conntrack: ast% cat /proc/modules | head ip_conntrack 45573 0 - Unloading 0xf8945000 [snip] Some bug in this module must be in an infinite loop in kernel space, as the process eats 100% of the CPU even when reniced to +19. Because of this the modprobe process cannot be killed and won't respond to interrupts. An attempt at getting a kernel stack trace failed, which is extremely unfortunate :( Cool to learn about /proc/sysrq-trigger, though! Did I miss anything? Cheers, j On Wed, 19 Nov 2008, Steve Totaro wrote: I was not implying that you upgrade anything but iptables. What is the output of ls /etc/init.d/ On Wed, Nov 19, 2008 at 8:02 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Hi Steve, [EMAIL PROTECTED] ~]# ls -ltr /etc/init.d lrwxrwxrwx 1 root root 11 Nov 29 2007 /etc/init.d - rc.d/init.d [EMAIL PROTECTED] ~]# Although I agree that updating the kernel et all would be a good idea, the whole point is to keep the machine running for 19 more days without the rogue process interfering with my voice quality. If I cannot unload the module or otherwise interrupt the process which is currently spinning in kernel space, no upgrade will be possible. I am quite sure that rebooting will fix this problem, but the puzzle was to fix it without doing so... Cheers, j On Wed, 19 Nov 2008, Steve Totaro wrote: Well then use whatever package manager you have. Apt-get I assume. Maybe that might help. What do you get with #ls -ltr /etc/init.d? -- Thanks, Steve Totaro +18887771888 (Toll Free) +12409381212 (Cell) +12024369784 (Skype) On Wed, Nov 19, 2008 at 7:19 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Its not Centos - there is no 'yum'. service iptables stop is what produced the hanging process in the first place - I think my big problem here is that a kernel module is broken, and there is no way to stop it, and there seems to be no way to unload it (in fact it is hung trying to do just that). Thanks for the suggestions, though! j On Wed, 19 Nov 2008, Steve Totaro wrote: YUM update? service iptables stop service iptables start? On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Hmm, I am more of a BSD guy I guess. I would expect a pipe to show a 'p' in a long ls. This is interesting though: [EMAIL PROTECTED] init.d]# cat /proc/modules | head ip_conntrack 45573 0 - Unloading 0xf8945000 [EMAIL PROTECTED] init.d]# rmmod -f ip_conntrack ERROR: Removing 'ip_conntrack': Device or resource busy (sigh) I am pretty sure ip_conntrack is part of the iptables stuff... j On Wed, 19 Nov 2008, Danny Nicholas wrote: /proc/modules is a pipe You can see what is in there by type cat /proc/modules|more -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:47 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle A good idea! The modprobe command is actually in the ps below - it is part of the /etc/init.d/iptables script, and apparently was trying to remove the ipt_state module. The result, however: [EMAIL PROTECTED] init.d]# rmmod ipt_state ERROR: Module ipt_state does not exist in /proc/modules (sigh). In fact /proc/modules is empty. [EMAIL PROTECTED] init.d]# ls -ltr /proc/modules -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules j On Wed, 19 Nov 2008, Danny Nicholas wrote: Your could try this History|grep modprobe Rmmod XXX where xxx is the parameter from the history|grep modprobe. This of course assumes that the command is in your last 1000 commands. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:20 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle Yes, the second 'ps' below showed the parent to be '1' (init), which means its real parent died already. Any attempt to flush the iptables hangs :( j On Wed, 19 Nov 2008, Danny
Re: [asterisk-users] puzzle
Jeff LaCoursiere wrote: Did I miss anything? Nope. You're dead on. -- Alex Balashov Evariste Systems Web: http://www.evaristesys.com/ Tel: (+1) (678) 954-0670 Direct : (+1) (678) 954-0671 Mobile : (+1) (706) 338-8599 ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] puzzle
No... isn't that a GUI? This is a colo'ed server running a prepaid calling card app. Cheers, j On Wed, 19 Nov 2008, Steve Totaro wrote: Are you using NetworkManager? On Wed, Nov 19, 2008 at 8:29 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Happy for all suggestions, of course! No offense intended with my reply. Not sure what you are trying to get at with init.d, but here you go: ast% ls /etc/init.d /etc/init.d@ Am guessing you expected a bit more than that, so allow me to assume what you are looking for (and sum up the state of the discussion): ast% ls -ltr /etc/init.d/iptables -rwxr-xr-x 1 root root 7135 Nov 11 2004 /etc/init.d/iptables* ast% This is of course the script that loads and unloads the kernel modules associated with iptables, and was run as iptables stop on November 3rd which caused the problem in discussion. One of the lines in this script does a modprobe -r ipt_state which hung. Apparently the actual module which hung while unloading is ip_conntrack: ast% cat /proc/modules | head ip_conntrack 45573 0 - Unloading 0xf8945000 [snip] Some bug in this module must be in an infinite loop in kernel space, as the process eats 100% of the CPU even when reniced to +19. Because of this the modprobe process cannot be killed and won't respond to interrupts. An attempt at getting a kernel stack trace failed, which is extremely unfortunate :( Cool to learn about /proc/sysrq-trigger, though! Did I miss anything? Cheers, j On Wed, 19 Nov 2008, Steve Totaro wrote: I was not implying that you upgrade anything but iptables. What is the output of ls /etc/init.d/ On Wed, Nov 19, 2008 at 8:02 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Hi Steve, [EMAIL PROTECTED] ~]# ls -ltr /etc/init.d lrwxrwxrwx 1 root root 11 Nov 29 2007 /etc/init.d - rc.d/init.d [EMAIL PROTECTED] ~]# Although I agree that updating the kernel et all would be a good idea, the whole point is to keep the machine running for 19 more days without the rogue process interfering with my voice quality. If I cannot unload the module or otherwise interrupt the process which is currently spinning in kernel space, no upgrade will be possible. I am quite sure that rebooting will fix this problem, but the puzzle was to fix it without doing so... Cheers, j On Wed, 19 Nov 2008, Steve Totaro wrote: Well then use whatever package manager you have. Apt-get I assume. Maybe that might help. What do you get with #ls -ltr /etc/init.d? -- Thanks, Steve Totaro +18887771888 (Toll Free) +12409381212 (Cell) +12024369784 (Skype) On Wed, Nov 19, 2008 at 7:19 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Its not Centos - there is no 'yum'. service iptables stop is what produced the hanging process in the first place - I think my big problem here is that a kernel module is broken, and there is no way to stop it, and there seems to be no way to unload it (in fact it is hung trying to do just that). Thanks for the suggestions, though! j On Wed, 19 Nov 2008, Steve Totaro wrote: YUM update? service iptables stop service iptables start? On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Hmm, I am more of a BSD guy I guess. I would expect a pipe to show a 'p' in a long ls. This is interesting though: [EMAIL PROTECTED] init.d]# cat /proc/modules | head ip_conntrack 45573 0 - Unloading 0xf8945000 [EMAIL PROTECTED] init.d]# rmmod -f ip_conntrack ERROR: Removing 'ip_conntrack': Device or resource busy (sigh) I am pretty sure ip_conntrack is part of the iptables stuff... j On Wed, 19 Nov 2008, Danny Nicholas wrote: /proc/modules is a pipe You can see what is in there by type cat /proc/modules|more -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:47 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle A good idea! The modprobe command is actually in the ps below - it is part of the /etc/init.d/iptables script, and apparently was trying to remove the ipt_state module. The result, however: [EMAIL PROTECTED] init.d]# rmmod ipt_state ERROR: Module ipt_state does not exist in /proc/modules (sigh). In fact /proc/modules is empty. [EMAIL PROTECTED] init.d]# ls -ltr /proc/modules -r--r--r-- 1 root root 0 Nov 19 14:46 /proc/modules j On Wed, 19 Nov 2008, Danny Nicholas wrote: Your could try this History|grep modprobe Rmmod XXX where xxx is the parameter from the history|grep modprobe. This of course assumes that the command is in your last 1000 commands. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:20 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle Yes, the second 'ps' below showed the parent
Re: [asterisk-users] puzzle
Well that is a puzzle, sort of like the crypto outside of the CIA. Just reboot and upgrade if you feel a need. BUT, if I were you, I would figure it out and hit 1,000, but if it is a production box, then just reboot, time is money. I see quite a few people with the same issue while googling, old kernels and various distros. One fix was to stop NetworkManager. I don't know about your particular server, but did you set it up? If not, maybe service NetworkManager stop may do the trick. Thanks, Steve Totaro On Wed, Nov 19, 2008 at 8:58 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: No... isn't that a GUI? This is a colo'ed server running a prepaid calling card app. Cheers, j On Wed, 19 Nov 2008, Steve Totaro wrote: Are you using NetworkManager? On Wed, Nov 19, 2008 at 8:29 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Happy for all suggestions, of course! No offense intended with my reply. Not sure what you are trying to get at with init.d, but here you go: ast% ls /etc/init.d /etc/init.d@ Am guessing you expected a bit more than that, so allow me to assume what you are looking for (and sum up the state of the discussion): ast% ls -ltr /etc/init.d/iptables -rwxr-xr-x 1 root root 7135 Nov 11 2004 /etc/init.d/iptables* ast% This is of course the script that loads and unloads the kernel modules associated with iptables, and was run as iptables stop on November 3rd which caused the problem in discussion. One of the lines in this script does a modprobe -r ipt_state which hung. Apparently the actual module which hung while unloading is ip_conntrack: ast% cat /proc/modules | head ip_conntrack 45573 0 - Unloading 0xf8945000 [snip] Some bug in this module must be in an infinite loop in kernel space, as the process eats 100% of the CPU even when reniced to +19. Because of this the modprobe process cannot be killed and won't respond to interrupts. An attempt at getting a kernel stack trace failed, which is extremely unfortunate :( Cool to learn about /proc/sysrq-trigger, though! Did I miss anything? Cheers, j On Wed, 19 Nov 2008, Steve Totaro wrote: I was not implying that you upgrade anything but iptables. What is the output of ls /etc/init.d/ On Wed, Nov 19, 2008 at 8:02 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Hi Steve, [EMAIL PROTECTED] ~]# ls -ltr /etc/init.d lrwxrwxrwx 1 root root 11 Nov 29 2007 /etc/init.d - rc.d/init.d [EMAIL PROTECTED] ~]# Although I agree that updating the kernel et all would be a good idea, the whole point is to keep the machine running for 19 more days without the rogue process interfering with my voice quality. If I cannot unload the module or otherwise interrupt the process which is currently spinning in kernel space, no upgrade will be possible. I am quite sure that rebooting will fix this problem, but the puzzle was to fix it without doing so... Cheers, j On Wed, 19 Nov 2008, Steve Totaro wrote: Well then use whatever package manager you have. Apt-get I assume. Maybe that might help. What do you get with #ls -ltr /etc/init.d? -- Thanks, Steve Totaro +18887771888 (Toll Free) +12409381212 (Cell) +12024369784 (Skype) On Wed, Nov 19, 2008 at 7:19 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Its not Centos - there is no 'yum'. service iptables stop is what produced the hanging process in the first place - I think my big problem here is that a kernel module is broken, and there is no way to stop it, and there seems to be no way to unload it (in fact it is hung trying to do just that). Thanks for the suggestions, though! j On Wed, 19 Nov 2008, Steve Totaro wrote: YUM update? service iptables stop service iptables start? On Wed, Nov 19, 2008 at 4:46 PM, Jeff LaCoursiere [EMAIL PROTECTED] wrote: Hmm, I am more of a BSD guy I guess. I would expect a pipe to show a 'p' in a long ls. This is interesting though: [EMAIL PROTECTED] init.d]# cat /proc/modules | head ip_conntrack 45573 0 - Unloading 0xf8945000 [EMAIL PROTECTED] init.d]# rmmod -f ip_conntrack ERROR: Removing 'ip_conntrack': Device or resource busy (sigh) I am pretty sure ip_conntrack is part of the iptables stuff... j On Wed, 19 Nov 2008, Danny Nicholas wrote: /proc/modules is a pipe You can see what is in there by type cat /proc/modules|more -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff LaCoursiere Sent: Wednesday, November 19, 2008 2:47 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] puzzle A good idea! The modprobe command is actually in the ps below - it is part of the /etc/init.d/iptables script, and apparently was trying to remove the ipt_state module. The result, however: [EMAIL PROTECTED] init.d]# rmmod ipt_state ERROR: Module ipt_state does not exist in /proc/modules (sigh). In fact /proc/modules is empty. [EMAIL PROTECTED] init.d]# ls -ltr /proc
