Hello all, oook , it looks interesting .. :) try that: ps -eo pcpu,pid,user,args | sort -k 1 -r | grep bear
in short, the bellow example is dual Xeon box with apache qmail mysql kannel with postgresql based sqlbox model name : Pentium III (Coppermine) stepping : 3 cpu MHz : 797.515 cache size : 256 KB s...@bubu:[Sat Mar 27 01:32:14]:[~]$ ps -eo pcpu,pid,user,args | sort -k 1 -r | grep bear 0.0 10391 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 10390 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4654 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4653 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4613 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4612 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4570 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4569 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4568 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4567 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4566 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4563 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4562 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4561 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4560 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4559 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4558 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4557 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4556 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4554 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf 0.0 4552 root /opt/kannel-mblox/sbin/bearerbox -d -P -p /var/run/mblox-bearerbox.pid /opt/kannel/etc/4636/wavecom.ttyS0.smskannel.conf if you run pool of modems, may be this is the reason for the CPU usage, writing to all the devices and waiting for the answer. try this one: http://www.daveltd.com/src/util/ttyload/ and check are there any peaks of there are any, we have something to work on .. in general the idea is to tail -f logs | grep+count for a suspicious events while watching the ttyload at least I would do that cheers On 03/27/2010 12:12 AM, Alvaro Cornejo wrote: > Nikos > > "Thanks God" it was "just" my laptop motherboard that died, not the server. > > In this moment I have no traffic and bearerbox still at 6-8% > > [r...@srvcom1 ~]# top > > top - 15:50:54 up 26 days, 15:23, 1 user, load average: 0.08, 0.27, 0.24 > Tasks: 218 total, 1 running, 204 sleeping, 0 stopped, 13 zombie > Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st > Mem: 2074712k total, 1989044k used, 85668k free, 42872k buffers > Swap: 2031608k total, 344k used, 2031264k free, 859020k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 2337 root 20 0 420m 13m 2448 S 7 0.6 0:25.80 bearerbox > 2716 root 20 0 2332 1088 800 R 0 0.1 0:00.07 top > 1 root 20 0 2044 652 560 S 0 0.0 0:10.44 init > 2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd > 3 root RT -5 0 0 0 S 0 0.0 0:00.95 migration/0 > 4 root 15 -5 0 0 0 S 0 0.0 0:02.18 ksoftirqd/0 > > > [r...@srvcom1 ~]# mpstat > Linux 2.6.22.14-72.fc6 (srvcom1.smsglobal.com.mx) 03/26/2010 > > 03:55:49 PM CPU %user %nice %sys %iowait %irq %soft > %steal %idle intr/s > 03:55:49 PM all 2.37 0.00 0.84 0.67 0.02 0.03 > 0.00 96.07 37.07 > > > [r...@srvcom1 ~]# iostat 1 10 > Linux 2.6.22.14-72.fc6 (srvcom1.smsglobal.com.mx) 03/26/2010 > > avg-cpu: %user %nice %system %iowait %steal %idle > 2.37 0.00 0.89 0.67 0.00 96.07 > > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > sda 6.33 79.19 176.27 182289095 405742931 > dm-0 23.24 79.19 176.26 182280322 405722192 > dm-1 0.00 0.00 0.01 1632 19936 > > avg-cpu: %user %nice %system %iowait %steal %idle > 11.00 0.00 9.50 0.00 0.00 79.50 > > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > sda 0.00 0.00 0.00 0 0 > dm-0 0.00 0.00 0.00 0 0 > dm-1 0.00 0.00 0.00 0 0 > > avg-cpu: %user %nice %system %iowait %steal %idle > 14.29 0.00 11.28 0.00 0.00 74.44 > > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > sda 0.00 0.00 0.00 0 0 > dm-0 0.00 0.00 0.00 0 0 > dm-1 0.00 0.00 0.00 0 0 > > avg-cpu: %user %nice %system %iowait %steal %idle > 12.47 0.00 12.47 0.00 0.00 75.06 > > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > sda 0.00 0.00 0.00 0 0 > dm-0 0.00 0.00 0.00 0 0 > dm-1 0.00 0.00 0.00 0 0 > > avg-cpu: %user %nice %system %iowait %steal %idle > 11.97 0.00 11.97 3.24 0.00 72.82 > > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > sda 21.00 0.00 496.00 0 496 > dm-0 62.00 0.00 496.00 0 496 > dm-1 0.00 0.00 0.00 0 0 > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.00 0.00 0.00 0.00 0.00 100.00 > > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > sda 0.00 0.00 0.00 0 0 > dm-0 0.00 0.00 0.00 0 0 > dm-1 0.00 0.00 0.00 0 0 > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.00 0.00 0.00 0.00 0.00 100.00 > > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > sda 0.00 0.00 0.00 0 0 > dm-0 0.00 0.00 0.00 0 0 > dm-1 0.00 0.00 0.00 0 0 > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.00 0.00 0.00 0.00 0.00 100.00 > > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > sda 0.00 0.00 0.00 0 0 > dm-0 0.00 0.00 0.00 0 0 > dm-1 0.00 0.00 0.00 0 0 > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.50 0.00 0.50 0.00 0.00 99.00 > > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > sda 0.00 0.00 0.00 0 0 > dm-0 0.00 0.00 0.00 0 0 > dm-1 0.00 0.00 0.00 0 0 > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.00 0.00 0.25 4.00 0.00 95.75 > > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > sda 131.00 0.00 1680.00 0 1680 > dm-0 210.00 0.00 1680.00 0 1680 > dm-1 0.00 0.00 0.00 0 0 > > [r...@srvcom1 ~]# > > > About debugging... have no idea what that looks like ;=(( ... should I > eat it with a salad? :=) > > > I do use the CNMI to receive MO in kannel but sometimes it fails and > messages start to clog the modems so I do also use sim-buffering in > order to keep modems free of messages. Sim-buffering looks for the > messages once per minute. I do have only about 20 modems. > > [r...@srvcom1 ~]# tail -f /var/log/kannel/com1_t1.log > 2010-03-26 16:05:54 [2337] [7] DEBUG: AT2[t1]: <-- OK > 2010-03-26 16:05:58 [2337] [7] DEBUG: AT2[t1]: --> AT+CSQ^M > 2010-03-26 16:05:58 [2337] [7] DEBUG: AT2[t1]: <-- +CSQ: 26,0 > 2010-03-26 16:05:58 [2337] [7] DEBUG: AT2[t1]: <-- OK > 2010-03-26 16:06:56 [2337] [7] DEBUG: AT2[t1]: --> AT+CPMS?^M > 2010-03-26 16:06:56 [2337] [7] DEBUG: AT2[t1]: <-- +CPMS: > "SM",0,20,"SM",0,20,"SM",0,20 > 2010-03-26 16:06:56 [2337] [7] DEBUG: AT2[tl1]: <-- OK > 2010-03-26 16:07:00 [2337] [7] DEBUG: AT2[t1]: --> AT+CSQ"^M > 2010-03-26 16:07:00 [2337] [7] DEBUG: AT2[t1]: <-- +CSQ: 26,0 > 2010-03-26 16:07:00 [2337] [7] DEBUG: AT2[t1]: <-- OK > 2010-03-26 16:07:58 [2337] [7] DEBUG: AT2[t1]: --> AT+CPMS?^M > 2010-03-26 16:07:58 [2337] [7] DEBUG: AT2[t1]: <-- +CPMS: > "SM",0,20,"SM",0,20,"SM",0,20 > 2010-03-26 16:07:58 [2337] [7] DEBUG: AT2[t1]: <-- OK > 2010-03-26 16:08:02 [2337] [7] DEBUG: AT2[t1]: --> AT+CSQ"^M > 2010-03-26 16:08:02 [2337] [7] DEBUG: AT2[t1]: <-- +CSQ: 26,0 > 2010-03-26 16:08:02 [2337] [7] DEBUG: AT2[t1]: <-- OK > 2010-03-26 16:09:00 [2337] [7] DEBUG: AT2[t1]: --> AT+CPMS?^M > 2010-03-26 16:09:00 [2337] [7] DEBUG: AT2[t1]: <-- +CPMS: > "SM",0,20,"SM",0,20,"SM",0,20 > 2010-03-26 16:09:00 [2337] [7] DEBUG: AT2[t1]: <-- OK > > > Regards > > Alvaro > > > > > |-----------------------------------------------------------------------------------------------------------------| > Envíe y Reciba Datos y mensajes de Texto (SMS) hacia y desde cualquier > celular y Nextel > en el Perú, México y en mas de 180 paises. Use aplicaciones 2 vias via > SMS y GPRS online > Visitenos en www.perusms.NET www.smsglobal.com.mx y > www.pravcom.com > > > > 2010/3/26 Nikos Balkanas <[email protected]>: >> Pretty descent system. I hope it isn't the system with the fried motherboard >> :-) >> >> Can you post again the output of top, or better mpstat or sar? Seems someone >> deleted this from the thread. >> >> 8% is reasonably low under traffic. However, when there is none, it is a >> cause for concern. I presume you have maximum level logs and nothing shows >> there either. Something strange is happening with your setup. In my case, >> starting up bearerbox without *any* traffic is @0% CPU as expected. Try that >> on another server with no traffic. >> >> You should run a process debugger, truss in Solaris, to see what system >> calls it makes during that 8%. Maybe the periodic polling of your modem >> bank? Are you using SIM buffering? >> >> BR, >> Nikos >> ----- Original Message ----- From: "Alvaro Cornejo" >> <[email protected]> >> To: "Nikos Balkanas" <[email protected]> >> Cc: <[email protected]>; "users" <[email protected]> >> Sent: Friday, March 26, 2010 11:11 PM >> Subject: Re: bearebox eating cpu time >> >> >> Hi Nikkos >> >> I do have a dual cpu dual core xeon with 4GB Ram >> >> [r...@srvcom1 ~]# dmesg | grep -i "hz" >> Detected 1596.575 MHz processor. >> hpet0: 3 64-bit timers, 14318180 Hz >> CPU0: Intel(R) Xeon(R) CPU 5110 @ 1.60GHz stepping 06 >> CPU1: Intel(R) Xeon(R) CPU 5110 @ 1.60GHz stepping 06 >> CPU2: Intel(R) Xeon(R) CPU 5110 @ 1.60GHz stepping 06 >> CPU3: Intel(R) Xeon(R) CPU 5110 @ 1.60GHz stepping 06 >> ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx >> eth0: Broadcom NetXtreme II BCM5708 1000Base-T (B1) PCI-X 64-bit 133MHz >> found at mem f8000000, IRQ 16, node addr 0015c5e7e0a1 >> eth1: Broadcom NetXtreme II BCM5708 1000Base-T (B1) PCI-X 64-bit 133MHz >> found at mem f4000000, IRQ 16, node addr 0015c5e7e0a3 >> >> >> Maybe I expressed the issue wrong. >> >> By eating cpu time I mean that even if there is no traffic MO/MT >> bearerbox process uses the same amount of cpu time. >> >> Regards >> >> Alvaro >> >> |-----------------------------------------------------------------------------------------------------------------| >> EnvΓe y Reciba Datos y mensajes de Texto (SMS) hacia y desde cualquier >> celular y Nextel >> en el PerΓΊ, MΓ©xico y en mas de 180 paises. Use aplicaciones 2 vias via >> SMS y GPRS online >> Visitenos en www.perusms.NET www.smsglobal.com.mx y >> www.pravcom.com >> >> >> >> 2010/3/26 Nikos Balkanas <[email protected]>: >>> >>> Well, if you had a store filesystem problem, you would get waiting I/O CPU >>> in your top, which you don't. You can verify with: >>> >>> iostat 1 10 >>> >>> Anyway, why do you say that 8% CPU is excessive? It seems normal to me. >>> How >>> much did it use to be? What is your CPU hardware? >>> >>> BR, >>> Nikos >>> ----- Original Message ----- From: "Alvaro Cornejo" >>> <[email protected]> >>> To: <[email protected]> >>> Cc: "users" <[email protected]> >>> Sent: Friday, March 26, 2010 4:43 AM >>> Subject: Re: bearebox eating cpu time >>> >>> >>> I monitor kannel queue in order to manage the amount of kannel queued >>> messages and be able to loadbalance the messages from different >>> customers. Therefore I never have more that 1000 messages in kannel >>> queue. Usually arround 400. I do have a pool of modems so that is >>> aprox a 5 min queue. >>> >>> Therefore I don't think the problem comes from there. >>> >>> Any other idea? >>> >>> Thanks >>> >>> Alvaro >>> >>> >>> >>> |-----------------------------------------------------------------------------------------------------------------| >>> EnvΞ½e y Reciba Datos y mensajes de Texto (SMS) hacia y desde cualquier >>> celular y Nextel >>> en el PerΟ , MΞΉxico y en mas de 180 paises. Use aplicaciones 2 vias via >>> SMS y GPRS online >>> Β Β Β Β Β Β Visitenos en www.perusms.NET www.smsglobal.com.mx y >>> www.pravcom.com >>> >>> >>> >>> On Thu, Mar 25, 2010 at 9:57 PM, Β <[email protected]> wrote: >>>> >>>> You get the dubious benefit of losing your store if the server crashes ;) >>>> >>>> However, if you're using ext3 for the store and it grows too big, this is >>>> known to cause io problems which could result into high load. Using a >>>> separate ext2 partition for the store is actually a very good idea, >>>> though >>>> I'd check the rest of the setup for other possible bottlenecks as well. >>>> >>>> Regards, >>>> >>>> Alex >>>> >>>> BlackBerry de movistar, allΞ½ donde estΞΉs estΞ± tu oficin@ >>>> >>>> ________________________________ >>>> From: Alvaro Cornejo <[email protected]> >>>> Date: Thu, 25 Mar 2010 21:37:39 -0400 >>>> To: Benaiad<[email protected]> >>>> Cc: users<[email protected]> >>>> Subject: Re: bearebox eating cpu time >>>> Hi Benaiad >>>> >>>> I don't get it. Why should I use ramdisk for MT? >>>> >>>> Do you mean move kannel to run from the ramdisk? What advantage might I >>>> get >>>> if the rest of the information (mysql) and my app is still running from >>>> HD? >>>> >>>> Regards >>>> >>>> Alvaro >>>> >>>> >>>> >>>> >>>> |-----------------------------------------------------------------------------------------------------------------| >>>> EnvΞ½e y Reciba Datos y mensajes de Texto (SMS) hacia y desde cualquier >>>> celular y Nextel >>>> en el PerΟ , MΞΉxico y en mas de 180 paises. Use aplicaciones 2 vias via >>>> SMS >>>> y >>>> GPRS online >>>> Visitenos en www.perusms.NET www.smsglobal.com.mx y >>>> www.pravcom.com >>>> >>>> >>>> On Thu, Mar 25, 2010 at 9:05 PM, Benaiad <[email protected]> wrote: >>>>> >>>>> Hi Alvaro, >>>>> I suggest you to try ramdisk for MT files. >>>>> Regards. >>>>> -- >>>>> Benaiad >>>> >>> >>> >> >> > >
