I had a question about memory last year. I think Scott answered it for me. When
looking at MRTG on a system in a stellite office, I would think the system was
consuming memory over the first few days after a reboot. I assumed since the
memory was not freed up, it was a problem, but this was not so.
top - 06:11:43 up 138 days, 16:45, 2 users, load average: 0.00, 0.01, 0.00
Tasks: 112 total, 1 running, 111 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.2%sy, 0.0%ni, 99.7%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2075652k total, 1969664k used, 105988k free, 159432k buffers
Swap: 2031608k total, 1088k used, 2030520k free, 1407508k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7428 root 15 0 9848 2796 2264 S 0 0.1 0:00.06 sshd
7452 root 15 0 2188 1020 808 R 0 0.0 0:00.32 top
1 root 15 0 2064 588 508 S 0 0.0 0:00.69 init
2 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
4 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/1
6 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
7 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
8 root 10 -5 0 0 0 S 0 0.0 0:00.32 events/0
9 root 10 -5 0 0 0 S 0 0.0 0:00.28 events/1
10 root 10 -5 0 0 0 S 0 0.0 0:00.00 khelper
11 root 10 -5 0 0 0 S 0 0.0 0:00.00 kthread
15 root 10 -5 0 0 0 S 0 0.0 0:00.95 kblockd/0
16 root 10 -5 0 0 0 S 0 0.0 0:02.02 kblockd/1
17 root 16 -5 0 0 0 S 0 0.0 0:00.00 kacpid
108 root 15 -5 0 0 0 S 0 0.0 0:00.00 cqueue/0
I might suggest that either the system doesn't have enough RAM for the load on
the media server for the number of users or RAM is simply assumed to be the
issue here. What is the equivalent version of sipx to the scs version they are
using?
>>> "Kathleen Eccles" <[email protected]> 01/23/09 11:50 PM >>>
-------- Forwarded Message --------
From: K Eccles (JIRA) <[email protected]>
Reply-To: [email protected]
To: [email protected]
Subject: [SFtrack] Updated: (XECS-2108) System becomes unresponsive over
time, requires reboot
Date: Fri, 23 Jan 2009 22:46:17 -0600 (CST)
[
http://track.sipfoundry.org/browse/XECS-2108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
K Eccles updated XECS-2108:
---------------------------
Attachment: svnDiffxecs2108Jan23.p0
This problem still exists, no better, no worse. The repeated CANCELs are not
retries and they are not caused by our transaction timers. Instead, the
incoming "100 Trying" for the CANCEL request is processed as it it were a
provisional response for the matching INVITE. In this case, the INVITE has
been cancelled. The stack assumes it was waiting for a provisional response
before it could send the CANCEL, so the Cancel is (re)sent now.
Th attached patch avoids (re)sending the CANCEL when we get this odd
provisional response by checking for the existence of a cancel message in the
transaction . This cancel message pointer does not get populated until just
before the message is actually sent. This check would not impact any normal
request retries or transaction timers.
> System becomes unresponsive over time, requires reboot
> ------------------------------------------------------
>
> Key: XECS-2108
> URL: http://track.sipfoundry.org/browse/XECS-2108
> Project: sipXecs
> Issue Type: Bug
> Affects Versions: 3.10.1
> Environment: SCS 2.0
> Reporter: Hesham Abdelrahman
> Assignee: Scott Lawrence
> Priority: Blocker
> Attachments: extract.tar.gz, freememory.JPG,
> sipx-configuration.tar.gz, snapshot.bmp, svnDiffxecs2108Jan23.p0,
> topcommand.doc
>
>
> Custmer has to reboot system everyday, loss of memory. would moving to 2.1
> fix this issue, if this is correct what cleanup we need to do
> attached 3 files, i have 2 more files a snap shot with debug set to 5, it is
> too big but you can download it from the cr.
> login as: root
> [email protected]'s password:
> Last login: Tue Jan 13 16:55:14 2009 from moorepc.dss.pueblo.cnty
> [r...@scs500fsc ~]# top top - 11:20:16 up 23:03, 1 user, load average:
> 0.00, 0.00, 0.00
> Tasks: 127 total, 1 running, 126 sleeping, 0 stopped, 0 zombie
> Cpu(s): 0.7%us, 0.0%sy, 0.0%ni, 99.2%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 2031980k total, 1965180k used, 66800k free, 128988k buffers
> Swap: 4194296k total, 0k used, 4194296k free, 1101004k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 5230 sipxchan 18 0 817m 221m 27m S 0 11.2 1:19.89 java
> 4822 root 15 0 190m 189m 22m S 0 9.6 0:00.74 sipconf
> 5233 root 23 0 147m 146m 24m S 0 7.4 2:53.06 sipXvxml
> 5002 sipxchan 18 0 79424 34m 4008 S 0 1.7 5:14.00 sipXproxy
> 4652 root 15 0 40556 27m 4008 S 0 1.4 0:36.48
> yum-updatesd
> 5000 sipxchan 18 0 691m 24m 9988 S 0 1.2 0:00.78 java
> 2486 mysql 15 0 134m 19m 4368 S 0 1.0 0:03.38 mysqld
> 4561 sipxchan 18 0 36256 18m 5244 S 0 0.9 0:05.64 watchdog
> 5240 sipxchan 15 0 29488 15m 2576 S 0 0.8 0:00.89
> sipxconfig-agen
> 5234 sipxchan 15 0 28600 14m 2872 S 0 0.7 0:20.81 ruby
> 5243 sipxchan 18 0 56692 11m 7852 S 0 0.6 1:00.32
> sipregistrar
> 4820 sipxchan 18 0 46252 11m 8300 S 0 0.6 0:52.78 sipstatus
> 6340 postgres 15 0 21820 10m 9.8m S 0 0.6 0:02.78 postmaster
> 5208 postgres 16 0 21652 9068 8168 S 0 0.4 0:04.81 postmaster
> 2557 postgres 15 0 21028 8796 8380 S 0 0.4 0:00.05 postmaster
> 31689 postgres 15 0 21572 8124 7264 S 0 0.4 0:00.21 postmaster
> 4999 sipxchan 18 0 77308 7704 4796 S 0 0.4 0:05.52 sipxpark
> 4765 root 24 0 61400 7096 4736 S 0 0.3 0:00.11 confbridge
> 13278 postgres 15 0 22532 5620 4136 S 0 0.3 0:00.53 postmaster
> 2294 root 15 0 25920 5220 3096 S 0 0.3 0:00.54 snmpd
> 6428 postgres 15 0 22480 5192 3648 S 0 0.3 0:00.97 postmaster
> 5232 sipxchan 18 0 38828 4744 3884 S 0 0.2 0:00.03 sipxrls
> 13276 postgres 15 0 21724 4488 3320 S 0 0.2 0:00.75 postmaster
> 9796 postgres 15 0 21720 4448 3280 S 0 0.2 0:00.32 postmaster
> 7862 postgres 15 0 21720 4428 3260 S 0 0.2 0:00.36 postmaster
> 4821 sipxchan 18 0 56148 4384 3332 S 0 0.2 0:00.15
> sipxpresence
> 7861 postgres 15 0 21736 4376 3200 S 0 0.2 0:00.21 postmaster
> 7860 postgres 15 0 21720 4352 3196 S 0 0.2 0:00.09 postmaster
> 2366 ntp 15 0 4316 4316 3312 S 0 0.2 0:00.00 ntpd
> 6429 postgres 15 0 21704 4232 3148 S 0 0.2 0:00.37 postmaster
> 11426 postgres 15 0 21704 4052 2976 S 0 0.2 0:00.48 postmaster
> 11427 postgres 15 0 21704 4028 2952 S 0 0.2 0:00.25 postmaster
> 9795 postgres 15 0 21704 4024 2952 S 0 0.2 0:00.08 postmaster
> 6430 postgres 15 0 21696 4000 2940 S 0 0.2 0:00.06 postmaster
> 2006 root 14 -3 10080 3924 2292 S 0 0.2 0:00.21 python
> 5231 sipxchan 18 0 15196 3920 3344 S 0 0.2 0:00.01 sipxacd
> 4603 root 18 0 10056 3880 2476 S 0 0.2 0:00.20 httpd
> 6308 sipxchan 18 0 10316 3560 1996 S 0 0.2 0:00.03 httpd
> 6194 sipxchan 15 0 10188 3556 1996 S 0 0.2 0:00.05 httpd
> 6172 sipxchan 17 0 10204 3552 1996 S 0 0.2 0:00.08 httpd
> 6324 sipxchan 15 0 10316 3536 1996 S 0 0.2 0:00.01 httpd
> 6368 sipxchan 17 0 10204 3532 1996 S 0 0.2 0:00.02 httpd
> 4667 haldaemo 18 0 5384 3524 1600 S 0 0.2 0:00.41 hald
> 6338 sipxchan 18 0 10204 3488 1932 S 0 0.2 0:00.03 httpd
> 9797 postgres 20 0 21568 3172 2324 S 0 0.2 0:00.01 postmaster
> 2587 root 18 0 9296 3092 1720 S 0 0.2 0:00.91 sendmail
> 2093 named 25 0 48084 2928 1912 S 0 0.1 0:00.07 named
> 2553 postgres 18 0 20892 2836 2448 S 0 0.1 0:00.11 postmaster
> 6482 root 15 0 8996 2724 2208 S 0 0.1 0:00.02 sshd
> 11425 postgres 15 0 21572 2700 1900 S 0 0.1 0:00.00 postmaster
> 13277 postgres 15 0 21572 2700 1900 S 0 0.1 0:00.00 postmaster
> 4691 root 18 0 15856 2520 1996 S 0 0.1 3:34.04
>
> *** NOTES 2009-01-14 16:43:47 Eastern Daylight Time brigitba
> The SCS is just about completely down now, and outbound calls cannot be made,
> and we cant receive inbound calls. Will reboot but we took the top command
> capture
> top - 14:39:42 up 1 day, 2:22, 1 user, load average: 0.68, 0.69, 1.55
> Tasks: 126 total, 1 running, 125 sleeping, 0 stopped, 0 zombie
> Cpu(s): 0.7%us, 0.2%sy, 0.0%ni, 98.2%id, 1.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 2031980k total, 1980348k used, 51632k free, 1760k buffers
> Swap: 4194296k total, 705276k used, 3489020k free, 195156k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 5002 sipxchan 18 0 1423m 1.3g 3860 S 2 65.0 117:01.94 sipXproxy
> 4822 root 15 0 190m 189m 22m S 0 9.6 0:00.85 sipconf
> 5233 root 18 0 140m 138m 24m S 0 7.0 3:51.51 sipXvxml
> 5230 sipxchan 18 0 817m 62m 7588 S 0 3.2 1:23.68 java
> 5234 sipxchan 15 0 28600 11m 2236 S 0 0.6 0:26.30 ruby
> 5243 sipxchan 18 0 56692 9628 6800 S 0 0.5 1:16.63
> sipregistrar
> 4820 sipxchan 18 0 46232 8704 7352 S 0 0.4 0:59.74 sipstatus
> 5240 sipxchan 15 0 29548 8396 2036 S 0 0.4 0:01.06
> sipxconfig-agen
> 4561 sipxchan 18 0 36256 5560 5244 S 0 0.3 0:06.47 watchdog
> 4999 sipxchan 18 0 77308 5076 4796 S 0 0.2 0:07.41 sipxpark
> 2366 ntp 15 0 4316 4316 3312 S 0 0.2 0:00.00 ntpd
> 4765 root 24 0 61400 4228 3892 S 0 0.2 0:00.13 confbridge
> 5232 sipxchan 18 0 38828 4048 3884 S 0 0.2 0:00.06 sipxrls
> 2486 mysql 16 0 134m 3868 2688 S 0 0.2 0:03.99 mysqld
> 4821 sipxchan 18 0 56148 3520 3332 S 0 0.2 0:00.28
> sipxpresence
> 14396 postgres 16 0 21572 3508 2872 S 0 0.2 0:00.00 postmaster
> 5231 sipxchan 18 0 15196 3364 3344 S 0 0.2 0:00.01 sipxacd
> 11427 postgres 19 0 21736 3180 2412 S 0 0.2 0:00.33 postmaster
> 13853 sipxchan 17 0 10320 2976 1996 S 0 0.1 0:00.06 httpd
> 13792 sipxchan 17 0 10204 2932 1996 S 0 0.1 0:00.11 httpd
> 13788 sipxchan 20 0 10204 2912 1948 S 0 0.1 0:00.14 httpd
> 14208 sipxchan 15 0 10204 2888 1996 S 0 0.1 0:00.01 httpd
> 13995 sipxchan 16 0 10204 2860 1932 S 0 0.1 0:00.08 httpd
> 4603 root 18 0 10056 2656 2108 S 0 0.1 0:00.22 httpd
> 6340 postgres 15 0 21676 2568 2172 S 0 0.1 0:03.83 postmaster
> 5208 postgres 18 0 21652 2488 2168 S 0 0.1 0:06.64 postmaster
> 4691 root 18 0 15856 2176 1984 S 0 0.1 4:04.70 cfsserver
> 9795 postgres 15 0 21736 2136 1772 S 0 0.1 0:00.37 postmaster
> 2006 root 12 -3 10080 2020 1588 S 0 0.1 0:00.24 python
> 2587 root 18 0 9296 1964 1372 S 0 0.1 0:01.06 sendmail
> 2294 root 15 0 25920 1884 1448 S 0 0.1 0:00.62 snmpd
> 6482 root 15 0 9152 1804 1692 S 0 0.1 0:01.48 sshd
> 5000 sipxchan 18 0 691m 1764 1280 S 0 0.1 0:00.80 java
> 13278 postgres 15 0 22532 1724 1696 S 0 0.1 0:00.53 postmaster
> 7860 postgres 15 0 21720 1712 1684 S 0 0.1 0:00.22 postmaster
> 9796 postgres 15 0 21720 1700 1672 S 0 0.1 0:00.48 postmaster
> 13276 postgres 15 0 21724 1696 1668 S 0 0.1 0:00.75 postmaster
> 6428 postgres 15 0 22480 1664 1636 S 0 0.1 0:00.97 postmaster
> 6429 postgres 15 0 21704 1664 1636 S 0 0.1 0:00.37 postmaster
> 7862 postgres 15 0 21720 1652 1624 S 0 0.1 0:00.36 postmaster
> 7861 postgres 15 0 21736 1648 1620 S 0 0.1 0:00.21 postmaster
> 11426 postgres 15 0 21704 1640 1612 S 0 0.1 0:00.48 postmaster
> 6430 postgres 15 0 21696 1636 1608 S 0 0.1 0:00.06 postmaster
> 2553 postgres 18 0 20892 1628 1504 S 0 0.1 0:00.12 postmaster
> 2093 named 25 0 48084 1564 1192 S 0 0.1 0:00.11 named
> 9797 postgres 17 0 21568 1484 1456 S 0 0.1 0:00.01 postmaster
> 2557 postgres 15 0 21028 1416 1272 S 0 0.1 0:00.07 postmaster
> 11425 postgres 15 0 21572 1396 1368 S 0 0.1 0:00.01 postmaster
> 13277 postgres 15 0 21572 1396 1368 S 0 0.1 0:00.00 postmaster
> 4652 root 15 0 40556 1344 1240 S 0 0.1 0:41.46
> yum-updatesd
> 6487 root 15 0 4504 1168 1024 S 0 0.1 0:00.01 bash
> 4667 haldaemo 18 0 5384 1160 800 S 0 0.1 0:00.42 hald
>
> *** NOTES 2009-01-14 16:52:21 Eastern Daylight Time brigitba
> I had a screen shot of memory usage as of today
>
> *** NOTES 2009-01-14 17:01:00 Eastern Daylight Time brigitba
> Ok, ive just restarted the scs, here are the top readings
> top - 14:58:22 up 7 min, 1 user, load average: 0.08, 0.25, 0.16
> Tasks: 116 total, 1 running, 115 sleeping, 0 stopped, 0 zombie
> Cpu(s): 0.2%us, 0.0%sy, 0.0%ni, 99.7%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 2031980k total, 965428k used, 1066552k free, 21952k buffers
> Swap: 4194296k total, 0k used, 4194296k free, 395912k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 4821 root 22 0 190m 189m 22m S 0 9.6 0:00.33 sipconf
> 5228 sipxchan 18 0 796m 185m 26m S 0 9.3 0:37.41 java
> 5231 root 18 0 81720 79m 24m S 0 4.0 0:01.51 sipXvxml
> 5002 sipxchan 18 0 56944 22m 4132 S 0 1.1 0:05.62 sipXproxy
> 4997 sipxchan 18 0 690m 22m 9952 S 0 1.1 0:00.61 java
> 4560 sipxchan 18 0 36320 18m 5236 S 0 0.9 0:00.11 watchdog
> 2485 mysql 15 0 134m 18m 3632 S 0 0.9 0:00.05 mysqld
> 5239 sipxchan 15 0 27016 13m 2896 S 0 0.7 0:00.44 ruby
> 6310 postgres 15 0 21828 10m 9.8m S 0 0.5 0:00.22 postmaster
> 5242 sipxchan 15 0 24636 10m 2524 S 0 0.5 0:00.19
> sipxconfig-agen
> 4646 root 15 0 23932 10m 2840 S 0 0.5 0:00.04
> yum-updatesd
> 4819 sipxchan 18 0 35608 8436 5572 S 0 0.4 0:00.87 sipstatus
> 5249 sipxchan 18 0 47328 7476 4324 S 0 0.4 0:00.98
> sipregistrar
> 4768 root 23 0 62180 7232 4740 S 0 0.4 0:00.11 confbridge
> 2556 postgres 15 0 21028 7180 6752 S 0 0.4 0:00.00 postmaster
> 4996 sipxchan 18 0 74364 6748 4796 S 0 0.3 0:00.13 sipxpark
> 6418 postgres 15 0 22608 5292 3712 S 0 0.3 0:00.54 postmaster
> 2293 root 15 0 25916 5216 3096 S 0 0.3 0:00.10 snmpd
> 5230 sipxchan 18 0 38800 4720 3852 S 0 0.2 0:00.02 sipxrls
> 4820 sipxchan 18 0 57176 4396 3336 S 0 0.2 0:00.02
> sipxpresence
> 2365 ntp 15 0 4320 4320 3312 S 0 0.2 0:00.00 ntpd
> 5224 postgres 16 0 21652 4228 3328 S 0 0.2 0:00.05 postmaster
> 6345 postgres 16 0 21568 4044 3156 S 0 0.2 0:00.00 postmaster
> 2005 root 12 -3 10084 3924 2292 S 0 0.2 0:00.01 python
> 5229 sipxchan 18 0 15192 3916 3344 S 0 0.2 0:00.01 sipxacd
> 4666 haldaemo 18 0 5384 3620 1688 S 0 0.2 0:00.41 hald
> 4647 sipxchan 15 0 10200 3560 1996 S 0 0.2 0:00.09 httpd
> 4650 sipxchan 15 0 10200 3556 1996 S 0 0.2 0:00.09 httpd
> 6311 sipxchan 15 0 10200 3556 1996 S 0 0.2 0:00.07 httpd
> 4648 sipxchan 15 0 10200 3548 1996 S 0 0.2 0:00.07 httpd
> 4651 sipxchan 17 0 10312 3548 1996 S 0 0.2 0:00.07 httpd
> 6474 sipxchan 19 0 10312 3532 1996 S 0 0.2 0:00.03 httpd
> 4649 sipxchan 16 0 10200 3496 1936 S 0 0.2 0:00.10 httpd
> 4602 root 18 0 10052 3264 1876 S 0 0.2 0:00.06 httpd
> 6420 postgres 15 0 21568 2916 2100 S 0 0.1 0:00.00 postmaster
> 2092 named 25 0 48080 2888 1904 S 0 0.1 0:00.00 named
> 2552 postgres 18 0 20892 2836 2448 S 0 0.1 0:00.10 postmaster
> 6702 root 15 0 8996 2724 2208 S 0 0.1 0:00.02 sshd
> 2586 root 17 0 9140 2720 1564 S 0 0.1 0:00.00 sendmail
> 4690 root 18 0 15848 2540 1996 S 0 0.1 0:01.61 cfsserver
> 6419 postgres 16 0 21308 2032 1288 S 0 0.1 0:00.00 postmaster
> 2327 root 18 0 9676 1960 1356 S 0 0.1 0:00.00 cupsd
> 2585 root 15 0 8964 1664 664 S 0 0.1 0:00.00 sendmail
> 2593 smmsp 25 0 8016 1460 612 S 0 0.1 0:00.00 sendmail
> 6716 root 15 0 4504 1436 1216 S 0 0.1 0:00.01 bash
> 2594 smmsp 15 0 8016 1364 516 S 0 0.1 0:00.00 sendmail
> 2226 root 25 0 12824 1308 520 S 0 0.1 0:00.02 pcscd
> 2449 root 25 0 4500 1208 1048 S 0 0.1 0:00.00 mysqld_safe
> 2253 root 25 0 9364 1128 880 S 0 0.1 0:00.00 automount
> 2624 root 18 0 5232 1116 584 S 0 0.1 0:00.00 crond
> 2312 root 18 0 6148 1028 664 S 0 0.1 0:00.00 sshd
> 6739 root 15 0 2172 1028 796 R 0 0.1 0:00.03 top
_______________________________________________
sipx-dev mailing list
[email protected]
List Archive: http://list.sipfoundry.org/archive/sipx-dev
Unsubscribe: http://list.sipfoundry.org/mailman/listinfo/sipx-dev
_______________________________________________
sipx-dev mailing list
[email protected]
List Archive: http://list.sipfoundry.org/archive/sipx-dev
Unsubscribe: http://list.sipfoundry.org/mailman/listinfo/sipx-dev
