Hi,
This is my first post on this forum I am also new to Clearwater IMS. In fact I
am babysitting our Clearwater IMS while our developer is on leave.
We have a manual install of bono/sprout/homestead/ibcf/dns being used as an
interface between Voip UE’s and a PSTN, running on 5 VM’s. They are connected
via a private virtual switch, and only the bono and ibcf nodes are connected to
an external network interface for external sip signalling. This setup is
working with no problems. At present we have four of these IMSs setup. Two of
them in test sites (one for us one for the customer) and two in production
sites. Only one production site is carrying live traffic.
Last week I got an alert to say that one of the production ibcf VMs was running
out of free memory. Fortunately this was on the production IMS that was not
carrying live traffic. My colleague had warned me that the VMs were slowly
using up memory and that I should check them periodically to make sure that the
free memory was not getting to low.
The ibcf free memory was down to less than 6%. I restarted the VM and this
sorted out the problem. I also checked the other VMs in the both the
production IMSs and they were all above 30%. At this point I thought that the
problem had been resolved.
However, today we received the same alaert for the ibcf VM. Once again, the
VM, was down to less than 6% free memory.
[bono]nec@ibcf:~$ free
total used free shared buffers cached
Mem: 4041496 3809652 231844 224 151596 3247032
-/+ buffers/cache: 411024 3630472
Swap: 2093052 320 2092732
[bono]nec@ibcf:~$
I restarted the VM again and this dropped the free memory back to about 90%
[bono]nec@ibcf:~$ free
total used free shared buffers cached
Mem: 4041496 386116 3655380 444 14528 150728
-/+ buffers/cache: 220860 3820636
Swap: 2093052 0 2093052
[bono]nec@ibcf:~$
I have attached a file with some of the checking that I did before and after
the restart.
Has anyone else experienced this problem and are there any checks that can be
done to try and isolate the leaky process.
Regards,
Gerard
Gerard Ratcliffe
Technical Specialist
NEC New Zealand Limited
NEC House, Level 6, 40 Taranaki Street, PO Box: 1936, Wellington 6011, New
Zealand
T: 043816235, M: 0275037273, F: +6443811110
[email protected]<mailto:[email protected]>
nz.nec.com<http://nz.nec.com>
[cid:[email protected]]
Please consider the environment before printing this email
Attention:
The information contained in this message and or attachments is intended only
for the person or entity to which it is addressed and may contain confidential
and/or privileged material. Any review, retransmission, dissemination, copying
or other use of, or taking of any action in reliance upon, this information by
persons or entities other than the intended recipient is prohibited. If you
received this in error, please contact the sender and delete the material from
any system and destroy any copies. NEC has no liability for any act or omission
in reliance on the email or any attachment. Before opening this email or any
attachment(s), please check them for viruses. NEC is not responsible for any
viruses in this email or any attachment(s); any changes made to this email or
any attachment(s) after they are sent; or any effects this email or any
attachment(s) have on your network or computer system.
[bono]nec@ibcf:~$ sudo monit summary
[sudo] password for nec:
The Monit daemon 5.8.1 uptime: 1d 2h 39m
Process 'snmpd_process' Running
Process 'restund_process' Execution failed
Program 'poll_restund' Not monitored
Process 'ntp_process' Running
System 'node-ibcf' Running
Process 'clearwater_queue_manager' Running
Process 'etcd_process' Not monitored
Program 'poll_etcd_cluster' Not monitored
Program 'poll_etcd' Not monitored
Process 'clearwater_diags_monitor_process' Running
Process 'clearwater_config_manager' Running
Process 'clearwater_cluster_manager' Running
Process 'bono_process' Running
Program 'poll_bono' Status ok
[bono]nec@ibcf:~$
[bono]nec@ibcf:~$ free
total used free shared buffers cached
Mem: 4041496 3809652 231844 224 151596 3247032
-/+ buffers/cache: 411024 3630472
Swap: 2093052 320 2092732
[bono]nec@ibcf:~$
ps -eo pmem,rss,pcpu,vsize,pid,cmd | sort -k 5 -n | head -300
[bono]nec@ibcf:~$ ps -eo pmem,rss,pcpu,vsize,pid,cmd | sort -k 5 -n | head -300
%MEM RSS %CPU VSZ PID CMD
0.0 2836 0.0 33636 1 /sbin/init
0.0 0 0.0 0 2 [kthreadd]
0.0 0 0.0 0 3 [ksoftirqd/0]
0.0 0 0.0 0 5 [kworker/0:0H]
0.0 0 0.0 0 7 [rcu_sched]
0.0 0 0.0 0 8 [rcuos/0]
0.0 0 0.0 0 9 [rcuos/1]
0.0 0 0.0 0 10 [rcuos/2]
0.0 0 0.0 0 11 [rcuos/3]
0.0 0 0.0 0 12 [rcuos/4]
0.0 0 0.0 0 13 [rcuos/5]
0.0 0 0.0 0 14 [rcuos/6]
0.0 0 0.0 0 15 [rcuos/7]
0.0 0 0.0 0 16 [rcuos/8]
0.0 0 0.0 0 17 [rcuos/9]
0.0 0 0.0 0 18 [rcuos/10]
0.0 0 0.0 0 19 [rcuos/11]
0.0 0 0.0 0 20 [rcuos/12]
0.0 0 0.0 0 21 [rcuos/13]
0.0 0 0.0 0 22 [rcuos/14]
0.0 0 0.0 0 23 [rcuos/15]
0.0 0 0.0 0 24 [rcuos/16]
0.0 0 0.0 0 25 [rcuos/17]
0.0 0 0.0 0 26 [rcuos/18]
0.0 0 0.0 0 27 [rcuos/19]
0.0 0 0.0 0 28 [rcuos/20]
0.0 0 0.0 0 29 [rcuos/21]
0.0 0 0.0 0 30 [rcuos/22]
0.0 0 0.0 0 31 [rcuos/23]
0.0 0 0.0 0 32 [rcuos/24]
0.0 0 0.0 0 33 [rcuos/25]
0.0 0 0.0 0 34 [rcuos/26]
0.0 0 0.0 0 35 [rcuos/27]
0.0 0 0.0 0 36 [rcuos/28]
0.0 0 0.0 0 37 [rcuos/29]
0.0 0 0.0 0 38 [rcuos/30]
0.0 0 0.0 0 39 [rcuos/31]
0.0 0 0.0 0 40 [rcuos/32]
0.0 0 0.0 0 41 [rcuos/33]
0.0 0 0.0 0 42 [rcuos/34]
0.0 0 0.0 0 43 [rcuos/35]
0.0 0 0.0 0 44 [rcuos/36]
0.0 0 0.0 0 45 [rcuos/37]
0.0 0 0.0 0 46 [rcuos/38]
0.0 0 0.0 0 47 [rcuos/39]
0.0 0 0.0 0 48 [rcuos/40]
0.0 0 0.0 0 49 [rcuos/41]
0.0 0 0.0 0 50 [rcuos/42]
0.0 0 0.0 0 51 [rcuos/43]
0.0 0 0.0 0 52 [rcuos/44]
0.0 0 0.0 0 53 [rcuos/45]
0.0 0 0.0 0 54 [rcuos/46]
0.0 0 0.0 0 55 [rcuos/47]
0.0 0 0.0 0 56 [rcuos/48]
0.0 0 0.0 0 57 [rcuos/49]
0.0 0 0.0 0 58 [rcuos/50]
0.0 0 0.0 0 59 [rcuos/51]
0.0 0 0.0 0 60 [rcuos/52]
0.0 0 0.0 0 61 [rcuos/53]
0.0 0 0.0 0 62 [rcuos/54]
0.0 0 0.0 0 63 [rcuos/55]
0.0 0 0.0 0 64 [rcuos/56]
0.0 0 0.0 0 65 [rcuos/57]
0.0 0 0.0 0 66 [rcuos/58]
0.0 0 0.0 0 67 [rcuos/59]
0.0 0 0.0 0 68 [rcuos/60]
0.0 0 0.0 0 69 [rcuos/61]
0.0 0 0.0 0 70 [rcuos/62]
0.0 0 0.0 0 71 [rcuos/63]
0.0 0 0.0 0 72 [rcu_bh]
0.0 0 0.0 0 73 [rcuob/0]
0.0 0 0.0 0 74 [rcuob/1]
0.0 0 0.0 0 75 [rcuob/2]
0.0 0 0.0 0 76 [rcuob/3]
0.0 0 0.0 0 77 [rcuob/4]
0.0 0 0.0 0 78 [rcuob/5]
0.0 0 0.0 0 79 [rcuob/6]
0.0 0 0.0 0 80 [rcuob/7]
0.0 0 0.0 0 81 [rcuob/8]
0.0 0 0.0 0 82 [rcuob/9]
0.0 0 0.0 0 83 [rcuob/10]
0.0 0 0.0 0 84 [rcuob/11]
0.0 0 0.0 0 85 [rcuob/12]
0.0 0 0.0 0 86 [rcuob/13]
0.0 0 0.0 0 87 [rcuob/14]
0.0 0 0.0 0 88 [rcuob/15]
0.0 0 0.0 0 89 [rcuob/16]
0.0 0 0.0 0 90 [rcuob/17]
0.0 0 0.0 0 91 [rcuob/18]
0.0 0 0.0 0 92 [rcuob/19]
0.0 0 0.0 0 93 [rcuob/20]
0.0 0 0.0 0 94 [rcuob/21]
0.0 0 0.0 0 95 [rcuob/22]
0.0 0 0.0 0 96 [rcuob/23]
0.0 0 0.0 0 97 [rcuob/24]
0.0 0 0.0 0 98 [rcuob/25]
0.0 0 0.0 0 99 [rcuob/26]
0.0 0 0.0 0 100 [rcuob/27]
0.0 0 0.0 0 101 [rcuob/28]
0.0 0 0.0 0 102 [rcuob/29]
0.0 0 0.0 0 103 [rcuob/30]
0.0 0 0.0 0 104 [rcuob/31]
0.0 0 0.0 0 105 [rcuob/32]
0.0 0 0.0 0 106 [rcuob/33]
0.0 0 0.0 0 107 [rcuob/34]
0.0 0 0.0 0 108 [rcuob/35]
0.0 0 0.0 0 109 [rcuob/36]
0.0 0 0.0 0 110 [rcuob/37]
0.0 0 0.0 0 111 [rcuob/38]
0.0 0 0.0 0 112 [rcuob/39]
0.0 0 0.0 0 113 [rcuob/40]
0.0 0 0.0 0 114 [rcuob/41]
0.0 0 0.0 0 115 [rcuob/42]
0.0 0 0.0 0 116 [rcuob/43]
0.0 0 0.0 0 117 [rcuob/44]
0.0 0 0.0 0 118 [rcuob/45]
0.0 0 0.0 0 119 [rcuob/46]
0.0 0 0.0 0 120 [rcuob/47]
0.0 0 0.0 0 121 [rcuob/48]
0.0 0 0.0 0 122 [rcuob/49]
0.0 0 0.0 0 123 [rcuob/50]
0.0 0 0.0 0 124 [rcuob/51]
0.0 0 0.0 0 125 [rcuob/52]
0.0 0 0.0 0 126 [rcuob/53]
0.0 0 0.0 0 127 [rcuob/54]
0.0 0 0.0 0 128 [rcuob/55]
0.0 0 0.0 0 129 [rcuob/56]
0.0 0 0.0 0 130 [rcuob/57]
0.0 0 0.0 0 131 [rcuob/58]
0.0 0 0.0 0 132 [rcuob/59]
0.0 0 0.0 0 133 [rcuob/60]
0.0 0 0.0 0 134 [rcuob/61]
0.0 0 0.0 0 135 [rcuob/62]
0.0 0 0.0 0 136 [rcuob/63]
0.0 0 0.0 0 137 [migration/0]
0.0 0 0.0 0 138 [watchdog/0]
0.0 0 0.0 0 139 [khelper]
0.0 0 0.0 0 140 [kdevtmpfs]
0.0 0 0.0 0 141 [netns]
0.0 0 0.0 0 142 [writeback]
0.0 0 0.0 0 143 [kintegrityd]
0.0 0 0.0 0 144 [bioset]
0.0 0 0.0 0 145 [kworker/u129:0]
0.0 0 0.0 0 146 [kblockd]
0.0 0 0.0 0 147 [ata_sff]
0.0 0 0.0 0 148 [khubd]
0.0 0 0.0 0 149 [md]
0.0 0 0.0 0 150 [devfreq_wq]
0.0 0 0.0 0 153 [khungtaskd]
0.0 0 0.0 0 154 [kswapd0]
0.0 0 0.0 0 155 [ksmd]
0.0 0 0.0 0 156 [khugepaged]
0.0 0 0.0 0 157 [fsnotify_mark]
0.0 0 0.0 0 158 [ecryptfs-kthrea]
0.0 0 0.0 0 159 [crypto]
0.0 0 0.0 0 171 [kthrotld]
0.0 0 0.0 0 173 [scsi_eh_0]
0.0 0 0.0 0 174 [scsi_eh_1]
0.0 0 0.0 0 195 [deferwq]
0.0 0 0.0 0 196 [charger_manager]
0.0 0 0.0 0 245 [hv_vmbus_con]
0.0 0 0.0 0 246 [hv_vmbus_ctl]
0.0 0 0.0 0 247 [kworker/0:2]
0.0 0 0.0 0 248 [hv_vmbus_ctl]
0.0 0 0.0 0 249 [hv_vmbus_ctl]
0.0 0 0.0 0 250 [hv_vmbus_ctl]
0.0 0 0.0 0 251 [kpsmoused]
0.0 0 0.0 0 252 [hv_vmbus_ctl]
0.0 0 0.0 0 253 [hv_vmbus_ctl]
0.0 0 0.0 0 254 [hv_vmbus_ctl]
0.0 0 0.0 0 255 [kworker/0:3]
0.0 0 0.0 0 256 [hv_vmbus_ctl]
0.0 0 0.0 0 258 [hv_vmbus_ctl]
0.0 0 0.0 0 259 [hv_vmbus_ctl]
0.0 0 0.0 0 260 [hv_vmbus_ctl]
0.0 0 0.0 0 261 [hv_vmbus_ctl]
0.0 0 0.0 0 263 [scsi_eh_2]
0.0 0 0.0 0 264 [scsi_eh_3]
0.0 0 0.0 0 272 [kdmflush]
0.0 0 0.0 0 273 [bioset]
0.0 0 0.0 0 275 [kdmflush]
0.0 0 0.0 0 277 [bioset]
0.0 0 0.0 0 291 [jbd2/dm-0-8]
0.0 0 0.0 0 292 [ext4-rsv-conver]
0.0 656 0.0 19472 432 upstart-udev-bridge --daemon
0.0 1660 0.0 51368 437 /lib/systemd/systemd-udevd --daemon
0.0 0 0.0 0 549 [ext4-rsv-conver]
0.0 1252 0.0 39224 663 dbus-daemon --system --fork
0.0 1812 0.0 43448 746 /lib/systemd/systemd-logind
0.4 17172 0.0 256228 751 rsyslogd
0.0 740 0.0 15404 958 upstart-file-bridge --daemon
0.0 640 0.0 15388 961 upstart-socket-bridge --daemon
0.0 964 0.0 14540 1151 /sbin/getty -8 38400 tty4
0.0 960 0.0 14540 1154 /sbin/getty -8 38400 tty5
0.0 652 0.0 4440 1161 /bin/sh -e /proc/self/fd/9
0.0 956 0.0 14540 1162 /sbin/getty -8 38400 tty2
0.0 952 0.0 14540 1163 /sbin/getty -8 38400 tty3
0.0 956 0.0 14540 1166 /sbin/getty -8 38400 tty6
0.0 3084 0.0 61368 1191 /usr/sbin/sshd -D
0.0 576 0.0 4324 1199
/usr/lib/linux-tools/3.13.0-74-generic/hv_vss_daemon
0.0 0 0.0 0 1208 [hv_vmbus_ctl]
0.0 812 0.0 7072 1210
/usr/lib/linux-tools/3.13.0-74-generic/hv_kvp_daemon
0.0 664 0.0 4364 1215 acpid -c /etc/acpi/events -s /var/run/acpid.socket
0.0 1040 0.0 23652 1216 cron
0.0 160 0.0 19136 1217 atd
0.0 900 0.0 29748 1237 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid
-u dnsmasq -r /var/ run/dnsmasq/resolv.conf -7
/etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new
0.0 1764 0.0 18160 1326 /bin/bash
/usr/share/clearwater/bin/clearwater_diags_monitor
0.0 376 0.0 6496 1338 inotifywait -e create -qq .
0.1 4376 0.0 52232 1350 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp
-I -smux mteTrigge r mteTriggerConf -p /var/run/snmpd.pid
1.3 55152 0.1 766324 1453 /usr/share/clearwater/bin/bono
--domain=bbip.spark.co.nz --localhost=
192.168.10.100,ibcf.bbip.spark.co.nz
--alias=10.10.10.110,ibcf.bbip06.spark.co.nz --pcscf=5060,505
8 --webrtc-port=5062 --routing-proxy=sprout.bbip.spark.co.nz,5054,50,600
--sas=0.0.0.0,[email protected] bip.spark.co.nz
--dns-server=127.0.0.1 --pjsip-threads=1 --worker-threads=1
--analytics=/var/log/b ono --log-file=/var/log/bono
--log-level=2 --pidfile=/var/run/bono/bono.pid --ibcf=10.10.10.11,10.
10.10.31,10.10.10.32,10.10.10.33,10.10.10.34,10.10.10.41,10.10.10.42,10.10.10.43,10.10.10.44
--non -registering-pbxes=
--additional-domains=bbip06.spark.co.nz
0.3 13000 0.0 68036 1464
/usr/share/clearwater/clearwater-cluster-manager/env/bin/python /usr/
share/clearwater/bin/clearwater-cluster-manager
--mgmt-local-ip=192.168.10.100 --sig-local-ip=192.
168.10.100 --local-site=site1 --remote-site= --signaling-namespace=
--etcd-key=clearwater --etcd-c luster-key=bono
--log-level=2 --log-directory=/var/log/clearwater-cluster-manager
--pidfile=/var/r un/clearwater-cluster-manager.pid
0.3 15696 0.0 246452 1485
/usr/share/clearwater/clearwater-config-manager/env/bin/python /usr/s
hare/clearwater/bin/clearwater-config-manager
--local-ip=192.168.10.100 --local-site=site1 --log-l
evel=2 --log-directory=/var/log/clearwater-config-manager
--pidfile=/var/run/clearwater-config-man ager.pid
--etcd-key=clearwater
0.0 0 0.0 0 1502 [kworker/u129:1]
0.0 2704 0.9 104464 1504 /usr/bin/monit -I -c /etc/monit/monitrc
0.0 1244 0.0 12708 1517 /usr/share/clearwater/bin/clearwater_socket_factory
0.7 29864 1.0 46676 1521 /usr/bin/etcd --listen-client-urls
http://192.168.10.100:4000 --adver tise-client-urls
http://192.168.10.100:4000 --data-dir /var/lib/clearwater-etcd/192.168.10.100
--n ame 192-168-10-100 --debug
--initial-advertise-peer-urls http://192.168.10.100:2380 --listen-peer-
urls http://192.168.10.100:2380
0.4 18640 0.0 470112 1544
/usr/share/clearwater/clearwater-queue-manager/env/bin/python /usr/sh
are/clearwater/bin/clearwater-queue-manager
--local-ip=192.168.10.100 --local-site=site1 --log-lev
el=2 --log-directory=/var/log/clearwater-queue-manager
--pidfile=/var/run/clearwater-queue-manager .pid
--etcd-key=clearwater --node-type=bono
0.0 964 0.0 14540 1598 /sbin/getty -8 38400 tty1
0.0 0 0.0 0 1609 [kauditd]
0.0 2048 0.0 31444 1910 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 105:112
0.0 1076 0.0 25108 1968 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 105:112
0.0 0 0.0 0 30542 [kworker/u128:1]
0.1 4256 0.0 105632 30815 sshd: nec [priv]
0.0 1904 0.0 105632 30905 sshd: nec@pts/0
0.0 3520 0.0 21076 30906 -bash
0.0 0 0.0 0 31245 [poll_bono.sh] <defunct>
0.0 888 0.0 8736 31262 ps -eo pmem,rss,pcpu,vsize,pid,cmd
0.0 844 0.0 14436 31263 sort -k 5 -n
0.0 676 0.0 5928 31264 head -300
0.0 0 0.0 0 61392 [kworker/u128:2]
[bono]nec@ibcf:~$
[bono]nec@ibcf:~$ sudo init 6
[sudo] password for nec:
[bono]nec@ibcf:~$ Connection to ibcf closed by remote host.
Connection to ibcf closed.
############################### After Restart
############################################
[root@wnnipomm01 ~]# ssh nec@ibcf
Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.13.0-74-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Tue Oct 4 09:35:48 NZDT 2016
System load: 1.12 Processes: 219
Usage of /: 2.2% of 122.67GB Users logged in: 0
Memory usage: 2% IP address for eth1: 192.168.10.100
Swap usage: 0%
Graph this data and manage this system at:
https://landscape.canonical.com/
Last login: Tue Oct 4 09:11:03 2016 from 192.168.10.230
[bono]nec@ibcf:~$
[bono]nec@ibcf:~$ free
total used free shared buffers cached
Mem: 4041496 386116 3655380 444 14528 150728
-/+ buffers/cache: 220860 3820636
Swap: 2093052 0 2093052
[bono]nec@ibcf:~$ sudo monit summary
[sudo] password for nec:
Sorry, try again.
[sudo] password for nec:
The Monit daemon 5.8.1 uptime: 2m
Process 'snmpd_process' Running
Process 'restund_process' Execution failed
Program 'poll_restund' Not monitored
Process 'ntp_process' Running
System 'node-ibcf' Running
Process 'clearwater_queue_manager' Running
Process 'etcd_process' Not monitored
Program 'poll_etcd_cluster' Not monitored
Program 'poll_etcd' Not monitored
Process 'clearwater_diags_monitor_process' Running
Process 'clearwater_config_manager' Running
Process 'clearwater_cluster_manager' Running
Process 'bono_process' Running
Program 'poll_bono' Status ok
[bono]nec@ibcf:~$
[bono]nec@ibcf:~$
[bono]nec@ibcf:~$ sudo monit summary
The Monit daemon 5.8.1 uptime: 2m
Process 'snmpd_process' Running
Process 'restund_process' Execution failed
Program 'poll_restund' Not monitored
Process 'ntp_process' PID changed
System 'node-ibcf' Running
Process 'clearwater_queue_manager' Running
Process 'etcd_process' Not monitored
Program 'poll_etcd_cluster' Not monitored
Program 'poll_etcd' Not monitored
Process 'clearwater_diags_monitor_process' Running
Process 'clearwater_config_manager' Running
Process 'clearwater_cluster_manager' Running
Process 'bono_process' Running
Program 'poll_bono' Status ok
[bono]nec@ibcf:~$
[bono]nec@ibcf:~$
[bono]nec@ibcf:~$ sudo monit summary
The Monit daemon 5.8.1 uptime: 3m
Process 'snmpd_process' Running
Process 'restund_process' Execution failed
Program 'poll_restund' Not monitored
Process 'ntp_process' PID changed
System 'node-ibcf' Running
Process 'clearwater_queue_manager' Running
Process 'etcd_process' Not monitored
Program 'poll_etcd_cluster' Not monitored
Program 'poll_etcd' Not monitored
Process 'clearwater_diags_monitor_process' Running
Process 'clearwater_config_manager' Running
Process 'clearwater_cluster_manager' Running
Process 'bono_process' Running
Program 'poll_bono' Status ok
[bono]nec@ibcf:~
[bono]nec@ibcf:~$ ps -eo pmem,rss,pcpu,vsize,pid,cmd | sort -k 5 -n | head -300
%MEM RSS %CPU VSZ PID CMD
0.0 2932 0.2 33628 1 /sbin/init
0.0 0 0.0 0 2 [kthreadd]
0.0 0 0.0 0 3 [ksoftirqd/0]
0.0 0 0.0 0 5 [kworker/0:0H]
0.0 0 0.0 0 6 [kworker/u128:0]
0.0 0 0.0 0 7 [rcu_sched]
0.0 0 0.0 0 8 [rcuos/0]
0.0 0 0.0 0 9 [rcuos/1]
0.0 0 0.0 0 10 [rcuos/2]
0.0 0 0.0 0 11 [rcuos/3]
0.0 0 0.0 0 12 [rcuos/4]
0.0 0 0.0 0 13 [rcuos/5]
0.0 0 0.0 0 14 [rcuos/6]
0.0 0 0.0 0 15 [rcuos/7]
0.0 0 0.0 0 16 [rcuos/8]
0.0 0 0.0 0 17 [rcuos/9]
0.0 0 0.0 0 18 [rcuos/10]
0.0 0 0.0 0 19 [rcuos/11]
0.0 0 0.0 0 20 [rcuos/12]
0.0 0 0.0 0 21 [rcuos/13]
0.0 0 0.0 0 22 [rcuos/14]
0.0 0 0.0 0 23 [rcuos/15]
0.0 0 0.0 0 24 [rcuos/16]
0.0 0 0.0 0 25 [rcuos/17]
0.0 0 0.0 0 26 [rcuos/18]
0.0 0 0.0 0 27 [rcuos/19]
0.0 0 0.0 0 28 [rcuos/20]
0.0 0 0.0 0 29 [rcuos/21]
0.0 0 0.0 0 30 [rcuos/22]
0.0 0 0.0 0 31 [rcuos/23]
0.0 0 0.0 0 32 [rcuos/24]
0.0 0 0.0 0 33 [rcuos/25]
0.0 0 0.0 0 34 [rcuos/26]
0.0 0 0.0 0 35 [rcuos/27]
0.0 0 0.0 0 36 [rcuos/28]
0.0 0 0.0 0 37 [rcuos/29]
0.0 0 0.0 0 38 [rcuos/30]
0.0 0 0.0 0 39 [rcuos/31]
0.0 0 0.0 0 40 [rcuos/32]
0.0 0 0.0 0 41 [rcuos/33]
0.0 0 0.0 0 42 [rcuos/34]
0.0 0 0.0 0 43 [rcuos/35]
0.0 0 0.0 0 44 [rcuos/36]
0.0 0 0.0 0 45 [rcuos/37]
0.0 0 0.0 0 46 [rcuos/38]
0.0 0 0.0 0 47 [rcuos/39]
0.0 0 0.0 0 48 [rcuos/40]
0.0 0 0.0 0 49 [rcuos/41]
0.0 0 0.0 0 50 [rcuos/42]
0.0 0 0.0 0 51 [rcuos/43]
0.0 0 0.0 0 52 [rcuos/44]
0.0 0 0.0 0 53 [rcuos/45]
0.0 0 0.0 0 54 [rcuos/46]
0.0 0 0.0 0 55 [rcuos/47]
0.0 0 0.0 0 56 [rcuos/48]
0.0 0 0.0 0 57 [rcuos/49]
0.0 0 0.0 0 58 [rcuos/50]
0.0 0 0.0 0 59 [rcuos/51]
0.0 0 0.0 0 60 [rcuos/52]
0.0 0 0.0 0 61 [rcuos/53]
0.0 0 0.0 0 62 [rcuos/54]
0.0 0 0.0 0 63 [rcuos/55]
0.0 0 0.0 0 64 [rcuos/56]
0.0 0 0.0 0 65 [rcuos/57]
0.0 0 0.0 0 66 [rcuos/58]
0.0 0 0.0 0 67 [rcuos/59]
0.0 0 0.0 0 68 [rcuos/60]
0.0 0 0.0 0 69 [rcuos/61]
0.0 0 0.0 0 70 [rcuos/62]
0.0 0 0.0 0 71 [rcuos/63]
0.0 0 0.0 0 72 [rcu_bh]
0.0 0 0.0 0 73 [rcuob/0]
0.0 0 0.0 0 74 [rcuob/1]
0.0 0 0.0 0 75 [rcuob/2]
0.0 0 0.0 0 76 [rcuob/3]
0.0 0 0.0 0 77 [rcuob/4]
0.0 0 0.0 0 78 [rcuob/5]
0.0 0 0.0 0 79 [rcuob/6]
0.0 0 0.0 0 80 [rcuob/7]
0.0 0 0.0 0 81 [rcuob/8]
0.0 0 0.0 0 82 [rcuob/9]
0.0 0 0.0 0 83 [rcuob/10]
0.0 0 0.0 0 84 [rcuob/11]
0.0 0 0.0 0 85 [rcuob/12]
0.0 0 0.0 0 86 [rcuob/13]
0.0 0 0.0 0 87 [rcuob/14]
0.0 0 0.0 0 88 [rcuob/15]
0.0 0 0.0 0 89 [rcuob/16]
0.0 0 0.0 0 90 [rcuob/17]
0.0 0 0.0 0 91 [rcuob/18]
0.0 0 0.0 0 92 [rcuob/19]
0.0 0 0.0 0 93 [rcuob/20]
0.0 0 0.0 0 94 [rcuob/21]
0.0 0 0.0 0 95 [rcuob/22]
0.0 0 0.0 0 96 [rcuob/23]
0.0 0 0.0 0 97 [rcuob/24]
0.0 0 0.0 0 98 [rcuob/25]
0.0 0 0.0 0 99 [rcuob/26]
0.0 0 0.0 0 100 [rcuob/27]
0.0 0 0.0 0 101 [rcuob/28]
0.0 0 0.0 0 102 [rcuob/29]
0.0 0 0.0 0 103 [rcuob/30]
0.0 0 0.0 0 104 [rcuob/31]
0.0 0 0.0 0 105 [rcuob/32]
0.0 0 0.0 0 106 [rcuob/33]
0.0 0 0.0 0 107 [rcuob/34]
0.0 0 0.0 0 108 [rcuob/35]
0.0 0 0.0 0 109 [rcuob/36]
0.0 0 0.0 0 110 [rcuob/37]
0.0 0 0.0 0 111 [rcuob/38]
0.0 0 0.0 0 112 [rcuob/39]
0.0 0 0.0 0 113 [rcuob/40]
0.0 0 0.0 0 114 [rcuob/41]
0.0 0 0.0 0 115 [rcuob/42]
0.0 0 0.0 0 116 [rcuob/43]
0.0 0 0.0 0 117 [rcuob/44]
0.0 0 0.0 0 118 [rcuob/45]
0.0 0 0.0 0 119 [rcuob/46]
0.0 0 0.0 0 120 [rcuob/47]
0.0 0 0.0 0 121 [rcuob/48]
0.0 0 0.0 0 122 [rcuob/49]
0.0 0 0.0 0 123 [rcuob/50]
0.0 0 0.0 0 124 [rcuob/51]
0.0 0 0.0 0 125 [rcuob/52]
0.0 0 0.0 0 126 [rcuob/53]
0.0 0 0.0 0 127 [rcuob/54]
0.0 0 0.0 0 128 [rcuob/55]
0.0 0 0.0 0 129 [rcuob/56]
0.0 0 0.0 0 130 [rcuob/57]
0.0 0 0.0 0 131 [rcuob/58]
0.0 0 0.0 0 132 [rcuob/59]
0.0 0 0.0 0 133 [rcuob/60]
0.0 0 0.0 0 134 [rcuob/61]
0.0 0 0.0 0 135 [rcuob/62]
0.0 0 0.0 0 136 [rcuob/63]
0.0 0 0.0 0 137 [migration/0]
0.0 0 0.0 0 138 [watchdog/0]
0.0 0 0.0 0 139 [khelper]
0.0 0 0.0 0 140 [kdevtmpfs]
0.0 0 0.0 0 141 [netns]
0.0 0 0.0 0 142 [writeback]
0.0 0 0.0 0 143 [kintegrityd]
0.0 0 0.0 0 144 [bioset]
0.0 0 0.0 0 145 [kworker/u129:0]
0.0 0 0.0 0 146 [kblockd]
0.0 0 0.0 0 147 [ata_sff]
0.0 0 0.0 0 148 [khubd]
0.0 0 0.0 0 149 [md]
0.0 0 0.0 0 150 [devfreq_wq]
0.0 0 0.0 0 151 [kworker/0:1]
0.0 0 0.0 0 153 [khungtaskd]
0.0 0 0.0 0 154 [kswapd0]
0.0 0 0.0 0 155 [ksmd]
0.0 0 0.0 0 156 [khugepaged]
0.0 0 0.0 0 157 [fsnotify_mark]
0.0 0 0.0 0 158 [ecryptfs-kthrea]
0.0 0 0.0 0 159 [crypto]
0.0 0 0.0 0 171 [kthrotld]
0.0 0 0.0 0 172 [kworker/u128:1]
0.0 0 0.0 0 173 [scsi_eh_0]
0.0 0 0.0 0 174 [scsi_eh_1]
0.0 0 0.0 0 195 [deferwq]
0.0 0 0.0 0 196 [charger_manager]
0.0 0 0.0 0 245 [kpsmoused]
0.0 0 0.0 0 246 [hv_vmbus_con]
0.0 0 0.0 0 248 [hv_vmbus_ctl]
0.0 0 0.0 0 249 [hv_vmbus_ctl]
0.0 0 0.0 0 250 [hv_vmbus_ctl]
0.0 0 0.0 0 251 [hv_vmbus_ctl]
0.0 0 0.0 0 252 [hv_vmbus_ctl]
0.0 0 0.0 0 253 [hv_vmbus_ctl]
0.0 0 0.0 0 254 [hv_vmbus_ctl]
0.0 0 0.0 0 255 [hv_vmbus_ctl]
0.0 0 0.0 0 257 [hv_vmbus_ctl]
0.0 0 0.0 0 258 [hv_vmbus_ctl]
0.0 0 0.0 0 259 [hv_vmbus_ctl]
0.0 0 0.0 0 260 [hv_vmbus_ctl]
0.0 0 0.0 0 262 [scsi_eh_2]
0.0 0 0.0 0 263 [scsi_eh_3]
0.0 0 0.0 0 269 [kdmflush]
0.0 0 0.0 0 270 [bioset]
0.0 0 0.0 0 272 [kdmflush]
0.0 0 0.0 0 274 [bioset]
0.0 0 0.0 0 288 [jbd2/dm-0-8]
0.0 0 0.0 0 289 [ext4-rsv-conver]
0.0 648 0.0 19472 438 upstart-udev-bridge --daemon
0.0 1740 0.0 51492 443 /lib/systemd/systemd-udevd --daemon
0.0 0 0.0 0 625 [ext4-rsv-conver]
0.0 1260 0.0 39232 663 dbus-daemon --system --fork
0.0 1820 0.0 43448 687 /lib/systemd/systemd-logind
0.0 1336 0.0 255840 690 rsyslogd
0.0 0 0.0 0 711 [kworker/0:3]
0.0 696 0.0 15404 835 upstart-file-bridge --daemon
0.0 624 0.0 15256 850 upstart-socket-bridge --daemon
0.0 960 0.0 14540 1155 /sbin/getty -8 38400 tty4
0.0 952 0.0 14540 1158 /sbin/getty -8 38400 tty5
0.0 656 0.0 4440 1165 /bin/sh -e /proc/self/fd/9
0.0 960 0.0 14540 1166 /sbin/getty -8 38400 tty2
0.0 952 0.0 14540 1167 /sbin/getty -8 38400 tty3
0.0 952 0.0 14540 1170 /sbin/getty -8 38400 tty6
0.0 3088 0.0 61368 1195 /usr/sbin/sshd -D
0.0 572 0.0 4324 1200
/usr/lib/linux-tools/3.13.0-74-generic/hv_vss_daemon
0.0 0 0.0 0 1212 [hv_vmbus_ctl]
0.0 660 0.0 4364 1215 acpid -c /etc/acpi/events -s /var/run/acpid.socket
0.0 1044 0.0 23652 1218 cron
0.0 164 0.0 19136 1219 atd
0.0 808 0.0 7072 1221
/usr/lib/linux-tools/3.13.0-74-generic/hv_kvp_daemon
0.0 896 0.0 29748 1242 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid
-u dnsmasq -r /var/run/dnsmasq/resolv.conf -7
/etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new
0.0 1764 0.0 18160 1330 /bin/bash
/usr/share/clearwater/bin/clearwater_diags_monitor
0.0 372 0.0 6496 1339 inotifywait -e create -qq .
0.1 4376 0.0 52240 1354 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp
-I -smux mteTrigger mteTriggerConf -p /var/run/snmpd.pid
1.3 54148 0.2 700784 1457 /usr/share/clearwater/bin/bono
--domain=bbip.spark.co.nz --localhost=192.168.10.100,ibcf.bbip.spark.co.nz
--alias=10.10.10.110,ibcf.bbip06.spark.co.nz --pcscf=5060,5058
--webrtc-port=5062 --routing-proxy=sprout.bbip.spark.co.nz,5054,50,600
--sas=0.0.0.0,[email protected] --dns-server=127.0.0.1
--pjsip-threads=1 --worker-threads=1 --analytics=/var/log/bono
--log-file=/var/log/bono --log-level=2 --pidfile=/var/run/bono/bono.pid
--ibcf=10.10.10.11,10.10.10.31,10.10.10.32,10.10.10.33,10.10.10.34,10.10.10.41,10.10.10.42,10.10.10.43,10.10.10.44
--non-registering-pbxes= --additional-domains=bbip06.spark.co.nz
0.3 13160 0.0 68116 1468
/usr/share/clearwater/clearwater-cluster-manager/env/bin/python
/usr/share/clearwater/bin/clearwater-cluster-manager
--mgmt-local-ip=192.168.10.100 --sig-local-ip=192.168.10.100 --local-site=site1
--remote-site= --signaling-namespace= --etcd-key=clearwater
--etcd-cluster-key=bono --log-level=2
--log-directory=/var/log/clearwater-cluster-manager
--pidfile=/var/run/clearwater-cluster-manager.pid
0.3 15532 0.1 246320 1489
/usr/share/clearwater/clearwater-config-manager/env/bin/python
/usr/share/clearwater/bin/clearwater-config-manager --local-ip=192.168.10.100
--local-site=site1 --log-level=2
--log-directory=/var/log/clearwater-config-manager
--pidfile=/var/run/clearwater-config-manager.pid --etcd-key=clearwater
0.0 2564 1.0 104464 1507 /usr/bin/monit -I -c /etc/monit/monitrc
0.0 0 0.0 0 1518 [kworker/u129:1]
0.0 1244 0.0 12708 1521 /usr/share/clearwater/bin/clearwater_socket_factory
0.7 29924 1.6 46676 1525 /usr/bin/etcd --listen-client-urls
http://192.168.10.100:4000 --advertise-client-urls http://192.168.10.100:4000
--data-dir /var/lib/clearwater-etcd/192.168.10.100 --name 192-168-10-100
--debug --initial-advertise-peer-urls http://192.168.10.100:2380
--listen-peer-urls http://192.168.10.100:2380
0.4 16480 0.1 470112 1548
/usr/share/clearwater/clearwater-queue-manager/env/bin/python
/usr/share/clearwater/bin/clearwater-queue-manager --local-ip=192.168.10.100
--local-site=site1 --log-level=2
--log-directory=/var/log/clearwater-queue-manager
--pidfile=/var/run/clearwater-queue-manager.pid --etcd-key=clearwater
--node-type=bono
0.0 964 0.0 14540 1602 /sbin/getty -8 38400 tty1
0.0 0 0.0 0 1612 [kauditd]
0.1 4252 0.0 105632 1616 sshd: nec [priv]
0.0 1908 0.0 105632 1664 sshd: nec@pts/0
0.0 3748 0.0 21160 1665 -bash
0.0 2036 0.0 31444 1925 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 105:112
0.0 1068 0.0 25108 1950 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 105:112
0.0 0 0.0 0 1992 [poll_bono.sh] <defunct>
0.0 884 0.0 8736 2019 ps -eo pmem,rss,pcpu,vsize,pid,cmd
0.0 836 0.0 14436 2020 sort -k 5 -n
0.0 676 0.0 5928 2021 head -300
[bono]nec@ibcf:~$_______________________________________________
Clearwater mailing list
[email protected]
http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
