HI Matt, That did the trick. Utilization is down to average 8%. Thanks Joe
-----Original Message----- From: Matt Williams [mailto:[email protected]] Sent: Monday, April 07, 2014 12:53 PM To: joep Cc: [email protected] Subject: RE: Clearwater Digest, Vol 12, Issue 33 Joe, Thanks! Yes, this is the issue I just tracked down. The issue is that the Service Assurance Server client code retries immediately on failure due to using the wrong clock. Combined with the fact that the AWS AMI for Clearwater defaults the SAS to "localhost" (which fails the connection attempt immediately), this causes homestead to tight-loop. I've coded a fix for the Service Assurance Server client code, and it's out for review. I'll also fix the AWS AMI to specify a SAS of 0.0.0.0 to disable the connection. In the meantime, please can you - remove the "sas_server=localhost" line from /etc/clearwater/config - type "sudo monit restart homestead" to restart the homestead process - let me know if this helps? Thanks, Matt -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of joep Sent: 07 April 2014 20:25 To: [email protected] Subject: Re: [Clearwater] Clearwater Digest, Vol 12, Issue 33 HI Matt, cat homestead_20140407_1900.txt show never ending lines examples of which are shown below : 07-04-2014 19:00:11.840 Error sas.cpp:343: Failed to connect to SAS localhost:6761 : 111 Connection refused 07-04-2014 19:00:11.840 Status sas.cpp:279: Attempting to connect to SAS localhost and I had to stop the print. Not sure where the Service Assurance Server is located since I didn't set up our Metaswitch SAS on the All-In_One Clearwater installation and cannot recall any installation instructions on doing this. Re the command-line please see below. [cw-aio]ubuntu@ec2-54-85-220-241:/$ ps -eaf | grep homestead root 1447 1 0 Apr05 ? 00:00:23 /usr/share/clearwater/homestead/env/bin/python -m metaswitch.crest.main --background --worker-processes 1 ubuntu 24290 23630 0 19:06 pts/0 00:00:00 cat homestead_20140407_1900.txt ubuntu 25048 23630 0 19:08 pts/0 00:00:00 cat homestead_20140407_1900.txt 995 30176 1 85 19:19 ? 00:01:43 /usr/share/clearwater/bin/homestead --diameter-conf /var/lib/homestead/homestead.conf --http 172.31.8.119 --http-threads 50 --dest-realm example.com --dest-host 0.0.0.0 --server-name sip:ec2-54-85-220-241.compute-1.amazonaws.com:5054 --impu-cache-ttl 0 --hss-reregistration-time 1800 --sprout-http-name ec2-54-85-220-241.compute-1.amazonaws.com:9888 -a /var/log/homestead -F /var/log/homestead -L 2 --sas localhost,[email protected] root 31194 1509 0 19:21 ? 00:00:00 /bin/bash /usr/share/clearwater/bin/poll_homestead.sh root 31195 1509 0 19:21 ? 00:00:00 /bin/bash /usr/share/clearwater/bin/poll_homestead-prov.sh ubuntu 31207 23630 0 19:21 pts/0 00:00:00 grep --color=auto homestead [cw-aio]ubuntu@ec2-54-85-220-241:/$ Regards Joe -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of [email protected] Sent: Monday, April 07, 2014 11:29 AM To: [email protected] Subject: Clearwater Digest, Vol 12, Issue 33 Send Clearwater mailing list submissions to [email protected] To subscribe or unsubscribe via the World Wide Web, visit http://lists.projectclearwater.org/listinfo/clearwater or, via email, send a message with subject or body 'help' to [email protected] You can reach the person managing the list at [email protected] When replying, please edit your Subject line so it is more specific than "Re: Contents of Clearwater digest..." Today's Topics: 1. Re: All-In-One CUP utilization (Matt Williams) ---------------------------------------------------------------------- Message: 1 Date: Mon, 7 Apr 2014 18:20:08 +0000 From: Matt Williams <[email protected]> To: PCS1 - Joe Pilcher <[email protected]>, Eleanor Merry <[email protected]> Cc: "[email protected]" <[email protected]> Subject: Re: [Clearwater] All-In-One CUP utilization Message-ID: <[email protected]> Content-Type: text/plain; charset="us-ascii" Joe, Thanks for your email. Two quick questions. * Please could you share the latest logs from /var/log/homestead/homestead*.txt? * Please could you share the full command-line to homestead (e.g. via "ps -eaf | grep homestead")? I've just tracked down a bug in our logging infrastructure that could cause a tight loop, and I'd like to check if that's what you're hitting. Thanks, Matt From: [email protected] [mailto:[email protected]] On Behalf Of joep Sent: 07 April 2014 18:37 To: Eleanor Merry; [email protected] Subject: Re: [Clearwater] All-In-One CUP utilization Hi Ellie, Please see below for UTOP/TOP Thanks Joe CPU[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||99.3%] Tasks: 60, 313 thr; 2 running Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||2708/3750MB] Load average: 1.24 1.52 1.55 Swp[ 0/0MB] Uptime: 1 day, 17:59:36 PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 28515 homestead 20 0 2553M 1021M 3408 R 92.0 27.2 30:42.44 /usr/share/clearwater/bin/homestead --diameter-conf /var/lib/homestead/homestead.conf --http 172.31.8.119 --http-t 28506 homestead 20 0 2553M 1021M 3408 S 92.0 27.2 30:44.09 /usr/share/clearwater/bin/homestead --diameter-conf /var/lib/homestead/homestead.conf --http 172.31.8.119 --http-t 12363 ubuntu 20 0 25448 3200 1424 R 2.0 0.1 0:01.01 htop 1641 sprout 20 0 1280M 41868 0 S 0.0 1.1 5:40.69 /usr/share/clearwater/bin/sprout --domain example.com --localhost 172.31.8.119 --sprout-domain ec2-54-85-220-241.c 1470 sprout 20 0 1280M 41868 0 S 0.0 1.1 6:24.48 /usr/share/clearwater/bin/sprout --domain example.com --localhost 172.31.8.119 --sprout-domain ec2-54-85-220-241.c 24725 cassandra 20 0 2193M 1355M 17476 S 0.0 36.2 0:05.00 jsvc.exec -user cassandra -home /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/../ -pidfile /var/run/cassandra.pid -err 1 root 20 0 24344 1796 880 S 0.0 0.0 0:00.54 /sbin/init 251 root 20 0 17236 372 188 S 0.0 0.0 0:00.05 upstart-udev-bridge --daemon 253 root 20 0 21516 1180 696 S 0.0 0.0 0:00.05 /sbin/udevd --daemon 300 root 20 0 21468 644 252 S 0.0 0.0 0:00.00 /sbin/udevd --daemon 301 root 20 0 21468 616 228 S 0.0 0.0 0:00.00 /sbin/udevd --daemon 372 root 20 0 15192 192 4 S 0.0 0.0 0:00.00 upstart-socket-bridge --daemon 452 root 20 0 7268 944 448 S 0.0 0.0 0:00.15 dhclient3 -e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -1 eth0 645 root 20 0 50036 1492 884 S 0.0 0.0 0:00.07 /usr/sbin/sshd -D 669 syslog 20 0 243M 1456 740 S 0.0 0.0 0:02.04 rsyslogd -c5 670 syslog 20 0 243M 1456 740 S 0.0 0.0 0:00.10 rsyslogd -c5 671 syslog 20 0 243M 1456 740 S 0.0 0.0 0:00.09 rsyslogd -c5 661 syslog 20 0 243M 1456 740 S 0.0 0.0 0:13.45 rsyslogd -c5 662 messagebu 20 0 23820 588 280 S 0.0 0.0 0:00.00 dbus-daemon --system --fork --activation=upstart 720 root 20 0 14508 912 748 S 0.0 0.0 0:00.00 /sbin/getty -8 38400 tty4 728 root 20 0 14508 912 748 S 0.0 0.0 0:00.00 /sbin/getty -8 38400 tty5 737 root 20 0 14508 908 748 S 0.0 0.0 0:00.00 /sbin/getty -8 38400 tty2 739 root 20 0 14508 908 748 S 0.0 0.0 0:00.00 /sbin/getty -8 38400 tty3 742 root 20 0 14508 908 748 S 0.0 0.0 0:00.00 /sbin/getty -8 38400 tty6 749 dnsmasq 20 0 27544 792 544 S 0.0 0.0 0:00.45 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -r /var/run/dnsmasq/resolv.conf -7 /etc/dnsmasq.d,.dp 755 root 20 0 4332 640 496 S 0.0 0.0 0:00.00 acpid -c /etc/acpi/events -s /var/run/acpid.socket 756 root 20 0 19116 976 740 S 0.0 0.0 0:00.37 cron 757 daemon 20 0 16912 284 120 S 0.0 0.0 0:00.00 atd 851 whoopsie 20 0 183M 1980 756 S 0.0 0.1 0:00.00 whoopsie 840 whoopsie 20 0 183M 1980 756 S 0.0 0.1 0:00.01 whoopsie 887 mysql 20 0 545M 37640 1228 S 0.0 1.0 0:00.00 /usr/sbin/mysqld 888 mysql 20 0 545M 37640 1228 S 0.0 1.0 0:00.00 /usr/sbin/mysqld 889 mysql 20 0 545M 37640 1228 S 0.0 1.0 0:00.00 /usr/sbin/mysqld 890 mysql 20 0 545M 37640 1228 S 0.0 1.0 0:00.00 /usr/sbin/mysqld 891 mysql 20 0 545M 37640 1228 S 0.0 1.0 0:00.00 /usr/sbin/mysqld 892 mysql 20 0 545M 37640 1228 S 0.0 1.0 0:00.00 /usr/sbin/mysqld 893 mysql 20 0 545M 37640 1228 S 0.0 1.0 0:00.00 /usr/sbin/mysqld 894 mysql 20 0 545M 37640 1228 S 0.0 1.0 0:00.00 /usr/sbin/mysqld 895 mysql 20 0 545M 37640 1228 S 0.0 1.0 0:00.00 /usr/sbin/mysqld 896 mysql 20 0 545M 37640 1228 S 0.0 1.0 0:00.00 /usr/sbin/mysqld 898 mysql 20 0 545M 37640 1228 S 0.0 1.0 0:10.85 /usr/sbin/mysqld 899 mysql 20 0 545M 37640 1228 S 0.0 1.0 0:21.39 /usr/sbin/mysqld 900 mysql 20 0 545M 37640 1228 S 0.0 1.0 0:00.81 /usr/sbin/mysqld 901 mysql 20 0 545M 37640 1228 S 0.0 1.0 0:00.00 /usr/sbin/mysqld 1081 mysql 20 0 545M 37640 1228 S 0.0 1.0 0:00.00 /usr/sbin/mysqld F1Help F2Setup F3SearchF4FilterF5Tree F6SortByF7Nice -F8Nice +F9Kill F10Quit Using username "ubuntu". Authenticating with public key "imported-openssh-key" Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-31-virtual x86_64) * Documentation: https://help.ubuntu.com/ System information disabled due to load higher than 1.0 155 packages can be updated. 65 updates are security updates. Get cloud support with Ubuntu Advantage Cloud Guest http://www.ubuntu.com/business/services/cloud *** /dev/xvda1 will be checked for errors at next reboot *** [cw-aio]ubuntu@ec2-54-85-220-241:~$ sudo utop sudo: utop: command not found [cw-aio]ubuntu@ec2-54-85-220-241:~$ htop [1]+ Stopped htop [cw-aio]ubuntu@ec2-54-85-220-241:~$ top top - 17:35:52 up 1 day, 18:04, 1 user, load average: 1.05, 1.24, 1.42 Tasks: 97 total, 1 running, 90 sleeping, 1 stopped, 5 zombie Cpu(s): 16.2%us, 80.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 3.7%si, 0.0%st Mem: 3840492k total, 3802312k used, 38180k free, 2600k buffers Swap: 0k total, 0k used, 0k free, 879356k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28506 homestea 20 0 2745m 1.1g 3408 S 78.7 31.2 35:10.48 homestead 3 root 20 0 0 0 0 S 5.3 0.0 146:38.02 ksoftirqd/0 25 root 20 0 0 0 0 S 0.3 0.0 1:10.34 kswapd0 1477 root 20 0 353m 3660 1320 S 0.3 0.1 9:39.80 chronos 13500 ubuntu 20 0 17344 1272 956 R 0.3 0.0 0:00.37 top 24639 cassandr 20 0 2193m 1.3g 17m S 0.3 36.2 1:18.32 jsvc 1 root 20 0 24344 1796 880 S 0.0 0.0 0:00.54 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0 5 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kworker/u:0 6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 7 root RT 0 0 0 0 S 0.0 0.0 0:00.98 watchdog/0 8 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 cpuset 9 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper 10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs 11 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns 12 root 20 0 0 0 0 S 0.0 0.0 0:00.02 xenwatch 13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 xenbus 14 root 20 0 0 0 0 S 0.0 0.0 0:00.49 sync_supers 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 bdi-default 16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd 17 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd 18 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ata_sff 19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khubd 20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md 21 root 20 0 0 0 0 S 0.0 0.0 0:06.91 kworker/0:1 23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u:1 24 root 20 0 0 0 0 S 0.0 0.0 0:00.09 khungtaskd 26 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd 27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 fsnotify_mark 28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ecryptfs-kthrea 29 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto 37 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld 38 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khvcd 57 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 devfreq_wq 155 root 20 0 0 0 0 S 0.0 0.0 0:11.52 jbd2/xvda1-8 156 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-dio-unwrit 251 root 20 0 17236 372 188 S 0.0 0.0 0:00.05 upstart-udev-br 253 root 20 0 21516 1180 696 S 0.0 0.0 0:00.05 udevd 300 root 20 0 21468 644 252 S 0.0 0.0 0:00.00 udevd 301 root 20 0 21468 616 228 S 0.0 0.0 0:00.00 udevd 372 root 20 0 15192 192 4 S 0.0 0.0 0:00.00 upstart-socket- 452 root 20 0 7268 744 248 S 0.0 0.0 0:00.15 dhclient3 608 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kjournald 645 root 20 0 50036 1492 884 S 0.0 0.0 0:00.07 sshd [2]+ Stopped top [cw-aio]ubuntu@ec2-54-85-220-241:~$ clear [cw-aio]ubuntu@ec2-54-85-220-241:~$ top top - 17:36:07 up 1 day, 18:05, 1 user, load average: 1.04, 1.23, 1.41 Tasks: 96 total, 1 running, 93 sleeping, 2 stopped, 0 zombie Cpu(s): 8.8%us, 88.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 3.1%si, 0.0%st Mem: 3840492k total, 3811408k used, 29084k free, 2644k buffers Swap: 0k total, 0k used, 0k free, 879944k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28506 homestea 20 0 2745m 1.1g 3408 S 90.3 31.3 35:23.32 homestead 3 root 20 0 0 0 0 S 6.0 0.0 146:38.90 ksoftirqd/0 1320 bono 20 0 818m 42m 1156 S 0.3 1.1 7:13.08 bono 1477 root 20 0 353m 3660 1320 S 0.3 0.1 9:39.86 chronos 1509 root 20 0 101m 1552 916 S 0.3 0.0 2:07.55 monit 15021 ubuntu 20 0 17344 1280 956 R 0.3 0.0 0:00.02 top 24639 cassandr 20 0 2193m 1.3g 17m S 0.3 36.2 1:18.38 jsvc 1 root 20 0 24344 1796 880 S 0.0 0.0 0:00.54 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0 5 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kworker/u:0 6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 7 root RT 0 0 0 0 S 0.0 0.0 0:00.98 watchdog/0 8 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 cpuset 9 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper 10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs 11 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns 12 root 20 0 0 0 0 S 0.0 0.0 0:00.02 xenwatch 13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 xenbus 14 root 20 0 0 0 0 S 0.0 0.0 0:00.49 sync_supers 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 bdi-default 16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd 17 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd 18 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ata_sff 19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khubd 20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md 21 root 20 0 0 0 0 S 0.0 0.0 0:06.91 kworker/0:1 23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u:1 24 root 20 0 0 0 0 S 0.0 0.0 0:00.09 khungtaskd 25 root 20 0 0 0 0 S 0.0 0.0 1:10.34 kswapd0 26 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd 27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 fsnotify_mark 28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ecryptfs-kthrea 29 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto 37 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld 38 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khvcd 57 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 devfreq_wq 155 root 20 0 0 0 0 S 0.0 0.0 0:11.52 jbd2/xvda1-8 156 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-dio-unwrit 251 root 20 0 17236 372 188 S 0.0 0.0 0:00.05 upstart-udev-br 253 root 20 0 21516 1180 696 S 0.0 0.0 0:00.05 udevd 300 root 20 0 21468 644 252 S 0.0 0.0 0:00.00 udevd 301 root 20 0 21468 616 228 S 0.0 0.0 0:00.00 udevd 372 root 20 0 15192 192 4 S 0.0 0.0 0:00.00 upstart-socket- 452 root 20 0 7268 744 248 S 0.0 0.0 0:00.15 dhclient3 From: Eleanor Merry [mailto:[email protected]] Sent: Monday, April 07, 2014 2:43 AM To: joep; [email protected]<mailto:[email protected]> Subject: RE: All-In-One CUP utilization Hi Joe, Thanks for highlighting this! Can you please run 'top' when the CPU utilization is high, and let me know which process(es) is using all the memory? Thanks, Ellie From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of joep Sent: 04 April 2014 23:09 To: [email protected]<mailto:[email protected]> Subject: [Clearwater] All-In-One CUP utilization A couple of days ago I noticed the CPU CPU utilization on the Ubuntu server that is running the Clearwater All-In-One image had shot up to 100% for a protracted amount of time. I brought the utilization back to normal by restarting the server. However as you can see from the attached it is back again to around 70% and looks like may go back up to 100% soon. It could be this is a server problem and not software related but thought should send this info out. Thanks Joe -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.projectclearwater.org/pipermail/clearwater/attachments/20140407/cf7ac565/attachment.html> ------------------------------ _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater End of Clearwater Digest, Vol 12, Issue 33 ****************************************** _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater
