Hi Ashish, The restund configuration file looks correct.
It looks like there's a problem with the restund monit file, where it tries to abort the restund process if the poll_restund script fails. However this doesn't actually restart the restund process. We're going to fix this up. In the meantime, can you please manually restart restund (run 'sudo service restund stop' and allow monit to restart it automatically), and see if that fixes this up. If it doesn't, then can you please send me your /var/log/syslog file (as this is where restund logs to) and the /var/log/monit.log file. Ellie -----Original Message----- From: Ashish Maheshwari -X (ashimahe - L & T TECHNOLOGY SERVICES LIMITED at Cisco) [mailto:[email protected]] Sent: 03 March 2015 09:21 To: Eleanor Merry; [email protected] Subject: RE: SCSCF sending repetetive MAR to HSS Hi Ellie, Below the snapshot of my restund.conf file where I am using turn_relay_addr as 50.50.13.5(local IP) and turn_relay_snat_addr 10.105.244.117 (public IP) please have a look if anything else need to correct. # restund.conf # core daemon yes debug no realm example.com syncinterval 600 udp_listen 50.50.13.5:3478 udp_sockbuf_size 524288 tcp_listen 50.50.13.5:3478 # modules module_path /usr/share/clearwater/restund/lib module stat.so module httpdb.so module binding.so module auth.so module turn.so module syslog.so module status.so # auth auth_nonce_expiry 3600 # turn turn_max_allocations 512 turn_max_lifetime 600 turn_relay_addr 50.50.13.5 turn_relay_snat_addr 10.105.244.117 turn_relay_addr6 ::1 # Cope with broken TURN clients turn_extended_channels yes # httpdb httpdb_url http://hs.example.com:8888/impi/%s/digest # Relax TURN security to support buggy clients httpdb_uri_workaround yes httpdb_uri_workaround_password # syslog syslog_facility 24 # status status_udp_addr 127.0.0.1 status_udp_port 33000 status_http_addr 127.0.0.1 status_http_port 8080 ~ ~ Thanks & Regards, Ashish Maheshwari -----Original Message----- From: Eleanor Merry [mailto:[email protected]] Sent: 28 February 2015 00:33 To: Ashish Maheshwari -X (ashimahe - L & T TECHNOLOGY SERVICES LIMITED at Cisco); [email protected] Subject: RE: SCSCF sending repetetive MAR to HSS Hi Ashish, Can you send across your configuration in /etc/clearwater/restund.conf? Can you also check that it has the correct IP addresses for the turn_relay_addr (should be the local IP) and turn_relay_snat_addr (should be your public IP)? 'moniter' and 'restart' are not the same. 'monitor' tells monit to start monitoring a process again; monit will then start the process if it isn't running. 'restart' tells monit to stop and start the process (even if it wasn't running in the first place). Typically, you shouldn't run restart as it can cause the process to start up slowly - instead we recommend you run 'sudo service <process> stop' and allow monit to (automatically) start the process. If a process is showing 'Not monitored', run sudo monit monitor <process> to get monit to watch the process again. If the process is showing as failed, then you shouldn't need to do anything - monit will restart the process itself. Ellie -----Original Message----- From: Ashish Maheshwari -X (ashimahe - L & T TECHNOLOGY SERVICES LIMITED at Cisco) [mailto:[email protected]] Sent: 26 February 2015 07:56 To: Eleanor Merry; [email protected] Subject: RE: SCSCF sending repetetive MAR to HSS Hi Ellie, Now the registration happened successfully but still program poll_restund status showing failed, would like to share that I didn't use any stun server while configuring Zoiper sip client , is this might the cause for this ??? Program 'poll_restund' status Status failed monitoring status Monitored last started Wed, 25 Feb 2015 23:59:53 last exit value 1 data collected Wed, 25 Feb 2015 23:59:53 Program 'poll_cassandra_ring' status Waiting monitoring status Waiting last started Thu, 26 Feb 2015 00:05:18 last exit value 0 data collected Thu, 26 Feb 2015 00:05:18 I analyze the monit logs and see the error : [UTC Feb 26 00:00:44] info : 'poll_restund' exec: /etc/init.d/restund [UTC Feb 26 00:00:44] error : 'cassandra' uptime test failed for /var/run/cassandra.pid -- current uptime is 20 seconds [UTC Feb 26 00:00:44] info : 'cassandra' exec: /bin/true [UTC Feb 26 00:00:54] error : 'poll_restund' STUN poll failed to 50.50.13.5 One more query : is remonitor or restart both are same thing right ? is this general command to start a service when its showing Not Monitored or status failed " sudo monit monitor <process name>" ?? Thanks & Regards, Ashish Maheshwari -----Original Message----- From: Eleanor Merry [mailto:[email protected]] Sent: 26 February 2015 02:41 To: Ashish Maheshwari -X (ashimahe - L & T TECHNOLOGY SERVICES LIMITED at Cisco); [email protected] Cc: Chinna Morampalle (cmorampa); Satheesh Marappan (sathkuma); Anil Jain -X (anilja - Aricent Technology Holdings Limited at Cisco) Subject: RE: SCSCF sending repetetive MAR to HSS Hi Ashish, Can you also send the /etc/memcached_11211.conf file across too? This should include the following: # Specify which IP address to listen on. The default is to listen on all IP addresses # This parameter is one of the only security measures that memcached has, so make sure # it's listening on a firewalled interface. -l <local IP address> Can you check that the IP address has been filled in, and is set to the correct IP address for the box? Thanks, Ellie -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Eleanor Merry Sent: 25 February 2015 21:04 To: Ashish Maheshwari -X (ashimahe - L & T TECHNOLOGY SERVICES LIMITED at Cisco); [email protected] Cc: Chinna Morampalle (cmorampa); Satheesh Marappan (sathkuma); Anil Jain -X (anilja - Aricent Technology Holdings Limited at Cisco) Subject: Re: [Clearwater] SCSCF sending repetetive MAR to HSS Hi Ashish, Can you send across your memcached cluster setting file (in /etc/clearwater/cluster_settings) and the most recent sprout log? To get this, can you restart sprout (use 'sudo service sprout stop'), attempt a register, and then send just the most recent /var/log/sprout/sprout_* file. Also, can you confirm that the memcached service is running ('sudo service memcached status'), and remonitor it with monit ('sudo monit monitor memcached'). Restund is our STUN/TURN server, and it's used by Bono; poll_restund is a script that monit uses to check that the restund process is correctly responding to requests. Is this still failing in monit? Can you send over the /var/log/monit.log file? Thanks, Ellie From: Ashish Maheshwari -X (ashimahe - L & T TECHNOLOGY SERVICES LIMITED at Cisco) [mailto:[email protected]] Sent: 25 February 2015 13:25 To: Eleanor Merry; [email protected] Cc: Chinna Morampalle (cmorampa); Satheesh Marappan (sathkuma); Anil Jain -X (anilja - Aricent Technology Holdings Limited at Cisco) Subject: FW: SCSCF sending repetetive MAR to HSS Hi Ellie, I completely re -installed the latest version of CW IMS OVF and did the integration with external HSS but after that when I check the status of all the services I found : Program 'poll_memcached' status Status failed monitoring status Monitored last started Wed, 25 Feb 2015 05:19:21 last exit value 1 data collected Wed, 25 Feb 2015 05:19:21 Process 'memcached' status Not monitored monitoring status Not monitored data collected Wed, 25 Feb 2015 03:44:07 Program 'poll_restund' status Status failed monitoring status Monitored last started Wed, 25 Feb 2015 05:45:28 last exit value 1 data collected Wed, 25 Feb 2015 05:45:28 What is this poll_restund process and how we can make it running.... To trouble shoot this memcached issue I took help from your online help : Memcached logs to /var/log/memcached.log. It logs very little by default, but it is possible to make it more verbose by editing /etc/memcached_11211.conf, uncommenting the -vvline, and running sudo monit restart memcached. In attached memcached logs in I can see : failed to listen on TCP port 11211: Invalid argument getaddrinfo(): Temporary failure in name resolution I also restart sprout service to capture the logs to see when sprout initially communicates to memcached but didn't get too much information from it. I used the attached config file and also attaching pcap capture ,syslog , config file and memcached logs not able to attached sprout logs as huge in size but I am seeing the same error as you mentioned in the below mail. 20-02-2015 04:07:13.921 UTC Debug memcachedstore.cpp:649: memcached_add command for av\\[email protected]\46fa0be12dbafeef failed on replica 0, rc = 47 (SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY), expiry = 40 (140124113689584) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY, host: 10.105.244.117:11211 -> libmemcached/connect.cc:633 20-02-2015 04:07:13.921 UTC Error memcachedstore.cpp:708: Failed to write data for av\\[email protected]\46fa0be12dbafeef to 1 replicas 20-02-2015 04:07:13.921 UTC Error avstore.cpp:73: Failed to write Authentication Vector for private_id [email protected]<mailto:[email protected]> As memcached service is not working I am still facing repetitive MAR from SCSF to HSS. Can you please help me to resolve this issue on priority as we totally blocked. Thanks in advance. Thanks & Regards, Ashish Maheshwari From: Eleanor Merry [mailto:[email protected]] Sent: 20 February 2015 19:11 To: Ashish Maheshwari -X (ashimahe - L & T TECHNOLOGY SERVICES LIMITED at Cisco); [email protected]<mailto:[email protected]> Cc: Chinna Morampalle (cmorampa); Satheesh Marappan (sathkuma) Subject: RE: SCSCF sending repetetive MAR to HSS Hi Ashish, It looks like there's a problem with memcached - from the Sprout logs: 20-02-2015 04:07:13.921 UTC Debug memcachedstore.cpp:649: memcached_add command for av\\[email protected]\46fa0be12dbafeef failed on replica 0, rc = 47 (SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY), expiry = 40 (140124113689584) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY, host: 10.105.244.117:11211 -> libmemcached/connect.cc:633 20-02-2015 04:07:13.921 UTC Error memcachedstore.cpp:708: Failed to write data for av\\[email protected]\46fa0be12dbafeef to 1 replicas 20-02-2015 04:07:13.921 UTC Error avstore.cpp:73: Failed to write Authentication Vector for private_id [email protected]<mailto:[email protected]> Are there any memcached logs in /var/log/memcached.log or in /var/log/syslog? Also, can you check if there are any reported errors about memcached when Sprout initially attempts to connect to memcached - you'll need to restart Sprout ('sudo service sprout stop' - it'll be restarted by monit). The memcached error is causing the repetitive MARs because when the S-CSCF receives the REGISTER for the P-CSCF, it first checks its authorization store for the subscriber. On the initial REGISTER this is empty, so the S-CSCF makes a MAR to the HSS. It then writes the returned auth information into the store (this step is failing in your test), and returns a 401 to the P-CSCF. Then when the S-CSCF receives the REGISTER with authorization, it checks this against its stored information. However in your case, Sprout looks up the subscriber in the auth store, doesn't find anything (as the write before failed), so makes a MAR to the HSS. We should probably return a better code to the P-CSCF if the write fails (a 500 rather than a 401) - I'll look into this. In the meantime, can you check the memcached process? Ellie From: Ashish Maheshwari -X (ashimahe - L & T TECHNOLOGY SERVICES LIMITED at Cisco) [mailto:[email protected]] Sent: 20 February 2015 11:45 To: Eleanor Merry; [email protected]<mailto:[email protected]> Cc: Chinna Morampalle (cmorampa); Satheesh Marappan (sathkuma) Subject: SCSCF sending repetetive MAR to HSS Hi Ellie, I am facing the issue that SCSCF sending repetitive MAR request to HSS instead of SAR even though first MAA diameter success. I am describing the detailed call flow: UE----SIP Register -------PCSCF(Bono) PCSCF-----sip reg frwrd------SCSCF(sprout) SCSCF-------MAR------------HSS HSS-------MAA------------SCSCF SCSCF--------401 unaut------PCSCF PCSCF----------401 unaut----UE UE----SIP Register -------PCSCF(Bono) PCSCF-----sip reg frwrd------SCSCF(sprout) SCSCF-------MAR------------HSS (instead of SAR) I am attaching pcap,monit status, config,bono and sprout logs file. Regards, Ashish Thanks & Regards, Ashish Maheshwari _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater
