Hi Ashish,
Can you send across your memcached cluster setting file (in
/etc/clearwater/cluster_settings) and the most recent sprout log? To get this,
can you restart sprout (use 'sudo service sprout stop'), attempt a register,
and then send just the most recent /var/log/sprout/sprout_* file.
Also, can you confirm that the memcached service is running ('sudo service
memcached status'), and remonitor it with monit ('sudo monit monitor
memcached').
Restund is our STUN/TURN server, and it's used by Bono; poll_restund is a
script that monit uses to check that the restund process is correctly
responding to requests. Is this still failing in monit? Can you send over the
/var/log/monit.log file?
Thanks,
Ellie
From: Ashish Maheshwari -X (ashimahe - L & T TECHNOLOGY SERVICES LIMITED at
Cisco) [mailto:[email protected]]
Sent: 25 February 2015 13:25
To: Eleanor Merry; [email protected]
Cc: Chinna Morampalle (cmorampa); Satheesh Marappan (sathkuma); Anil Jain -X
(anilja - Aricent Technology Holdings Limited at Cisco)
Subject: FW: SCSCF sending repetetive MAR to HSS
Hi Ellie,
I completely re -installed the latest version of CW IMS OVF and did the
integration with external HSS but after that when I check the status of all the
services I found :
Program 'poll_memcached'
status Status failed
monitoring status Monitored
last started Wed, 25 Feb 2015 05:19:21
last exit value 1
data collected Wed, 25 Feb 2015 05:19:21
Process 'memcached'
status Not monitored
monitoring status Not monitored
data collected Wed, 25 Feb 2015 03:44:07
Program 'poll_restund'
status Status failed
monitoring status Monitored
last started Wed, 25 Feb 2015 05:45:28
last exit value 1
data collected Wed, 25 Feb 2015 05:45:28
What is this poll_restund process and how we can make it running....
To trouble shoot this memcached issue I took help from your online help :
Memcached logs to /var/log/memcached.log. It logs very little by default, but
it is possible to make it more verbose by editing /etc/memcached_11211.conf,
uncommenting the -vvline, and running sudo monit restart memcached.
In attached memcached logs in I can see :
failed to listen on TCP port 11211: Invalid argument
getaddrinfo(): Temporary failure in name resolution
I also restart sprout service to capture the logs to see when sprout initially
communicates to memcached but didn't get too much information from it.
I used the attached config file and also attaching pcap capture ,syslog ,
config file and memcached logs not able to attached sprout logs as huge in size
but I am seeing the same error as you mentioned in the below mail.
20-02-2015 04:07:13.921 UTC Debug memcachedstore.cpp:649: memcached_add command
for av\\[email protected]\46fa0be12dbafeef failed on replica 0, rc =
47 (SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY), expiry = 40
(140124113689584) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY, host:
10.105.244.117:11211 -> libmemcached/connect.cc:633
20-02-2015 04:07:13.921 UTC Error memcachedstore.cpp:708: Failed to write data
for av\\[email protected]\46fa0be12dbafeef to 1 replicas
20-02-2015 04:07:13.921 UTC Error avstore.cpp:73: Failed to write
Authentication Vector for private_id
[email protected]<mailto:[email protected]>
As memcached service is not working I am still facing repetitive MAR from SCSF
to HSS.
Can you please help me to resolve this issue on priority as we totally blocked.
Thanks in advance.
Thanks & Regards,
Ashish Maheshwari
From: Eleanor Merry [mailto:[email protected]]
Sent: 20 February 2015 19:11
To: Ashish Maheshwari -X (ashimahe - L & T TECHNOLOGY SERVICES LIMITED at
Cisco);
[email protected]<mailto:[email protected]>
Cc: Chinna Morampalle (cmorampa); Satheesh Marappan (sathkuma)
Subject: RE: SCSCF sending repetetive MAR to HSS
Hi Ashish,
It looks like there's a problem with memcached - from the Sprout logs:
20-02-2015 04:07:13.921 UTC Debug memcachedstore.cpp:649: memcached_add command
for av\\[email protected]\46fa0be12dbafeef failed on replica 0, rc =
47 (SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY), expiry = 40
(140124113689584) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY, host:
10.105.244.117:11211 -> libmemcached/connect.cc:633
20-02-2015 04:07:13.921 UTC Error memcachedstore.cpp:708: Failed to write data
for av\\[email protected]\46fa0be12dbafeef to 1 replicas
20-02-2015 04:07:13.921 UTC Error avstore.cpp:73: Failed to write
Authentication Vector for private_id
[email protected]<mailto:[email protected]>
Are there any memcached logs in /var/log/memcached.log or in /var/log/syslog?
Also, can you check if there are any reported errors about memcached when
Sprout initially attempts to connect to memcached - you'll need to restart
Sprout ('sudo service sprout stop' - it'll be restarted by monit).
The memcached error is causing the repetitive MARs because when the S-CSCF
receives the REGISTER for the P-CSCF, it first checks its authorization store
for the subscriber. On the initial REGISTER this is empty, so the S-CSCF makes
a MAR to the HSS. It then writes the returned auth information into the store
(this step is failing in your test), and returns a 401 to the P-CSCF. Then when
the S-CSCF receives the REGISTER with authorization, it checks this against its
stored information. However in your case, Sprout looks up the subscriber in the
auth store, doesn't find anything (as the write before failed), so makes a MAR
to the HSS.
We should probably return a better code to the P-CSCF if the write fails (a 500
rather than a 401) - I'll look into this. In the meantime, can you check the
memcached process?
Ellie
From: Ashish Maheshwari -X (ashimahe - L & T TECHNOLOGY SERVICES LIMITED at
Cisco) [mailto:[email protected]]
Sent: 20 February 2015 11:45
To: Eleanor Merry;
[email protected]<mailto:[email protected]>
Cc: Chinna Morampalle (cmorampa); Satheesh Marappan (sathkuma)
Subject: SCSCF sending repetetive MAR to HSS
Hi Ellie,
I am facing the issue that SCSCF sending repetitive MAR request to HSS instead
of SAR even though first MAA diameter success.
I am describing the detailed call flow:
UE----SIP Register -------PCSCF(Bono)
PCSCF-----sip reg frwrd------SCSCF(sprout)
SCSCF-------MAR------------HSS
HSS-------MAA------------SCSCF
SCSCF--------401 unaut------PCSCF
PCSCF----------401 unaut----UE
UE----SIP Register -------PCSCF(Bono)
PCSCF-----sip reg frwrd------SCSCF(sprout)
SCSCF-------MAR------------HSS (instead of SAR)
I am attaching pcap,monit status, config,bono and sprout logs file.
Regards,
Ashish
Thanks & Regards,
Ashish Maheshwari
_______________________________________________
Clearwater mailing list
[email protected]
http://lists.projectclearwater.org/listinfo/clearwater
