Hello James,
I'm ccing this reply to the mailing list, for which I hope you don't
mind. You have been very helpful. I have seen similar REGISTER problems
from others earlier. Your answers could be useful for them who may
encounter the same issue regarding to REGSITER 403 error. Also a bit
more inputs from the community may be helpful.
1) On homestead node, the access log shows that there is no GET
in-coming between the time 27-03-2014 15 <tel:27-03-2014%2015>:28:34.713
(sprout Sending HTTP request ) and 27-03-2014 15:28:34.833 (Received
HTTP error response - Time out):
27-03-2014 15:28:16.619 200 GET /ping
27-03-2014 15:28:26.605 200 GET /ping
27-03-2014 15:28:36.595 200 GET /ping
27-03-2014 15:28:46.613 200 GET /ping
2) How to configure homestead to get impi/impu? and what's impi/impu,
are these two tables, if you please elaborate? I didn't find where these
two are mentioned in the document on github.
3) I didn't use HSS. I understand that Homestead is only an HSS mirror.
But without HSS, homestead still works (at least for previous releases
before YoJimMo), for which I believe hs stores the user authentication
vector locally, without diametering it to the HSS.
4) After I ran cqlsh on homestead box, `use homestead_cache;` and
`select * from impi;` `select * from impu;` , here is the output
(nothing in impi nor in impu):
[homestead]ubuntu@hs:/var/log/homestead$ cqlsh
Connected to Test Cluster at localhost:9160.
[cqlsh 3.1.8 | Cassandra 1.2.12 | CQL spec 3.0.0 | Thrift protocol 19.36.1]
Use HELP for help.
cqlsh> use homestead_cache
... select * from impi
... select * from impu
...
I guess among the above 4 steps, 1) says the most obvious: homestead
didn't receive GET request from sprout.
After the following further attempts, I'm convinced that homestead
didn't receive the GET is because sprout didn't initiate the TCP connection:
a) [homestead]ubuntu@hs:/var/log/homestead$ sudo netstat | grep hs:
Among the list of network connections, there is no tcp connection
between sprout and hs.
b)
[sprout]ubuntu@sprout:/var/log/sprout$ nslookup hs.clearwater.demo
Server: 127.0.0.1
Address: 127.0.0.1#53
Non-authoritative answer:
Name: hs.clearwater.demo
Address: 10.8.0.20
c)
[sprout]ubuntu@sprout:/var/log/sprout$ telnet hs.clearwater.demo 8888
Trying 10.8.0.20...
Connected to hs.clearwater.demo.
Escape character is '^]'.
Regards,
Lilin
On 27/03/2014 12:01 PM, James Coleman wrote:
Humm. Your monit status does look okay.
Maybe also check homestead/access_<date>_<time>.log
To see if sprout GET does hit homestead at that time.
The messages in homestead at time 15:28:35.148 do seem to be at the
right time and probably caused by the sprout message ?
How have you configured homestead to get impi/impu?
I use an open IMS core HSS separate box.
If you are using a HSS like this then tcpdump capturing all on
DIAMETER port 3868 is useful.
Also run cqlsh on homestead box, `use homestead_cache;` and `select *
from impi;` `select * from impu;`
maybe clearing out impu entries would help . . .
depending on how your hss works . . .
James.
On 27 March 2014 15:35, Lilin <[email protected]
<mailto:[email protected]>> wrote:
Hello James,
Thanks for your quick reply!
The "sudo monit status all" gives the following, I believe they
are all ok status:
Program 'poll_homestead'
status Status ok
Process 'homestead'
status Running
Program 'poll_homestead-prov'
status Status ok
Process 'cassandra'
status Running
System 'system_hs.clearwater.demo'
status Running
I didn the REGISTER again at this time instant: 27-03-2014
15:28:34.833
27-03-2014 15 <tel:27-03-2014%2015>:28:34.713 Debug
httpconnection.cpp:335: Sending HTTP request :
http://hs.clearwater.demo:8888/impi/6505550373%40clearwater.demo/av?impu=sip%3A6505550373%40clearwater.demo
(try 1) on new connection
27-03-2014 15 <tel:27-03-2014%2015>:28:34.833 Debug
httpconnection.cpp:358: Received HTTP error response :
http://hs.clearwater.demo:8888/impi/6505550373%40clearwater.demo/av?impu=sip%3A6505550373%40clearwater.demo
: Timeout was reached
Around this time instant, here is what homestead's log says:
27-03-2014 15 <tel:27-03-2014%2015>:28:26.605 Debug
zmq_lvc.cpp:251: Clearing message cache for 0x7fdfe80008c0
27-03-2014 15 <tel:27-03-2014%2015>:28:26.605 Debug
zmq_lvc.cpp:168: Update to H_incoming_requests statistic
27-03-2014 15 <tel:27-03-2014%2015>:28:26.605 Debug
zmq_lvc.cpp:251: Clearing message cache for 0x7fdfe8017a80
27-03-2014 15 <tel:27-03-2014%2015>:28:35.148 Debug freeDiameter:
'STATE_CLOSED' <-- 'FDEVP_PSM_TIMEOUT' ((nil),0) ''
27-03-2014 15 <tel:27-03-2014%2015>:28:35.148 Debug freeDiameter:
: Connecting...
27-03-2014 15 <tel:27-03-2014%2015>:28:35.148 Debug freeDiameter:
'STATE_CLOSED' -> 'STATE_WAITCNXACK' ''
27-03-2014 15 <tel:27-03-2014%2015>:28:35.148 Debug freeDiameter:
Peer timeout reset to 10 seconds
27-03-2014 15 <tel:27-03-2014%2015>:28:35.148 Debug freeDiameter:
'' in state 'STATE_WAITCNXACK' waiting for next event.
Then you'll notice that there is a 7-second time gap around that
time instant, between 27-03-2014 15
<tel:27-03-2014%2015>:28:26.605 and 27-03-2014 15:28:35.148
To my understanding, it means that homestead didn't receive
anything from sprout, that's why homestead didn't have any log
between the 7 seconds window where sprout sent request on
27-03-2014 15 <tel:27-03-2014%2015>:28:34.713 and was returned
with a time out error at 27-03-2014 15:28:34.833 (120 milliseconds
after)
Regards,
Lilin
On 27/03/2014 11:09 AM, James Coleman wrote:
I like the telnet :)
Crude but effective.
I think your config is okay.
Sprout is doing the right thing.
Even on homestead there is a service . . . but . . . homestead
doesn't return.
What does `sudo monit status all` on homestead box show?
Is all okay with homestead, cassandra, chronos e.t.c. ?
If not `sudo monit restart all`
AND what is in homestead log at the time from sprout sending
request to sprout getting timeout?
James.
On 27 March 2014 14:53, Lilin <[email protected]
<mailto:[email protected]>> wrote:
Hello,
This email is an addition to my previous email reporting the
403 error for REGISTER message.
I narrow down the cause of the problem is that sprout is
unable to initiate a TCP connection to homestead:
On sprout node, I ran telnet hs.clearwater.demo 8888, the
outputs are:
Trying 10.8.0.20...
Connected to hs.clearwater.demo.
Escape character is '^]'.
While telnet connection is active, on homestead, I ran sudo
netstat -anp |grep :8888, the output contains:
10.8.0.20:8888 <http://10.8.0.20:8888> 10.8.0.18:38373
<http://10.8.0.18:38373> ESTABLISHED
This rules out the possible causes, such as network
disconnection condition or port restriction on either of the
two nodes in question here.
Also after I terminate the telnet connection on sprout node,
then check the netstat on homestead node, the TCP connection
between sprout and homestead is gone...
So now the problem (sprout TIME OUT when reaching out to hs
-> 403 error returned for REGISTER) has been narrowed down to
that sprout is unable to establish the connection to
homestead. Since I have checked on sprout node that
/etc/clearwater/config has the following:
# Deployment definitions
home_domain=clearwater.demo
sprout_hostname=sprout.clearwater.demo
chronos_hostname=10.8.0.18:7253 <http://10.8.0.18:7253>
hs_hostname=hs.clearwater.demo:8888
hs_provisioning_hostname=hs.clearwater.demo:8889
xdms_hostname=homer.clearwater.demo:7888
I'd be really appreciated if you could give me some insights
on how to fix the 403 error. Thanks!
Regards,
Lilin
_______________________________________________
Clearwater mailing list
[email protected]
<mailto:[email protected]>
http://lists.projectclearwater.org/listinfo/clearwater
<http://www.linkedin.com/company/76647?trk=fc_badge>
openmindnetworks.com <http://openmindnetworks.com>
<http://www.linkedin.com/company/76647?trk=fc_badge>
openmindnetworks.com <http://openmindnetworks.com>
_______________________________________________
Clearwater mailing list
[email protected]
http://lists.projectclearwater.org/listinfo/clearwater
