Hi Lilin, James, I think that the problem looks like requests from sprout just aren't getting to homestead. Project Clearwater doesn't need an HSS - if one isn't configured in the deployment then Homestead will only look in its cache. Can you do a network capture on homestead to confirm whether there is any traffic from sprout?
I'd like to see what response sprout receives from the DNS server during a register attempt. Can you please send me a network capture from sprout for ports 8888 and 53 (as a .pcap file), and the sprout debug logs for an attempted register? I'm asking this as telnet defaults to using only IPv4 addresses, but curl (which is what sprout uses when contacting homestead) defaults to using both IPv4 and IPv6 addresses. I wonder if there's an issue where sprout is doing both A and AAAA lookups for hs.clearwater.demo, but then times out waiting for the IPv6 address. To test this, can you please add a value for hs.clearwater.demo to your /etc/hosts file? This will override the value that the DNS server returns, so hopefully you will then be able to register (and if not then it narrows down where the issue can be). Can you let me know if the /etc/hosts change works, and send over the homestead/sprout captures when you have them? Ellie From: [email protected] [mailto:[email protected]] On Behalf Of James Coleman Sent: 27 March 2014 17:02 To: Lilin Cc: [email protected] Subject: Re: [Clearwater] [YoJimMo} sprout unable to initiate a tcp connection to homestead Of course I don't mind. :) First answer was a bit speculative so I just sent it to you Your problem symptoms looks a bit like when sprout talks to homestead okay but homestead can't talk to HSS okay. The sprout -> homestead GET is very quick so would not be easy to see an estabblished connection. Use tcpdump :-) capture port 8888 on sprout side and homestead side to see for sure . . . tcpdump -i any -s 0 -w /tmp/8888.pcap "tcp port 8888" -vvv Could you use curl to do the http GET? e.g. response that I get curl -v http://10.x.x.x:8888/impi/%2B3538999999%40openims.test/av?impu=sip%3A%2B3538999999%40openims.test {"digest":{"ha1":"b9a6fe5e0dc62bd017a603bdb8d0e59e","realm":"openims.test","qop":"auth"}} For my setup with homestead connecting to open IMS core HSS . . . when REGISTER is done . . . sprout does GET impi from homestead . . . . . . Homestead looks in cassandra impu or impi table first for entry. . . . And then goes out over DIAMETER to HSS if impu entry not found. . . . (impis not cached in cassandra as the authorisation is done with impis) In cqlsh you need a semicolon after each command: use homestead_cache; select * from impi; select * from impu; There is never anything in my impi table. I see the impu table contains the IFC xml for each impu which was previously registered. James On 27 March 2014 16:30, Lilin <[email protected]<mailto:[email protected]>> wrote: Hello James, I'm ccing this reply to the mailing list, for which I hope you don't mind. You have been very helpful. I have seen similar REGISTER problems from others earlier. Your answers could be useful for them who may encounter the same issue regarding to REGSITER 403 error. Also a bit more inputs from the community may be helpful. 1) On homestead node, the access log shows that there is no GET in-coming between the time 27-03-2014 15<tel:27-03-2014%2015>:28:34.713 (sprout Sending HTTP request ) and 27-03-2014 15:28:34.833 (Received HTTP error response - Time out): 27-03-2014 15<tel:27-03-2014%2015>:28:16.619 200 GET /ping 27-03-2014 15<tel:27-03-2014%2015>:28:26.605 200 GET /ping 27-03-2014 15<tel:27-03-2014%2015>:28:36.595 200 GET /ping 27-03-2014 15<tel:27-03-2014%2015>:28:46.613 200 GET /ping 2) How to configure homestead to get impi/impu? and what's impi/impu, are these two tables, if you please elaborate? I didn't find where these two are mentioned in the document on github. 3) I didn't use HSS. I understand that Homestead is only an HSS mirror. But without HSS, homestead still works (at least for previous releases before YoJimMo), for which I believe hs stores the user authentication vector locally, without diametering it to the HSS. 4) After I ran cqlsh on homestead box, `use homestead_cache;` and `select * from impi;` `select * from impu;` , here is the output (nothing in impi nor in impu): [homestead]ubuntu@hs:/var/log/homestead$ cqlsh Connected to Test Cluster at localhost:9160. [cqlsh 3.1.8 | Cassandra 1.2.12 | CQL spec 3.0.0 | Thrift protocol 19.36.1] Use HELP for help. cqlsh> use homestead_cache ... select * from impi ... select * from impu ... I guess among the above 4 steps, 1) says the most obvious: homestead didn't receive GET request from sprout. After the following further attempts, I'm convinced that homestead didn't receive the GET is because sprout didn't initiate the TCP connection: a) [homestead]ubuntu@hs:/var/log/homestead$ sudo netstat | grep hs: Among the list of network connections, there is no tcp connection between sprout and hs. b) [sprout]ubuntu@sprout:/var/log/sprout$ nslookup hs.clearwater.demo Server: 127.0.0.1 Address: 127.0.0.1#53 Non-authoritative answer: Name: hs.clearwater.demo Address: 10.8.0.20 c) [sprout]ubuntu@sprout:/var/log/sprout$ telnet hs.clearwater.demo 8888 Trying 10.8.0.20... Connected to hs.clearwater.demo. Escape character is '^]'. Regards, Lilin On 27/03/2014 12:01 PM, James Coleman wrote: Humm. Your monit status does look okay. Maybe also check homestead/access_<date>_<time>.log To see if sprout GET does hit homestead at that time. The messages in homestead at time 15:28:35.148 do seem to be at the right time and probably caused by the sprout message ? How have you configured homestead to get impi/impu? I use an open IMS core HSS separate box. If you are using a HSS like this then tcpdump capturing all on DIAMETER port 3868 is useful. Also run cqlsh on homestead box, `use homestead_cache;` and `select * from impi;` `select * from impu;` maybe clearing out impu entries would help . . . depending on how your hss works . . . James. On 27 March 2014 15:35, Lilin <[email protected]<mailto:[email protected]>> wrote: Hello James, Thanks for your quick reply! The "sudo monit status all" gives the following, I believe they are all ok status: Program 'poll_homestead' status Status ok Process 'homestead' status Running Program 'poll_homestead-prov' status Status ok Process 'cassandra' status Running System 'system_hs.clearwater.demo' status Running I didn the REGISTER again at this time instant: 27-03-2014 15<tel:27-03-2014%2015>:28:34.833 27-03-2014 15<tel:27-03-2014%2015>:28:34.713 Debug httpconnection.cpp:335: Sending HTTP request : http://hs.clearwater.demo:8888/impi/6505550373%40clearwater.demo/av?impu=sip%3A6505550373%40clearwater.demo (try 1) on new connection 27-03-2014 15<tel:27-03-2014%2015>:28:34.833 Debug httpconnection.cpp:358: Received HTTP error response : http://hs.clearwater.demo:8888/impi/6505550373%40clearwater.demo/av?impu=sip%3A6505550373%40clearwater.demo : Timeout was reached Around this time instant, here is what homestead's log says: 27-03-2014 15<tel:27-03-2014%2015>:28:26.605 Debug zmq_lvc.cpp:251: Clearing message cache for 0x7fdfe80008c0 27-03-2014 15<tel:27-03-2014%2015>:28:26.605 Debug zmq_lvc.cpp:168: Update to H_incoming_requests statistic 27-03-2014 15<tel:27-03-2014%2015>:28:26.605 Debug zmq_lvc.cpp:251: Clearing message cache for 0x7fdfe8017a80 27-03-2014 15<tel:27-03-2014%2015>:28:35.148 Debug freeDiameter: 'STATE_CLOSED' <-- 'FDEVP_PSM_TIMEOUT' ((nil),0) '' 27-03-2014 15<tel:27-03-2014%2015>:28:35.148 Debug freeDiameter: : Connecting... 27-03-2014 15<tel:27-03-2014%2015>:28:35.148 Debug freeDiameter: 'STATE_CLOSED' -> 'STATE_WAITCNXACK' '' 27-03-2014 15<tel:27-03-2014%2015>:28:35.148 Debug freeDiameter: Peer timeout reset to 10 seconds 27-03-2014 15<tel:27-03-2014%2015>:28:35.148 Debug freeDiameter: '' in state 'STATE_WAITCNXACK' waiting for next event. Then you'll notice that there is a 7-second time gap around that time instant, between 27-03-2014 15<tel:27-03-2014%2015>:28:26.605 and 27-03-2014 15<tel:27-03-2014%2015>:28:35.148 To my understanding, it means that homestead didn't receive anything from sprout, that's why homestead didn't have any log between the 7 seconds window where sprout sent request on 27-03-2014 15<tel:27-03-2014%2015>:28:34.713 and was returned with a time out error at 27-03-2014 15:28:34.833 (120 milliseconds after) Regards, Lilin On 27/03/2014 11:09 AM, James Coleman wrote: I like the telnet :) Crude but effective. I think your config is okay. Sprout is doing the right thing. Even on homestead there is a service . . . but . . . homestead doesn't return. What does `sudo monit status all` on homestead box show? Is all okay with homestead, cassandra, chronos e.t.c. ? If not `sudo monit restart all` AND what is in homestead log at the time from sprout sending request to sprout getting timeout? James. On 27 March 2014 14:53, Lilin <[email protected]<mailto:[email protected]>> wrote: Hello, This email is an addition to my previous email reporting the 403 error for REGISTER message. I narrow down the cause of the problem is that sprout is unable to initiate a TCP connection to homestead: On sprout node, I ran telnet hs.clearwater.demo 8888, the outputs are: Trying 10.8.0.20... Connected to hs.clearwater.demo. Escape character is '^]'. While telnet connection is active, on homestead, I ran sudo netstat -anp |grep :8888, the output contains: 10.8.0.20:8888<http://10.8.0.20:8888> 10.8.0.18:38373<http://10.8.0.18:38373> ESTABLISHED This rules out the possible causes, such as network disconnection condition or port restriction on either of the two nodes in question here. Also after I terminate the telnet connection on sprout node, then check the netstat on homestead node, the TCP connection between sprout and homestead is gone... So now the problem (sprout TIME OUT when reaching out to hs -> 403 error returned for REGISTER) has been narrowed down to that sprout is unable to establish the connection to homestead. Since I have checked on sprout node that /etc/clearwater/config has the following: # Deployment definitions home_domain=clearwater.demo sprout_hostname=sprout.clearwater.demo chronos_hostname=10.8.0.18:7253<http://10.8.0.18:7253> hs_hostname=hs.clearwater.demo:8888 hs_provisioning_hostname=hs.clearwater.demo:8889 xdms_hostname=homer.clearwater.demo:7888 I'd be really appreciated if you could give me some insights on how to fix the 403 error. Thanks! Regards, Lilin _______________________________________________ Clearwater mailing list [email protected]<mailto:[email protected]> http://lists.projectclearwater.org/listinfo/clearwater [Image removed by sender.] [Image removed by sender.]<http://www.linkedin.com/company/76647?trk=fc_badge> openmindnetworks.com<http://openmindnetworks.com> [Image removed by sender.] [Image removed by sender.]<http://www.linkedin.com/company/76647?trk=fc_badge> openmindnetworks.com<http://openmindnetworks.com> [Image removed by sender.] [Image removed by sender.]<http://www.linkedin.com/company/76647?trk=fc_badge> openmindnetworks.com<http://openmindnetworks.com>
<<inline: ~WRD269.jpg>>
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater
