Of course I don't mind. :) First answer was a bit speculative so I just sent it to you
Your problem symptoms looks a bit like when sprout talks to homestead okay but homestead can't talk to HSS okay. The sprout -> homestead GET is very quick so would not be easy to see an estabblished connection. Use tcpdump :-) capture port 8888 on sprout side and homestead side to see for sure . . . tcpdump -i any -s 0 -w /tmp/8888.pcap "tcp port 8888" -vvv Could you use curl to do the http GET? e.g. response that I get curl -v http://10.x.x.x:8888/impi/%2B3538999999%40openims.test/av?impu=sip%3A%2B3538999999%40openims.test {"digest":{"ha1":"b9a6fe5e0dc62bd017a603bdb8d0e59e","realm":"openims.test","qop":"auth"}} For my setup with homestead connecting to open IMS core HSS . . . when REGISTER is done . . . sprout does GET impi from homestead . . . . . . Homestead looks in cassandra impu or impi table first for entry. . . . And then goes out over DIAMETER to HSS if impu entry not found. . . . (impis not cached in cassandra as the authorisation is done with impis) In cqlsh you need a semicolon after each command: use homestead_cache; select * from impi; select * from impu; There is never anything in my impi table. I see the impu table contains the IFC xml for each impu which was previously registered. James On 27 March 2014 16:30, Lilin <[email protected]> wrote: > Hello James, > > I'm ccing this reply to the mailing list, for which I hope you don't mind. > You have been very helpful. I have seen similar REGISTER problems from > others earlier. Your answers could be useful for them who may encounter the > same issue regarding to REGSITER 403 error. Also a bit more inputs from the > community may be helpful. > > 1) On homestead node, the access log shows that there is no GET in-coming > between the time 27-03-2014 15 <27-03-2014%2015>:28:34.713 (sprout > Sending HTTP request ) and 27-03-2014 15:28:34.833 (Received HTTP error > response - Time out): > 27-03-2014 15:28:16.619 200 GET /ping > 27-03-2014 15:28:26.605 200 GET /ping > 27-03-2014 15:28:36.595 200 GET /ping > 27-03-2014 15:28:46.613 200 GET /ping > > 2) How to configure homestead to get impi/impu? and what's impi/impu, are > these two tables, if you please elaborate? I didn't find where these two > are mentioned in the document on github. > > 3) I didn't use HSS. I understand that Homestead is only an HSS mirror. > But without HSS, homestead still works (at least for previous releases > before YoJimMo), for which I believe hs stores the user authentication > vector locally, without diametering it to the HSS. > > 4) After I ran cqlsh on homestead box, `use homestead_cache;` and `select > * from impi;` `select * from impu;` , here is the output (nothing in impi > nor in impu): > [homestead]ubuntu@hs:/var/log/homestead$ cqlsh > Connected to Test Cluster at localhost:9160. > [cqlsh 3.1.8 | Cassandra 1.2.12 | CQL spec 3.0.0 | Thrift protocol 19.36.1] > Use HELP for help. > cqlsh> use homestead_cache > ... select * from impi > ... select * from impu > ... > > I guess among the above 4 steps, 1) says the most obvious: homestead > didn't receive GET request from sprout. > > After the following further attempts, I'm convinced that homestead didn't > receive the GET is because sprout didn't initiate the TCP connection: > a) [homestead]ubuntu@hs:/var/log/homestead$ sudo netstat | grep hs: > Among the list of network connections, there is no tcp connection between > sprout and hs. > > b) > [sprout]ubuntu@sprout:/var/log/sprout$ nslookup hs.clearwater.demo > Server: 127.0.0.1 > Address: 127.0.0.1#53 > > Non-authoritative answer: > Name: hs.clearwater.demo > Address: 10.8.0.20 > > c) > [sprout]ubuntu@sprout:/var/log/sprout$ telnet hs.clearwater.demo 8888 > > Trying 10.8.0.20... > Connected to hs.clearwater.demo. > Escape character is '^]'. > > Regards, > Lilin > > > On 27/03/2014 12:01 PM, James Coleman wrote: > > Humm. Your monit status does look okay. > > Maybe also check homestead/access_<date>_<time>.log > To see if sprout GET does hit homestead at that time. > > The messages in homestead at time 15:28:35.148 do seem to be at the > right time and probably caused by the sprout message ? > How have you configured homestead to get impi/impu? > I use an open IMS core HSS separate box. > If you are using a HSS like this then tcpdump capturing all on DIAMETER > port 3868 is useful. > > Also run cqlsh on homestead box, `use homestead_cache;` and `select * > from impi;` `select * from impu;` > maybe clearing out impu entries would help . . . > depending on how your hss works . . . > > > James. > > > > > > > On 27 March 2014 15:35, Lilin <[email protected]> wrote: > >> Hello James, >> >> Thanks for your quick reply! >> >> The "sudo monit status all" gives the following, I believe they are all >> ok status: >> Program 'poll_homestead' >> status Status ok >> Process 'homestead' >> status Running >> Program 'poll_homestead-prov' >> status Status ok >> Process 'cassandra' >> status Running >> System 'system_hs.clearwater.demo' >> status Running >> >> I didn the REGISTER again at this time instant: 27-03-2014 15:28:34.833 >> 27-03-2014 15:28:34.713 Debug httpconnection.cpp:335: Sending HTTP >> request : >> http://hs.clearwater.demo:8888/impi/6505550373%40clearwater.demo/av?impu=sip%3A6505550373%40clearwater.demo(try >> 1) on new connection >> 27-03-2014 15:28:34.833 Debug httpconnection.cpp:358: Received HTTP >> error response : >> http://hs.clearwater.demo:8888/impi/6505550373%40clearwater.demo/av?impu=sip%3A6505550373%40clearwater.demo: >> Timeout was reached >> >> Around this time instant, here is what homestead's log says: >> 27-03-2014 15:28:26.605 Debug zmq_lvc.cpp:251: Clearing message cache >> for 0x7fdfe80008c0 >> 27-03-2014 15:28:26.605 Debug zmq_lvc.cpp:168: Update to >> H_incoming_requests statistic >> 27-03-2014 15:28:26.605 Debug zmq_lvc.cpp:251: Clearing message cache >> for 0x7fdfe8017a80 >> 27-03-2014 15:28:35.148 Debug freeDiameter: 'STATE_CLOSED' <-- >> 'FDEVP_PSM_TIMEOUT' ((nil),0) '' >> 27-03-2014 15:28:35.148 Debug freeDiameter: : Connecting... >> 27-03-2014 15:28:35.148 Debug freeDiameter: 'STATE_CLOSED' -> >> 'STATE_WAITCNXACK' '' >> 27-03-2014 15:28:35.148 Debug freeDiameter: Peer timeout reset to 10 >> seconds >> 27-03-2014 15:28:35.148 Debug freeDiameter: '' in state >> 'STATE_WAITCNXACK' waiting for next event. >> >> Then you'll notice that there is a 7-second time gap around that time >> instant, between 27-03-2014 15:28:26.605 and 27-03-2014 15:28:35.148 >> To my understanding, it means that homestead didn't receive anything from >> sprout, that's why homestead didn't have any log between the 7 seconds >> window where sprout sent request on 27-03-2014 15:28:34.713 and was >> returned with a time out error at 27-03-2014 15:28:34.833 (120 milliseconds >> after) >> >> Regards, >> Lilin >> >> >> On 27/03/2014 11:09 AM, James Coleman wrote: >> >> I like the telnet :) >> Crude but effective. >> >> I think your config is okay. >> Sprout is doing the right thing. >> Even on homestead there is a service . . . but . . . homestead doesn't >> return. >> >> What does `sudo monit status all` on homestead box show? >> Is all okay with homestead, cassandra, chronos e.t.c. ? >> If not `sudo monit restart all` >> >> AND what is in homestead log at the time from sprout sending request to >> sprout getting timeout? >> >> James. >> >> >> On 27 March 2014 14:53, Lilin <[email protected]> wrote: >> >>> Hello, >>> >>> This email is an addition to my previous email reporting the 403 error >>> for REGISTER message. >>> >>> I narrow down the cause of the problem is that sprout is unable to >>> initiate a TCP connection to homestead: >>> >>> On sprout node, I ran telnet hs.clearwater.demo 8888, the outputs are: >>> Trying 10.8.0.20... >>> Connected to hs.clearwater.demo. >>> Escape character is '^]'. >>> >>> While telnet connection is active, on homestead, I ran sudo netstat >>> -anp |grep :8888, the output contains: >>> 10.8.0.20:8888 10.8.0.18:38373 ESTABLISHED >>> This rules out the possible causes, such as network disconnection >>> condition or port restriction on either of the two nodes in question here. >>> Also after I terminate the telnet connection on sprout node, then check >>> the netstat on homestead node, the TCP connection between sprout and >>> homestead is gone... >>> >>> So now the problem (sprout TIME OUT when reaching out to hs -> 403 error >>> returned for REGISTER) has been narrowed down to that sprout is unable to >>> establish the connection to homestead. Since I have checked on sprout node >>> that /etc/clearwater/config has the following: >>> # Deployment definitions >>> home_domain=clearwater.demo >>> sprout_hostname=sprout.clearwater.demo >>> chronos_hostname=10.8.0.18:7253 >>> hs_hostname=hs.clearwater.demo:8888 >>> hs_provisioning_hostname=hs.clearwater.demo:8889 >>> xdms_hostname=homer.clearwater.demo:7888 >>> >>> I'd be really appreciated if you could give me some insights on how to >>> fix the 403 error. Thanks! >>> >>> Regards, >>> Lilin >>> _______________________________________________ >>> Clearwater mailing list >>> [email protected] >>> http://lists.projectclearwater.org/listinfo/clearwater >>> >> >> >> >> >> <http://www.linkedin.com/company/76647?trk=fc_badge> >> >> openmindnetworks.com >> >> >> > > > > <http://www.linkedin.com/company/76647?trk=fc_badge> > > openmindnetworks.com > > > -- <http://www.linkedin.com/company/76647?trk=fc_badge> openmindnetworks.com
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater
