Hi, Agalya!

As you have already noticed, the timeout fix has made its way to 2.2, so from my perspective, all should be good now regarding the previous issues.


Your tests are very interesting, but I need more info in order to isolate the cause of the crash. Here are some things to consider:

- first and foremost: keep in mind that it's difficult to help debug code that I cannot see (REST PUT). If you were to make it crash while doing 10,000 REST POSTs, I would be able to jump in and attempt to replicate the crash.

- do your workers have enough PKG memory? (use "opensipsctl fifo get_statistics pkmem:" during runtime to find out)

- 10,000 calls is a nice number, but what was your CPS rate? Also, how many OpenSIPS processes did you run with?

- I noticed you enabled QM_MALLOC, which is good, but did you also enable DBG_MALLOC for it? This would help catch any memory corruption earliest possible.

Best regards,

Liviu Chircu
OpenSIPS Developer
http://www.opensips-solutions.com

On 10.10.2016 20:09, Ramachandran, Agalya (Contractor) wrote:
Hi Liviu/team,

There is one item pending from my mail thread - opensips crash when doing load 
test.
Please find the details of that message below.

Hi Bogdan,

When I try to apply patch, I see all the code are already available in 2.2. So 
I did not add any patch.
But did a code addition for 'REST_PUT' request in rest_client.c, 
rest_methods.c, rest_methohs.h
Tried to perform load test. Observed 2 issues.

1)      Tried to load 10,000 calls - But route[resume_http] is called only for 
9985 calls.
Every time approximately 10-20 calls, route[resume_http]  is not called. But if 
I see the tcpdump, I am seeing 10,000 HTTP request and 10,000 HTTP 200 OK 
responses.
When printing the response in resume_http for every call-id, 10-20 calls 
response is not printed - which means resume is not called for these calls.
Am not filtering any response code.
route[resume_http] {
         xlog("L_INFO","Resp $rc in HTTP PUT!\n");
         xlog("L_INFO","route[relay] The content received from SM for $rm: 
[callId=$ci] : $var(body) in HTTP PUT\n");
         .....................
}
    For first few calls, $rc is 1(ASYNC_DONE), for others $rc is -5(ASYNC_SYNC)

2)      Tried to load more than 10,000 calls. When it reaches 25,000 calls 
after that some point opensips crashes.
I have two core dump generated as of now.
Coredump1:
--------------
(gdb) bt
#0  0x00007f8defacc840 in Curl_num_addresses () from /lib64/libcurl.so.4
#1  0x00007f8defaf09ee in Curl_connecthost () from /lib64/libcurl.so.4
#2  0x00007f8defae34ff in Curl_setup_conn () from /lib64/libcurl.so.4
#3  0x00007f8defae370c in Curl_connect () from /lib64/libcurl.so.4
#4  0x00007f8defaf32c0 in multi_runsingle () from /lib64/libcurl.so.4
#5  0x00007f8defaf4121 in curl_multi_perform () from /lib64/libcurl.so.4
#6  0x00007f8defd2b65b in start_async_http_req (msg=msg@entry=0x7f8df3740fc0, 
method=method@entry=REST_CLIENT_PUT,
     url=0x7f8df36ceb88 
"http://test.comcast.net/RTCGSessionManager/rest/tel/session/createroom?";, 
req_body=<optimized out>,
     req_ctype=<optimized out>, out_handle=out_handle@entry=0x7f8df3743300, 
body=body@entry=0x7f8df3743308, ctype=0x7f8df3743318)
     at rest_methods.c:265
#7  0x00007f8defd31b56 in w_async_rest_put (msg=0x7f8df3740fc0, 
resume_f=0x7ffd4afd8c90, resume_param=0x7ffd4afd8c88,
     gp_url=<optimized out>, gp_body=<optimized out>, gp_ctype=<optimized out>, 
body_pv=0x7f8df36f85a8 "N",
     ctype_pv=0x7f8df36f8700 "N", code_pv=0x7f8df36f88c0 "N") at 
rest_client.c:544
#8  0x00007f8df1031841 in t_handle_async (msg=0x7f8df3740fc0, a=0x7f8df36cf0c8, 
resume_route=2) at async.c:240
#9  0x0000000000424056 in do_action (a=0x7f8df36cf2d0, msg=0x7f8df3740fc0) at 
action.c:1863
#10 0x000000000041c330 in run_action_list (a=0x7f8df36cf2d0, 
msg=0x7f8df3740fc0) at action.c:172
#11 0x0000000000420637 in do_action (a=0x7f8df36cf3f8, msg=0x7f8df3740fc0) at 
action.c:1108
#12 0x000000000041c330 in run_action_list (a=0x7f8df36ce568, 
msg=0x7f8df3740fc0) at action.c:172
#13 0x000000000041c1fd in run_actions (a=0x7f8df36ce568, msg=0x7f8df3740fc0) at 
action.c:137
#14 0x000000000041ed55 in do_action (a=0x7f8df36ca080, msg=0x7f8df3740fc0) at 
action.c:745
#15 0x000000000041c330 in run_action_list (a=0x7f8df36c9e78, 
msg=0x7f8df3740fc0) at action.c:172
#16 0x0000000000420637 in do_action (a=0x7f8df36ca1a8, msg=0x7f8df3740fc0) at 
action.c:1108
#17 0x000000000041c330 in run_action_list (a=0x7f8df36c2560, 
msg=0x7f8df3740fc0) at action.c:172
#18 0x000000000041c1fd in run_actions (a=0x7f8df36c2560, msg=0x7f8df3740fc0) at 
action.c:137
#19 0x000000000041c3fd in run_top_route (a=0x7f8df36c2560, msg=0x7f8df3740fc0) 
at action.c:204
#20 0x000000000042bb64 in receive_msg (
     buf=0x7eb340 <buf.8031> "INVITE sip:++12001000004@10.0.0.1:5060 SIP/2.0\r\nTo: sut 
<sip:+19086774567@10.0.0.0.1:5060;user=phone>\r\nFrom: sipp 
<sip:sipp@10.0.0.2:5060;user=phone>;tag=26939SIPpTag0014404\r\nCall-ID: 14"..., len=837,
     rcv_info=0x7ffd4afdae00, existing_context=0x0) at receive.c:208
#21 0x0000000000521838 in udp_read_req (si=0x7f8df36bda50, 
bytes_read=0x7ffd4afdaec8) at net/proto_udp/proto_udp.c:192
#22 0x000000000050d05b in handle_io (fm=0x7f8df371e128, idx=0, event_type=1) at 
net/net_udp.c:259
#23 0x000000000050ba2b in io_wait_loop_epoll (h=0x80be40 <_worker_io>, t=1, 
repeat=0) at net/../io_wait_loop.h:221
#24 0x000000000050d3db in udp_rcv_loop (si=0x7f8df36bda50) at net/net_udp.c:311
#25 0x000000000050d973 in udp_start_processes (chd_rank=0x7d8128 
<chd_rank.10725>, startup_done=0x0) at net/net_udp.c:375
#26 0x0000000000495241 in main_loop () at main.c:671
#27 0x0000000000497cec in main (argc=3, argv=0x7ffd4afdb1c8) at main.c:1271

Coredump2:
--------------
(gdb) bt
#0  0x00000000004affeb in qm_detach_free (qm=0x7f5aa3ff4010, 
frag=0x7f5aa40c50f0) at mem/q_malloc.c:281
#1  0x00000000004b1233 in qm_free (qm=0x7f5aa3ff4010, p=0x7f5aa40c50b0, 
file=0x7f5aa069fbf5 "rest_client.c",
     func=0x7f5aa06a01fd <__FUNCTION__.8581> "osips_free", line=205) at 
mem/q_malloc.c:494
#2  0x00007f5aa04774f6 in curl_thread_create_thunk () from /lib64/libcurl.so.4
#3  0x00007f5a9edb9dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f5aa42eb28d in clone () from /lib64/libc.so.6

Regards,
Agalya



-----Original Message-----
From: users-boun...@lists.opensips.org 
[mailto:users-boun...@lists.opensips.org] On Behalf Of Liviu Chircu
Sent: Monday, October 10, 2016 4:53 AM
To: OpenSIPS devel mailling list <de...@lists.opensips.org>; OpenSIPS users mailling 
list <users@lists.opensips.org>
Subject: [OpenSIPS-Users] Pending OpenSIPS minor releases: Last minute bug 
fixes!

Hi, everyone!

A new series of OpenSIPS minor releases is planned for next week, October 19th.

If you have any pending GitHub issues / mailing list bug-threads in OpenSIPS 
1.11, 2.1 or 2.2 which are still not resolved, this would be a good time to 
bump them!

Best regards,

--
Liviu Chircu
OpenSIPS Developer
http://www.opensips-solutions.com


_______________________________________________
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Reply via email to