Hi Bogdan,
When I try to apply patch, I see all the code are already available in 2.2. So
I did not add any patch.
But did a code addition for 'REST_PUT' request in rest_client.c,
rest_methods.c, rest_methohs.h
Tried to perform load test. Observed 2 issues.
1) Tried to load 10,000 calls - But route[resume_http] is called only for
9985 calls.
Every time approximately 10-20 calls, route[resume_http] is not called. But if
I see the tcpdump, I am seeing 10,000 HTTP request and 10,000 HTTP 200 OK
responses.
When printing the response in resume_http for every call-id, 10-20 calls
response is not printed - which means resume is not called for these calls.
Am not filtering any response code.
route[resume_http] {
xlog("L_INFO","Resp $rc in HTTP PUT!\n");
xlog("L_INFO","route[relay] The content received from SM for $rm:
[callId=$ci] : $var(body) in HTTP PUT\n");
.....................
}
For first few calls, $rc is 1(ASYNC_DONE), for others $rc is
-5(ASYNC_SYNC)
2) Tried to load more than 10,000 calls. When it reaches 25,000 calls
after that some point opensips crashes.
I have two core dump generated as of now.
Coredump1:
--------------
(gdb) bt
#0 0x00007f8defacc840 in Curl_num_addresses () from /lib64/libcurl.so.4
#1 0x00007f8defaf09ee in Curl_connecthost () from /lib64/libcurl.so.4
#2 0x00007f8defae34ff in Curl_setup_conn () from /lib64/libcurl.so.4
#3 0x00007f8defae370c in Curl_connect () from /lib64/libcurl.so.4
#4 0x00007f8defaf32c0 in multi_runsingle () from /lib64/libcurl.so.4
#5 0x00007f8defaf4121 in curl_multi_perform () from /lib64/libcurl.so.4
#6 0x00007f8defd2b65b in start_async_http_req (msg=msg@entry=0x7f8df3740fc0,
method=method@entry=REST_CLIENT_PUT,
url=0x7f8df36ceb88
"http://test.comcast.net/RTCGSessionManager/rest/tel/session/createroom?";,
req_body=<optimized out>,
req_ctype=<optimized out>, out_handle=out_handle@entry=0x7f8df3743300,
body=body@entry=0x7f8df3743308, ctype=0x7f8df3743318)
at rest_methods.c:265
#7 0x00007f8defd31b56 in w_async_rest_put (msg=0x7f8df3740fc0,
resume_f=0x7ffd4afd8c90, resume_param=0x7ffd4afd8c88,
gp_url=<optimized out>, gp_body=<optimized out>, gp_ctype=<optimized out>,
body_pv=0x7f8df36f85a8 "N",
ctype_pv=0x7f8df36f8700 "N", code_pv=0x7f8df36f88c0 "N") at
rest_client.c:544
#8 0x00007f8df1031841 in t_handle_async (msg=0x7f8df3740fc0, a=0x7f8df36cf0c8,
resume_route=2) at async.c:240
#9 0x0000000000424056 in do_action (a=0x7f8df36cf2d0, msg=0x7f8df3740fc0) at
action.c:1863
#10 0x000000000041c330 in run_action_list (a=0x7f8df36cf2d0,
msg=0x7f8df3740fc0) at action.c:172
#11 0x0000000000420637 in do_action (a=0x7f8df36cf3f8, msg=0x7f8df3740fc0) at
action.c:1108
#12 0x000000000041c330 in run_action_list (a=0x7f8df36ce568,
msg=0x7f8df3740fc0) at action.c:172
#13 0x000000000041c1fd in run_actions (a=0x7f8df36ce568, msg=0x7f8df3740fc0) at
action.c:137
#14 0x000000000041ed55 in do_action (a=0x7f8df36ca080, msg=0x7f8df3740fc0) at
action.c:745
#15 0x000000000041c330 in run_action_list (a=0x7f8df36c9e78,
msg=0x7f8df3740fc0) at action.c:172
#16 0x0000000000420637 in do_action (a=0x7f8df36ca1a8, msg=0x7f8df3740fc0) at
action.c:1108
#17 0x000000000041c330 in run_action_list (a=0x7f8df36c2560,
msg=0x7f8df3740fc0) at action.c:172
#18 0x000000000041c1fd in run_actions (a=0x7f8df36c2560, msg=0x7f8df3740fc0) at
action.c:137
#19 0x000000000041c3fd in run_top_route (a=0x7f8df36c2560, msg=0x7f8df3740fc0)
at action.c:204
#20 0x000000000042bb64 in receive_msg (
buf=0x7eb340 <buf.8031> "INVITE sip:[email protected]:5060
SIP/2.0\r\nTo: sut <sip:[email protected]:5060;user=phone>\r\nFrom: sipp
<sip:[email protected]:5060;user=phone>;tag=26939SIPpTag0014404\r\nCall-ID: 14"...,
len=837,
rcv_info=0x7ffd4afdae00, existing_context=0x0) at receive.c:208
#21 0x0000000000521838 in udp_read_req (si=0x7f8df36bda50,
bytes_read=0x7ffd4afdaec8) at net/proto_udp/proto_udp.c:192
#22 0x000000000050d05b in handle_io (fm=0x7f8df371e128, idx=0, event_type=1) at
net/net_udp.c:259
#23 0x000000000050ba2b in io_wait_loop_epoll (h=0x80be40 <_worker_io>, t=1,
repeat=0) at net/../io_wait_loop.h:221
#24 0x000000000050d3db in udp_rcv_loop (si=0x7f8df36bda50) at net/net_udp.c:311
#25 0x000000000050d973 in udp_start_processes (chd_rank=0x7d8128
<chd_rank.10725>, startup_done=0x0) at net/net_udp.c:375
#26 0x0000000000495241 in main_loop () at main.c:671
#27 0x0000000000497cec in main (argc=3, argv=0x7ffd4afdb1c8) at main.c:1271
Coredump2:
--------------
(gdb) bt
#0 0x00000000004affeb in qm_detach_free (qm=0x7f5aa3ff4010,
frag=0x7f5aa40c50f0) at mem/q_malloc.c:281
#1 0x00000000004b1233 in qm_free (qm=0x7f5aa3ff4010, p=0x7f5aa40c50b0,
file=0x7f5aa069fbf5 "rest_client.c",
func=0x7f5aa06a01fd <__FUNCTION__.8581> "osips_free", line=205) at
mem/q_malloc.c:494
#2 0x00007f5aa04774f6 in curl_thread_create_thunk () from /lib64/libcurl.so.4
#3 0x00007f5a9edb9dc5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f5aa42eb28d in clone () from /lib64/libc.so.6
Regards,
Agalya
From: Bogdan-Andrei Iancu [mailto:[email protected]]
Sent: Wednesday, October 05, 2016 3:58 PM
To: Ramachandran, Agalya (Contractor) <[email protected]>;
OpenSIPS users mailling list <[email protected]>
Subject: Re: opensips crash when doing load test
That's a good news :)
Try to add one patch at a time (incrementally) and let's see when it starts to
crash again.
Thanks and regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
On 05.10.2016 22:55, Ramachandran, Agalya (Contractor) wrote:
Hi Bogdan,
With 2.2 version, am not seeing any crash. All looks good.
Regards,
Agalya
From: [email protected]<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Ramachandran, Agalya
(Contractor)
Sent: Wednesday, October 05, 2016 10:50 AM
To: Bogdan-Andrei Iancu <[email protected]><mailto:[email protected]>;
OpenSIPS users mailling list
<[email protected]><mailto:[email protected]>
Subject: Re: [OpenSIPS-Users] opensips crash when doing load test
Sure. I will test and share the results by end of day or tomorrow.
Regards,
Agalya
From: Bogdan-Andrei Iancu [mailto:[email protected]]
Sent: Wednesday, October 05, 2016 10:47 AM
To: Ramachandran, Agalya (Contractor)
<[email protected]<mailto:[email protected]>>;
OpenSIPS users mailling list
<[email protected]<mailto:[email protected]>>
Subject: Re: opensips crash when doing load test
Yes, that is ok (without the REST) - just to be sure which patch introduces the
bug.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
On 05.10.2016 17:45, Ramachandran, Agalya (Contractor) wrote:
Hi Bogdan,
I have observed one more thing. In the same source code, when opensips acts as
pure proxy no HTTP call is made opensips is not crashing.
I suspect the crashes may be due to the new async fix provided.
I can do load test in 2.2, but it will be just pure-proxy load test and not
with the REST queries made.
If that is OK, I can test and share the results.
Regards,
Agalya
From: Bogdan-Andrei Iancu [mailto:[email protected]]
Sent: Wednesday, October 05, 2016 9:46 AM
To: Ramachandran, Agalya (Contractor)
<[email protected]><mailto:[email protected]>;
OpenSIPS users mailling list
<[email protected]><mailto:[email protected]>
Subject: Re: opensips crash when doing load test
Hi Agalya,
That is really weird, as that field (my_T) is initialized (with the address of
the transaction) when the transaction is created (for all the possible
branches) and never reset or set again.
To avoid any regression due the patches you applied, could you please run your
load test with the stock 2.2 code from git ? And see if the crash occurs again.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
On 04.10.2016 19:51, Ramachandran, Agalya (Contractor) wrote:
Hi Bogdan,
Please find the values. Any common folder you have access for the outside world
to put files?
If yes, share me the details. I can upload the core file there.
(gdb) print r_buf
$1 = (struct retr_buf *) 0x7fded2484098
(gdb) print *r_buf
$2 = {activ_type = 0, buffer = {
s = 0x7fded0ca79f0 "INVITE
sip:[email protected]:5060<mailto:sip:[email protected]:5060>
SIP/2.0\r\nRecord-Route: <sip:96.119.240.35;lr>\r\nTo: sut
<sip:[email protected]:5060;user=phone><mailto:sip:[email protected]:5060;user=phone>\r\nFrom:
sipp
<sip:[email protected]:5060;user=phone<mailto:sip:[email protected]:5060;user=phone>"...,
len = 968}, dst = {proto = 1,
proto_reserved1 = 0, to = {s = {sa_family = 2, sa_data =
"\023\304`w\347\375\000\000\000\000\000\000\000"}, sin = {
sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 4259805024},
sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {
sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 4259805024,
sin6_addr = {__in6_u = {
__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0,
0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
sin6_scope_id = 0}}, send_sock = 0x7fded841dcb0}, retr_timer = {next_tl
= 0x7fded1af3518, prev_tl = 0x7fded15c8ae0,
ld_tl = 0x0, time_out = 169500000, timer_list = 0x7fded0acbc48, deleted =
1, set = 0}, fr_timer = {next_tl = 0x0, prev_tl = 0x0,
ld_tl = 0x0, time_out = 165, timer_list = 0x7fded8005b00 <detached_timer>,
deleted = 0, set = 0}, retr_list = RT_T2, my_T = 0x0,
branch = 0}
Regards,
Agalya
From: Bogdan-Andrei Iancu [mailto:[email protected]]
Sent: Tuesday, October 04, 2016 11:52 AM
To: Ramachandran, Agalya (Contractor)
<[email protected]><mailto:[email protected]>;
OpenSIPS users mailling list
<[email protected]><mailto:[email protected]>
Subject: Re: opensips crash when doing load test
Could you also print r_buf and *r_buf ?
_______________________________________________
Users mailing list
[email protected]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
