Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[Henning Westerholt]

Kamailio crashed after 5 minutes, version kamailio-5.4.0-dev1

```
[logs every 2s the following error]
Nov 18 21:26:28 customer-1-nbg1-1 
/usr/local/kamailio-devel/sbin/kamailio[24200]: ERROR: statsc 
[statsc_mod.c:151]: statsc_svalue(): statistic free_size not found
Nov 18 21:26:28 customer-1-nbg1-1 
/usr/local/kamailio-devel/sbin/kamailio[24200]: ERROR: statsc 
[statsc_mod.c:151]: statsc_svalue(): statistic used_size not found
Nov 18 21:26:28 customer-1-nbg1-1 
/usr/local/kamailio-devel/sbin/kamailio[24200]: ERROR: statsc 
[statsc_mod.c:151]: statsc_svalue(): statistic real_used_size not found
Nov 18 21:26:28 customer-1-nbg1-1 
/usr/local/kamailio-devel/sbin/kamailio[24200]: ERROR: statsc 
[statsc_mod.c:151]: statsc_svalue(): statistic rcv_requests not found
Nov 18 21:26:30 customer-1-nbg1-1 
/usr/local/kamailio-devel/sbin/kamailio[24200]: ERROR: statsc 
[statsc_mod.c:151]: statsc_svalue(): statistic free_size not found
Nov 18 21:26:30 customer-1-nbg1-1 
/usr/local/kamailio-devel/sbin/kamailio[24200]: ERROR: statsc 
[statsc_mod.c:151]: statsc_svalue(): statistic used_size not found
Nov 18 21:26:30 customer-1-nbg1-1 
/usr/local/kamailio-devel/sbin/kamailio[24200]: ERROR: statsc 
[statsc_mod.c:151]: statsc_svalue(): statistic real_used_size not found
Nov 18 21:26:30 customer-1-nbg1-1 
/usr/local/kamailio-devel/sbin/kamailio[24200]: ERROR: statsc 
[statsc_mod.c:151]: statsc_svalue(): statistic rcv_requests not found
Nov 18 21:26:32 customer-1-nbg1-1 kernel: [626011.807865] kamailio[24200]: 
segfault at 5dd2fe78 ip 5623b1b9487e sp 7ffcb3a908e0 error 4 in 
kamailio[5623b1b61000+39c000]
Nov 18 21:26:32 customer-1-nbg1-1 kernel: [626011.816407] Code: c7 e8 59 fb ff 
ff 90 c9 c3 55 48 89 e5 41 54 53 48 89 7d d8 89 75 d4 bb 00 00 00 00 48 8b 45 
d8 48 89 45 e8 eb 58 48 8b 45 e8 <0f> b6 00 0f be c0 c1 e0 18 89 c2 48 8b 45 e8 
48 83 c0 01 0f b6 00
Nov 18 21:26:32 customer-1-nbg1-1 
/usr/local/kamailio-devel/sbin/kamailio[24189]: ALERT:  [main.c:767]: 
handle_sigs(): child process 24200 exited by a signal 11
Nov 18 21:26:32 customer-1-nbg1-1 
/usr/local/kamailio-devel/sbin/kamailio[24189]: ALERT:  [main.c:770]: 
handle_sigs(): core was generated
Nov 18 21:26:32 customer-1-nbg1-1 
/usr/local/kamailio-devel/sbin/kamailio[24189]: INFO:  [main.c:793]: 
handle_sigs(): terminating due to SIGCHLD
Nov 18 21:26:32 customer-1-nbg1-1 
/usr/local/kamailio-devel/sbin/kamailio[24198]: INFO:  [main.c:848]: 
sig_usr(): signal 15 received
```

```
Core was generated by `/usr/local/kamailio-devel/sbin/kamailio -f 
/etc/kamailio/kamailio.cfg -L /usr/l'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x5623b1b9487e in get_hash1_raw (s=0x5dd2fe78 , len=9) at core/hashes.h:109
109 hash_update_str(s, s+len, p, v, h);
(gdb) bt
#0  0x5623b1b9487e in get_hash1_raw (s=0x5dd2fe78 , len=9) at core/hashes.h:109
#1  0x5623b1b94e2b in str_hash_get (ht=0x5623b20050e0 , 
key=0x5dd2fe78 , len=9) at 
core/str_hash.h:96
#2  0x5623b1b96bb3 in cnt_hash_lookup (group=0x7ffcb3a909d0, 
name=0x7f032f7aab80) at core/counters.c:430
#3  0x5623b1b9748d in counter_lookup_str (handle=0x7ffcb3a909ee, 
group=0x7ffcb3a909d0, name=0x7f032f7aab80) at core/counters.c:551
#4  0x7f033417b247 in get_stat (name=0x7f032f7aab80) at 
../../core/counters.h:197
#5  0x7f033417bf57 in statsc_svalue (name=0x7f032f7aab80, 
res=0x7f032f7aaf4b) at statsc_mod.c:149
#6  0x7f033417dd5c in statsc_timer (ticks=81421045, param=0x0) at 
statsc_mod.c:278
#7  0x5623b1d76a27 in sr_wtimer_exec (ticks=81421045, param=0x0) at 
core/timer_proc.c:433
#8  0x5623b1d75b3a in fork_sync_timer (child_id=-1, desc=0x5623b1f674c1 
"secondary timer", make_sock=1, f=0x5623b1d766df , param=0x0, 
interval=1000)
at core/timer_proc.c:267
#9  0x5623b1d76da1 in sr_wtimer_start () at core/timer_proc.c:459
#10 0x5623b1b703ff in main_loop () at main.c:1746
#11 0x5623b1b78ac2 in main (argc=5, argv=0x7ffcb3a91258) at main.c:2804

```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-555201030___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[Henning Westerholt]

Thanks, I have configured it on one of my tests servers. Lets see tomorrow. :-)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-555169237___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[ekke56]

used configuration:

debug=2
log_stderror=no
log_facility=LOG_LOCAL3
children=8
disable_tcp=yes

listen=udp:XXX.XXX.XXX.XX:5060

#!ifdef WITH_SRCPATH
mpath="modules/"
#!else
mpath="/usr/local/lib64/kamailio/modules/"
#!endif

loadmodule "statsc.so"

modparam("statsc", "interval", 2)
modparam("statsc", "track", "req.received=rcv_requests")
modparam("statsc", "items", 200)

request_route {
exit;
}


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-555030714___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[Henning Westerholt]

Thanks for the feedback, this looks certainly like an overflow or something 
similar. You mentioned that you can reproduce this with a small cfg, could you 
provide it also in the issue?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-554277911___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[ekke56]

Hi!

I now have two new servers with kamailio 5.2.4 and centos 8 which have exactly 
the same symptoms, crash every 9 hours and 41 minutes. Traceback:

0  0x0046b7d6 in get_hash1_raw (s=0x5dcbf2d2 , len=9) at core/hashes.h:109
#1  0x0046bab6 in str_hash_get (ht=0xa61680 , 
key=0x5dcbf2d2 , len=9) at 
core/str_hash.h:94
#2  0x0046c9d0 in cnt_hash_lookup (group=0x7fff3acb37a0, 
name=0x7f75ceb93b80) at core/counters.c:420
#3  0x0046d221 in counter_lookup_str (handle=0x7fff3acb37be, 
group=0x7fff3acb37a0, name=0x7f75ceb93b80) at core/counters.c:541
#4  0x7f75ce71734b in get_stat (name=0x7f75ceb93b80) at 
../../core/counters.h:197
#5  0x7f75ce71805b in statsc_svalue (name=0x7f75ceb93b80, 
res=0x7f75ceb93f4b) at statsc_mod.c:149
#6  0x7f75ce719aff in statsc_timer (ticks=94505445, param=0x0) at 
statsc_mod.c:274
#7  0x005d6087 in sr_wtimer_exec (ticks=94505445, param=0x0) at 
core/timer_proc.c:390
#8  0x005d52a8 in fork_sync_timer (child_id=-1, desc=0x7bd391 
"secondary timer", make_sock=1, f=0x5d5d80 , param=0x0, 
interval=1000) at core/timer_proc.c:224
#9  0x005d63bc in sr_wtimer_start () at core/timer_proc.c:416
#10 0x00425873 in main_loop () at main.c:1729
#11 0x0042c4d1 in main (argc=7, argv=0x7fff3acb3dd8) at main.c:2696




From: Henning Westerholt 
Sent: 08 November 2019 16:29
To: kamailio/kamailio 
Cc: Ekke Einberg ; Author 

Subject: Re: [kamailio/kamailio] statsc module - server crashes every 10 hours 
(#2101)


This is interesting. So you are saying that it happens only on one server, even 
if the others also use stats.

Do you see e.g. any memory errors in the dmesg or system logging? Do you have a 
chance to e.g. to a memory test on this particular server?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub,
 or 
unsubscribe.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-553394422___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[Henning Westerholt]

This is interesting. So you are saying that it happens only on one server, even 
if the others also use stats. 

Do you see e.g. any memory errors in the dmesg or system logging? Do you have a 
chance to e.g. to a memory test on this particular server?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-551848491___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[ekke56]

Hi!

We have total 5 kamailio servers all running as virtual servers. So far it 
seems that the problem occurs on only one server.
The crash happens every modparam("statsc", "interval", 300) * 2 minutes

If I set this to 2 it crashes every four minutes.

ekke


From: Henning Westerholt 
Sent: 08 November 2019 13:48
To: kamailio/kamailio 
Cc: Ekke Einberg ; Author 

Subject: Re: [kamailio/kamailio] statsc module - server crashes every 10 hours 
(#2101)


Hello Ekke thanks for the feedback. About the question regarding reproducing - 
maybe could sort out eventual other causes.
Do you see this only on this server, or also on another machine? Is the server 
a actual hardware or a virtual server? Maybe you can move it to another host to 
see if its maybe something related to memory or other hardware problems?
Does the crash happens everytime the same time (like 10h as you mentioned), or 
does it change?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub,
 or 
unsubscribe.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-551735862___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[Henning Westerholt]

Hello Ekke thanks for the feedback. About the question regarding reproducing - 
maybe could sort out eventual other causes. 
Do you see this only on this server, or also on another machine? Is the server 
a actual hardware or a virtual server? Maybe you can move it to another host to 
see if its maybe something related to memory or other hardware problems?
Does the crash happens everytime the same time (like 10h as you mentioned), or 
does it change?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-551651645___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[ekke56]

I have essentially deleted whole configuration and loaded minimum modules but 
the issue still occurs.
Should we pursue it further or (sigh) declare it unreproducable?

ekke

PS I have another question regarding TM module and t_replicate, do you know who 
could help?

From: Daniel-Constantin Mierla 
Sent: 24 October 2019 13:58
To: kamailio/kamailio 
Cc: Ekke Einberg ; Author 

Subject: Re: [kamailio/kamailio] statsc module - server crashes every 10 hours 
(#2101)


I loaded statsc in a test system and runs for almost 1 week without any issue.

Do you fetch the statsc data via RPC periodically? How often?

Can you give the list of loaded modules and modparams (replace sensitive data 
like ip addresses, passwords ...) ?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub,
 or 
unsubscribe.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-551632215___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[ekke56]

I will try to isolate the issue next week, when im back in the office.

From: Daniel-Constantin Mierla 
Sent: 24 October 2019 13:57:55
To: kamailio/kamailio 
Cc: Ekke Einberg ; Author 

Subject: Re: [kamailio/kamailio] statsc module - server crashes every 10 hours 
(#2101)


I loaded statsc in a test system and runs for almost 1 week without any issue.

Do you fetch the statsc data via RPC periodically? How often?

Can you give the list of loaded modules and modparams (replace sensitive data 
like ip addresses, passwords ...) ?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub,
 or 
unsubscribe.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-545876343___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[Daniel-Constantin Mierla]

I loaded statsc in a test system and runs for almost 1 week without any issue.

Do you fetch the statsc data via RPC periodically? How often?

Can you give the list of loaded modules and modparams (replace sensitive data 
like ip addresses, passwords ...) ?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-545864552___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[ekke56]

There are no custom modules or patches. No changes to downloaded code.

From: Daniel-Constantin Mierla 
Sent: 24 October 2019 13:34:33
To: kamailio/kamailio 
Cc: Ekke Einberg ; Author 

Subject: Re: [kamailio/kamailio] statsc module - server crashes every 10 hours 
(#2101)


Do you have custom modules or custom patches in that kamailio instance? Or is 
just stock kamailio? It looks like some addresses are overwritten...

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub,
 or 
unsubscribe.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-545860322___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[Daniel-Constantin Mierla]

Do you have custom modules or custom patches in that kamailio instance? Or is 
just stock kamailio? It looks like some addresses are overwritten...

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-545856777___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[ekke56]

I am out of country, hence the delay:

(gdb) frame 6
#6  0x7eff7d65db79 in statsc_timer (ticks=113205942, param=0x0) at 
statsc_mod.c:274
274 statsc_svalue(>rname, sm->vals + n);
(gdb) p *sm
$1 = {sname = {s = 0x5da5103e , len = 
1571098986}, rname = {s = 0x5da51296 , len = 
9}, vals = 0x7eff79693bb3, next = 0x7eff79693f40}
(gdb)


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-543124028___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[Daniel-Constantin Mierla]

Are you able to provide the output for last commands?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-543073786___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[Daniel-Constantin Mierla]

Now I need also the output for gdb commands:

```
frame 6
p *sm
```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-542351187___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[ekke56]

Sure:
(gdb) frame 5
#5  0x7eff7d65c113 in statsc_svalue (name=0x7eff79693b80, 
res=0x7eff79693f4b) at statsc_mod.c:149
149 stat = get_stat(name);
(gdb) p name
$1 = (str *) 0x7eff79693b80
(gdb) p *name
$2 = {s = 0x5da51296 , len = 9}
(gdb)
$3 = {s = 0x5da51296 , len = 9}

loadmodule "statsc.so"
modparam("statsc", "interval", 300)
modparam("statsc", "track", "req.received=rcv_requests")
modparam("statsc", "track", 
"req.received=rcv_requests;rpl.received=rcv_replies")
modparam("statsc", "items", 200)

ekke
From: Daniel-Constantin Mierla 
Sent: Tuesday, October 15, 2019 4:11 PM
To: kamailio/kamailio 
Cc: Ekke Einberg ; Author 

Subject: Re: [kamailio/kamailio] statsc module - server crashes every 10 hours 
(#2101)


It seems to originate in statsc module, I updated the title to reflect that. 
Can give the output from gdb for following commands:

frame 5

p name

p *name

Also, can you provide the modparams you set for statsc module?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub,
 or 
unsubscribe.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-542206335___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[Daniel-Constantin Mierla]

It seems to originate in statsc module, I updated the title to reflect that. 
Can give the output from gdb for following commands:

```
frame 5
p name
p *name
```

Also, can you provide the modparams you set for statsc module?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-542203719___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev