Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[ekke56]

used configuration:

debug=2
log_stderror=no
log_facility=LOG_LOCAL3
children=8
disable_tcp=yes

listen=udp:XXX.XXX.XXX.XX:5060

#!ifdef WITH_SRCPATH
mpath="modules/"
#!else
mpath="/usr/local/lib64/kamailio/modules/"
#!endif

loadmodule "statsc.so"

modparam("statsc", "interval", 2)
modparam("statsc", "track", "req.received=rcv_requests")
modparam("statsc", "items", 200)

request_route {
exit;
}


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-555030714___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[ekke56]

Hi!

I now have two new servers with kamailio 5.2.4 and centos 8 which have exactly 
the same symptoms, crash every 9 hours and 41 minutes. Traceback:

0  0x0046b7d6 in get_hash1_raw (s=0x5dcbf2d2 , len=9) at core/hashes.h:109
#1  0x0046bab6 in str_hash_get (ht=0xa61680 , 
key=0x5dcbf2d2 , len=9) at 
core/str_hash.h:94
#2  0x0046c9d0 in cnt_hash_lookup (group=0x7fff3acb37a0, 
name=0x7f75ceb93b80) at core/counters.c:420
#3  0x0046d221 in counter_lookup_str (handle=0x7fff3acb37be, 
group=0x7fff3acb37a0, name=0x7f75ceb93b80) at core/counters.c:541
#4  0x7f75ce71734b in get_stat (name=0x7f75ceb93b80) at 
../../core/counters.h:197
#5  0x7f75ce71805b in statsc_svalue (name=0x7f75ceb93b80, 
res=0x7f75ceb93f4b) at statsc_mod.c:149
#6  0x7f75ce719aff in statsc_timer (ticks=94505445, param=0x0) at 
statsc_mod.c:274
#7  0x005d6087 in sr_wtimer_exec (ticks=94505445, param=0x0) at 
core/timer_proc.c:390
#8  0x005d52a8 in fork_sync_timer (child_id=-1, desc=0x7bd391 
"secondary timer", make_sock=1, f=0x5d5d80 , param=0x0, 
interval=1000) at core/timer_proc.c:224
#9  0x005d63bc in sr_wtimer_start () at core/timer_proc.c:416
#10 0x00425873 in main_loop () at main.c:1729
#11 0x0042c4d1 in main (argc=7, argv=0x7fff3acb3dd8) at main.c:2696




From: Henning Westerholt 
Sent: 08 November 2019 16:29
To: kamailio/kamailio 
Cc: Ekke Einberg ; Author 

Subject: Re: [kamailio/kamailio] statsc module - server crashes every 10 hours 
(#2101)


This is interesting. So you are saying that it happens only on one server, even 
if the others also use stats.

Do you see e.g. any memory errors in the dmesg or system logging? Do you have a 
chance to e.g. to a memory test on this particular server?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub,
 or 
unsubscribe.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-553394422___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[ekke56]

Hi!

We have total 5 kamailio servers all running as virtual servers. So far it 
seems that the problem occurs on only one server.
The crash happens every modparam("statsc", "interval", 300) * 2 minutes

If I set this to 2 it crashes every four minutes.

ekke


From: Henning Westerholt 
Sent: 08 November 2019 13:48
To: kamailio/kamailio 
Cc: Ekke Einberg ; Author 

Subject: Re: [kamailio/kamailio] statsc module - server crashes every 10 hours 
(#2101)


Hello Ekke thanks for the feedback. About the question regarding reproducing - 
maybe could sort out eventual other causes.
Do you see this only on this server, or also on another machine? Is the server 
a actual hardware or a virtual server? Maybe you can move it to another host to 
see if its maybe something related to memory or other hardware problems?
Does the crash happens everytime the same time (like 10h as you mentioned), or 
does it change?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub,
 or 
unsubscribe.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-551735862___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[ekke56]

I have essentially deleted whole configuration and loaded minimum modules but 
the issue still occurs.
Should we pursue it further or (sigh) declare it unreproducable?

ekke

PS I have another question regarding TM module and t_replicate, do you know who 
could help?

From: Daniel-Constantin Mierla 
Sent: 24 October 2019 13:58
To: kamailio/kamailio 
Cc: Ekke Einberg ; Author 

Subject: Re: [kamailio/kamailio] statsc module - server crashes every 10 hours 
(#2101)


I loaded statsc in a test system and runs for almost 1 week without any issue.

Do you fetch the statsc data via RPC periodically? How often?

Can you give the list of loaded modules and modparams (replace sensitive data 
like ip addresses, passwords ...) ?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub,
 or 
unsubscribe.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-551632215___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[ekke56]

I will try to isolate the issue next week, when im back in the office.

From: Daniel-Constantin Mierla 
Sent: 24 October 2019 13:57:55
To: kamailio/kamailio 
Cc: Ekke Einberg ; Author 

Subject: Re: [kamailio/kamailio] statsc module - server crashes every 10 hours 
(#2101)


I loaded statsc in a test system and runs for almost 1 week without any issue.

Do you fetch the statsc data via RPC periodically? How often?

Can you give the list of loaded modules and modparams (replace sensitive data 
like ip addresses, passwords ...) ?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub,
 or 
unsubscribe.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-545876343___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[ekke56]

There are no custom modules or patches. No changes to downloaded code.

From: Daniel-Constantin Mierla 
Sent: 24 October 2019 13:34:33
To: kamailio/kamailio 
Cc: Ekke Einberg ; Author 

Subject: Re: [kamailio/kamailio] statsc module - server crashes every 10 hours 
(#2101)


Do you have custom modules or custom patches in that kamailio instance? Or is 
just stock kamailio? It looks like some addresses are overwritten...

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub,
 or 
unsubscribe.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-545860322___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[ekke56]

I am out of country, hence the delay:

(gdb) frame 6
#6  0x7eff7d65db79 in statsc_timer (ticks=113205942, param=0x0) at 
statsc_mod.c:274
274 statsc_svalue(>rname, sm->vals + n);
(gdb) p *sm
$1 = {sname = {s = 0x5da5103e , len = 
1571098986}, rname = {s = 0x5da51296 , len = 
9}, vals = 0x7eff79693bb3, next = 0x7eff79693f40}
(gdb)


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-543124028___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] statsc module - server crashes every 10 hours (#2101)

[ekke56]

Sure:
(gdb) frame 5
#5  0x7eff7d65c113 in statsc_svalue (name=0x7eff79693b80, 
res=0x7eff79693f4b) at statsc_mod.c:149
149 stat = get_stat(name);
(gdb) p name
$1 = (str *) 0x7eff79693b80
(gdb) p *name
$2 = {s = 0x5da51296 , len = 9}
(gdb)
$3 = {s = 0x5da51296 , len = 9}

loadmodule "statsc.so"
modparam("statsc", "interval", 300)
modparam("statsc", "track", "req.received=rcv_requests")
modparam("statsc", "track", 
"req.received=rcv_requests;rpl.received=rcv_replies")
modparam("statsc", "items", 200)

ekke
From: Daniel-Constantin Mierla 
Sent: Tuesday, October 15, 2019 4:11 PM
To: kamailio/kamailio 
Cc: Ekke Einberg ; Author 

Subject: Re: [kamailio/kamailio] statsc module - server crashes every 10 hours 
(#2101)


It seems to originate in statsc module, I updated the title to reflect that. 
Can give the output from gdb for following commands:

frame 5

p name

p *name

Also, can you provide the modparams you set for statsc module?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub,
 or 
unsubscribe.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101#issuecomment-542206335___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

[sr-dev] [kamailio/kamailio] server crashes every 10 hours (#2101)

[ekke56]

### Description



### Troubleshooting
coredump data below. 

 Reproduction



 Debugging Data



```
(gdb)   bt full
#0  0x0046ccdd in get_hash1_raw (s=0x5da51296 , len=9) at core/hashes.h:109
p = 0x5da51296 
v = 4100353960
h = 0
#1  0x0046cfb6 in str_hash_get (ht=0xa6d6c0 , 
key=0x5da51296 , len=9) at core/str_hash.h:94
h = 0
e = 0x19
#2  0x0046debe in cnt_hash_lookup (group=0x7ffc2db9d7c0, 
name=0x7eff79693b80) at core/counters.c:420
e = 0x7eff80c48c88
first = 0xbd8191bc2e950d55
cnt_rec = 0x7eff80c48cb0
#3  0x0046e752 in counter_lookup_str (handle=0x7ffc2db9d7d0, 
group=0x7ffc2db9d7c0, name=0x7eff79693b80) at core/counters.c:541
cnt_rec = 0x19
#4  0x7eff7d65b447 in get_stat (name=0x7eff79693b80) at 
../../core/counters.h:197
h = {id = 13}
grp = {s = 0x0, len = 0}
#5  0x7eff7d65c113 in statsc_svalue (name=0x7eff79693b80, 
res=0x7eff79693f4b) at statsc_mod.c:149
stat = 0xd
__FUNCTION__ = "statsc_svalue"
#6  0x7eff7d65db79 in statsc_timer (ticks=113205942, param=0x0) at 
statsc_mod.c:274
sm = 0x7eff79693b70
tn = 1571099286
n = 115
__FUNCTION__ = "statsc_timer"
#7  0x005db7f9 in sr_wtimer_exec (ticks=113205942, param=0x0) at 
core/timer_proc.c:390
wt = 0x7eff80c56f48
wn = 0x0
wp = 0x0
cs = 0
__FUNCTION__ = "sr_wtimer_exec"
#8  0x005daa29 in fork_sync_timer (child_id=-1, desc=0x7c8a71 
"secondary timer", make_sock=1, f=0x5db4f2 , param=0x0, 
interval=1000) at core/timer_proc.c:224
pid = 0
ts1 = 1811295077
ts2 = 1000
#9  0x005dbb26 in sr_wtimer_start () at core/timer_proc.c:416
__FUNCTION__ = "sr_wtimer_start"
#10 0x00425a33 in main_loop () at main.c:1729
i = 8
pid = 3615
si = 0x0
si_desc = "udp receiver child=7 
sock=91.216.86.57:5060\000\377~\000\000@۹-\374\177\000\000\020\207A\000\000\000\000\000\020\336\271-\374\177\000\000\323c{\000\000\000\000\000\031\000\000\000\000\000\000\000\000\000\000\004\000\000\000\000\000\000\200\000\000\000\000\000__:\201\377~\000\000\250\346\260v\377~\000\000\270\314À\377~\000"
nrprocs = 8
woneinit = 1
__FUNCTION__ = "main_loop"
#11 0x0042c72a in main (argc=7, argv=0x7ffc2db9def8) at main.c:2696
cfg_stream = 0x1ee3010
c = -1
r = 0
tmp = 0x7ffc2db9ef2c ""
tmp_len = 0
port = 0
proto = 0
options = 0x76a9a0 
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
---Type  to continue, or q  to quit---  info locals
seed = 2620966285
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x2
p = 0x0
st = {st_dev = 20, st_ino = 22346, st_nlink = 2, st_mode = 16832, 
st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 
4096, st_blocks = 0, st_atim = {tv_sec = 1568107728, tv_nsec = 45800}, 
st_mtim = {
tv_sec = 1571064422, tv_nsec = 5700}, st_ctim = {tv_sec = 
1571064422, tv_nsec = 5700}, __unused = {0, 0, 0}}
__FUNCTION__ = "main"
(gdb)   list
104 register unsigned v;
105 register unsigned h;
106
107 h=0;
108
109 hash_update_str(s, s+len, p, v, h);
110 return hash_finish(h);
111 }
112
113

```

 Log Messages



```
no relevant messages
```

 SIP Traffic



```
seems to be triggered by internal timer
```

### Possible Solutions



### Additional Information

  * **Kamailio Version** - output of `kamailio -v`

```
version: kamailio 5.2.4 (x86_64/linux) 759867
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, 
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, 
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, 
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024, 
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 759867
compiled on 12:20:22 Sep 11 2019 with gcc 4.8.5

```

* **Operating System**:



```
CentOS Linux release 7.4.1708 (Core)
Linux xxx 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 x86_64 
x86_64 x86_64 GNU/Linux
```


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2101___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev