## Description
kamailio dispatcher cache is inaccessible and ds_select_domain is failing. all
new calls are rejected.
```
# kamctl kamcmd dispatcher.list
error: 500 - no destination sets
```
## Troubleshooting
This is happened is production after a dispatcher reload and we are unable to
reproduced it again. we restarted kamailio service and that fixed the issue.
## Reproduction
I think this happened after we did a 10 simultaneous RPC command to reload the
dispatcher cache.
## Log Messages
```
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: INFO: <script>:
[XX.XX.XX.62:1900]: /http_rpc/dispatcher/dispatcher.reload?arg=
hdr(Host)=<XX.XX.XX.7:5060> hdr(Via)=<SIP/2.0/TCP XX.XX.XX.62:1900>
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: INFO: <script>:
[XX.XX.XX.62:1900]: /http_rpc/dispatcher/dispatcher.reload?arg=
hdr(Authorization)=<yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww> =
<yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww>
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: INFO: <script>:
[XX.XX.XX.62:1900]: Successful authentication
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>:
[XX.XX.XX.54:3314]: http_rpc detected
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>:
[XX.XX.XX.54:3314]: /http_rpc/dispatcher/dispatcher.reload?arg=
hdr(Host)=<XX.XX.XX.7:5060> hdr(Via)=<SIP/2.0/TCP XX.XX.XX.54:3314>
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>:
[XX.XX.XX.54:3314]: /http_rpc/dispatcher/dispatcher.reload?arg=
hdr(Authorization)=<yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww> =
<yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww>
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>:
[XX.XX.XX.54:3314]: Successful authentication
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69c949b0), called from dispatcher: dispatch.c: ds_avl_destroy(3032),
first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69ae3840), called from dispatcher: dispatch.c: ds_avl_destroy(3032),
first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f696c5458), called from dispatcher: dispatch.c: ds_avl_destroy(3032),
first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69ce8610), called from dispatcher: dispatch.c: ds_avl_destroy(3037),
first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f698e3cd0), called from dispatcher: dispatch.c: ds_avl_destroy(3038),
first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f697498c8), called from dispatcher: dispatch.c: ds_avl_destroy(3037),
first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69662058), called from dispatcher: dispatch.c: ds_avl_destroy(3038),
first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f699b1990), called from dispatcher: dispatch.c: ds_avl_destroy(3037),
first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69663320), called from dispatcher: dispatch.c: ds_avl_destroy(3038),
first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69a4f418), called from dispatcher: dispatch.c: ds_avl_destroy(3037),
first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69663928), called from dispatcher: dispatch.c: ds_avl_destroy(3038),
first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f699125a0), called from dispatcher: dispatch.c: ds_avl_destroy(3032),
first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f698b4c30), called from dispatcher: dispatch.c: ds_avl_destroy(3037),
first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69acbee8), called from dispatcher: dispatch.c: ds_avl_destroy(3038),
first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69716f28), called from dispatcher: dispatch.c: ds_avl_destroy(3032),
first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69978d18), called from dispatcher: dispatch.c: ds_avl_destroy(3037),
first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69662d38), called from dispatcher: dispatch.c: ds_avl_destroy(3038),
first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f698c4e70), called from dispatcher: dispatch.c: ds_avl_destroy(3032),
first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69b13130), called from dispatcher: dispatch.c: ds_avl_destroy(3037),
first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f699027a0), called from dispatcher: dispatch.c: ds_avl_destroy(3038),
first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f6987f500), called from dispatcher: dispatch.c: ds_avl_destroy(3037),
first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69663f10), called from dispatcher: dispatch.c: ds_avl_destroy(3038),
first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69664400), called from dispatcher: dispatch.c: ds_avl_destroy(3032),
first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69667bd0), called from dispatcher: dispatch.c: ds_avl_destroy(3037),
first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69661a50), called from dispatcher: dispatch.c: ds_avl_destroy(3038),
first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f69857670), called from dispatcher: dispatch.c: reindex_dests(648), first
free dispatcher: dispatch.c: reindex_dests(648) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer
(0x7f6f6987f500), called from dispatcher: dispatch.c: reindex_dests(648), first
free dispatcher: dispatch.c: reindex_dests(648) - aborting
Apr 26 10:28:55 kamailio-gateway-01 /usr/sbin/kamailio[32182]: INFO: <script>:
===== redis sip code insert: SIP Code: 183 * time: 1556274535 * Rebtel-CID:
[email protected] * Call-Id:
4b6b9685-c797-4e13-8cf5-cd6f7d23a39f
Apr 26 10:28:55 kamailio-gateway-01 /usr/sbin/kamailio[32191]: INFO: <script>:
===== redis sip code insert: SIP Code: 181 * time: 1556274535 * Rebtel-CID:
[email protected] * Call-Id:
d74592af-d240-481f-92bc-6b6f1010887c
Apr 26 10:28:56 kamailio-gateway-01 /usr/sbin/kamailio[32187]: INFO: <script>:
===== redis sip code insert: SIP Code: 487 * time: 1556274536 * Rebtel-CID:
12-12-73EB33B1-5CC2DD5B000379CB-51039700 * Call-Id:
b8078da6-7dff-4689-a28b-76268200a01a
Apr 26 10:28:56 kamailio-gateway-01 /usr/sbin/kamailio[32187]: INFO: <script>:
===== redis sip code insert: SIP Code: 487 * time: 1556274523 * Rebtel-CID:
12-12-73EB33B1-5CC2DD5B000379CB-51039700 * Call-Id:
b8078da6-7dff-4689-a28b-76268200a01a
Apr 26 10:28:56 kamailio-gateway-01 /usr/sbin/kamailio[32206]: ERROR:
dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32210]: ERROR:
dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32207]: ERROR:
dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32195]: ERROR:
dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32209]: ERROR:
dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32202]: ERROR:
dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
```
## Possible Solutions
Restarting Kamailio
## Additional Information
Kamailio Version - output of kamailio -v
```
version: kamailio 5.1.3 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC,
TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled on 13:05:41 Apr 24 2018 with gcc 4.8.5
```
## Operating System:
CentOS Linux release 7.4.1708 (Core)
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1939
_______________________________________________
Kamailio (SER) - Development Mailing List
[email protected]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
