kuzaxak created an issue (kamailio/kamailio#4809)
### Description
Concurrent `permissions` address (or trusted) reloads can corrupt shared memory
when more than one worker process rebuilds the table at the same time.
The `permissions` module keeps two hash-table slots per set in shared memory —
`perm_addr_table_1/_2`, `perm_subnet_table_1/_2`, `perm_domain_table_1/_2`
(and `perm_trust_table_1/_2` for trusted) — plus a shared "current" pointer for
each (`perm_addr_table`, `perm_subnet_table`, `perm_domain_table`,
`perm_trust_table`), all allocated with `shm_malloc` in `init_addresses()` /
`init_trusted()`. `reload_address_table()` (`src/modules/permissions/address.c`)
swaps between the two slots with an unlocked select / empty / populate / publish
sequence:
```c
int reload_address_table(void)
{
/* Choose new hash table and free its old contents */
if(*perm_addr_table == perm_addr_table_1) { /* (A) read current */
empty_addr_hash_table(perm_addr_table_2); /* (B) shm_free slot
*/
atg.address_table = perm_addr_table_2;
} else {
empty_addr_hash_table(perm_addr_table_1);
atg.address_table = perm_addr_table_1;
}
/* ... same select/empty for the subnet and domain tables ... */
ret = reload_address_file_table(&atg); /* (C) repopulate
*/
if(ret != 1) {
return ret;
}
*perm_addr_table = atg.address_table; /* (D) publish
*/
*perm_subnet_table = atg.subnet_table;
*perm_domain_table = atg.domain_table;
}
```
`reload_trusted_table()` (`trusted.c`) has the same unlocked
select / `empty_hash_table` / populate / publish shape on
`perm_trust_table_1/_2`.
Because the slot pointers and the backing tables all live in shared memory and
nothing serializes (A) through (D), two worker processes can interleave:
1. Worker 1 and Worker 2 both read `*perm_addr_table == perm_addr_table_1` at
(A).
2. Both select the **same** inactive slot `perm_addr_table_2`.
3. Both call `empty_addr_hash_table(perm_addr_table_2)` at (B) and both
repopulate
that same slot at (C).
`empty_addr_hash_table()` (`hash.c`) `shm_free()`s every node in the slot, so
the
second worker walks and frees a list the first has already freed (and both
insert
into it concurrently): a double free / use-after-free of the address hash nodes,
their subnet-table tags, and domain entries.
The only guard is the check-then-set in `rpc_check_reload()` (`rpc.c`):
```c
if(*perm_rpc_reload_time != 0
&& *perm_rpc_reload_time > time(NULL) - perm_reload_delta) {
LM_ERR("ongoing reload\n");
rpc->fault(ctx, 500, "ongoing reload");
return -1;
}
// we are reloading, don't allow new reloads
*perm_rpc_reload_time = time(NULL) + 86400; /* master; 6.0.x sets time(NULL)
*/
```
This is itself a racy check-then-set: two callers can both read the old
`perm_rpc_reload_time`, both pass the comparison, and both proceed before either
writes. It is not held across the table swap, and it is opened wide by
`modparam("permissions", "reload_delta", 0)`. On `master` the flag is set to
`time(NULL) + 86400` for the reload's duration, which narrows the window to the
non-atomic check→set gap but does **not** close it. On the 6.0 branch it is set
to plain `time(NULL)`, so with `reload_delta=0` it essentially never rejects and
concurrent reloads flow freely.
`permissions.addressReload` / `permissions.trustedReload` are ordinary RPC
commands, so any RPC transport served by more than one process can run them
concurrently. The `ctl` binrpc socket alone would not show this — it forks a
single RPC handler — which is likely why the race stays rare in the field until
a
datagram/HTTP RPC transport with a worker pool (or several concurrent RPC
clients) is in use.
Expected behavior: concurrent reload requests should not corrupt shared memory.
At most one address/trusted table rebuild should be active at a time, or later
callers should return without touching the inactive slot.
The same unguarded sequence is present on both branches checked:
- `master`: `b3a8384d9d10343317b3df83c595f5cfe570d14a` (builds as `6.2.0-dev1`)
- `6.0`: reproduced on the `6.0.7` release
This was originally observed in a production deployment (kamailio 6.0.x) under
automation that fires frequent `permissions.addressReload` with a low
`reload_delta`: with the default `mem_safety=1` the double free is logged and
ignored, after which the cascading heap corruption crashes a worker
(signal 6/11) and/or leaks shm.
### Troubleshooting
#### Reproduction
A self-contained reproducer archive is attached to this issue:
`kamailio-permissions-reload-race-repro-with-pcap.zip`. It builds a pristine
base
and base+fix from the same source tree and drives 16 `jsonrpcs` datagram workers
into concurrent `permissions.addressReload`.
[kamailio-permissions-reload-race-repro-with-pcap.zip](https://github.com/user-attachments/files/29596047/kamailio-permissions-reload-race-repro-with-pcap.zip)
To reproduce from the attached archive:
```sh
unzip kamailio-permissions-reload-race-repro-with-pcap.zip -d
kamailio-permissions-reload-race-repro
cd kamailio-permissions-reload-race-repro
./run.sh # unpatched master -> double-free reproduced
./run.sh --patched # master + reload lock -> survives
./run.sh --base 6.0.7 # reproduce on the 6.0.7 release base
```
The repro (defaults: 32 UDP senders x 25 s):
- builds unpatched `master` from `b3a8384d9d10343317b3df83c595f5cfe570d14a`
- uses a file-backed address list of 8000 records (widens the empty/repopulate
window)
- sets `modparam("permissions", "reload_delta", 0)` (see note below)
- drives reloads over `jsonrpcs` with a datagram transport and a worker pool
(`modparam("jsonrpcs", "dgram_workers", 16)`), so up to 16 processes execute
`reload_address_table()` at once
- starts kamailio with `-x q_malloc` (so the `DBG_QM_MALLOC` double-free
detector
is active; `-w /cores` drops a core on crash)
- floods `permissions.addressReload` JSON-RPC datagrams from 32 parallel senders
(see `stress/flood.py`)
This setup only makes the race window easy to hit within seconds; it does not
create the race. `reload_delta` does not close it: the `rpc_check_reload()` gate
is itself a non-atomic check-then-set on shared memory, so two workers can pass
it simultaneously just as they can pass the `*perm_addr_table` select above.
Observed on unpatched `master` in the committed run
(`logs/kam-unpatched.full.log.gz`), `-x q_malloc`, default `mem_safety=1` (the
double free is logged and ignored — as in the production incident — then the
cascading corruption crashes a worker):
```text
double-free BUG lines : 905 (q_malloc 'freeing already freed pointer')
worker signal/core lines : 2 (child process 24 exited by signal 11, core
dumped)
reload-gate rejections : 235098 (ERR 'ongoing reload' = concurrent reloads
seen)
```
The 905 double-free lines break down by "first free" site:
- 556 `empty_addr_hash_table(615)` — two workers emptied the same inactive slot
- 346 `addr_hash_table_insert(462)` — one worker freed a node the other had just
inserted
- 2 `subnet_table_insert(666)`, 1 `empty_subnet_table(843)`
Either way it is the same slot being emptied and repopulated by two processes at
once. Note the 235,098 `ongoing reload` rejections: workers constantly saw a
reload already in flight — exactly the window in which two of them proceed
together — yet the race still fired 905 times.
The patched build still logs `ongoing reload` — the fix does not change the RPC
gate — but the reload critical section is now serialized, so the concurrent
reloads that do get through no longer corrupt the shared tables.
#### Debugging Data
`bt full` from the crashed worker core (`logs/coredump-bt-unpatched.txt`): a
`jsonrpcs` datagram worker crashes directly inside `reload_address_table()`,
freeing a fragment a concurrent worker corrupted (note the clobbered
`atg.address_table = 0x6a46439c` at frame #8):
```text
Core was generated by `/usr/local/sbin/kamailio -DD -E -x q_malloc -m 128 -M 16
-w /cores -f /etc/kamailio/kam.cfg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000ffffaf1a4248 in ?? () from /lib/aarch64-linux-gnu/libc.so.6
#1 0x0000ffffaf1562b0 in ?? () from /lib/aarch64-linux-gnu/libc.so.6
#2 0x0000ffffaf1570e8 [PAC] in ?? () from /lib/aarch64-linux-gnu/libc.so.6
#3 0x0000ffffaf14bf34 [PAC] in fprintf () from /lib/aarch64-linux-gnu/libc.so.6
#4 qm_debug_check_frag (qm=0xffffa5ef0000, f=0xffffa601e2c0,
file="permissions: hash.c", line=615, efile="core/mem/q_malloc.c", eline=569)
at core/mem/q_malloc.c:130
p = 0x5a600ac40
__func__ = "qm_debug_check_frag"
#5 qm_free (qmp=0xffffa5ef0000, p=0xffffa601e300, file="permissions: hash.c",
func="empty_addr_hash_table", line=615, mname="permissions") at
core/mem/q_malloc.c:569
#6 qm_shm_free (..., func="empty_addr_hash_table", line=615, ...) at
core/mem/q_malloc.c:1602
#7 empty_addr_hash_table (table=0xffffa5f8f210) at hash.c:615
i = 24
np = 0xffffa601e300
next = 0xffffa62aee70
__func__ = "empty_addr_hash_table"
#8 reload_address_table () at address.c:470
ret = 0
atg = {address_table = 0x6a46439c, subnet_table = 0xffffadf85298
<jsonrpc_init_reply+2108>, domain_table = 0xaaaad0aebc20}
__func__ = "reload_address_table"
#9 reload_address_table_cmd () at address.c:535
#10 rpc_address_reload (rpc=..., c=...) at rpc.c:113
#11 jsonrpc_exec_ex (cmd=..., rpath=0x0, spath=...) at jsonrpcs_mod.c:1594
#12 jsonrpc_dgram_server (rx_sock=4) at jsonrpcs_sock.c:660
#13 jsonrpc_dgram_process (rank=11) at jsonrpcs_sock.c:482
[ #14-#20: jsonrpc_dgram_child_init -> child_init -> init_mod_child ->
init_child -> main_loop -> main ]
```
The `scmd` local at frame #11 in the full backtrace shows the driving request:
`{"jsonrpc":"2.0","method":"permissions.addressReload","id":354}`.
#### Log Messages
Allocator double-free and crash from `logs/kam-unpatched.full.log.gz` (unpatched
`master`). The two "first free" sites are the two ways the race manifests:
```text
CRITICAL: <core> [core/mem/q_malloc.c:578]: qm_free(): BUG: freeing already
freed pointer (0xffffa62818b0), called from permissions: hash.c:
empty_addr_hash_table(615), first free permissions: hash.c:
empty_addr_hash_table(615) - ignoring
CRITICAL: <core> [core/mem/q_malloc.c:578]: qm_free(): BUG: freeing already
freed pointer (0xffffa6214f30), called from permissions: hash.c:
empty_addr_hash_table(615), first free permissions: hash.c:
addr_hash_table_insert(462) - ignoring
...
ALERT: <core> [main.c:824]: handle_sigs(): child process 24 exited by a signal
11
ALERT: <core> [main.c:828]: handle_sigs(): core was generated
```
Same signature on the 6.0.7 base (`logs/kam-unpatched-6.0.7.full.log.gz`;
`q_malloc.c:555`, `empty_addr_hash_table(614)`):
```text
CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already
freed pointer (0xffff7e52a880), called from permissions: hash.c:
empty_addr_hash_table(614), first free permissions: hash.c:
empty_addr_hash_table(614) - ignoring
```
Which q_malloc message appears varies with churn — a clean `freeing already
freed
pointer` versus a clobbered fragment header that segfaults inside
`qm_debug_check_frag` (frame #4 above) — but both are the same race and both
crash a worker.
#### SIP Traffic
The race is not driven by SIP: the trigger is a flood of JSON-RPC
`permissions.addressReload` datagrams to the `jsonrpcs` UDP socket
(udp/5062), dispatched across the 16-worker datagram pool. The SIP receive loop
is idle (it only replies `200 OK` so the node is demonstrably alive during and
after the stress). The captured RPC flood and the generator are in the attached
archive:
```text
pcap/rpc-unpatched.pcap (full capture, unpatched run — 65,640
datagrams)
pcap/rpc-unpatched-sample.pcap (first 800 datagrams, for convenience)
pcap/rpc-patched-sample.pcap (first 800 datagrams, patched run)
stress/flood.py (traffic generator)
```
### Possible Solutions
Serialize the whole select / empty / populate / publish critical section with a
single permissions-global lock in shared memory (`perm_reload_lock`), taken at
the start of `reload_address_table()` / `reload_trusted_table()` and released on
every success and error return. This does not touch the address/trusted lookup
hot path — those read the active slot without the lock. I tested this with a
`gen_lock_t *perm_reload_lock` allocated in `mod_init` and destroyed in
`destroy`; under the same stress it produced 0 BUG lines and no crash (see the
patched run above).
I can submit a pull request if the approach is acceptable.
```diff
--- a/src/modules/permissions/address.c
+++ b/src/modules/permissions/address.c
@@ -466,6 +466,7 @@
address_tables_group_t atg;
/* Choose new hash table and free its old contents */
+ lock_get(perm_reload_lock);
if(*perm_addr_table == perm_addr_table_1) {
empty_addr_hash_table(perm_addr_table_2);
atg.address_table = perm_addr_table_2;
@@ -498,12 +499,14 @@
ret = reload_address_file_table(&atg);
}
if(ret != 1) {
+ lock_release(perm_reload_lock);
return ret;
}
*perm_addr_table = atg.address_table;
*perm_subnet_table = atg.subnet_table;
*perm_domain_table = atg.domain_table;
+ lock_release(perm_reload_lock);
LM_DBG("address table reloaded successfully.\n");
--- a/src/modules/permissions/permissions.c
+++ b/src/modules/permissions/permissions.c
@@ -95,6 +95,7 @@
static int perm_check_all_branches = 1;
time_t *perm_rpc_reload_time = NULL;
+gen_lock_t *perm_reload_lock = NULL;
int _perm_max_subnets = 512;
int _perm_load_backends = 0xFFFF;
@@ -642,6 +643,21 @@
return -1;
}
*perm_rpc_reload_time = 0;
+
+ perm_reload_lock = lock_alloc();
+ if(!perm_reload_lock) {
+ shm_free(perm_rpc_reload_time);
+ perm_rpc_reload_time = NULL;
+ SHM_MEM_ERROR;
+ return -1;
+ }
+ if(lock_init(perm_reload_lock) == 0) {
+ lock_dealloc(perm_reload_lock);
+ shm_free(perm_rpc_reload_time);
+ perm_rpc_reload_time = NULL;
+ LM_ERR("could not initialize reload lock\n");
+ return -1;
+ }
if(perm_reload_delta < 0)
perm_reload_delta = 5;
@@ -737,6 +753,12 @@
if(perm_rpc_reload_time != NULL) {
shm_free(perm_rpc_reload_time);
perm_rpc_reload_time = 0;
+ }
+
+ if(perm_reload_lock != NULL) {
+ lock_destroy(perm_reload_lock);
+ lock_dealloc(perm_reload_lock);
+ perm_reload_lock = NULL;
}
for(i = 0; i < perm_rules_num; i++) {
--- a/src/modules/permissions/permissions.h
+++ b/src/modules/permissions/permissions.h
@@ -30,6 +30,7 @@
#include "../../core/sr_module.h"
#include "../../lib/srdb1/db.h"
#include "../../core/pvar.h"
+#include "../../core/locking.h"
#include "rule.h"
#define DEFAULT_ALLOW_FILE "permissions.allow"
@@ -73,6 +74,7 @@
#define PERM_LOAD_DENYFILE (1 << 3)
extern int _perm_load_backends; /* */
extern time_t *perm_rpc_reload_time;
+extern gen_lock_t *perm_reload_lock;
typedef struct int_or_pvar
{
--- a/src/modules/permissions/trusted.c 2026-07-02 09:16:45
+++ b/src/modules/permissions/trusted.c 2026-07-02 13:31:11
@@ -98,6 +98,7 @@
}
/* Choose new hash table and free its old contents */
+ lock_get(perm_reload_lock);
if(*perm_trust_table == perm_trust_table_1) {
new_hash_table = perm_trust_table_2;
} else {
@@ -258,6 +259,7 @@
LM_ERR("hash table problem\n");
perm_dbf.free_result(perm_db_handle, res);
empty_hash_table(new_hash_table);
+ lock_release(perm_reload_lock);
return -1;
}
LM_DBG("tuple <%s, %s, %s, %s, %s> inserted into trusted hash "
@@ -269,6 +271,7 @@
perm_dbf.free_result(perm_db_handle, res);
*perm_trust_table = new_hash_table;
+ lock_release(perm_reload_lock);
LM_DBG("trusted table reloaded successfully.\n");
@@ -278,6 +281,7 @@
LM_ERR("database problem - invalid record\n");
perm_dbf.free_result(perm_db_handle, res);
empty_hash_table(new_hash_table);
+ lock_release(perm_reload_lock);
return -1;
}
```
### Additional Information
- [x] LLM/AI Assistants were involved in discovery, analysis or submitting of
the issue
- [x] It happened in a production deployment
- [x] It happened in a testing or development deployment
- [x] Testing or fuzzing tools were used to generate SIP traffic when it
happened
* **Kamailio Version** - output of `kamailio -v`
```text
version: kamailio 6.2.0-dev1 (aarch64/linux)
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC,
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, DBG_QM_MALLOC, DBG_F_MALLOC,
DBG_TLSF_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES,
TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE
262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled on 10:40:50 Jul 2 2026 with gcc 14.2.0
```
* **Operating System**:
```text
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
NAME="Debian GNU/Linux"
VERSION_ID="13"
VERSION="13 (trixie)"
VERSION_CODENAME=trixie
DEBIAN_VERSION_FULL=13.5
ID=debian
Linux 6.10.14-linuxkit #1 SMP Thu Aug 14 19:26:13 UTC 2025 aarch64 GNU/Linux
```
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/4809
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/[email protected]>_______________________________________________
Kamailio - Development Mailing List -- [email protected]
To unsubscribe send an email to [email protected]
Important: keep the mailing list in the recipients, do not reply only to the
sender!
