kuzaxak created an issue (kamailio/kamailio#4799)
### Description
A Kamailio worker (the tm **slow-timer** process) crashes with `SIGSEGV` in
`branch_builder()` during **DNS failover of a locally-generated UAC
transaction** — i.e. a transaction created *without* an originating SIP message
(dispatcher/keepalive `OPTIONS`, `uac` module request, etc.).
On final-response (FR) timeout the slow timer runs:
`final_response_handler()` → `add_uac_dns_fallback(t, msg=NULL, ...)` → (with
`reparse_on_dns_failover` enabled) `add_uac_from_buf(t, request=NULL, ...)` →
`print_uac_request_from_buf(t, i_req=NULL, ...)`.
`print_uac_request_from_buf()` dereferences `i_req` unconditionally
(`src/modules/tm/t_fwd.c`):
```c
/* ... we calculate branch ... */
if(!t_calc_branch(t, branch, i_req->add_to_branch_s,
&i_req->add_to_branch_len)) {
...
branch_str.s = i_req->add_to_branch_s;
branch_str.len = i_req->add_to_branch_len;
```
`add_to_branch_s` is an array member, so with `i_req == NULL` it evaluates to
the address `offsetof(sip_msg, add_to_branch_s)` (≈ `0x628` in our build) and
`&i_req->add_to_branch_len` ≈ `0x664` (Δ = `0x3c`, the scratch-buffer size).
Those bogus pointers become the output buffer passed to `t_calc_branch()` →
`branch_builder()` (`src/core/msg_translator.c`), which writes the magic cookie
/ `*len` through them → `SIGSEGV`. (`add_uac_from_buf()` *also* dereferences
`request` directly, e.g. `request->first_line.u.request.method.len`, so the
whole path is unusable when the originating message is NULL.)
Because a tm worker dies, the master aborts and the whole process restarts.
Under traffic this becomes a crash loop (recurring every ~70–90 s in our
deployment).
### Troubleshooting
#### Reproduction
Happens when all of the following hold:
- core: `use_dns_failover = on` (DNS cache on)
- tm: `reparse_on_dns_failover` enabled (the backtrace goes through
`add_uac_from_buf`)
- a transaction generated locally **without** a SIP message (e.g. dispatcher
keepalive `OPTIONS` via `t_uac`, or the `uac` module)
- that request times out with no reply, and its destination still has another
DNS target (`dns_srv_handle_next()` returns true)
In our case it was triggered by dispatcher-style keepalive `OPTIONS` to a
destination that had DNS-failover candidates.
#### Debugging Data
Symbolized backtrace (binary built with DWARF; site-specific IPs/hostnames
sanitized to placeholders):
```
#0 branch_builder (hash_index=..., label=0, char_v=..., branch=1,
branch_str=0x628 <error: Cannot access memory at address 0x628>,
len=0x664)
at src/core/msg_translator.c
begin = 0x628 <error: Cannot access memory at address 0x628>
#1 t_calc_branch (t=..., b=1, branch=0x628, branch_len=0x664)
at src/modules/tm/t_msgbuilder.c:1819
#2 print_uac_request_from_buf (t=..., i_req=0x0, branch=1, uri=..., len=...,
dst=...,
buf=0x... "OPTIONS sip: <TARGET_IP>:5060;transport=udp;conn_id=<uuid>
SIP/2.0\r\n"
"Via: SIP/2.0/UDP <proxy-host>:5065;branch=z9hG4bK...",
buf_len=493)
at src/modules/tm/t_fwd.c:655
branch_str = {s = 0x0, len = 0}
#3 add_uac_from_buf (t=..., request=0x0, uri=..., buf=0x... "OPTIONS sip:
<TARGET_IP>:5060;...",
buf_len=493) at src/modules/tm/t_fwd.c:900
#4 add_uac_dns_fallback (t=..., msg=0x0, old_uac=..., lock_replies=0)
at src/modules/tm/t_fwd.c:1056
#5 final_response_handler (r_buf=..., t=...) at src/modules/tm/timer.c:455
#6 retr_buf_handler (ticks=..., tl=..., p=...) at src/modules/tm/timer.c:534
#7 slow_timer_main () at src/core/timer.c:1105
#8 main_loop () at src/main.c:1996
#9 main (...) at src/main.c:3407
```
Crash site (`src/modules/tm/t_fwd.c::print_uac_request_from_buf`, identical in
6.0.6 / 6.0.7 / master):
```c
static char *print_uac_request_from_buf(struct cell *t, struct sip_msg *i_req,
int branch, str *uri, unsigned int *len, struct dest_info *dst,
char *buf, short buf_len)
{
char *shbuf;
str branch_str;
...
shbuf = 0;
/* ... we calculate branch ... */
if(!t_calc_branch(
t, branch, i_req->add_to_branch_s, &i_req->add_to_branch_len)) {
/* i_req == NULL here */
```
#### Log Messages
```
ALERT: <core> [main.c:815]: handle_sigs(): child process NN exited by a
signal 11
ALERT: <core> [main.c:819]: handle_sigs(): core was generated
CRITICAL: <core> [core/pass_fd.c:284]: receive_fd(): EOF on 60
```
#### SIP Traffic
The locally-generated keepalive `OPTIONS` (the transaction that has no
originating msg), sanitized. The crash is **independent of the URI content** —
it is the NULL `i_req` on the failover path:
```
OPTIONS sip: <TARGET_IP>:5060;transport=udp;conn_id=<uuid> SIP/2.0
Via: SIP/2.0/UDP <proxy-host>:5065;branch=z9hG4bK...
To: <sip: <TARGET_IP>:5060;transport=udp;conn_id=<uuid>>
From: <sip:proxy@<proxy-domain>>;tag=...
CSeq: 10 OPTIONS
Call-ID: ...
Max-Forwards: 70
Content-Length: 0
User-Agent: ...
```
### Possible Solutions
Minimal fix — decline DNS failover when there is no originating message (both
the `reparse_on_dns_failover` path via `add_uac_from_buf()` and the `add_uac()`
path require it). Applies cleanly to current `master`:
```diff
--- a/src/modules/tm/t_fwd.c
+++ b/src/modules/tm/t_fwd.c
@@ -1076,6 +1076,18 @@
int ret;
ret = -1;
+
+ /* A locally generated UAC transaction (e.g. dispatcher/keepalive
OPTIONS
+ * sent via t_uac, the uac module, ...) has no originating SIP message.
+ * The reparse/rebuild failover paths below dereference it
unconditionally
+ * (add_uac_from_buf() / print_uac_request_from_buf() ->
+ * i_req->add_to_branch_s, and add_uac() -> msg->rcv), which segfaults
when
+ * msg==NULL. Decline DNS failover for msg-less transactions. */
+ if(msg == NULL) {
+ LM_DBG("no originating request - skipping dns failover for a"
+ " locally generated transaction\n");
+ return ret;
+ }
if(cfg_get(core, core_cfg, use_dns_failover)
&& !((t->flags & (T_DONT_FORK | T_DISABLE_FAILOVER))
|| uac_dont_fork(old_uac))
```
Recommended defense-in-depth (independent of the above): NULL-guard `i_req` at
the top of `print_uac_request_from_buf()` and `request` in
`add_uac_from_buf()`, returning an error instead of dereferencing. Happy to
open a PR.
### Additional Information
* **Kamailio Version** - output of `kamailio -v`:
```
version: kamailio 6.0.6 (x86_64/linux)
flags: ... USE_DNS_CACHE, USE_DNS_FAILOVER, ...
```
The affected files `src/modules/tm/t_fwd.c` and `src/modules/tm/t_msgbuilder.c`
are **byte-identical to upstream 6.0.6, 6.0.7 and current master** (verified) —
the unguarded `i_req` dereference is present on `master`
(`print_uac_request_from_buf`, ~line 714). **6.0.6, 6.0.7 and master are all
affected.**
* **Operating System**: Amazon Linux 2023, kernel 6.12.x, x86_64.
Related but distinct (and never fixed): #2987 (locally-generated transaction +
DNS failover, bad-URI symptom).
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/4799
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/[email protected]>_______________________________________________
Kamailio - Development Mailing List -- [email protected]
To unsubscribe send an email to [email protected]
Important: keep the mailing list in the recipients, do not reply only to the
sender!
