## Pre-Submission Checklist - [X] Commit message has the format required by CONTRIBUTING guide - [X] Commits are split per component (core, individual modules, libs, utils, ...) - [X] Each component has a single commit (if not, squash them into one commit) - [X] Code is formatted with `clang-format` using the config file `.clang-format` from source code folder - [ ] No commits to README files for modules (changes must be done to docbook files in `doc/` subfolder, the README file is autogenerated)
## Type Of Change - [ ] Small bug fix (non-breaking change which fixes an issue) - [X] New feature (non-breaking change which adds new functionality) - [ ] Breaking change (fix or feature that would change existing functionality) ## Checklist: <!-- Go over all points below, and after creating the PR, tick the checkboxes that apply --> - [ ] PR should be backported to stable branches - [X] Tested changes locally - [ ] Related to issue #XXXX (replace XXXX with an open issue number) ## Description This PR adds a new operating mode for kamailio's TCP layer, selected by the existing global `tcp_main_threads`: - `0` — fd-passing (unchanged) - `1` — fd-passing + OpenSSL trampoline threads (was `>0`) - `2` — **full TCP reactor (new):** `PROC_TCP_MAIN` owns every TCP fd and performs all network I/O; the TCP worker processes become pure SIP engines. In mode 2 there is **no fd-passing**. Connections are accepted, read, reassembled, written, opened, closed, and timed out entirely inside `PROC_TCP_MAIN`. Reassembled SIP messages are dispatched to worker processes as small shared-memory task objects over a kernel-load-balanced datagram socket. The mode is opt-in and additive; modes 0 and 1 are unchanged. The motivation is architectural: **handling all network I/O in one process is simpler than shuttling connection file descriptors between processes with `sendmsg(SCM_RIGHTS)`.** --- ### Background: the fd-passing model In the current TCP architecture, `PROC_TCP_MAIN` owns the listen sockets and accepts connections, then **hands each connection's file descriptor to a TCP worker** via `sendmsg()`/`SCM_RIGHTS` over a UNIX socket. The worker then owns that fd and does the reads, reassembly, `receive_msg()`, and writes for that connection. This has served kamailio well, but the fd-passing at its core carries recurring structural costs: 1. **Every handoff is a control-message syscall.** Passing an fd is a `sendmsg`/`recvmsg` pair carrying ancillary `SCM_RIGHTS` data; the kernel installs (dups) the fd into the receiver's descriptor table. This happens on every inbound accept→worker handoff, and the async write path adds further fd movement to let a process write on a connection it does not currently hold. 2. **A passed fd is duplicated across processes.** After a pass, the same connection fd exists in *both* processes until closed in both. Descriptor accounting doubles for in-flight connections, and close/teardown must be coordinated across processes to avoid leaking or closing too early. 3. **Connection ownership is smeared across processes.** Because an fd can move, more than one process can end up wanting to write to the same connection, and **outbound connection creation to the same peer can race across workers** (multiple workers independently opening a socket to the same destination). 4. **The write path is the most complex part of the subsystem.** Writing on a connection you do not own requires either passing the fd back or driving a fd-shuffling async write queue. This is a large, subtle body of code precisely because ownership is distributed. 5. **Connection state is scattered.** Read buffers, partial-message reassembly state, write queues, and per-connection timers live in whichever worker currently holds the fd. Reasoning about, monitoring, and reaping connections means reasoning about distributed state across N workers. --- ### What this PR introduces (mode 2) `PROC_TCP_MAIN` runs one `io_wait` thread plus a small pool of I/O worker threads (harcoded to 8 at the moment). It owns all TCP fds for their entire lifetime. - **Reads:** `PROC_TCP_MAIN` reads and reassembles complete SIP messages. Each assembled message is copied into a shared-memory task object and its pointer is written to an `AF_UNIX`/`SOCK_DGRAM` socketpair. All TCP workers `recvfrom()` that one socket; **the kernel load-balances** delivery across them — no explicit worker assignment or affinity is needed. - **Writes:** any process that wants to send posts a *write request* to `PROC_TCP_MAIN` (`CONN_WRITE_REQ`) with the payload in shared memory. `PROC_TCP_MAIN` queues and transmits it. **No fd is passed for writes.** - **Outbound:** a worker with no usable connection posts a *connect request* (`CONN_CONNECT_REQ`); `PROC_TCP_MAIN` performs the `socket()`+`connect()`, hashes the connection, and sends the first payload. Creation is serialized in one process. - **Workers:** own no fds, run no io_wait loop, do no raw socket I/O, and (see below) run no OpenSSL. They are stateless SIP processors fed by the dispatch socket. Inside `PROC_TCP_MAIN`, per-connection work (read/reassemble, encrypt/decrypt, write) is handed to the thread pool under a single-owner-per-connection rule, so the io_wait thread never blocks on a slow read, a large write, or a TLS handshake. --- ### Why: the value of owning all network I/O in one process #### 1. No `SCM_RIGHTS` shuttling The per-connection `sendmsg`/`recvmsg` fd handoff disappears from the hot path. Inbound connections are never passed; writes are never passed; the async write path no longer moves descriptors. What crosses a process boundary instead is a single pointer (to a shm task) over a datagram socket — a fixed, tiny, ancillary-data-free transfer that the kernel fans out for free. #### 2. No cross-process fd duplication or split lifecycle Each connection fd exists in exactly one process. There is no window where the same descriptor lives in two descriptor tables, and no cross-process protocol for "who closes it." Descriptor accounting and connection teardown become local and unambiguous. #### 3. Multi-writer races are eliminated by construction Only `PROC_TCP_MAIN` ever opens, writes, or closes a connection. The outbound-connect-to-same-peer race is gone because creation is serialized in one process. "Two workers writing the same socket" cannot occur because workers hold no sockets. #### 4. The write path collapses "Write on a connection you don't own" becomes "post a write request to the one owner." The distributed async-write / fd-shuffle machinery is replaced by a single queue owned by a single process. Less code, fewer states, fewer corners. #### 5. Connection lifecycle is centralized One hash table, one timer wheel, one reaper, in one process. Idle timeouts, partial-read timeouts, and error teardown are local decisions on local state rather than a cross-process dance. #### 6. Clean separation of concerns → independent scaling The reactor does network I/O; the workers do SIP logic. Workers become CPU-bound, stateless message processors with no I/O responsibilities, fed by a kernel-balanced queue. The number of I/O threads and the number of SIP processors are now independent knobs rather than both being tied to "one fd owner per worker." #### 7. TLS: the single-process constraint, resolved rather than worked around OpenSSL's per-connection `SSL` objects and the shared `SSL_CTX` carry process-local heap pointers and assume in-process locking. They **cannot be shared across processes** without installing custom shared-memory crypto allocators *and* `PROCESS_SHARED` pthread-mutex hooks — a fragile arrangement. The practical consequence under fd-passing is that a TLS connection's cryptographic state cannot follow its fd to an arbitrary worker. `tcp_main_threads=1` already acknowledges this: its trampoline relays each OpenSSL operation from the worker to a dedicated thread *inside* `PROC_TCP_MAIN` over a per-worker socketpair — keeping the socket in the worker but the crypto in `PROC_TCP_MAIN`. It works, but it is inherently a **two-mechanism** design: the fd lives in one process and its crypto in another, coordinated on every call, with one relay thread and socketpair pair per worker. Mode 2 dissolves the split. The fd **and** its `SSL` object live and are used in the same process, so the single-process constraint is satisfied *by construction* — no relay, no cross-process crypto, no allocator/mutex hooks. Since the crypto had to be centralized in `PROC_TCP_MAIN` anyway, centralizing the I/O alongside it is the natural simplification, and it removes the trampoline relay entirely. One pool thread owns a connection (and thus its `SSL`) at a time, so OpenSSL is driven correctly with nothing more than modern OpenSSL's own in-process thread safety. --- ### Design notes - **Dispatch:** assembled messages are shm task objects (receive-info + message buffer); the pointer is a single `uintptr_t` written to the dispatch socketpair — well within `PIPE_BUF`, so delivery is atomic and lossless. Workers cast back and call `receive_msg()`, then free the task. - **In-process pool:** a `io_wait` thread owns file descriptors/timers/hash/lifecycle exclusively; pool threads run read/write/crypto on a connection that has been *shielded* (removed from io_wait and the timer) for the duration of a job, then signal completion back to the io_wait thread. This keeps a strict, small concurrency contract: connection metadata mutation stays on one thread. - **Cross-process wakeups** use a `PROCESS_SHARED`, robust condition variable with owner-died recovery. --- ### Compatibility and safety - **Opt-in.** Behaviour is unchanged unless `tcp_main_threads=2` is set. - **Modes 0 and 1 are unchanged** — verified: the reworked paths are gated on `tcp_main_threads == 2`; the legacy fd-passing and trampoline code paths are not modified. - **Incremental.** The series builds the mode step by step (dispatch socket, task struct, read path, write path, outbound path, then the in-process thread pool, then TLS offload), each step preserving the mode-0 path. --- ### Testing / validation - **No regression** on the plain-TCP path in mode 2, including the outbound dispatcher probe socket. - **Inbound TLS** (TLS 1.2 and 1.3): handshake + REGISTER + in-dialog request/ response handled entirely through the reactor and its pool. - **Writes from a non-TCP process:** a config that relays inbound TLS to UDP and back exercises a UDP receiver issuing a TLS write — validating that *any* process can write without owning the fd. - **Outbound TLS:** connection opened in `PROC_TCP_MAIN`, handshake driven by the pool, OPTIONS ping answered (dispatcher marks the target active). - **Concurrency soak:** 256 simultaneous TLS UACs, repeated. No crash; connections drain back to baseline; shared memory and fragmentation flat across repeated runs (no leak); no TLS record corruption under contention (confirms per-thread crypto isolation). - TODO: add a container to the kamailio test repo for this use case --- ### Known limitations / follow-ups (tracked) - **WS/WSS** are not yet offloaded to the pool in mode 2 (handled inline on the io_wait thread). Functional, but the websocket framing layer still needs a thread-safety audit before offload. - **`event_route[tls:connection-out]`** does not fire under `tcp_main_threads > 0` (the handshake completes where the outbound-send context is not populated). It is documented as unsupported in this mode pending a rework that runs the event on a worker. This is a pre-existing property of TLS-in-`PROC_TCP_MAIN` (modes 1 and 2), not introduced here. - Mode 2 relies on OpenSSL ≥ 1.1.0 (in-process thread safety); this should be made an explicit requirement. --- ### Risk & rollback The feature is gated behind a single global that defaults off. Setting `tcp_main_threads` back to `0` (or `1`) fully restores the prior behaviour with no residual effect, because those code paths are untouched. The blast radius of a problem is therefore bounded to deployments that explicitly opt into mode 2. You can view, comment on, or merge this pull request online at: https://github.com/kamailio/kamailio/pull/4800 -- Commit Summary -- * core: tcp reactor - add mode 2 dispatch socketpair * core: tcp reactor - add dispatch task struct * core: tcp reactor - skip trampoline threads and socketpairs in mode 2 * core: tcp reactor - direct-call OpenSSL trampoline in mode 2 * core: tcp reactor - dispatch reassembled SIP messages to workers * core: tcp reactor - route reassembled messages through dispatch * core: tcp reactor - read and reassemble in tcp_main (handle_tcpconn_ev) * core: tcp reactor - CONN_WRITE_REQ to queue worker writes in tcp_main * core: tcp reactor - workers receive dispatched SIP tasks (F_TCP_REACTOR) * core: tcp reactor - tcp_send ships existing-connection writes to tcp_main * core: tcp reactor - encrypt queued TLS writes in tcp_main * core: tcp reactor - open outbound connections in tcp_main (CONN_CONNECT_REQ) * core: tcp reactor - handle self-initiated sends inside PROC_TCP_MAIN * core: tcp reactor - reap stalled partial SIP message reads * core: tcp reactor - document worker busy-tracking bypass in mode 2 * core: tcp reactor - create the dispatch socketpair before forking * core: tcp reactor - read from c->s via _tconfd() in tcp_main * core: tcp reactor - fix self-directed writes from PROC_TCP_MAIN * core: tcp reactor - add process-shared robust condition variable * core: tcp reactor - pool, job and write-queue data structures * core: tcp reactor - add the pool thread routine and helpers * core: tcp reactor - offload plain-TCP reads to the pool * core: tcp reactor - keep conn->flags io_wait-exclusive under POOL_BUSY * core: tcp reactor - offload writes to the pool with wsq staging * core: tcp reactor - defer close of a POOL_BUSY connection * core: tcp reactor - guard the idle timeout against POOL_BUSY connections * core: tcp reactor - document the pool concurrency contract * core: tcp reactor - offload TLS reads and writes to the pool -- File Changes -- A src/core/tcp_cond.c (142) A src/core/tcp_cond.h (69) M src/core/tcp_conn.h (37) M src/core/tcp_main.c (1300) M src/core/tcp_mtops.c (62) M src/core/tcp_mtops.h (4) A src/core/tcp_reactor.c (77) A src/core/tcp_reactor.h (152) M src/core/tcp_read.c (100) M src/core/tcp_read.h (3) M src/core/tcp_server.h (6) M src/modules/tls/tls_server.c (21) M src/modules/tls_wolfssl/tls_server.c (17) -- Patch Links -- https://github.com/kamailio/kamailio/pull/4800.patch https://github.com/kamailio/kamailio/pull/4800.diff -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/pull/4800 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/pull/[email protected]>
_______________________________________________ Kamailio - Development Mailing List -- [email protected] To unsubscribe send an email to [email protected] Important: keep the mailing list in the recipients, do not reply only to the sender!
