cmcfarlen opened a new pull request, #13821:
URL: https://github.com/apache/trafficserver/pull/13821
The inactivity cop walks every connection on its thread once a second, so
its cost grows with connection count rather than with the number of timeouts
actually due. On a busy host each net thread pauses its poll loop for
milliseconds every second, with occasional passes long enough to show up as
latency.
This replaces the walk with an intrusive timer wheel: each `NetEvent` is
scheduled at its earliest deadline, and a cop pass only looks at connections
whose deadline has come due.
## What changes
- `InactivityCop` moves out of `UnixNet.cc` into `InactivityCop.h` and takes
its `NetHandler` by injection, so it can be benchmarked in isolation.
- New `TimerWheel` in `include/tscore/TimerWheel.h`: an intrusive,
single-level wheel with a delegate concept for firing, plus unit tests.
- Deadlines are scheduled through a single `rearm_timer()` path on
`NetEvent`, called from `startCop` / `stopCop` and from the timeout setters.
The wheel may hold a deadline that is earlier than the real one, never later.
That's what lets `netActivity()`, the hot I/O path, push a deadline later
without touching the wheel: when the bucket comes due, the cop re-reads the
real deadline and re-arms anything not yet due.
- `cop_list`, `cop_link` and the full active-queue scan the cop forced every
tick (added for TS-4131) are removed. The wheel already fires active timeouts,
so that scan was duplicate work.
- Each pass is capped at a configurable budget,
`proxy.config.net.inactivity_cop_budget` (default 4096). When a pass hits the
budget it yields to the poll loop and continues 1 ms later.
- New metrics: `inactivity_cop_passes`, `_visited`, `_fired`,
`_pass_time_us`, `_pass_max_us` and `_budget_exhausted`.
- An opt-in audit (`inactivity_cop_audit` debug tag) finds connections with
a live deadline that the wheel will never visit. That's the TS-4131 failure
mode, made detectable instead of papered over by a full scan.
- Developer-guide page on network timeouts, plus records and metrics docs.
`214b029ecf` ("Report a timeout when evicting a connection from a full
queue") is a standalone bug fix that later tests depend on; I can split it into
its own PR first if reviewers prefer.
## Results
**Benchmark** (`test_net '[inactivity_cop]'`, 100k connections, none due): a
pass drops from ~15.9 ms to ~0.1 µs. More importantly, the number of
`get_mutex()` and `get_thread()` calls per pass falls from N to 0. Those counts
are exact, so they don't depend on machine noise.
**Production trial**: two identical production hosts, same hardware (64
cores), same traffic (~171k open connections each), measured in the same minute
with a bpftrace uprobe on `InactivityCop::check_inactivity`:
| | before | after |
|---|---|---|
| average pass | 1,323 µs | 673 µs |
| passes over 2 ms | 57 / min | 5 / min |
| worst pass | 22.0 ms | 6.5 ms |
| cop CPU | 0.085 cores | 0.043 cores |
On a host with ~358k connections the unmodified cop averaged 4.2 ms per
pass, with a 12 ms worst case, so the old cost grows faster than linearly with
connection count. The CPU saving is small; the shorter, more predictable
poll-loop pauses are the main benefit.
Timeout behaviour matched: keep-alive and HTTP/2 inactivity close rates on
the trial host agreed with its unmodified siblings to within 0.1%.
## Testing
- New unit tests for `TimerWheel` and the cop (`[inactivity_cop]`),
including rearm, cancel, wraparound, budget, multi-tick passes, and the audit.
- `ctest` passes locally on macOS, except `test_jsonrpcserver`, which fails
there on clean master too.
- Autests: `timeout` passes (on an earlier base).
Draft so it can get eyes on the design while I find another host for a
higher-connection trial.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
