cmcfarlen opened a new pull request, #13506: URL: https://github.com/apache/trafficserver/pull/13506
Publishes per-upstream-server connection metrics aggregated per hostname, using hidden metrics as the inputs and a derived metric for the aggregation, so the aggregation work happens on the stat sync task rather than on every connection. > **Depends on #13505**, which adds the hidden-metric store and `Derived::add_source` / `Op::MAX` that this builds on. The first 8 commits here are that PR; only the last two are new. Please merge #13505 first, after which this diff reduces to those two commits. ### `metric_enabled` becomes a level `proxy.config.http.per_server.connection.metric_enabled` changes from a flag to a level: | Value | Behavior | |---|---| | `0` | No per server metrics. | | `1` | Per group metrics stay hidden; the per hostname aggregates are published. | | `2` | As `1`, plus the per group metrics are mirrored into the published store. | The per group metrics are **always** created in the hidden store whenever metrics are enabled at all. Only what is registered for publication varies by level, so changing the level at runtime never has to migrate a metric between the two stores. ### Metrics Three hidden per group metrics (`current_connection`, `total_connection`, `blocked_connection`) feed four published per hostname aggregates: those three summed across the hostname's groups, plus `current_connection_max`, the `MAX` of the groups' current counts. `current_connection_max` is deliberately the maximum of the groups' *instantaneous* counts, not a monotone high-water mark. It answers "how close is the busiest group of this hostname to `per_server.connection.max` right now", which matters because that limit is enforced per group rather than per hostname. Because it rises and falls, a monitoring system can compute max-over-time over any window from it; a monotone value would collapse the time dimension and only report that a peak happened at some point, not when. There is no per group `current_connection_max` — it exists only as an aggregate. Aggregates are registered only for the `both` match type, the only one with more than one group per hostname. For `host`, `Group::metric_name` already returns the bare FQDN, so a host aggregate would collide with the single group's own name on one metric. ### No new hot-path work `TxnState::reserve`, `release` and `blocked` are unchanged. They already branch on `_count_metric != nullptr` and use the typed mutators, and `createHiddenPtr` returns the same pointer types, so those branches keep working with the metrics simply being hidden now. `git diff master -- include/iocore/net/ConnectionTracker.h` shows no logic added inside them. ### Also fixes the per group peak count `Group::_count_max`, reported as the `max` field of the connection tracker group dump, had two defects, both pre-existing: - `update_max_count()` made a single `compare_exchange_weak` attempt with no retry, so a racing update — or a spurious failure of the weak form, which is permitted by the standard and does occur on LL/SC architectures — silently discarded the sample. - It was only called when a maximum was configured. With metrics enabled and no configured maximum the count was reserved and then discarded, so the peak stayed at zero. ### Testing `tests/gold_tests/origin_connection/per_server_connection_max.test.py` is extended to cover: - Level 1: host aggregates published, and all three per group names absent from a normal query. - Level 2: host aggregates plus the per group metrics published. - Both levels: per group metrics visible via `traffic_ctl metric match per_server --include-hidden`. - **An aggregate genuinely spanning two groups** — one hostname mapped to two origin ports under `match: both`, with different concurrency per group (2 and 3) so the `SUM` (5) and the `MAX` (3) are distinguishable rather than coincidentally equal. Nothing previously covered a multi-group aggregate. - **Both gauges draining back to 0** after traffic stops and a further sync interval passes. This is the assertion that distinguishes an instantaneous gauge from a monotone peak, which could never satisfy it. Aggregates are recomputed on `ET_TASK` every `REC_RAW_STAT_SYNC_INTERVAL_MS` (5000 ms) and no record drives that interval, so the test sleeps past a tick rather than trying to configure a faster one; reading too early would silently compare against zeros. The autest passes. Documentation is added for `metric_enabled` and `metric_prefix`, neither of which was documented before, plus the metrics themselves under the monitoring guide. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
