dmsolr opened a new pull request, #7674:
URL: https://github.com/apache/shenyu/pull/7674
## Which issue is this PR closes?
Closes #7647.
Tracker: #7650.
## What does this PR change?
`UpstreamCheckTask` schedules fixed-delay rounds, submits every registered
upstream to a separate executor and waits for the round inside a synchronized
section. Large batches or slow endpoints can increase round duration and delay
subsequent checks, but today there is no way to tell from the outside:
- whether a round is falling behind the configured `checkInterval`;
- whether the executor or the endpoints are the bottleneck;
- how stale the published health state actually is.
Issue #7647 asks for exactly these signals to be recorded **before** any
scheduling change is evaluated ("Record round duration, submitted/pending
checks, endpoint latency and health-state age before changing scheduling").
This PR adds the observability layer only — no scheduling, locking, threshold
or race-condition change. Bounding in-flight work, batching or staggering stays
measurement-gated and can be a follow-up once operators have real numbers.
### Per-round signals
All readable via new public getters on `UpstreamCheckTask`:
| Getter | Meaning |
| --- | --- |
| `getLastRoundDurationMillis()` | Monotonic `System.nanoTime()` elapsed
across the whole synchronized round (submit + wait). Reflects real time to
converge, not publisher-construction time. |
| `getLastRoundSubmittedChecks()` | How many probes were dispatched this
round. |
| `getLastRoundPendingChecks()` | How many of those were still pending when
the round entered `waitFinish()`. A persistently high value points at the
executor, not the endpoints. |
| `getLastRoundMaxEndpointLatencyMillis()` | Worst single-probe latency
observed this round. Only probes with health-check enabled contribute (probes
short-circuited by `!upstream.isHealthCheckEnabled()` are not measured). |
| `getLastRoundOldestHealthStateMillis()` | Max age across healthy and
unhealthy buckets, computed from `Upstream.getLastHealthTimestamp()` /
`getLastUnhealthyTimestamp()`. Grows when rounds fall behind `checkInterval`. |
| `getLastRoundCompletedAtMillis()` | Wall-clock finish timestamp. |
### Cumulative counters
`getTotalRounds()`, `getTotalSubmittedChecks()`,
`getTotalEndpointLatencyMillis()` let operators compute long-run averages
without a metrics backend.
### Logging
A single DEBUG-level summary line per round:
```
[Health Check] round summary: durationMs=..., submitted=...,
pendingAtWait=..., maxEndpointLatencyMs=..., oldestHealthStateMs=...,
healthySelectors=..., unhealthySelectors=...
```
DEBUG (not INFO) keeps log volume aligned with #7643. The existing INFO
lines on state transitions (`server is back online` / `server is offline`) and
`print()` are unchanged.
### Implementation notes
- Per-round scratch counters (`AtomicInteger currentRoundSubmittedChecks`,
`AtomicLong currentRoundMaxEndpointLatencyMillis`) are reset at the start of
every `healthCheck()` and copied into the `volatile lastRound*` fields once the
round terminates, so callers never observe a half-updated snapshot.
- `lastRoundPendingChecks` is captured after `doHealthCheck()` and before
`waitFinish()` by walking the `futures` list once with `isDone()`.
- `computeOldestHealthStateMillis()` walks both buckets once per round; the
cost is bounded by the same population the round is already probing.
- Endpoint latency uses `System.nanoTime()` around
`UpstreamCheckUtils.checkUrl(...)` only, in a `finally` block, so the counter
is updated even when the probe throws.
- `recordRoundMetrics` runs in `finally`, so the counters still advance when
the round aborts on exception.
## Tests
New tests in `UpstreamCheckTaskTest`, all deterministic (invoke `run()`
directly instead of relying on the scheduled executor):
- `roundMetricsAreRecordedAfterRunCompletes` — after one synchronous `run()`
with four registered upstreams, all last-round fields are populated and
`submitted` covers every upstream.
- `emptyPopulationStillRecordsRoundCompletion` — `totalRounds` advances even
with zero upstreams, so operators can distinguish "task not running" from "task
running with nothing to check"; `maxEndpointLatency` stays zero.
- `oldestHealthStateAgeReflectsStaleTimestamp` — an upstream whose
`lastHealthTimestamp` is 5 s old surfaces as an oldest-age of at least 4 s, so
the metric does not collapse to zero.
- `perRoundCountersAreResetBetweenRounds` — a second `run()` resets the
per-round scratch counters instead of leaking values from the previous round.
Module run:
```
./mvnw test -pl shenyu-loadbalancer -Dtest=UpstreamCheckTaskTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0
./mvnw checkstyle:check -pl shenyu-loadbalancer
You have 0 Checkstyle violations.
```
## Measurement status
The tracker issue requires a benchmark with increasing upstream counts and
partial slow/failing endpoint populations, plus a probe-enabled configuration
note (bootstrap currently disables this feature by default). This environment
does not have a load generator, controllable slow upstreams or JFR profiling
harness, so no numeric before/after is attached here.
The change is source-grounded and does not require a benchmark to be safe:
it only reads state that is already maintained, plus two `System.nanoTime()`
calls per round and one per probe. Once merged, operators running with
health-check enabled can collect the numbers the issue asks for directly from
these getters, and a follow-up PR can decide whether bounded in-flight work /
batching / staggering is actually justified.
## Compatibility
- No public API removed or renamed. Only new getters are added.
- No scheduling, locking, threshold or race-condition change. The
`synchronized (lock)` scope, `tryStartHealthCheck` / `finishHealthCheck` gating
and `triggerRemoveAll` race protection are unchanged.
- No new runtime dependency.
- No metric cardinality expansion: all signals are scalars per task
instance, no per-upstream or per-selector labels are added.
- Healthy/unhealthy transition semantics and the #6502 recovery scenario are
untouched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
