jineshparakh opened a new pull request, #19500:
URL: https://github.com/apache/pinot/pull/19500
## Summary
A freshly (re)started broker reports Ready as soon as Helix routing
converges, but its query serve path
is still cold at that instant: the SQL compile path, the scatter gather and
reduce code, and the response
build path are all uncompiled (interpreted or only partially JIT compiled),
and per query caches are
empty. The first burst of real traffic therefore runs the serve path cold,
which on a constrained broker
saturates CPU and inflates latency enough that clients cancel (the cold
start cancellation storm).
This PR warms the serve path with real probe queries **before** readiness is
granted, behind a readiness
gate. When enabled the broker reports `STARTING` until the probes have
driven the query path warm (or a
budget expires), so the first real traffic hits an already warm broker. It
is off by default, opt in per
deployment, bounded so it can never stall a rolling restart, and it
complements the server pre-connect
change (which removes the broker to server connect and TLS handshake from
the first query; this change
removes the serve path cold start).
## What it does
After Helix converges, a background thread runs probe queries through
`QueryRouter`, deliberately
bypassing access control, query quota, and the query log, so warmup needs no
synthetic identity and
pollutes no customer facing surface. Each probe exercises the full serve
path: compile, route, scatter,
receive and deserialize the server DataTables, and reduce. A small local no
network step (stage 1) also
compiles a throwaway query and serializes a synthetic response, warming the
SQL compile and JSON response
paths even if no server is reachable.
The exit condition is a **depth floor OR a budget ceiling**, never a latency
guess:
- **Default probe** (`SELECT * FROM "<t>" LIMIT 1` over a greedy set cover
of tables that covers every
routable server): warmup declares the broker warm once at least
`minIterations` probes have completed
successfully. This is a depth floor that guarantees enough invocations to
drive the query path's JIT to
its top tier. A latency target is deliberately not used: a trivial probe
reaches low latency after a
handful of iterations while the code is still only partially compiled, so
latency is a false early exit
signal (this was verified experimentally, see below).
- **Custom probe query** (`warmup.query`): one or more representative
production query shapes (separated
by `;`), all run every round, exiting on the same `minIterations` depth
floor or the budget. A
probe-count floor is a cost-independent warm signal; latency plateauing is
not (a trivial query reaches a
stable latency while still cold), so it is deliberately not used. Several
shapes together warm several
serve-path branches, so a scan, an aggregation, a group by and a join can
each warm their own branch.
- **Budget**: a hard ceiling measured from Helix convergence. Readiness
opens when it expires whatever the
probe progress, so a slow or unreachable server can never stall a rolling
restart. The gate also opens
if warmup throws or is interrupted.
Probes are fired `concurrency` at a time per round. Serial (1, the default)
warms the serve path; a higher
value additionally warms the concurrency step (channel lock contention,
concurrent scatter gather and
reduce, GC under load) that the first real burst hits.
This is **single stage (SSE) only**. The multi stage (gRPC) and time series
paths are unaffected.
## Flow
Warmup runs after Helix convergence, behind the readiness gate. Every path
is bounded by the budget
(it never hangs), and the gate opens whether warmup succeeds or times out.
The default probe picks
its tables by a deterministic greedy set cover over routable servers; a
custom `warmup.query`
bypasses selection.
```text
warmUp(config, deadline) after Helix convergence, behind the
STARTING readiness gate
|
|- warmUpLocal() stage 1: compile "SELECT 1" + serialize a
response (no network)
|- create probe pool (concurrency daemon threads)
|
|- hasCustomQuery() ?
|
|=== NO: AUTO-SELECT === default probe: SELECT * FROM "<t>" LIMIT 1
| |- tables = hasCustomTables ? filterRoutableTables() (keep only
routable)
| | : selectProbeTables() (greedy
set-cover, see below)
| |- tables empty (routing not ready)? -> backoff + retry, until budget
| |- reportServerCoverage() ONCE -> WARN +
BROKER_WARMUP_UNCOVERED_SERVERS gauge
| |- each round: roundRobinBatch(tables, probeSeq, concurrency) ->
probe()
| '- EXIT: successfulProbes >= minIterations -> TRUE | budget
expired -> FALSE
|
|=== YES: CUSTOM QUERY === one or more shapes separated by ';'
| |- compile-validate each query (skip bad; none valid -> FALSE)
| |- each round: run ALL queries x concurrency -> probe()
| '- EXIT: successfulProbes >= minIterations -> TRUE | budget -> FALSE
(same floor as default)
|
'- return TRUE | FALSE -> readiness gate opens EITHER WAY (bounded by
budget; never hangs)
selectProbeTables(maxTables) greedy set-cover, deterministic
uncovered = servers THIS broker routes to (union of servingInstances
over routable tables,
NOT the cluster-wide
getRoutableServerInstanceMap)
candidates = routable tables, SORTED (same choice on every
broker / restart)
for table in candidates:
stop if |selected| == maxTables OR uncovered is empty
servers = servingInstances(table) (null/empty -> skip)
if servers cover >= 1 uncovered server: select table; remove them
from uncovered
return selected
# maxTables hit with servers still uncovered -> reportServerCoverage()
warns + sets the gauge
probe(request, timeout)
getTableRouteInfo(raw name)
-> shouldRouteOffline / shouldRouteRealtime -> typedRequest(typed name)
per leg
-> calculateRoutes -> submitQuery -> getFinalResponses -> reduce
(best-effort)
-> return latency (or -1 on failure; never throws out)
```
## Configuration
| Property | Default | Description |
|---|---|---|
| `pinot.broker.startup.warmup.enabled` | `false` | Run probe queries at
startup and gate readiness on warmup. Off by default. |
| `pinot.broker.startup.warmup.budgetMs` | `15000` | Hard ceiling on the
whole warmup (from Helix convergence). Readiness opens when it expires. |
| `pinot.broker.startup.warmup.minIterations` | `1000` | Default probe depth
floor: warmup exits once this many probes have completed, or the budget
expires. Enough invocations to reach top tier JIT. |
| `pinot.broker.startup.warmup.maxTables` | `5` | Upper bound on tables
probed by the default probe (a set cover covering every routable server).
Governs coverage breadth: raise it if `BROKER_WARMUP_UNCOVERED_SERVERS` is
non-zero. |
| `pinot.broker.startup.warmup.query` | `` | Optional. One or more complete
probe queries (each names its own table), separated by `;` when more than one,
so several representative shapes warm several serve paths. Empty means auto
select the default probe. All configured queries run every round and exit on
the same `minIterations` floor. |
| `pinot.broker.startup.warmup.tables` | `` | Optional comma separated
tables to probe with the default query. Empty means auto select via set cover.
Ignored when `warmup.query` is set. |
| `pinot.broker.startup.warmup.concurrency` | `1` | Probes fired
concurrently per round. Serial by default. |
## Metrics
| Metric | Type | Meaning |
|---|---|---|
| `BROKER_WARM` (gauge) | gauge | `1` once warmup has completed and
readiness may be granted, `0` while warming. Always `1` when warmup is
disabled. |
| `STARTUP_WARMUP_DURATION_MS` | timer | Wall clock duration of warmup, from
Helix convergence to readiness being released. |
| `BROKER_WARMUP_UNCOVERED_SERVERS` | gauge | Number of servers this broker
routes to (the union of serving instances over its routable tables,
tenant-scoped, not the cluster-wide server map) that the chosen probe tables do
not, between them, cover. `0` means the probes span every server this broker
routes to. Non-zero means `maxTables` capped the set cover short (raise it) or
a custom `warmup.tables` list omits some servers. Recorded once per warmup run;
not recorded when warmup is disabled. |
## Performance
### Setup
All numbers below are from a **TLS-enabled** single broker test cluster (a
real ZK plus controller plus
server plus broker, not a synthetic mock), with both client to broker and
broker to server TLS on:
- One offline table: about **4.0 million documents** across **41 segments**,
**MMAP** load mode, single
replica.
- **Open loop** synthetic cold load (step arrival, no ramp) at a fixed
arrival rate. Because arrivals do
not wait for responses, under elevated cold latency the in flight count
builds to several hundred
concurrent requests.
- Each run is split into a **cold** window `[0, 8 s)` and a **warm** window
`[150, 180 s)` measured from
the first traffic at readiness. A cancellation is a query whose client
round trip exceeds 2 s.
- **5 cold restarts per arm** (each arm = one setting of the warmup config),
toggled via controller
cluster config so the operator stays running.
### Gain
On a **constrained (2 vCPU) broker at about 288 queries/second**, deep
warmup nearly eliminates the cold
start cancellation storm (median / mean / range of cold cancellation rate, 5
restarts per arm):
```
warmup OFF (pre-connect only) : 39.4% / 39.6% / [22.1, 52.7] serve
time (timeUsed) avg 1095 ms
warmup ON, deep : 4.8% / 3.5% / [ 0.0, 6.3] serve
time (timeUsed) avg 115 ms
```
That is about a **91% reduction** in cold cancellations, with cold serve
time down about **9.5x**
(1095 ms to 115 ms) and peak concurrent in flight down about **4x** (1008 to
250), because faster serve
means lower latency, which by Little's law means lower concurrency, which
avoids the CPU saturation that
drove the storm. Every warm window across every arm was 0% cancellations at
about 2.5 ms wall.
A shallow warmup is not enough: an early exit after only 2 to 3 probes left
the path only partially
compiled and the cancellation rate at about 29%, which is why the shipped
default exits on a probe count
floor (`minIterations`) rather than on a latency target.
## Testing
- **Unit** (`BrokerWarmupTest`, 20 tests): config parsing including the
`concurrency`, `minIterations` and
`maxTables` floors, the custom `query` (multiple queries separated by `;`)
and `tables` overrides, and
the no op interface default; the greedy set cover selection (covers every
server, skips redundant tables,
respects `maxTables`, handles an empty cluster, ignores tables without
serving instances); the coverage
check (`uncoveredRoutableServers` and the
`BROKER_WARMUP_UNCOVERED_SERVERS` gauge emission); the
round-robin batch rotation (advances across rounds, wraps, survives index
overflow); the
offline/realtime/hybrid probe routing decision across every table-type
combination; the custom-table
filter; the tenant-scoped coverage universe (a cluster server no table
routes to is not counted); and
the `concurrency` upper clamp.
- **Integration** (`BrokerStartupWarmupIntegrationTest`, 6 tests): brings up
a real ZK plus controller plus
server plus broker with an offline table and asserts, against a live
server: (1) with warmup enabled the
readiness gate opens and the broker's `ServiceStatus` reaches `GOOD`; (2)
the default auto-select path
(`RoutingManager` set cover to `QueryRouter` probe to reduce) reaches its
probe-count floor and returns
`true`; (3) the custom multi-query path warms and returns `true`; (4)
concurrent probes (`concurrency`
greater than 1) reach the floor; (5) an explicit `tables` list warms; and
(6) an unreachable floor with a
short budget returns `false` promptly rather than hanging.
## Backward compatibility
- Off by default, so existing deployments are unchanged.
- Readiness reports the existing `STARTING` status, with no new status enum
value, so mixed version
broker/controller peers are unaffected.
- The `BROKER_WARM` gauge always reads `1` when warmup is disabled, so
dashboards read identically on
untouched deployments.
- No wire protocol or serialization changes.
## Relationship to server pre-connect
This change is independent of and complementary to the broker startup server
pre-connect change.
Pre-connect removes the broker to server connect and TLS handshake from the
first query; this change
removes the serve path cold start. Each is independently valuable and either
can be enabled without the
other. When both are present they compose: each registers its own `STARTING`
readiness gate, and readiness
is granted only once both are satisfied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
