AlinsRan opened a new pull request, #13941:
URL: https://github.com/apache/apisix/pull/13941
### Description
A node that joins an Upstream today takes its full share of traffic
immediately. For a JVM that has not JIT-compiled yet, a service with an empty
local cache, or one still filling its connection pools, the scale-out meant to
add capacity is what drives its latency up or knocks it over. Tuning node
weights by hand does not survive autoscaling or rolling releases.
This adds `warm_up_conf` to the Upstream: a node the gateway observes for
the first time takes a reduced share of the traffic and ramps back to its
configured weight over `slow_start_time_seconds`.
```json
{
"type": "roundrobin",
"nodes": [{"host": "10.0.0.10", "port": 8080, "weight": 100}],
"warm_up_conf": {
"slow_start_time_seconds": 300,
"min_weight_percent": 1,
"interval": 1,
"aggression": 1,
"startup_grace_period_seconds": 180
}
}
```
Without `warm_up_conf` nothing changes: no state is created, the picker
cache key is untouched, and no shared dict is written.
#### Relation to #12991
#12991 proposed the same feature with the same four core fields and the same
weight curve, which this keeps. What it does not keep is where the start of a
ramp comes from. There, the Admin API wrote `update_time` onto each node and
persisted it to etcd. That makes the ramp depend on the clock and the honesty
of whoever writes the configuration, pollutes the declarative config with
runtime state, and does nothing for nodes that never pass through the Admin API
- every node from service discovery.
Here the data plane decides on its own. `apisix/slow_start.lua` compares the
node set of each picker build against the one the previous build recorded in a
new `upstream-slow-start` shared dict, and generates every ramp start locally
with `ngx.now()`. Nothing is read from the node configuration, and nothing is
written back to etcd.
#### Lifecycle
- **Baseline.** The node set an Upstream has when the gateway first builds a
picker for it is mature. That covers a cold start, a full resync, and turning
`warm_up_conf` on for an Upstream that is already serving.
- **New nodes** added afterwards ramp from `min_weight_percent` over
`slow_start_time_seconds`. A node an active health check keeps out of the
picker starts its ramp when it first becomes pickable, not when the
configuration first mentioned it.
- **Tombstone.** A node that leaves the Upstream keeps its state for one
window, so coming back within it resumes the ramp. Coming back later, or being
kept out of the picker by a health check for longer than a window, starts over
- a different process is probably answering on that address.
- **Identity** is host and port; a domain node is identified by its
configured hostname, so weight, priority, metadata, order, and DNS rotation do
not restart anything.
- **`startup_grace_period_seconds`** keeps nodes that arrive late during a
restart mature, so start-up ordering does not manufacture new nodes.
#### Weight curve
Same as #12991 and Envoy's `SlowStartConfig`:
```
elapsed = clamp(now - ramp_start, 0, slow_start_time_seconds)
time_factor = max(elapsed, 1) / slow_start_time_seconds
ratio = max(min_weight_percent / 100, time_factor ^ (1 / aggression))
weight = floor(configured_weight * ratio) -- 0 stays 0, otherwise at
least 1
```
#### Request path
While any node ramps, the picker cache key carries the current `interval`
bucket, so a worker rebuilds the picker once per bucket rather than doing
per-node work per request. Once every node is mature the key settles on a
stable suffix, and the rebuild that causes is the one that restores the full
weights. The hot path does a single `shdict:get` of an aggregate deadline. A
request and its retries keep the same picker.
#### Consistency across workers
Workers reconcile in parallel and the shared dict has no compare-and-set, so
state is never read and then written back blindly:
- **Presence and eligibility are tracked separately.** Presence comes from
the configuration, identical in every worker, and drives the snapshot and the
tombstone. Eligibility is each worker's own health view, and only decides when
a ramp starts or is interrupted. Tombstoning off the eligible set would let one
worker's transient health opinion drop a node the others still serve.
- **A ramp start is only created with `add`**, and only replaced through an
election - an `add` on a key naming the start being replaced, whose winner
every other worker adopts. Refreshes use `expire`, which leaves the value
alone. This avoids a lock on purpose: `create_server_picker` can be reached
from the balancer phase on a retry, where a sleeping lock is not allowed.
#### Scope
This first step ramps HTTP `roundrobin` Upstreams whose nodes share one
priority. Anything else is rejected at the Admin API instead of being accepted
and silently ignored:
- another balancer type;
- nodes with different priorities - the highest tier is drained first, so a
ramp inside one tier cannot hold traffic back;
- `interval` greater than `slow_start_time_seconds`;
- a stream route that reaches such an Upstream, directly, through an
embedded one, or through a service - plus the reverse check when `warm_up_conf`
is added to an Upstream a stream route already reaches;
- the Upstreams of the `traffic-split` plugin, which are rebuilt per request
and have no stable scope.
These checks run on the configuration entry points only. A configuration
written to etcd directly, or embedded in a route or a service, never runs them,
and must not take a whole Upstream out of service over a field that only shapes
a ramp. At runtime the module therefore fails open: in the stream subsystem,
with mixed priorities, or without the shared dict, nodes keep their configured
weights and one error is logged per Upstream.
#### Worth knowing
- A picker rebuild restarts the round-robin cursor, which favours the
heaviest node for its first picks. It averages out over a bucket with real
traffic; on an Upstream seeing only a handful of requests per `interval`, a
ramping node can get even less than its weight asks for.
- A ramp only moves traffic between nodes. A single-node Upstream, or one
whose nodes are all new, still sends every request to them.
- Every APISIX instance ramps from the moment it observed the node;
instances are not synchronised.
- `upstream-slow-start` defaults to 10m. Its size is bounded by the nodes of
Upstreams that enable slow start, and eviction fails open: a node whose state
is lost keeps its configured weight.
#### Which issue(s) this PR fixes:
Related to #7992 and #10832, and supersedes #12991.
### Checklist
- [x] I have explained the need for this PR and the problem it solves
- [x] I have explained the changes or the new features added to this PR
- [x] I have added tests corresponding to this change
- [x] I have updated the documentation to reflect this change
- [x] I have verified that this change is backward compatible (If not,
please discuss on the [APISIX mailing
list](https://github.com/apache/apisix/tree/master#community) first)
#### Tests
- `t/node/upstream-slow-start.t`: the weight curve and its clamping; a node
added to a running Upstream ramping while the existing one stays at full
weight; the ramp ending at the configured weights; an unrelated change not
restarting it; the tombstone resuming a ramp; a route-embedded Upstream as its
own scope; the picker key moving once per `interval` and then settling; a node
an active health check sidelines keeping its lifecycle; a sidelined node
starting its window on first entry; and two workers racing on the same state,
replayed deterministically by injecting another worker's write between a read
and the write that follows it.
- `t/admin/upstream-slow-start.t`: field ranges, defaults, every rejection
above in both directions, an unreadable etcd reference failing validation
rather than the request, declarative validation through
`/apisix/admin/configs/validate`, the data plane keeping an Upstream whose
`warm_up_conf` it cannot honour, and enabling `warm_up_conf` in a cluster with
no stream route at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
