strelok89 opened a new pull request, #3855:
URL: https://github.com/apache/celeborn/pull/3855
### What changes were proposed in this pull request?
Three values for `charts/celeborn`, all disabled by default.
**`worker.zoneAwareReplication`** renders one worker StatefulSet per zone
instead of a single
one for the whole cluster. Each carries a `celeborn.apache.org/zone` label
as part of its
selector, so a zone's worker count is declared rather than inferred from
scheduling.
`worker.replicas` becomes the total across zones, split `ceil(replicas /
zones)`, with a
per-zone override; `zones[].nodeSelector` merges over `worker.nodeSelector`
and
`zones[].affinity` replaces `worker.affinity`. All zones share the existing
headless service,
so the service and pod monitor keep selecting every zone.
**`worker.zoneAwareReplication.metricsLabel`** passes the zone into
`celeborn.metrics.extraLabels` so the worker stamps a `zone` label on
everything it emits. It
is injected as a system property, since master and worker share one config
map:
`Utils.loadDefaultCelebornProperties` applies the file with `setIfMissing`,
so a system
property takes precedence, and `start-worker.sh` builds `CELEBORN_JAVA_OPTS`
from
`CELEBORN_WORKER_JAVA_OPTS`, so it composes with whatever `worker.env` sets.
**`worker.autoscaling`** creates a KEDA `ScaledObject` per worker
StatefulSet — one per zone
with the above enabled. Two triggers ship, on the two resources whose
exhaustion takes a
worker out of service, both scoped by the `zone` label and both excluding a
decommissioning
worker:
- `diskUsage`, on `DeviceCelebornFreeBytes / DeviceCelebornTotalBytes`,
default 0.70.
- `memoryUsage`, on `DirectMemoryUsageRatio`, default 0.70 — below
`celeborn.worker.directMemoryRatioToPauseReceive` (0.85) so capacity
arrives before pushes
pause.
Both use `metricType: Value`, since a ratio must not be divided across
replicas.
`worker.autoscaling.triggers` appends user triggers, rendered through `tpl`
against the zone's
context.
`worker.autoscaling.drain` adds a `preStop` hook calling `POST
/api/v1/workers/exit` with
`DECOMMISSION`, because Kubernetes scale-in only sends `SIGTERM`, which
never enters the
decommission path — the worker would leave with its shuffle data. The hook
must not fire on a
rolling update or node drain, where waiting for shuffle data to expire would
block every pod
replacement for hours, so it distinguishes them by ordinal: the StatefulSet
controller lowers
`spec.replicas` before deleting pods on a scale-in, so an ordinal at or
above the desired count
is being removed for good, while anything else shuts down gracefully and
returns. A desired
count that cannot be read falls back to a graceful shutdown, so a failed
lookup cannot stall a
rollout. The hook reads the count with the pod's own service account, so the
role gains `get`
on `statefulsets/scale` when the drain is enabled.
### Why are the changes needed?
Workers all live in one StatefulSet at a fixed replica count, which leaves
two gaps.
Their spread across availability zones is uneven. The scheduler decides
where a worker lands,
so pod-level spreading (`worker.affinity`, or
`worker.topologySpreadConstraints` from
CELEBORN-2446) balances it only on a best-effort basis: the balance is
re-derived on every
scheduling decision, holds while the constraint is satisfiable, and degrades
quietly when it is
not — a zone out of capacity for the workers' instance type, or several
nodes replaced at once.
And a zone cannot be scaled at all. Load per zone is genuinely uneven when
applications select
workers by tag (`celeborn.tags.tagsExpr`), so the fleet has to be sized for
the busiest zone
and left that size. A single StatefulSet also cannot be rolled or paused one
zone at a time.
Note that this controls where worker pods run and how many there are. It is
not zone-aware data
placement: which workers a client may use is decided in the master by worker
tags, not by pod
topology.
### Does this PR resolve a correctness bug?
<!-- Check if yes. The `correctness` label will be added/removed
automatically. -->
- [ ] Yes
### Does this PR introduce _any_ user-facing change?
<!-- Check if yes. -->
- [x] Yes
New chart values: `worker.zoneAwareReplication` and `worker.autoscaling`.
Both default to
disabled and the rendered output with default values is unchanged, so
existing releases are
unaffected until they opt in. `charts/celeborn/README.md` documents both,
including that
enabling zone-aware replication renames the worker StatefulSet and so
recreates every worker —
a maintenance-window change rather than a rolling update.
### How was this patch tested?
- `helm unittest charts/celeborn --file "tests/**/*_test.yaml" --strict` —
112 tests pass, 27
of them new, covering per-zone StatefulSets and selectors, the replica
split and per-zone
override, the node selector merge, the zone metric label and its
composition with existing
extra labels, the per-zone ScaledObjects and their default triggers, the
`preStop` hook and
its RBAC rule, and each `fail` guard.
- `helm lint`, with default values and with `charts/celeborn/ci/values.yaml`.
- `helm template` with default values, diffed against `main`: byte-identical.
- `helm template` with three zones and autoscaling enabled: three
StatefulSets with distinct
names and selectors, integer replica counts, one shared `serviceName`,
unchanged service and
pod monitor selectors, and one ScaledObject per zone targeting its own
StatefulSet.
- The trigger queries were run against a live Celeborn deployment's
Prometheus to confirm the
metric names exist and each query returns the single element KEDA requires.
- The `preStop` decision logic was exercised as a shell script over
scale-in, rolling update,
unreadable replica count, missing pod name, and an unreachable exit
endpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
