This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/iggy.git
The following commit(s) were added to refs/heads/master by this push:
new e6ec86950 feat(helm): deploy a cluster with one release per node
(#4035)
e6ec86950 is described below
commit e6ec869500c3b362bf7f4b85650b7fc6df9a5fa1
Author: Grzegorz Koszyk <[email protected]>
AuthorDate: Wed Sep 2 16:17:27 2026 +0200
feat(helm): deploy a cluster with one release per node (#4035)
---
helm/charts/iggy/Chart.yaml | 4 +-
helm/charts/iggy/README.md | 211 ++++++++++++++++++++--
helm/charts/iggy/README.md.gotmpl | 170 +++++++++++++++++-
helm/charts/iggy/examples/cluster-3-node.yaml | 130 ++++++++++++++
helm/charts/iggy/templates/_helpers.tpl | 234 +++++++++++++++++++++++++
helm/charts/iggy/templates/deployment.yaml | 75 +++++++-
helm/charts/iggy/templates/hpa.yaml | 61 -------
helm/charts/iggy/templates/server-secrets.yaml | 44 +++++
helm/charts/iggy/templates/service.yaml | 6 +-
helm/charts/iggy/values.yaml | 132 ++++++++++++--
scripts/ci/test-helm.sh | 72 ++++++--
11 files changed, 1019 insertions(+), 120 deletions(-)
diff --git a/helm/charts/iggy/Chart.yaml b/helm/charts/iggy/Chart.yaml
index 08f86a803..a52150924 100644
--- a/helm/charts/iggy/Chart.yaml
+++ b/helm/charts/iggy/Chart.yaml
@@ -20,8 +20,8 @@ apiVersion: v2
name: iggy
description: A Helm chart for Apache Iggy server and web-ui
type: application
-version: 0.5.0
-appVersion: "0.7.0"
+version: 0.6.0
+appVersion: "0.9.0-edge.6"
sources:
- https://github.com/apache/iggy
keywords:
diff --git a/helm/charts/iggy/README.md b/helm/charts/iggy/README.md
index bfd38cb7a..8026d99ab 100644
--- a/helm/charts/iggy/README.md
+++ b/helm/charts/iggy/README.md
@@ -2,7 +2,7 @@
A Helm chart for Apache Iggy server and web-ui
-
+
## Prerequisites
@@ -83,6 +83,156 @@ If Prometheus Operator is installed and you want
monitoring, set
`server.serviceMonitor.enabled=true` in `custom-values.yaml` or pass it on the
command line with `--set server.serviceMonitor.enabled=true`.
+## Cluster Mode
+
+The server runs a Viewstamped Replication cluster when `cluster.enabled` is set
+and each node is told which roster entry it is. The chart models this as **one
+release per node**: every release is handed the same roster and overrides only
+its own identity.
+
+`helm/charts/iggy/examples/cluster-3-node.yaml` is a ready three-node roster.
+Point the `ip` values at the nodes you will pin the releases to, then:
+
+```bash
+for i in 0 1 2; do
+ helm upgrade --install "iggy-n$i" ./helm/charts/iggy \
+ -f ./helm/charts/iggy/examples/cluster-3-node.yaml \
+ --set "server.cluster.selfReplicaId=$i" \
+ --set "server.nodeSelector.kubernetes\.io/hostname=node-$i"
+done
+```
+
+The chart turns `server.cluster` into the server's `IGGY_CLUSTER_*` environment
+variables and passes `--replica-id`, so no configuration file is mounted. The
+values mirror the server's `[[cluster.nodes]]` config one for one.
+
+### Why hostNetwork and node pinning are required
+
+The server's consensus listener binds `cluster.nodes[*].ip` **verbatim**,
unlike
+the client listeners, which keep their own bind address and take only the port
+from the roster. A pod therefore has to own the address its roster entry names.
+Pod IPs are neither stable nor known before install, and a Service ClusterIP is
+not an address a pod can bind, so the workable layout today is `hostNetwork:
+true` with each release pinned to a node and its roster `ip` set to that node's
+IP. Node IPs are known up front and survive a pod restart, which is what lets a
+replica come back and rejoin.
+
+The cost is real: the server pod shares the node's network namespace and its
+ports. Weigh it before running this in a shared cluster.
+
+If a release lands on a node whose IP is not the one in its roster entry, the
+server refuses to start rather than misbehaving quietly:
+
+```text
+Error: ShardJoinFailures { failures: [ShardJoinFailure { shard_id: 0,
+ kind: Error(Iggy(CannotBindToSocket("10.0.1.11:9090"))) }] }
+```
+
+Check the `nodeSelector` on that release against the `ip` in its roster entry.
+
+### Upgrades
+
+Host ports make a rolling update impossible: the replacement pod cannot bind
+ports the outgoing pod still holds, so it stays `Pending` while the Deployment
+waits for it to become ready. The chart therefore switches the server to the
+`Recreate` strategy whenever `hostNetwork` is set, which takes the node down
+for the length of the restart. Roll **one release at a time** and let the
+cluster regain quorum before starting the next.
+
+Turning `hostNetwork` on for a release that already exists moves the Deployment
+from `RollingUpdate` to `Recreate`. Under Helm 4's server-side apply that is
+rejected, because the patch cannot drop the `rollingUpdate` block the API
server
+defaulted in:
+
+```text
+Deployment.apps "iggy-n0" is invalid: spec.strategy.rollingUpdate: Forbidden:
+ may not be specified when strategy `type` is 'Recreate'
+```
+
+On Helm 4, run that one upgrade with `helm upgrade --server-side=false`, which
+replaces the strategy in place. Helm 3 has no such flag, since it does not use
+server-side apply. Set `server.strategy` explicitly if you want a different
+strategy.
+
+### Roster rules
+
+* Every node runs the **identical** `server.cluster.nodes` list. Only
+ `selfReplicaId` differs between releases.
+* `replicaId` values are unique and cover `0..N-1` for an `N`-node roster.
+* `ports.tcpReplica` is required on every entry. In cluster mode the server
+ takes every listener port from the roster and will not fall back to defaults,
+ because two nodes on one host would otherwise race for the same socket. The
+ remaining ports default to `server.ports`.
+* `server.cluster.name` is hashed into the on-disk cluster id on first boot.
+ Changing it later makes the server refuse to start against existing data.
+* `ip` must be a literal IP. Hostnames are rejected. Use
+ `advertisedAddress` for the name clients dial, which does accept DNS.
+
+### Secrets
+
+Four settings have to carry the byte-identical value on every node, so they
+belong in one Secret that all releases reference rather than in each release's
+values. Create it in the release namespace before the first install:
+
+```bash
+JWT="$(head -c 32 /dev/urandom | base64)"
+kubectl create secret generic iggy-cluster-secrets \
+ --from-literal=username=iggy \
+ --from-literal=password="$(head -c 24 /dev/urandom | base64)" \
+ --from-literal=clusterSharedSecret="$(head -c 32 /dev/urandom | base64)" \
+ --from-literal=jwtEncodingSecret="$JWT" \
+ --from-literal=jwtDecodingSecret="$JWT" \
+ --from-literal=encryptionKey="$(head -c 32 /dev/urandom | base64)"
+```
+
+`examples/cluster-3-node.yaml` points `server.users.root`, `server.encryption`,
+`server.jwt` and `server.cluster.auth` at that one Secret. All four also accept
+an inline value, which the chart turns into a Secret of its own:
+`<release>-root-credentials` for the root username and password,
+`<release>-secrets` for the other three. That is convenient for a single node
+and a poor fit for a cluster, since the value then lives in every release's
+stored values.
+
+* **`server.users.root`** seeds the root user. Every node creates it locally
+ from its own `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD`, so a password
that
+ differs on one release logs in on that node only, and a first cluster boot
+ refuses to start without both.
+* **`server.cluster.auth`** makes every replica connection complete an
+ authenticated handshake or be rejected. The key is at least 32 bytes of
+ CSPRNG output. Turning it on or off is a coordinated restart of the whole
+ cluster: a node that authenticates cannot talk to one that does not.
+* **`server.encryption`** encrypts message payloads and state commands at rest
+ with AES-256-GCM, under a 32-byte base64 key. A node holding a different key
+ cannot read what its peers wrote.
+* **`server.jwt`** makes HTTP bearer tokens valid on every node and survive a
+ restart. With `cluster.auth` enabled the server already derives a
cluster-wide
+ key from the replica PSK, so setting the JWT secrets is an alternative to
that
+ rather than an addition; setting them anyway keeps tokens working if auth is
+ later turned off.
+
+None of the encryption, JWT and replica-auth values is written to the data
+directory, and each is masked as `******` in the startup log.
+
+### Storage
+
+A replica cannot move between nodes without invalidating its roster entry, so
+give each release storage that stays on its node, and size
+`server.persistence` per node rather than for the cluster. A replica that
starts
+on an empty volume rejoins by state transfer, which is correct but re-reads the
+whole dataset from its peers. `server.persistence.enabled: false` puts the
+replica on `emptyDir` and forces exactly that on every pod replacement, so the
+cluster example enables persistence and leaves `storageClass` for you to point
+at a node-local provisioner.
+
+### What the chart refuses
+
+`server.replicaCount > 1` fails at render time. Scaling the server Deployment
+produces N independent servers behind one Service, all writing the same PVC
+subpath with no lock between them, which corrupts the data directory while
+`helm --wait` still reports success. The chart ships no HorizontalPodAutoscaler
+for the same reason. Cluster size is a roster decision, so add a node to
+`server.cluster.nodes` and install another release instead.
+
## Uninstallation
```bash
@@ -148,7 +298,7 @@ If a previous local smoke install failed and left resources
behind, reset the sm
scripts/ci/test-helm.sh cleanup-smoke
```
-On Apple Silicon hosts, the released `apache/iggy:0.7.0` `arm64` image may
still fail during the runtime smoke path in kind. If your Docker setup supports
amd64 emulation well enough, you can try recreating the dedicated smoke cluster
with:
+On Apple Silicon hosts, the released `arm64` server image may still fail
during the runtime smoke path in kind. If your Docker setup supports amd64
emulation well enough, you can try recreating the dedicated smoke cluster with:
```bash
HELM_SMOKE_KIND_PLATFORM=linux/amd64 scripts/ci/setup-helm-smoke-cluster.sh
@@ -219,12 +369,18 @@ Ensure the server binds to `0.0.0.0` instead of
`127.0.0.1`. This is configured
A wildcard bind says which interfaces accept connections, not where clients
reach the pod, so the server also needs the address to publish in cluster
-metadata. Server builds that carry the setting refuse to start without it;
-older ones, including the `0.7.0` this chart pins by default, have no such
-refusal and log the variable as unknown and ignored. Either way the chart
-sets `IGGY_NODE_ADVERTISED_ADDRESS` to the in-cluster Service DNS name;
-override it with `server.advertisedAddress` when clients arrive through a
-LoadBalancer or an Ingress.
+metadata. Server builds that carry the setting refuse to start without it.
+`0.9.0-edge.6`, the image this chart pins, predates it:
+that build logs `IGGY_NODE_ADVERTISED_ADDRESS` as an unknown variable and
+publishes the bind address, so the chart's default stays inert until the pinned
+image moves past it. Either way the chart sets `IGGY_NODE_ADVERTISED_ADDRESS`
to
+the in-cluster Service DNS name; override it with `server.advertisedAddress`
+when clients arrive through a LoadBalancer or an Ingress.
+
+In cluster mode the server ignores the variable altogether and publishes each
+node's roster `ip`, or its `advertisedAddress` when the entry carries one, so
+the chart leaves the variable out there and refuses a render that sets
+`server.advertisedAddress` alongside `server.cluster.enabled`.
Declaring `IGGY_NODE_ADVERTISED_ADDRESS` in `server.env` yourself works too:
the chart then leaves its own default out, so the variable is declared once.
@@ -325,10 +481,6 @@ pre-commit install
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| additionalLabels | object | `{}` | Additional labels for all resources |
-| autoscaling.enabled | bool | `false` | Enable horizontal pod autoscaling |
-| autoscaling.maxReplicas | int | `100` | Maximum replicas for autoscaling |
-| autoscaling.minReplicas | int | `1` | Minimum replicas for autoscaling |
-| autoscaling.targetCPUUtilizationPercentage | int | `80` | Target CPU
utilization for autoscaling |
| fullnameOverride | string | `""` | Override full release name |
| imagePullSecrets | list | `[]` | Image pull secrets for private registries |
| nameOverride | string | `""` | Override chart name |
@@ -336,19 +488,44 @@ pre-commit install
| podSecurityContext | object | `{"seccompProfile":{"type":"Unconfined"}}` |
Pod security context (server uses io_uring, requires unconfined seccomp) |
| resources | object | `{}` | Resource limits and requests for server |
| securityContext | object | `{"capabilities":{"add":["IPC_LOCK"]}}` |
Container security context (server requires IPC_LOCK for io_uring) |
-| server | object |
`{"advertisedAddress":"","affinity":{},"enabled":true,"env":[{"name":"RUST_LOG","value":"info"},{"name":"IGGY_HTTP_ADDRESS","value":"0.0.0.0:3000"},{"name":"IGGY_TCP_ADDRESS","value":"0.0.0.0:8090"},{"name":"IGGY_QUIC_ADDRESS","value":"0.0.0.0:8080"},{"name":"IGGY_WEBSOCKET_ADDRESS","value":"0.0.0.0:8092"}],"image":{"pullPolicy":"Always","repository":"apache/iggy","tag":"0.7.0"},"ingress":{"annotations":{},"className":"","enabled":false,"hosts":[{"host":"chart-example
[...]
-| server.advertisedAddress | string | `""` | Client-facing address published
in cluster metadata. Declaring `IGGY_NODE_ADVERTISED_ADDRESS` in `server.env`
instead also works, but setting both is refused at render time. Empty falls
back to the in-cluster Service DNS name. |
+| server | object |
`{"advertisedAddress":"","affinity":{},"cluster":{"auth":{"enabled":false,"existingSecret":{"name":"","previousSharedSecretKey":"clusterPreviousSharedSecret","sharedSecretKey":"clusterSharedSecret"},"previousSharedSecret":"","sharedSecret":""},"enabled":false,"name":"iggy-cluster","nodes":[],"requireHostNetwork":true,"selfReplicaId":0},"enabled":true,"encryption":{"enabled":false,"existingSecret":{"key":"encryptionKey","name":""},"key":""},"env":[{"name":"RUST_LOG","v
[...]
+| server.advertisedAddress | string | `""` | Client-facing address published
in cluster metadata. Declaring `IGGY_NODE_ADVERTISED_ADDRESS` in `server.env`
instead also works, but setting both is refused at render time. Empty falls
back to the in-cluster Service DNS name. Ignored in cluster mode, where the
address comes from the node's roster entry, so setting both is refused there
too. |
| server.affinity | object | `{}` | Affinity rules for server pods |
+| server.cluster.auth | object |
`{"enabled":false,"existingSecret":{"name":"","previousSharedSecretKey":"clusterPreviousSharedSecret","sharedSecretKey":"clusterSharedSecret"},"previousSharedSecret":"","sharedSecret":""}`
| Replica-to-replica authentication on the consensus port. When enabled every
peer must complete an authenticated handshake or be rejected, and a shared
secret becomes mandatory. Enabling it on a running cluster is a
coordinated-restart change, not a rolling one. |
+| server.cluster.auth.enabled | bool | `false` | Require the authenticated
replica handshake |
+| server.cluster.auth.existingSecret.name | string | `""` | Name of an
existing Secret holding the pre-shared keys |
+| server.cluster.auth.existingSecret.previousSharedSecretKey | string |
`"clusterPreviousSharedSecret"` | Key inside that Secret holding the retiring
shared secret |
+| server.cluster.auth.existingSecret.sharedSecretKey | string |
`"clusterSharedSecret"` | Key inside that Secret holding the active shared
secret |
+| server.cluster.auth.previousSharedSecret | string | `""` | Retiring key,
accepted for verification only while a rotation is in flight. Leave empty
outside a rotation. |
+| server.cluster.auth.sharedSecret | string | `""` | Cluster-wide pre-shared
key, at least 32 bytes of CSPRNG output, byte-identical on every node. Ignored
when `existingSecret.name` is set. |
+| server.cluster.enabled | bool | `false` | Enable cluster (VSR consensus)
mode. One Helm release per node: every release shares the same `nodes` roster
and overrides only `selfReplicaId`. See the Cluster Mode section of the chart
README. |
+| server.cluster.name | string | `"iggy-cluster"` | Cluster name,
byte-identical on every node. Hashed into the on-disk cluster id on first boot,
so changing it later means starting from an empty data directory. |
+| server.cluster.nodes | list | `[]` | Cluster roster, mirroring the server's
`[[cluster.nodes]]` config. Every node runs the identical list. `ip` is the
replica-plane address: this node's consensus listener binds it verbatim and
every peer dials it verbatim, so it must be a literal IP that the pod itself
owns. With `server.hostNetwork` that is the node IP. `ports.tcpReplica` is
required on every entry; the remaining ports default to `server.ports`. |
+| server.cluster.requireHostNetwork | bool | `true` | Refuse to render a
cluster node without `server.hostNetwork`. The replica listener binds the
roster `ip` verbatim, which no pod owns on the cluster network, so the pod
would die at boot with `CannotBindToSocket`. Set this to false only when the
roster `ip` is an address the pod itself holds. |
+| server.cluster.selfReplicaId | int | `0` | Which `nodes` entry this release
runs, matched against `replicaId`. |
| server.enabled | bool | `true` | Enable the Iggy server deployment |
+| server.encryption | object |
`{"enabled":false,"existingSecret":{"key":"encryptionKey","name":""},"key":""}`
| Server-side encryption of message payloads and state commands, using
AES-256-GCM. Every node of a cluster must hold the identical key, or it cannot
read data another node wrote. |
+| server.encryption.enabled | bool | `false` | Enable encryption at rest |
+| server.encryption.existingSecret.key | string | `"encryptionKey"` | Key
inside that Secret |
+| server.encryption.existingSecret.name | string | `""` | Name of an existing
Secret holding the encryption key |
+| server.encryption.key | string | `""` | 32-byte key, base64 encoded. Ignored
when `existingSecret.name` is set. Prefer `existingSecret` outside development:
a value here is stored in the Helm release and readable by anyone who can read
it. |
| server.env | list |
`[{"name":"RUST_LOG","value":"info"},{"name":"IGGY_HTTP_ADDRESS","value":"0.0.0.0:3000"},{"name":"IGGY_TCP_ADDRESS","value":"0.0.0.0:8090"},{"name":"IGGY_QUIC_ADDRESS","value":"0.0.0.0:8080"},{"name":"IGGY_WEBSOCKET_ADDRESS","value":"0.0.0.0:8092"}]`
| Environment variables for the server container |
+| server.extraArgs | list | `[]` | Extra command-line arguments appended to
the server entrypoint, e.g. `["--with-default-root-credentials"]` for a
throwaway development install. `--replica-id` is not one of them: the chart
passes it already whenever `cluster.enabled` is set. |
+| server.hostNetwork | bool | `false` | Run the server pod in the host network
namespace. Required for cluster mode, where the replica listener binds the
roster IP verbatim. |
| server.image.pullPolicy | string | `"Always"` | Image pull policy |
| server.image.repository | string | `"apache/iggy"` | Server image repository
|
-| server.image.tag | string | `"0.7.0"` | Server image tag (overrides chart
appVersion) |
+| server.image.tag | string | `""` | Server image tag. Empty uses the chart
appVersion. |
| server.ingress.annotations | object | `{}` | Ingress annotations
(controller-specific) |
| server.ingress.className | string | `""` | Ingress class name
(controller-neutral) |
| server.ingress.enabled | bool | `false` | Enable ingress for the server |
| server.ingress.hosts | list |
`[{"host":"chart-example.local","paths":[{"path":"/","pathType":"ImplementationSpecific"}]}]`
| Ingress hosts configuration |
| server.ingress.tls | list | `[]` | Ingress TLS configuration |
+| server.jwt | object |
`{"decodingSecret":"","encodingSecret":"","existingSecret":{"decodingSecretKey":"jwtDecodingSecret","encodingSecretKey":"jwtEncodingSecret","name":""}}`
| Secrets used to sign and validate HTTP bearer tokens. Left unset, each node
generates a random secret on every start, which invalidates tokens across
restarts and keeps them node-local. Setting the identical secret on every node
makes bearers valid cluster-wide and activates follower to primary HTTP
forwarding; [...]
+| server.jwt.decodingSecret | string | `""` | Decoding secret. Ignored when
`existingSecret.name` is set. |
+| server.jwt.encodingSecret | string | `""` | Encoding secret. Ignored when
`existingSecret.name` is set. |
+| server.jwt.existingSecret.decodingSecretKey | string | `"jwtDecodingSecret"`
| Key inside that Secret holding the decoding secret |
+| server.jwt.existingSecret.encodingSecretKey | string | `"jwtEncodingSecret"`
| Key inside that Secret holding the encoding secret |
+| server.jwt.existingSecret.name | string | `""` | Name of an existing Secret
holding the JWT secrets |
| server.nodeSelector | object | `{}` | Node selector for server pods |
| server.persistence.accessMode | string | `"ReadWriteOnce"` | PVC access mode
|
| server.persistence.annotations | object | `{}` | PVC annotations |
@@ -357,8 +534,9 @@ pre-commit install
| server.persistence.size | string | `"8Gi"` | PVC storage size |
| server.persistence.storageClass | string | `""` | Storage class for PVC
(empty uses default provisioner) |
| server.ports.http | int | `3000` | HTTP API port |
-| server.ports.quic | int | `8080` | QUIC protocol port |
+| server.ports.quic | int | `8080` | QUIC protocol port (UDP) |
| server.ports.tcp | int | `8090` | TCP protocol port |
+| server.ports.websocket | int | `8092` | WebSocket protocol port |
| server.replicaCount | int | `1` | Number of server replicas |
| server.service.port | int | `3000` | Service port for the server |
| server.service.type | string | `"ClusterIP"` | Service type for the server |
@@ -370,6 +548,7 @@ pre-commit install
| server.serviceMonitor.namespace | string | `""` | Namespace to deploy the
ServiceMonitor |
| server.serviceMonitor.path | string | `"/metrics"` | Path to scrape metrics
from |
| server.serviceMonitor.scrapeTimeout | string | `"10s"` | Timeout for scrape
metrics request |
+| server.strategy | object | `{}` | Deployment update strategy. Empty lets the
chart choose: `Recreate` when `hostNetwork` is set, because a rolling update
would wait forever for a replacement pod that cannot bind host ports the
outgoing pod still holds, and the Kubernetes default otherwise. |
| server.tolerations | list | `[]` | Tolerations for server pods |
| server.users.root.createSecret | bool | `true` | Create a secret for the
root user credentials |
| server.users.root.existingSecret.name | string | `""` | Name of existing
secret for root credentials |
diff --git a/helm/charts/iggy/README.md.gotmpl
b/helm/charts/iggy/README.md.gotmpl
index a36d30adc..ec1fb50d1 100644
--- a/helm/charts/iggy/README.md.gotmpl
+++ b/helm/charts/iggy/README.md.gotmpl
@@ -101,6 +101,156 @@ If Prometheus Operator is installed and you want
monitoring, set
`server.serviceMonitor.enabled=true` in `custom-values.yaml` or pass it on the
command line with `--set server.serviceMonitor.enabled=true`.
+## Cluster Mode
+
+The server runs a Viewstamped Replication cluster when `cluster.enabled` is set
+and each node is told which roster entry it is. The chart models this as **one
+release per node**: every release is handed the same roster and overrides only
+its own identity.
+
+`helm/charts/iggy/examples/cluster-3-node.yaml` is a ready three-node roster.
+Point the `ip` values at the nodes you will pin the releases to, then:
+
+```bash
+for i in 0 1 2; do
+ helm upgrade --install "iggy-n$i" ./helm/charts/iggy \
+ -f ./helm/charts/iggy/examples/cluster-3-node.yaml \
+ --set "server.cluster.selfReplicaId=$i" \
+ --set "server.nodeSelector.kubernetes\.io/hostname=node-$i"
+done
+```
+
+The chart turns `server.cluster` into the server's `IGGY_CLUSTER_*` environment
+variables and passes `--replica-id`, so no configuration file is mounted. The
+values mirror the server's `[[cluster.nodes]]` config one for one.
+
+### Why hostNetwork and node pinning are required
+
+The server's consensus listener binds `cluster.nodes[*].ip` **verbatim**,
unlike
+the client listeners, which keep their own bind address and take only the port
+from the roster. A pod therefore has to own the address its roster entry names.
+Pod IPs are neither stable nor known before install, and a Service ClusterIP is
+not an address a pod can bind, so the workable layout today is `hostNetwork:
+true` with each release pinned to a node and its roster `ip` set to that node's
+IP. Node IPs are known up front and survive a pod restart, which is what lets a
+replica come back and rejoin.
+
+The cost is real: the server pod shares the node's network namespace and its
+ports. Weigh it before running this in a shared cluster.
+
+If a release lands on a node whose IP is not the one in its roster entry, the
+server refuses to start rather than misbehaving quietly:
+
+```text
+Error: ShardJoinFailures { failures: [ShardJoinFailure { shard_id: 0,
+ kind: Error(Iggy(CannotBindToSocket("10.0.1.11:9090"))) }] }
+```
+
+Check the `nodeSelector` on that release against the `ip` in its roster entry.
+
+### Upgrades
+
+Host ports make a rolling update impossible: the replacement pod cannot bind
+ports the outgoing pod still holds, so it stays `Pending` while the Deployment
+waits for it to become ready. The chart therefore switches the server to the
+`Recreate` strategy whenever `hostNetwork` is set, which takes the node down
+for the length of the restart. Roll **one release at a time** and let the
+cluster regain quorum before starting the next.
+
+Turning `hostNetwork` on for a release that already exists moves the Deployment
+from `RollingUpdate` to `Recreate`. Under Helm 4's server-side apply that is
+rejected, because the patch cannot drop the `rollingUpdate` block the API
server
+defaulted in:
+
+```text
+Deployment.apps "iggy-n0" is invalid: spec.strategy.rollingUpdate: Forbidden:
+ may not be specified when strategy `type` is 'Recreate'
+```
+
+On Helm 4, run that one upgrade with `helm upgrade --server-side=false`, which
+replaces the strategy in place. Helm 3 has no such flag, since it does not use
+server-side apply. Set `server.strategy` explicitly if you want a different
+strategy.
+
+### Roster rules
+
+* Every node runs the **identical** `server.cluster.nodes` list. Only
+ `selfReplicaId` differs between releases.
+* `replicaId` values are unique and cover `0..N-1` for an `N`-node roster.
+* `ports.tcpReplica` is required on every entry. In cluster mode the server
+ takes every listener port from the roster and will not fall back to defaults,
+ because two nodes on one host would otherwise race for the same socket. The
+ remaining ports default to `server.ports`.
+* `server.cluster.name` is hashed into the on-disk cluster id on first boot.
+ Changing it later makes the server refuse to start against existing data.
+* `ip` must be a literal IP. Hostnames are rejected. Use
+ `advertisedAddress` for the name clients dial, which does accept DNS.
+
+### Secrets
+
+Four settings have to carry the byte-identical value on every node, so they
+belong in one Secret that all releases reference rather than in each release's
+values. Create it in the release namespace before the first install:
+
+```bash
+JWT="$(head -c 32 /dev/urandom | base64)"
+kubectl create secret generic iggy-cluster-secrets \
+ --from-literal=username=iggy \
+ --from-literal=password="$(head -c 24 /dev/urandom | base64)" \
+ --from-literal=clusterSharedSecret="$(head -c 32 /dev/urandom | base64)" \
+ --from-literal=jwtEncodingSecret="$JWT" \
+ --from-literal=jwtDecodingSecret="$JWT" \
+ --from-literal=encryptionKey="$(head -c 32 /dev/urandom | base64)"
+```
+
+`examples/cluster-3-node.yaml` points `server.users.root`, `server.encryption`,
+`server.jwt` and `server.cluster.auth` at that one Secret. All four also accept
+an inline value, which the chart turns into a Secret of its own:
+`<release>-root-credentials` for the root username and password,
+`<release>-secrets` for the other three. That is convenient for a single node
+and a poor fit for a cluster, since the value then lives in every release's
+stored values.
+
+* **`server.users.root`** seeds the root user. Every node creates it locally
+ from its own `IGGY_ROOT_USERNAME` and `IGGY_ROOT_PASSWORD`, so a password
that
+ differs on one release logs in on that node only, and a first cluster boot
+ refuses to start without both.
+* **`server.cluster.auth`** makes every replica connection complete an
+ authenticated handshake or be rejected. The key is at least 32 bytes of
+ CSPRNG output. Turning it on or off is a coordinated restart of the whole
+ cluster: a node that authenticates cannot talk to one that does not.
+* **`server.encryption`** encrypts message payloads and state commands at rest
+ with AES-256-GCM, under a 32-byte base64 key. A node holding a different key
+ cannot read what its peers wrote.
+* **`server.jwt`** makes HTTP bearer tokens valid on every node and survive a
+ restart. With `cluster.auth` enabled the server already derives a
cluster-wide
+ key from the replica PSK, so setting the JWT secrets is an alternative to
that
+ rather than an addition; setting them anyway keeps tokens working if auth is
+ later turned off.
+
+None of the encryption, JWT and replica-auth values is written to the data
+directory, and each is masked as `******` in the startup log.
+
+### Storage
+
+A replica cannot move between nodes without invalidating its roster entry, so
+give each release storage that stays on its node, and size
+`server.persistence` per node rather than for the cluster. A replica that
starts
+on an empty volume rejoins by state transfer, which is correct but re-reads the
+whole dataset from its peers. `server.persistence.enabled: false` puts the
+replica on `emptyDir` and forces exactly that on every pod replacement, so the
+cluster example enables persistence and leaves `storageClass` for you to point
+at a node-local provisioner.
+
+### What the chart refuses
+
+`server.replicaCount > 1` fails at render time. Scaling the server Deployment
+produces N independent servers behind one Service, all writing the same PVC
+subpath with no lock between them, which corrupts the data directory while
+`helm --wait` still reports success. The chart ships no HorizontalPodAutoscaler
+for the same reason. Cluster size is a roster decision, so add a node to
+`server.cluster.nodes` and install another release instead.
+
## Uninstallation
```bash
@@ -166,7 +316,7 @@ If a previous local smoke install failed and left resources
behind, reset the sm
scripts/ci/test-helm.sh cleanup-smoke
```
-On Apple Silicon hosts, the released `apache/iggy:0.7.0` `arm64` image may
still fail during the runtime smoke path in kind. If your Docker setup supports
amd64 emulation well enough, you can try recreating the dedicated smoke cluster
with:
+On Apple Silicon hosts, the released `arm64` server image may still fail
during the runtime smoke path in kind. If your Docker setup supports amd64
emulation well enough, you can try recreating the dedicated smoke cluster with:
```bash
HELM_SMOKE_KIND_PLATFORM=linux/amd64 scripts/ci/setup-helm-smoke-cluster.sh
@@ -237,12 +387,18 @@ Ensure the server binds to `0.0.0.0` instead of
`127.0.0.1`. This is configured
A wildcard bind says which interfaces accept connections, not where clients
reach the pod, so the server also needs the address to publish in cluster
-metadata. Server builds that carry the setting refuse to start without it;
-older ones, including the `0.7.0` this chart pins by default, have no such
-refusal and log the variable as unknown and ignored. Either way the chart
-sets `IGGY_NODE_ADVERTISED_ADDRESS` to the in-cluster Service DNS name;
-override it with `server.advertisedAddress` when clients arrive through a
-LoadBalancer or an Ingress.
+metadata. Server builds that carry the setting refuse to start without it.
+`{{ template "chart.appVersion" . }}`, the image this chart pins, predates it:
+that build logs `IGGY_NODE_ADVERTISED_ADDRESS` as an unknown variable and
+publishes the bind address, so the chart's default stays inert until the pinned
+image moves past it. Either way the chart sets `IGGY_NODE_ADVERTISED_ADDRESS`
to
+the in-cluster Service DNS name; override it with `server.advertisedAddress`
+when clients arrive through a LoadBalancer or an Ingress.
+
+In cluster mode the server ignores the variable altogether and publishes each
+node's roster `ip`, or its `advertisedAddress` when the entry carries one, so
+the chart leaves the variable out there and refuses a render that sets
+`server.advertisedAddress` alongside `server.cluster.enabled`.
Declaring `IGGY_NODE_ADVERTISED_ADDRESS` in `server.env` yourself works too:
the chart then leaves its own default out, so the variable is declared once.
diff --git a/helm/charts/iggy/examples/cluster-3-node.yaml
b/helm/charts/iggy/examples/cluster-3-node.yaml
new file mode 100644
index 000000000..96a9329ed
--- /dev/null
+++ b/helm/charts/iggy/examples/cluster-3-node.yaml
@@ -0,0 +1,130 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Shared roster for a three-node Iggy cluster. Install one release per node,
+# overriding only the identity:
+#
+# for i in 0 1 2; do
+# helm upgrade --install "iggy-n$i" ./helm/charts/iggy \
+# -f ./helm/charts/iggy/examples/cluster-3-node.yaml \
+# --set "server.cluster.selfReplicaId=$i" \
+# --set "server.nodeSelector.kubernetes\.io/hostname=node-$i"
+# done
+#
+# Replace the `ip` values with the IPs of the nodes you pin each release to.
+# The server binds this address verbatim for replica traffic, so with
+# hostNetwork it must be the node's own IP.
+#
+# Every secret below has to be byte-identical on every node, so they live in
one
+# Secret that all three releases reference rather than in this file. Create it
+# once, in the release namespace, before the first install:
+#
+# JWT="$(head -c 32 /dev/urandom | base64)"
+# kubectl create secret generic iggy-cluster-secrets \
+# --from-literal=username=iggy \
+# --from-literal=password="$(head -c 24 /dev/urandom | base64)" \
+# --from-literal=clusterSharedSecret="$(head -c 32 /dev/urandom | base64)"
\
+# --from-literal=jwtEncodingSecret="$JWT" \
+# --from-literal=jwtDecodingSecret="$JWT" \
+# --from-literal=encryptionKey="$(head -c 32 /dev/urandom | base64)"
+#
+# The JWT encoding and decoding secrets are one HMAC key, which is why both
+# entries carry the same value. Two different values make every token from
+# /users/login fail with 401, and the server only warns about the mismatch.
+
+ui:
+ enabled: false
+
+server:
+ hostNetwork: true
+
+ # The root user is created locally on each node out of that node's own
+ # IGGY_ROOT_* environment, so the credentials have to match across the
+ # releases the same way the keys below do.
+ users:
+ root:
+ createSecret: false
+ existingSecret:
+ name: iggy-cluster-secrets
+
+ # A release is pinned to one node, so its storage has to stay on that node.
+ # On emptyDir every pod replacement wipes the replica, which then refetches
+ # the whole dataset from its peers.
+ persistence:
+ enabled: true
+ # Name a node-local class here (local-path, topolvm, a local volume
+ # provisioner). Empty takes the default provisioner, which may hand out
+ # storage that does not stay on the pinned node.
+ storageClass: ""
+ size: 8Gi
+
+ # Encrypts message payloads and state commands at rest with AES-256-GCM. The
+ # key is cluster-wide: a node that holds a different one cannot read what its
+ # peers wrote.
+ encryption:
+ enabled: true
+ existingSecret:
+ name: iggy-cluster-secrets
+
+ # Makes bearer tokens valid on every node and survive a restart. Redundant
+ # while cluster.auth is enabled, which derives the same key from the replica
+ # PSK, but set explicitly here so tokens keep working if auth is turned off.
+ jwt:
+ existingSecret:
+ name: iggy-cluster-secrets
+
+ cluster:
+ enabled: true
+ name: iggy-cluster
+
+ # Every replica connection completes an authenticated handshake or is
+ # rejected. Turning this on or off is a coordinated restart of the whole
+ # cluster, not a rolling one.
+ auth:
+ enabled: true
+ existingSecret:
+ name: iggy-cluster-secrets
+ nodes:
+ - name: iggy-node-0
+ ip: 10.0.1.10
+ replicaId: 0
+ ports:
+ tcpReplica: 9090
+ - name: iggy-node-1
+ ip: 10.0.1.11
+ replicaId: 1
+ ports:
+ tcpReplica: 9090
+ - name: iggy-node-2
+ ip: 10.0.1.12
+ replicaId: 2
+ ports:
+ tcpReplica: 9090
+
+ env:
+ - name: RUST_LOG
+ value: info
+ # Bind interface only: in cluster mode every listener port comes from the
+ # roster entry for this node.
+ - name: IGGY_HTTP_ADDRESS
+ value: "0.0.0.0:3000"
+ - name: IGGY_TCP_ADDRESS
+ value: "0.0.0.0:8090"
+ - name: IGGY_QUIC_ADDRESS
+ value: "0.0.0.0:8080"
+ - name: IGGY_WEBSOCKET_ADDRESS
+ value: "0.0.0.0:8092"
diff --git a/helm/charts/iggy/templates/_helpers.tpl
b/helm/charts/iggy/templates/_helpers.tpl
index 92f69780e..f6943c8c4 100644
--- a/helm/charts/iggy/templates/_helpers.tpl
+++ b/helm/charts/iggy/templates/_helpers.tpl
@@ -97,3 +97,237 @@ Create the name of the service account to use
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}
+
+{{/*
+Validate the cluster roster and fail the render with an actionable message.
+Every check here is one the server would otherwise only reject at boot, after
+the pod is already scheduled.
+*/}}
+{{- define "iggy.validateCluster" -}}
+ {{- $cluster := .Values.server.cluster }}
+ {{- if not $cluster.nodes }}
+ {{- fail "server.cluster.enabled is true but server.cluster.nodes is
empty. Every node runs the identical roster; see the Cluster Mode section of
the chart README." }}
+ {{- end }}
+ {{- if and $cluster.requireHostNetwork (not .Values.server.hostNetwork) }}
+ {{- fail "server.cluster.enabled is true but server.hostNetwork is false.
The replica listener binds the roster ip verbatim, which is not an address a
pod owns on the cluster network, so the pod dies at boot with
CannotBindToSocket. Set server.hostNetwork to true, or
server.cluster.requireHostNetwork to false if the roster ip really is one this
pod owns." }}
+ {{- end }}
+ {{- $root := .Values.server.users.root }}
+ {{- if and (not $root.createSecret) (not $root.existingSecret.name) }}
+ {{- fail "server.users.root has neither createSecret nor
existingSecret.name, so the container receives no IGGY_ROOT_USERNAME or
IGGY_ROOT_PASSWORD. A first cluster boot refuses to start without both, because
every node creates root locally and the credentials have to come out identical
on all of them." }}
+ {{- end }}
+ {{- $count := len $cluster.nodes }}
+ {{- $seen := dict }}
+ {{- range $cluster.nodes }}
+ {{- if not .name }}
+ {{- fail "every server.cluster.nodes entry needs a name" }}
+ {{- end }}
+ {{- if not .ip }}
+ {{- fail (printf "server.cluster.nodes entry %q has no ip. The replica
listener binds this address verbatim, so it must be a literal IP the pod owns."
.name) }}
+ {{- end }}
+ {{- $ip := toString .ip }}
+ {{- if not (regexMatch "^[0-9a-fA-F.:]+$" $ip) }}
+ {{- fail (printf "server.cluster.nodes entry %q has ip %q, which the
server parses as an IP address and rejects. Use
server.cluster.nodes[*].advertisedAddress for the hostname clients dial." .name
$ip) }}
+ {{- end }}
+ {{- if or (eq $ip "0.0.0.0") (eq $ip "::") }}
+ {{- fail (printf "server.cluster.nodes entry %q has the wildcard ip %q.
Every peer dials this address verbatim, so it has to name one interface." .name
$ip) }}
+ {{- end }}
+ {{- if kindIs "invalid" .replicaId }}
+ {{- fail (printf "server.cluster.nodes entry %q has no replicaId" .name)
}}
+ {{- end }}
+ {{- $id := int .replicaId }}
+ {{- if or (lt $id 0) (ge $id $count) }}
+ {{- fail (printf "server.cluster.nodes entry %q has replicaId %d, which
is outside 0..%d for a %d-node roster" .name $id (sub $count 1) $count) }}
+ {{- end }}
+ {{- if hasKey $seen (printf "%d" $id) }}
+ {{- fail (printf "server.cluster.nodes has two entries with replicaId
%d; ids must be unique" $id) }}
+ {{- end }}
+ {{- $_ := set $seen (printf "%d" $id) .name }}
+ {{- if not (and .ports .ports.tcpReplica) }}
+ {{- fail (printf "server.cluster.nodes entry %q has no ports.tcpReplica.
In cluster mode the server takes every listener port from the roster and
refuses to start without it." .name) }}
+ {{- end }}
+ {{- end }}
+ {{- if not (hasKey $seen (printf "%d" (int $cluster.selfReplicaId))) }}
+ {{- fail (printf "server.cluster.selfReplicaId is %d but no
server.cluster.nodes entry declares that replicaId. Each release picks its own
identity out of the shared roster." (int $cluster.selfReplicaId)) }}
+ {{- end }}
+{{- end }}
+
+{{/*
+The ports this release's server binds, as a JSON object. In cluster mode the
+server takes every listener port from its own roster entry and never falls back
+to the top-level ones, so a node whose entry names other ports has to reach the
+container ports, the probes and the Service targets as well. `server.ports`
+supplies the per-field default there and the whole answer outside cluster mode.
+`tcpReplica` has no top-level default because the roster owns it: it is
+mandatory on every entry and there is no replica listener outside cluster mode.
+*/}}
+{{- define "iggy.serverPorts" -}}
+ {{- $ports := .Values.server.ports }}
+ {{- $resolved := dict "http" $ports.http "quic" $ports.quic "tcp" $ports.tcp
"websocket" $ports.websocket }}
+ {{- if .Values.server.cluster.enabled }}
+ {{- $selfReplicaId := int .Values.server.cluster.selfReplicaId }}
+ {{- range .Values.server.cluster.nodes }}
+ {{- if eq (int .replicaId) $selfReplicaId }}
+ {{- range $name, $port := (default (dict) .ports) }}
+ {{- if $port }}
+ {{- $_ := set $resolved $name $port }}
+ {{- end }}
+ {{- end }}
+ {{- end }}
+ {{- end }}
+ {{- end }}
+ {{- $resolved | toJson }}
+{{- end }}
+
+{{/*
+Render the roster as IGGY_CLUSTER_* environment variables. The server accepts
+the whole cluster config this way, so the chart needs no config file mount.
+*/}}
+{{- define "iggy.clusterEnv" -}}
+ {{- $ports := .Values.server.ports }}
+- name: IGGY_CLUSTER_ENABLED
+ value: "true"
+- name: IGGY_CLUSTER_NAME
+ value: {{ .Values.server.cluster.name | quote }}
+ {{- range .Values.server.cluster.nodes }}
+ {{- $id := int .replicaId }}
+- name: IGGY_CLUSTER_NODES_{{ $id }}_NAME
+ value: {{ .name | quote }}
+- name: IGGY_CLUSTER_NODES_{{ $id }}_IP
+ value: {{ .ip | quote }}
+- name: IGGY_CLUSTER_NODES_{{ $id }}_REPLICA_ID
+ value: {{ $id | quote }}
+ {{- if .advertisedAddress }}
+- name: IGGY_CLUSTER_NODES_{{ $id }}_ADVERTISED_ADDRESS
+ value: {{ .advertisedAddress | quote }}
+ {{- end }}
+- name: IGGY_CLUSTER_NODES_{{ $id }}_PORTS_TCP
+ value: {{ (default $ports.tcp (and .ports .ports.tcp)) | quote }}
+- name: IGGY_CLUSTER_NODES_{{ $id }}_PORTS_QUIC
+ value: {{ (default $ports.quic (and .ports .ports.quic)) | quote }}
+- name: IGGY_CLUSTER_NODES_{{ $id }}_PORTS_HTTP
+ value: {{ (default $ports.http (and .ports .ports.http)) | quote }}
+- name: IGGY_CLUSTER_NODES_{{ $id }}_PORTS_WEBSOCKET
+ value: {{ (default $ports.websocket (and .ports .ports.websocket)) | quote }}
+- name: IGGY_CLUSTER_NODES_{{ $id }}_PORTS_TCP_REPLICA
+ value: {{ .ports.tcpReplica | quote }}
+ {{- end }}
+{{- end }}
+
+{{/*
+Name of the Secret holding the cluster-wide secrets the chart generates from
+inline values. Each of encryption, JWT and replica auth may instead point at a
+Secret the operator made, which is the path a multi-node cluster wants: the
+values have to be byte-identical on every node, so they belong in one object
+every release references rather than in each release's values.
+*/}}
+{{- define "iggy.secretName" -}}
+ {{- printf "%s-secrets" (include "iggy.fullname" .) }}
+{{- end }}
+
+{{/*
+True when any secret has to be generated by the chart, i.e. an inline value is
+set for a feature that is switched on and no existing Secret was named for it.
+*/}}
+{{- define "iggy.createsSecret" -}}
+ {{- $server := .Values.server }}
+ {{- $create := false }}
+ {{- if and $server.encryption.enabled (not
$server.encryption.existingSecret.name) }}
+ {{- $create = true }}
+ {{- end }}
+ {{- if and (or $server.jwt.encodingSecret $server.jwt.decodingSecret) (not
$server.jwt.existingSecret.name) }}
+ {{- $create = true }}
+ {{- end }}
+ {{- if and $server.cluster.auth.enabled (not
$server.cluster.auth.existingSecret.name) }}
+ {{- $create = true }}
+ {{- end }}
+ {{- if $create }}true{{ end }}
+{{- end }}
+
+{{/*
+Validate the secret configuration. The server enforces all of this at boot, so
+catching it during render only saves a scheduling round trip, but a cluster
+whose PSK differs between nodes fails as a handshake rejection rather than as
+anything that names the cause.
+*/}}
+{{- define "iggy.validateSecrets" -}}
+ {{- $server := .Values.server }}
+ {{- if $server.encryption.enabled }}
+ {{- if and (not $server.encryption.key) (not
$server.encryption.existingSecret.name) }}
+ {{- fail "server.encryption.enabled is true but no key was given. Set
server.encryption.key to a base64-encoded 32-byte key, or point
server.encryption.existingSecret.name at a Secret holding one." }}
+ {{- end }}
+ {{- if and $server.encryption.key (not
$server.encryption.existingSecret.name) }}
+ {{- if ne (len (b64dec $server.encryption.key)) 32 }}
+ {{- fail (printf "server.encryption.key decodes to %d bytes.
AES-256-GCM takes a base64-encoded 32-byte key, and the server fails the boot
with 'Invalid encryption key' on anything else." (len (b64dec
$server.encryption.key))) }}
+ {{- end }}
+ {{- end }}
+ {{- end }}
+ {{- if $server.cluster.auth.enabled }}
+ {{- if not $server.cluster.enabled }}
+ {{- fail "server.cluster.auth.enabled is true but server.cluster.enabled
is false. Replica authentication only applies to the consensus port, which
exists in cluster mode." }}
+ {{- end }}
+ {{- if and (not $server.cluster.auth.sharedSecret) (not
$server.cluster.auth.existingSecret.name) }}
+ {{- fail "server.cluster.auth.enabled is true but no shared secret was
given. Set server.cluster.auth.sharedSecret, or point
server.cluster.auth.existingSecret.name at a Secret holding one. Every node
needs the byte-identical value." }}
+ {{- end }}
+ {{- if and $server.cluster.auth.sharedSecret (lt (len
$server.cluster.auth.sharedSecret) 32) }}
+ {{- fail (printf "server.cluster.auth.sharedSecret is %d bytes; the
server requires at least 32 bytes of CSPRNG output." (len
$server.cluster.auth.sharedSecret)) }}
+ {{- end }}
+ {{- if $server.cluster.auth.previousSharedSecret }}
+ {{- if lt (len $server.cluster.auth.previousSharedSecret) 32 }}
+ {{- fail (printf "server.cluster.auth.previousSharedSecret is %d
bytes; the server requires at least 32 bytes of CSPRNG output." (len
$server.cluster.auth.previousSharedSecret)) }}
+ {{- end }}
+ {{- if eq $server.cluster.auth.previousSharedSecret
$server.cluster.auth.sharedSecret }}
+ {{- fail "server.cluster.auth.previousSharedSecret equals
server.cluster.auth.sharedSecret, which the server rejects as a no-op rotation
window. Leave it empty outside a rotation." }}
+ {{- end }}
+ {{- end }}
+ {{- end }}
+{{- end }}
+
+{{/*
+Environment entries for the secret-backed settings, each sourced from whichever
+Secret owns it.
+*/}}
+{{- define "iggy.secretEnv" -}}
+ {{- $server := .Values.server }}
+ {{- $generated := include "iggy.secretName" . }}
+ {{- if $server.encryption.enabled }}
+- name: IGGY_SYSTEM_ENCRYPTION_ENABLED
+ value: "true"
+- name: IGGY_SYSTEM_ENCRYPTION_KEY
+ valueFrom:
+ secretKeyRef:
+ name: {{ default $generated $server.encryption.existingSecret.name }}
+ key: {{ ternary $server.encryption.existingSecret.key "encryptionKey"
(ne $server.encryption.existingSecret.name "") }}
+ {{- end }}
+ {{- if or $server.jwt.existingSecret.name $server.jwt.encodingSecret }}
+- name: IGGY_HTTP_JWT_ENCODING_SECRET
+ valueFrom:
+ secretKeyRef:
+ name: {{ default $generated $server.jwt.existingSecret.name }}
+ key: {{ ternary $server.jwt.existingSecret.encodingSecretKey
"jwtEncodingSecret" (ne $server.jwt.existingSecret.name "") }}
+ {{- end }}
+ {{- if or $server.jwt.existingSecret.name $server.jwt.decodingSecret }}
+- name: IGGY_HTTP_JWT_DECODING_SECRET
+ valueFrom:
+ secretKeyRef:
+ name: {{ default $generated $server.jwt.existingSecret.name }}
+ key: {{ ternary $server.jwt.existingSecret.decodingSecretKey
"jwtDecodingSecret" (ne $server.jwt.existingSecret.name "") }}
+ optional: true
+ {{- end }}
+ {{- if $server.cluster.auth.enabled }}
+- name: IGGY_CLUSTER_AUTH_ENABLED
+ value: "true"
+- name: IGGY_CLUSTER_AUTH_SHARED_SECRET
+ valueFrom:
+ secretKeyRef:
+ name: {{ default $generated $server.cluster.auth.existingSecret.name }}
+ key: {{ ternary $server.cluster.auth.existingSecret.sharedSecretKey
"clusterSharedSecret" (ne $server.cluster.auth.existingSecret.name "") }}
+ {{- if or $server.cluster.auth.previousSharedSecret
$server.cluster.auth.existingSecret.name }}
+- name: IGGY_CLUSTER_AUTH_PREVIOUS_SHARED_SECRET
+ valueFrom:
+ secretKeyRef:
+ name: {{ default $generated $server.cluster.auth.existingSecret.name }}
+ key: {{ ternary
$server.cluster.auth.existingSecret.previousSharedSecretKey
"clusterPreviousSharedSecret" (ne $server.cluster.auth.existingSecret.name "")
}}
+ optional: true
+ {{- end }}
+ {{- end }}
+{{- end }}
diff --git a/helm/charts/iggy/templates/deployment.yaml
b/helm/charts/iggy/templates/deployment.yaml
index 4e7b6751e..c99ac9dc9 100644
--- a/helm/charts/iggy/templates/deployment.yaml
+++ b/helm/charts/iggy/templates/deployment.yaml
@@ -19,6 +19,14 @@
{{- if hasKey .Values.server "podSecurityContext" }}
{{- fail "server.podSecurityContext has been moved to podSecurityContext
(root level). Please update your values." }}
{{- end }}
+ {{- if gt (int .Values.server.replicaCount) 1 }}
+ {{- fail "server.replicaCount > 1 is refused: the replicas share one PVC
at one subPath and one advertised address, so they would run as independent
servers writing the same data directory, and the server takes no lock against
that. Run cluster mode instead: one release per node, each with
server.cluster.enabled and its own server.cluster.selfReplicaId. See the
Cluster Mode section of the chart README." }}
+ {{- end }}
+ {{- if .Values.server.cluster.enabled }}
+ {{- include "iggy.validateCluster" . }}
+ {{- end }}
+ {{- include "iggy.validateSecrets" . }}
+ {{- $ports := include "iggy.serverPorts" . | fromJson }}
---
apiVersion: apps/v1
kind: Deployment
@@ -27,17 +35,38 @@ metadata:
labels:
{{- include "iggy.labels" . | nindent 4 }}
spec:
- {{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.server.replicaCount }}
+ {{- if .Values.server.strategy }}
+ strategy:
+ {{- toYaml .Values.server.strategy | nindent 4 }}
+ {{- else if .Values.server.hostNetwork }}
+ strategy:
+ type: Recreate
{{- end }}
selector:
matchLabels:
{{- include "iggy.selectorLabels" . | nindent 6 }}
template:
metadata:
- {{- with .Values.podAnnotations }}
+ {{- $secretsChecksum := "" }}
+ {{- if (include "iggy.createsSecret" .) }}
+ {{- $secretsChecksum = (include (print $.Template.BasePath
"/server-secrets.yaml") . | sha256sum) }}
+ {{- end }}
+ {{- $rootChecksum := "" }}
+ {{- if and .Values.server.users.root.createSecret (not
.Values.server.users.root.existingSecret.name) }}
+ {{- $rootChecksum = (include (print $.Template.BasePath
"/root-user-credentials.yaml") . | sha256sum) }}
+ {{- end }}
+ {{- if or $secretsChecksum $rootChecksum .Values.podAnnotations }}
annotations:
+ {{- if $secretsChecksum }}
+ checksum/server-secrets: {{ $secretsChecksum }}
+ {{- end }}
+ {{- if $rootChecksum }}
+ checksum/root-user-credentials: {{ $rootChecksum }}
+ {{- end }}
+ {{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
+ {{- end }}
{{- end }}
labels:
{{- include "iggy.labels" . | nindent 8 }}
@@ -47,6 +76,11 @@ spec:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "iggy.serviceAccountName" . }}
+ enableServiceLinks: false
+ {{- if .Values.server.hostNetwork }}
+ hostNetwork: true
+ dnsPolicy: ClusterFirstWithHostNet
+ {{- end }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
@@ -55,16 +89,34 @@ spec:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.server.image.repository }}:{{
.Values.server.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.server.image.pullPolicy }}
+ {{- if or .Values.server.cluster.enabled .Values.server.extraArgs }}
+ args:
+ {{- if .Values.server.cluster.enabled }}
+ - "--replica-id"
+ - {{ .Values.server.cluster.selfReplicaId | quote }}
+ {{- end }}
+ {{- range .Values.server.extraArgs }}
+ - {{ . | quote }}
+ {{- end }}
+ {{- end }}
ports:
- name: http
- containerPort: {{ .Values.server.ports.http }}
+ containerPort: {{ int $ports.http }}
protocol: TCP
- name: tcp
- containerPort: {{ .Values.server.ports.tcp }}
+ containerPort: {{ int $ports.tcp }}
+ protocol: TCP
+ - name: websocket
+ containerPort: {{ int $ports.websocket }}
protocol: TCP
- name: quic
- containerPort: {{ .Values.server.ports.quic }}
+ containerPort: {{ int $ports.quic }}
+ protocol: UDP
+ {{- if .Values.server.cluster.enabled }}
+ - name: tcp-replica
+ containerPort: {{ int $ports.tcpReplica }}
protocol: TCP
+ {{- end }}
env:
{{- if .Values.server.users.root.existingSecret.name }}
- name: IGGY_ROOT_USERNAME
@@ -98,13 +150,22 @@ spec:
{{- $declaredInEnv = true }}
{{- end }}
{{- end }}
+ {{- if and .Values.server.cluster.enabled .Values.server.advertisedAddress }}
+ {{- fail "server.advertisedAddress is set together with
server.cluster.enabled. In cluster mode the server ignores
IGGY_NODE_ADVERTISED_ADDRESS and publishes the roster entry's ip, or its
advertisedAddress when one is given, so set it on this node's
server.cluster.nodes entry instead." }}
+ {{- end }}
{{- if and $declaredInEnv .Values.server.advertisedAddress }}
{{- fail "IGGY_NODE_ADVERTISED_ADDRESS is set in server.env and
server.advertisedAddress is also set. The server.env entry wins and
server.advertisedAddress is ignored. Please set only one." }}
{{- end }}
- {{- if not $declaredInEnv }}
+ {{- if and (not $declaredInEnv) (not .Values.server.cluster.enabled) }}
- name: IGGY_NODE_ADVERTISED_ADDRESS
value: {{ .Values.server.advertisedAddress | default (printf
"%s.%s.svc.cluster.local" (include "iggy.fullname" .) .Release.Namespace) |
quote }}
{{- end }}
+ {{- if .Values.server.cluster.enabled }}
+ {{- include "iggy.clusterEnv" . | nindent 12 }}
+ {{- end }}
+ {{- with (include "iggy.secretEnv" . | trim) }}
+ {{- . | nindent 12 }}
+ {{- end }}
{{- if .Values.server.env }}
{{- range .Values.server.env }}
- name: {{ .name }}
@@ -173,7 +234,7 @@ spec:
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
- {{- include "iggy-ui.labels" . | nindent 8 }}-ui
+ {{- include "iggy-ui.labels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
diff --git a/helm/charts/iggy/templates/hpa.yaml
b/helm/charts/iggy/templates/hpa.yaml
deleted file mode 100644
index d1f658129..000000000
--- a/helm/charts/iggy/templates/hpa.yaml
+++ /dev/null
@@ -1,61 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-{{ if .Values.autoscaling.enabled -}}
- {{- if semverCompare ">=1.23-0" .Capabilities.KubeVersion.GitVersion -}}
-apiVersion: autoscaling/v2
- {{- else -}}
-apiVersion: autoscaling/v2beta2
- {{- end }}
-kind: HorizontalPodAutoscaler
-metadata:
- name: {{ include "iggy.fullname" . }}
- labels:
- {{- include "iggy.labels" . | nindent 4 }}
-spec:
- scaleTargetRef:
- apiVersion: apps/v1
- kind: Deployment
- name: {{ include "iggy.fullname" . }}
- minReplicas: {{ .Values.autoscaling.minReplicas }}
- maxReplicas: {{ .Values.autoscaling.maxReplicas }}
- metrics:
- {{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
- - type: Resource
- resource:
- name: cpu
- {{- if semverCompare ">=1.23-0" .Capabilities.KubeVersion.GitVersion }}
- target:
- type: Utilization
- averageUtilization: {{
.Values.autoscaling.targetCPUUtilizationPercentage }}
- {{- else }}
- targetAverageUtilization: {{
.Values.autoscaling.targetCPUUtilizationPercentage }}
- {{- end }}
- {{- end }}
- {{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
- - type: Resource
- resource:
- name: memory
- {{- if semverCompare ">=1.23-0" .Capabilities.KubeVersion.GitVersion }}
- target:
- type: Utilization
- averageUtilization: {{
.Values.autoscaling.targetMemoryUtilizationPercentage }}
- {{- else }}
- targetAverageUtilization: {{
.Values.autoscaling.targetMemoryUtilizationPercentage }}
- {{- end }}
- {{- end }}
-{{- end }}
diff --git a/helm/charts/iggy/templates/server-secrets.yaml
b/helm/charts/iggy/templates/server-secrets.yaml
new file mode 100644
index 000000000..e75e6e498
--- /dev/null
+++ b/helm/charts/iggy/templates/server-secrets.yaml
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+{{- if and .Values.server.enabled (include "iggy.createsSecret" .) }}
+apiVersion: v1
+kind: Secret
+metadata:
+ name: {{ include "iggy.secretName" . }}
+ labels:
+ {{- include "iggy.labels" . | nindent 4 }}
+type: Opaque
+stringData:
+ {{- if and .Values.server.encryption.enabled (not
.Values.server.encryption.existingSecret.name) }}
+ encryptionKey: {{ .Values.server.encryption.key | quote }}
+ {{- end }}
+ {{- if not .Values.server.jwt.existingSecret.name }}
+ {{- with .Values.server.jwt.encodingSecret }}
+ jwtEncodingSecret: {{ . | quote }}
+ {{- end }}
+ {{- with .Values.server.jwt.decodingSecret }}
+ jwtDecodingSecret: {{ . | quote }}
+ {{- end }}
+ {{- end }}
+ {{- if and .Values.server.cluster.auth.enabled (not
.Values.server.cluster.auth.existingSecret.name) }}
+ clusterSharedSecret: {{ .Values.server.cluster.auth.sharedSecret | quote }}
+ {{- with .Values.server.cluster.auth.previousSharedSecret }}
+ clusterPreviousSharedSecret: {{ . | quote }}
+ {{- end }}
+ {{- end }}
+{{- end }}
diff --git a/helm/charts/iggy/templates/service.yaml
b/helm/charts/iggy/templates/service.yaml
index 982004424..4294e3828 100644
--- a/helm/charts/iggy/templates/service.yaml
+++ b/helm/charts/iggy/templates/service.yaml
@@ -32,11 +32,15 @@ spec:
- name: quic
port: {{ .Values.server.ports.quic }}
targetPort: quic
- protocol: TCP
+ protocol: UDP
- name: tcp
port: {{ .Values.server.ports.tcp }}
targetPort: tcp
protocol: TCP
+ - name: websocket
+ port: {{ .Values.server.ports.websocket }}
+ targetPort: websocket
+ protocol: TCP
selector:
{{- include "iggy.selectorLabels" . | nindent 4 }}
{{- end }}
diff --git a/helm/charts/iggy/values.yaml b/helm/charts/iggy/values.yaml
index be9757156..ab1a5f728 100644
--- a/helm/charts/iggy/values.yaml
+++ b/helm/charts/iggy/values.yaml
@@ -20,7 +20,8 @@ server:
# -- Client-facing address published in cluster metadata. Declaring
# `IGGY_NODE_ADVERTISED_ADDRESS` in `server.env` instead also works, but
# setting both is refused at render time. Empty falls back to the in-cluster
- # Service DNS name.
+ # Service DNS name. Ignored in cluster mode, where the address comes from the
+ # node's roster entry, so setting both is refused there too.
advertisedAddress: ""
# -- Enable the Iggy server deployment
enabled: true
@@ -31,15 +32,17 @@ server:
repository: apache/iggy
# -- Image pull policy
pullPolicy: Always
- # -- Server image tag (overrides chart appVersion)
- tag: "0.7.0"
+ # -- Server image tag. Empty uses the chart appVersion.
+ tag: ""
ports:
# -- HTTP API port
http: 3000
- # -- QUIC protocol port
+ # -- QUIC protocol port (UDP)
quic: 8080
# -- TCP protocol port
tcp: 8090
+ # -- WebSocket protocol port
+ websocket: 8092
service:
# -- Service type for the server
@@ -100,6 +103,115 @@ server:
# -- PVC storage size
size: 8Gi
+ # -- Server-side encryption of message payloads and state commands, using
+ # AES-256-GCM. Every node of a cluster must hold the identical key, or it
+ # cannot read data another node wrote.
+ encryption:
+ # -- Enable encryption at rest
+ enabled: false
+ # -- 32-byte key, base64 encoded. Ignored when `existingSecret.name` is
set.
+ # Prefer `existingSecret` outside development: a value here is stored in
the
+ # Helm release and readable by anyone who can read it.
+ key: ""
+ existingSecret:
+ # -- Name of an existing Secret holding the encryption key
+ name: ""
+ # -- Key inside that Secret
+ key: encryptionKey
+
+ # -- Secrets used to sign and validate HTTP bearer tokens. Left unset, each
+ # node generates a random secret on every start, which invalidates tokens
+ # across restarts and keeps them node-local. Setting the identical secret on
+ # every node makes bearers valid cluster-wide and activates follower to
+ # primary HTTP forwarding; `cluster.auth.enabled` derives the same thing from
+ # the replica PSK, so it is an alternative rather than an addition.
+ jwt:
+ # -- Encoding secret. Ignored when `existingSecret.name` is set.
+ encodingSecret: ""
+ # -- Decoding secret. Ignored when `existingSecret.name` is set.
+ decodingSecret: ""
+ existingSecret:
+ # -- Name of an existing Secret holding the JWT secrets
+ name: ""
+ # -- Key inside that Secret holding the encoding secret
+ encodingSecretKey: jwtEncodingSecret
+ # -- Key inside that Secret holding the decoding secret
+ decodingSecretKey: jwtDecodingSecret
+
+ cluster:
+ # -- Enable cluster (VSR consensus) mode. One Helm release per node: every
+ # release shares the same `nodes` roster and overrides only
`selfReplicaId`.
+ # See the Cluster Mode section of the chart README.
+ enabled: false
+ # -- Cluster name, byte-identical on every node. Hashed into the on-disk
+ # cluster id on first boot, so changing it later means starting from an
+ # empty data directory.
+ name: iggy-cluster
+ # -- Which `nodes` entry this release runs, matched against `replicaId`.
+ selfReplicaId: 0
+ # -- Refuse to render a cluster node without `server.hostNetwork`. The
+ # replica listener binds the roster `ip` verbatim, which no pod owns on the
+ # cluster network, so the pod would die at boot with `CannotBindToSocket`.
+ # Set this to false only when the roster `ip` is an address the pod itself
+ # holds.
+ requireHostNetwork: true
+ # -- Replica-to-replica authentication on the consensus port. When enabled
+ # every peer must complete an authenticated handshake or be rejected, and a
+ # shared secret becomes mandatory. Enabling it on a running cluster is a
+ # coordinated-restart change, not a rolling one.
+ auth:
+ # -- Require the authenticated replica handshake
+ enabled: false
+ # -- Cluster-wide pre-shared key, at least 32 bytes of CSPRNG output,
+ # byte-identical on every node. Ignored when `existingSecret.name` is
set.
+ sharedSecret: ""
+ # -- Retiring key, accepted for verification only while a rotation is in
+ # flight. Leave empty outside a rotation.
+ previousSharedSecret: ""
+ existingSecret:
+ # -- Name of an existing Secret holding the pre-shared keys
+ name: ""
+ # -- Key inside that Secret holding the active shared secret
+ sharedSecretKey: clusterSharedSecret
+ # -- Key inside that Secret holding the retiring shared secret
+ previousSharedSecretKey: clusterPreviousSharedSecret
+
+ # -- Cluster roster, mirroring the server's `[[cluster.nodes]]` config.
+ # Every node runs the identical list. `ip` is the replica-plane address:
+ # this node's consensus listener binds it verbatim and every peer dials it
+ # verbatim, so it must be a literal IP that the pod itself owns. With
+ # `server.hostNetwork` that is the node IP. `ports.tcpReplica` is required
+ # on every entry; the remaining ports default to `server.ports`.
+ nodes: []
+ # nodes:
+ # - name: iggy-node-0
+ # ip: 10.0.1.5
+ # replicaId: 0
+ # # -- Optional client-facing address, a literal IP or a DNS hostname.
+ # advertisedAddress: ""
+ # ports:
+ # tcp: 8090
+ # quic: 8080
+ # http: 3000
+ # websocket: 8092
+ # tcpReplica: 9090
+
+ # -- Run the server pod in the host network namespace. Required for cluster
+ # mode, where the replica listener binds the roster IP verbatim.
+ hostNetwork: false
+
+ # -- Deployment update strategy. Empty lets the chart choose: `Recreate` when
+ # `hostNetwork` is set, because a rolling update would wait forever for a
+ # replacement pod that cannot bind host ports the outgoing pod still holds,
+ # and the Kubernetes default otherwise.
+ strategy: {}
+
+ # -- Extra command-line arguments appended to the server entrypoint, e.g.
+ # `["--with-default-root-credentials"]` for a throwaway development install.
+ # `--replica-id` is not one of them: the chart passes it already whenever
+ # `cluster.enabled` is set.
+ extraArgs: []
+
# -- Environment variables for the server container
env:
- name: RUST_LOG
@@ -232,15 +344,3 @@ securityContext:
# -- Resource limits and requests for server
resources: {}
-
-autoscaling:
- # -- Enable horizontal pod autoscaling
- enabled: false
- # -- Minimum replicas for autoscaling
- minReplicas: 1
- # -- Maximum replicas for autoscaling
- maxReplicas: 100
- # -- Target CPU utilization for autoscaling
- targetCPUUtilizationPercentage: 80
- # -- Target memory utilization for autoscaling (optional)
- # targetMemoryUtilizationPercentage: 80
diff --git a/scripts/ci/test-helm.sh b/scripts/ci/test-helm.sh
index 370b2f00b..531522314 100755
--- a/scripts/ci/test-helm.sh
+++ b/scripts/ci/test-helm.sh
@@ -53,6 +53,9 @@
HELM_SMOKE_GATEWAY_NAME="${HELM_SMOKE_GATEWAY_NAME:-iggy-smoke-gateway}"
HELM_SMOKE_GATEWAY_PF_PORT="${HELM_SMOKE_GATEWAY_PF_PORT:-8080}"
HELM_SMOKE_KIND_NAME="${HELM_SMOKE_KIND_NAME:-iggy-helm-smoke}"
HELM_SMOKE_SERVER_CPU_ALLOCATION="${HELM_SMOKE_SERVER_CPU_ALLOCATION:-1}"
+# A well-formed 32-byte AES-256-GCM key, base64 encoded. Render-only: it never
+# reaches a running server.
+HELM_TEST_ENCRYPTION_KEY="AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8="
# HELM_SMOKE_GATEWAY_NAMESPACE and HELM_SMOKE_GATEWAY_NAME must match the
# defaults in scripts/ci/setup-helm-smoke-cluster.sh - the HTTPRoute
@@ -197,12 +200,10 @@ validate() {
local chart_version
local chart_app_version
- local server_image_tag
local ui_image_tag
chart_version="$(extract_chart_field version)"
chart_app_version="$(extract_chart_field appVersion)"
- server_image_tag="$(extract_values_tag server)"
ui_image_tag="$(extract_values_tag ui)"
prepare_render_dir
@@ -217,19 +218,16 @@ validate() {
grep -q "helm.sh/chart: iggy-${chart_version}"
"$HELM_RENDER_DIR/default.yaml"
grep -q "helm.sh/chart: iggy-ui-${chart_version}"
"$HELM_RENDER_DIR/default.yaml"
grep -q "app.kubernetes.io/version: \"${chart_app_version}\""
"$HELM_RENDER_DIR/default.yaml"
- grep -q "image: \"apache/iggy:${server_image_tag}\""
"$HELM_RENDER_DIR/default.yaml"
+ grep -q "image: \"apache/iggy:${chart_app_version}\""
"$HELM_RENDER_DIR/default.yaml"
grep -q "image: \"apache/iggy-web-ui:${ui_image_tag}\""
"$HELM_RENDER_DIR/default.yaml"
helm template iggy "$CHART_DIR" \
--set server.persistence.enabled=true \
- --set autoscaling.enabled=true \
- --set autoscaling.targetCPUUtilizationPercentage=80 \
--set server.ingress.enabled=true \
--set ui.ingress.enabled=true \
--set server.serviceMonitor.enabled=true \
> "$HELM_RENDER_DIR/all-features.yaml"
grep -q '^kind: PersistentVolumeClaim$' "$HELM_RENDER_DIR/all-features.yaml"
- grep -q '^kind: HorizontalPodAutoscaler$'
"$HELM_RENDER_DIR/all-features.yaml"
test "$(grep -c '^kind: Ingress$' "$HELM_RENDER_DIR/all-features.yaml")" -eq
2
test "$(extract_kind_names "$HELM_RENDER_DIR/all-features.yaml" Ingress |
sort -u | wc -l | tr -d ' ')" -eq 2
extract_kind_names "$HELM_RENDER_DIR/all-features.yaml" Ingress | grep -qx
'iggy'
@@ -239,14 +237,10 @@ validate() {
helm template iggy "$CHART_DIR" \
--kube-version 1.18.0 \
--api-versions networking.k8s.io/v1beta1 \
- --api-versions autoscaling/v2beta2 \
- --set autoscaling.enabled=true \
- --set autoscaling.targetCPUUtilizationPercentage=80 \
--set server.ingress.enabled=true \
--set ui.ingress.enabled=true \
> "$HELM_RENDER_DIR/legacy-k8s-1.18.yaml"
test "$(grep -c '^apiVersion: networking.k8s.io/v1beta1$'
"$HELM_RENDER_DIR/legacy-k8s-1.18.yaml")" -eq 2
- grep -q '^apiVersion: autoscaling/v2beta2$'
"$HELM_RENDER_DIR/legacy-k8s-1.18.yaml"
helm template iggy "$CHART_DIR" --set ui.enabled=false >
"$HELM_RENDER_DIR/server-only.yaml"
test "$(grep -c '^kind: Deployment$' "$HELM_RENDER_DIR/server-only.yaml")"
-eq 1
@@ -266,11 +260,69 @@ validate() {
fi
grep -q 'name: supersecret' "$HELM_RENDER_DIR/existing-secret.yaml"
+ helm template iggy "$CHART_DIR" \
+ -f "$CHART_DIR/examples/cluster-3-node.yaml" \
+ --set server.cluster.selfReplicaId=1 \
+ > "$HELM_RENDER_DIR/cluster.yaml"
+ grep -q 'name: IGGY_CLUSTER_ENABLED' "$HELM_RENDER_DIR/cluster.yaml"
+ grep -q 'name: IGGY_CLUSTER_NODES_2_PORTS_TCP_REPLICA'
"$HELM_RENDER_DIR/cluster.yaml"
+ grep -q -- '- "--replica-id"' "$HELM_RENDER_DIR/cluster.yaml"
+ grep -q '^ hostNetwork: true$' "$HELM_RENDER_DIR/cluster.yaml"
+ grep -q 'name: tcp-replica' "$HELM_RENDER_DIR/cluster.yaml"
+
+ grep -q 'name: IGGY_CLUSTER_AUTH_SHARED_SECRET'
"$HELM_RENDER_DIR/cluster.yaml"
+ grep -q 'name: IGGY_SYSTEM_ENCRYPTION_KEY' "$HELM_RENDER_DIR/cluster.yaml"
+ grep -q 'name: IGGY_HTTP_JWT_ENCODING_SECRET' "$HELM_RENDER_DIR/cluster.yaml"
+ if grep -qE '^ +(encryptionKey|clusterSharedSecret|jwtEncodingSecret):'
"$HELM_RENDER_DIR/cluster.yaml"; then
+ echo "Error: cluster render inlined a secret value instead of referencing
the existing Secret" >&2
+ exit 1
+ fi
+
+ helm template iggy "$CHART_DIR" \
+ --set server.encryption.enabled=true \
+ --set-string server.encryption.key="$HELM_TEST_ENCRYPTION_KEY" \
+ > "$HELM_RENDER_DIR/generated-secret.yaml"
+ grep -q "^ encryptionKey: \"${HELM_TEST_ENCRYPTION_KEY}\"$"
"$HELM_RENDER_DIR/generated-secret.yaml"
+ grep -q '^ name: iggy-secrets$' "$HELM_RENDER_DIR/generated-secret.yaml"
+ grep -q '^ name: iggy-secrets$'
"$HELM_RENDER_DIR/generated-secret.yaml"
+ grep -q '^ key: encryptionKey$'
"$HELM_RENDER_DIR/generated-secret.yaml"
+ test "$(grep -c '^kind: Secret$' "$HELM_RENDER_DIR/generated-secret.yaml")"
-eq 2
+
+ assert_render_rejected "server.replicaCount=3" --set server.replicaCount=3
+ assert_render_rejected "encryption without a key" --set
server.encryption.enabled=true
+ assert_render_rejected "an encryption key that is not 32 bytes" \
+ --set server.encryption.enabled=true \
+ --set-string server.encryption.key=bm90LTMyLWJ5dGVz
+ assert_render_rejected "replica auth without a secret" \
+ -f "$CHART_DIR/examples/cluster-3-node.yaml" \
+ --set server.cluster.auth.existingSecret.name="" \
+ --set server.cluster.selfReplicaId=0
+ assert_render_rejected "a shared secret under 32 bytes" \
+ -f "$CHART_DIR/examples/cluster-3-node.yaml" \
+ --set server.cluster.auth.existingSecret.name="" \
+ --set-string server.cluster.auth.sharedSecret=tooshort \
+ --set server.cluster.selfReplicaId=0
+ assert_render_rejected "cluster without a roster" --set
server.cluster.enabled=true
+ assert_render_rejected "selfReplicaId outside the roster" \
+ -f "$CHART_DIR/examples/cluster-3-node.yaml" --set
server.cluster.selfReplicaId=9
+
validate_yamllint
validate_helmfmt
validate_helm_docs
}
+# Assert that `helm template` refuses a values combination the chart guards
+# against. A guard that stops failing is a silent data-corruption regression,
+# so the render succeeding is the error case here.
+assert_render_rejected() {
+ local description="$1"
+ shift
+ if helm template iggy "$CHART_DIR" "$@" > /dev/null 2>&1; then
+ echo "Error: chart rendered ${description}, but that combination must be
refused" >&2
+ exit 1
+ fi
+}
+
# PID of the kubectl port-forward process started by smoke().
# Stored at script scope so the EXIT trap can kill it on any code path.
HELM_SMOKE_GW_PF_PID=""
