bitflicker64 commented on code in PR #494: URL: https://github.com/apache/hugegraph-doc/pull/494#discussion_r4192319417
########## content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md: ########## @@ -0,0 +1,523 @@ +--- +title: "Operate HugeGraph on Kubernetes" +linkTitle: "Operations on Kubernetes (Helm)" +weight: 5 +search_keywords: + - helm + - kubernetes + - operations + - disaster recovery + - networkpolicy +--- + +### 1 Scope + +This page is for operating a HugeGraph cluster that the Helm chart already +deployed. Installing, upgrading, and uninstalling are on the +[deployment page](/docs/quickstart/hugegraph/hugegraph-helm/); the full +parameter reference stays in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration). + +Commands below assume the release is named `hugegraph` in namespace +`hugegraph`; substitute your own names. Two credentials come up repeatedly, +both read from chart-managed Secrets: + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin \ + -o jsonpath='{.data.password}' | base64 --decode)" +PD_SECRET="$(kubectl get secret -n hugegraph hugegraph-pd-auth \ + -o jsonpath='{.data.secret-key}' | base64 --decode)" +``` + +### 2 Ports and health + +| Component | Port | Purpose | +|------|-------------|---------| +| PD | `8686` | gRPC (Store and Server clients) | +| PD | `8620` | REST / health probes | +| PD | `8610` | Raft | +| Store | `8500` | gRPC | +| Store | `8510` | Raft | +| Store | `8520` | REST / health probes | +| Server | `8080` | Gremlin and REST API | + +All ports are configurable through values; changing `server.port` updates the +listener, container port, and Service together. A stalled component (process +alive but frozen) is ended by its liveness probe: the default 20 s period and +3-failure threshold bound the blast radius of a stalled Store at roughly one +minute, and raft moves its partition leaders within seconds of the restart. + +Reach the API through a port-forward. It runs in the foreground, so keep +it in one terminal: + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +``` + +and call the API from a second terminal, with `PASSWORD` set as above: + +```bash +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/graphs +``` + +### 3 Scheduling + +Every component (`pd`, `store`, `server`, `hubble`) exposes `nodeSelector`, +`tolerations`, `affinity`, `topologySpreadConstraints`, and +`priorityClassName`. Pinning Store to labeled nodes is just: + +```yaml +store: + nodeSelector: + hugegraph/role: storage +``` + +`antiAffinity` (`required` | `preferred` | `disabled`) renders a hostname +pod-anti-affinity preset for `pd`, `store`, and `server`; Hubble is +single-replica by design and has no such key. Setting a raw `affinity` +replaces the preset entirely. All three default to `preferred`, so the chart +schedules on clusters with fewer nodes than replicas. The trade: under node +pressure the scheduler may co-locate replicas, and a single node failure can +then take more than one PD or Store replica with it. Production clusters with +enough nodes should pin `pd.antiAffinity` and `store.antiAffinity` to +`required`, as `values-cluster.yaml` does. + +### 4 Partition sharding + +A fresh install seeds PD with a partition shard count of 3 when +`store.replicas` is at least 3, and 1 otherwise; without the seed, the PD +image pins `partition.default-shard-count` to 1 and chart-deployed clusters +would run without store-level HA. The chart passes the value as +`-Dpartition.default-shard-count` in PD's `JAVA_OPTS`, so the image's JVM +auto-sizing is unaffected. + +**The seed applies at first bootstrap only.** PD persists the shard count the +first time it starts with empty storage; from then on the stored value is +authoritative, and changing `pd.partition.defaultShardCount` or scaling +`store.replicas` later has no effect on an initialized cluster. To change +the shard count of a running cluster, use PD's own config API (odd values +only, at most the live store count), then trigger +`GET /v1/task/patrolPartitions` and expect shard-group reallocation. + +The shard count also fixes the initial partition count: +`store.replicas x storeMaxShardCount / shardCount`, computed once at +bootstrap. With the image default `store-max-shard-count` of 12, a default +3-store install gets 12 partitions; raise +`pd.partition.storeMaxShardCount` (likewise seeded once) when more +partitions are wanted. + +An explicit `pd.partition.defaultShardCount` must be odd and at most +`store.replicas`; the chart rejects other values at render time because PD +would otherwise clamp or refuse them silently. + +### 5 NetworkPolicy + +`networkPolicy.enabled` renders one NetworkPolicy per component (PD, Store, +Server, and Hubble when enabled). Each policy isolates its own Pods in both +directions and lists the traffic they need, so apply order does not matter. +It is off in `values.yaml`, because the base values cannot know who your +clients are, and on in `values-cluster.yaml`. + +It only works when the cluster's network plugin enforces NetworkPolicy (kind +v0.25 or later, k3s, Calico, Cilium); other plugins accept the objects and +enforce nothing. To check, run a Pod without chart labels in another +namespace and `curl` the PD client Service on the REST port: it must time +out. + +With the policies on, the release admits only its own traffic: + +| To | From, ports | +|---|---| +| PD | PD: raft, gRPC. Store, Server, and Hubble in `pd` mode: gRPC, REST | +| Store | Store: raft. Server: gRPC, REST. Hubble in `pd` mode: REST | +| Server | Hubble and the `helm test` Pod: `server.port` | +| Hubble | nothing (port-forward uses loopback and needs no rule) | + +Every component may also resolve DNS on port 53. Nothing outside the release +is admitted unless it is listed in `networkPolicy.<component>.extraIngress`, +including the Ingress controller and the clients of a NodePort or +LoadBalancer Service. Exposing PD, Server or Hubble that way, or setting +`server.advertiseUrl`, with an empty `extraIngress` fails the render instead +of opening the port. The check sees only exposure the chart creates; a +Service, Gateway route or proxy you add yourself needs its own entry. An +in-cluster Hubble in `pd` mode combined with `server.advertiseUrl` also +fails the render until `networkPolicy.hubble.extraEgress` admits the +advertised destination: PD hands Hubble that URL for discovery, and +Hubble's egress policy otherwise reaches only the release's own Pods. + +Two egress facts to plan around. PD, Store and Server reach nothing outside +the release except DNS, so features that call out do not work with the +policies on; the one default caller is the Store image, which downloads +`libjemalloc.so` from github.com on every start. With the policies on that +connection times out after about two minutes and the Store starts without +jemalloc (measured on kind: Ready after 151 s instead of 11 s); the same +happens on any cluster without internet access. And the `helm test` Pod is +selected by no chart policy, so under a namespace-wide default-deny policy +of your own you must allow it egress to `server.port` and DNS. + +**Letting other workloads in.** Every rule must name its peers in `from`; +to admit any address, write an `ipBlock` such as `0.0.0.0/0` explicitly. A +`namespaceSelector` and a `podSelector` in the same peer must both match; +as two separate peers, either one is enough. What a NodePort or +LoadBalancer client looks like from the Pod depends on the network plugin, +`externalTrafficPolicy`, and the node the request arrives on. Measured with +a NodePort Server on two-node kind clusters: + +- kindnet, and Cilium with kube-proxy replacement: a call to the Server's + own node arrived with the client address; through the other node it + arrived with that node's address. +- Calico: through the other node the call arrived from that node's tunnel + address inside the Pod CIDR. +- Cilium with kube-proxy: no `ipBlock` rule admitted NodePort traffic, + because Cilium identifies node addresses by its own node identities + rather than by CIDR. + +Test with the plugin you run and name the CIDR you see arriving. A worked +example admitting an Ingress controller, an application namespace, and +Prometheus: + +```yaml +networkPolicy: + server: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: ingress-nginx + podSelector: + matchLabels: + app.kubernetes.io/name: ingress-nginx + ports: + - port: 8080 + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: apps + ports: + - port: 8080 + pd: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: monitoring + ports: + - port: 8620 +``` + +### 6 Rolling Store images safely + +Store rolling updates advance on a listener check, not on shard recovery, +so the controller can replace the next Store while the previous one is +still rejoining its shard groups. `values-cluster.yaml` therefore sets +`store.updateStrategy.type=OnDelete` (set it yourself on any other +production values): an upgrade then updates the StatefulSet without +replacing any Store Pod, and you delete Store Pods one at a time, checking +between deletions. `OnDelete` only stops automatic advancement; deleting +Pods without the checks below carries the same risk. + +`Up` in PD is not that check. PD marks a Store `Up` at registration, before +the Store has restored any partition, and a stopped Store stays `Up` in +every shard group until its keep-alive entry expires (300 s on current +images): a Pod deleted and back inside that window never leaves `Up` at +all. Start with the Pod instead: + +```bash +kubectl -n hugegraph wait --for=condition=Ready \ + pod/hugegraph-store-<ordinal> --timeout=10m +``` + +Then check shard membership and leadership per group, read from the PD +leader (see Disaster Recovery below for finding the leader): + +```bash +curl -s -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/shardGroups | jq ' + .shardGroups[] | {id: (.id // 0), + shards: [.shards[] | {storeId, role}], + leaders: [.shards[] | select(.role=="Leader")] | length}' +``` + +Delete the next Store only when the replaced Pod is `Ready`, its Store id +shows a fresh `lastHeartBeat` in `/v1/stores`, and every group reports its +full shard count with exactly one `Leader`. + +Know what this does not prove: the shard list is PD's membership record, +not a statement that the Store caught up on the raft log. No endpoint on +current images reports restoration-complete. For a closer look, +port-forward the replaced Store and read its own view of a group: +`GET :8520/v1/partition/<groupId>` returns the raft role, leader, and +committed index (`logIndex`) that Store holds, and fails while the Store +is down; compare the index with a peer Store rather than reading it +alone. This route never fills its `term` field, so `term` always reads 0 +there. The plural `GET :8520/v1/partitions` answers 500 on any Store that +follows a group on images built before +[apache/hugegraph#3232](https://github.com/apache/hugegraph/pull/3232) +(merged 2026-09-24); after it, it answers 200 on every Store, with `conf` +and `peers` null for followed groups. The per-group path works on both. + +Leave a margin after the membership check, keep `store.pdb.minAvailable` +at `replicas - 1` so an accidental second eviction is refused, and treat a +group that is short a shard or has no leader as a stop. A real +restoration-complete signal is upstream work, tracked in +[apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229). + +### 7 Disaster recovery + +What PD automates on current builds is narrow: a 60-second patrol only +marks Stores that stopped sending heartbeats as `Offline`. There is no +automatic re-replication; re-placing the replicas of a lost Store, +reconciling shard groups, and processing tombstoned Stores run only when a +partition patrol is triggered explicitly. + +The task endpoints execute locally on the PD that receives them, and a +follower answers with an empty success while doing nothing. +Port-forwarding the client Service selects an arbitrary PD, so identify +the leader first, then port-forward that Pod (two terminals: the forward +runs in the foreground): + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-pd-client 8620:8620 +# Read .data.pdLeader.raftUrl; its host names the leader Pod. +curl -su "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/members +# Stop the Service forward, then forward the leader Pod instead. +kubectl port-forward -n hugegraph pod/<leader-pod> 8620:8620 +# Reconcile shard groups and process tombstoned Stores. +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/patrolPartitions +# If the patrol repaired a shard group, wait 180 s first (see below). +# Spread Raft leaders, then partition data. +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balanceLeaders +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balancePartitions +``` + +Read `/v1/members` again after the tasks: if leadership moved mid-sequence, +the later tasks ran on a follower and did nothing. PD sets a +balance-shard flag for 180 s in two cases: `patrolPartitions` sets it +whenever it reallocates a shard group whose shard count is wrong, and +`balancePartitions` sets it even when it moves nothing. Inside that +window `balanceLeaders` is refused and `balancePartitions` returns +without doing anything, so wait at least 180 s after a patrol that +repaired a group, and after `balancePartitions`, before the next +balancing call. On images built before +[apache/hugegraph#3233](https://github.com/apache/hugegraph/pull/3233) +(merged 2026-09-24) the refusal is a bare HTTP 500 whose reason appears +only in the PD log; after it, the reason comes back in the body as +`{"status":1001,"error":"balance shard is processing, please try later!"}`. + +Telling a real run from a no-op takes the PD leader's log, because the +responses do not: `patrolPartitions` answers the same empty success on the +leader and on a follower, whether or not it repaired anything (look for +`reallocShards`, `shardOffline` or `storeTurnoff` lines, or diff +`/v1/shardGroups` before and after), and `balancePartitions` answers `{}` +on the leader and an empty body on a follower. `balanceLeaders` is the one +call whose body carries the work. Distinguishable task responses are +upstream work, tracked in +[apache/hugegraph#3231](https://github.com/apache/hugegraph/issues/3231). + +Run `patrolPartitions` after replacing a Store that is not coming back, +`balancePartitions` once the cluster is stable again, and `balanceLeaders` +after restarts that skewed leader placement. + +**Losing a Store volume.** A Store rebuilt with an empty PVC recovers in +place on images carrying +[apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234) +(merged 2026-09-24); on every earlier image, including all published +release images, it does not. In both cases the replacement registers under +a new Store ID while its Pod name, DNS name, and raft address are +unchanged, so `/v1/stores` lists two IDs at one address. + +On post-#3234 images the retirement works: find the old ID in `/v1/stores` +(the row at the replaced Pod's address that is not the newly registered +one), `POST /v1/store/<oldId>` with `{"storeState":"Tombstone"}` on the PD +leader, run `GET /v1/task/patrolPartitions`, and wait; verify every shard +group is back to full shard count with one leader, no group names the old +ID, and the replaced Store's own `:8520/v1/partition/<groupId>` answers +200 for every group; then `DELETE /v1/store/<oldId>` to erase the retired +record. Measured on a 3+3+3 install with images built from `master` at +`dbb6663a`: the replacement was Ready in 156 s, all 12 groups converged +onto the new ID 1 s after the Tombstone and patrol (the empty Store caught +up by raft snapshot install), and a continuous writer lost 0 acknowledged +writes. + +On images without #3234 the same retirement runs and does not repair the +groups: PD fires the configuration change, but the group leader sees the +address already in the group, so jraft has nothing to add and the group +keeps the old ID. Measured: 20 minutes and three patrols later, all 12 +groups still listed the retired ID and the replacement Store held no +partitions. Nothing in the health surface shows the failure; `/v1/stores`, +cluster state, Hubble, and Pod readiness all read healthy while every +shard group runs on two live replicas. The one check that shows it is the +replaced Store's own `:8520/v1/partition/<groupId>`. + +So the default remains: replace a Store Pod, keep its PVC (the Store id +lives in the data path, and the Pod comes back under the same id). Treat +the empty-PVC procedure as recovery for post-#3234 images only; on a +released image a genuinely lost volume leaves the cluster degraded, and +expect to rebuild rather than to recover in place. + +Periodic leader balancing is tracked in +[apache/hugegraph#3135](https://github.com/apache/hugegraph/issues/3135); +disaster-recovery metrics in +[apache/hugegraph#3136](https://github.com/apache/hugegraph/issues/3136). + +### 8 Scaling + +PD and Store reserve the maximum StatefulSet ordinal in their resource +names, so scaling never renames a PersistentVolumeClaim or shifts a Pod +identity; both are capped at 99 replicas. Server scales freely through +`server.replicas` or `server.hpa` (with HPA on, the Deployment omits +`spec.replicas` so upgrades do not overwrite the autoscaler). + +A staged rollout cannot be written in a values file (the schema requires +one replica per component); stage it with `kubectl scale statefulset +hugegraph-store --replicas=0` and scale back up when ready. The Servers +wait, not ready, until Stores register, and the next `helm upgrade` +restores the full topology from values. + +Changing PD or Store replicas on a live release is not a values change: +raft and shard membership are persisted, and Pods alone do not reconfigure +them. The chart rejects both directions for PD and a shrink for Store by +reading the live StatefulSet, so fresh installs at any count are +unaffected. + +**PD, either direction.** The rendered peer list reaches raft only as its +bootstrap configuration, which an initialized group ignores; a 3-to-5 +upgrade leaves the voting configuration at three, and a 3-to-1 shrink +loses quorum outright. Membership changes go through PD's peer-list +change, which the PD client API and the PD REST route +`POST /v1/members/change` both expose; the route needs the same Basic +authentication as the other PD REST routes, and the chart wraps neither. +Change the membership through PD, confirm it in `/v1/members`, scale the +live StatefulSet, then `helm upgrade` with the matching value; until that +sequence is verified on your own build, install the PD count you intend +to keep. + +**Store, shrinking.** Draining is a state transition, not a balance: +neither `patrolPartitions` nor `balancePartitions` retires a healthy +Store. Retire the leaving Stores the way Disaster Recovery retires a +replaced one: + +1. Check the remaining Stores still cover the persisted shard count + (`pd.partition.defaultShardCount`; empty derives 3 when + `store.replicas` is at least 3). +2. Map the ordinals the shrink will delete (the highest ones) to Store + ids through `/v1/stores`, matching on the Pod address. +3. `POST /v1/store/<id>` with `{"storeState":"Tombstone"}` for each Review Comment: Important: Step 3 cannot succeed on a cluster this chart deployed. The chart sets `HG_PD_INITIAL_STORE_COUNT` to `store.replicas` ([pd-statefulset.yaml L123-124](https://github.com/apache/hugegraph/blob/8fd9841ebc6770df6119fdce07249fa9e558ae9e/helm/hugegraph/templates/pd-statefulset.yaml#L123-L124)), and that value is PD's `minStoreCount` ([PDConfig L63-64](https://github.com/apache/hugegraph/blob/8fd9841ebc6770df6119fdce07249fa9e558ae9e/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/config/PDConfig.java#L63-L64)). `POST /v1/store/<id>` goes through `PDRestService.updateStore` into `StoreNodeService.updateStore`, which refuses a Tombstone for a Store whose last state is `Up` when `getStores().size() - 1 < minStoreCount` ([StoreNodeService L271-278](https://github.com/apache/hugegraph/blob/8fd9841ebc6770df6119fdce07249fa9e558ae9e/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/StoreNodeService.java#L271-L278)). On a fresh 5-Store install, retiring a hea lthy Store gives 5 - 1 < 5, so PD returns `LESS_ACTIVE_STORE` ("The number of active stores is less then 5"). `StoreAPI.setStore` serializes that exception into the JSON body instead of failing the request, so an operator following the steps may not notice. Step 4 then never finishes. The Disaster Recovery retirement passes this check only because the replacement adds a second record at the same address. I traced this in the source at apache/hugegraph master 8fd9841eb and did not run it on a cluster. Please say that PD refuses to Tombstone an `Up` Store while the chart pins `pd.initial-store-count` to the current `store.replicas`. Then either document a path that works or mark Store shrink as unsupported for now, as the page already does for PD. Please make the same change in the Chinese page, section 8 step 3 (line 297). ########## content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md: ########## @@ -0,0 +1,523 @@ +--- +title: "Operate HugeGraph on Kubernetes" +linkTitle: "Operations on Kubernetes (Helm)" +weight: 5 +search_keywords: + - helm + - kubernetes + - operations + - disaster recovery + - networkpolicy +--- + +### 1 Scope + +This page is for operating a HugeGraph cluster that the Helm chart already +deployed. Installing, upgrading, and uninstalling are on the +[deployment page](/docs/quickstart/hugegraph/hugegraph-helm/); the full +parameter reference stays in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration). + +Commands below assume the release is named `hugegraph` in namespace +`hugegraph`; substitute your own names. Two credentials come up repeatedly, +both read from chart-managed Secrets: + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin \ + -o jsonpath='{.data.password}' | base64 --decode)" +PD_SECRET="$(kubectl get secret -n hugegraph hugegraph-pd-auth \ + -o jsonpath='{.data.secret-key}' | base64 --decode)" +``` + +### 2 Ports and health + +| Component | Port | Purpose | +|------|-------------|---------| +| PD | `8686` | gRPC (Store and Server clients) | +| PD | `8620` | REST / health probes | +| PD | `8610` | Raft | +| Store | `8500` | gRPC | +| Store | `8510` | Raft | +| Store | `8520` | REST / health probes | +| Server | `8080` | Gremlin and REST API | + +All ports are configurable through values; changing `server.port` updates the +listener, container port, and Service together. A stalled component (process +alive but frozen) is ended by its liveness probe: the default 20 s period and +3-failure threshold bound the blast radius of a stalled Store at roughly one +minute, and raft moves its partition leaders within seconds of the restart. + +Reach the API through a port-forward. It runs in the foreground, so keep +it in one terminal: + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +``` + +and call the API from a second terminal, with `PASSWORD` set as above: + +```bash +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/graphs +``` + +### 3 Scheduling + +Every component (`pd`, `store`, `server`, `hubble`) exposes `nodeSelector`, +`tolerations`, `affinity`, `topologySpreadConstraints`, and +`priorityClassName`. Pinning Store to labeled nodes is just: + +```yaml +store: + nodeSelector: + hugegraph/role: storage +``` + +`antiAffinity` (`required` | `preferred` | `disabled`) renders a hostname +pod-anti-affinity preset for `pd`, `store`, and `server`; Hubble is +single-replica by design and has no such key. Setting a raw `affinity` +replaces the preset entirely. All three default to `preferred`, so the chart +schedules on clusters with fewer nodes than replicas. The trade: under node +pressure the scheduler may co-locate replicas, and a single node failure can +then take more than one PD or Store replica with it. Production clusters with +enough nodes should pin `pd.antiAffinity` and `store.antiAffinity` to +`required`, as `values-cluster.yaml` does. + +### 4 Partition sharding + +A fresh install seeds PD with a partition shard count of 3 when +`store.replicas` is at least 3, and 1 otherwise; without the seed, the PD +image pins `partition.default-shard-count` to 1 and chart-deployed clusters +would run without store-level HA. The chart passes the value as +`-Dpartition.default-shard-count` in PD's `JAVA_OPTS`, so the image's JVM +auto-sizing is unaffected. + +**The seed applies at first bootstrap only.** PD persists the shard count the +first time it starts with empty storage; from then on the stored value is +authoritative, and changing `pd.partition.defaultShardCount` or scaling +`store.replicas` later has no effect on an initialized cluster. To change +the shard count of a running cluster, use PD's own config API (odd values +only, at most the live store count), then trigger +`GET /v1/task/patrolPartitions` and expect shard-group reallocation. + +The shard count also fixes the initial partition count: +`store.replicas x storeMaxShardCount / shardCount`, computed once at +bootstrap. With the image default `store-max-shard-count` of 12, a default +3-store install gets 12 partitions; raise +`pd.partition.storeMaxShardCount` (likewise seeded once) when more +partitions are wanted. + +An explicit `pd.partition.defaultShardCount` must be odd and at most +`store.replicas`; the chart rejects other values at render time because PD +would otherwise clamp or refuse them silently. + +### 5 NetworkPolicy + +`networkPolicy.enabled` renders one NetworkPolicy per component (PD, Store, +Server, and Hubble when enabled). Each policy isolates its own Pods in both +directions and lists the traffic they need, so apply order does not matter. +It is off in `values.yaml`, because the base values cannot know who your +clients are, and on in `values-cluster.yaml`. + +It only works when the cluster's network plugin enforces NetworkPolicy (kind +v0.25 or later, k3s, Calico, Cilium); other plugins accept the objects and +enforce nothing. To check, run a Pod without chart labels in another +namespace and `curl` the PD client Service on the REST port: it must time +out. + +With the policies on, the release admits only its own traffic: + +| To | From, ports | +|---|---| +| PD | PD: raft, gRPC. Store, Server, and Hubble in `pd` mode: gRPC, REST | +| Store | Store: raft. Server: gRPC, REST. Hubble in `pd` mode: REST | +| Server | Hubble and the `helm test` Pod: `server.port` | +| Hubble | nothing (port-forward uses loopback and needs no rule) | + +Every component may also resolve DNS on port 53. Nothing outside the release +is admitted unless it is listed in `networkPolicy.<component>.extraIngress`, +including the Ingress controller and the clients of a NodePort or +LoadBalancer Service. Exposing PD, Server or Hubble that way, or setting +`server.advertiseUrl`, with an empty `extraIngress` fails the render instead +of opening the port. The check sees only exposure the chart creates; a +Service, Gateway route or proxy you add yourself needs its own entry. An +in-cluster Hubble in `pd` mode combined with `server.advertiseUrl` also +fails the render until `networkPolicy.hubble.extraEgress` admits the +advertised destination: PD hands Hubble that URL for discovery, and +Hubble's egress policy otherwise reaches only the release's own Pods. + +Two egress facts to plan around. PD, Store and Server reach nothing outside +the release except DNS, so features that call out do not work with the +policies on; the one default caller is the Store image, which downloads +`libjemalloc.so` from github.com on every start. With the policies on that +connection times out after about two minutes and the Store starts without +jemalloc (measured on kind: Ready after 151 s instead of 11 s); the same +happens on any cluster without internet access. And the `helm test` Pod is +selected by no chart policy, so under a namespace-wide default-deny policy +of your own you must allow it egress to `server.port` and DNS. + +**Letting other workloads in.** Every rule must name its peers in `from`; +to admit any address, write an `ipBlock` such as `0.0.0.0/0` explicitly. A +`namespaceSelector` and a `podSelector` in the same peer must both match; +as two separate peers, either one is enough. What a NodePort or +LoadBalancer client looks like from the Pod depends on the network plugin, +`externalTrafficPolicy`, and the node the request arrives on. Measured with +a NodePort Server on two-node kind clusters: + +- kindnet, and Cilium with kube-proxy replacement: a call to the Server's + own node arrived with the client address; through the other node it + arrived with that node's address. +- Calico: through the other node the call arrived from that node's tunnel + address inside the Pod CIDR. +- Cilium with kube-proxy: no `ipBlock` rule admitted NodePort traffic, + because Cilium identifies node addresses by its own node identities + rather than by CIDR. + +Test with the plugin you run and name the CIDR you see arriving. A worked +example admitting an Ingress controller, an application namespace, and +Prometheus: + +```yaml +networkPolicy: + server: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: ingress-nginx + podSelector: + matchLabels: + app.kubernetes.io/name: ingress-nginx + ports: + - port: 8080 + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: apps + ports: + - port: 8080 + pd: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: monitoring + ports: + - port: 8620 +``` + +### 6 Rolling Store images safely + +Store rolling updates advance on a listener check, not on shard recovery, +so the controller can replace the next Store while the previous one is +still rejoining its shard groups. `values-cluster.yaml` therefore sets +`store.updateStrategy.type=OnDelete` (set it yourself on any other +production values): an upgrade then updates the StatefulSet without +replacing any Store Pod, and you delete Store Pods one at a time, checking +between deletions. `OnDelete` only stops automatic advancement; deleting +Pods without the checks below carries the same risk. + +`Up` in PD is not that check. PD marks a Store `Up` at registration, before +the Store has restored any partition, and a stopped Store stays `Up` in +every shard group until its keep-alive entry expires (300 s on current +images): a Pod deleted and back inside that window never leaves `Up` at +all. Start with the Pod instead: + +```bash +kubectl -n hugegraph wait --for=condition=Ready \ + pod/hugegraph-store-<ordinal> --timeout=10m +``` + +Then check shard membership and leadership per group, read from the PD +leader (see Disaster Recovery below for finding the leader): + +```bash +curl -s -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/shardGroups | jq ' + .shardGroups[] | {id: (.id // 0), + shards: [.shards[] | {storeId, role}], + leaders: [.shards[] | select(.role=="Leader")] | length}' +``` + +Delete the next Store only when the replaced Pod is `Ready`, its Store id +shows a fresh `lastHeartBeat` in `/v1/stores`, and every group reports its +full shard count with exactly one `Leader`. + +Know what this does not prove: the shard list is PD's membership record, +not a statement that the Store caught up on the raft log. No endpoint on +current images reports restoration-complete. For a closer look, +port-forward the replaced Store and read its own view of a group: +`GET :8520/v1/partition/<groupId>` returns the raft role, leader, and +committed index (`logIndex`) that Store holds, and fails while the Store +is down; compare the index with a peer Store rather than reading it +alone. This route never fills its `term` field, so `term` always reads 0 +there. The plural `GET :8520/v1/partitions` answers 500 on any Store that +follows a group on images built before +[apache/hugegraph#3232](https://github.com/apache/hugegraph/pull/3232) +(merged 2026-09-24); after it, it answers 200 on every Store, with `conf` +and `peers` null for followed groups. The per-group path works on both. + +Leave a margin after the membership check, keep `store.pdb.minAvailable` +at `replicas - 1` so an accidental second eviction is refused, and treat a +group that is short a shard or has no leader as a stop. A real +restoration-complete signal is upstream work, tracked in +[apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229). + +### 7 Disaster recovery + +What PD automates on current builds is narrow: a 60-second patrol only +marks Stores that stopped sending heartbeats as `Offline`. There is no +automatic re-replication; re-placing the replicas of a lost Store, +reconciling shard groups, and processing tombstoned Stores run only when a +partition patrol is triggered explicitly. + +The task endpoints execute locally on the PD that receives them, and a +follower answers with an empty success while doing nothing. +Port-forwarding the client Service selects an arbitrary PD, so identify +the leader first, then port-forward that Pod (two terminals: the forward +runs in the foreground): + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-pd-client 8620:8620 +# Read .data.pdLeader.raftUrl; its host names the leader Pod. +curl -su "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/members +# Stop the Service forward, then forward the leader Pod instead. +kubectl port-forward -n hugegraph pod/<leader-pod> 8620:8620 +# Reconcile shard groups and process tombstoned Stores. +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/patrolPartitions +# If the patrol repaired a shard group, wait 180 s first (see below). +# Spread Raft leaders, then partition data. +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balanceLeaders +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balancePartitions +``` + +Read `/v1/members` again after the tasks: if leadership moved mid-sequence, +the later tasks ran on a follower and did nothing. PD sets a +balance-shard flag for 180 s in two cases: `patrolPartitions` sets it +whenever it reallocates a shard group whose shard count is wrong, and +`balancePartitions` sets it even when it moves nothing. Inside that +window `balanceLeaders` is refused and `balancePartitions` returns +without doing anything, so wait at least 180 s after a patrol that +repaired a group, and after `balancePartitions`, before the next +balancing call. On images built before +[apache/hugegraph#3233](https://github.com/apache/hugegraph/pull/3233) +(merged 2026-09-24) the refusal is a bare HTTP 500 whose reason appears +only in the PD log; after it, the reason comes back in the body as +`{"status":1001,"error":"balance shard is processing, please try later!"}`. + +Telling a real run from a no-op takes the PD leader's log, because the +responses do not: `patrolPartitions` answers the same empty success on the +leader and on a follower, whether or not it repaired anything (look for +`reallocShards`, `shardOffline` or `storeTurnoff` lines, or diff +`/v1/shardGroups` before and after), and `balancePartitions` answers `{}` +on the leader and an empty body on a follower. `balanceLeaders` is the one +call whose body carries the work. Distinguishable task responses are +upstream work, tracked in +[apache/hugegraph#3231](https://github.com/apache/hugegraph/issues/3231). + +Run `patrolPartitions` after replacing a Store that is not coming back, +`balancePartitions` once the cluster is stable again, and `balanceLeaders` +after restarts that skewed leader placement. + +**Losing a Store volume.** A Store rebuilt with an empty PVC recovers in +place on images carrying +[apache/hugegraph#3234](https://github.com/apache/hugegraph/pull/3234) +(merged 2026-09-24); on every earlier image, including all published +release images, it does not. In both cases the replacement registers under +a new Store ID while its Pod name, DNS name, and raft address are +unchanged, so `/v1/stores` lists two IDs at one address. + +On post-#3234 images the retirement works: find the old ID in `/v1/stores` +(the row at the replaced Pod's address that is not the newly registered +one), `POST /v1/store/<oldId>` with `{"storeState":"Tombstone"}` on the PD +leader, run `GET /v1/task/patrolPartitions`, and wait; verify every shard +group is back to full shard count with one leader, no group names the old +ID, and the replaced Store's own `:8520/v1/partition/<groupId>` answers +200 for every group; then `DELETE /v1/store/<oldId>` to erase the retired +record. Measured on a 3+3+3 install with images built from `master` at +`dbb6663a`: the replacement was Ready in 156 s, all 12 groups converged +onto the new ID 1 s after the Tombstone and patrol (the empty Store caught +up by raft snapshot install), and a continuous writer lost 0 acknowledged +writes. + +On images without #3234 the same retirement runs and does not repair the +groups: PD fires the configuration change, but the group leader sees the +address already in the group, so jraft has nothing to add and the group +keeps the old ID. Measured: 20 minutes and three patrols later, all 12 +groups still listed the retired ID and the replacement Store held no +partitions. Nothing in the health surface shows the failure; `/v1/stores`, +cluster state, Hubble, and Pod readiness all read healthy while every +shard group runs on two live replicas. The one check that shows it is the +replaced Store's own `:8520/v1/partition/<groupId>`. + +So the default remains: replace a Store Pod, keep its PVC (the Store id +lives in the data path, and the Pod comes back under the same id). Treat +the empty-PVC procedure as recovery for post-#3234 images only; on a +released image a genuinely lost volume leaves the cluster degraded, and +expect to rebuild rather than to recover in place. + +Periodic leader balancing is tracked in +[apache/hugegraph#3135](https://github.com/apache/hugegraph/issues/3135); +disaster-recovery metrics in +[apache/hugegraph#3136](https://github.com/apache/hugegraph/issues/3136). + +### 8 Scaling + +PD and Store reserve the maximum StatefulSet ordinal in their resource +names, so scaling never renames a PersistentVolumeClaim or shifts a Pod +identity; both are capped at 99 replicas. Server scales freely through +`server.replicas` or `server.hpa` (with HPA on, the Deployment omits +`spec.replicas` so upgrades do not overwrite the autoscaler). + +A staged rollout cannot be written in a values file (the schema requires +one replica per component); stage it with `kubectl scale statefulset +hugegraph-store --replicas=0` and scale back up when ready. The Servers +wait, not ready, until Stores register, and the next `helm upgrade` +restores the full topology from values. + +Changing PD or Store replicas on a live release is not a values change: +raft and shard membership are persisted, and Pods alone do not reconfigure +them. The chart rejects both directions for PD and a shrink for Store by Review Comment: Minor: This section and the deployment page treat Store growth as a plain values change, but three things about it go undocumented. (1) With the default `store.pdb.minAvailable: 2`, `--set store.replicas=5` fails to render with "store.pdb.minAvailable must be at least store.replicas - 1" ([_helpers.tpl L780-781](https://github.com/apache/hugegraph/blob/8fd9841ebc6770df6119fdce07249fa9e558ae9e/helm/hugegraph/templates/_helpers.tpl#L780-L781)); `store.pdb.minAvailable` has to move with the replica count in both directions, which also affects the `helm upgrade` in step 5 of the shrink below. (2) `HG_PD_INITIAL_STORE_LIST` and `HG_PD_INITIAL_STORE_COUNT` live in the PD Pod template, so growing Store rolls every PD Pod. A `helm template` diff of `store.replicas` 3 against 5 shows only those two PD env values changing. (3) PD schedules no balance task ([TaskScheduleService.init](https://github.com/apache/hugegraph/blob/8fd9841ebc6770df6119fdce07249fa9e558ae9e/hugegraph-pd/hg-pd-core/src /main/java/org/apache/hugegraph/pd/TaskScheduleService.java#L121)), so new Stores hold no partitions until `balancePartitions` runs on the PD leader. Please add a short "Store, growing" paragraph that covers these three points and then the check from section 6 (full shard count, one Leader). Please sync the Chinese page, section 8 (line 283). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
