bitflicker64 commented on code in PR #494: URL: https://github.com/apache/hugegraph-doc/pull/494#discussion_r4166026864
########## content/en/docs/quickstart/hugegraph/hugegraph-helm.md: ########## @@ -0,0 +1,282 @@ +--- +title: "Deploy on Kubernetes with Helm" +linkTitle: "Deploy on Kubernetes (Helm)" +weight: 4 +search_keywords: + - helm + - kubernetes + - k8s +--- + +### 1 Overview + +The Helm chart deploys a distributed HugeGraph cluster on Kubernetes: PD, Store, and Server, plus the optional +Hubble UI. It lives in the main repository under +[`helm/hugegraph`](https://github.com/apache/hugegraph/tree/master/helm/hugegraph). + +| Component | Workload | Default replicas | Purpose | +|-----------|----------|------------------|---------| +| PD | StatefulSet + PVC | 3 | Placement driver: a Raft group tracking Stores and partitions | +| Store | StatefulSet + PVC | 3 | Graph data storage (HStore) | +| Server | Deployment | 3 | Gremlin and REST query layer | +| Hubble | Deployment | 0 (off) | Web UI, enabled with `hubble.enabled=true` | + +A distributed HugeGraph cluster has a startup contract (no `init-store` on Server, every Server using PD for graph +metadata, Store waiting for a PD quorum, one PD REST secret shared by three readers). The chart encodes that +contract so operators do not have to; the details are in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#chart-details). +Day-2 work (NetworkPolicy details, disaster recovery, scaling, safe Store +rolls, running Hubble outside the cluster) is on the +[operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/). + +```mermaid +flowchart LR + subgraph Kubernetes cluster + PD[PD StatefulSet<br>Raft group of 3, PVC] + Store[Store StatefulSet<br>3 replicas, PVC] -->|register, heartbeat| PD + Server[Server Deployment<br>3 replicas] -->|metadata, discovery| PD + Server -->|gRPC read/write| Store + Hubble[Hubble Deployment<br>optional UI] -->|discovers Servers via PD| PD + end + Client[Client / hugegraph-client] -->|REST / Gremlin| Server +``` + +Startup order is enforced by the chart, not by the operator: PD pods elect a leader first, each Store pod waits in +an init container until a majority of PD peers report ready, and Servers keep restarting their storage wait until +Stores have registered. A fresh install converges without manual steps. + +### 2 Prerequisites + +- Kubernetes 1.23 or later (the chart renders `autoscaling/v2` and `policy/v1`) +- Helm 3; the `--reset-then-reuse-values` flag mentioned under Upgrade needs Helm 3.14 or later +- Dynamic volume provisioning: a default StorageClass, or an explicit `storageClassName` for PD and Store +- Memory for nine JVMs in the default topology; see the resource note under Install + +The chart requires component images that carry the PD readiness endpoint and PD REST authentication (both merged +for the release after 1.7.0). The default image tags already point at builds that include them; 1.7.0 images are +not supported. + +### 3 Install + +#### 3.1 Get the chart + +The chart is not published to a chart repository yet, so install it from the source tree: + +```bash +git clone https://github.com/apache/hugegraph.git +cd hugegraph +``` + +#### 3.2 Install with default values + +First confirm `kubectl` points at the intended cluster and that it can provision volumes. PVCs stuck in `Pending` +for want of a StorageClass are the most common first-run failure: + +```bash +kubectl config current-context +kubectl get storageclass +``` + +Then install: + +```bash +helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace --wait --timeout 15m +``` + +`--wait` makes Helm block until every workload is ready, which for a distributed cluster is the signal that PD +elected a leader, Stores registered, and Servers came up; without it `helm install` returns as soon as the objects +are created. A fresh cluster normally converges in a few minutes; the 15 minute timeout leaves room for slow image +pulls. + +Two defaults to know before going further: + +- **No resources are set.** Every pod is BestEffort and each JVM sizes its heap against total node memory. That is + fine on a single node; on a multi-node cluster the heaps oversubscribe the nodes and pods abort. Use + `values-cluster.yaml` or set `resources` per component for anything beyond a laptop. +- **Image tags track `latest`** until the next HugeGraph release is published, with `pullPolicy: Always`. Pin tags + or digests for production. + +#### 3.3 Topology presets + +The chart ships three values files: + +| File | Topology | Intended use | +|------|----------|--------------| +| `values.yaml` | 3 PD + 3 Store + 3 Server | Default; preferred anti-affinity, auth on, Hubble off | +| `values-single.yaml` | 1 + 1 + 1 | Single-node development and CI; smaller PVCs | +| `values-cluster.yaml` | 3 + 3 + 3 | Production starting point: JVM heap and resource settings, PD/Store PodDisruptionBudgets, `required` anti-affinity, NetworkPolicy on | + +```bash +helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace \ + -f helm/hugegraph/values-single.yaml --wait --timeout 15m +``` + +`values-cluster.yaml` is a starting point, not a capacity guarantee: recalculate resources for your graph size and +traffic. One of its numbers deserves a note: it requests 5Gi and limits at 8Gi of memory per Store, well above the +1Gi heap, because the Store's RocksDB caches live outside the JVM heap (a 4Gi limit OOM-killed Stores after about +1 GB of data). Scale both numbers with data size. The full parameter reference (every component, probe, +scheduling, and Secret knob) is kept in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration). + +#### 3.4 Verify the install + +```bash +helm test hugegraph --namespace hugegraph +``` + +Read the generated admin password and call the API: + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin -o jsonpath='{.data.password}' | base64 --decode)" +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions Review Comment: Fixed in 686d3b930. Section 3.4 now starts the port-forward in its own block and terminal, then reads the password and runs `curl` from a second terminal. The operations page health check and its per-Pod Gremlin check got the same split, in EN and CN. ########## content/en/docs/quickstart/hugegraph/hugegraph-helm.md: ########## @@ -0,0 +1,282 @@ +--- +title: "Deploy on Kubernetes with Helm" +linkTitle: "Deploy on Kubernetes (Helm)" +weight: 4 +search_keywords: + - helm + - kubernetes + - k8s +--- + +### 1 Overview + +The Helm chart deploys a distributed HugeGraph cluster on Kubernetes: PD, Store, and Server, plus the optional +Hubble UI. It lives in the main repository under +[`helm/hugegraph`](https://github.com/apache/hugegraph/tree/master/helm/hugegraph). + +| Component | Workload | Default replicas | Purpose | +|-----------|----------|------------------|---------| +| PD | StatefulSet + PVC | 3 | Placement driver: a Raft group tracking Stores and partitions | +| Store | StatefulSet + PVC | 3 | Graph data storage (HStore) | +| Server | Deployment | 3 | Gremlin and REST query layer | +| Hubble | Deployment | 0 (off) | Web UI, enabled with `hubble.enabled=true` | + +A distributed HugeGraph cluster has a startup contract (no `init-store` on Server, every Server using PD for graph +metadata, Store waiting for a PD quorum, one PD REST secret shared by three readers). The chart encodes that +contract so operators do not have to; the details are in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#chart-details). +Day-2 work (NetworkPolicy details, disaster recovery, scaling, safe Store +rolls, running Hubble outside the cluster) is on the +[operations page](/docs/quickstart/hugegraph/hugegraph-helm-operations/). + +```mermaid +flowchart LR + subgraph Kubernetes cluster + PD[PD StatefulSet<br>Raft group of 3, PVC] + Store[Store StatefulSet<br>3 replicas, PVC] -->|register, heartbeat| PD + Server[Server Deployment<br>3 replicas] -->|metadata, discovery| PD + Server -->|gRPC read/write| Store + Hubble[Hubble Deployment<br>optional UI] -->|discovers Servers via PD| PD + end + Client[Client / hugegraph-client] -->|REST / Gremlin| Server +``` + +Startup order is enforced by the chart, not by the operator: PD pods elect a leader first, each Store pod waits in +an init container until a majority of PD peers report ready, and Servers keep restarting their storage wait until +Stores have registered. A fresh install converges without manual steps. + +### 2 Prerequisites + +- Kubernetes 1.23 or later (the chart renders `autoscaling/v2` and `policy/v1`) +- Helm 3; the `--reset-then-reuse-values` flag mentioned under Upgrade needs Helm 3.14 or later +- Dynamic volume provisioning: a default StorageClass, or an explicit `storageClassName` for PD and Store +- Memory for nine JVMs in the default topology; see the resource note under Install + +The chart requires component images that carry the PD readiness endpoint and PD REST authentication (both merged +for the release after 1.7.0). The default image tags already point at builds that include them; 1.7.0 images are +not supported. + +### 3 Install + +#### 3.1 Get the chart + +The chart is not published to a chart repository yet, so install it from the source tree: + +```bash +git clone https://github.com/apache/hugegraph.git +cd hugegraph +``` + +#### 3.2 Install with default values + +First confirm `kubectl` points at the intended cluster and that it can provision volumes. PVCs stuck in `Pending` +for want of a StorageClass are the most common first-run failure: + +```bash +kubectl config current-context +kubectl get storageclass +``` + +Then install: + +```bash +helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace --wait --timeout 15m +``` + +`--wait` makes Helm block until every workload is ready, which for a distributed cluster is the signal that PD +elected a leader, Stores registered, and Servers came up; without it `helm install` returns as soon as the objects +are created. A fresh cluster normally converges in a few minutes; the 15 minute timeout leaves room for slow image +pulls. + +Two defaults to know before going further: + +- **No resources are set.** Every pod is BestEffort and each JVM sizes its heap against total node memory. That is + fine on a single node; on a multi-node cluster the heaps oversubscribe the nodes and pods abort. Use + `values-cluster.yaml` or set `resources` per component for anything beyond a laptop. +- **Image tags track `latest`** until the next HugeGraph release is published, with `pullPolicy: Always`. Pin tags + or digests for production. + +#### 3.3 Topology presets + +The chart ships three values files: + +| File | Topology | Intended use | +|------|----------|--------------| +| `values.yaml` | 3 PD + 3 Store + 3 Server | Default; preferred anti-affinity, auth on, Hubble off | +| `values-single.yaml` | 1 + 1 + 1 | Single-node development and CI; smaller PVCs | +| `values-cluster.yaml` | 3 + 3 + 3 | Production starting point: JVM heap and resource settings, PD/Store PodDisruptionBudgets, `required` anti-affinity, NetworkPolicy on | + +```bash +helm install hugegraph ./helm/hugegraph --namespace hugegraph --create-namespace \ + -f helm/hugegraph/values-single.yaml --wait --timeout 15m +``` + +`values-cluster.yaml` is a starting point, not a capacity guarantee: recalculate resources for your graph size and +traffic. One of its numbers deserves a note: it requests 5Gi and limits at 8Gi of memory per Store, well above the +1Gi heap, because the Store's RocksDB caches live outside the JVM heap (a 4Gi limit OOM-killed Stores after about +1 GB of data). Scale both numbers with data size. The full parameter reference (every component, probe, +scheduling, and Secret knob) is kept in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration). + +#### 3.4 Verify the install + +```bash +helm test hugegraph --namespace hugegraph +``` + +Read the generated admin password and call the API: + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin -o jsonpath='{.data.password}' | base64 --decode)" +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +``` + +The commands above assume the release is named `hugegraph`; with another name, substitute the release-prefixed +resource names (`kubectl get svc,secret -n <namespace>` lists them). The post-install notes printed by +`helm install` repeat these commands with the right names filled in. + +### 4 Authentication and Secrets + +Authentication is on by default and the chart manages three Secrets. Each credential resolves in the same order: +an `existingSecret` you created wins, then an inline value, then a random value generated at install time. + +| Secret | Key | Used for | Bring your own with | +|--------|-----|----------|---------------------| +| `<release>-admin` | `password` | Server admin account, Hubble login | `server.auth.admin.existingSecret` | +| `<release>-auth-token` | `token_secret` | JWT signing key shared by all Server replicas | `server.auth.token.existingSecret` | +| `<release>-pd-auth` | `secret-key` | PD REST authentication, read by PD, Server, and Hubble | `pd.auth.existingSecret` | + +To manage a credential yourself, create the Secret before installing and point the matching `existingSecret` value +at it; the chart never modifies a Secret it did not create. Value constraints: the admin password must not contain +newlines, carriage returns, or backslashes; the JWT key must be at least 32 bytes; the PD secret must be printable Review Comment: Fixed in 5e4cbd25c. Section 4 now says the admin password must not start or end with whitespace, next to the newline, carriage-return and backslash rules, in EN and CN. ########## content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md: ########## @@ -0,0 +1,483 @@ +--- +title: "Operate HugeGraph on Kubernetes" +linkTitle: "Operations on Kubernetes (Helm)" +weight: 5 +search_keywords: + - helm + - kubernetes + - operations + - disaster recovery + - networkpolicy +--- + +### 1 Scope + +This page is for operating a HugeGraph cluster that the Helm chart already +deployed. Installing, upgrading, and uninstalling are on the +[deployment page](/docs/quickstart/hugegraph/hugegraph-helm/); the full +parameter reference stays in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration). + +Commands below assume the release is named `hugegraph` in namespace +`hugegraph`; substitute your own names. Two credentials come up repeatedly, +both read from chart-managed Secrets: + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin \ + -o jsonpath='{.data.password}' | base64 --decode)" +PD_SECRET="$(kubectl get secret -n hugegraph hugegraph-pd-auth \ + -o jsonpath='{.data.secret-key}' | base64 --decode)" +``` + +### 2 Ports and health + +| Component | Port | Purpose | +|------|-------------|---------| +| PD | `8686` | gRPC (Store and Server clients) | +| PD | `8620` | REST / health probes | +| PD | `8610` | Raft | +| Store | `8500` | gRPC | +| Store | `8510` | Raft | +| Store | `8520` | REST / health probes | +| Server | `8080` | Gremlin and REST API | + +All ports are configurable through values; changing `server.port` updates the +listener, container port, and Service together. A stalled component (process +alive but frozen) is ended by its liveness probe: the default 20 s period and +3-failure threshold bound the blast radius of a stalled Store at roughly one +minute, and raft moves its partition leaders within seconds of the restart. + +Reach the API through a port-forward: + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/graphs +``` + +### 3 Scheduling + +Every component (`pd`, `store`, `server`, `hubble`) exposes `nodeSelector`, +`tolerations`, `affinity`, `topologySpreadConstraints`, and +`priorityClassName`. Pinning Store to labeled nodes is just: + +```yaml +store: + nodeSelector: + hugegraph/role: storage +``` + +`antiAffinity` (`required` | `preferred` | `disabled`) renders a hostname +pod-anti-affinity preset for `pd`, `store`, and `server`; Hubble is +single-replica by design and has no such key. Setting a raw `affinity` +replaces the preset entirely. All three default to `preferred`, so the chart +schedules on clusters with fewer nodes than replicas. The trade: under node +pressure the scheduler may co-locate replicas, and a single node failure can +then take more than one PD or Store replica with it. Production clusters with +enough nodes should pin `pd.antiAffinity` and `store.antiAffinity` to +`required`, as `values-cluster.yaml` does. + +### 4 Partition sharding + +A fresh install seeds PD with a partition shard count of 3 when +`store.replicas` is at least 3, and 1 otherwise; without the seed, the PD +image pins `partition.default-shard-count` to 1 and chart-deployed clusters +would run without store-level HA. The chart passes the value as +`-Dpartition.default-shard-count` in PD's `JAVA_OPTS`, so the image's JVM +auto-sizing is unaffected. + +**The seed applies at first bootstrap only.** PD persists the shard count the +first time it starts with empty storage; from then on the stored value is +authoritative, and changing `pd.partition.defaultShardCount` or scaling +`store.replicas` later has no effect on an initialized cluster. To change +the shard count of a running cluster, use PD's own config API (odd values +only, at most the live store count), then trigger +`GET /v1/task/patrolPartitions` and expect shard-group reallocation. + +The shard count also fixes the initial partition count: +`store.replicas x storeMaxShardCount / shardCount`, computed once at +bootstrap. With the image default `store-max-shard-count` of 12, a default +3-store install gets 12 partitions; raise +`pd.partition.storeMaxShardCount` (likewise seeded once) when more +partitions are wanted. + +An explicit `pd.partition.defaultShardCount` must be odd and at most +`store.replicas`; the chart rejects other values at render time because PD +would otherwise clamp or refuse them silently. + +### 5 NetworkPolicy + +`networkPolicy.enabled` renders one NetworkPolicy per component (PD, Store, +Server, and Hubble when enabled). Each policy isolates its own Pods in both +directions and lists the traffic they need, so apply order does not matter. +It is off in `values.yaml`, because the base values cannot know who your +clients are, and on in `values-cluster.yaml`. + +It only works when the cluster's network plugin enforces NetworkPolicy (kind +v0.25 or later, k3s, Calico, Cilium); other plugins accept the objects and +enforce nothing. To check, run a Pod without chart labels in another +namespace and `curl` the PD client Service on the REST port: it must time +out. + +With the policies on, the release admits only its own traffic: + +| To | From, ports | +|---|---| +| PD | PD: raft, gRPC. Store, Server, and Hubble in `pd` mode: gRPC, REST | +| Store | Store: raft. Server: gRPC, REST. Hubble in `pd` mode: REST | +| Server | Hubble and the `helm test` Pod: `server.port` | +| Hubble | nothing (port-forward uses loopback and needs no rule) | + +Every component may also resolve DNS on port 53. Nothing outside the release +is admitted unless it is listed in `networkPolicy.<component>.extraIngress`, +including the Ingress controller and the clients of a NodePort or +LoadBalancer Service. Exposing PD, Server or Hubble that way, or setting +`server.advertiseUrl`, with an empty `extraIngress` fails the render instead +of opening the port. The check sees only exposure the chart creates; a +Service, Gateway route or proxy you add yourself needs its own entry. + +Two egress facts to plan around. PD, Store and Server reach nothing outside +the release except DNS, so features that call out do not work with the +policies on; the one default caller is the Store image, which downloads +`libjemalloc.so` from github.com on every start. With the policies on that +connection times out after about two minutes and the Store starts without +jemalloc (measured on kind: Ready after 151 s instead of 11 s); the same +happens on any cluster without internet access. And the `helm test` Pod is +selected by no chart policy, so under a namespace-wide default-deny policy +of your own you must allow it egress to `server.port` and DNS. + +**Letting other workloads in.** Every rule must name its peers in `from`; +to admit any address, write an `ipBlock` such as `0.0.0.0/0` explicitly. A +`namespaceSelector` and a `podSelector` in the same peer must both match; +as two separate peers, either one is enough. What a NodePort or +LoadBalancer client looks like from the Pod depends on the network plugin, +`externalTrafficPolicy`, and the node the request arrives on. Measured with +a NodePort Server on two-node kind clusters: + +- kindnet, and Cilium with kube-proxy replacement: a call to the Server's + own node arrived with the client address; through the other node it + arrived with that node's address. +- Calico: through the other node the call arrived from that node's tunnel + address inside the Pod CIDR. +- Cilium with kube-proxy: no `ipBlock` rule admitted NodePort traffic, + because Cilium identifies node addresses by its own node identities + rather than by CIDR. + +Test with the plugin you run and name the CIDR you see arriving. A worked +example admitting an Ingress controller, an application namespace, and +Prometheus: + +```yaml +networkPolicy: + server: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: ingress-nginx + podSelector: + matchLabels: + app.kubernetes.io/name: ingress-nginx + ports: + - port: 8080 + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: apps + ports: + - port: 8080 + pd: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: monitoring + ports: + - port: 8620 +``` + +### 6 Rolling Store images safely + +Store rolling updates advance on a listener check, not on shard recovery, +so the controller can replace the next Store while the previous one is +still rejoining its shard groups. For a production image roll, set +`store.updateStrategy.type=OnDelete` and delete Store Pods one at a time, +checking between deletions. + +`Up` in PD is not that check. PD marks a Store `Up` at registration, before +the Store has restored any partition, and a stopped Store stays `Up` in +every shard group until its keep-alive entry expires (300 s on current +images): a Pod deleted and back inside that window never leaves `Up` at +all. Start with the Pod instead: + +```bash +kubectl -n hugegraph wait --for=condition=Ready \ + pod/hugegraph-store-<ordinal> --timeout=10m +``` + +Then check shard membership and leadership per group, read from the PD +leader (see Disaster Recovery below for finding the leader): + +```bash +curl -s -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/shardGroups | jq ' + .shardGroups[] | {id: (.id // 0), + shards: [.shards[] | {storeId, role}], + leaders: [.shards[] | select(.role=="Leader")] | length}' +``` + +Delete the next Store only when the replaced Pod is `Ready`, its Store id +shows a fresh `lastHeartBeat` in `/v1/stores`, and every group reports its +full shard count with exactly one `Leader`. + +Know what this does not prove: the shard list is PD's membership record, +not a statement that the Store caught up on the raft log. No endpoint on +current images reports restoration-complete. For a closer look, +port-forward the replaced Store and read its own view of a group: +`GET :8520/v1/partition/<groupId>` returns the raft role, term, and Review Comment: Fixed in 795d5f835. Confirmed on apache/hugegraph master (176fb56dd): `PartitionAPI.getPartition()` sets role, leader and `logIndex` but never `term`, while the plural handler calls `setTerm`. Section 6 now lists role, leader and committed index (`logIndex`), compares only the index with a peer Store, and says `term` always reads 0 on that route. CN updated. ########## content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md: ########## @@ -0,0 +1,483 @@ +--- +title: "Operate HugeGraph on Kubernetes" +linkTitle: "Operations on Kubernetes (Helm)" +weight: 5 +search_keywords: + - helm + - kubernetes + - operations + - disaster recovery + - networkpolicy +--- + +### 1 Scope + +This page is for operating a HugeGraph cluster that the Helm chart already +deployed. Installing, upgrading, and uninstalling are on the +[deployment page](/docs/quickstart/hugegraph/hugegraph-helm/); the full +parameter reference stays in the +[chart README](https://github.com/apache/hugegraph/tree/master/helm/hugegraph#configuration). + +Commands below assume the release is named `hugegraph` in namespace +`hugegraph`; substitute your own names. Two credentials come up repeatedly, +both read from chart-managed Secrets: + +```bash +PASSWORD="$(kubectl get secret -n hugegraph hugegraph-admin \ + -o jsonpath='{.data.password}' | base64 --decode)" +PD_SECRET="$(kubectl get secret -n hugegraph hugegraph-pd-auth \ + -o jsonpath='{.data.secret-key}' | base64 --decode)" +``` + +### 2 Ports and health + +| Component | Port | Purpose | +|------|-------------|---------| +| PD | `8686` | gRPC (Store and Server clients) | +| PD | `8620` | REST / health probes | +| PD | `8610` | Raft | +| Store | `8500` | gRPC | +| Store | `8510` | Raft | +| Store | `8520` | REST / health probes | +| Server | `8080` | Gremlin and REST API | + +All ports are configurable through values; changing `server.port` updates the +listener, container port, and Service together. A stalled component (process +alive but frozen) is ended by its liveness probe: the default 20 s period and +3-failure threshold bound the blast radius of a stalled Store at roughly one +minute, and raft moves its partition leaders within seconds of the restart. + +Reach the API through a port-forward: + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-server 8080:8080 +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/versions +curl --user "admin:${PASSWORD}" http://127.0.0.1:8080/graphs +``` + +### 3 Scheduling + +Every component (`pd`, `store`, `server`, `hubble`) exposes `nodeSelector`, +`tolerations`, `affinity`, `topologySpreadConstraints`, and +`priorityClassName`. Pinning Store to labeled nodes is just: + +```yaml +store: + nodeSelector: + hugegraph/role: storage +``` + +`antiAffinity` (`required` | `preferred` | `disabled`) renders a hostname +pod-anti-affinity preset for `pd`, `store`, and `server`; Hubble is +single-replica by design and has no such key. Setting a raw `affinity` +replaces the preset entirely. All three default to `preferred`, so the chart +schedules on clusters with fewer nodes than replicas. The trade: under node +pressure the scheduler may co-locate replicas, and a single node failure can +then take more than one PD or Store replica with it. Production clusters with +enough nodes should pin `pd.antiAffinity` and `store.antiAffinity` to +`required`, as `values-cluster.yaml` does. + +### 4 Partition sharding + +A fresh install seeds PD with a partition shard count of 3 when +`store.replicas` is at least 3, and 1 otherwise; without the seed, the PD +image pins `partition.default-shard-count` to 1 and chart-deployed clusters +would run without store-level HA. The chart passes the value as +`-Dpartition.default-shard-count` in PD's `JAVA_OPTS`, so the image's JVM +auto-sizing is unaffected. + +**The seed applies at first bootstrap only.** PD persists the shard count the +first time it starts with empty storage; from then on the stored value is +authoritative, and changing `pd.partition.defaultShardCount` or scaling +`store.replicas` later has no effect on an initialized cluster. To change +the shard count of a running cluster, use PD's own config API (odd values +only, at most the live store count), then trigger +`GET /v1/task/patrolPartitions` and expect shard-group reallocation. + +The shard count also fixes the initial partition count: +`store.replicas x storeMaxShardCount / shardCount`, computed once at +bootstrap. With the image default `store-max-shard-count` of 12, a default +3-store install gets 12 partitions; raise +`pd.partition.storeMaxShardCount` (likewise seeded once) when more +partitions are wanted. + +An explicit `pd.partition.defaultShardCount` must be odd and at most +`store.replicas`; the chart rejects other values at render time because PD +would otherwise clamp or refuse them silently. + +### 5 NetworkPolicy + +`networkPolicy.enabled` renders one NetworkPolicy per component (PD, Store, +Server, and Hubble when enabled). Each policy isolates its own Pods in both +directions and lists the traffic they need, so apply order does not matter. +It is off in `values.yaml`, because the base values cannot know who your +clients are, and on in `values-cluster.yaml`. + +It only works when the cluster's network plugin enforces NetworkPolicy (kind +v0.25 or later, k3s, Calico, Cilium); other plugins accept the objects and +enforce nothing. To check, run a Pod without chart labels in another +namespace and `curl` the PD client Service on the REST port: it must time +out. + +With the policies on, the release admits only its own traffic: + +| To | From, ports | +|---|---| +| PD | PD: raft, gRPC. Store, Server, and Hubble in `pd` mode: gRPC, REST | +| Store | Store: raft. Server: gRPC, REST. Hubble in `pd` mode: REST | +| Server | Hubble and the `helm test` Pod: `server.port` | +| Hubble | nothing (port-forward uses loopback and needs no rule) | + +Every component may also resolve DNS on port 53. Nothing outside the release +is admitted unless it is listed in `networkPolicy.<component>.extraIngress`, +including the Ingress controller and the clients of a NodePort or +LoadBalancer Service. Exposing PD, Server or Hubble that way, or setting +`server.advertiseUrl`, with an empty `extraIngress` fails the render instead +of opening the port. The check sees only exposure the chart creates; a +Service, Gateway route or proxy you add yourself needs its own entry. + +Two egress facts to plan around. PD, Store and Server reach nothing outside +the release except DNS, so features that call out do not work with the +policies on; the one default caller is the Store image, which downloads +`libjemalloc.so` from github.com on every start. With the policies on that +connection times out after about two minutes and the Store starts without +jemalloc (measured on kind: Ready after 151 s instead of 11 s); the same +happens on any cluster without internet access. And the `helm test` Pod is +selected by no chart policy, so under a namespace-wide default-deny policy +of your own you must allow it egress to `server.port` and DNS. + +**Letting other workloads in.** Every rule must name its peers in `from`; +to admit any address, write an `ipBlock` such as `0.0.0.0/0` explicitly. A +`namespaceSelector` and a `podSelector` in the same peer must both match; +as two separate peers, either one is enough. What a NodePort or +LoadBalancer client looks like from the Pod depends on the network plugin, +`externalTrafficPolicy`, and the node the request arrives on. Measured with +a NodePort Server on two-node kind clusters: + +- kindnet, and Cilium with kube-proxy replacement: a call to the Server's + own node arrived with the client address; through the other node it + arrived with that node's address. +- Calico: through the other node the call arrived from that node's tunnel + address inside the Pod CIDR. +- Cilium with kube-proxy: no `ipBlock` rule admitted NodePort traffic, + because Cilium identifies node addresses by its own node identities + rather than by CIDR. + +Test with the plugin you run and name the CIDR you see arriving. A worked +example admitting an Ingress controller, an application namespace, and +Prometheus: + +```yaml +networkPolicy: + server: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: ingress-nginx + podSelector: + matchLabels: + app.kubernetes.io/name: ingress-nginx + ports: + - port: 8080 + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: apps + ports: + - port: 8080 + pd: + extraIngress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: monitoring + ports: + - port: 8620 +``` + +### 6 Rolling Store images safely + +Store rolling updates advance on a listener check, not on shard recovery, +so the controller can replace the next Store while the previous one is +still rejoining its shard groups. For a production image roll, set +`store.updateStrategy.type=OnDelete` and delete Store Pods one at a time, +checking between deletions. + +`Up` in PD is not that check. PD marks a Store `Up` at registration, before +the Store has restored any partition, and a stopped Store stays `Up` in +every shard group until its keep-alive entry expires (300 s on current +images): a Pod deleted and back inside that window never leaves `Up` at +all. Start with the Pod instead: + +```bash +kubectl -n hugegraph wait --for=condition=Ready \ + pod/hugegraph-store-<ordinal> --timeout=10m +``` + +Then check shard membership and leadership per group, read from the PD +leader (see Disaster Recovery below for finding the leader): + +```bash +curl -s -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/shardGroups | jq ' + .shardGroups[] | {id: (.id // 0), + shards: [.shards[] | {storeId, role}], + leaders: [.shards[] | select(.role=="Leader")] | length}' +``` + +Delete the next Store only when the replaced Pod is `Ready`, its Store id +shows a fresh `lastHeartBeat` in `/v1/stores`, and every group reports its +full shard count with exactly one `Leader`. + +Know what this does not prove: the shard list is PD's membership record, +not a statement that the Store caught up on the raft log. No endpoint on +current images reports restoration-complete. For a closer look, +port-forward the replaced Store and read its own view of a group: +`GET :8520/v1/partition/<groupId>` returns the raft role, term, and +committed index that Store holds, and fails while the Store is down; +compare term and index with a peer Store rather than reading them alone. +The plural `GET :8520/v1/partitions` answers 500 on any Store that follows +a group on images built before +[apache/hugegraph#3232](https://github.com/apache/hugegraph/pull/3232) +(merged 2026-09-24); after it, it answers 200 on every Store, with `conf` +and `peers` null for followed groups. The per-group path works on both. + +Leave a margin after the membership check, keep `store.pdb.minAvailable` +at `replicas - 1` so an accidental second eviction is refused, and treat a +group that is short a shard or has no leader as a stop. A real +restoration-complete signal is upstream work, tracked in +[apache/hugegraph#3229](https://github.com/apache/hugegraph/issues/3229). + +### 7 Disaster recovery + +What PD automates on current builds is narrow: a 60-second patrol only +marks Stores that stopped sending heartbeats as `Offline`. There is no +automatic re-replication; re-placing the replicas of a lost Store, +reconciling shard groups, and processing tombstoned Stores run only when a +partition patrol is triggered explicitly. + +The task endpoints execute locally on the PD that receives them, and a +follower answers with an empty success while doing nothing. +Port-forwarding the client Service selects an arbitrary PD, so identify +the leader first, then port-forward that Pod (two terminals: the forward +runs in the foreground): + +```bash +kubectl port-forward -n hugegraph svc/hugegraph-pd-client 8620:8620 +# Read .data.pdLeader.raftUrl; its host names the leader Pod. +curl -su "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/members +# Stop the Service forward, then forward the leader Pod instead. +kubectl port-forward -n hugegraph pod/<leader-pod> 8620:8620 +# Reconcile shard groups and process tombstoned Stores. +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/patrolPartitions +# Spread Raft leaders, then partition data. +curl -u "hg:${PD_SECRET}" http://127.0.0.1:8620/v1/task/balanceLeaders Review Comment: Fixed in 795d5f835. Confirmed in `TaskScheduleService.patrolPartitions()`: a shard group whose count differs from the configured shard count gets `reallocShards` and `BALANCE_SHARD_KEY` for 180 s. Section 7 now says both a reallocating patrol and `balancePartitions` set the flag, that inside the window `balanceLeaders` is refused and `balancePartitions` does nothing, and to wait 180 s after a patrol that repaired a group. The command block has a comment at that step. CN updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
