This is an automated email from the ASF dual-hosted git repository.
imbajin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hugegraph.git
The following commit(s) were added to refs/heads/master by this push:
new e85b36a49 feat(pd): add quorum-aware /v1/ready endpoint and raft
gauges (#3185)
e85b36a49 is described below
commit e85b36a49b5834f720b71de18addd9767ae8951a
Author: KAI <[email protected]>
AuthorDate: Tue Sep 8 19:45:08 2026 +0530
feat(pd): add quorum-aware /v1/ready endpoint and raft gauges (#3185)
---
docker/README.md | 26 +++
hugegraph-pd/README.md | 2 +
hugegraph-pd/docs/api-reference.md | 74 ++++++
.../org/apache/hugegraph/pd/raft/RaftEngine.java | 171 +++++++++++++-
.../apache/hugegraph/pd/raft/RaftStateMachine.java | 42 ++++
hugegraph-pd/hg-pd-service/pom.xml | 6 +
.../org/apache/hugegraph/pd/metrics/PDMetrics.java | 22 ++
.../org/apache/hugegraph/pd/rest/StoreAPI.java | 31 +++
.../rest/interceptor/AuthenticationConfigurer.java | 3 +-
.../hugegraph/pd/rest/StoreAPIReadyTest.java | 144 ++++++++++++
.../apache/hugegraph/pd/core/PDCoreSuiteTest.java | 2 +
.../hugegraph/pd/raft/RaftEngineReadinessTest.java | 257 +++++++++++++++++++++
.../org/apache/hugegraph/pd/rest/RestApiTest.java | 57 +++++
.../src/assembly/travis/test-start-hugegraph-pd.sh | 28 +++
hugegraph-store/docs/deployment-guide.md | 20 +-
15 files changed, 879 insertions(+), 6 deletions(-)
diff --git a/docker/README.md b/docker/README.md
index 8ed7ea8fa..6fcfdee42 100644
--- a/docker/README.md
+++ b/docker/README.md
@@ -202,6 +202,32 @@ done
curl -fsS http://localhost:8088/about
```
+PD answers two unauthenticated probe endpoints. `/v1/health` is liveness only:
+it returns `200` as soon as the REST listener is up, even when the PD has no
+raft leader. `/v1/ready` returns `200` only while the PD sees a raft leader,
+and `503` otherwise. Each PD answers for itself: a single PD elects itself, and
+in a three-PD group the two that can reach each other elect a leader and turn
+ready, while a partitioned third keeps answering `503` until it sees that
+leader.
+
+The healthchecks in these files still gate on `/v1/health`, because
+`/v1/ready` ships from the next release onwards while the files run published
+images. Two things to know before pointing them at readiness:
+
+- Match on the body, not the status code. As of 1.7.0 PD answers `200` with
+ `{"status":-1,"error":"Unauthorized!"}` on every path its auth interceptor
+ does not exclude, a path that does not exist included, so a status-only
+ probe reads a PD too old to have `/v1/ready` as ready. The body match holds
+ whichever status a refusal carries. Gate with
+ `curl -fsS http://localhost:8620/v1/ready | grep -q '"ready":true'` instead.
+- Pin `HUGEGRAPH_VERSION` to a release that carries the endpoint, or build the
+ images from source with `docker-compose.dev.yml`.
+
+The `HEALTHCHECK` baked into `hugegraph-pd/Dockerfile` is `/v1/health` as well.
+Both compose files override it, so it governs `docker run` and anything else
+inheriting the image probe, and those keep reading a PD without a quorum as
+healthy.
+
Open `http://localhost:8088` and sign in as `admin` with the password from
`.env`.
diff --git a/hugegraph-pd/README.md b/hugegraph-pd/README.md
index 794dba9b9..c940dae4c 100644
--- a/hugegraph-pd/README.md
+++ b/hugegraph-pd/README.md
@@ -284,6 +284,8 @@ docker/docker-compose-3pd-3store-3server.yml
PD exposes metrics via REST API at:
- Health check: `http://<pd-host>:8620/actuator/health`
+- Liveness: `http://<pd-host>:8620/v1/health` (REST listener is up)
+- Readiness: `http://<pd-host>:8620/v1/ready` (`200` only while the PD sees a
raft leader)
- Metrics: `http://<pd-host>:8620/actuator/metrics`
## Community
diff --git a/hugegraph-pd/docs/api-reference.md
b/hugegraph-pd/docs/api-reference.md
index aa8cce847..58b969886 100644
--- a/hugegraph-pd/docs/api-reference.md
+++ b/hugegraph-pd/docs/api-reference.md
@@ -774,6 +774,56 @@ curl http://localhost:8620/actuator/health
}
```
+### Liveness and Readiness
+
+Two unauthenticated endpoints are meant for probes and startup gates:
+
+| Endpoint | Meaning | Status |
+|----------|---------|--------|
+| `GET /v1/health` | Liveness: the REST listener is up. Does not consult raft.
| always `200` |
+| `GET /v1/ready` | Readiness: the raft node is active and sees a leader, so
this PD is inside a quorum. | `200` when ready, `503` otherwise |
+
+```bash
+curl -i http://localhost:8620/v1/ready
+```
+
+**Response** (leader of a healthy cluster):
+```json
+{
+ "ready": true,
+ "state": "STATE_LEADER",
+ "isLeader": true
+}
+```
+
+A follower reports `"state": "STATE_FOLLOWER"` with `"isLeader": false`. When
+the quorum is lost the PD keeps answering `/v1/health` with `200` but
+`/v1/ready` turns into `503` with `"ready": false`. Being unauthenticated, the
+body carries no cluster addresses; the leader's address stays on `/v1/members`.
+
+The answer is served from state the raft callbacks maintain rather than from
+the raft node, so it stays prompt while an election is running and never waits
+on the node lock. `state` is therefore the last change raft announced. A PD
+reports `STATE_UNINITIALIZED` with `"ready": false` from process start until
+its first raft callback, which is the ordinary startup window before a quorum
+first forms, and jraft emits no callback for candidacy or leadership transfer,
+so a candidate reports `STATE_FOLLOWER` with `"ready": false`.
+
+Point Kubernetes readiness probes, `depends_on` healthchecks and any
+"wait for PD" script at `/v1/ready`; keep liveness probes on `/v1/health`
+so a PD that merely lost its leader is not restarted.
+
+Match on the body rather than on the status code alone. A PD that predates this
+endpoint does not reliably answer `404` for it: `RestAuthentication` refuses a
+request it does not exclude by writing an error envelope, and as of 1.7.0 it
+does so without setting a status, so an unknown path answers `200` with
+`{"status":-1,"error":"Unauthorized!"}`. A status-only probe therefore reads
+such a PD as ready. The body match holds whichever status a refusal carries: a
+shell gate should use
+`curl -fsS http://<pd-host>:8620/v1/ready | grep -q '"ready":true'`, and a
+Kubernetes `httpGet` probe should be paired with a PD image that carries the
+endpoint.
+
### Metrics
```bash
@@ -796,6 +846,30 @@ pd_store_count{state="Offline"} 0.0
pd_partition_count 36.0
```
+#### Raft membership gauges
+
+Exported on `/actuator/prometheus` for alerting on quorum loss:
+
+| Gauge | Value |
+|-------|-------|
+| `hg_raft_leader` | `1` on the raft leader, `0` elsewhere |
+| `hg_raft_has_leader` | `1` while this PD sees a leader (is inside a quorum),
`0` otherwise |
+| `hg_raft_alive_peers` | Number of alive peers on the leader, itself
included; `NaN` elsewhere |
+
+`hg_raft_alive_peers` counts the peers the leader has heard from within the
+leader lease timeout, which jraft derives as 90% of the election timeout by
+default.
+
+A cluster has lost its quorum when `sum(hg_raft_leader) == 0` or when
+`hg_raft_has_leader == 0` on every member. Both are briefly true during a
+normal election, so alert on them with a `for:` clause longer than the
+election timeout rather than on the instantaneous value.
+
+Do not aggregate `hg_raft_alive_peers` across instances: it is `NaN` on every
+node but the leader, and one `NaN` sample turns the result of `sum()` or
+`avg()` into `NaN` as well. Select the leader's series instead, for example
+`hg_raft_alive_peers and on(instance) (hg_raft_leader == 1)`.
+
### Partition API
#### List Partitions
diff --git
a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftEngine.java
b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftEngine.java
index 314c9e57e..d39384909 100644
---
a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftEngine.java
+++
b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftEngine.java
@@ -27,6 +27,8 @@ import java.util.Set;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicReference;
@@ -46,6 +48,7 @@ import com.alipay.sofa.jraft.RaftGroupService;
import com.alipay.sofa.jraft.Status;
import com.alipay.sofa.jraft.conf.Configuration;
import com.alipay.sofa.jraft.core.Replicator;
+import com.alipay.sofa.jraft.core.State;
import com.alipay.sofa.jraft.entity.PeerId;
import com.alipay.sofa.jraft.entity.Task;
import com.alipay.sofa.jraft.error.RaftError;
@@ -63,14 +66,24 @@ import lombok.extern.slf4j.Slf4j;
@Slf4j
public class RaftEngine {
+ /**
+ * Refresh period of the alive peer count behind {@code
hg_raft_alive_peers}. jraft's own
+ * step-down timer walks the same peer set every half election timeout, so
a one second
+ * poll adds nothing next to the work the node already does, and it keeps
the gauge fresh
+ * well inside any scrape interval.
+ */
+ private static final long ALIVE_PEERS_REFRESH_MS = 1000L;
+
private volatile static RaftEngine instance = new RaftEngine();
private RaftStateMachine stateMachine;
private String groupId = "pd_raft";
private PDConfig.Raft config;
private RaftGroupService raftGroupService;
private RpcServer rpcServer;
- private Node raftNode;
+ private volatile Node raftNode;
private RaftRpcClient raftRpcClient;
+ private volatile int alivePeerCount = -1;
+ private ScheduledExecutorService alivePeersRefresher;
public RaftEngine() {
this.stateMachine = new RaftStateMachine();
@@ -133,6 +146,7 @@ public class RaftEngine {
this.raftGroupService =
new RaftGroupService(groupId, serverId, nodeOptions,
rpcServer, true);
this.raftNode = raftGroupService.start(false);
+ startAlivePeersRefresher();
log.info("RaftEngine start successfully: id = {}, peers list = {}",
groupId,
nodeOptions.getInitialConf().getPeers());
return this.raftNode != null;
@@ -182,6 +196,24 @@ public class RaftEngine {
}
public void shutDown() {
+ if (this.alivePeersRefresher != null) {
+ this.alivePeersRefresher.shutdownNow();
+ try {
+ // Best effort: shutdownNow only interrupts, and a refresh
parked in
+ // listAlivePeers waits on a lock acquire the interrupt does
not break,
+ // for up to the raft rpc connect timeout per unreachable
peer. A refresh
+ // that outlives this wait may publish one stale count over
the reset
+ // below; acceptable while shutDown has no production caller.
+ if (!this.alivePeersRefresher.awaitTermination(1,
TimeUnit.SECONDS)) {
+ log.warn("Raft alive-peers refresher still running after
shutdown; " +
+ "hg_raft_alive_peers may briefly report a stale
value");
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ this.alivePeersRefresher = null;
+ }
+ this.alivePeerCount = -1;
if (this.raftGroupService != null) {
this.raftGroupService.shutdown();
try {
@@ -203,7 +235,139 @@ public class RaftEngine {
}
public boolean isLeader() {
- return this.raftNode.isLeader(true);
+ Node node = this.raftNode;
+ return node != null && node.isLeader(true);
+ }
+
+ /**
+ * Whether this PD is ready in the sense {@code GET /v1/ready} answers:
the raft node
+ * is active and sees a leader.
+ * <p>
+ * A follower only keeps its leader while heartbeats keep arriving inside
the election
+ * timeout, and a leader only keeps its role while it can reach a quorum.
Seeing a leader
+ * therefore means this node is part of a quorum from its own point of
view, which is the
+ * signal a readiness probe needs. Served from the state machine
callbacks, not from the
+ * raft node, so it never waits on the node lock.
+ * <p>
+ * Derived from {@link #getRaftStatus()} so {@code hg_raft_has_leader}
cannot drift from
+ * the endpoint it is documented to mirror.
+ */
+ public boolean hasLeader() {
+ return getRaftStatus().isReady();
+ }
+
+ /**
+ * Take a view of the local raft state from the volatile copies the state
machine
+ * callbacks maintain, never from the raft node itself. During an election
jraft holds
+ * the node lock while it reconnects to peers, so a probe that read the
node stalled for
+ * the connect timeout instead of answering its 503 promptly. All fields
derive from one
+ * read of one immutable view, swapped in whole per callback, so they
cannot contradict
+ * each other.
+ * <p>
+ * The state reported is the last one a callback announced: leader,
follower, error or
+ * shutdown, and {@code STATE_UNINITIALIZED} until the first callback
runs. jraft emits
+ * no callback for candidacy or leadership transfer, so a candidate reads
as a follower
+ * that sees no leader, which yields the same not-ready answer. The view
trails the node
+ * by whatever sits in the FSM queue ahead of the announcement, which is
the price of
+ * never waiting on the node lock.
+ * <p>
+ * A missing raft node needs no separate branch: before {@link #init} the
view still
+ * holds its initial {@code STATE_UNINITIALIZED}, and {@code shutDown()}
drops the node
+ * only after {@code join()} has let the shutdown callback announce
+ * {@code STATE_SHUTDOWN}. Neither state is active, so neither reads as
ready, and a
+ * branch on the node would report an uninitialized PD where the callback
said error or
+ * shutdown.
+ */
+ public RaftStatus getRaftStatus() {
+ RaftStateMachine.ProbeView view = this.stateMachine.getProbeView();
+ return new RaftStatus(view.state.isActive() && view.seesLeader,
+ view.state.name(), State.STATE_LEADER ==
view.state);
+ }
+
+ /**
+ * Immutable view of the raft state behind {@code GET /v1/ready}. It
carries no cluster
+ * addresses: the endpoint is unauthenticated, and the leader's address
stays on the
+ * authenticated {@code /v1/members}.
+ */
+ public static final class RaftStatus {
+
+ private final boolean ready;
+ private final String state;
+ private final boolean localLeader;
+
+ RaftStatus(boolean ready, String state, boolean localLeader) {
+ this.ready = ready;
+ this.state = state;
+ this.localLeader = localLeader;
+ }
+
+ public boolean isReady() {
+ return this.ready;
+ }
+
+ /**
+ * @return the jraft node state name, never null
+ */
+ public String getState() {
+ return this.state;
+ }
+
+ public boolean isLocalLeader() {
+ return this.localLeader;
+ }
+ }
+
+ /**
+ * Number of raft peers, this node included, that the leader has heard
from within the
+ * leader lease timeout, which jraft derives as 90% of the election
timeout by default.
+ * Only the leader tracks replication state, so any other node reports -1.
+ * <p>
+ * This is the value the refresher last published, at most one refresh
period old, and -1
+ * until the first refresh runs. The count cannot be read here: {@code
listAlivePeers}
+ * takes the node read lock before it checks for leadership, so a caller
that read the
+ * node would wait out whoever holds the write lock, which during an
election is a
+ * per-peer reconnect bounded only by the rpc connect timeout.
+ */
+ public int getAlivePeerCount() {
+ return this.alivePeerCount;
+ }
+
+ private void startAlivePeersRefresher() {
+ ScheduledExecutorService refresher =
+ Executors.newSingleThreadScheduledExecutor(runnable -> {
+ Thread thread = new Thread(runnable,
"pd-raft-alive-peers");
+ thread.setDaemon(true);
+ return thread;
+ });
+ refresher.scheduleWithFixedDelay(this::refreshAlivePeerCount, 0,
+ ALIVE_PEERS_REFRESH_MS,
TimeUnit.MILLISECONDS);
+ this.alivePeersRefresher = refresher;
+ }
+
+ /**
+ * Read the alive peer count from the raft node and publish it for the
gauge. Runs on the
+ * refresher thread, never on a request thread, because the read can block
for as long as
+ * an election holds the node write lock. A fixed delay schedule means a
blocked refresh
+ * only delays the next one, and the gauge keeps reporting the last value
meanwhile.
+ */
+ void refreshAlivePeerCount() {
+ Node node = this.raftNode;
+ if (node == null || !this.stateMachine.isLeader()) {
+ this.alivePeerCount = -1;
+ return;
+ }
+ try {
+ this.alivePeerCount = node.listAlivePeers().size();
+ } catch (IllegalStateException e) {
+ // Lost leadership between the check and the call
+ this.alivePeerCount = -1;
+ } catch (Throwable e) {
+ // scheduleWithFixedDelay cancels every later run if the task
throws,
+ // and an Error escaping here would leave the gauge serving its
last
+ // value forever, so this catch has to be wider than Exception.
+ log.warn("Failed to refresh the raft alive peer count", e);
+ this.alivePeerCount = -1;
+ }
}
/**
@@ -232,7 +396,8 @@ public class RaftEngine {
}
public PeerId getLeader() {
- return raftNode.getLeaderId();
+ Node node = this.raftNode;
+ return node == null ? null : node.getLeaderId();
}
/**
diff --git
a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftStateMachine.java
b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftStateMachine.java
index 6fad3347f..aab90e533 100644
---
a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftStateMachine.java
+++
b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftStateMachine.java
@@ -36,6 +36,7 @@ import com.alipay.sofa.jraft.Closure;
import com.alipay.sofa.jraft.Iterator;
import com.alipay.sofa.jraft.Status;
import com.alipay.sofa.jraft.conf.Configuration;
+import com.alipay.sofa.jraft.core.State;
import com.alipay.sofa.jraft.core.StateMachineAdapter;
import com.alipay.sofa.jraft.entity.LeaderChangeContext;
import com.alipay.sofa.jraft.entity.LocalFileMetaOutter;
@@ -56,6 +57,27 @@ public class RaftStateMachine extends StateMachineAdapter {
private static final String SNAPSHOT_DIR_NAME = "snapshot";
private static final String SNAPSHOT_ARCHIVE_NAME = "snapshot.zip";
private final AtomicLong leaderTerm = new AtomicLong(-1);
+ // Kept for the readiness probe and the raft gauges, which must answer
without
+ // touching the raft node: during an election jraft holds the node lock
while it
+ // reconnects to peers, and a reader stalls for the connect timeout. One
volatile
+ // immutable pair, swapped in whole per callback, so a reader can never
see the
+ // state of one announcement with the leader visibility of another.
+ private volatile ProbeView probeView = new
ProbeView(State.STATE_UNINITIALIZED, false);
+
+ /**
+ * Immutable pair of the last announced node state and leader visibility.
Written only
+ * from the FSM callback thread; both fields always come from the same
announcement.
+ */
+ static final class ProbeView {
+
+ final State state;
+ final boolean seesLeader;
+
+ ProbeView(State state, boolean seesLeader) {
+ this.state = state;
+ this.seesLeader = seesLeader;
+ }
+ }
private List<RaftTaskHandler> taskHandlers;
private List<RaftStateListener> stateListeners;
@@ -76,6 +98,15 @@ public class RaftStateMachine extends StateMachineAdapter {
return this.leaderTerm.get() > 0;
}
+ /**
+ * The last state and leader visibility a raft callback announced, as one
immutable
+ * view. jraft emits no callback for candidacy or leadership transfer, so
a candidate
+ * reads as a follower that sees no leader.
+ */
+ ProbeView getProbeView() {
+ return this.probeView;
+ }
+
@Override
public void onApply(Iterator iter) {
while (iter.hasNext()) {
@@ -105,17 +136,24 @@ public class RaftStateMachine extends StateMachineAdapter
{
@Override
public void onError(final RaftException e) {
+ // A node that errors after it was leader keeps a positive leaderTerm
otherwise, and
+ // the alive-peer refresher would go on reading listAlivePeers off a
broken node
+ this.leaderTerm.set(-1);
+ this.probeView = new ProbeView(State.STATE_ERROR, false);
log.error("Raft StateMachine on error {}", e);
}
@Override
public void onShutdown() {
+ this.leaderTerm.set(-1);
+ this.probeView = new ProbeView(State.STATE_SHUTDOWN, false);
super.onShutdown();
}
@Override
public void onLeaderStart(final long term) {
this.leaderTerm.set(term);
+ this.probeView = new ProbeView(State.STATE_LEADER, true);
super.onLeaderStart(term);
log.info("Raft becomes leader");
@@ -129,12 +167,14 @@ public class RaftStateMachine extends StateMachineAdapter
{
@Override
public void onLeaderStop(final Status status) {
this.leaderTerm.set(-1);
+ this.probeView = new ProbeView(State.STATE_FOLLOWER, false);
super.onLeaderStop(status);
log.info("Raft lost leader ");
}
@Override
public void onStartFollowing(final LeaderChangeContext ctx) {
+ this.probeView = new ProbeView(State.STATE_FOLLOWER, true);
super.onStartFollowing(ctx);
Utils.runInThread(() -> {
if (!CollectionUtils.isEmpty(stateListeners)) {
@@ -145,6 +185,8 @@ public class RaftStateMachine extends StateMachineAdapter {
@Override
public void onStopFollowing(final LeaderChangeContext ctx) {
+ // Callbacks run on the single FSM thread, so reading our own field is
safe
+ this.probeView = new ProbeView(this.probeView.state, false);
super.onStopFollowing(ctx);
}
diff --git a/hugegraph-pd/hg-pd-service/pom.xml
b/hugegraph-pd/hg-pd-service/pom.xml
index ee78863f3..eef38b407 100644
--- a/hugegraph-pd/hg-pd-service/pom.xml
+++ b/hugegraph-pd/hg-pd-service/pom.xml
@@ -162,6 +162,12 @@
<artifactId>log4j-jul</artifactId>
<version>2.17.2</version>
</dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>${junit.version}</version>
+ <scope>test</scope>
+ </dependency>
</dependencies>
<build>
<plugins>
diff --git
a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/metrics/PDMetrics.java
b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/metrics/PDMetrics.java
index 483974a01..793aa24b0 100644
---
a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/metrics/PDMetrics.java
+++
b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/metrics/PDMetrics.java
@@ -33,6 +33,7 @@ import org.apache.hugegraph.pd.common.PDException;
import org.apache.hugegraph.pd.grpc.Metapb;
import org.apache.hugegraph.pd.grpc.Metapb.ShardGroup;
import org.apache.hugegraph.pd.model.GraphStatistics;
+import org.apache.hugegraph.pd.raft.RaftEngine;
import org.apache.hugegraph.pd.service.PDRestService;
import org.apache.hugegraph.pd.service.PDService;
@@ -76,7 +77,28 @@ public final class PDMetrics {
Gauge.builder(PREFIX + ".terms", () -> setTerms())
.description("term of partitions in PD")
.register(registry);
+ registerRaftMeters();
+ }
+ /**
+ * Raft membership gauges so operators can alert on quorum loss. They
mirror what
+ * {@code GET /v1/ready} answers: a PD that sees no leader is outside a
quorum.
+ */
+ private void registerRaftMeters() {
+ RaftEngine raft = RaftEngine.getInstance();
+ Gauge.builder(PREFIX + ".raft.leader", () ->
raft.getRaftStatus().isLocalLeader() ? 1 : 0)
+ .description("1 if this PD is the raft leader, 0 otherwise")
+ .register(registry);
+ Gauge.builder(PREFIX + ".raft.has.leader", () -> raft.hasLeader() ? 1
: 0)
+ .description("1 if this PD sees a raft leader, i.e. is part of a
quorum, 0 otherwise")
+ .register(registry);
+ Gauge.builder(PREFIX + ".raft.alive.peers", () -> {
+ int alive = raft.getAlivePeerCount();
+ return alive < 0 ? Double.NaN : alive;
+ })
+ .description("Number of raft peers, itself included, the leader
has heard from " +
+ "within the leader lease timeout; NaN on non-leader
nodes")
+ .register(registry);
}
private long updateGraphs() {
diff --git
a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/StoreAPI.java
b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/StoreAPI.java
index 2cddb29fe..b3c30ca9e 100644
---
a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/StoreAPI.java
+++
b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/StoreAPI.java
@@ -22,6 +22,7 @@ import java.util.ArrayList;
import java.util.Date;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
@@ -31,11 +32,14 @@ import org.apache.hugegraph.pd.grpc.Pdpb;
import org.apache.hugegraph.pd.model.RestApiResponse;
import org.apache.hugegraph.pd.model.StoreRestRequest;
import org.apache.hugegraph.pd.model.TimeRangeRequest;
+import org.apache.hugegraph.pd.raft.RaftEngine;
import org.apache.hugegraph.pd.service.PDRestService;
import org.apache.hugegraph.pd.util.DateUtil;
import org.apache.hugegraph.pd.util.StoreRestAddressUtil;
import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
+import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
@@ -378,6 +382,10 @@ public class StoreAPI extends API {
* Check Service Health Status
* This interface is used to check the health status of the service by
accessing the /health
* path via a GET request.
+ * <p>
+ * This is a liveness signal only: it answers 200 as soon as the REST
listener is up and
+ * does not consult the raft state. Use {@link #checkReady()} to find out
whether this PD
+ * can actually serve.
*
* @return Returns a string indicating the service's health status.
Typically, an empty
* string indicates the service is healthy.
@@ -386,4 +394,27 @@ public class StoreAPI extends API {
public Serializable checkHealthy() {
return "";
}
+
+ /**
+ * Check Service Readiness
+ * Answers 200 only when this PD is part of a raft quorum, that is, the
raft node is active
+ * and knows the current leader. Otherwise answers 503 so that anything
gating on PD
+ * (the compose healthcheck in front of Stores, a Kubernetes readiness
probe)
+ * is held back until the PD can serve. Like /health this endpoint needs
no authentication.
+ *
+ * @return JSON with the readiness flag, the local raft state and whether
this node is the
+ * leader. It carries no cluster addresses, since the endpoint is
unauthenticated.
+ */
+ @GetMapping(value = "/ready", produces = MediaType.APPLICATION_JSON_VALUE)
+ public ResponseEntity<Map<String, Object>> checkReady() {
+ RaftEngine.RaftStatus status =
RaftEngine.getInstance().getRaftStatus();
+
+ Map<String, Object> body = new LinkedHashMap<>();
+ body.put("ready", status.isReady());
+ body.put("state", status.getState());
+ body.put("isLeader", status.isLocalLeader());
+ HttpStatus httpStatus = status.isReady() ? HttpStatus.OK
+ :
HttpStatus.SERVICE_UNAVAILABLE;
+ return ResponseEntity.status(httpStatus).body(body);
+ }
}
diff --git
a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/interceptor/AuthenticationConfigurer.java
b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/interceptor/AuthenticationConfigurer.java
index 7d1041696..d4b1e026d 100644
---
a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/interceptor/AuthenticationConfigurer.java
+++
b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/interceptor/AuthenticationConfigurer.java
@@ -32,6 +32,7 @@ public class AuthenticationConfigurer implements
WebMvcConfigurer {
public void addInterceptors(InterceptorRegistry registry) {
registry.addInterceptor(restAuthentication)
.addPathPatterns("/**")
- .excludePathPatterns("/actuator/*", "/v1/health",
"/v1/prom/targets/*");
+ .excludePathPatterns("/actuator/*", "/v1/health", "/v1/ready",
+ "/v1/prom/targets/*");
}
}
diff --git
a/hugegraph-pd/hg-pd-service/src/test/java/org/apache/hugegraph/pd/rest/StoreAPIReadyTest.java
b/hugegraph-pd/hg-pd-service/src/test/java/org/apache/hugegraph/pd/rest/StoreAPIReadyTest.java
new file mode 100644
index 000000000..75e9581cb
--- /dev/null
+++
b/hugegraph-pd/hg-pd-service/src/test/java/org/apache/hugegraph/pd/rest/StoreAPIReadyTest.java
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.pd.rest;
+
+import java.lang.reflect.Proxy;
+import java.util.Map;
+
+import org.apache.hugegraph.pd.raft.RaftEngine;
+import org.apache.hugegraph.pd.raft.RaftStateMachine;
+import org.apache.hugegraph.testutil.Whitebox;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.springframework.http.HttpStatus;
+import org.springframework.http.ResponseEntity;
+
+import com.alipay.sofa.jraft.Node;
+import com.alipay.sofa.jraft.Status;
+import com.alipay.sofa.jraft.core.State;
+import com.alipay.sofa.jraft.entity.LeaderChangeContext;
+import com.alipay.sofa.jraft.entity.PeerId;
+
+/**
+ * Pins the status a readiness probe actually receives from {@code GET
/v1/ready}: 503 while
+ * this PD is outside a raft quorum, 200 once it is inside one, with the same
body either way.
+ * The live REST suite can only reach the 200 path, because the PD it talks to
is a single node
+ * group that is always its own leader.
+ */
+public class StoreAPIReadyTest {
+
+ private static final PeerId LEADER = new PeerId("10.0.0.1", 8610);
+
+ private final StoreAPI api = new StoreAPI();
+
+ private Node originalRaftNode;
+ private RaftStateMachine originalStateMachine;
+ private RaftStateMachine stateMachine;
+
+ @Before
+ public void setUp() {
+ RaftEngine engine = RaftEngine.getInstance();
+ this.originalRaftNode = engine.getRaftNode();
+ this.originalStateMachine = Whitebox.getInternalState(engine,
"stateMachine");
+
+ // A fresh machine so callbacks fired here reach no listeners of the
real one
+ this.stateMachine = new RaftStateMachine();
+ Whitebox.setInternalState(engine, "stateMachine", this.stateMachine);
+ }
+
+ @After
+ public void tearDown() {
+ RaftEngine engine = RaftEngine.getInstance();
+ Whitebox.setInternalState(engine, "raftNode", this.originalRaftNode);
+ Whitebox.setInternalState(engine, "stateMachine",
this.originalStateMachine);
+ }
+
+ @Test
+ public void testNotReadyBeforeRaftStartsAnswers503() {
+ Whitebox.setInternalState(RaftEngine.getInstance(), "raftNode", null);
+
+ ResponseEntity<Map<String, Object>> response = this.api.checkReady();
+
+ Assert.assertEquals(HttpStatus.SERVICE_UNAVAILABLE,
response.getStatusCode());
+ Map<String, Object> body = response.getBody();
+ Assert.assertNotNull(body);
+ Assert.assertEquals(Boolean.FALSE, body.get("ready"));
+ Assert.assertEquals(Boolean.FALSE, body.get("isLeader"));
+ Assert.assertEquals(State.STATE_UNINITIALIZED.name(),
body.get("state"));
+ }
+
+ @Test
+ public void testLosingTheLeaderAnswers503() {
+ Whitebox.setInternalState(RaftEngine.getInstance(), "raftNode",
untouchableNode());
+ // The quorum-loss shape: heartbeats arrive, then they stop
+ this.stateMachine.onStartFollowing(context());
+ this.stateMachine.onStopFollowing(context());
+
+ ResponseEntity<Map<String, Object>> response = this.api.checkReady();
+
+ Assert.assertEquals(HttpStatus.SERVICE_UNAVAILABLE,
response.getStatusCode());
+ Map<String, Object> body = response.getBody();
+ Assert.assertNotNull(body);
+ Assert.assertEquals(Boolean.FALSE, body.get("ready"));
+ Assert.assertEquals(Boolean.FALSE, body.get("isLeader"));
+ Assert.assertEquals(State.STATE_FOLLOWER.name(), body.get("state"));
+ }
+
+ @Test
+ public void testLeaderAnswers200() {
+ Whitebox.setInternalState(RaftEngine.getInstance(), "raftNode",
untouchableNode());
+ this.stateMachine.onLeaderStart(5);
+
+ ResponseEntity<Map<String, Object>> response = this.api.checkReady();
+
+ Assert.assertEquals(HttpStatus.OK, response.getStatusCode());
+ Map<String, Object> body = response.getBody();
+ Assert.assertNotNull(body);
+ Assert.assertEquals(Boolean.TRUE, body.get("ready"));
+ Assert.assertEquals(Boolean.TRUE, body.get("isLeader"));
+ Assert.assertEquals(State.STATE_LEADER.name(), body.get("state"));
+ }
+
+ private static LeaderChangeContext context() {
+ return new LeaderChangeContext(LEADER, 5, Status.OK());
+ }
+
+ /**
+ * A started raft node that fails the test if the endpoint calls it.
Readiness is served
+ * from the state machine callbacks precisely so that it never waits on
the node lock.
+ */
+ private static Node untouchableNode() {
+ return (Node) Proxy.newProxyInstance(
+ Node.class.getClassLoader(), new Class<?>[]{Node.class},
+ (proxy, method, args) -> {
+ switch (method.getName()) {
+ case "toString":
+ return "untouchable raft node";
+ case "hashCode":
+ return System.identityHashCode(proxy);
+ case "equals":
+ return proxy == args[0];
+ default:
+ throw new AssertionError("the ready path must not
touch the raft " +
+ "node, it called " +
method.getName());
+ }
+ });
+ }
+}
diff --git
a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDCoreSuiteTest.java
b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDCoreSuiteTest.java
index 95b044c76..bdacf7d37 100644
---
a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDCoreSuiteTest.java
+++
b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDCoreSuiteTest.java
@@ -22,6 +22,7 @@ import org.apache.hugegraph.pd.core.store.HgKVStoreImplTest;
import org.apache.hugegraph.pd.raft.IpAuthHandlerTest;
import org.apache.hugegraph.pd.raft.RaftEngineIpAuthIntegrationTest;
import org.apache.hugegraph.pd.raft.RaftEngineLeaderAddressTest;
+import org.apache.hugegraph.pd.raft.RaftEngineReadinessTest;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
@@ -43,6 +44,7 @@ import lombok.extern.slf4j.Slf4j;
IpAuthHandlerTest.class,
RaftEngineIpAuthIntegrationTest.class,
RaftEngineLeaderAddressTest.class,
+ RaftEngineReadinessTest.class,
// StoreNodeServiceTest.class,
})
@Slf4j
diff --git
a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/raft/RaftEngineReadinessTest.java
b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/raft/RaftEngineReadinessTest.java
new file mode 100644
index 000000000..08c35287d
--- /dev/null
+++
b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/raft/RaftEngineReadinessTest.java
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.pd.raft;
+
+import java.util.Arrays;
+
+import org.apache.hugegraph.testutil.Whitebox;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import com.alipay.sofa.jraft.Node;
+import com.alipay.sofa.jraft.Status;
+import com.alipay.sofa.jraft.core.State;
+import com.alipay.sofa.jraft.entity.LeaderChangeContext;
+import com.alipay.sofa.jraft.entity.PeerId;
+import com.alipay.sofa.jraft.error.RaftException;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoInteractions;
+import static org.mockito.Mockito.when;
+
+/**
+ * Covers the raft-aware readiness signal behind {@code GET /v1/ready} and the
+ * {@code hg.raft.*} gauges. The signal is served from the volatile copies the
state machine
+ * callbacks maintain, never from the raft node, so the probe stays prompt
while an election
+ * holds the node lock; these tests drive the callbacks the way jraft does.
The alive peer
+ * count is the one value that has to come off the node, so it is refreshed
explicitly here,
+ * the way the refresher thread does it in a running PD.
+ */
+public class RaftEngineReadinessTest {
+
+ private static final PeerId LEADER = new PeerId("10.0.0.1", 8610);
+
+ private Node originalRaftNode;
+ private RaftStateMachine originalStateMachine;
+
+ private Node mockNode;
+ private RaftStateMachine stateMachine;
+
+ @Before
+ public void setUp() {
+ RaftEngine engine = RaftEngine.getInstance();
+ originalRaftNode = engine.getRaftNode();
+ originalStateMachine = Whitebox.getInternalState(engine,
"stateMachine");
+
+ // A fresh machine so callbacks fired here reach no listeners other
tests registered
+ mockNode = mock(Node.class);
+ stateMachine = new RaftStateMachine();
+ Whitebox.setInternalState(engine, "raftNode", mockNode);
+ Whitebox.setInternalState(engine, "stateMachine", stateMachine);
+ // The count is cached on the singleton, so clear what an earlier test
published
+ engine.refreshAlivePeerCount();
+ }
+
+ @After
+ public void tearDown() {
+ RaftEngine engine = RaftEngine.getInstance();
+ Whitebox.setInternalState(engine, "raftNode", originalRaftNode);
+ Whitebox.setInternalState(engine, "stateMachine",
originalStateMachine);
+ }
+
+ private static LeaderChangeContext ctx() {
+ return new LeaderChangeContext(LEADER, 5, Status.OK());
+ }
+
+ @Test
+ public void testNotReadyBeforeRaftNodeStarts() {
+ Whitebox.setInternalState(RaftEngine.getInstance(), "raftNode", null);
+ RaftEngine engine = RaftEngine.getInstance();
+ engine.refreshAlivePeerCount();
+
+ RaftEngine.RaftStatus status = engine.getRaftStatus();
+ Assert.assertFalse(status.isReady());
+ Assert.assertFalse(status.isLocalLeader());
+ Assert.assertEquals(State.STATE_UNINITIALIZED.name(),
status.getState());
+ Assert.assertFalse(engine.hasLeader());
+ Assert.assertEquals(-1, engine.getAlivePeerCount());
+ }
+
+ @Test
+ public void testStartedNodeWithoutAnyCallbackIsNotReady() {
+ RaftEngine.RaftStatus status =
RaftEngine.getInstance().getRaftStatus();
+ Assert.assertFalse(status.isReady());
+ Assert.assertEquals(State.STATE_UNINITIALIZED.name(),
status.getState());
+ }
+
+ @Test
+ public void testLeaderIsReady() {
+ stateMachine.onLeaderStart(5);
+ when(mockNode.listAlivePeers()).thenReturn(Arrays.asList(LEADER, new
PeerId("b", 1),
+ new
PeerId("c", 1)));
+ RaftEngine engine = RaftEngine.getInstance();
+ engine.refreshAlivePeerCount();
+
+ RaftEngine.RaftStatus status = engine.getRaftStatus();
+ Assert.assertTrue(status.isReady());
+ Assert.assertTrue(status.isLocalLeader());
+ Assert.assertEquals(State.STATE_LEADER.name(), status.getState());
+ Assert.assertTrue(engine.hasLeader());
+ Assert.assertEquals(3, engine.getAlivePeerCount());
+ }
+
+ @Test
+ public void testFollowerWithLeaderIsReady() {
+ stateMachine.onStartFollowing(ctx());
+ RaftEngine engine = RaftEngine.getInstance();
+ engine.refreshAlivePeerCount();
+
+ RaftEngine.RaftStatus status = engine.getRaftStatus();
+ Assert.assertTrue(status.isReady());
+ Assert.assertFalse(status.isLocalLeader());
+ Assert.assertEquals(State.STATE_FOLLOWER.name(), status.getState());
+ Assert.assertTrue(engine.hasLeader());
+ // Only the leader tracks replication, followers cannot count alive
peers
+ Assert.assertEquals(-1, engine.getAlivePeerCount());
+ }
+
+ @Test
+ public void testFollowerLosingItsLeaderTurnsNotReady() {
+ // The quorum-loss shape from the issue: heartbeats stop, jraft
announces the loss
+ stateMachine.onStartFollowing(ctx());
+ stateMachine.onStopFollowing(ctx());
+ RaftEngine engine = RaftEngine.getInstance();
+
+ RaftEngine.RaftStatus status = engine.getRaftStatus();
+ Assert.assertFalse(status.isReady());
+ Assert.assertEquals(State.STATE_FOLLOWER.name(), status.getState());
+ Assert.assertFalse(engine.hasLeader());
+ }
+
+ @Test
+ public void testLeaderSteppingDownTurnsNotReady() {
+ stateMachine.onLeaderStart(5);
+ stateMachine.onLeaderStop(Status.OK());
+ RaftEngine engine = RaftEngine.getInstance();
+ engine.refreshAlivePeerCount();
+
+ RaftEngine.RaftStatus status = engine.getRaftStatus();
+ Assert.assertFalse(status.isReady());
+ Assert.assertFalse(status.isLocalLeader());
+ Assert.assertFalse(engine.hasLeader());
+ Assert.assertEquals(-1, engine.getAlivePeerCount());
+ }
+
+ @Test
+ public void testErrorAndShutdownAreNotReadyEvenAfterLeadership() {
+ stateMachine.onLeaderStart(5);
+ stateMachine.onError(mock(RaftException.class));
+ Assert.assertFalse(RaftEngine.getInstance().getRaftStatus().isReady());
+ Assert.assertEquals(State.STATE_ERROR.name(),
+
RaftEngine.getInstance().getRaftStatus().getState());
+
+ stateMachine.onShutdown();
+ Assert.assertFalse(RaftEngine.getInstance().getRaftStatus().isReady());
+ Assert.assertEquals(State.STATE_SHUTDOWN.name(),
+
RaftEngine.getInstance().getRaftStatus().getState());
+ }
+
+ @Test
+ public void testProbeNeverTouchesTheRaftNode() {
+ // The point of serving from callbacks: an election holds the node
lock while jraft
+ // reconnects to peers, so the probe and the gauges must answer
without the node.
+ // getAlivePeerCount() included: it reads what the refresher thread
published
+ stateMachine.onStartFollowing(ctx());
+ stateMachine.onStopFollowing(ctx());
+ RaftEngine engine = RaftEngine.getInstance();
+
+ engine.getRaftStatus();
+ engine.hasLeader();
+ engine.getAlivePeerCount();
+
+ verifyNoInteractions(mockNode);
+ }
+
+ @Test
+ public void testAlivePeerCountSurvivesLeadershipLossRace() {
+ stateMachine.onLeaderStart(5);
+ when(mockNode.listAlivePeers()).thenThrow(new
IllegalStateException("Not leader"));
+ RaftEngine.getInstance().refreshAlivePeerCount();
+
+ Assert.assertEquals(-1, RaftEngine.getInstance().getAlivePeerCount());
+ }
+
+ @Test
+ public void testShutdownStateSurvivesTheNodeBeingDropped() {
+ // shutDown() drops the raft node after the shutdown callback ran, so
the probe has
+ // to keep reporting what the callback announced rather than falling
back to
+ // uninitialized, which would read as a PD that has not started yet
+ stateMachine.onLeaderStart(5);
+ stateMachine.onShutdown();
+ Whitebox.setInternalState(RaftEngine.getInstance(), "raftNode", null);
+ RaftEngine engine = RaftEngine.getInstance();
+
+ RaftEngine.RaftStatus status = engine.getRaftStatus();
+ Assert.assertFalse(status.isReady());
+ Assert.assertFalse(status.isLocalLeader());
+ Assert.assertEquals(State.STATE_SHUTDOWN.name(), status.getState());
+ Assert.assertFalse(engine.hasLeader());
+ }
+
+ @Test
+ public void testAlivePeerCountStopsAfterALeaderErrorsOrShutsDown() {
+ // The terminal callbacks clear the leader term, so the refresher
stops reading the
+ // count off a node that errored or shut down and the gauge falls back
to NaN
+ stateMachine.onLeaderStart(5);
+ when(mockNode.listAlivePeers()).thenReturn(Arrays.asList(LEADER, new
PeerId("b", 1)));
+ RaftEngine engine = RaftEngine.getInstance();
+ engine.refreshAlivePeerCount();
+ Assert.assertEquals(2, engine.getAlivePeerCount());
+
+ stateMachine.onError(mock(RaftException.class));
+ engine.refreshAlivePeerCount();
+ Assert.assertEquals(-1, engine.getAlivePeerCount());
+
+ stateMachine.onLeaderStart(6);
+ engine.refreshAlivePeerCount();
+ Assert.assertEquals(2, engine.getAlivePeerCount());
+
+ stateMachine.onShutdown();
+ engine.refreshAlivePeerCount();
+ Assert.assertEquals(-1, engine.getAlivePeerCount());
+ // Twice, once per leader phase: the terminal states never reached the
node
+ verify(mockNode, times(2)).listAlivePeers();
+ }
+
+ @Test
+ public void testAlivePeerCountOnALeaderAlsoSkipsTheNode() {
+ // Even on a leader, where the count means something, the gauge reads
the published
+ // value: listAlivePeers takes the node read lock before it checks for
leadership, so
+ // a scrape that called it would wait out an election that holds the
write lock
+ stateMachine.onLeaderStart(5);
+ RaftEngine engine = RaftEngine.getInstance();
+
+ engine.getAlivePeerCount();
+
+ verifyNoInteractions(mockNode);
+ }
+}
diff --git
a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java
b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java
index fb2b71d48..da90f6f0f 100644
---
a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java
+++
b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java
@@ -25,6 +25,7 @@ import java.net.http.HttpResponse;
import org.json.JSONException;
import org.json.JSONObject;
+import org.junit.Assert;
import org.junit.Test;
public class RestApiTest extends BaseServerTest {
@@ -62,6 +63,62 @@ public class RestApiTest extends BaseServerTest {
assert obj.getInt("status") == 0;
}
+ @Test
+ public void testHealthNeedsNoAuth() throws URISyntaxException, IOException,
+ InterruptedException {
+ String url = pdRestAddr + "/v1/health";
+ HttpRequest request = HttpRequest.newBuilder().uri(new
URI(url)).GET().build();
+ HttpResponse<String> response = client.send(request,
HttpResponse.BodyHandlers.ofString());
+ Assert.assertEquals(200, response.statusCode());
+ // A 200 alone does not prove the path is anonymous: as of 1.7.0 the
auth interceptor
+ // refuses with 200 and an error envelope. checkHealthy() returns an
empty body, which
+ // separates the two whichever status a refusal carries.
+ Assert.assertTrue("expected an empty body, got " + response.body(),
+ response.body().isEmpty());
+ }
+
+ @Test
+ public void testReadyNeedsNoAuthAndReflectsRaft() throws
URISyntaxException, IOException,
+
InterruptedException, JSONException {
+ // The CI PD is a single-node raft group, so it is its own leader and
must be ready
+ String url = pdRestAddr + "/v1/ready";
+ HttpRequest request = HttpRequest.newBuilder().uri(new
URI(url)).GET().build();
+ HttpResponse<String> response = client.send(request,
HttpResponse.BodyHandlers.ofString());
+ Assert.assertEquals("expected 200, body=" + response.body(), 200,
response.statusCode());
+ JSONObject obj = new JSONObject(response.body());
+ Assert.assertTrue(obj.getBoolean("ready"));
+ Assert.assertTrue(obj.getBoolean("isLeader"));
+ Assert.assertEquals("STATE_LEADER", obj.getString("state"));
+ // Unauthenticated, so it must not disclose cluster addresses
+ Assert.assertFalse("the anonymous body must not carry the leader
address",
+ obj.has("leader"));
+ }
+
+ @Test
+ public void testRaftGaugesExported() throws URISyntaxException,
IOException,
+ InterruptedException {
+ String url = pdRestAddr + "/actuator/prometheus";
+ HttpRequest request = HttpRequest.newBuilder().uri(new
URI(url)).GET().build();
+ HttpResponse<String> response = client.send(request,
HttpResponse.BodyHandlers.ofString());
+ Assert.assertEquals(200, response.statusCode());
+ String body = response.body();
+ // Micrometer only writes a {...} block when the meter carries tags,
and the sample
+ // line is the one that starts with the metric name, unlike its HELP
and TYPE lines
+ Assert.assertTrue("missing hg_raft_leader gauge",
+ body.matches("(?sm).*^hg_raft_leader(\\{[^}]*\\})?
.*"));
+ Assert.assertTrue("missing hg_raft_has_leader gauge",
+
body.matches("(?sm).*^hg_raft_has_leader(\\{[^}]*\\})? .*"));
+ Assert.assertTrue("missing hg_raft_alive_peers gauge",
+
body.matches("(?sm).*^hg_raft_alive_peers(\\{[^}]*\\})? .*"));
+ // Single-node CI cluster: this PD is the leader and hears from itself
+ Assert.assertTrue("hg_raft_leader should be 1 on a single-node leader",
+ body.matches("(?sm).*^hg_raft_leader(\\{[^}]*\\})?
1\\.0.*"));
+ Assert.assertTrue("hg_raft_has_leader should be 1 on a single-node
leader",
+
body.matches("(?sm).*^hg_raft_has_leader(\\{[^}]*\\})? 1\\.0.*"));
+ Assert.assertTrue("hg_raft_alive_peers should be 1 on a single-node
leader",
+
body.matches("(?sm).*^hg_raft_alive_peers(\\{[^}]*\\})? 1\\.0.*"));
+ }
+
@Test
public void testQueryClusterMembers() throws URISyntaxException,
IOException,
InterruptedException,
JSONException {
diff --git
a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-start-hugegraph-pd.sh
b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-start-hugegraph-pd.sh
index ab73255b8..754e2baae 100755
---
a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-start-hugegraph-pd.sh
+++
b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-start-hugegraph-pd.sh
@@ -111,6 +111,27 @@ wait_for_pd() {
return 1
}
+# Wait until PD readiness answers, or timeout. Same gate the docs recommend:
+# -f rejects the 503, the body match rejects a 200 that is an auth envelope
rather
+# than a readiness answer. No credentials, and a single-node group elects
itself.
+# Captured rather than piped: under pipefail a grep -q SIGPIPE could misread a
+# ready PD, and wait_for_pd() above uses the same shape. The curl timeouts
bound
+# each attempt and the loop bounds itself on the wall clock, so a PD that
accepts
+# the connection and then stops answering cannot park this loop past
STARTUP_WAIT.
+# A per-round counter could not: an attempt costs up to --max-time plus the
sleep,
+# which is more than the counter would add.
+wait_for_pd_ready() {
+ local deadline=$((SECONDS + STARTUP_WAIT))
+ while (( SECONDS < deadline )); do
+ local body
+ body=$(curl --connect-timeout 2 --max-time 5 -fsS \
+ "$PD_URL/v1/ready" 2>/dev/null || true)
+ grep -Eq '"ready"[[:space:]]*:[[:space:]]*true' <<<"$body" && return 0
+ sleep 2
+ done
+ return 1
+}
+
# Wait until bin/pid is non-empty or timeout
wait_for_pid_file() {
local elapsed=0
@@ -199,6 +220,13 @@ else
fail "PD health endpoint not responding after ${STARTUP_WAIT}s"
fi
+info "Waiting up to ${STARTUP_WAIT}s for PD readiness endpoint..."
+if wait_for_pd_ready; then
+ pass "PD readiness endpoint reports a raft leader at $PD_URL/v1/ready"
+else
+ fail "PD readiness endpoint did not report ready=true after
${STARTUP_WAIT}s"
+fi
+
cleanup
# ── test 2: foreground mode blocks ───────────────────────────────────────────
diff --git a/hugegraph-store/docs/deployment-guide.md
b/hugegraph-store/docs/deployment-guide.md
index de07904d6..40c70b0e0 100644
--- a/hugegraph-store/docs/deployment-guide.md
+++ b/hugegraph-store/docs/deployment-guide.md
@@ -719,10 +719,15 @@ environment:
```
**Startup ordering** is enforced via `depends_on` with `condition:
service_healthy`:
-1. PD nodes start first and must pass healthchecks (`/v1/health`)
+1. PD nodes start first and must pass healthchecks (`/v1/health`, liveness
only)
2. Store nodes start after all PD nodes are healthy
3. Server nodes start after all Store nodes are healthy
+`/v1/health` answers `200` as soon as the PD REST listener is up, so step 1
does
+not wait for a raft quorum to form. PD also serves `/v1/ready`, which answers
+`200` only while the PD sees a raft leader; `docker/README.md` covers what
+pointing the healthchecks at it requires.
+
> **Note**: The deprecated env var names (`GRPC_HOST`, `RAFT_ADDRESS`,
> `RAFT_PEERS`, `PD_ADDRESS`, `BACKEND`, `PD_PEERS`) still work but log a
> warning. Use the `HG_*` prefixed names for new deployments.
**Deploy**:
@@ -855,13 +860,24 @@ kubectl port-forward svc/hugegraph-store 8500:8500 -n
hugegraph
### Health Check
```bash
-# PD health
+# PD liveness (REST listener up)
curl http://192.168.1.10:8620/v1/health
+# PD readiness (200 only while the PD sees a raft leader, 503 otherwise)
+curl -i http://192.168.1.10:8620/v1/ready
+
# Store health
curl http://192.168.1.20:8520/v1/health
```
+> **Note**: `/v1/ready` ships from the release after `1.7.0`, so the Docker
+> examples above, which pin `HUGEGRAPH_VERSION=1.7.0`, need a newer tag or
+> images built from source before this check means anything. On `1.7.0` the PD
+> answers `200` with `{"status":-1,"error":"Unauthorized!"}` on any path its
+> auth interceptor does not exclude, `/v1/ready` included, so match on the body
+> rather than the status code. See
+> [docker/README.md](../../docker/README.md) for the details.
+
### Cluster Status
```bash
