bitflicker64 opened a new issue, #3389:
URL: https://github.com/apache/hugegraph/issues/3389
### Bug Type (问题类型)
logic (逻辑设计问题)
### Before submit
- [x] I have confirmed and searched that there are no similar problems in
the historical issue and documents
### Environment (环境信息)
- Server Version: PD on `master` at `ebb8f4e17` (1.8.0-SNAPSHOT), and the KV
read change of #3383 at its head `db0d771f7`
- Backend: PD's raft metadata store (jraft-core 1.3.9)
- OS: any; found by reading the code on 2026-10-22, see "How this was found"
below
- Data Size: N/A
### Expected & Actual behavior (期望与实际表现)
While a PD leader cannot reach a raft quorum, a request that needs the
quorum should fail or be redirected without holding a gRPC worker for long, so
the RPCs that need no quorum keep being served.
What happens instead: three request paths wait on a worker of the one shared
gRPC executor (`hg-grpc`) until raft answers, and the executor has a fixed
number of workers in front of an unbounded queue.
1. **The executor.** `GRpcServerConfig.configure` (lines 37 to 41) installs
`HgExecutorUtil.createExecutor("hg-grpc", core, max, queue)` for the whole PD
gRPC server. `PDConfig.ThreadPoolGrpc` defaults to `thread.pool.grpc.core=600`,
`max=1000` and `queue=Integer.MAX_VALUE`, which becomes a `LinkedBlockingQueue`
of that capacity (`HgExecutorUtil.java` lines 101 to 107). A
`ThreadPoolExecutor` adds threads above the core size only when the queue is
full, so in practice the server has 600 workers, and every request past them
waits in the queue.
2. **KV writes wait without a timeout.** Every write of `RaftKVStore` blocks
on `applyOperation(...).get()` with no deadline (`RaftKVStore.java` lines 76,
99, 109, 119, 128, 138, 157 and 210). Without a quorum the entry does not
commit, and the future completes when jraft fails the pending task at step-down
(`NodeImpl.stepDown` calls `ballotBox.clearPendingTasks()`, which runs each
closure with `EPERM "Leader stepped down"`).
3. **Id reads wait for a ReadIndex.** `IdMetaStore.inLeaderTerm` (line 108)
calls `waitReadIndexInTerm`, added by #3368, which waits up to
`raft.rpc-timeout` (10 s by default; `RaftEngine.waitReadIndex`, lines 663 and
690 to 713).
4. **KV `get` and `scanPrefix` (#3383) wait for the same ReadIndex**, also
up to `raft.rpc-timeout` (`KvServiceGrpcImpl.java` lines 166 and 270 at
`db0d771f7`).
How long a worker is held: PD sets no election timeout, so jraft's default
of 1000 ms applies, with a leader lease of 90% of it. A leader that loses its
quorum steps down about 1 s later. After that, new requests see `isLeader() ==
false` and are redirected without waiting. A write that was waiting is released
at the step-down. A ReadIndex that was already waiting (paths 3 and 4) holds
its worker until it fails or reaches its 10 s deadline.
So a burst of about 600 requests that arrive in the second before the
step-down can occupy every worker. RPCs that need no quorum then queue behind
them, for about a second for writes and up to 10 s for reads. Nothing is lost,
since the queue is unbounded, but the delay is real. This was raised in the
review of #3383 for the KV reads, and paths 2 and 3 show the same holds on
master.
One possible direction, for the reads: let the handler return at once and
answer from a small executor once the ReadIndex completes. `RaftEngine` already
builds a `CompletableFuture` for the wait, so it could return that future
instead of blocking on it. The write path would need the same treatment, or a
deadline. Other directions are a bound on the number of concurrent waits, or a
separate executor for the RPCs that need raft.
### How this was found
By reading the code at `ebb8f4e17` and the jraft-core 1.3.9 sources, on
2026-10-22. **No quorum-loss test was run**, so the counts and times above come
from the configuration defaults and the code paths, not from a measurement.
### Vertex/Edge example (问题点 / 边数据举例)
N/A
### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]