SebastianGruza commented on issue #3172:
URL: https://github.com/apache/hugegraph/issues/3172#issuecomment-5647357426
### The cause of problems 2 and 3, and of problem 1 as well
All three symptoms are the same exception. In 1.7.0 `RaftRpcClient.init()`
calls `rpcClient.init(null)`, so jraft's `BoltRpcClient.opts` is null, and
`internalCallAsyncWithRpc()` passes `invokeCtx = null`. In jraft 1.3.13
`BoltRpcClient.getBoltInvokeCtx(null)` evaluates
`this.opts.isEnableRpcChecksum()` and throws a `NullPointerException` with no
message, synchronously, before anything is sent. It is caught by the `catch
(Throwable t)` that logs only `t.getMessage()`, hence `failed to call rpc to X.
null`. Every `GetMemberRequest` therefore fails, for every peer including the
node itself, which is why localhost "fails" too. It is not the IP whitelist:
`IpAuthHandler` only appeared in May 2026, 1.7.0 does not have it.
Stack trace from the official `apache-hugegraph-incubating-1.7.0`, follower,
`GET /v1/members`:
```
MemberAPI.getMembers(MemberAPI.java:77)
RaftEngine.getLeaderGrpcAddress(RaftEngine.java:242)
RaftRpcClient.getGrpcAddress(RaftRpcClient.java:56)
RaftRpcClient.internalCallAsyncWithRpc(RaftRpcClient.java:80)
BoltRpcClient.invokeAsync(BoltRpcClient.java:113)
BoltRpcClient.getBoltInvokeCtx(BoltRpcClient.java:135) <-
NullPointerException
```
The same RPC serves `getLeaderGrpcAddress()` (problem 1) and `getMembers()`
(problem 3), so on 1.7.0 a follower can never redirect to the leader and
`/v1/members` never has correct data.
### Master fixed it by accident
Commit `b9a3dd9` ("chore(ci): enable hugegraph-struct tests", #3038) changed
`invokeCtx = null` to `new InvokeContext()`, which bypasses the `this.opts`
branch. The same commit carries the `getLeaderGrpcAddress()` hardening known as
#2961. `rpcClient.init(null)` stayed, so `opts` is still null and any future
path with an empty context returns to this NPE. Nobody has linked that commit
to this issue.
### Reproduction on a lab: 1.7.0 vs master, 3 PD nodes
Three VMs, one raft group, the same script for both builds
([`cluster/pd3.sh`](https://github.com/SebastianGruza/hugegraph-validation/blob/master/cluster/pd3.sh)):
fresh start, check every node, stop all, restart with the previous leader
last, check again.
| step | official 1.7.0 | master `60c8803` |
|---|---|---|
| fresh start, `/v1/members` on the leader | `pdLeader: null`, 3 × Offline,
`grpcUrl: ""` | leader 235, 3 × Up, every `grpcUrl` filled |
| fresh start, `/v1/members` on a follower | HTTP 500, `ExecutionException:
NullPointerException` from `getLeaderGrpcAddress` | same answer as on the
leader (redirected) |
| restart all, leader last | the leader moved, behaviour unchanged:
followers 500, leader 3 × Offline | leader 235 → 236, all three nodes answer, 3
× Up, one transient `failed to call rpc` on the node that came up first |
| a store pointing at a follower | does not register (`Exception in storage
registration ... pd=192.168.80.235:8686`) | n/a |
| single node, query to itself | ~9 400 lines of `failed to call rpc to
127.0.0.1:8610. null` within 10 s of start | none, `/v1/members` Up with
`grpcUrl` |
A stronger conclusion than the issue text: on 1.7.0 a follower is broken
from the first start, not only after a leader change. The restart merely
exposes it, because until then every client happened to hit the leader. It also
explains why a #2961 backport always takes the fallback path (leader address
from the local gRPC port): the real RPC has never worked on 1.7.0, not once.
### What I would suggest
- A 1.7.x backport is two lines: `new InvokeContext()` instead of `null` in
`internalCallAsyncWithRpc()`, and, properly, `rpcClient.init(rpcOptions)`
instead of `init(null)` so that `opts` is not null.
- On master the same `init(rpcOptions)`, plus `log.error("failed to call rpc
to {}", endpoint, t)` so that the next message-less NPE does not cost anyone a
week.
- The closed #3195 treated the symptom (a static address map); with this
cause the map is not needed.
Script, the logs of both cycles and the stack trace excerpt:
[`results/issue-3172/`](https://github.com/SebastianGruza/hugegraph-validation/tree/master/results/issue-3172).
If anyone wants, I can test the backport on the same setup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
