imbajin commented on PR #3130:
URL: https://github.com/apache/hugegraph/pull/3130#issuecomment-5283161857
## K8s A/B result: recovery exists, but the current PR breaks the in-flight
retry
I ran a resource-bounded `1 PD + 1 Store + 1 Server` test on Kubernetes. The
Store was registered and consumed as the stable FQDN
`pr3130-hugegraph-store-0.pr3130-hugegraph-store.pr-3130-hstore.svc:8500`, not
as a Pod IP. For each phase I deleted only that Store Pod, then verified a new
Pod UID/IP with the same PVC, Store ID, and FQDN. The Server Pod/JVM stayed
unchanged within each phase.
| Phase | Store replacement | In-flight write | First observable later
success | Server restart |
|---|---|---|---|---|
| current base + #3126, without #3130 | UID changed; IP `192.168.194.8 ->
.10`; Store ID unchanged | HTTP 500 after **128.19s**, `UNAVAILABLE: io
exception` | HTTP 201 at about **t+199s** from deletion | no |
| exact current-base + PR merge (`4066d2ff`, image `sha256:2c4b914d...`) |
UID changed; IP `192.168.194.10 -> .12`; Store ID unchanged | HTTP 500 after
**38.39s**, `The gRPC client is closed` | the next transaction succeeded
immediately; then **10/10** writes succeeded | no |
The merge image includes #3126 and has `networkaddress.cache.ttl=30`.
Runtime name-resolution behavior is from gRPC core/API 1.47.0 (the distribution
still has protobuf-related 1.39.0 artifacts).
### What this proves
```text
Store transport failure
-> NOT_WORK evicts HgStoreNode and closes its channel
-> NodeTxExecutor retry keeps its HgStoreSession cached by nodeId
-> cached session still points to the evicted HgStoreNode
-> node::isCurrent == false
-> every remaining attempt in the same transaction sees
"The gRPC client is closed"
next HTTP transaction
-> transaction/session map is recreated
-> current Store node/session is acquired
-> succeeds immediately
```
This matches the independent source-level analysis. The failure is no longer
hypothetical: the current PR shortens the failure to the configured retry
window, but it cannot recover the request that encountered the Store
replacement.
### Required refactor before merge
1. **Fix logical session generation first.**
`NodeTxExecutor.openNodeSession()` must reject/replace a cached session whose
`HgStoreNode` is no longer the current node (or stop caching it across retry
attempts). A retry after `NOT_WORK` must acquire a fresh node + session.
2. **Keep the stable Kubernetes FQDN as the target.** Do not turn Pod IPs
into application-owned identity.
3. **Keep one bounded channel lifecycle per logical target and let gRPC own
DNS/address updates.** Add only a deliberate reconnect/backoff trigger if the
measured ~199s native recovery is unacceptable; avoid the current 5-second
`InetAddress` fingerprint plus full 32-channel/stub-pool replacement.
4. Add a regression test for: same Store ID + same FQDN + new IP, first
unary request fails transport, the *same retrying transaction* reacquires a
current session and succeeds. Then cover stream/state/healthy clients, which
currently have separate lifecycle paths.
One test nuance: short client-side probe timeouts can still leave
Server-side requests running; several timed-out probes later committed.
Acceptance therefore needs unique operation IDs and must distinguish “HTTP
response observed” from “write eventually committed”.
Based on this K8s run, I would not merge the current channel-refresh
implementation as-is. The small, coherent fix is **stable FQDN + native gRPC
resolution + generation-aware HugeGraph node/session retry**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
