bitflicker64 opened a new issue, #3478:
URL: https://github.com/apache/hugegraph/issues/3478
### Bug Type (问题类型)
logic (逻辑设计问题)
### Before submit
- [x] I have confirmed and searched that there are no similar problems in
the historical issue and documents
### Environment (环境信息)
- Server Version: `master` at `8828d8238` (1.8.0-SNAPSHOT), Server with
HStore and the PD meta driver
- Backend: HStore (PD + Store)
- OS: found by reading the code, not reproduced on a cluster (see "How this
was checked")
- Data Size: N/A
### Expected & Actual behavior (期望与实际表现)
**Expected:** when a PD watch behind a Server meta listener ends, the Server
either starts it again or makes the loss visible to the code that depends on
it, so a long-lived Server does not keep working on events it will never get.
**Actual:** `KvClient` treats nine gRPC status codes as non-retryable. A
watch that ends with one of them is removed and never started again, and the
only notice is an error handler that the Server leaves empty. Every listener
registered through `PdMetaDriver` is affected.
The path on `master`:
1. `KvClient` lists the codes it does not retry: `INVALID_ARGUMENT`,
`NOT_FOUND`, `ALREADY_EXISTS`, `PERMISSION_DENIED`, `FAILED_PRECONDITION`,
`OUT_OF_RANGE`, `UNIMPLEMENTED`, `DATA_LOSS`, `UNAUTHENTICATED`.
https://github.com/apache/hugegraph/blob/8828d8238ad578f0d8ac25010c3dd6d4b3ba0e07/hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/KvClient.java#L73-L82
2. For one of these, `stopWatch` removes the subscription, logs `Watch for
key {} stopped after a non-retryable error` at ERROR level, and calls the
subscription's `errorConsumer` once.
https://github.com/apache/hugegraph/blob/8828d8238ad578f0d8ac25010c3dd6d4b3ba0e07/hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/KvClient.java#L472-L486
3. `PdMetaDriver.listen` and `listenPrefix` call the two-argument `KvClient`
methods, whose `errorConsumer` is `throwable -> { }`.
https://github.com/apache/hugegraph/blob/8828d8238ad578f0d8ac25010c3dd6d4b3ba0e07/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/meta/PdMetaDriver.java#L111-L125
https://github.com/apache/hugegraph/blob/8828d8238ad578f0d8ac25010c3dd6d4b3ba0e07/hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/KvClient.java#L252-L263
4. So nothing above the PD client learns that the watch is gone. For the
schema cache this is permanent: `CachedSchemaTransactionV2` registers its
listener once per JVM and never resets `metaEventListenerRegistered`, so no
later transaction registers again.
https://github.com/apache/hugegraph/blob/8828d8238ad578f0d8ac25010c3dd6d4b3ba0e07/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CachedSchemaTransactionV2.java#L234-L255
The result for the schema cache: that Server keeps the schema it has cached
until it is restarted, and the one sign is a single ERROR line in its log. The
same holds for the other listeners that `MetaManager` registers through the PD
driver (graph, graph space and service add/update/remove, graph clear, the
graph, vertex and edge cache clears, auth events, REST and Gremlin
configuration updates).
**Where the codes can come from**
- PD does not answer a watch with any of them itself. Outside `KvClient` and
its test, the sources under `hugegraph-pd` use none of the nine as a gRPC
status, and a refused watch on a follower is `responseObserver.onError(new
PDException(-1, msg))`, which the client sees as `UNKNOWN` and retries. The
auth interceptor rejects a call by throwing a plain `RuntimeException`, not a
gRPC status.
- They can come from the gRPC layer. gRPC Java 1.55.3 (the version in the
root `pom.xml`) maps an HTTP answer without a gRPC status as follows: 401 to
`UNAUTHENTICATED`, 403 to `PERMISSION_DENIED`, 404 to `UNIMPLEMENTED`. So a
proxy, gateway or mesh sidecar between Server and PD that answers the watch
stream with 401, 403 or 404 once ends the watch for good. A Server whose PD
address points at an endpoint without the KV watch service gets `UNIMPLEMENTED`
as well.
So a deployment where Server talks to PD directly, such as the Helm chart,
is not expected to hit this. It matters once something sits between them, or if
PD starts to return real auth statuses.
**How this was checked**
- Code read at `8828d8238`, links above.
- `KvClientTest.testPermanentErrorStopsReconnect` on `master` asserts the
client behaviour for all nine codes: one watch call, no reconnect scheduled.
- The search over PD sources: `git grep -nwE
"(PERMISSION_DENIED|UNAUTHENTICATED|FAILED_PRECONDITION|INVALID_ARGUMENT|NOT_FOUND|ALREADY_EXISTS|OUT_OF_RANGE|UNIMPLEMENTED|DATA_LOSS)"
8828d8238 -- 'hugegraph-pd/*.java'`. It matches `KvClient.java`,
`KvClientTest.java`, and `NOT_FOUND` as the protobuf
`Pdpb.ErrorType.NOT_FOUND`, which is a response field and not a gRPC status.
- The HTTP mapping: `javap -p -c` on
`io.grpc.internal.GrpcUtil.httpStatusToGrpcCode` from `grpc-core-1.55.3.jar`.
- Not done: a cluster run with a proxy that injects one of these answers.
**Related**
- #3152 and #3157 made the retryable errors retry until the watch is back,
and introduced the split into retryable and non-retryable codes.
- #3036 asked for the schema cache listener to come back after a meta
reconnect.
- Found while going through review comments on #3477. That PR adds a
callback that runs each time the watch starts and clears the schema caches
there. A stopped watch never starts again, so the callback does not run for
this case, and #3477 lists it under its limits without changing it.
**Possible directions**
1. Let `MetaDriver.listen` take a callback for a stopped watch and pass it
down to `KvClient` as the `errorConsumer`. `CachedSchemaTransactionV2` would
drop its caches there and allow the listener to be registered again.
2. Or handle it once in `PdMetaDriver` for every listener: register the
watch again with a slow backoff after a stop, so each listener does not need
its own recovery.
Either one needs a limit on how often a watch that keeps getting refused is
registered again, since a wrong endpoint answers the same way every time.
### Vertex/Edge example (问题点 / 边数据举例)
N/A
### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]