bitflicker64 opened a new issue, #3478:
URL: https://github.com/apache/hugegraph/issues/3478

   ### Bug Type (问题类型)
   
   logic (逻辑设计问题)
   
   ### Before submit
   
   - [x] I have confirmed and searched that there are no similar problems in 
the historical issue and documents
   
   ### Environment (环境信息)
   
   - Server Version: `master` at `8828d8238` (1.8.0-SNAPSHOT), Server with 
HStore and the PD meta driver
   - Backend: HStore (PD + Store)
   - OS: found by reading the code, not reproduced on a cluster (see "How this 
was checked")
   - Data Size: N/A
   
   ### Expected & Actual behavior (期望与实际表现)
   
   **Expected:** when a PD watch behind a Server meta listener ends, the Server 
either starts it again or makes the loss visible to the code that depends on 
it, so a long-lived Server does not keep working on events it will never get.
   
   **Actual:** `KvClient` treats nine gRPC status codes as non-retryable. A 
watch that ends with one of them is removed and never started again, and the 
only notice is an error handler that the Server leaves empty. Every listener 
registered through `PdMetaDriver` is affected.
   
   The path on `master`:
   
   1. `KvClient` lists the codes it does not retry: `INVALID_ARGUMENT`, 
`NOT_FOUND`, `ALREADY_EXISTS`, `PERMISSION_DENIED`, `FAILED_PRECONDITION`, 
`OUT_OF_RANGE`, `UNIMPLEMENTED`, `DATA_LOSS`, `UNAUTHENTICATED`.
      
https://github.com/apache/hugegraph/blob/8828d8238ad578f0d8ac25010c3dd6d4b3ba0e07/hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/KvClient.java#L73-L82
   2. For one of these, `stopWatch` removes the subscription, logs `Watch for 
key {} stopped after a non-retryable error` at ERROR level, and calls the 
subscription's `errorConsumer` once.
      
https://github.com/apache/hugegraph/blob/8828d8238ad578f0d8ac25010c3dd6d4b3ba0e07/hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/KvClient.java#L472-L486
   3. `PdMetaDriver.listen` and `listenPrefix` call the two-argument `KvClient` 
methods, whose `errorConsumer` is `throwable -> { }`.
      
https://github.com/apache/hugegraph/blob/8828d8238ad578f0d8ac25010c3dd6d4b3ba0e07/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/meta/PdMetaDriver.java#L111-L125
      
https://github.com/apache/hugegraph/blob/8828d8238ad578f0d8ac25010c3dd6d4b3ba0e07/hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/KvClient.java#L252-L263
   4. So nothing above the PD client learns that the watch is gone. For the 
schema cache this is permanent: `CachedSchemaTransactionV2` registers its 
listener once per JVM and never resets `metaEventListenerRegistered`, so no 
later transaction registers again.
      
https://github.com/apache/hugegraph/blob/8828d8238ad578f0d8ac25010c3dd6d4b3ba0e07/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CachedSchemaTransactionV2.java#L234-L255
   
   The result for the schema cache: that Server keeps the schema it has cached 
until it is restarted, and the one sign is a single ERROR line in its log. The 
same holds for the other listeners that `MetaManager` registers through the PD 
driver (graph, graph space and service add/update/remove, graph clear, the 
graph, vertex and edge cache clears, auth events, REST and Gremlin 
configuration updates).
   
   **Where the codes can come from**
   
   - PD does not answer a watch with any of them itself. Outside `KvClient` and 
its test, the sources under `hugegraph-pd` use none of the nine as a gRPC 
status, and a refused watch on a follower is `responseObserver.onError(new 
PDException(-1, msg))`, which the client sees as `UNKNOWN` and retries. The 
auth interceptor rejects a call by throwing a plain `RuntimeException`, not a 
gRPC status.
   - They can come from the gRPC layer. gRPC Java 1.55.3 (the version in the 
root `pom.xml`) maps an HTTP answer without a gRPC status as follows: 401 to 
`UNAUTHENTICATED`, 403 to `PERMISSION_DENIED`, 404 to `UNIMPLEMENTED`. So a 
proxy, gateway or mesh sidecar between Server and PD that answers the watch 
stream with 401, 403 or 404 once ends the watch for good. A Server whose PD 
address points at an endpoint without the KV watch service gets `UNIMPLEMENTED` 
as well.
   
   So a deployment where Server talks to PD directly, such as the Helm chart, 
is not expected to hit this. It matters once something sits between them, or if 
PD starts to return real auth statuses.
   
   **How this was checked**
   
   - Code read at `8828d8238`, links above.
   - `KvClientTest.testPermanentErrorStopsReconnect` on `master` asserts the 
client behaviour for all nine codes: one watch call, no reconnect scheduled.
   - The search over PD sources: `git grep -nwE 
"(PERMISSION_DENIED|UNAUTHENTICATED|FAILED_PRECONDITION|INVALID_ARGUMENT|NOT_FOUND|ALREADY_EXISTS|OUT_OF_RANGE|UNIMPLEMENTED|DATA_LOSS)"
 8828d8238 -- 'hugegraph-pd/*.java'`. It matches `KvClient.java`, 
`KvClientTest.java`, and `NOT_FOUND` as the protobuf 
`Pdpb.ErrorType.NOT_FOUND`, which is a response field and not a gRPC status.
   - The HTTP mapping: `javap -p -c` on 
`io.grpc.internal.GrpcUtil.httpStatusToGrpcCode` from `grpc-core-1.55.3.jar`.
   - Not done: a cluster run with a proxy that injects one of these answers.
   
   **Related**
   
   - #3152 and #3157 made the retryable errors retry until the watch is back, 
and introduced the split into retryable and non-retryable codes.
   - #3036 asked for the schema cache listener to come back after a meta 
reconnect.
   - Found while going through review comments on #3477. That PR adds a 
callback that runs each time the watch starts and clears the schema caches 
there. A stopped watch never starts again, so the callback does not run for 
this case, and #3477 lists it under its limits without changing it.
   
   **Possible directions**
   
   1. Let `MetaDriver.listen` take a callback for a stopped watch and pass it 
down to `KvClient` as the `errorConsumer`. `CachedSchemaTransactionV2` would 
drop its caches there and allow the listener to be registered again.
   2. Or handle it once in `PdMetaDriver` for every listener: register the 
watch again with a slow backoff after a stop, so each listener does not need 
its own recovery.
   
   Either one needs a limit on how often a watch that keeps getting refused is 
registered again, since a wrong endpoint answers the same way every time.
   
   ### Vertex/Edge example (问题点 / 边数据举例)
   
   N/A
   
   ### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
   
   N/A
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to