Re: ERROR [] - DEADLINE_EXCEEDED: deadline exceeded after 19.999959545s.

[Sheng Wu]

This means timeout in gRPC.
Generally, the OAP server or storage database can't support this load.

Could you provide more context and maybe metrics of self-observability?

Sheng Wu 吴晟
Twitter, wusheng1108

Wangpeng (Forest, ISCP SRE) <forest.w...@huawei.com> 于2021年10月22日周五 下午4:27写道：
>
> Hi Community:
>          We are running skywalking 8.7.0 cluster with 8 nodes.
>          Recently, there are a lot of following ERRORs.
> I thought maybe it's the server side consume capability problem. So I had 
> increase METRICS_L2_AGGREGATION_THREAD to 40, but it doesn't work.
>          Can you give me some leads about this problem?
>
> 2021-10-08 23:18:11,790 - 
> org.apache.skywalking.oap.server.core.remote.client.GRPCRemoteClient - 224 
> [grpc-default-executor-18748] ERROR [] - DEADLINE_EXCEEDED: deadline exceeded 
> after 19.999959545s. [remote_addr=/12.11.0.105:11800]
> io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 
> 19.999959545s. [remote_addr=/12.11.0.105:11800]
>          at io.grpc.Status.asRuntimeException(Status.java:533) 
> ~[grpc-api-1.32.1.jar:1.32.1]
>          at 
> io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
>  [grpc-stub-1.32.1.jar:1.32.1]
>          at 
> io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:413) 
> [grpc-core-1.32.1.jar:1.32.1]
>          at 
> io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66) 
> [grpc-core-1.32.1.jar:1.32.1]
>          at 
> io.grpc.internal.ClientCallImpl$1CloseInContext.runInContext(ClientCallImpl.java:403)
>  [grpc-core-1.32.1.jar:1.32.1]
>          at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) 
> [grpc-core-1.32.1.jar:1.32.1]
>          at 
> io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) 
> [grpc-core-1.32.1.jar:1.32.1]
>          at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>  [?:?]
>          at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>  [?:?]
>          at java.lang.Thread.run(Thread.java:834) [?:?]
> 2021-10-08 23:18:11,934 - 
> org.apache.skywalking.oap.server.core.remote.client.GRPCRemoteClient - 224 
> [grpc-default-executor-18734] ERROR [] - DEADLINE_EXCEEDED: deadline exceeded 
> after 19.999995159s. [remote_addr=/12.11.0.59:11800]
> io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 
> 19.999995159s. [remote_addr=/12.11.0.59:11800]
>          at io.grpc.Status.asRuntimeException(Status.java:533) 
> ~[grpc-api-1.32.1.jar:1.32.1]
>          at 
> io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
>  [grpc-stub-1.32.1.jar:1.32.1]
>          at 
> io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:413) 
> [grpc-core-1.32.1.jar:1.32.1]
>          at 
> io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66) 
> [grpc-core-1.32.1.jar:1.32.1]
>          at 
> io.grpc.internal.ClientCallImpl$1CloseInContext.runInContext(ClientCallImpl.java:403)
>  [grpc-core-1.32.1.jar:1.32.1]
>          at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) 
> [grpc-core-1.32.1.jar:1.32.1]
>          at 
> io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) 
> [grpc-core-1.32.1.jar:1.32.1]
>          at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>  [?:?]
>          at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>  [?:?]
>          at java.lang.Thread.run(Thread.java:834) [?:?]
>
> Thank you very much!