pantovad commented on issue #6244:
URL: https://github.com/apache/skywalking/issues/6244#issuecomment-770753690
Hello,
I did the debug of the agent while trying to do the profiling on it and
found few IMO indicative things. Frequent occurrences of the following errors:
```
ERROR 2021-02-01 09:54:50:328 SkywalkingAgent-7-JVMService-consume-0
JVMMetricsSender : send JVM metrics to Collector fail.
org.apache.skywalking.apm.dependencies.io.grpc.StatusRuntimeException:
DEADLINE_EXCEEDED: deadline exceeded after 29.948960917s.
[buffered_nanos=420756690,
remote_addr=skywalking.cryptobroker.ch/217.20.194.174:11800]
at
org.apache.skywalking.apm.dependencies.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
at
org.apache.skywalking.apm.dependencies.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
at
org.apache.skywalking.apm.dependencies.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
at
org.apache.skywalking.apm.network.language.agent.v3.JVMMetricReportServiceGrpc$JVMMetricReportServiceBlockingStub.collect(JVMMetricReportServiceGrpc.java:181)
at
org.apache.skywalking.apm.agent.core.jvm.JVMMetricsSender.run(JVMMetricsSender.java:82)
at
org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
```
```
ERROR 2021-02-01 09:55:18:319 SkywalkingAgent-8-ServiceManagementClient-0
ServiceManagementClient : ServiceManagementClient execute fail.
org.apache.skywalking.apm.dependencies.io.grpc.StatusRuntimeException:
DEADLINE_EXCEEDED: deadline exceeded after 29.992329509s.
[remote_addr=skywalking.cryptobroker.
ch/217.20.194.174:11800]
at
org.apache.skywalking.apm.dependencies.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
at
org.apache.skywalking.apm.dependencies.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
at
org.apache.skywalking.apm.dependencies.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
at
org.apache.skywalking.apm.network.management.v3.ManagementServiceGrpc$ManagementServiceBlockingStub.reportInstanceProperties(ManagementServiceGrpc.java:253
)
at
org.apache.skywalking.apm.agent.core.remote.ServiceManagementClient.run(ServiceManagementClient.java:120)
at
org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
```
```
ERROR 2021-02-01 09:55:31:810 grpc-default-executor-0
TraceSegmentServiceClient : Send UpstreamSegment to collector fail with a grpc
internal exception.
org.apache.skywalking.apm.dependencies.io.grpc.StatusRuntimeException:
DEADLINE_EXCEEDED: deadline exceeded after 29.999527355s.
[remote_addr=skywalking.cryptobroker.ch/217.20.194.174:11800]
at
org.apache.skywalking.apm.dependencies.io.grpc.Status.asRuntimeException(Status.java:533)
at
org.apache.skywalking.apm.dependencies.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
at
org.apache.skywalking.apm.dependencies.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:413)
at
org.apache.skywalking.apm.dependencies.io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
at
org.apache.skywalking.apm.dependencies.io.grpc.internal.ClientCallImpl$1CloseInContext.runInContext(ClientCallImpl.java:403)
at
org.apache.skywalking.apm.dependencies.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.skywalking.apm.dependencies.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
```
and one specific for the profile:
```
ERROR 2021-02-01 09:59:05:135 grpc-default-executor-1 ProfileSnapshotSender
: Send profile segment snapshot to collector fail with a grpc internal
exception.
org.apache.skywalking.apm.dependencies.io.grpc.StatusRuntimeException:
DEADLINE_EXCEEDED: deadline exceeded after 29.995320900s.
[remote_addr=skywalking.cryptobroker.ch/217.20.194.174:11800]
at
org.apache.skywalking.apm.dependencies.io.grpc.Status.asRuntimeException(Status.java:533)
at
org.apache.skywalking.apm.dependencies.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
at
org.apache.skywalking.apm.dependencies.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:413)
at
org.apache.skywalking.apm.dependencies.io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
at
org.apache.skywalking.apm.dependencies.io.grpc.internal.ClientCallImpl$1CloseInContext.runInContext(ClientCallImpl.java:403)
at
org.apache.skywalking.apm.dependencies.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.skywalking.apm.dependencies.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
```
I can send you the log file containing debug (to a private e-mail or so ...).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
