killGC opened a new issue #5569:
URL: https://github.com/apache/skywalking/issues/5569
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [ ] Question or discussion
- [ ] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
___
### Question
- What do you want to know?
skywalking oap cluster run in two server，i observer oap server and es
resource using is nomal
what reason maybe cause this？
```java
2020-09-27 16:45:45,570 -
org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer - 110
[pool-13-thread-1] INFO [] - Update the cache of service instance inventory,
instance id: 549
2020-09-27 16:45:45,570 -
org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer - 110
[pool-13-thread-1] INFO [] - Update the cache of service instance inventory,
instance id: 462
2020-09-27 16:45:45,570 -
org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer - 110
[pool-13-thread-1] INFO [] - Update the cache of service instance inventory,
instance id: 66
2020-09-27 16:45:49,459 -
org.apache.skywalking.oap.server.receiver.trace.provider.UninstrumentedGatewaysConfig
- 81 [pool-10-thread-1] INFO [] - Updating uninstrumented gateways with: null
2020-09-27 16:45:52,717 -
org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler
- 70 [grpcServerPool-1-thread-35] ERROR [] - CANCELLED: cancelled before
receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half
close
at io.grpc.Status.asRuntimeException(Status.java:524)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2020-09-27 16:45:52,717 -
org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler
- 70 [grpcServerPool-1-thread-45] ERROR [] - CANCELLED: cancelled before
receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half
close
at io.grpc.Status.asRuntimeException(Status.java:524)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2020-09-27 16:45:52,717 -
org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler
- 70 [grpcServerPool-1-thread-5] ERROR [] - CANCELLED: cancelled before
receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half
close
at io.grpc.Status.asRuntimeException(Status.java:524)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2020-09-27 16:45:52,718 -
org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler
- 70 [grpcServerPool-1-thread-8] ERROR [] - CANCELLED: cancelled before
receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half
close
at io.grpc.Status.asRuntimeException(Status.java:524)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2020-09-27 16:45:52,718 -
org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler
- 70 [grpcServerPool-1-thread-50] ERROR [] - CANCELLED: cancelled before
receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half
close
at io.grpc.Status.asRuntimeException(Status.java:524)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2020-09-27 16:45:52,718 -
org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler
- 70 [grpcServerPool-1-thread-37] ERROR [] - CANCELLED: cancelled before
receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half
close
at io.grpc.Status.asRuntimeException(Status.java:524)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2020-09-27 16:45:52,718 -
org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler
- 70 [grpcServerPool-1-thread-13] ERROR [] - CANCELLED: cancelled before
receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half
close
at io.grpc.Status.asRuntimeException(Status.java:524)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2020-09-27 16:45:52,718 -
org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler
- 70 [grpcServerPool-1-thread-34] ERROR [] - CANCELLED: cancelled before
receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half
close
at io.grpc.Status.asRuntimeException(Status.java:524)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2020-09-27 16:45:52,718 -
org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler
- 70 [grpcServerPool-1-thread-40] ERROR [] - CANCELLED: cancelled before
receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half
close
at io.grpc.Status.asRuntimeException(Status.java:524)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
```java
___
### Bug
- Which version of SkyWalking, OS and JRE?
skywalking 7.0, os linux, jre jdk8
- Which company or project?
- What happened?
If possible, provide a way to reproduce the error. e.g. demo application,
component version.
___
### Requirement or improvement
- Please describe your requirements or improvement suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
