EvanLjp opened a new issue #5486:
URL: https://github.com/apache/skywalking/issues/5486
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [ ] Question or discussion
___
### Question
- What do you want to know?
the roles of oap have receiver and aggregator in skywalking.
Recently, I found the connection between receiver and aggregator is not
stable.
The receiver cannot push metric data to the aggregator.
```
log:
2020-09-15 13:03:55 [grpc-default-executor-928] [ERROR]
[org.apache.skywalking.oap.server.core.remote.client.GRPCRemoteClient] [N/A] -
UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.Status.asRuntimeException(Status.java:533)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:449)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:521)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:641)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:529)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:703)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:692)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_231]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_231]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: /10.122.0.143:9091
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
~[?:1.8.0_231]
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
~[?:1.8.0_231]
at
io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:688)
~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
~[netty-transport-4.1.42.Final.jar:4.1.42.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
~[netty-common-4.1.42.Final.jar:4.1.42.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
~[netty-common-4.1.42.Final.jar:4.1.42.Final]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[netty-common-4.1.42.Final.jar:4.1.42.Final]
... 1 more
```
But when I delete the pod, all things are normal.No more exception logs.
When I read the source code, Oap does not appear to check connectivity, only
check are there any new IP changes.
@wu-sheng thx for your helping.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
