leego86 opened a new issue #6030:
URL: https://github.com/apache/skywalking/issues/6030
### Bug
- Which version of SkyWalking, OS, and JRE?
**version** APM for ES7 8.2 & 8.3
**OS** CentOS Linux release 7.3.1611 (Core)
**JRE** 1.8.0_171
- Which company or project?
LPPZ
- What happened?
The Rocketbot can display some data,but log file is printing errors
1. oap.log is printing a lot of errors, and the file is grouing bigger and
bigger
```
Dec 18, 2020 6:24:10 PM io.grpc.internal.SerializingExecutor run
SEVERE: Exception while executing runnable
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable@7d968be6
io.grpc.StatusRuntimeException: INTERNAL: Invalid protobuf byte sequence
at io.grpc.Status.asRuntimeException(Status.java:524)
at
io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:218)
at
io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:118)
at io.grpc.MethodDescriptor.parseRequest(MethodDescriptor.java:296)
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309)
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292)
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:782)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
message had invalid UTF-8.
at
com.google.protobuf.InvalidProtocolBufferException.invalidUtf8(InvalidProtocolBufferException.java:141)
at com.google.protobuf.Utf8$DecodeUtil.handleTwoBytes(Utf8.java:1909)
at com.google.protobuf.Utf8$DecodeUtil.access$700(Utf8.java:1883)
at com.google.protobuf.Utf8$UnsafeProcessor.decodeUtf8(Utf8.java:1411)
at com.google.protobuf.Utf8.decodeUtf8(Utf8.java:340)
at
com.google.protobuf.CodedInputStream$ArrayDecoder.readStringRequireUtf8(CodedInputStream.java:809)
at
org.apache.skywalking.apm.network.language.agent.v3.SegmentObject.<init>(SegmentObject.java:65)
at
org.apache.skywalking.apm.network.language.agent.v3.SegmentObject.<init>(SegmentObject.java:16)
at
org.apache.skywalking.apm.network.language.agent.v3.SegmentObject$1.parsePartialFrom(SegmentObject.java:1656)
at
org.apache.skywalking.apm.network.language.agent.v3.SegmentObject$1.parsePartialFrom(SegmentObject.java:1650)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:86)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48)
at
io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parseFrom(ProtoLiteUtils.java:223)
at
io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:215)
... 10 more
```
2. skywalking-oap-server.log print error too,but the frequency is much
lower ,like every 12s print some errors
```
2020-12-18 18:06:14,205 -
org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker -
192 [pool-12-thread-1] ERROR [] - Can't split instance id into 2 parts, .1_
org.apache.skywalking.oap.server.core.UnexpectedException: Can't split
instance id into 2 parts, .1_
at
org.apache.skywalking.oap.server.core.analysis.IDManager$ServiceInstanceID.analysisId(IDManager.java:121)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.alarm.provider.NotifyHandler.notify(NotifyHandler.java:78)
~[server-alarm-plugin-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.alarm.AlarmEntrance.forward(AlarmEntrance.java:39)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.analysis.worker.AlarmNotifyWorker.in(AlarmNotifyWorker.java:41)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.analysis.worker.AlarmNotifyWorker.in(AlarmNotifyWorker.java:30)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.lambda$nextWorker$2(MetricsPersistentWorker.java:199)
~[server-core-8.3.0.jar:8.3.0]
at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_171]
at
org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.nextWorker(MetricsPersistentWorker.java:199)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.flushDataToStorage(MetricsPersistentWorker.java:182)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.prepareBatch(MetricsPersistentWorker.java:141)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.analysis.worker.PersistenceWorker.buildBatchRequests(PersistenceWorker.java:74)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$extractDataAndSave$2(PersistenceTimer.java:107)
~[server-core-8.3.0.jar:8.3.0]
at java.util.ArrayList.forEach(ArrayList.java:1257) ~[?:1.8.0_171]
at
org.apache.skywalking.oap.server.core.storage.PersistenceTimer.extractDataAndSave(PersistenceTimer.java:102)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$start$0(PersistenceTimer.java:78)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33)
[apm-util-8.3.0.jar:8.3.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_171]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[?:1.8.0_171]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_171]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_171]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_171]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
2020-12-18 18:06:14,212 -
org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker -
192 [pool-12-thread-1] ERROR [] - Can't split instance id into 2 parts, .1_
org.apache.skywalking.oap.server.core.UnexpectedException: Can't split
instance id into 2 parts, .1_
at
org.apache.skywalking.oap.server.core.analysis.IDManager$ServiceInstanceID.analysisId(IDManager.java:121)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.alarm.provider.NotifyHandler.notify(NotifyHandler.java:78)
~[server-alarm-plugin-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.alarm.AlarmEntrance.forward(AlarmEntrance.java:39)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.analysis.worker.AlarmNotifyWorker.in(AlarmNotifyWorker.java:41)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.analysis.worker.AlarmNotifyWorker.in(AlarmNotifyWorker.java:30)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.lambda$nextWorker$2(MetricsPersistentWorker.java:199)
~[server-core-8.3.0.jar:8.3.0]
at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_171]
at
org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.nextWorker(MetricsPersistentWorker.java:199)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.flushDataToStorage(MetricsPersistentWorker.java:182)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.prepareBatch(MetricsPersistentWorker.java:141)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.analysis.worker.PersistenceWorker.buildBatchRequests(PersistenceWorker.java:74)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$extractDataAndSave$2(PersistenceTimer.java:107)
~[server-core-8.3.0.jar:8.3.0]
at java.util.ArrayList.forEach(ArrayList.java:1257) ~[?:1.8.0_171]
at
org.apache.skywalking.oap.server.core.storage.PersistenceTimer.extractDataAndSave(PersistenceTimer.java:102)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$start$0(PersistenceTimer.java:78)
~[server-core-8.3.0.jar:8.3.0]
at
org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33)
[apm-util-8.3.0.jar:8.3.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_171]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[?:1.8.0_171]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_171]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_171]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_171]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
```
I want to known that
1. how to solve the problems
2. how can I config **oap.log** rolling to prevent disk from becoming full
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
