sunliang3 opened a new issue #1952: CPU utilization 100% & elasticsearch 429 too many requests URL: https://github.com/apache/incubator-skywalking/issues/1952 Please answer these questions before submitting your issue. - Why do you submit this issue? - [-] Question or discussion - [ ] Bug - [ ] Requirement - [ ] Feature or performance improvement ### Bug - Which version of SkyWalking, OS and JRE? 6.0.0-alpha - Which company or project? - What happen? If possible, provide a way for reproducing the error. e.g. demo application, component version. After the service runs for a period of time, the CPU utilization reaches 100%. Print stack information to find nearly one thousand pieces of information as follows: "grpc-default-executor-1226" #1549 daemon prio=5 os_prio=0 tid=0x00002b7264108800 nid=0x305f sleeping[0x00002b7328230000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.skywalking.apm.commons.datacarrier.buffer.Buffer.save(Buffer.java:64) at org.apache.skywalking.apm.commons.datacarrier.buffer.Channels.save(Channels.java:52) at org.apache.skywalking.apm.commons.datacarrier.DataCarrier.produce(DataCarrier.java:88) at org.apache.skywalking.oap.server.core.analysis.worker.RecordPersistentWorker.in(RecordPersistentWorker.java:53) at org.apache.skywalking.oap.server.core.analysis.worker.RecordProcess.in(RecordProcess.java:39) at org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentDispatcher.dispatch(SegmentDispatcher.java:44) at org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentDispatcher.dispatch(SegmentDispatcher.java:28) at org.apache.skywalking.oap.server.core.analysis.DispatcherManager.forward(DispatcherManager.java:71) at org.apache.skywalking.oap.server.core.source.SourceReceiverImpl.receive(SourceReceiverImpl.java:35) at org.apache.skywalking.oap.server.receiver.trace.provider.parser.listener.segment.SegmentSpanListener.build(SegmentSpanListener.java:98) at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParse$$Lambda$262/1249197766.accept(Unknown Source) at java.lang.Iterable.forEach(Iterable.java:75) at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParse.notifyListenerToBuild(SegmentParse.java:169) at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParse.parse(SegmentParse.java:77) at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParse$Producer.send(SegmentParse.java:234) at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v5.grpc.TraceSegmentServiceHandler$1.onNext(TraceSegmentServiceHandler.java :49) at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v5.grpc.TraceSegmentServiceHandler$1.onNext(TraceSegmentServiceHandler.java :43) at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248) at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:252) at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:629) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) I also got many errors in the skywalking-oap-server.log as follows: org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker -1943661 [DataCarrier.IndicatorPersistentWorker.service_inventory.Consumser.0.Thread] ERROR [] - Elasticsearch exception [type=es_rejected_execution_exception, reason=rejected execution of org.elasticsearch.transport.TransportService$7@110d08cf on EsThreadPoolExecutor[name = WRvqRnR/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@513e0b13[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 1802831]]] org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=es_rejected_execution_exception, reason=rejected execution of org.elasticsearch.transport.TransportService$7@110d08cf on EsThreadPoolExecutor[name = WRvqRnR/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@513e0b13[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 1802831]]] at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:653) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:628) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:535) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.update(RestHighLevelClient.java:366) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.forceUpdate(ElasticSearchClient.java:180) ~[library-client-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.RegisterEsDAO.forceUpdate(RegisterEsDAO.java:78) ~[storage-elasticsearch-plugin-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.lambda$onWork$0(RegisterPersistentWorker.java:79) ~[server-core-6.0.0-alpha.jar:6.0.0-alpha] at java.util.HashMap$Values.forEach(HashMap.java:981) [?:1.8.0_171] at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.onWork(RegisterPersistentWorker.java:74) [server-core-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.access$100(RegisterPersistentWorker.java:35) [server-core-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker$PersistentConsumer.consume(RegisterPersistentWorker.java:120) [server-core-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.consume(ConsumerThread.java:101) [apm-datacarrier-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.run(ConsumerThread.java:68) [apm-datacarrier-6.0.0-alpha.jar:6.0.0-alpha] Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [...], URI [/service_inventory/type/question-center_0_0/_update?refresh=true&timeout=1m], status line [HTTP/1.1 429 Too Many Requests] {"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[WRvqRnR][10.6.23.132:9300][indices:data/write/update[s]]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@110d08cf on EsThreadPoolExecutor[name = WRvqRnR/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@513e0b13[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 1802831]]"},"status":429} at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:705) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:522) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.update(RestHighLevelClient.java:366) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.forceUpdate(ElasticSearchClient.java:180) ~[library-client-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.RegisterEsDAO.forceUpdate(RegisterEsDAO.java:78) ~[storage-elasticsearch-plugin-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.lambda$onWork$0(RegisterPersistentWorker.java:79) ~[server-core-6.0.0-alpha.jar:6.0.0-alpha] at java.util.HashMap$Values.forEach(HashMap.java:981) [?:1.8.0_171] at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.onWork(RegisterPersistentWorker.java:74) [server-core-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.access$100(RegisterPersistentWorker.java:35) [server-core-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker$PersistentConsumer.consume(RegisterPersistentWorker.java:120) [server-core-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.consume(ConsumerThread.java:101) [apm-datacarrier-6.0.0-alpha.jar:6.0.0-alpha] at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.run(ConsumerThread.java:68) [apm-datacarrier-6.0.0-alpha.jar:6.0.0-alpha] Caused by: org.elasticsearch.client.ResponseException: method [POST], host [...], URI [/service_inventory/type/question-center_0_0/_update?refresh=true&timeout=1m], status line [HTTP/1.1 429 Too Many Requests] {"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[WRvqRnR][10.6.23.132:9300][indices:data/write/update[s]]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@110d08cf on EsThreadPoolExecutor[name = WRvqRnR/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@513e0b13[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 1802831]]"},"status":429} at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2] at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119) ~[httpcore-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) ~[httpasyncclient-4.1.2.jar:4.1.2] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.2.jar:4.1.2] at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.2.jar:4.1.2] at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) ~[httpcore-nio-4.4.5.jar:4.4.5] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_171] ___ ### Requirement or improvement - Please describe about your requirements or improvement suggestions.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
