sodared opened a new issue #5505:
URL: https://github.com/apache/skywalking/issues/5505
Please answer these questions before submitting your issue.
- Why do you submit this issue?
Question
___
### Question
- What do you want to know?
What caused the failed information in log
___
### Bug
- Which version of SkyWalking, OS and JRE?
SkyWalking6.1.0 CentOS7 Elasticsearch6.8 jdk1.8
- What happened?
If possible, provide a way to reproduce the error. e.g. demo application,
component version.
IN skywalking-oap-server.log sometimes exist error
################################################################################
org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker
- 113 [DataCarrier.REGISTER_L2.BulkConsumePool.0.Thread] ERROR [] -
Elasticsearch exception [type=es_rejected_execution_exception, reason=rejected
execution of processing of [26498660][indices:data/write/bulk[s][p]]: request:
BulkShardRequest [[elasticsearch_service_instance_inventory][1]] containing
[index
{[elasticsearch_service_instance_inventory][type][2_db832d795d464164afc35af6d9ceb76f_0_0],
source[{"sequence":171,"heartbeat_time":1600312410914,"service_id":2,"address_id":0,"name":"common-pid:17364@ecs-test","is_address":0,"instance_uuid":"db832d795d464164afc35af6d9ceb76f","register_time":1599613931154,"properties":"{\"os_name\":\"Linux\",\"host_name\":\"ecs-test\",\"process_no\":\"17364\",\"language\":\"java\",\"ipv4s\":\"[\\\"192.9.250.121\\\"]\"}"}]}]
and a refresh, target allocation id: iDMVGloXS5i5ebYP9lTpZA, primary term: 2
on EsThreadPoolExecutor[name = node-1/write, queue capacity = 200, or
g.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6ec213bb[Running,
pool size = 16, active threads = 16, queued tasks = 200, completed tasks =
3999038]]]
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception
[type=es_rejected_execution_exception, reason=rejected execution of processing
of [26498660][indices:data/write/bulk[s][p]]: request: BulkShardRequest
[[elasticsearch_service_instance_inventory][1]] containing [index
{[elasticsearch_service_instance_inventory][type][2_db832d795d464164afc35af6d9ceb76f_0_0],
source[{"sequence":171,"heartbeat_time":1600312410914,"service_id":2,"address_id":0,"name":"common-pid:17364@ecs-test","is_address":0,"instance_uuid":"db832d795d464164afc35af6d9ceb76f","register_time":1599613931154,"properties":"{\"os_name\":\"Linux\",\"host_name\":\"ecs-test\",\"process_no\":\"17364\",\"language\":\"java\",\"ipv4s\":\"[\\\"192.9.250.121\\\"]\"}"}]}]
and a refresh, target allocation id: iDMVGloXS5i5ebYP9lTpZA, primary term: 2
on EsThreadPoolExecutor[name = node-1/write, queue capacity = 200,
org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6ec213bb[Running,
pool size = 16, activ
e threads = 16, queued tasks = 200, completed tasks = 3999038]]]
at
org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
~[elasticsearch-6.3.2.jar:6.3.2]
at
org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:653)
~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at
org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:628)
~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at
org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:535)
~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at
org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508)
~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at
org.elasticsearch.client.RestHighLevelClient.update(RestHighLevelClient.java:366)
~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at
org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.forceUpdate(ElasticSearchClient.java:213)
~[library-client-6.1.0.jar:6.1.0]
at
org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.RegisterEsDAO.forceUpdate(RegisterEsDAO.java:60)
~[storage-elasticsearch-plugin-6.1.0.jar:6.1.0]
at
org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.lambda$onWork$0(RegisterPersistentWorker.java:90)
~[server-core-6.1.0.jar:6.1.0]
at java.util.HashMap$Values.forEach(HashMap.java:981) [?:1.8.0_212]
at
org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.onWork(RegisterPersistentWorker.java:85)
[server-core-6.1.0.jar:6.1.0]
at
org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.access$100(RegisterPersistentWorker.java:36)
[server-core-6.1.0.jar:6.1.0]
at
org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker$PersistentConsumer.consume(RegisterPersistentWorker.java:142)
[server-core-6.1.0.jar:6.1.0]
at
org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.consume(MultipleChannelsConsumer.java:80)
[apm-datacarrier-6.1.0.jar:6.1.0]
at
org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:49)
[apm-datacarrier-6.1.0.jar:6.1.0]
Suppressed: org.elasticsearch.client.ResponseException: method [POST], host
[http://elasticsearch.eis.smec-cn.com:9200], URI
[/elasticsearch_service_instance_inventory/type/2_db832d795d464164afc35af6d9ceb76f_0_0/_update?refresh=true&timeout=1m],
status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[node-1][192.9.250.113:9300][indices:data/write/update[s]]"}],"type":"es_rejected_execution_exception","reason":"rejected
execution of processing of [26498660][indices:data/write/bulk[s][p]]: request:
BulkShardRequest [[elasticsearch_service_instance_inventory][1]] containing
[index
{[elasticsearch_service_instance_inventory][type][2_db832d795d464164afc35af6d9ceb76f_0_0],
source[{\"sequence\":171,\"heartbeat_time\":1600312410914,\"service_id\":2,\"address_id\":0,\"name\":\"common-pid:17364@ecs-test\",\"is_address\":0,\"instance_uuid\":\"db832d795d464164afc35af6d9ceb76f\",\"register_time\":1599613931154,\"properties\":\"{\\\"os_name\\\":\\\"Linux\\\",\\\"host_name\\\":\\\"ecs-test\\\",\\\"process_no\\\":\\\"17364\\\",\\\"language\\\":\\\"java\\\",\\\"ipv4s\\\":\\\"[\\\\\\\"192.9.250.121\\\\\\\"]\\\"}\"}]}]
and a refresh, target allocation id: iDMVGloXS5i5ebYP9lTpZA, primary term: 2
on EsThreadPoolExecutor[name =
node-1/write, queue capacity = 200,
org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6ec213bb[Running,
pool size = 16, active threads = 16, queued tasks = 200, completed tasks =
3999038]]"},"status":429}
at
org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:705)
~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at
org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at
org.elasticsearch.client.RestClient.performRequest(RestClient.java:198)
~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at
org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:522)
~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at
org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508)
~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at
org.elasticsearch.client.RestHighLevelClient.update(RestHighLevelClient.java:366)
~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at
org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.forceUpdate(ElasticSearchClient.java:213)
~[library-client-6.1.0.jar:6.1.0]
at
org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.RegisterEsDAO.forceUpdate(RegisterEsDAO.java:60)
~[storage-elasticsearch-plugin-6.1.0.jar:6.1.0]
at
org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.lambda$onWork$0(RegisterPersistentWorker.java:90)
~[server-core-6.1.0.jar:6.1.0]
at java.util.HashMap$Values.forEach(HashMap.java:981)
[?:1.8.0_212]
at
org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.onWork(RegisterPersistentWorker.java:85)
[server-core-6.1.0.jar:6.1.0]
at
org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.access$100(RegisterPersistentWorker.java:36)
[server-core-6.1.0.jar:6.1.0]
at
org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker$PersistentConsumer.consume(RegisterPersistentWorker.java:142)
[server-core-6.1.0.jar:6.1.0]
at
org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.consume(MultipleChannelsConsumer.java:80)
[apm-datacarrier-6.1.0.jar:6.1.0]
at
org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:49)
[apm-datacarrier-6.1.0.jar:6.1.0]
Caused by: org.elasticsearch.client.ResponseException: method [POST],
host [http://elasticsearch.eis.smec-cn.com:9200], URI
[/elasticsearch_service_instance_inventory/type/2_db832d795d464164afc35af6d9ceb76f_0_0/_update?refresh=true&timeout=1m],
status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[node-1][192.9.250.113:9300][indices:data/write/update[s]]"}],"type":"es_rejected_execution_exception","reason":"rejected
execution of processing of [26498660][indices:data/write/bulk[s][p]]: request:
BulkShardRequest [[elasticsearch_service_instance_inventory][1]] containing
[index
{[elasticsearch_service_instance_inventory][type][2_db832d795d464164afc35af6d9ceb76f_0_0],
source[{\"sequence\":171,\"heartbeat_time\":1600312410914,\"service_id\":2,\"address_id\":0,\"name\":\"common-pid:17364@ecs-test\",\"is_address\":0,\"instance_uuid\":\"db832d795d464164afc35af6d9ceb76f\",\"register_time\":1599613931154,\"properties\":\"{\\\"os_name\\\":\\\"Linux\\\",\\\"host_name\\\":\\\"ecs-test\\\",\\\"process_no\\\":\\\"17364\\\",\\\"language\\\":\\\"java\\\",\\\"ipv4s\\\":\\\"[\\\\\\\"192.9.250.121\\\\\\\"]\\\"}\"}]}]
and a refresh, target allocation id: iDMVGloXS5i5ebYP9lTpZA, primary term: 2
on EsThreadPoolExecutor[name =
node-1/write, queue capacity = 200,
org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6ec213bb[Running,
pool size = 16, active threads = 16, queued tasks = 200, completed tasks =
3999038]]"},"status":429}
at
org.elasticsearch.client.RestClient$1.completed(RestClient.java:377)
~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at
org.elasticsearch.client.RestClient$1.completed(RestClient.java:366)
~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at
org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
~[httpcore-4.4.5.jar:4.4.5]
at
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
~[httpasyncclient-4.1.2.jar:4.1.2]
at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
~[httpcore-nio-4.4.5.jar:4.4.5]
at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
~[httpcore-nio-4.4.5.jar:4.4.5]
at
org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
~[httpcore-nio-4.4.5.jar:4.4.5]
at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
~[httpasyncclient-4.1.2.jar:4.1.2]
at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
~[httpasyncclient-4.1.2.jar:4.1.2]
at
org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
~[httpcore-nio-4.4.5.jar:4.4.5]
at
org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
~[httpcore-nio-4.4.5.jar:4.4.5]
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
~[httpcore-nio-4.4.5.jar:4.4.5]
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
~[httpcore-nio-4.4.5.jar:4.4.5]
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
~[httpcore-nio-4.4.5.jar:4.4.5]
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
~[httpcore-nio-4.4.5.jar:4.4.5]
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
~[httpcore-nio-4.4.5.jar:4.4.5]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]
###################################################################################
___
### Requirement or improvement
- Please describe your requirements or improvement suggestions.
Please help me for this error.
appreciate!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
