piaoluoxiaobai opened a new issue #4191: ttl clean data error URL: https://github.com/apache/skywalking/issues/4191 Please answer these questions before submitting your issue. - Why do you submit this issue? - [ ] Question or discussion - [ ] Bug - [ ] Requirement - [ ] Feature or performance improvement ___ ### Question - What do you want to know? **how to solve clean historical data error** ___ ### Bug - Which version of SkyWalking, OS and JRE? 6.4.0 os:linux jre 1.8 - Which company or project? - What happen? If possible, provide a way for reproducing the error. e.g. demo application, component version. i use docker-compose deploy this, like this: `#skywalking的存储 elasticsearch: image: elasticsearch:6.8.1 container_name: elasticsearch restart: always ports: - 9200 - 9300 environment: - discovery.type=single-node - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m" ulimits: memlock: soft: -1 hard: -1 oap: image: skywalking-oap-tz:6.4.0 container_name: oap depends_on: - elasticsearch restart: always ports: - 11800 - 12800 environment: SW_STORAGE: elasticsearch SW_STORAGE_ES_CLUSTER_NODES: elasticsearch:9200 #采样率 The sample rate precision is 1/10000. 10000 means 100% sample in default. SW_TRACE_SAMPLE_RATE: 0 #每1000个请求执行一次 SW_STORAGE_ES_BULK_ACTIONS: 2000 #每40mb刷新一次内存块 SW_STORAGE_ES_BULK_SIZE: 40 #无论请求数量如何，每30s刷新一次堆 SW_STORAGE_ES_FLUSH_INTERVAL: 30 #并发请求的数量 SW_STORAGE_ES_CONCURRENT_REQUESTS: 4 #设置ttl删除日志 (月) SW_CORE_MONTH_METRIC_DATA_TTL: 0 SW_STORAGE_ES_MONTH_METRIC_DATA_TTL: 0 #天 SW_CORE_DAY_METRIC_DATA_TTL: 0 SW_STORAGE_ES_OTHER_METRIC_DATA_TTL: 0 SW_STORAGE_ES_RECORD_DATA_TTL: 1 #小时 SW_CORE_HOUR_METRIC_DATA_TTL: 24` **everything goes well，but when clean historical data，it can't work well with some error:** `2020-01-07 12:05:09,383 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer -79208718 [pool-13-thread-1] INFO [] - Beginning to remove expired metrics from the storage. 2020-01-07 12:05:09,398 - org.elasticsearch.client.RestClient -79208733 [I/O dispatcher 1] WARN [] - request [PUT http://elasticsearch:9200/endpoint_relation_server_side_month-202001?master_timeout=30s&timeout=30s] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in create index requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', and requests are expected to omit the type name in mapping definitions."] 2020-01-07 12:05:09,399 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer -79208734 [pool-13-thread-1] ERROR [] - Remove data in background failure. org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=resource_already_exists_exception, reason=index [endpoint_relation_server_side_month-202001/0d0aytdDRCm8jvpejAf-WA] already exists] at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:653) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:628) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:535) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.IndicesClient.create(IndicesClient.java:99) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.createIndex(ElasticSearchClient.java:168) ~[library-client-6.4.0.jar:6.4.0] at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.HistoryDeleteEsDAO.deleteHistory(HistoryDeleteEsDAO.java:75) ~[storage-elasticsearch-plugin-6.4.0.jar:6.4.0] at org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer.execute(DataTTLKeeperTimer.java:73) ~[server-core-6.4.0.jar:6.4.0] at org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer.lambda$delete$1(DataTTLKeeperTimer.java:66) ~[server-core-6.4.0.jar:6.4.0] at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_181] at org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer.delete(DataTTLKeeperTimer.java:64) ~[server-core-6.4.0.jar:6.4.0] at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:36) [apm-util-6.4.0.jar:6.4.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_181] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_181] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Suppressed: org.elasticsearch.client.ResponseException: method [PUT], host [http://elasticsearch:9200], URI [/endpoint_relation_server_side_month-202001?master_timeout=30s&timeout=30s], status line [HTTP/1.1 400 Bad Request] {"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [endpoint_relation_server_side_month-202001/0d0aytdDRCm8jvpejAf-WA] already exists","index_uuid":"0d0aytdDRCm8jvpejAf-WA","index":"endpoint_relation_server_side_month-202001"}],"type":"resource_already_exists_exception","reason":"index [endpoint_relation_server_side_month-202001/0d0aytdDRCm8jvpejAf-WA] already exists","index_uuid":"0d0aytdDRCm8jvpejAf-WA","index":"endpoint_relation_server_side_month-202001"},"status":400} at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:705) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:522) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.IndicesClient.create(IndicesClient.java:99) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2] at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.createIndex(ElasticSearchClient.java:168) ~[library-client-6.4.0.jar:6.4.0] at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.HistoryDeleteEsDAO.deleteHistory(HistoryDeleteEsDAO.java:75) ~[storage-elasticsearch-plugin-6.4.0.jar:6.4.0] at org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer.execute(DataTTLKeeperTimer.java:73) ~[server-core-6.4.0.jar:6.4.0] at org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer.lambda$delete$1(DataTTLKeeperTimer.java:66) ~[server-core-6.4.0.jar:6.4.0] at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_181] at org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer.delete(DataTTLKeeperTimer.java:64) ~[server-core-6.4.0.jar:6.4.0] at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:36) [apm-util-6.4.0.jar:6.4.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_181] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_181] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by: org.elasticsearch.client.ResponseException: method [PUT], host [http://elasticsearch:9200], URI [/endpoint_relation_server_side_month-202001?master_timeout=30s&timeout=30s], status line [HTTP/1.1 400 Bad Request] {"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [endpoint_relation_server_side_month-202001/0d0aytdDRCm8jvpejAf-WA] already exists","index_uuid":"0d0aytdDRCm8jvpejAf-WA","index":"endpoint_relation_server_side_month-202001"}],"type":"resource_already_exists_exception","reason":"index [endpoint_relation_server_side_month-202001/0d0aytdDRCm8jvpejAf-WA] already exists","index_uuid":"0d0aytdDRCm8jvpejAf-WA","index":"endpoint_relation_server_side_month-202001"},"status":400} at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2] at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2] at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119) ~[httpcore-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) ~[httpasyncclient-4.1.2.jar:4.1.2] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.2.jar:4.1.2] at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.2.jar:4.1.2] at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) ~[httpcore-nio-4.4.5.jar:4.4.5] ... 1 more 2020-01-07 12:05:12,840 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -79212175 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 2039 millis, size: 1906 2020-01-07 12:05:18,179 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -79217514 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 2030 millis, size: 1577` ### Requirement or improvement - Please describe about your requirements or improvement suggestions. i am sure elasticsearch is healthy, how can i solve it
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
