nsatish8 opened a new issue #12124:
URL: https://github.com/apache/druid/issues/12124
I ran a purge for 5 days after few hours are deleted from s3 a failure
happened. When I attempted the purge again it found zero records to purge.
Based on where it failed I know there are 4+ days of files to be removed from
s3.
Failure from s3:
2022-01-05T16:14:19,052 ERROR [task-runner-0-priority-0]
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Exception while
running
task[AbstractTask{id='api-issued_kill_xxxxxx_fpacbohk_2021-10-10T00:00:00.000Z_2021-10-15T00:00:00.000Z_2022-01-05T15:24:41.636Z',
groupId='api-issued_kill_xxxxxx_fpacbohk_2021-10-10T00:00:00.000Z_2021-10-15T00:00:00.000Z_2022-01-05T15:24:41.636Z',
taskResource=TaskResource{availabilityGroup='api-issued_kill_xxxxxx_fpacbohk_2021-10-10T00:00:00.000Z_2021-10-15T00:00:00.000Z_2022-01-05T15:24:41.636Z',
requiredCapacity=1}, dataSource='xxxxxx', context={forceTimeChunkLock=true,
useLineageBasedSegmentAllocation=true}}]
org.apache.druid.segment.loading.SegmentLoadingException: Couldn't kill
segment[xxxxxx_2021-10-11T01:00:00.000Z_2021-10-11T02:00:00.000Z_2021-10-11T04:17:02.819Z_64]:
[com.amazonaws.services.s3.model.AmazonS3Exception: Internal Server Error
(Service: Amazon S3; Status Code: 500; Error Code: 500 Internal Server Error;
Request ID: 2RKX5DTHYG26RA36; S3 Extended Request ID:
oQTNAzXTQwMfVIuj07oYF7GRP0cFS038dSSRLrCcjBwEWKUwdxpR3FpdpoljA1zBBwC7TtA+B2k=;
Proxy: null), S3 Extended Request ID:
oQTNAzXTQwMfVIuj07oYF7GRP0cFS038dSSRLrCcjBwEWKUwdxpR3FpdpoljA1zBBwC7TtA+B2k=]
at
org.apache.druid.storage.s3.S3DataSegmentKiller.kill(S3DataSegmentKiller.java:79)
~[?:?]
at
org.apache.druid.segment.loading.OmniDataSegmentKiller.kill(OmniDataSegmentKiller.java:45)
~[druid-server-0.22.1.jar:0.22.1]
at
org.apache.druid.indexing.common.task.KillUnusedSegmentsTask.run(KillUnusedSegmentsTask.java:118)
~[druid-indexing-service-0.22.1.jar:0.22.1]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:471)
[druid-indexing-service-0.22.1.jar:0.22.1]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:443)
[druid-indexing-service-0.22.1.jar:0.22.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_275]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Internal
Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Internal
Server Error; Request ID: 2RKX5DTHYG26RA36; S3 Extended Request ID:
oQTNAzXTQwMfVIuj07oYF7GRP0cFS038dSSRLrCcjBwEWKUwdxpR3FpdpoljA1zBBwC7TtA+B2k=;
Proxy: null)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1862)
~[aws-java-sdk-core-1.12.37.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1415)
~[aws-java-sdk-core-1.12.37.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)
~[aws-java-sdk-core-1.12.37.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)
~[aws-java-sdk-core-1.12.37.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)
~[aws-java-sdk-core-1.12.37.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
~[aws-java-sdk-core-1.12.37.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
~[aws-java-sdk-core-1.12.37.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
~[aws-java-sdk-core-1.12.37.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
~[aws-java-sdk-core-1.12.37.jar:?]
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
~[aws-java-sdk-core-1.12.37.jar:?]
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
~[aws-java-sdk-core-1.12.37.jar:?]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5437)
~[aws-java-sdk-s3-1.12.37.jar:?]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5384)
~[aws-java-sdk-s3-1.12.37.jar:?]
at
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1367)
~[aws-java-sdk-s3-1.12.37.jar:?]
at
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1341)
~[aws-java-sdk-s3-1.12.37.jar:?]
at
com.amazonaws.services.s3.AmazonS3Client.doesObjectExist(AmazonS3Client.java:1422)
~[aws-java-sdk-s3-1.12.37.jar:?]
at
org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3.doesObjectExist(ServerSideEncryptingAmazonS3.java:70)
~[?:?]
at
org.apache.druid.storage.s3.S3DataSegmentKiller.kill(S3DataSegmentKiller.java:73)
~[?:?]
... 8 more
2022-01-05T16:14:19,058 INFO [task-runner-0-priority-0]
org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed
with status: {
"id" :
"api-issued_kill_xxxxxx_fpacbohk_2021-10-10T00:00:00.000Z_2021-10-15T00:00:00.000Z_2022-01-05T15:24:41.636Z",
"status" : "FAILED",
"duration" : 2970118,
"errorMsg" : "org.apache.druid.segment.loading.SegmentLoadingException:
Couldn't kill segment[xxxxxx_2021-10-1...",
"location" : {
"host" : null,
"port" : -1,
"tlsPort" : -1
}
}
2022-01-05T16:14:19,066 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [ANNOUNCEMENTS]
2022-01-05T16:14:19,068 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [SERVER]
2022-01-05T16:14:19,076 INFO [main]
org.eclipse.jetty.server.AbstractConnector - Stopped
ServerConnector@35fb3209{HTTP/1.1, (http/1.1)}{0.0.0.0:8101}
2022-01-05T16:14:19,076 INFO [main] org.eclipse.jetty.server.session - node0
Stopped scavenging
2022-01-05T16:14:19,078 INFO [main]
org.eclipse.jetty.server.handler.ContextHandler - Stopped
o.e.j.s.ServletContextHandler@210c1b9d{/,null,STOPPED}
2022-01-05T16:14:19,079 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [NORMAL]
2022-01-05T16:14:19,080 INFO [main]
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting
graceful shutdown of
task[api-issued_kill_xxxxxx_fpacbohk_2021-10-10T00:00:00.000Z_2021-10-15T00:00:00.000Z_2022-01-05T15:24:41.636Z].
2022-01-05T16:14:19,098 INFO
[LookupExtractorFactoryContainerProvider-MainThread]
org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop
exited. Lookup notices are not handled anymore.
2022-01-05T16:14:19,109 INFO [Curator-Framework-0]
org.apache.curator.framework.imps.CuratorFrameworkImpl -
backgroundOperationsLoop exiting
2022-01-05T16:14:19,215 INFO [main] org.apache.zookeeper.ZooKeeper -
Session: 0x500000c081ca87a closed
2022-01-05T16:14:19,215 INFO [main-EventThread]
org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
0x500000c081ca87a
2022-01-05T16:14:19,224 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [INIT]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
