[
https://issues.apache.org/jira/browse/KAFKA-9877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17376281#comment-17376281
]
Kiran commented on KAFKA-9877:
------------------------------
I am seeing the same issue in kafka1.0 as well..
org.apache.kafka.common.errors.KafkaStorageException: Error while deleting
segments for testtopic-0 in dir /kafka-logs
Caused by: java.io.IOException: Delete of log 00000000000000000000.log.deleted
failed.
at kafka.log.LogSegment.delete(LogSegment.scala:496)
at
kafka.log.Log$$anonfun$kafka$log$Log$$deleteSeg$1$1.apply$mcV$sp(Log.scala:1596)
at kafka.log.Log$$anonfun$kafka$log$Log$$deleteSeg$1$1.apply(Log.scala:1596)
at kafka.log.Log$$anonfun$kafka$log$Log$$deleteSeg$1$1.apply(Log.scala:1596)
at kafka.log.Log.maybeHandleIOException(Log.scala:1669)
at kafka.log.Log.kafka$log$Log$$deleteSeg$1(Log.scala:1595)
at
kafka.log.Log$$anonfun$kafka$log$Log$$asyncDeleteSegment$1.apply$mcV$sp(Log.scala:1599)
at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)
at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:61)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I have compaction enabled for topic with below config:
segment.ms=100ms
delete.retention.ms=100ms
also, lot of below errros.
ERROR Error while processing data for partition testtopic1-18
(kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.KafkaStorageException: Replica 3 is in an
offline log directory for partition testtopic-10
> ERROR Shutdown broker because all log dirs in /tmp/kafka-logs have failed
> (kafka.log.LogManager)
> ------------------------------------------------------------------------------------------------
>
> Key: KAFKA-9877
> URL: https://issues.apache.org/jira/browse/KAFKA-9877
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
> Affects Versions: 2.1.1
> Environment: Redhat
> Reporter: Hawking Du
> Priority: Major
> Attachments: server-125.log
>
>
> There is a so confused problem around me long time.
> Kafka server often stop exceptionally seems caused by log clean process. Here
> are some of logs from server. Can anyone give me some ideas for fixing it.
> {code:java}
> [2020-04-04 02:07:57,410] INFO [GroupMetadataManager brokerId=5] Removed 0
> expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)[2020-04-04 02:07:57,410] INFO
> [GroupMetadataManager brokerId=5] Removed 0 expired offsets in 0
> milliseconds. (kafka.coordinator.group.GroupMetadataManager)[2020-04-04
> 02:17:57,410] INFO [GroupMetadataManager brokerId=5] Removed 0 expired
> offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)[2020-04-04 02:27:57,410] INFO
> [GroupMetadataManager brokerId=5] Removed 0 expired offsets in 0
> milliseconds. (kafka.coordinator.group.GroupMetadataManager)[2020-04-04
> 02:30:22,272] INFO [ProducerStateManager partition=__consumer_offsets-35]
> Writing producer snapshot at offset 741037
> (kafka.log.ProducerStateManager)[2020-04-04 02:30:22,274] INFO [Log
> partition=__consumer_offsets-35, dir=/tmp/kafka-logs] Rolled new log segment
> at offset 741037 in 3 ms. (kafka.log.Log)[2020-04-04 02:30:26,289] ERROR
> Failed to clean up log for __consumer_offsets-35 in dir /tmp/kafka-logs due
> to IOException
> (kafka.server.LogDirFailureChannel)java.nio.file.NoSuchFileException:
> /tmp/kafka-logs/__consumer_offsets-35/00000000000000000000.log at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at
> sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409) at
> sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at
> java.nio.file.Files.move(Files.java:1395) at
> org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:815) at
> org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:224) at
> kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:508) at
> kafka.log.Log.asyncDeleteSegment(Log.scala:1962) at
> kafka.log.Log.$anonfun$replaceSegments$6(Log.scala:2025) at
> kafka.log.Log.$anonfun$replaceSegments$6$adapted(Log.scala:2020) at
> scala.collection.immutable.List.foreach(List.scala:392) at
> kafka.log.Log.replaceSegments(Log.scala:2020) at
> kafka.log.Cleaner.cleanSegments(LogCleaner.scala:602) at
> kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:528) at
> kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:527) at
> scala.collection.immutable.List.foreach(List.scala:392) at
> kafka.log.Cleaner.doClean(LogCleaner.scala:527) at
> kafka.log.Cleaner.clean(LogCleaner.scala:501) at
> kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:359) at
> kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:328) at
> kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:307) at
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89) Suppressed:
> java.nio.file.NoSuchFileException:
> /tmp/kafka-logs/__consumer_offsets-35/00000000000000000000.log ->
> /tmp/kafka-logs/__consumer_offsets-35/00000000000000000000.log.deleted at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at
> sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396) at
> sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at
> java.nio.file.Files.move(Files.java:1395) at
> org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:812)
> ... 17 more[2020-04-04 02:30:26,296] INFO [ReplicaManager broker=5] Stopping
> serving replicas in dir /tmp/kafka-logs
> (kafka.server.ReplicaManager)[2020-04-04 02:30:26,302] INFO
> [ReplicaFetcherManager on broker 5] Removed fetcher for partitions
> Set(fitment-deduct-0, __consumer_offsets-22, __consumer_offsets-30,
> __consumer_offsets-4, __consumer_offsets-27, __consumer_offsets-7,
> __consumer_offsets-9, __consumer_offsets-46, __consumer_offsets-35,
> __consumer_offsets-23, __consumer_offsets-49, __consumer_offsets-47, test-0,
> __consumer_offsets-31, __consumer_offsets-42, __consumer_offsets-3,
> __consumer_offsets-18, __consumer_offsets-15, __consumer_offsets-24,
> ajhz-log-0, __consumer_offsets-38, __consumer_offsets-19,
> __consumer_offsets-11, bpinfo-sync-0, spinfo-sync-backup-0,
> __consumer_offsets-2, __consumer_offsets-43, __consumer_offsets-6,
> __consumer_offsets-14, __consumer_offsets-44, __consumer_offsets-39,
> __consumer_offsets-26, __consumer_offsets-29, __consumer_offsets-34,
> __consumer_offsets-10, video-log-0)
> (kafka.server.ReplicaFetcherManager)[2020-04-04 02:30:26,303] INFO
> [ReplicaAlterLogDirsManager on broker 5] Removed fetcher for partitions
> Set(fitment-deduct-0, __consumer_offsets-22, __consumer_offsets-30,
> __consumer_offsets-4, __consumer_offsets-27, __consumer_offsets-7,
> __consumer_offsets-9, __consumer_offsets-46, __consumer_offsets-35,
> __consumer_offsets-23, __consumer_offsets-49, __consumer_offsets-47, test-0,
> __consumer_offsets-31, __consumer_offsets-42, __consumer_offsets-3,
> __consumer_offsets-18, __consumer_offsets-15, __consumer_offsets-24,
> ajhz-log-0, __consumer_offsets-38, __consumer_offsets-19,
> __consumer_offsets-11, bpinfo-sync-0, spinfo-sync-backup-0,
> __consumer_offsets-2, __consumer_offsets-43, __consumer_offsets-6,
> __consumer_offsets-14, __consumer_offsets-44, __consumer_offsets-39,
> __consumer_offsets-26, __consumer_offsets-29, __consumer_offsets-34,
> __consumer_offsets-10, video-log-0)
> (kafka.server.ReplicaAlterLogDirsManager)[2020-04-04 02:30:26,330] INFO
> [ReplicaManager broker=5] Broker 5 stopped fetcher for partitions
> fitment-deduct-0,__consumer_offsets-22,__consumer_offsets-30,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,__consumer_offsets-35,__consumer_offsets-23,__consumer_offsets-49,__consumer_offsets-47,test-0,__consumer_offsets-31,__consumer_offsets-42,__consumer_offsets-3,__consumer_offsets-18,__consumer_offsets-15,__consumer_offsets-24,ajhz-log-0,__consumer_offsets-38,__consumer_offsets-19,__consumer_offsets-11,bpinfo-sync-0,spinfo-sync-backup-0,__consumer_offsets-2,__consumer_offsets-43,__consumer_offsets-6,__consumer_offsets-14,__consumer_offsets-44,__consumer_offsets-39,__consumer_offsets-26,__consumer_offsets-29,__consumer_offsets-34,__consumer_offsets-10,video-log-0
> and stopped moving logs for partitions because they are in the failed log
> directory /tmp/kafka-logs. (kafka.server.ReplicaManager)[2020-04-04
> 02:30:26,330] INFO Stopping serving logs in dir /tmp/kafka-logs
> (kafka.log.LogManager)[2020-04-04 02:30:26,347] ERROR Shutdown broker because
> all log dirs in /tmp/kafka-logs have failed (kafka.log.LogManager){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
