huzaifa kagazwala created KAFKA-7889:
----------------------------------------
Summary: Kafka broker crashes on windows when log compaction is
used
Key: KAFKA-7889
URL: https://issues.apache.org/jira/browse/KAFKA-7889
Project: Kafka
Issue Type: Bug
Components: log
Affects Versions: 2.1.0, 1.1.1, 1.1.0
Reporter: huzaifa kagazwala
I am running kafka on a Windows 10 machine tried the latest "kafka_2.12-2.1.0"
build. I created a topic with log compaction enabled for testing. After
publishing some data as soon as the first log rollover happens the server
crashes with the below error.
command used to create the topic.
kafka-topics.bat -create -zookeeper localhost:2181 -topic compact_log__test_2
-replication-factor 1 -partitions 2 -config segment.bytes=100000 -config
segment.ms=600000 -config min.cleanable.dirty.ratio=0.01 -config
compression.type=snappy -config cleanup.policy=compact -config
min.compaction.lag.ms=30000
Error Details:
[2019-01-31 11:12:09,763] INFO [ProducerStateManager
partition=compact_log__test_2-0] Writing producer snapshot at offset 998
(kafka.log.ProducerStateManager)
[2019-01-31 11:12:09,770] INFO [Log partition=compact_log__test_2-0,
dir=C:\kafka_logs] Rolled new log segment at offset 998 in 35 ms.
(kafka.log.Log)
[2019-01-31 11:12:09,791] INFO [ProducerStateManager
partition=compact_log__test_2-1] Writing producer snapshot at offset 1002
(kafka.log.ProducerStateManager)
[2019-01-31 11:12:09,796] INFO [Log partition=compact_log__test_2-1,
dir=C:\kafka_logs] Rolled new log segment at offset 1002 in 17 ms.
(kafka.log.Log)
[2019-01-31 11:12:34,711] ERROR Failed to clean up log for
compact_log__test_2-0 in dir C:\kafka_logs due to IOException
(kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException:
C:\kafka_logs\compact_log__test_2-0\00000000000000000000.timeindex.cleaned ->
C:\kafka_logs\compact_log__test_2-0\00000000000000000000.timeindex.swap: The
process cannot access the file because it is being used by another process.
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
at java.nio.file.Files.move(Unknown Source)
at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:809)
at kafka.log.AbstractIndex.renameTo(AbstractIndex.scala:205)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:490)
at kafka.log.Log.$anonfun$replaceSegments$4(Log.scala:1892)
at kafka.log.Log.$anonfun$replaceSegments$4$adapted(Log.scala:1892)
at scala.collection.immutable.List.foreach(List.scala:388)
at kafka.log.Log.replaceSegments(Log.scala:1892)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:583)
at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:515)
at kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:514)
at scala.collection.immutable.List.foreach(List.scala:388)
at kafka.log.Cleaner.doClean(LogCleaner.scala:514)
at kafka.log.Cleaner.clean(LogCleaner.scala:492)
at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:353)
at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:319)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:300)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
Suppressed: java.nio.file.FileSystemException:
C:\kafka_logs\compact_log__test_2-0\00000000000000000000.timeindex.cleaned ->
C:\kafka_logs\compact_log__test_2-0\00000000000000000000.timeindex.swap: The
process cannot access the file because it is being used by another process.
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
at java.nio.file.Files.move(Unknown Source)
at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:806)
... 16 more
[2019-01-31 11:12:34,716] INFO [ReplicaManager broker=0] Stopping serving
replicas in dir C:\kafka_logs (kafka.server.ReplicaManager)
[2019-01-31 11:12:34,723] INFO [ReplicaFetcherManager on broker 0] Removed
fetcher for partitions Set(__consumer_offsets-22, __consumer_offsets-30,
__consumer_offsets-8, __consumer_offsets-21, __consumer_offsets-4,
__consumer_offsets-27, __consumer_offsets-7, __consumer_offsets-9,
__consumer_offsets-46, __consumer_offsets-25, compact_log__test_2-0,
__consumer_offsets-35, __consumer_offsets-41, __consumer_offsets-33,
__consumer_offsets-23, __consumer_offsets-49, __consumer_offsets-47,
__consumer_offsets-16, __consumer_offsets-28, __consumer_offsets-31,
__consumer_offsets-36, __consumer_offsets-42, __consumer_offsets-3,
__consumer_offsets-18, __consumer_offsets-37, __consumer_offsets-15,
__consumer_offsets-24, __consumer_offsets-38, __consumer_offsets-17,
__consumer_offsets-48, __consumer_offsets-19, __consumer_offsets-11,
__consumer_offsets-13, __consumer_offsets-2, __consumer_offsets-43,
__consumer_offsets-6, __consumer_offsets-14, compact_log__test_2-1,
__consumer_offsets-20, __consumer_offsets-0, __consumer_offsets-44,
__consumer_offsets-39, __consumer_offsets-12, __consumer_offsets-45,
__consumer_offsets-1, __consumer_offsets-5, __consumer_offsets-26,
__consumer_offsets-29, __consumer_offsets-34, __consumer_offsets-10,
__consumer_offsets-32, __consumer_offsets-40)
(kafka.server.ReplicaFetcherManager)
[2019-01-31 11:12:34,724] INFO [ReplicaAlterLogDirsManager on broker 0] Removed
fetcher for partitions Set(__consumer_offsets-22, __consumer_offsets-30,
__consumer_offsets-8, __consumer_offsets-21, __consumer_offsets-4,
__consumer_offsets-27, __consumer_offsets-7, __consumer_offsets-9,
__consumer_offsets-46, __consumer_offsets-25, compact_log__test_2-0,
__consumer_offsets-35, __consumer_offsets-41, __consumer_offsets-33,
__consumer_offsets-23, __consumer_offsets-49, __consumer_offsets-47,
__consumer_offsets-16, __consumer_offsets-28, __consumer_offsets-31,
__consumer_offsets-36, __consumer_offsets-42, __consumer_offsets-3,
__consumer_offsets-18, __consumer_offsets-37, __consumer_offsets-15,
__consumer_offsets-24, __consumer_offsets-38, __consumer_offsets-17,
__consumer_offsets-48, __consumer_offsets-19, __consumer_offsets-11,
__consumer_offsets-13, __consumer_offsets-2, __consumer_offsets-43,
__consumer_offsets-6, __consumer_offsets-14, compact_log__test_2-1,
__consumer_offsets-20, __consumer_offsets-0, __consumer_offsets-44,
__consumer_offsets-39, __consumer_offsets-12, __consumer_offsets-45,
__consumer_offsets-1, __consumer_offsets-5, __consumer_offsets-26,
__consumer_offsets-29, __consumer_offsets-34, __consumer_offsets-10,
__consumer_offsets-32, __consumer_offsets-40)
(kafka.server.ReplicaAlterLogDirsManager)
[2019-01-31 11:12:34,751] INFO [ReplicaManager broker=0] Broker 0 stopped
fetcher for partitions
__consumer_offsets-22,__consumer_offsets-30,__consumer_offsets-8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,__consumer_offsets-25,compact_log__test_2-0,__consumer_offsets-35,__consumer_offsets-41,__consumer_offsets-33,__consumer_offsets-23,__consumer_offsets-49,__consumer_offsets-47,__consumer_offsets-16,__consumer_offsets-28,__consumer_offsets-31,__consumer_offsets-36,__consumer_offsets-42,__consumer_offsets-3,__consumer_offsets-18,__consumer_offsets-37,__consumer_offsets-15,__consumer_offsets-24,__consumer_offsets-38,__consumer_offsets-17,__consumer_offsets-48,__consumer_offsets-19,__consumer_offsets-11,__consumer_offsets-13,__consumer_offsets-2,__consumer_offsets-43,__consumer_offsets-6,__consumer_offsets-14,compact_log__test_2-1,__consumer_offsets-20,__consumer_offsets-0,__consumer_offsets-44,__consumer_offsets-39,__consumer_offsets-12,__consumer_offsets-45,__consumer_offsets-1,__consumer_offsets-5,__consumer_offsets-26,__consumer_offsets-29,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,__consumer_offsets-40
and stopped moving logs for partitions because they are in the failed log
directory C:\kafka_logs. (kafka.server.ReplicaManager)
[2019-01-31 11:12:34,753] INFO Stopping serving logs in dir C:\kafka_logs
(kafka.log.LogManager)
[2019-01-31 11:12:34,759] ERROR Shutdown broker because all log dirs in
C:\kafka_logs have failed (kafka.log.LogManager)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
