[jira] [Created] (KAFKA-3457) KafkaConsumer.committed(...) hangs forever if port number is wrong
Harald Kirsch created KAFKA-3457: Summary: KafkaConsumer.committed(...) hangs forever if port number is wrong Key: KAFKA-3457 URL: https://issues.apache.org/jira/browse/KAFKA-3457 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.9.0.1 Reporter: Harald Kirsch Create a KafkaConsumer with default settings but with a wrong host:port setting for bootstrap.servers. Have it in some consumer group, do not subscribe or assign partitions. Then call .committed(...) for a topic/partition combination a few times. It will hang on the 2nd or third call forever. In the debug log you will see that it repeats connections all over again. I waited many minutes and it never came back to throw an Exception. The connections problems should at least pop out on the WARNING log level. Likely the connection problems should throw an exception eventually. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete
[ https://issues.apache.org/jira/browse/KAFKA-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harald Kirsch updated KAFKA-3339: - Description: The api documentation for seekToBeginning and seekToEnd in org.apache.kafka.clients.consumer.KafkaConsumer these methods should remark that all subscribed partitions are seeked if no TopicPartition is provided. (was: The api documentation for seekToBeginning, seekToEnd in these methods should remark that all subscribed partitions are seeked if no TopicPartition is provided.) > org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, > seekToEnd incomplete > - > > Key: KAFKA-3339 > URL: https://issues.apache.org/jira/browse/KAFKA-3339 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 0.9.0.1 >Reporter: Harald Kirsch > > The api documentation for seekToBeginning and seekToEnd in > org.apache.kafka.clients.consumer.KafkaConsumer these methods should remark > that all subscribed partitions are seeked if no TopicPartition is provided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete
[ https://issues.apache.org/jira/browse/KAFKA-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harald Kirsch updated KAFKA-3339: - Description: The api documentation for seekToBeginning, seekToEnd in these methods should remark that all subscribed partitions are seeked if no TopicPartition is provided. (was: The api documentation for these methods should remark that all subscribed partitions are seeked if no TopicPartition is provided.) > org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, > seekToEnd incomplete > - > > Key: KAFKA-3339 > URL: https://issues.apache.org/jira/browse/KAFKA-3339 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 0.9.0.1 >Reporter: Harald Kirsch > > The api documentation for seekToBeginning, seekToEnd in these methods should > remark that all subscribed partitions are seeked if no TopicPartition is > provided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete
Harald Kirsch created KAFKA-3339: Summary: org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete Key: KAFKA-3339 URL: https://issues.apache.org/jira/browse/KAFKA-3339 Project: Kafka Issue Type: Improvement Components: clients Affects Versions: 0.9.0.1 Reporter: Harald Kirsch The api documentation for these methods should remark that all subscribed partitions are seeked if no TopicPartition is provided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete
[
https://issues.apache.org/jira/browse/KAFKA-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15182722#comment-15182722
]
Harald Kirsch commented on KAFKA-3339:
--
[~singhashish] A bit too much for a pull request. For both methods I would just
add something along the lines of:
If no {@code TopicPartition} is provided, all topic/partition pairs returned by
{@link #assignment} are repositioned.
> org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning,
> seekToEnd incomplete
> -
>
> Key: KAFKA-3339
> URL: https://issues.apache.org/jira/browse/KAFKA-3339
> Project: Kafka
> Issue Type: Improvement
> Components: clients
>Affects Versions: 0.9.0.1
>Reporter: Harald Kirsch
>
> The api documentation for seekToBeginning and seekToEnd in
> org.apache.kafka.clients.consumer.KafkaConsumer these methods should remark
> that all subscribed partitions are seeked if no TopicPartition is provided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[
https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377023#comment-15377023
]
Harald Kirsch commented on KAFKA-1194:
--
Having a very similar problem I would hope that the fix fixes this one too. The
message and stacktrace is slightly different. We are using the logcleaner with
compaction and get the below stack trace.
This is on Windows. The claim of the error message that another process has the
file open is misleading. I verified with procexp and handle search that only
the Kafka process has the file open, so it is likely blocking itself on this.
Any chance that the patch will fix this one too?
{noformat}
[2016-07-14 16:09:20,568] ERROR [kafka-log-cleaner-thread-0], Error due to
(kafka.log.LogCleaner)
kafka.common.KafkaStorageException: Failed to change the log file suffix from
.cleaned to .swap for log segment 0
at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:263)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:265)
at kafka.log.Log.replaceSegments(Log.scala:869)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:395)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:343)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:342)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Cleaner.clean(LogCleaner.scala:342)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:237)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:215)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.nio.file.FileSystemException:
d:\Search\kafka\__consumer_offsets-40\.log.cleaned ->
d:\Search\kafka\__consumer_offsets-40\.log.swap: The
process cannot access the file because it is being used by another process.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670)
at kafka.log.FileMessageSet.renameTo(FileMessageSet.scala:364)
... 10 more
Suppressed: java.nio.file.FileSystemException:
d:\Search\kafka\__consumer_offsets-40\.log.cleaned ->
d:\Search\kafka\__consumer_offsets-40\.log.swap: The
process cannot access the file because it is being used by another process.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667)
... 11 more
{noformat}
> The kafka broker cannot delete the old log files after the configured time
> --
>
> Key: KAFKA-1194
> URL: https://issues.apache.org/jira/browse/KAFKA-1194
> Project: Kafka
> Issue Type: Bug
> Components: log
>Affects Versions: 0.8.1
> Environment: window
>Reporter: Tao Qin
>Assignee: Jay Kreps
> Labels: features, patch
> Fix For: 0.10.1.0
>
> Attachments: KAFKA-1194.patch, kafka-1194-v1.patch,
> kafka-1194-v2.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> We tested it in windows environment, and set the log.retention.hours to 24
> hours.
> # The minimum age of a log file to be eligible for deletion
> log.retention.hours=24
> After several days, the kafka broker still cannot delete the old log file.
> And we get the following exceptions:
> [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task
> 'kafka-log-retention' (kafka.utils.KafkaScheduler)
> kafka.common.KafkaStorageException: Failed to change the log file suffix from
> to .deleted for log segment 1516723
> at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249)
> at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638)
> at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629)
> at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418)
> at
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[
https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405810#comment-15405810
]
Harald Kirsch commented on KAFKA-1194:
--
Just stumbled over yet another problem instance. During startup, Kafka notices
a corrupt log/index file and tries to repair it. Here is the stack trace:
{noformat}
[2016-08-03 13:56:17,467] INFO Found log file
d:\Search\kafka\fileshare-1\.log.swap from interrupted swap
operation, repairing. (kafka.log.Log)
[2016-08-03 13:56:18,436] ERROR There was an error in one of the threads during
logs loading: kafka.common.KafkaStorageException: Failed to change the index
file suffix from .swap to for log segment 0 (kafka.log.LogManager)
[2016-08-03 13:56:18,436] FATAL Fatal error during KafkaServer startup. Prepare
to shutdown (kafka.server.KafkaServer)
kafka.common.KafkaStorageException: Failed to change the index file suffix from
.swap to for log segment 0
at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:268)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:274)
at kafka.log.Log.replaceSegments(Log.scala:886)
at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:230)
at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:214)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
at kafka.log.Log.loadSegments(Log.scala:214)
at kafka.log.Log.(Log.scala:101)
at
kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:151)
at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:56)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.file.FileSystemException:
d:\Search\kafka\fileshare-1\.index.swap ->
d:\Search\kafka\fileshare-1\.index: The process cannot
access the file because it is being used by another process.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670)
at kafka.log.OffsetIndex.renameTo(OffsetIndex.scala:365)
... 14 more
Suppressed: java.nio.file.FileSystemException:
d:\Search\kafka\fileshare-1\.index.swap ->
d:\Search\kafka\fileshare-1\.index: The process cannot
access the file because it is being used by another process.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667)
... 15 more
[2016-08-03 13:56:18,451] INFO shutting down (kafka.server.KafkaServer)
[2016-08-03 13:56:18,467] INFO shut down completed (kafka.server.KafkaServer)
{noformat}
> The kafka broker cannot delete the old log files after the configured time
> --
>
> Key: KAFKA-1194
> URL: https://issues.apache.org/jira/browse/KAFKA-1194
> Project: Kafka
> Issue Type: Bug
> Components: log
>Affects Versions: 0.8.1
> Environment: window
>Reporter: Tao Qin
>Assignee: Jay Kreps
> Labels: features, patch
> Fix For: 0.10.1.0
>
> Attachments: KAFKA-1194.patch, kafka-1194-v1.patch,
> kafka-1194-v2.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> We tested it in windows environment, and set the log.retention.hours to 24
> hours.
> # The minimum age of a log file to be eligible for deletion
> log.retention.hours=24
> After several days, the kafka broker still cannot delete the old log file.
> And we get the following exceptions:
> [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task
> 'kafka-log-retention' (kafka.utils.KafkaScheduler)
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[
https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379380#comment-15379380
]
Harald Kirsch commented on KAFKA-1194:
--
It seems we are one step further but not yet there. I just cloned the master,
pulled in 1624.patch, installed and ran kafka on an existing log. The message
has changed from not being able to rename the log file to not being able to
rename the index file. Here is the full stack trace from the LogCleaner.
{noformat}
[2016-07-15 15:33:09,622] ERROR [kafka-log-cleaner-thread-0], Error due to
(kafka.log.LogCleaner)
kafka.common.KafkaStorageException: Failed to change the index file suffix from
to .deleted for log segment 0
at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:268)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:274)
at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:837)
at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:883)
at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:878)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Log.replaceSegments(Log.scala:878)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:395)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:343)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:342)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Cleaner.clean(LogCleaner.scala:342)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:237)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:215)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.nio.file.FileSystemException:
d:\Search\kafka\windream-9\.index ->
d:\Search\kafka\windream-9\.index.deleted: The process
cannot access the file because it is being used by another process.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670)
at kafka.log.OffsetIndex.renameTo(OffsetIndex.scala:365)
... 14 more
Suppressed: java.nio.file.FileSystemException:
d:\Search\kafka\windream-9\.index ->
d:\Search\kafka\windream-9\.index.deleted: The process
cannot access the file because it is being used by another process.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667)
... 15 more
[2016-07-15 15:33:09,622] INFO [kafka-log-cleaner-thread-0], Stopped
(kafka.log.LogCleaner)
{noformat}
Indeed the overly long {{.log}} file is now {{.log.deleted}}. There are
{{.log.swap}} and {{.index.swap}} and still just {{.index}}.
> The kafka broker cannot delete the old log files after the configured time
> --
>
> Key: KAFKA-1194
> URL: https://issues.apache.org/jira/browse/KAFKA-1194
> Project: Kafka
> Issue Type: Bug
> Components: log
>Affects Versions: 0.8.1
> Environment: window
>Reporter: Tao Qin
>Assignee: Jay Kreps
> Labels: features, patch
> Fix For: 0.10.1.0
>
> Attachments: KAFKA-1194.patch, kafka-1194-v1.patch,
> kafka-1194-v2.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> We tested it in windows environment, and set the log.retention.hours to 24
> hours.
> # The minimum age of a log file to be eligible for deletion
> log.retention.hours=24
> After several days, the kafka broker still cannot delete the old log file.
> And we get the following exceptions:
> [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task
> 'kafka-log-retention' (kafka.utils.KafkaScheduler)
> kafka.common.KafkaStorageException: Failed to change the log file suffix from
> to .deleted for log segment 1516723
> at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249)
>
[jira] [Comment Edited] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[
https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379380#comment-15379380
]
Harald Kirsch edited comment on KAFKA-1194 at 7/15/16 1:41 PM:
---
It seems we are one step further but not yet there. I just cloned the master,
pulled in 1624.patch, installed and ran kafka on an existing log. The message
has changed from not being able to rename the log file to not being able to
rename the index file. Here is the full stack trace from the LogCleaner.
{noformat}
[2016-07-15 15:33:09,622] ERROR [kafka-log-cleaner-thread-0], Error due to
(kafka.log.LogCleaner)
kafka.common.KafkaStorageException: Failed to change the index file suffix from
to .deleted for log segment 0
at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:268)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:274)
at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:837)
at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:883)
at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:878)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Log.replaceSegments(Log.scala:878)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:395)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:343)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:342)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Cleaner.clean(LogCleaner.scala:342)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:237)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:215)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.nio.file.FileSystemException:
d:\Search\kafka\windream-9\.index ->
d:\Search\kafka\windream-9\.index.deleted: The process
cannot access the file because it is being used by another process.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670)
at kafka.log.OffsetIndex.renameTo(OffsetIndex.scala:365)
... 14 more
Suppressed: java.nio.file.FileSystemException:
d:\Search\kafka\windream-9\.index ->
d:\Search\kafka\windream-9\.index.deleted: The process
cannot access the file because it is being used by another process.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667)
... 15 more
[2016-07-15 15:33:09,622] INFO [kafka-log-cleaner-thread-0], Stopped
(kafka.log.LogCleaner)
{noformat}
Indeed the overly long {{.log}} file is now {{.log.deleted}}. There are
{{.log.swap}} and {{.index.swap}} and still just {{.index}}.
For reference, the server.log lists:
{noformat}
memorymapped.file.updates.enabled = false
{noformat}
was (Author: haraldk):
It seems we are one step further but not yet there. I just cloned the master,
pulled in 1624.patch, installed and ran kafka on an existing log. The message
has changed from not being able to rename the log file to not being able to
rename the index file. Here is the full stack trace from the LogCleaner.
{noformat}
[2016-07-15 15:33:09,622] ERROR [kafka-log-cleaner-thread-0], Error due to
(kafka.log.LogCleaner)
kafka.common.KafkaStorageException: Failed to change the index file suffix from
to .deleted for log segment 0
at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:268)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:274)
at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:837)
at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:883)
at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:878)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Log.replaceSegments(Log.scala:878)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:395)
at
[jira] [Commented] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows
[ https://issues.apache.org/jira/browse/KAFKA-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515615#comment-15515615 ] Harald Kirsch commented on KAFKA-2170: -- [~soumyajitsahu] This is great news. I assume the pull request goes agains the most recent trunk. I will try this out no. > 10 LogTest cases failed for file.renameTo failed under windows > --- > > Key: KAFKA-2170 > URL: https://issues.apache.org/jira/browse/KAFKA-2170 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.10.1.0 > Environment: Windows >Reporter: Honghai Chen >Assignee: Jay Kreps > > get latest code from trunk, then run test > gradlew -i core:test --tests kafka.log.LogTest > Got 10 cases failed for same reason: > kafka.common.KafkaStorageException: Failed to change the log file suffix from > to .deleted for log segment 0 > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:259) > at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:756) > at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:747) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:514) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:514) > at scala.collection.immutable.List.foreach(List.scala:318) > at kafka.log.Log.deleteOldSegments(Log.scala:514) > at kafka.log.LogTest.testAsyncDelete(LogTest.scala:633) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41) > at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) > at org.junit.runners.ParentRunner.run(ParentRunner.java:220) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:48) > at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) > at > org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) > at > org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) > at $Proxy2.processTestClass(Unknown Source) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:105) > at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) > at > org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at >
[jira] [Commented] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows
[
https://issues.apache.org/jira/browse/KAFKA-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515792#comment-15515792
]
Harald Kirsch commented on KAFKA-2170:
--
Better, but not yet there, it seems. Here is what I did.
I applied the .diff file of the 1757 pull request as downloaded from github
with the patch command to 106a7456060750ab0604d290b8c1e33a57adbf20 from
http://git-wip-us.apache.org/repos/asf/kafka.git. It ran in just fin.
Build the tgz, put it on my test machine, ran kafka with these settings:
{code}
log.segment.bytes=6000111
log.cleaner.enable=true
log.cleanup.policy=compact
log.cleaner.min.cleanable.ratio=0.01
log.cleaner.backoff.ms=15000
log.segment.delete.delay.ms=600
auto.create.topics.enable=false
{code}
Ran in around 75 files as binary blobs between 10k and 6M in size twice. The
cleanup triggered and worked just fine.
Tried this a few times more, also with running the files in in quick succession
and it worked just fine.
Stopped Kafka with CTRL-C, it shut down nicely an restarted (mentioning just
for completeness, not sure whether this is relevant).
Tried this a few more times, running the files in roughly 15 times in quick
succession and it bombed out as shown follows:
{code}
kafka.common.KafkaStorageException: Failed to change the log file suffix from
to .deleted for log segment 695
at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:327)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:329)
at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:956)
at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:1002)
at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:997)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Log.replaceSegments(Log.scala:997)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:425)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:364)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:363)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Cleaner.clean(LogCleaner.scala:363)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:239)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:218)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.nio.file.FileAlreadyExistsException:
C:\Users\hk\tmp\kafka-data\hktest-0\0695.log ->
C:\Users\hk\tmp\kafka-data\hktest-0\0695.log.deleted
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:81)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670)
at kafka.log.FileMessageSet.renameTo(FileMessageSet.scala:431)
... 14 more
Suppressed: java.nio.file.AccessDeniedException:
C:\Users\hk\tmp\kafka-data\hktest-0\0695.log ->
C:\Users\hk\tmp\kafka-data\hktest-0\0695.log.deleted
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667)
... 15 more
[2016-09-23 10:12:36,254] INFO [kafka-log-cleaner-thread-0], Stopped
(kafka.log.LogCleaner)
{code}
Let me know if I need to provide more information or run a different experiment.
> 10 LogTest cases failed for file.renameTo failed under windows
> ---
>
> Key: KAFKA-2170
> URL: https://issues.apache.org/jira/browse/KAFKA-2170
> Project: Kafka
> Issue Type: Bug
> Components: log
>Affects Versions: 0.10.1.0
> Environment: Windows
>Reporter: Honghai Chen
>Assignee: Jay Kreps
>
> get latest code from trunk, then run test
> gradlew -i core:test --tests kafka.log.LogTest
> Got 10 cases failed for same reason:
> kafka.common.KafkaStorageException: Failed to change the log file suffix from
> to .deleted for log segment 0
> at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:259)
> at
[jira] [Comment Edited] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows
[
https://issues.apache.org/jira/browse/KAFKA-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515792#comment-15515792
]
Harald Kirsch edited comment on KAFKA-2170 at 9/23/16 8:37 AM:
---
Better, but not yet there, it seems. Here is what I did.
I applied the .diff file of the 1757 pull request as downloaded from github
with the patch command to 106a7456060750ab0604d290b8c1e33a57adbf20 from
http://git-wip-us.apache.org/repos/asf/kafka.git. It ran in just fin.
Build the tgz, put it on my test machine, ran kafka with these settings:
{code}
log.segment.bytes=6000111
log.cleaner.enable=true
log.cleanup.policy=compact
log.cleaner.min.cleanable.ratio=0.01
log.cleaner.backoff.ms=15000
log.segment.delete.delay.ms=600
auto.create.topics.enable=false
{code}
Ran in around 75 files as binary blobs between 10k and 6M in size twice. The
cleanup triggered and worked just fine.
Tried this a few times more, also with running the files in in quick succession
and it worked just fine.
Stopped Kafka with CTRL-C, it shut down nicely an restarted (mentioning just
for completeness, not sure whether this is relevant).
Tried this a few more times, running the files in roughly 15 times in quick
succession and it bombed out as shown follows:
{code}
kafka.common.KafkaStorageException: Failed to change the log file suffix from
to .deleted for log segment 695
at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:327)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:329)
at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:956)
at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:1002)
at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:997)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Log.replaceSegments(Log.scala:997)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:425)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:364)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:363)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Cleaner.clean(LogCleaner.scala:363)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:239)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:218)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.nio.file.FileAlreadyExistsException:
C:\Users\hk\tmp\kafka-data\hktest-0\0695.log ->
C:\Users\hk\tmp\kafka-data\hktest-0\0695.log.deleted
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:81)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670)
at kafka.log.FileMessageSet.renameTo(FileMessageSet.scala:431)
... 14 more
Suppressed: java.nio.file.AccessDeniedException:
C:\Users\hk\tmp\kafka-data\hktest-0\0695.log ->
C:\Users\hk\tmp\kafka-data\hktest-0\0695.log.deleted
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667)
... 15 more
[2016-09-23 10:12:36,254] INFO [kafka-log-cleaner-thread-0], Stopped
(kafka.log.LogCleaner)
{code}
Afterwards I restartet Kafka, which worked without complaints. I ran one round
on the 75 files and the logcleaner cleaned up just fine, so it looks like
you're pretty close to a working solution.
Let me know if I need to provide more information or run a different experiment.
was (Author: haraldk):
Better, but not yet there, it seems. Here is what I did.
I applied the .diff file of the 1757 pull request as downloaded from github
with the patch command to 106a7456060750ab0604d290b8c1e33a57adbf20 from
http://git-wip-us.apache.org/repos/asf/kafka.git. It ran in just fin.
Build the tgz, put it on my test machine, ran kafka with these settings:
{code}
log.segment.bytes=6000111
log.cleaner.enable=true
log.cleanup.policy=compact
log.cleaner.min.cleanable.ratio=0.01
log.cleaner.backoff.ms=15000
log.segment.delete.delay.ms=600
auto.create.topics.enable=false
[jira] [Commented] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows
[
https://issues.apache.org/jira/browse/KAFKA-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426278#comment-15426278
]
Harald Kirsch commented on KAFKA-2170:
--
The patch does not seem to work completely for compaction. I applied the patch
on 40b1dd3f495a59abef8a0cba5450526994c92c04, ran ./gradlew releaseTarGz and
unpacked on a windows 8.1. I ran the server with the following configs:
{noformat}
broker.id=0
listeners=PLAINTEXT://:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=C:\\Users\\hk\\tmp\\kafka-data
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=6000111
log.retention.check.interval.ms=30
zookeeper.connect=hal.intranet.raytion.com:2181/kafka
zookeeper.connection.timeout.ms=6000
log.cleaner.enable=true
log.cleanup.policy=compact
log.cleaner.min.cleanable.ratio=0.01
log.cleaner.backoff.ms=15000
log.segment.delete.delay.ms=600
auto.create.topics.enable=false
{noformat}
Then I fed the same set of docs twice. The cleaner logged a successful
activity, but the folder now contains all {{.index.deleted}} files like this:
{noformat}
ModeLastWriteTime Length Name
- --
-a---18.08.2016 12:33 0 .index
-18.08.2016 12:33176 .index.deleted
-a---18.08.2016 12:33 0 .log
-a---18.08.2016 12:33 0 0023.index
-18.08.2016 12:33192 0023.index.deleted
-a---18.08.2016 12:33 583752 0023.log
-a---18.08.2016 12:33 48 0048.index
-18.08.2016 12:33192 0048.index.deleted
-a---18.08.2016 12:335844328 0048.log
-a---18.08.2016 12:33 48 0073.index
-18.08.2016 12:33200 0073.index.deleted
-a---18.08.2016 12:335494169 0073.log
-a---18.08.2016 12:33 10485760 0099.index
-a---18.08.2016 12:331529831 0099.log
{noformat}
Then I fed the same set of messages a third time. As soon as the cleaner
started working thereafter, it bombed out like this:
{noformat}
[2016-08-18 13:10:43,254] ERROR [kafka-log-cleaner-thread-0], Error due to
(kafka.log.LogCleaner)
kafka.common.KafkaStorageException: Failed to change the index file suffix from
to .deleted for log segment 0
at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:263)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:269)
at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:832)
at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:878)
at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:873)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Log.replaceSegments(Log.scala:873)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:395)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:343)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:342)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Cleaner.clean(LogCleaner.scala:342)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:237)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:215)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.nio.file.FileAlreadyExistsException:
C:\Users\hk\tmp\kafka-data\hktest-0\.index ->
C:\Users\hk\tmp\kafka-data\hktest-0\.index.deleted
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:81)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670)
at kafka.log.OffsetIndex.renameTo(OffsetIndex.scala:364)
... 14 more
Suppressed: java.nio.file.AccessDeniedException:
C:\Users\hk\tmp\kafka-data\hktest-0\.index ->
C:\Users\hk\tmp\kafka-data\hktest-0\.index.deleted
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at
[jira] [Commented] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows
[ https://issues.apache.org/jira/browse/KAFKA-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438668#comment-15438668 ] Harald Kirsch commented on KAFKA-2170: -- Tried to added the patches. Since I had the 1757 in already, I only added the 1716 (one liner). The 1718 was in my code already too (prevent null DirectBuffer use). The test results are exactly as in my last comment, namely that the .index.deleted files are not really deleted (with no error message). The next time the cleaner runs, it complains about not being able to rename to the already existing file (no surprise). So the remaining problem seems to that even with a low log.segment.delete.delay.ms the files are not deleted. And also their counterparts without .delete, despite being empty, are not deleted. > 10 LogTest cases failed for file.renameTo failed under windows > --- > > Key: KAFKA-2170 > URL: https://issues.apache.org/jira/browse/KAFKA-2170 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.10.1.0 > Environment: Windows >Reporter: Honghai Chen >Assignee: Jay Kreps > > get latest code from trunk, then run test > gradlew -i core:test --tests kafka.log.LogTest > Got 10 cases failed for same reason: > kafka.common.KafkaStorageException: Failed to change the log file suffix from > to .deleted for log segment 0 > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:259) > at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:756) > at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:747) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:514) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:514) > at scala.collection.immutable.List.foreach(List.scala:318) > at kafka.log.Log.deleteOldSegments(Log.scala:514) > at kafka.log.LogTest.testAsyncDelete(LogTest.scala:633) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41) > at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) > at org.junit.runners.ParentRunner.run(ParentRunner.java:220) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:48) > at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) > at > org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) > at > org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) > at $Proxy2.processTestClass(Unknown Source) > at >
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532092#comment-15532092 ] Harald Kirsch commented on KAFKA-1194: -- [~abhit011] The patch described in https://issues.apache.org/jira/browse/KAFKA-2170?focusedCommentId=15515023=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15515023 nearly worked for me, but not quite. Maybe it furthers the issue if you could try the patch as well on a test system and report your findings. > The kafka broker cannot delete the old log files after the configured time > -- > > Key: KAFKA-1194 > URL: https://issues.apache.org/jira/browse/KAFKA-1194 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.8.1 > Environment: window >Reporter: Tao Qin >Assignee: Jay Kreps > Labels: features, patch > Fix For: 0.10.2.0 > > Attachments: KAFKA-1194.patch, kafka-1194-v1.patch, > kafka-1194-v2.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > We tested it in windows environment, and set the log.retention.hours to 24 > hours. > # The minimum age of a log file to be eligible for deletion > log.retention.hours=24 > After several days, the kafka broker still cannot delete the old log file. > And we get the following exceptions: > [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task > 'kafka-log-retention' (kafka.utils.KafkaScheduler) > kafka.common.KafkaStorageException: Failed to change the log file suffix from > to .deleted for log segment 1516723 > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249) > at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638) > at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at > scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) > at scala.collection.immutable.List.foreach(List.scala:76) > at kafka.log.Log.deleteOldSegments(Log.scala:418) > at > kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:284) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:316) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:314) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:743) > at scala.collection.Iterator$class.foreach(Iterator.scala:772) > at > scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:73) > at > scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615) > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:742) > at kafka.log.LogManager.cleanupLogs(LogManager.scala:314) > at > kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:143) > at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > I think this error happens because kafka tries to rename the log file when it > is still opened. So we should close the file first before rename. > The index file uses a special data structure, the MappedByteBuffer. Javadoc > describes it as: > A mapped byte buffer and the file mapping that it represents remain valid > until the buffer itself is garbage-collected. > Fortunately, I find a forceUnmap function in kafka code, and perhaps it can > be used to free the MappedByteBuffer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15701308#comment-15701308 ] Harald Kirsch commented on KAFKA-1194: -- See https://issues.apache.org/jira/browse/KAFKA-2170?focusedCommentId=15515792=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15515792 for a description of how I applied a patch (and which) and what I experienced. Maybe it works good enough to help you. But surely try this on a test system first. > The kafka broker cannot delete the old log files after the configured time > -- > > Key: KAFKA-1194 > URL: https://issues.apache.org/jira/browse/KAFKA-1194 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.8.1 > Environment: window >Reporter: Tao Qin >Assignee: Jay Kreps > Labels: features, patch > Fix For: 0.10.2.0 > > Attachments: KAFKA-1194.patch, kafka-1194-v1.patch, > kafka-1194-v2.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > We tested it in windows environment, and set the log.retention.hours to 24 > hours. > # The minimum age of a log file to be eligible for deletion > log.retention.hours=24 > After several days, the kafka broker still cannot delete the old log file. > And we get the following exceptions: > [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task > 'kafka-log-retention' (kafka.utils.KafkaScheduler) > kafka.common.KafkaStorageException: Failed to change the log file suffix from > to .deleted for log segment 1516723 > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249) > at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638) > at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at > scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) > at scala.collection.immutable.List.foreach(List.scala:76) > at kafka.log.Log.deleteOldSegments(Log.scala:418) > at > kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:284) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:316) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:314) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:743) > at scala.collection.Iterator$class.foreach(Iterator.scala:772) > at > scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:73) > at > scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615) > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:742) > at kafka.log.LogManager.cleanupLogs(LogManager.scala:314) > at > kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:143) > at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > I think this error happens because kafka tries to rename the log file when it > is still opened. So we should close the file first before rename. > The index file uses a special data structure, the MappedByteBuffer. Javadoc > describes it as: > A mapped byte buffer and the file mapping that it represents remain valid > until the buffer itself is garbage-collected. > Fortunately, I find a forceUnmap function in kafka code, and perhaps it can > be used to free the MappedByteBuffer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-4502) Exception during startup, append offset to swap file
Harald Kirsch created KAFKA-4502:
Summary: Exception during startup, append offset to swap file
Key: KAFKA-4502
URL: https://issues.apache.org/jira/browse/KAFKA-4502
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.10.2.0
Environment: Windows Server
Reporter: Harald Kirsch
During startup, the Kafka server throws the exception shown below with a bit of
pre-context.
We are using the so-called SiphonRelease
(https://github.com/Microsoft/Kafka/tree/SiphonRelease,
https://issues.apache.org/jira/browse/KAFKA-1194?focusedCommentId=15702991)
which tries to circumvent problems of the logCleaner to rename and delete
segments still memory mapped by the broker.
The trouble seems to be as follows: since in the SiphonRelease the LogCleaner
still sometimes crashes, we have a monitoring script that detects this and then
restarts the Windows Service (apache procrun based) running Kafka. My hunch is
that the combination of restart-service/procrun does not allow Kafka to shut
down smoothly, since when it starts we get tons of messages like:
{noformat}
[2016-12-05 23:30:20,704] WARN Found a corrupted index file due to requirement
failed: Corrupt index found, index file
(d:\Search\kafka\fileshare-0\00084814.index) has non-zero size but
the last offset is 84814 which is no larger than the base offset 84814.}.
deleting d:\Search\kafka\fileshare-0\00084814.timeindex,
d:\Search\kafka\fileshare-0\00084814.index and rebuilding index...
(kafka.log.Log)
{noformat}
While this seems fixable by Kafka, my hunch is that a leftover .swap file then
breaks it as follows:
{noformat}
[2016-12-05 23:32:34,676] INFO Found log file
d:\Search\kafka\windream-4\.log.swap from interrupted swap
operation, repairing. (kafka.log.Log)
[2016-12-05 23:32:34,957] ERROR There was an error in one of the threads during
logs loading: kafka.common.InvalidOffsetException: Attempt to append an offset
(110460) to position 182 no larger than the last offset appended (110735) to
d:\Search\kafka\windream-4\.index.swap.
(kafka.log.LogManager)
[2016-12-05 23:32:34,957] FATAL Fatal error during KafkaServer startup. Prepare
to shutdown (kafka.server.KafkaServer)
kafka.common.InvalidOffsetException: Attempt to append an offset (110460) to
position 182 no larger than the last offset appended (110735) to
d:\Search\kafka\windream-4\.index.swap.
at
kafka.log.OffsetIndex$$anonfun$append$1.apply$mcV$sp(OffsetIndex.scala:132)
at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:122)
at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:122)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.log.OffsetIndex.append(OffsetIndex.scala:122)
at kafka.log.LogSegment.recover(LogSegment.scala:224)
at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:248)
at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:232)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
at kafka.log.Log.loadSegments(Log.scala:232)
at kafka.log.Log.(Log.scala:108)
at
kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:151)
at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:58)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-4502) Exception during startup, append offset to swap file
[
https://issues.apache.org/jira/browse/KAFKA-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harald Kirsch updated KAFKA-4502:
-
Labels: (was: log)
> Exception during startup, append offset to swap file
>
>
> Key: KAFKA-4502
> URL: https://issues.apache.org/jira/browse/KAFKA-4502
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.10.2.0
> Environment: Windows Server
>Reporter: Harald Kirsch
>
> During startup, the Kafka server throws the exception shown below with a bit
> of pre-context.
> We are using the so-called SiphonRelease
> (https://github.com/Microsoft/Kafka/tree/SiphonRelease,
> https://issues.apache.org/jira/browse/KAFKA-1194?focusedCommentId=15702991)
> which tries to circumvent problems of the logCleaner to rename and delete
> segments still memory mapped by the broker.
> The trouble seems to be as follows: since in the SiphonRelease the LogCleaner
> still sometimes crashes, we have a monitoring script that detects this and
> then restarts the Windows Service (apache procrun based) running Kafka. My
> hunch is that the combination of restart-service/procrun does not allow Kafka
> to shut down smoothly, since when it starts we get tons of messages like:
> {noformat}
> [2016-12-05 23:30:20,704] WARN Found a corrupted index file due to
> requirement failed: Corrupt index found, index file
> (d:\Search\kafka\fileshare-0\00084814.index) has non-zero size
> but the last offset is 84814 which is no larger than the base offset 84814.}.
> deleting d:\Search\kafka\fileshare-0\00084814.timeindex,
> d:\Search\kafka\fileshare-0\00084814.index and rebuilding
> index... (kafka.log.Log)
> {noformat}
> While this seems fixable by Kafka, my hunch is that a leftover .swap file
> then breaks it as follows:
> {noformat}
> [2016-12-05 23:32:34,676] INFO Found log file
> d:\Search\kafka\windream-4\.log.swap from interrupted
> swap operation, repairing. (kafka.log.Log)
> [2016-12-05 23:32:34,957] ERROR There was an error in one of the threads
> during logs loading: kafka.common.InvalidOffsetException: Attempt to append
> an offset (110460) to position 182 no larger than the last offset appended
> (110735) to d:\Search\kafka\windream-4\.index.swap.
> (kafka.log.LogManager)
> [2016-12-05 23:32:34,957] FATAL Fatal error during KafkaServer startup.
> Prepare to shutdown (kafka.server.KafkaServer)
> kafka.common.InvalidOffsetException: Attempt to append an offset (110460) to
> position 182 no larger than the last offset appended (110735) to
> d:\Search\kafka\windream-4\.index.swap.
> at
> kafka.log.OffsetIndex$$anonfun$append$1.apply$mcV$sp(OffsetIndex.scala:132)
> at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:122)
> at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:122)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.log.OffsetIndex.append(OffsetIndex.scala:122)
> at kafka.log.LogSegment.recover(LogSegment.scala:224)
> at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:248)
> at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:232)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
> at kafka.log.Log.loadSegments(Log.scala:232)
> at kafka.log.Log.(Log.scala:108)
> at
> kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:151)
> at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:58)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[
https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712362#comment-15712362
]
Harald Kirsch commented on KAFKA-1194:
--
[~soumyajitsahu] Here are the server configs as logged:
{code}
[2016-11-29 16:23:13,731] INFO KafkaConfig values:
advertised.host.name = null
advertised.listeners = null
advertised.port = null
authorizer.class.name =
auto.create.topics.enable = false
auto.leader.rebalance.enable = true
background.threads = 10
broker.id = 0
broker.id.generation.enable = true
broker.rack = null
compression.type = producer
connections.max.idle.ms = 60
controlled.shutdown.enable = true
controlled.shutdown.max.retries = 3
controlled.shutdown.retry.backoff.ms = 5000
controller.socket.timeout.ms = 3
default.replication.factor = 1
delete.topic.enable = false
fetch.purgatory.purge.interval.requests = 1000
group.max.session.timeout.ms = 30
group.min.session.timeout.ms = 6000
host.name =
inter.broker.protocol.version = 0.10.1-IV2
leader.imbalance.check.interval.seconds = 300
leader.imbalance.per.broker.percentage = 10
listeners = PLAINTEXT://:9092
log.cleaner.backoff.ms = 15000
log.cleaner.dedupe.buffer.size = 134217728
log.cleaner.delete.retention.ms = 8640
log.cleaner.enable = true
log.cleaner.io.buffer.load.factor = 0.9
log.cleaner.io.buffer.size = 524288
log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
log.cleaner.min.cleanable.ratio = 0.1
log.cleaner.min.compaction.lag.ms = 0
log.cleaner.threads = 1
log.cleanup.policy = [compact]
log.dir = /tmp/kafka-logs
log.dirs = d:\Search\kafka
log.flush.interval.messages = 9223372036854775807
log.flush.interval.ms = null
log.flush.offset.checkpoint.interval.ms = 6
log.flush.scheduler.interval.ms = 9223372036854775807
log.index.interval.bytes = 4096
log.index.size.max.bytes = 10485760
log.message.format.version = 0.10.1-IV2
log.message.timestamp.difference.max.ms = 9223372036854775807
log.message.timestamp.type = CreateTime
log.preallocate = false
log.retention.bytes = -1
log.retention.check.interval.ms = 300100
log.retention.hours = 168
log.retention.minutes = null
log.retention.ms = null
log.roll.hours = 24
log.roll.jitter.hours = 0
log.roll.jitter.ms = null
log.roll.ms = null
log.segment.bytes = 200111000
log.segment.delete.delay.ms = 6
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides =
message.max.bytes = 20999000
metric.reporters = []
metrics.num.samples = 2
metrics.sample.window.ms = 3
min.insync.replicas = 1
num.io.threads = 36
num.network.threads = 36
num.partitions = 1
num.recovery.threads.per.data.dir = 1
num.replica.fetchers = 1
offset.metadata.max.bytes = 4096
offsets.commit.required.acks = -1
offsets.commit.timeout.ms = 5000
offsets.load.buffer.size = 5242880
offsets.retention.check.interval.ms = 60
offsets.retention.minutes = 2147483
offsets.topic.compression.codec = 0
offsets.topic.num.partitions = 50
offsets.topic.replication.factor = 3
offsets.topic.segment.bytes = 104857600
port = 9092
principal.builder.class = class
org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
producer.purgatory.purge.interval.requests = 1000
queued.max.requests = 500
quota.consumer.default = 9223372036854775807
quota.producer.default = 9223372036854775807
quota.window.num = 11
quota.window.size.seconds = 1
replica.fetch.backoff.ms = 1000
replica.fetch.max.bytes = 20999000
replica.fetch.min.bytes = 1
replica.fetch.response.max.bytes = 10485760
replica.fetch.wait.max.ms = 500
replica.high.watermark.checkpoint.interval.ms = 5000
replica.lag.time.max.ms = 1
replica.socket.receive.buffer.bytes = 65536
replica.socket.timeout.ms = 3
replication.quota.throttled.rate = 9223372036854775807
replication.quota.window.num = 11
replication.quota.window.size.seconds = 1
request.timeout.ms = 32
reserved.broker.max.id = 1000
sasl.enabled.mechanisms = [GSSAPI]
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 6
sasl.kerberos.principal.to.local.rules = [DEFAULT]
sasl.kerberos.service.name = null
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[
https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708281#comment-15708281
]
Harald Kirsch commented on KAFKA-1194:
--
The report of success was slightly exaggerted :-( After several hours of
flawless operation, cleaner threads manage to trip over their own feed, it
seems, with a variety of exceptions. I have seen this now with a single cleaner
thread as well as with 4 threads. With 4 threads, I got variant 1 below 2 times
then variant 2, then variant three but all within 10 seconds. I wonder in what
kind of mood (mode) the system is that this all happens within a few seconds.
Let me know if I can help with more information, details of the setup,
configuration details or experiments, debug logging.
Variant 1:
{noformat}
[2016-11-30 08:26:26,576] ERROR [kafka-log-cleaner-thread-1], Error due to
(kafka.log.LogCleaner)
kafka.common.KafkaStorageException: Failed to change the log file suffix from
to .deleted for log segment 58972
at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:327)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:329)
at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:956)
at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:1002)
at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:997)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Log.replaceSegments(Log.scala:997)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:425)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:364)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:363)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Cleaner.clean(LogCleaner.scala:363)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:239)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:218)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.nio.file.FileAlreadyExistsException:
d:\Search\kafka\fileshare-10\00058972.log ->
d:\Search\kafka\fileshare-10\00058972.log.deleted
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:81)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670)
at kafka.log.FileMessageSet.renameTo(FileMessageSet.scala:431)
... 14 more
Suppressed: java.nio.file.AccessDeniedException:
d:\Search\kafka\fileshare-10\00058972.log ->
d:\Search\kafka\fileshare-10\00058972.log.deleted
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667)
... 15 more
{noformat}
Variant 2:
{noformat}
[2016-11-30 08:26:30,467] ERROR [kafka-log-cleaner-thread-3], Error due to
(kafka.log.LogCleaner)
kafka.common.InvalidOffsetException: Attempt to append an offset (59264) to
position 61 no larger than the last offset appended (66994) to
d:\Search\kafka\fileshare-10\.index.swap.cleaned.
at
kafka.log.OffsetIndex$$anonfun$append$1.apply$mcV$sp(OffsetIndex.scala:132)
at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:122)
at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:122)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.log.OffsetIndex.append(OffsetIndex.scala:122)
at kafka.log.LogSegment.append(LogSegment.scala:105)
at kafka.log.Cleaner.cleanInto(LogCleaner.scala:504)
at
kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:404)
at
kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:400)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:400)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:364)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:363)
at scala.collection.immutable.List.foreach(List.scala:318)
at
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705770#comment-15705770 ] Harald Kirsch commented on KAFKA-1194: -- Now I tried exactly the SiphonRelease mentioned three comments up and it works great. So great indeed that after testing on QA I could move it to our production where it is happily eating away outdated segments while the rest of the operations is going on just fine. Many thanks to [~soumyajitsahu] for providing this release. > The kafka broker cannot delete the old log files after the configured time > -- > > Key: KAFKA-1194 > URL: https://issues.apache.org/jira/browse/KAFKA-1194 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.8.1 > Environment: window >Reporter: Tao Qin >Assignee: Jay Kreps > Labels: features, patch > Fix For: 0.10.2.0 > > Attachments: KAFKA-1194.patch, Untitled.jpg, kafka-1194-v1.patch, > kafka-1194-v2.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > We tested it in windows environment, and set the log.retention.hours to 24 > hours. > # The minimum age of a log file to be eligible for deletion > log.retention.hours=24 > After several days, the kafka broker still cannot delete the old log file. > And we get the following exceptions: > [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task > 'kafka-log-retention' (kafka.utils.KafkaScheduler) > kafka.common.KafkaStorageException: Failed to change the log file suffix from > to .deleted for log segment 1516723 > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249) > at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638) > at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at > scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) > at scala.collection.immutable.List.foreach(List.scala:76) > at kafka.log.Log.deleteOldSegments(Log.scala:418) > at > kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:284) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:316) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:314) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:743) > at scala.collection.Iterator$class.foreach(Iterator.scala:772) > at > scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:73) > at > scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615) > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:742) > at kafka.log.LogManager.cleanupLogs(LogManager.scala:314) > at > kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:143) > at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > I think this error happens because kafka tries to rename the log file when it > is still opened. So we should close the file first before rename. > The index file uses a special data structure, the MappedByteBuffer. Javadoc > describes it as: > A mapped byte buffer and the file mapping that it represents remain valid > until the buffer itself is garbage-collected. > Fortunately, I find a forceUnmap function in kafka code, and perhaps it can > be used to free the MappedByteBuffer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
