[
https://issues.apache.org/jira/browse/KAFKA-3330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Lin updated KAFKA-3330:
----------------------------
Description:
Were getting a number of failures of the log compaction thread with the
following error:
2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Beginning cleaning of log
__consumer_offsets-93.
2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Building offset map for
__consumer_offsets-93...
2016/02/02 00:13:59.048 [LogCleaner] Cleaner 0: Building offset map for log
__consumer_offsets-93 for 2 segments in offset range [11951210572,
11952632314).
2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Error
due to
java.lang.IllegalArgumentException: requirement failed: Last clean offset
is 11951210572 but segment base offset is 11950300163 for log
__consumer_offsets-93.
at scala.Predef$.require(Predef.scala:233) ~[scala-library-2.10.4.jar:?]
at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:561)
~[kafka_2.10-0.8.2.56.jar:?]
at kafka.log.Cleaner.clean(LogCleaner.scala:306)
~[kafka_2.10-0.8.2.56.jar:?]
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:217)
~[kafka_2.10-0.8.2.56.jar:?]
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:195)
~[kafka_2.10-0.8.2.56.jar:?]
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
~[kafka_2.10-0.8.2.56.jar:?]
2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Stopped
We found that this may be caused in the following scenario:
- we have three log segments with offset range [100, 200), [200, 300), and
[300, 400) respectively. 300 is the base offset of the active log segment. Log
cleaner offset checkpoint is also 300.
- After log is truncated to offset 220, the log segments become [100, 200),
[200, 220). The Log cleaner offset checkpoint is still 300.
- After new messages are appended to the log, the log segments become [100,
200), [200, 320), [320, 420). The Log cleaner offset checkpoint is still 300.
- Log cleaner cleans the log starting at offset 300. The require(offset ==
start) in Cleaner.buildOffsetMap() fails because the the offset 300 is not the
base offset of any segments.
To fix the problem, when the log is truncated to an offset smaller than cleaner
offset checkpoint, we should reset cleaner offset checkpoint to the base offset
of the active segment if this value is larger than the checkpointed offset.
> Decrease log cleaner offset checkpoint if the log is truncated
> --------------------------------------------------------------
>
> Key: KAFKA-3330
> URL: https://issues.apache.org/jira/browse/KAFKA-3330
> Project: Kafka
> Issue Type: Bug
> Reporter: Dong Lin
> Assignee: Dong Lin
>
> Were getting a number of failures of the log compaction thread with the
> following error:
> 2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Beginning cleaning of log
> __consumer_offsets-93.
> 2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Building offset map for
> __consumer_offsets-93...
> 2016/02/02 00:13:59.048 [LogCleaner] Cleaner 0: Building offset map for log
> __consumer_offsets-93 for 2 segments in offset range [11951210572,
> 11952632314).
> 2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Error
> due to
> java.lang.IllegalArgumentException: requirement failed: Last clean offset
> is 11951210572 but segment base offset is 11950300163 for log
> __consumer_offsets-93.
> at scala.Predef$.require(Predef.scala:233) ~[scala-library-2.10.4.jar:?]
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:561)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.Cleaner.clean(LogCleaner.scala:306)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:217)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:195)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> ~[kafka_2.10-0.8.2.56.jar:?]
> 2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Stopped
> We found that this may be caused in the following scenario:
> - we have three log segments with offset range [100, 200), [200, 300), and
> [300, 400) respectively. 300 is the base offset of the active log segment.
> Log cleaner offset checkpoint is also 300.
> - After log is truncated to offset 220, the log segments become [100, 200),
> [200, 220). The Log cleaner offset checkpoint is still 300.
> - After new messages are appended to the log, the log segments become [100,
> 200), [200, 320), [320, 420). The Log cleaner offset checkpoint is still 300.
> - Log cleaner cleans the log starting at offset 300. The require(offset ==
> start) in Cleaner.buildOffsetMap() fails because the the offset 300 is not
> the base offset of any segments.
> To fix the problem, when the log is truncated to an offset smaller than
> cleaner offset checkpoint, we should reset cleaner offset checkpoint to the
> base offset of the active segment if this value is larger than the
> checkpointed offset.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
