[
https://issues.apache.org/jira/browse/KAFKA-21203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18122121#comment-18122121
]
Jun Rao commented on KAFKA-21203:
---------------------------------
One way to address this issue is to change LogSegment.recover() so that it
continues the segment recovery after detecting an offset overflow, but stops
building the indexes (to avoid overflow). Once the recovery completes, it
throws an OffsetOverflow exception. The caller will split the recovered segment
and rebuild the indexes.
{code:java}
LogSegmentOffsetOverflowException overflow = null;
try {
for (RecordBatch batch : log.batches()) {
batch.ensureValid();
if (overflow == null) {
try {
ensureOffsetInRange(batch.lastOffset());
} catch (LogSegmentOffsetOverflowException e) {
overflow = e; // remember; from here on we only measure valid
bytes
}
}
if (overflow == null) {
if (batch.maxTimestamp() > maxTimestampSoFar())
maxTimestampAndOffsetSoFar = new
TimestampOffset(batch.maxTimestamp(), batch.lastOffset());
if (validBytes - lastIndexEntry > indexIntervalBytes) {
offsetIndex().append(batch.lastOffset(), validBytes);
timeIndex().maybeAppend(maxTimestampSoFar(),
shallowOffsetOfMaxTimestampSoFar());
lastIndexEntry = validBytes;
}
if (batch.magic() >= RecordBatch.MAGIC_VALUE_V2) {
// epoch cache + producer state updates
...
}
}
validBytes += batch.sizeInBytes(); // always count real bytes,
overflow batch included
}
} catch (CorruptRecordException | InvalidRecordException e) {
LOGGER.warn(...); // hit the preallocated zero tail -> end of records
}
...
if (overflow != null)
throw overflow; // rethrow AFTER padding is gone -> split sees a clean
segment
return truncated; {code}
> recovery could fail when a preallocated segment hits offset overflow
> --------------------------------------------------------------------
>
> Key: KAFKA-21203
> URL: https://issues.apache.org/jira/browse/KAFKA-21203
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Rao
> Priority: Minor
>
> During log recovery, we discover offset overflow in a segment. If an overflow
> is detected, it stops the recovery and calls splitOverflowedSegment() to
> split the segment. However, since the segment hasn't been fully recovered,
> there could be untrimmed padding bytes in the segment if preallocation is
> enabled. This will cause an exception in splitOverflowedSegment() and fail
> log loading. The following call sequence shows how this can happen.
>
> {code:java}
> LogLoader.recoverLog() LogLoader.java:456
> └─ retryOnOffsetOverflow(this::recoverLog) :210 (outer) / :330
> catch
> └─ recoverSegment(segment) :400
> └─ LogSegment.recover(...) LogSegment.java:482
> for (RecordBatch batch : log.batches()) :490
> batch.ensureValid() :491
> ensureOffsetInRange(batch.lastOffset()) :492 ──► throws
>
> LogSegmentOffsetOverflowException
> ── caught by retryOnOffsetOverflow ── LogLoader.java:330
> └─ LocalLog.splitOverflowedSegment(segment,...) LocalLog.java:909
> while (position < sourceRecords.sizeInBytes()) :926 ◄──
> sizeInBytes() incl. padding
> sourceRecords.batchesFrom(position).iterator().next() :927
> ──► CorruptRecordException
> newSegment.appendFromFile(sourceRecords, position) :930
> appendChunkFromFile →
> records.batchesFrom(position).iterator() LogSegment.java:294
> {code}
>
> Note that this issue likely won't happen in practice since the offset
> overflow case typically happens on existing rolled segment which has been
> trimmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
