Gaurav Narula created KAFKA-19200:
-------------------------------------

             Summary: Indexes should be sanity checked on startup
                 Key: KAFKA-19200
                 URL: https://issues.apache.org/jira/browse/KAFKA-19200
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 4.0.0, 3.9.0
            Reporter: Gaurav Narula


KAFKA-7283 removed sanity checks for indexes on broker startup as we
thought it doesn't add much benefit. It turns out that log segment
corruption may be independent of an index corruption.

An index corruption when not caught early is quite tricky to debug. We observed 
the following in production:

A corruption lead to a timeindex file on disk to be mostly filled with {{0}}s 
at the end. This file is then loaded in memory such that {{DirectByteBuffer}}'s 
{{position=limit=10485756}}. Note that this is 4 bytes short of 10MiB, the 
configured max index size.

At this point, the log segment is eligible to be rolled as {{TimeIndex#isFull}} 
will return {{true}}. We observe that the roll is attempted from 2 code paths:

1. ReplicaFetcherThread attempts to roll the log segment as it tries to append 
records
2. LogCleanerThread attempts to roll the log segment as it tries to clean 
segments that breach retentionMs

In both scenarios, {{LogSegment#onBecomeInactiveSegment}} is invoked which in 
turn invokes {{TimeIndex#maybeAppend(long timestamp, long offset, boolean 
skipFullCheck)}} with {{skipFullCheck=true}}, causing an append to an already 
full TimeIndex which fails by throwing a {{BufferOverException}}.

For (1), the exception causes the partition to be marked as failed, thereby 
causing an under-replicated partition. For (2), the LogCleanerThread shuts 
down, potentially causing a leak as other segments which are eligible for 
cleanup aren't cleaned up.

We should therefore reintroduce sanity checks on startup for indexes in 
{{LogSegment#sanityCheck}}, as that is invoked regardless of an unclean 
shutdown and it attempts to re-create the indexes if corruption is diagnosed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to