gaurav-narula commented on PR #22585:
URL: https://github.com/apache/kafka/pull/22585#issuecomment-5430596763
> Measured the cost of calling timeIndex().sanityCheck() during segment
loading by timing full log initialization (UnifiedLog.create()) on a directory
of pre-existing segments, with and without the check. The segments are minimal
(empty log files, single-entry trimmed time indexes), so the delta between runs
isolates the cost of materializing the lazy time index.
>
> Calling timeIndex() on a segment for the first time triggers LazyIndex to
construct the TimeIndex, which mmaps the file and reads the last entry from the
mapped region. sanityCheck() itself only compares cached values. The entire
cost is in materializing the lazy index.
>
> The only variable between runs is the presence of the
timeIndex().sanityCheck() call. Each number in the table is the average of 5
runs. Results (MacBook Pro M3, SSD):
>
> ```
> Segments Without check With check Delta
> 1,000 761 ms 960 ms +199 ms
> 5,000 3,417 ms 4,570 ms +1,153 ms
> 10,000 6,469 ms 8,860 ms +2,391 ms
> ```
Thanks for this PR and apologies for the late review. It would be useful to
know where the most amount of time is spent in the check. If most of the time
is spent mmapping the 10MB (assuming that's the max index size) file then maybe
we can optimize this with a different heuristic. Since the size of each entry
is deterministic and a corrupt index has entries filled with 0s, perhaps the
heuristic can be to look at the last two entries and check if they're both
filled with zeros. This would result in reading 1 page instead per index
instead of 2560 pages (assuming each page is 4K). This way we can add the check
cheaply for both offset and time indexes. WDYT?
> The cost scales linearly at ~0.24 ms per segment on this hardware. The
offset index is not materialized (stays lazy), so this is half the I/O that
[KAFKA-7283](https://issues.apache.org/jira/browse/KAFKA-7283) originally
removed.
Better to caveat this as half the _index_ related I/O as that issue removed
sanity checks for log segments as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
