[
https://issues.apache.org/jira/browse/KAFKA-593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Ye updated KAFKA-593:
--------------------------
Attachment: kafka_593_v1.diff
Basically the problem was during restarting the server and truncation from
existing log and index files.
When after the restart or truncation the existing index file is empty (so it's
always full), one of the conditions of maybeRoll() will be true, so a new log
segment will be rolled ---- we will end up with two log segments starting from
the same offset, one of them is empty.
To fix it, we change the function trimToSize() in OffsetIndex to
trimOrReallocate(), it does either trimming or reallocating the offset index
file (and memory mapping). At truncation or restart, it will do reallocation,
so that enough space for offset index is allocated.
We also check existing log segments at roll() function, and throws exception if
some segment exists with the same offset as the target offset. (This should not
happen)
> Empty log index file created when it shouldn't be empty
> -------------------------------------------------------
>
> Key: KAFKA-593
> URL: https://issues.apache.org/jira/browse/KAFKA-593
> Project: Kafka
> Issue Type: Bug
> Reporter: Yang Ye
> Attachments: kafka_583_zk_kafka_data.tar.gz, kafka_593_v1.diff
>
>
> We have met empty index file during system test when it shouldn't be empty.
> In this case, there're around 100 messages in each segment, each of size
> around 100 bytes, given the "logIndexIntervalBytes" 4096, there should be at
> least 2 log index entries, but we see empty index file. The kafka and
> zookeeper logs are attached
> [yye@yye-ld kafka_server_3_logs]$ cd test_1-2/
> [yye@yye-ld test_1-2]$ ls -l
> total 84
> -rw-r--r-- 1 yye eng 8 Oct 29 15:22 00000000000000000000.index
> -rw-r--r-- 1 yye eng 10248 Oct 29 15:22 00000000000000000000.log
> -rw-r--r-- 1 yye eng 8 Oct 29 15:22 00000000000000000100.index
> -rw-r--r-- 1 yye eng 10296 Oct 29 15:22 00000000000000000100.log
> -rw-r--r-- 1 yye eng 0 Oct 29 15:23 00000000000000000200.index
> -rw-r--r-- 1 yye eng 10293 Oct 29 15:23 00000000000000000200.log
> -rw-r--r-- 1 yye eng 0 Oct 29 15:23 00000000000000000300.index
> -rw-r--r-- 1 yye eng 10274 Oct 29 15:23 00000000000000000300.log
> -rw-r--r-- 1 yye eng 0 Oct 29 15:23 00000000000000000399.index
> -rw-r--r-- 1 yye eng 10276 Oct 29 15:23 00000000000000000399.log
> -rw-r--r-- 1 yye eng 0 Oct 29 15:23 00000000000000000498.index
> -rw-r--r-- 1 yye eng 10256 Oct 29 15:23 00000000000000000498.log
> -rw-r--r-- 1 yye eng 10485760 Oct 29 15:23 00000000000000000596.index
> -rw-r--r-- 1 yye eng 3564 Oct 29 15:23 00000000000000000596.log
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
