[ 
https://issues.apache.org/jira/browse/KAFKA-593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yang Ye updated KAFKA-593:
--------------------------

    Attachment: kafka_593_v1.diff


Basically the problem was during restarting the server and truncation from 
existing log and index files. 

When after the restart or truncation the existing index file is empty (so it's 
always full), one of the conditions of maybeRoll() will be true, so a new log 
segment will be rolled ---- we will end up with two log segments starting from 
the same offset, one of them is empty.

To fix it, we change the function trimToSize() in OffsetIndex to 
trimOrReallocate(), it does either trimming or reallocating the offset index 
file (and memory mapping). At truncation or restart, it will do reallocation, 
so that enough space for offset index is allocated.

We also check existing log segments at roll() function, and throws exception if 
some segment exists with the same offset as the target offset. (This should not 
happen)


                
> Empty log index file created when it shouldn't be empty
> -------------------------------------------------------
>
>                 Key: KAFKA-593
>                 URL: https://issues.apache.org/jira/browse/KAFKA-593
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Yang Ye
>         Attachments: kafka_583_zk_kafka_data.tar.gz, kafka_593_v1.diff
>
>
> We have met empty index file during system test when it shouldn't be empty. 
> In this case, there're around 100 messages in each segment, each of size 
> around 100 bytes, given the "logIndexIntervalBytes" 4096, there should be at 
> least 2 log index entries, but we see empty index file. The kafka and 
> zookeeper logs are attached
> [yye@yye-ld kafka_server_3_logs]$ cd test_1-2/
> [yye@yye-ld test_1-2]$ ls -l
> total 84
> -rw-r--r-- 1 yye eng        8 Oct 29 15:22 00000000000000000000.index
> -rw-r--r-- 1 yye eng    10248 Oct 29 15:22 00000000000000000000.log
> -rw-r--r-- 1 yye eng        8 Oct 29 15:22 00000000000000000100.index
> -rw-r--r-- 1 yye eng    10296 Oct 29 15:22 00000000000000000100.log
> -rw-r--r-- 1 yye eng        0 Oct 29 15:23 00000000000000000200.index
> -rw-r--r-- 1 yye eng    10293 Oct 29 15:23 00000000000000000200.log
> -rw-r--r-- 1 yye eng        0 Oct 29 15:23 00000000000000000300.index
> -rw-r--r-- 1 yye eng    10274 Oct 29 15:23 00000000000000000300.log
> -rw-r--r-- 1 yye eng        0 Oct 29 15:23 00000000000000000399.index
> -rw-r--r-- 1 yye eng    10276 Oct 29 15:23 00000000000000000399.log
> -rw-r--r-- 1 yye eng        0 Oct 29 15:23 00000000000000000498.index
> -rw-r--r-- 1 yye eng    10256 Oct 29 15:23 00000000000000000498.log
> -rw-r--r-- 1 yye eng 10485760 Oct 29 15:23 00000000000000000596.index
> -rw-r--r-- 1 yye eng     3564 Oct 29 15:23 00000000000000000596.log

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Reply via email to