Hi,
i ran DumpLogSegments.
*Broker 1*
offset: 23077447 position: 1073722324 isvalid: true payloadsize: 431
magic: 0 compresscodec: NoCompressionCodec crc: 895349554
*Broker 2*
offset: 23077447 position: 1073740131 isvalid: true payloadsize: 431
magic: 0 compresscodec: NoCompressionCodec crc: 895349554
Thanks
On 06/23/2015 04:52 AM, Joel Koshy wrote:
The replicas do not have to decompress/recompress so I don't think
that would contribute to this.
There may be some corner cases such as:
- Multiple unclean leadership elections in sequence
- Changing the compression codec for a topic on the fly - different
brokers may see this config change at almost (but not exactly) the
same time, but not sure if you are using that feature.
You may want to use the DumpLogSegments tool to actually compare the
offsets present in both log files.
On Mon, Jun 22, 2015 at 08:55:40AM -0700, Todd Palino wrote:
I assume that you are considering the data loss to be the difference in
size between the two directories? This is generally not a good guideline,
as the batching and compression will be different between the two replicas.
-Todd
On Mon, Jun 22, 2015 at 7:26 AM, Nirmal ram <[email protected]>
wrote:
Hi,
I noticed a data loss while storing in kafka logs.
Generally, leader hands the request to followers, is there a data loss in
that process?
topic 'jun8' with 2 replicas and 8 partitions
*Broker 1*[user@ jun8-6]$ ls -ltr
total 7337500
-rw-rw-r-- 1 user user 1073741311 Jun 22 12:45 00000000000015195331.log
-rw-rw-r-- 1 user user 1127512 Jun 22 12:45 00000000000015195331.index
-rw-rw-r-- 1 user user 1073741396 Jun 22 12:48 00000000000016509739.log
-rw-rw-r-- 1 user user 1108544 Jun 22 12:48 00000000000016509739.index
-rw-rw-r-- 1 user user 1073740645 Jun 22 12:52 00000000000017823869.log
-rw-rw-r-- 1 user user 1129064 Jun 22 12:52 00000000000017823869.index
-rw-rw-r-- 1 user user 1073741800 Jun 22 13:17 00000000000019136798.log
-rw-rw-r-- 1 user user 1161152 Jun 22 13:17 00000000000019136798.index
-rw-rw-r-- 1 user user 1073741509 Jun 22 13:21 00000000000020451309.log
-rw-rw-r-- 1 user user 1152448 Jun 22 13:21 00000000000020451309.index
*-rw-rw-r-- 1 user user 1073740588 Jun 22 13:39 00000000000021764229.log*
-rw-rw-r-- 1 user user 1241168 Jun 22 13:39 00000000000021764229.index
-rw-rw-r-- 1 user user 1062343875 Jun 22 13:42 00000000000023077448.log
-rw-rw-r-- 1 user user 10485760 Jun 22 13:42 00000000000023077448.index
[user@ jun8-6]$
*Broker 2*[user@ jun8-6]$ ls -ltr
total 7340468
-rw-rw-r-- 1 user user 1073741311 Jun 22 12:45 00000000000015195331.log
-rw-rw-r-- 1 user user 1857144 Jun 22 12:45 00000000000015195331.index
-rw-rw-r-- 1 user user 1073741396 Jun 22 12:48 00000000000016509739.log
-rw-rw-r-- 1 user user 1857168 Jun 22 12:48 00000000000016509739.index
-rw-rw-r-- 1 user user 1073740645 Jun 22 12:52 00000000000017823869.log
-rw-rw-r-- 1 user user 1857752 Jun 22 12:52 00000000000017823869.index
-rw-rw-r-- 1 user user 1073741800 Jun 22 13:17 00000000000019136798.log
-rw-rw-r-- 1 user user 1857440 Jun 22 13:17 00000000000019136798.index
-rw-rw-r-- 1 user user 1073741509 Jun 22 13:21 00000000000020451309.log
-rw-rw-r-- 1 user user 1856968 Jun 22 13:21 00000000000020451309.index
*-rw-rw-r-- 1 user user 1073722781 Jun 22 13:39 00000000000021764229.log*
-rw-rw-r-- 1 user user 1762288 Jun 22 13:39 00000000000021764229.index
-rw-rw-r-- 1 user user 10485760 Jun 22 13:42 00000000000023077448.index
-rw-rw-r-- 1 user user 1062343875 Jun 22 13:42 00000000000023077448.log
[user@ jun8-6]$
