It seems you might have run that on the last log segment. Can you run it on 00000000000021764229.log on both brokers and compare? I'm guessing there may be a message-set with a different compression codec that may be causing this.
Thanks, Joel On Tue, Jun 23, 2015 at 01:06:16PM +0530, nirmal wrote: > Hi, > i ran DumpLogSegments. > > *Broker 1* > offset: 23077447 position: 1073722324 isvalid: true payloadsize: 431 > magic: 0 compresscodec: NoCompressionCodec crc: 895349554 > > *Broker 2* > offset: 23077447 position: 1073740131 isvalid: true payloadsize: 431 > magic: 0 compresscodec: NoCompressionCodec crc: 895349554 > > Thanks > > > On 06/23/2015 04:52 AM, Joel Koshy wrote: > >The replicas do not have to decompress/recompress so I don't think > >that would contribute to this. > > > >There may be some corner cases such as: > >- Multiple unclean leadership elections in sequence > >- Changing the compression codec for a topic on the fly - different > > brokers may see this config change at almost (but not exactly) the > > same time, but not sure if you are using that feature. > > > >You may want to use the DumpLogSegments tool to actually compare the > >offsets present in both log files. > > > >On Mon, Jun 22, 2015 at 08:55:40AM -0700, Todd Palino wrote: > >>I assume that you are considering the data loss to be the difference in > >>size between the two directories? This is generally not a good guideline, > >>as the batching and compression will be different between the two replicas. > >> > >>-Todd > >> > >> > >>On Mon, Jun 22, 2015 at 7:26 AM, Nirmal ram <nirmal106110...@gmail.com> > >>wrote: > >> > >>>Hi, > >>> > >>>I noticed a data loss while storing in kafka logs. > >>>Generally, leader hands the request to followers, is there a data loss in > >>>that process? > >>> > >>>topic 'jun8' with 2 replicas and 8 partitions > >>> > >>>*Broker 1*[user@ jun8-6]$ ls -ltr > >>>total 7337500 > >>>-rw-rw-r-- 1 user user 1073741311 Jun 22 12:45 00000000000015195331.log > >>>-rw-rw-r-- 1 user user 1127512 Jun 22 12:45 00000000000015195331.index > >>>-rw-rw-r-- 1 user user 1073741396 Jun 22 12:48 00000000000016509739.log > >>>-rw-rw-r-- 1 user user 1108544 Jun 22 12:48 00000000000016509739.index > >>>-rw-rw-r-- 1 user user 1073740645 Jun 22 12:52 00000000000017823869.log > >>>-rw-rw-r-- 1 user user 1129064 Jun 22 12:52 00000000000017823869.index > >>>-rw-rw-r-- 1 user user 1073741800 Jun 22 13:17 00000000000019136798.log > >>>-rw-rw-r-- 1 user user 1161152 Jun 22 13:17 00000000000019136798.index > >>>-rw-rw-r-- 1 user user 1073741509 Jun 22 13:21 00000000000020451309.log > >>>-rw-rw-r-- 1 user user 1152448 Jun 22 13:21 00000000000020451309.index > >>>*-rw-rw-r-- 1 user user 1073740588 Jun 22 13:39 00000000000021764229.log* > >>>-rw-rw-r-- 1 user user 1241168 Jun 22 13:39 00000000000021764229.index > >>>-rw-rw-r-- 1 user user 1062343875 Jun 22 13:42 00000000000023077448.log > >>>-rw-rw-r-- 1 user user 10485760 Jun 22 13:42 00000000000023077448.index > >>>[user@ jun8-6]$ > >>> > >>> > >>> > >>>*Broker 2*[user@ jun8-6]$ ls -ltr > >>>total 7340468 > >>>-rw-rw-r-- 1 user user 1073741311 Jun 22 12:45 00000000000015195331.log > >>>-rw-rw-r-- 1 user user 1857144 Jun 22 12:45 00000000000015195331.index > >>>-rw-rw-r-- 1 user user 1073741396 Jun 22 12:48 00000000000016509739.log > >>>-rw-rw-r-- 1 user user 1857168 Jun 22 12:48 00000000000016509739.index > >>>-rw-rw-r-- 1 user user 1073740645 Jun 22 12:52 00000000000017823869.log > >>>-rw-rw-r-- 1 user user 1857752 Jun 22 12:52 00000000000017823869.index > >>>-rw-rw-r-- 1 user user 1073741800 Jun 22 13:17 00000000000019136798.log > >>>-rw-rw-r-- 1 user user 1857440 Jun 22 13:17 00000000000019136798.index > >>>-rw-rw-r-- 1 user user 1073741509 Jun 22 13:21 00000000000020451309.log > >>>-rw-rw-r-- 1 user user 1856968 Jun 22 13:21 00000000000020451309.index > >>>*-rw-rw-r-- 1 user user 1073722781 Jun 22 13:39 00000000000021764229.log* > >>>-rw-rw-r-- 1 user user 1762288 Jun 22 13:39 00000000000021764229.index > >>>-rw-rw-r-- 1 user user 10485760 Jun 22 13:42 00000000000023077448.index > >>>-rw-rw-r-- 1 user user 1062343875 Jun 22 13:42 00000000000023077448.log > >>>[user@ jun8-6]$ > >>> > -- Joel
