Thanks, Joel. I know I remember a case where we had a difference like this between two brokers, and it was not due to retention settings or some other problem, but I can't remember exactly what we determined it was.
-Todd On Mon, Jun 22, 2015 at 4:22 PM, Joel Koshy <[email protected]> wrote: > The replicas do not have to decompress/recompress so I don't think > that would contribute to this. > > There may be some corner cases such as: > - Multiple unclean leadership elections in sequence > - Changing the compression codec for a topic on the fly - different > brokers may see this config change at almost (but not exactly) the > same time, but not sure if you are using that feature. > > You may want to use the DumpLogSegments tool to actually compare the > offsets present in both log files. > > On Mon, Jun 22, 2015 at 08:55:40AM -0700, Todd Palino wrote: > > I assume that you are considering the data loss to be the difference in > > size between the two directories? This is generally not a good guideline, > > as the batching and compression will be different between the two > replicas. > > > > -Todd > > > > > > On Mon, Jun 22, 2015 at 7:26 AM, Nirmal ram <[email protected]> > > wrote: > > > > > Hi, > > > > > > I noticed a data loss while storing in kafka logs. > > > Generally, leader hands the request to followers, is there a data > loss in > > > that process? > > > > > > topic 'jun8' with 2 replicas and 8 partitions > > > > > > *Broker 1*[user@ jun8-6]$ ls -ltr > > > total 7337500 > > > -rw-rw-r-- 1 user user 1073741311 Jun 22 12:45 00000000000015195331.log > > > -rw-rw-r-- 1 user user 1127512 Jun 22 12:45 > 00000000000015195331.index > > > -rw-rw-r-- 1 user user 1073741396 Jun 22 12:48 00000000000016509739.log > > > -rw-rw-r-- 1 user user 1108544 Jun 22 12:48 > 00000000000016509739.index > > > -rw-rw-r-- 1 user user 1073740645 Jun 22 12:52 00000000000017823869.log > > > -rw-rw-r-- 1 user user 1129064 Jun 22 12:52 > 00000000000017823869.index > > > -rw-rw-r-- 1 user user 1073741800 Jun 22 13:17 00000000000019136798.log > > > -rw-rw-r-- 1 user user 1161152 Jun 22 13:17 > 00000000000019136798.index > > > -rw-rw-r-- 1 user user 1073741509 Jun 22 13:21 00000000000020451309.log > > > -rw-rw-r-- 1 user user 1152448 Jun 22 13:21 > 00000000000020451309.index > > > *-rw-rw-r-- 1 user user 1073740588 Jun 22 13:39 > 00000000000021764229.log* > > > -rw-rw-r-- 1 user user 1241168 Jun 22 13:39 > 00000000000021764229.index > > > -rw-rw-r-- 1 user user 1062343875 Jun 22 13:42 00000000000023077448.log > > > -rw-rw-r-- 1 user user 10485760 Jun 22 13:42 > 00000000000023077448.index > > > [user@ jun8-6]$ > > > > > > > > > > > > *Broker 2*[user@ jun8-6]$ ls -ltr > > > total 7340468 > > > -rw-rw-r-- 1 user user 1073741311 Jun 22 12:45 00000000000015195331.log > > > -rw-rw-r-- 1 user user 1857144 Jun 22 12:45 > 00000000000015195331.index > > > -rw-rw-r-- 1 user user 1073741396 Jun 22 12:48 00000000000016509739.log > > > -rw-rw-r-- 1 user user 1857168 Jun 22 12:48 > 00000000000016509739.index > > > -rw-rw-r-- 1 user user 1073740645 Jun 22 12:52 00000000000017823869.log > > > -rw-rw-r-- 1 user user 1857752 Jun 22 12:52 > 00000000000017823869.index > > > -rw-rw-r-- 1 user user 1073741800 Jun 22 13:17 00000000000019136798.log > > > -rw-rw-r-- 1 user user 1857440 Jun 22 13:17 > 00000000000019136798.index > > > -rw-rw-r-- 1 user user 1073741509 Jun 22 13:21 00000000000020451309.log > > > -rw-rw-r-- 1 user user 1856968 Jun 22 13:21 > 00000000000020451309.index > > > *-rw-rw-r-- 1 user user 1073722781 Jun 22 13:39 > 00000000000021764229.log* > > > -rw-rw-r-- 1 user user 1762288 Jun 22 13:39 > 00000000000021764229.index > > > -rw-rw-r-- 1 user user 10485760 Jun 22 13:42 > 00000000000023077448.index > > > -rw-rw-r-- 1 user user 1062343875 Jun 22 13:42 00000000000023077448.log > > > [user@ jun8-6]$ > > > > > -- > Joel >
