Neha, These are not deleted segments. Actually, most of them are new segments. Because the total disk usage shown with "du" is smaller than "ls", I thought it was because space not released for those segments that have expired. But the detailed du output indicates that it is those active segments that uses more disk spaces (shown 1GB in du but .5GB with ls).
-Xiaoyu On Wed, Aug 8, 2012 at 3:31 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > Xiaoyu, > > Thanks for sending these! However, could you also send the Kafka log > that says these segments are deleted ? > > Thanks, > Neha > > On Wed, Aug 8, 2012 at 3:17 PM, xiaoyu wang <xiaoyu.w...@gmail.com> wrote: > > Here is a sample. The highlighted lines are those showing different sizes > > in du and ls > > > > *du output > > * > > 513M ./00000001197773741184.kafka > > 1.0G ./00000001200458096891.kafka > > 1.0G ./00000001200994968784.kafka > > 1.0G ./00000001201531839884.kafka > > 1.0G ./00000001202068711094.kafka > > 513M ./00000001011479389046.kafka > > > > *ls output > > * > > -rw-r--r-- 1 kafka kafka 536871053 Aug 8 13:01 > 00000001197773741184.kafka > > -rw-r--r-- 1 kafka kafka 536871893 Aug 8 13:36 > 00000001200458096891.kafka > > -rw-r--r-- 1 kafka kafka 536871100 Aug 8 13:45 > 00000001200994968784.kafka > > -rw-r--r-- 1 kafka kafka 536871210 Aug 8 13:49 > 00000001201531839884.kafka > > -rw-r--r-- 1 kafka kafka 536871063 Aug 8 14:00 > 00000001202068711094.kafka > > -rw-r--r-- 1 kafka kafka 536871139 Aug 6 07:02 > 00000001011479389046.kafka > > > > On Wed, Aug 8, 2012 at 2:58 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > >> The mailing list does not accept attachments - you will need to either > >> paste here or externally, say on gist. > >> > >> Thanks, > >> > >> Joel > >> > >> On Wed, Aug 8, 2012 at 2:52 PM, xiaoyu wang <xiaoyu.w...@gmail.com> > wrote: > >> > >> > Neha, > >> > > >> > I have attached the du and ls output for one of the partitions. Looks > >> like > >> > for some segments, du is showing 1GB while ls shows 513MB. Let me > know if > >> > you need additional info. > >> > > >> > Thanks, > >> > > >> > -Xiaoyu > >> > > >> > > >> > > >> > On Wed, Aug 8, 2012 at 12:29 PM, Neha Narkhede < > neha.narkh...@gmail.com > >> >wrote: > >> > > >> >> Xiaoyu, > >> >> > >> >> Please could you paste the log statements that specify which files > are > >> >> deleted by the broker and also the detailed output by du on the Kafka > >> >> log directory ? > >> >> If the broker says it has garbage collected some log segments, space > >> >> should free up. > >> >> > >> >> Thanks, > >> >> Neha > >> >> > >> >> On Wed, Aug 8, 2012 at 12:22 PM, xiaoyu wang <xiaoyu.w...@gmail.com> > >> >> wrote: > >> >> > Hi all, > >> >> > > >> >> > I am running kafka-0.7.0 and found that even though expired > segments > >> are > >> >> > deleted by broker, the disk space is not released. So, du shows > that a > >> >> > partition is 450 GB, but the total size of the files in the > partition > >> is > >> >> > only about 250GB. Is this a bug? We are using raid10. > >> >> > > >> >> > Thanks, > >> >> > > >> >> > -Xiaoyu > >> >> > >> > > >> > > >> >
