Xiaoyu, Thanks for sending these! However, could you also send the Kafka log that says these segments are deleted ?
Thanks, Neha On Wed, Aug 8, 2012 at 3:17 PM, xiaoyu wang <xiaoyu.w...@gmail.com> wrote: > Here is a sample. The highlighted lines are those showing different sizes > in du and ls > > *du output > * > 513M ./00000001197773741184.kafka > 1.0G ./00000001200458096891.kafka > 1.0G ./00000001200994968784.kafka > 1.0G ./00000001201531839884.kafka > 1.0G ./00000001202068711094.kafka > 513M ./00000001011479389046.kafka > > *ls output > * > -rw-r--r-- 1 kafka kafka 536871053 Aug 8 13:01 00000001197773741184.kafka > -rw-r--r-- 1 kafka kafka 536871893 Aug 8 13:36 00000001200458096891.kafka > -rw-r--r-- 1 kafka kafka 536871100 Aug 8 13:45 00000001200994968784.kafka > -rw-r--r-- 1 kafka kafka 536871210 Aug 8 13:49 00000001201531839884.kafka > -rw-r--r-- 1 kafka kafka 536871063 Aug 8 14:00 00000001202068711094.kafka > -rw-r--r-- 1 kafka kafka 536871139 Aug 6 07:02 00000001011479389046.kafka > > On Wed, Aug 8, 2012 at 2:58 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > >> The mailing list does not accept attachments - you will need to either >> paste here or externally, say on gist. >> >> Thanks, >> >> Joel >> >> On Wed, Aug 8, 2012 at 2:52 PM, xiaoyu wang <xiaoyu.w...@gmail.com> wrote: >> >> > Neha, >> > >> > I have attached the du and ls output for one of the partitions. Looks >> like >> > for some segments, du is showing 1GB while ls shows 513MB. Let me know if >> > you need additional info. >> > >> > Thanks, >> > >> > -Xiaoyu >> > >> > >> > >> > On Wed, Aug 8, 2012 at 12:29 PM, Neha Narkhede <neha.narkh...@gmail.com >> >wrote: >> > >> >> Xiaoyu, >> >> >> >> Please could you paste the log statements that specify which files are >> >> deleted by the broker and also the detailed output by du on the Kafka >> >> log directory ? >> >> If the broker says it has garbage collected some log segments, space >> >> should free up. >> >> >> >> Thanks, >> >> Neha >> >> >> >> On Wed, Aug 8, 2012 at 12:22 PM, xiaoyu wang <xiaoyu.w...@gmail.com> >> >> wrote: >> >> > Hi all, >> >> > >> >> > I am running kafka-0.7.0 and found that even though expired segments >> are >> >> > deleted by broker, the disk space is not released. So, du shows that a >> >> > partition is 450 GB, but the total size of the files in the partition >> is >> >> > only about 250GB. Is this a bug? We are using raid10. >> >> > >> >> > Thanks, >> >> > >> >> > -Xiaoyu >> >> >> > >> > >>
