Does kafka pre-allocate 1GB for each segment? It shows that these segment files are all open and used by kafka broker.
On Wed, Aug 8, 2012 at 4:14 PM, xiaoyu wang <xiaoyu.w...@gmail.com> wrote: > Jun, > > this does not seem to be a du problem because after restarting brokers, du > and ls show the same file size. > > > On Wed, Aug 8, 2012 at 3:55 PM, Jun Rao <jun...@gmail.com> wrote: > >> Hmm, this seems like an issue with du. >> >> Thanks >> >> Jun >> >> On Wed, Aug 8, 2012 at 3:17 PM, xiaoyu wang <xiaoyu.w...@gmail.com> >> wrote: >> >> > Here is a sample. The highlighted lines are those showing different >> sizes >> > in du and ls >> > >> > *du output >> > * >> > 513M ./00000001197773741184.kafka >> > 1.0G ./00000001200458096891.kafka >> > 1.0G ./00000001200994968784.kafka >> > 1.0G ./00000001201531839884.kafka >> > 1.0G ./00000001202068711094.kafka >> > 513M ./00000001011479389046.kafka >> > >> > *ls output >> > * >> > -rw-r--r-- 1 kafka kafka 536871053 Aug 8 13:01 >> 00000001197773741184.kafka >> > -rw-r--r-- 1 kafka kafka 536871893 Aug 8 13:36 >> 00000001200458096891.kafka >> > -rw-r--r-- 1 kafka kafka 536871100 Aug 8 13:45 >> 00000001200994968784.kafka >> > -rw-r--r-- 1 kafka kafka 536871210 Aug 8 13:49 >> 00000001201531839884.kafka >> > -rw-r--r-- 1 kafka kafka 536871063 Aug 8 14:00 >> 00000001202068711094.kafka >> > -rw-r--r-- 1 kafka kafka 536871139 Aug 6 07:02 >> 00000001011479389046.kafka >> > >> > On Wed, Aug 8, 2012 at 2:58 PM, Joel Koshy <jjkosh...@gmail.com> wrote: >> > >> > > The mailing list does not accept attachments - you will need to either >> > > paste here or externally, say on gist. >> > > >> > > Thanks, >> > > >> > > Joel >> > > >> > > On Wed, Aug 8, 2012 at 2:52 PM, xiaoyu wang <xiaoyu.w...@gmail.com> >> > wrote: >> > > >> > > > Neha, >> > > > >> > > > I have attached the du and ls output for one of the partitions. >> Looks >> > > like >> > > > for some segments, du is showing 1GB while ls shows 513MB. Let me >> know >> > if >> > > > you need additional info. >> > > > >> > > > Thanks, >> > > > >> > > > -Xiaoyu >> > > > >> > > > >> > > > >> > > > On Wed, Aug 8, 2012 at 12:29 PM, Neha Narkhede < >> > neha.narkh...@gmail.com >> > > >wrote: >> > > > >> > > >> Xiaoyu, >> > > >> >> > > >> Please could you paste the log statements that specify which files >> are >> > > >> deleted by the broker and also the detailed output by du on the >> Kafka >> > > >> log directory ? >> > > >> If the broker says it has garbage collected some log segments, >> space >> > > >> should free up. >> > > >> >> > > >> Thanks, >> > > >> Neha >> > > >> >> > > >> On Wed, Aug 8, 2012 at 12:22 PM, xiaoyu wang < >> xiaoyu.w...@gmail.com> >> > > >> wrote: >> > > >> > Hi all, >> > > >> > >> > > >> > I am running kafka-0.7.0 and found that even though expired >> segments >> > > are >> > > >> > deleted by broker, the disk space is not released. So, du shows >> > that a >> > > >> > partition is 450 GB, but the total size of the files in the >> > partition >> > > is >> > > >> > only about 250GB. Is this a bug? We are using raid10. >> > > >> > >> > > >> > Thanks, >> > > >> > >> > > >> > -Xiaoyu >> > > >> >> > > > >> > > > >> > > >> > >> > >
