Hi Ari, The chukwa dataflow documentation (http://incubator.apache.org/chukwa/docs/r0.4.0/dataflow.html) mentions - "Collectors write chunks to logs/*.chukwa files until a 64MB chunk size is reached or a given time interval has passed." I think this should be corrected.
Thanks, Himanshu On Tue, Jul 12, 2011 at 4:18 PM, Ariel Rabkin <[email protected]> wrote: > Howdy. > > There isn't such a check; if there's documentation somewhere that > suggests there is, let us know where and we can fix it. In general, > the goal is to have .done files as large as possible while compatible > with SLAs; there wasn't any intent to have them only be one block > long. > > --Ari > > On Tue, Jul 12, 2011 at 3:57 PM, Himanshu Gahlot > <[email protected]> wrote: >> Hi, >> >> We have a system where the .chukwa files produced are larger than >> 64MB, but I do not see them being closed by collector and convert to >> .done file. They are closing only at the scheduled time and hence are >> greater than the block size (64MB). I do not see a check to close >> files larger than a block size in SeqFileWriter class. Where is this >> check made in the code ? >> >> Thanks, >> Himanshu >> > > > > -- > Ari Rabkin [email protected] > UC Berkeley Computer Science Department >
