I work with Shweta and this is something we've been considering for a while. The idea is that collectors could be configured to close their files at some fixed offset after a the end of the interval. In our case, we'd like files to close at 30 seconds after the end of a 5 minute period. This will help us with a couple of edge cases that the current approach causes for us. Backward compatibility and default behavior would of course be maintained.
On Tue, May 31, 2011 at 2:12 PM, Eric Yang <[email protected]> wrote: > +1 for JIRA, nice to have feature. > > Regards, > Eric > > On 5/31/11 2:01 PM, "Ariel Rabkin" <[email protected]> wrote: > > Also. > > This sounds like something fairly site-specific. It would be really > good to have unit tests so we don't break this > feature in the future. > > --Ari > > On Tue, May 31, 2011 at 2:01 PM, Ariel Rabkin <[email protected]> wrote: > > Not quite sure I understand the use case, but if it seems useful, > > sure, open a JIRA. > > > > --Ari > > > > On Tue, May 31, 2011 at 11:33 AM, Shweta Shah <[email protected]> > wrote: > >> Hi, > >> > >> We have a system built on top of Chukwa that introduces the need to have > all > >> sequence files for a given rotateInterval closed and available at a > >> predictable time. Currently, we are experiencing some drift in the > times > >> that the sequence files are closed due to the way the TimerTask is > scheduled > >> in the SeqFileWriter class. We would like to submit a solution that > will > >> allow people to configure the time all collectors should close their > files > >> for processing in a given interval, while still supporting the default > >> functionality. We have written the code and are currently testing the > >> functionality, but would like to know if anyone has any feedback on this > >> development before submitting a JIRA ticket. > >> > >> - Shweta > >> > > > > > > > > -- > > Ari Rabkin [email protected] > > UC Berkeley Computer Science Department > > > > > > -- > Ari Rabkin [email protected] > UC Berkeley Computer Science Department > >
