> On Aug. 24, 2012, 5:27 p.m., Brock Noland wrote: > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, > > line 367 > > <https://reviews.apache.org/r/6756/diff/2/?file=145258#file145258line367> > > > > We can get this same information from the inflights. It will have a > > very slight cost of generating the list but that should be trivial (less > > than a millisecond) compared with the disk operations we do turning a > > checkpoint taking many milliseconds. > > > > With the advantage that is makes this more understandable.
Brock, Using a bunch of sources and sinks, and reducing the roll size(causing more frequent rolling) I was able to get the inflights upto around 35000. And traversing and inserting each one tool a total of around 120 ms(including the time required to convert each to a FlumeEventPointer to get the file ID). That said, I will move the data structure holding the fileIDs to InflightWrapper, so the understandability issue is solved and would not take hundreds of millis in the checkpoint to generate this list. - Hari ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/6756/#review10741 ----------------------------------------------------------- On Aug. 24, 2012, 4:33 a.m., Hari Shreedharan wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/6756/ > ----------------------------------------------------------- > > (Updated Aug. 24, 2012, 4:33 a.m.) > > > Review request for Flume, Arvind Prabhakar and Brock Noland. > > > Description > ------- > > Keep track of file ids of non-committed transactions also. Remove them only > when the transaction gets committed. > > > This addresses bug FLUME-1428. > https://issues.apache.org/jira/browse/FLUME-1428 > > > Diffs > ----- > > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java > 766c59a > > flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java > 2093839 > > Diff: https://reviews.apache.org/r/6756/diff/ > > > Testing > ------- > > Added a unit test for this case. > > > Thanks, > > Hari Shreedharan > >
