> On Aug. 24, 2012, 5:27 p.m., Brock Noland wrote: > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, > > line 367 > > <https://reviews.apache.org/r/6756/diff/2/?file=145258#file145258line367> > > > > We can get this same information from the inflights. It will have a > > very slight cost of generating the list but that should be trivial (less > > than a millisecond) compared with the disk operations we do turning a > > checkpoint taking many milliseconds. > > > > With the advantage that is makes this more understandable. > > Hari Shreedharan wrote: > Brock, > > Using a bunch of sources and sinks, and reducing the roll size(causing > more frequent rolling) I was able to get the inflights upto around 35000. And > traversing and inserting each one tool a total of around 120 ms(including the > time required to convert each to a FlumeEventPointer to get the file ID). > That said, I will move the data structure holding the fileIDs to > InflightWrapper, so the understandability issue is solved and would not take > hundreds of millis in the checkpoint to generate this list.
To be clear slightly modified SEQ and NULL which commit transactions after 10000 events each and sleep every now and then. - Hari ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/6756/#review10741 ----------------------------------------------------------- On Aug. 24, 2012, 4:33 a.m., Hari Shreedharan wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/6756/ > ----------------------------------------------------------- > > (Updated Aug. 24, 2012, 4:33 a.m.) > > > Review request for Flume, Arvind Prabhakar and Brock Noland. > > > Description > ------- > > Keep track of file ids of non-committed transactions also. Remove them only > when the transaction gets committed. > > > This addresses bug FLUME-1428. > https://issues.apache.org/jira/browse/FLUME-1428 > > > Diffs > ----- > > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java > 766c59a > > flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java > 2093839 > > Diff: https://reviews.apache.org/r/6756/diff/ > > > Testing > ------- > > Added a unit test for this case. > > > Thanks, > > Hari Shreedharan > >
