----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/6683/#review10515 -----------------------------------------------------------
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java <https://reviews.apache.org/r/6683/#comment22504> should we lock this file to ensure two threads don't do this write at the same time? - Brock Noland On Aug. 18, 2012, 8:40 a.m., Hari Shreedharan wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/6683/ > ----------------------------------------------------------- > > (Updated Aug. 18, 2012, 8:40 a.m.) > > > Review request for Flume and Brock Noland. > > > Description > ------- > > Flume Event Queue now keeps a copy of the event pointers to uncommitted puts > and takes. It serializes these on every checkpoint and deserializes these on > replay and reinserts these into either the event queue(for takes) or to the > replay queue(for puts). > > I could have used the PutList and TakeList of the transaction for this, but I > didn't really like the approach. I don't want to be sharing this kind of data > between multiple layers, since that makes it complex to change the > FlumeEventQueue implementation without causing major changes in > FileBackedTransaction. Also it would lead to a number of cross layer calls to > read data - which makes the approach less clean. > With my current approach, by localizing most changes to the FlumeEventQueue > class, only a couple of function calls would need to be removed/modified. > Agreed that this is going to be some memory overhead, but this is > insignificant compared to the event queue size itself. This would be hardly a > few MB extra in memory - but if that gives me cleaner implementation, I would > prefer that. > > > This addresses bug FLUME-1437. > https://issues.apache.org/jira/browse/FLUME-1437 > > > Diffs > ----- > > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java > e7735e8 > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java > 9bfee2d > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java > 11f1e1f > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java > bbca62c > > flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java > 7ec5916 > > flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java > 1d5a0f9 > > flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java > 569b7c7 > > flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java > e0b5e3f > > Diff: https://reviews.apache.org/r/6683/diff/ > > > Testing > ------- > > Added 4 new unit tests (2 to TestFileChannel.java to test the actual use > case, and 2 to TestFlumeEventQueue.java to test the actual functionality of > serialization/deserialization). > > > Thanks, > > Hari Shreedharan > >
