----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/6683/ -----------------------------------------------------------
(Updated Aug. 21, 2012, 3:01 a.m.) Review request for Flume and Brock Noland. Changes ------- Removing "queue.remove(FlumeEventPointer.fromLong(eventPointer));" from Replayhandler. Despite this fix, there is one more race condition: https://issues.apache.org/jira/browse/FLUME-1498 Description ------- Flume Event Queue now keeps a copy of the event pointers to uncommitted puts and takes. It serializes these on every checkpoint and deserializes these on replay and reinserts these into either the event queue(for takes) or to the replay queue(for puts). I could have used the PutList and TakeList of the transaction for this, but I didn't really like the approach. I don't want to be sharing this kind of data between multiple layers, since that makes it complex to change the FlumeEventQueue implementation without causing major changes in FileBackedTransaction. Also it would lead to a number of cross layer calls to read data - which makes the approach less clean. With my current approach, by localizing most changes to the FlumeEventQueue class, only a couple of function calls would need to be removed/modified. Agreed that this is going to be some memory overhead, but this is insignificant compared to the event queue size itself. This would be hardly a few MB extra in memory - but if that gives me cleaner implementation, I would prefer that. This addresses bug FLUME-1437. https://issues.apache.org/jira/browse/FLUME-1437 Diffs (updated) ----- flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java e7735e8 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 9bfee2d flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 11f1e1f flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java bbca62c flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java 7ec5916 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 1d5a0f9 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java 569b7c7 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java e0b5e3f Diff: https://reviews.apache.org/r/6683/diff/ Testing ------- Added 4 new unit tests (2 to TestFileChannel.java to test the actual use case, and 2 to TestFlumeEventQueue.java to test the actual functionality of serialization/deserialization). Thanks, Hari Shreedharan
