----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/12060/ -----------------------------------------------------------
Review request for Flume. Description ------- Revised design for Spillable Mem Channel. We no longer have Spillable channel config pointing to another channel (by name) as in the previous design. Spillable Channel instead derives from FileChannel (as per https://issues.apache.org/jira/browse/FLUME-1227?focusedCommentId=13628201&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13628201) Essence of this design: - SC derives from File channel and maintains an in memory queue. If memory queue is full, events are sent to disk overflow (i.e. File channel). - SC maintains a 'Drain-Order' queue (DOQ) for remembering the order in which the incoming events were interleaved between main memory and - Put transaction: All the elements in the putList are committed to mem queue if it has space, else written to disk (ie file channel). Head of DOQ is updated to indicate where the elements put. - Take transaction : Tail of DOQ is consulted to determine whether the next set of events are to be taken from the memory queue or from disk overflow. DOQ's tail is updated after events are taken out. This addresses bug FLUME-1227. https://issues.apache.org/jira/browse/FLUME-1227 Diffs ----- flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 36f150b flume-ng-channels/flume-spillable-memory-channel/pom.xml PRE-CREATION flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java PRE-CREATION flume-ng-channels/flume-spillable-memory-channel/src/test/java/org/apache/flume/channel/TestSpillableMemoryChannel.java PRE-CREATION flume-ng-channels/pom.xml 6485d47 flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java 26f4dd7 flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java 15b8cc3 flume-ng-core/src/main/java/org/apache/flume/ChannelFullException.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 1370e66 flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java 688323d flume-ng-dist/pom.xml f20ff1e flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgentConfiguration.java 6204bc5 flume-ng-node/pom.xml 5cdd9e2 flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java e63c601 pom.xml 7f6f824 Diff: https://reviews.apache.org/r/12060/diff/ Testing ------- Wrote a set of Unit tests. A few are failing and need to be finished up. Thanks, Roshan Naik
