----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/12060/#review27964 -----------------------------------------------------------
flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java <https://reviews.apache.org/r/12060/#comment54413> made this change in others places too flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java <https://reviews.apache.org/r/12060/#comment54414> made this fix in other locations too flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java <https://reviews.apache.org/r/12060/#comment54429> Thanks. good catch. flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java <https://reviews.apache.org/r/12060/#comment54418> In this instance, it would nice to make an exception for readability .. since the comment needs to appear before the else flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java <https://reviews.apache.org/r/12060/#comment54428> Thanks. Good catch. - Roshan Naik On Oct. 24, 2013, 10:18 p.m., Roshan Naik wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/12060/ > ----------------------------------------------------------- > > (Updated Oct. 24, 2013, 10:18 p.m.) > > > Review request for Flume. > > > Bugs: FLUME-1227 > https://issues.apache.org/jira/browse/FLUME-1227 > > > Repository: flume-git > > > Description > ------- > > Revised design for Spillable Mem Channel. > We no longer have Spillable channel config pointing to another channel (by > name) as in the previous design. > > Spillable Channel instead derives from FileChannel (as per > https://issues.apache.org/jira/browse/FLUME-1227?focusedCommentId=13628201&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13628201) > > Essence of this design: > - SC derives from File channel and maintains an in memory queue. If memory > queue is full, events are sent to disk overflow (i.e. File channel). > - SC maintains a 'Drain-Order' queue (DOQ) for remembering the order in which > the incoming events were interleaved between main memory and > - Put transaction: All the elements in the putList are committed to mem queue > if it has space, else written to disk (ie file channel). Head of DOQ is > updated to indicate where the elements put. > - Take transaction : Tail of DOQ is consulted to determine whether the next > set of events are to be taken from the memory queue or from disk overflow. > DOQ's tail is updated after events are taken out. > > > SC Configuration: > - Accepts all the File Channel settings > - Introduces one additional setting: 'memoryCapacity' which indicates the > number of items it can hold in memory > > > Sample config ... > > a1.channels = c1 > a1.sinks = logger > a1.sources = src > > a1.sources.src.type = exec > a1.sources.src.command = seq 1 100000 > a1.sources.src.batchSize = 10 > a1.sources.src.channels = c1 > > a1.sinks.logger.type = logger > a1.sinks.logger.channel = c1 > > a1.channels.c1.type = spillablememory > a1.channels.c1.checkpointDir = /tmp/flume/checkpoint > a1.channels.c1.dataDirs = /tmp/flume/data > a1.channels.c1.memoryCapacity = 10 > a1.channels.c1.keep-alive = 2 > > > Diffs > ----- > > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java > 36f150b > flume-ng-channels/flume-spillable-memory-channel/pom.xml PRE-CREATION > > flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java > PRE-CREATION > > flume-ng-channels/flume-spillable-memory-channel/src/test/java/org/apache/flume/channel/TestSpillableMemoryChannel.java > PRE-CREATION > flume-ng-channels/pom.xml 5832ab4 > > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java > 26f4dd7 > > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java > 15b8cc3 > flume-ng-core/src/main/java/org/apache/flume/ChannelFullException.java > PRE-CREATION > flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java > 1370e66 > flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java > 688323d > flume-ng-dist/pom.xml 2d0ee47 > flume-ng-doc/sphinx/FlumeUserGuide.rst 98859ce > > flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgentConfiguration.java > 6204bc5 > flume-ng-node/pom.xml f1b0c65 > pom.xml 267925f > > Diff: https://reviews.apache.org/r/12060/diff/ > > > Testing > ------- > > Wrote a set of Unit tests. A few are failing and need to be finished up. > > > File Attachments > ---------------- > > Revised design doc > > https://reviews.apache.org/media/uploaded/files/2013/06/24/SpillableMemory_Channel_Design_2_1.pdf > > > Thanks, > > Roshan Naik > >
