-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12060/
-----------------------------------------------------------
(Updated Dec. 18, 2013, 11:43 a.m.)
Review request for Flume.
Changes
-------
updating with patch v8.
This patch fixes the issues with
- the 80char line,
- The counter issue you noted plus other counters also fixed.. additional unit
tests added to check counters.
- testTotalSemaphore failure.. (now renamed to testCounters)
Bugs: FLUME-1227
https://issues.apache.org/jira/browse/FLUME-1227
Repository: flume-git
Description
-------
Revised design for Spillable Mem Channel.
We no longer have Spillable channel config pointing to another channel (by
name) as in the previous design.
Spillable Channel instead derives from FileChannel (as per
https://issues.apache.org/jira/browse/FLUME-1227?focusedCommentId=13628201&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13628201)
Essence of this design:
- SC derives from File channel and maintains an in memory queue. If memory
queue is full, events are sent to disk overflow (i.e. File channel).
- SC maintains a 'Drain-Order' queue (DOQ) for remembering the order in which
the incoming events were interleaved between main memory and
- Put transaction: All the elements in the putList are committed to mem queue
if it has space, else written to disk (ie file channel). Head of DOQ is updated
to indicate where the elements put.
- Take transaction : Tail of DOQ is consulted to determine whether the next set
of events are to be taken from the memory queue or from disk overflow. DOQ's
tail is updated after events are taken out.
SC Configuration:
- Accepts all the File Channel settings
- Introduces one additional setting: 'memoryCapacity' which indicates the
number of items it can hold in memory
Sample config ...
a1.channels = c1
a1.sinks = logger
a1.sources = src
a1.sources.src.type = exec
a1.sources.src.command = seq 1 100000
a1.sources.src.batchSize = 10
a1.sources.src.channels = c1
a1.sinks.logger.type = logger
a1.sinks.logger.channel = c1
a1.channels.c1.type = spillablememory
a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
a1.channels.c1.dataDirs = /tmp/flume/data
a1.channels.c1.memoryCapacity = 10
a1.channels.c1.keep-alive = 2
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
2cd7f03
flume-ng-channels/flume-spillable-memory-channel/pom.xml PRE-CREATION
flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
PRE-CREATION
flume-ng-channels/flume-spillable-memory-channel/src/test/java/org/apache/flume/channel/TestSpillableMemoryChannel.java
PRE-CREATION
flume-ng-channels/pom.xml 5832ab4
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java
26f4dd7
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java
15b8cc3
flume-ng-core/src/main/java/org/apache/flume/ChannelFullException.java
PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java
1370e66
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java
f10a79f
flume-ng-dist/pom.xml 2d0ee47
flume-ng-doc/sphinx/FlumeUserGuide.rst 3088400
flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgentConfiguration.java
6204bc5
flume-ng-node/pom.xml f1b0c65
pom.xml 3c741c3
Diff: https://reviews.apache.org/r/12060/diff/
Testing
-------
Wrote a set of Unit tests. A few are failing and need to be finished up.
File Attachments
----------------
Revised design doc
https://reviews.apache.org/media/uploaded/files/2013/06/24/SpillableMemory_Channel_Design_2_1.pdf
Thanks,
Roshan Naik
