-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12060/#review27847
-----------------------------------------------------------
Ship it!
Roshan,
This looks good to go, but I am fairly sure this needs considerable testing and
improvements before we call it production ready. So let's commit this for now,
but mark it as alpha and experimental in the user docs and the javadocs. Let's
iterate on this over to make sure it is production ready.
Also can you please make sure the formats are correct. An if looks like:
if (condition) {
} else {
}
similarly for for/while etc. An IDE can probably fix most of it up for you.
Please make these fixes and attach the patch to the jira.
flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
<https://reviews.apache.org/r/12060/#comment54213>
Nit: Space before and after += makes it more readable.
flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
<https://reviews.apache.org/r/12060/#comment54211>
Extra space after if (should generally be before the paranthesis).
flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
<https://reviews.apache.org/r/12060/#comment54214>
same as above
flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
<https://reviews.apache.org/r/12060/#comment54215>
same as above
flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
<https://reviews.apache.org/r/12060/#comment54212>
same as above
flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
<https://reviews.apache.org/r/12060/#comment54216>
the spaces in the if are off. an if should look like:
if (condition) {
} else {
}
flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
<https://reviews.apache.org/r/12060/#comment54217>
additional space after return
flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
<https://reviews.apache.org/r/12060/#comment54218>
Space after !
flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
<https://reviews.apache.org/r/12060/#comment54219>
additional space within (), missing space before.
flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
<https://reviews.apache.org/r/12060/#comment54220>
space issue
flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
<https://reviews.apache.org/r/12060/#comment54221>
space issue
- Hari Shreedharan
On Oct. 24, 2013, 10:18 p.m., Roshan Naik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12060/
> -----------------------------------------------------------
>
> (Updated Oct. 24, 2013, 10:18 p.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-1227
> https://issues.apache.org/jira/browse/FLUME-1227
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> Revised design for Spillable Mem Channel.
> We no longer have Spillable channel config pointing to another channel (by
> name) as in the previous design.
>
> Spillable Channel instead derives from FileChannel (as per
> https://issues.apache.org/jira/browse/FLUME-1227?focusedCommentId=13628201&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13628201)
>
> Essence of this design:
> - SC derives from File channel and maintains an in memory queue. If memory
> queue is full, events are sent to disk overflow (i.e. File channel).
> - SC maintains a 'Drain-Order' queue (DOQ) for remembering the order in which
> the incoming events were interleaved between main memory and
> - Put transaction: All the elements in the putList are committed to mem queue
> if it has space, else written to disk (ie file channel). Head of DOQ is
> updated to indicate where the elements put.
> - Take transaction : Tail of DOQ is consulted to determine whether the next
> set of events are to be taken from the memory queue or from disk overflow.
> DOQ's tail is updated after events are taken out.
>
>
> SC Configuration:
> - Accepts all the File Channel settings
> - Introduces one additional setting: 'memoryCapacity' which indicates the
> number of items it can hold in memory
>
>
> Sample config ...
>
> a1.channels = c1
> a1.sinks = logger
> a1.sources = src
>
> a1.sources.src.type = exec
> a1.sources.src.command = seq 1 100000
> a1.sources.src.batchSize = 10
> a1.sources.src.channels = c1
>
> a1.sinks.logger.type = logger
> a1.sinks.logger.channel = c1
>
> a1.channels.c1.type = spillablememory
> a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
> a1.channels.c1.dataDirs = /tmp/flume/data
> a1.channels.c1.memoryCapacity = 10
> a1.channels.c1.keep-alive = 2
>
>
> Diffs
> -----
>
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
> 36f150b
> flume-ng-channels/flume-spillable-memory-channel/pom.xml PRE-CREATION
>
> flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
> PRE-CREATION
>
> flume-ng-channels/flume-spillable-memory-channel/src/test/java/org/apache/flume/channel/TestSpillableMemoryChannel.java
> PRE-CREATION
> flume-ng-channels/pom.xml 5832ab4
>
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java
> 26f4dd7
>
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java
> 15b8cc3
> flume-ng-core/src/main/java/org/apache/flume/ChannelFullException.java
> PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java
> 1370e66
> flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java
> 688323d
> flume-ng-dist/pom.xml 2d0ee47
> flume-ng-doc/sphinx/FlumeUserGuide.rst 98859ce
>
> flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgentConfiguration.java
> 6204bc5
> flume-ng-node/pom.xml f1b0c65
> pom.xml 267925f
>
> Diff: https://reviews.apache.org/r/12060/diff/
>
>
> Testing
> -------
>
> Wrote a set of Unit tests. A few are failing and need to be finished up.
>
>
> File Attachments
> ----------------
>
> Revised design doc
>
> https://reviews.apache.org/media/uploaded/files/2013/06/24/SpillableMemory_Channel_Design_2_1.pdf
>
>
> Thanks,
>
> Roshan Naik
>
>
