[
https://issues.apache.org/jira/browse/FLUME-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13282018#comment-13282018
]
Juhani Connolly commented on FLUME-1201:
----------------------------------------
this works for me too.
I do like the current buffer store in scribed, but the exact mechanics of it
can certainly not be setup with flume.
If we're happy with it not being configurable, limited to just memory/file,
then we could just implement it as a separate channel, though I suspect there
would be a fair bit of code repetition with existing channels, which in turn
would make for more maintenance.
The point about a compound channel quadratically increasing things to test for
is valid... Sort of... Ultimately though, it should not need to be aware of the
channels below it, and should only rely on the backing channels fulfilling
their contract, testing it would involve testing that only it works as intended.
Again though, if only having a memory channel spilling to file is enough, doing
an explicit implementation of that could work.
> Create a buffer channel, that stores overflow from a fast, low capacity
> channel to a slower high capacity channel
> -----------------------------------------------------------------------------------------------------------------
>
> Key: FLUME-1201
> URL: https://issues.apache.org/jira/browse/FLUME-1201
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Reporter: Juhani Connolly
> Assignee: Juhani Connolly
> Fix For: v1.2.0
>
>
> As it stands, users need to make a choice between either slower channels or
> faster ones. The current "middle ground" consists of RecoverableFileChannel
> which still stores everything to disk. MemoryChannel on the other hand has
> limited capacity or a high memory footprint.
> I propose to implement a buffer channel, somewhat like the buffer store in
> scribed.
> It would normally act as a proxy to the primary channel. Should this channel
> be unable to receive events(normally because it is at capacity, but perhaps
> some future channels may have other failure cases) it would switch states to
> buffering, storing new events to the secondary channel.
> In buffering, items continue to be read from the primary channel, and it
> attempts to "refill" itself from the secondary. Once the secondary is found
> to be empty, operation switches back to streaming mode, with items going
> directly to the primary.
> The main objective of this would thus be to have a high throughput channel as
> the primary mode of operation, allowing it to switch over when takes are not
> keeping up with puts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
