[
https://issues.apache.org/jira/browse/FLUME-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154883#comment-13154883
]
Praveen Ramachandra commented on FLUME-851:
-------------------------------------------
Hi Sammer,
I think this can work even without any custom time fields. We can look at the
scribe's mirror its implementation. All of scribe code is 6k C code, hopefully
it is simple to understand and mimic or better yet improve on it. At best we
can make it configurable at a channel level, with system wide default.
Ideal Behavior
When there is event backlog and new events come to the system, newer events are
delivered ahead of "backlogged" events. Once the backlog is cleared or when
there is no backlog in the system there is no deliberate/explicit change to the
order in which events are delivered i.e., system default behavior kicks in.
Implemented Behavior
Create 2 channels, one called backlog-bucket and the other sunnyday-bucket.
While the source is delivering the events to the bucket, it first checks if
there is more than "x" events in sunnyday-bucket, if true it delivers events to
backlog-bucket. Any throttling applied while draining the events from buckets is
applied only to backlog-bucket. The idea is that the platform would have
allocated enough resources to deal with sunny-day scenario with some added
buffer.
Hope this helps
> Scribe style of handling events especially when there is a large event
> backlog built-up
> ---------------------------------------------------------------------------------------
>
> Key: FLUME-851
> URL: https://issues.apache.org/jira/browse/FLUME-851
> Project: Flume
> Issue Type: Improvement
> Components: Channel, Node
> Affects Versions: NG alpha 1
> Environment: All
> Reporter: Praveen Ramachandra
>
> Newer events are of much higher value to be delivered than old events that
> have been accumulated.
> Scribe does this very well and would like to incorporate it in flume-ng.
> Currently we use a dirty hack to realize this important/critical feature. If
> this is incorporated in the platform, users of flume-ng will benefit from
> this immensly
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
