[
https://issues.apache.org/jira/browse/FLUME-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13738649#comment-13738649
]
Hari Shreedharan commented on FLUME-2155:
-----------------------------------------
Here are some numbers for just the current remove algorithm:
Assume a queue with 1 million elements - where element 400,000 to 450,000 are
removed.
Each remove causes every element above it to get moved down. So remove element#
400000 causes 399,999 moves, removing 400,001 causes another 399,999 (since
there are only 399,999 elements before that now), removing 400,002 causes
another 399,999 etc.
Total number of copy operations = 399,999 * 50,000 = 19999950000 -> This
probably will take forever to complete.
With the new algorithm, all the 50,000 elements are simply marked as removed.
We start from 1 millionth element and move up and finally hit a REMOVED_MARKER
at 450,000 (which is the first place where writePos is not changed with
readPos) - so the first copy is 399,999 to 450,000 so the total moves =
399,999. I noticed that there is a bug in the compact pseudo-code. Will fix and
update.
> Improve replay time
> -------------------
>
> Key: FLUME-2155
> URL: https://issues.apache.org/jira/browse/FLUME-2155
> Project: Flume
> Issue Type: Bug
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Attachments: SmartReplay.pdf
>
>
> File Channel has scaled so well that people now run channels with sizes in
> 100's of millions of events. Turns out, replay can be crazy slow even between
> checkpoints at this scale - because of the remove() method in FlumeEventQueue
> moving every pointer that follows the one being removed (1 remove causes 99
> million+ moves for a channel of 100 million!). There are several ways of
> improving - one being move at the end of replay - sort of like a compaction.
> Another is to use the fact that all removes happen from the top of the queue,
> so move the first "k" events out to hashset and remove from there - we can
> find k using the write id of the last checkpoint and the current one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
