[
https://issues.apache.org/jira/browse/FLUME-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229216#comment-13229216
]
[email protected] commented on FLUME-896:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4325/
-----------------------------------------------------------
(Updated 2012-03-14 14:11:18.627099)
Review request for Flume.
Changes
-------
Once again, for anyone watching. I am not proposing this for immediate commit.
It's more of a thought process/discussion.
The attached is less sexy, but would work in your scenario. Basically we read
all the sequence ids into a large array (for memory savings), sort it, and then
only replay entries not in the array. The memory consumption here is not
terrible. Let's say we had 5GB of events on disk at 100 bytes an item. That
translates into 53687091 items at 8 bytes an long that's 400MB of ram replay
the 5GB of events.
Summary
-------
Attached patch not meant for commit. Just posting here for easy review.
This addresses bug FLUME-896.
https://issues.apache.org/jira/browse/FLUME-896
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/pom.xml ee2d20f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
a279453
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelEvent.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/wal/WAL.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/wal/WALDataFile.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/wal/WALEntry.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/wal/WALReplyResult.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
ab66998
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/wal/TestWAL.java
PRE-CREATION
Diff: https://reviews.apache.org/r/4325/diff
Testing
-------
Thanks,
Brock
> Implement file write ahead log channel
> --------------------------------------
>
> Key: FLUME-896
> URL: https://issues.apache.org/jira/browse/FLUME-896
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Affects Versions: NG alpha 1
> Reporter: E. Sammer
> Assignee: E. Sammer
> Fix For: v1.2.0
>
> Attachments: FLUME-896-1.patch
>
>
> Implement a channel that uses a regular file system and a write ahead log for
> durable event delivery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
