[
https://issues.apache.org/jira/browse/FLUME-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229072#comment-13229072
]
[email protected] commented on FLUME-896:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4325/#review5926
-----------------------------------------------------------
This looks like a great start on a tough task...
I'm unsure about one thing that doesn't seem to be covered by tests:
When parallel threads each have their own transactions, there is no guarantee
that FileChannel.commitSequenceId is called in seqid order. Some threads will
already have committed some or all of their entries, but because of a lower
commitSequenceId entry the entries will still be "revived" in replay.
e.g. thread a and b do some puts/takes and commit
action | seqId | lastSequenceId
a.put | 1
b.put | 2
b.put | 3
a.put | 4
b.take | | 3
a.commit
b.commit
at this point the sequenceId file would just contain 3
events would contain 1,2,3,4
channel a would contain 1,4
channel b would contain 2
Now the process dies, we bring it back, replay the WAL:
- replaying the sequenceId file(s) would detect 3 as the oldest commitSequenceId
- only entry 4 would be > 3 and thus restored to entries
- entries 1,2 which were never take()d have gone missing.
Does this fall within the purview of best effort delivery(so it is acceptable
to lose these since we are not guaranteeing delivery)?
Or am I missing something?
- Juhani
On 2012-03-14 01:41:47, Brock Noland wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4325/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-14 01:41:47)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Attached patch not meant for commit. Just posting here for easy review.
bq.
bq.
bq. This addresses bug FLUME-896.
bq. https://issues.apache.org/jira/browse/FLUME-896
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-channels/flume-file-channel/pom.xml ee2d20f
bq.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
a279453
bq.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelEvent.java
PRE-CREATION
bq.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/wal/WAL.java
PRE-CREATION
bq.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/wal/WALDataFile.java
PRE-CREATION
bq.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/wal/WALEntry.java
PRE-CREATION
bq.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/wal/WALReplyResult.java
PRE-CREATION
bq.
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
ab66998
bq.
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/wal/TestWAL.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/4325/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Brock
bq.
bq.
> Implement file write ahead log channel
> --------------------------------------
>
> Key: FLUME-896
> URL: https://issues.apache.org/jira/browse/FLUME-896
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Affects Versions: NG alpha 1
> Reporter: E. Sammer
> Assignee: E. Sammer
> Fix For: v1.2.0
>
> Attachments: FLUME-896-1.patch
>
>
> Implement a channel that uses a regular file system and a write ahead log for
> durable event delivery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
