[
https://issues.apache.org/jira/browse/FLUME-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229165#comment-13229165
]
[email protected] commented on FLUME-896:
-----------------------------------------------------
bq. On 2012-03-14 08:27:28, Juhani Connolly wrote:
bq. > This looks like a great start on a tough task...
bq. >
bq. > I'm unsure about one thing that doesn't seem to be covered by tests:
bq. >
bq. > When parallel threads each have their own transactions, there is no
guarantee that FileChannel.commitSequenceId is called in seqid order. Some
threads will already have committed some or all of their entries, but because
of a lower commitSequenceId entry the entries will still be "revived" in replay.
bq. >
bq. > e.g. thread a and b do some puts/takes and commit
bq. >
bq. > action | seqId | lastSequenceId
bq. > a.put | 1
bq. > b.put | 2
bq. > b.put | 3
bq. > a.put | 4
bq. > b.take | | 3
bq. > a.commit
bq. > b.commit
bq. >
bq. > at this point the sequenceId file would just contain 3
bq. > events would contain 1,2,3,4
bq. > channel a would contain 1,4
bq. > channel b would contain 2
bq. >
bq. > Now the process dies, we bring it back, replay the WAL:
bq. >
bq. > - replaying the sequenceId file(s) would detect 3 as the oldest
commitSequenceId
bq. > - only entry 4 would be > 3 and thus restored to entries
bq. > - entries 1,2 which were never take()d have gone missing.
bq. >
bq. > Does this fall within the purview of best effort delivery(so it is
acceptable to lose these since we are not guaranteeing delivery)?
bq. > Or am I missing something?
bq.
bq. Juhani Connolly wrote:
bq. I was going to write a unit test but there's no mechanism to force a
WAL write. While changing implementation for testing isn't good, I think it may
make sense to roll the wall when doClose is called on FileChannel.
bq.
bq. Juhani Connolly wrote:
bq. Also, I got the example wrong(because of the order take's are made
in)... the lastSequenceId should be 2, resulting in 3,4, getting recovered but
1 still missing.
bq.
bq. With that being said, have you considered treating take's and put's as
separate entries in the wal, with commit id's assigned to them at commit time.
You can then periodically "compact" the wal by replaying its contents, removing
matched takes/puts(you may want to look at how this is done in HBase)?
Yep I think you are correct, say we have two threads doing takes. Thread 1
does a take and gets Event(SeqID=5), Thread 2 does a take and gets
Events(SeqID=6). Thread 2 commits and Thread 1 rollback. At this point
Event(SeqID=5) is stored only in memory as it will not get replayed if the
process should restart.
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4325/#review5926
-----------------------------------------------------------
On 2012-03-14 01:41:47, Brock Noland wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4325/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-14 01:41:47)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Attached patch not meant for commit. Just posting here for easy review.
bq.
bq.
bq. This addresses bug FLUME-896.
bq. https://issues.apache.org/jira/browse/FLUME-896
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-channels/flume-file-channel/pom.xml ee2d20f
bq.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
a279453
bq.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelEvent.java
PRE-CREATION
bq.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/wal/WAL.java
PRE-CREATION
bq.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/wal/WALDataFile.java
PRE-CREATION
bq.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/wal/WALEntry.java
PRE-CREATION
bq.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/wal/WALReplyResult.java
PRE-CREATION
bq.
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
ab66998
bq.
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/wal/TestWAL.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/4325/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Brock
bq.
bq.
> Implement file write ahead log channel
> --------------------------------------
>
> Key: FLUME-896
> URL: https://issues.apache.org/jira/browse/FLUME-896
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Affects Versions: NG alpha 1
> Reporter: E. Sammer
> Assignee: E. Sammer
> Fix For: v1.2.0
>
> Attachments: FLUME-896-1.patch
>
>
> Implement a channel that uses a regular file system and a write ahead log for
> durable event delivery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
