[
https://issues.apache.org/jira/browse/FLUME-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182439#comment-13182439
]
Praveen Ramachandra commented on FLUME-896:
-------------------------------------------
Hi Peter,
I am not sure if this is the best approach for a system that has to deal with
lots and lots of data.
Some of the things we need to validate against
1. What if the event source produce an event at a time, essentially transaction
boundary is one event
2. What is the system behavior if 99.99% of the transactions doesn't do a
rollback i.e., actually does the commit
3. How do we support applications who need a relaxed guarantees as opposed to
complete guarantees like a database system.
On the contrary, some questions that we could ask ourselves is
1. Why cant we store the uncommitted data in memory. If the server (i.e., flume
node) goes down then only uncommitted data is lost and hence is not an issue.
Application can deal with such scenarios
2. Even if we persist uncommitted data, how does a source request to resurrect
a transaction once a node comes back up. There is no such thing as "oh! this is
the addressable handle of my last transaction, please continue from where we
left off". Even if we can build such complex semantics, important question is
do we really need to?
As I have said in the past please (please) look at current implementations that
solve the same problem and port the working implementation (e.g., scribe,
kafka). We don't have the baggage of JMS, JDBC etc., and we can build the
reliability and keep it tunable for individual application needs.
Hope this helps.
--
Regards,
Praveen Ramachandra
> Implement file write ahead log channel
> --------------------------------------
>
> Key: FLUME-896
> URL: https://issues.apache.org/jira/browse/FLUME-896
> Project: Flume
> Issue Type: New Feature
> Components: Channel
> Affects Versions: NG alpha 1
> Reporter: E. Sammer
> Assignee: Peter Newcomb
> Fix For: v1.1.0
>
>
> Implement a channel that uses a regular file system and a write ahead log for
> durable event delivery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
