[
https://issues.apache.org/jira/browse/FLUME-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13749028#comment-13749028
]
Hari Shreedharan commented on FLUME-2173:
-----------------------------------------
Here is an initial stage algorithm I can think of:
# Insert uuid at the first agent when the event is received (or when the event
is created at the client SDK).
# At the destination HDFS (presumably we should be able to support this in all
sinks, including custom ones by abstracting this out into another library), at
the time of take, do a create for FLUME/<uuid>.
# If the create succeeds, this agent "owns" that event and writes it out to
HDFS.
# If the create fails, it means another agent will eventually write the event
out - so drop the event.
# After a configured time period for which we guarantee that the event will not
duplicate, delete the path which was created (where to do this is a good
question - presumably any agent should be able to do it).
This algorithm seems to guarantee that an event will eventually be written
since there is at least one agent that will not drop it, and barring hdfs
reporting false failures (actually writing the events but throwing exceptions)
and hdfs timeouts (due to which we don't know if the event really got written
or not), this algorithm should not cause duplicates.
Also, this algorithm assumes that an agent which dies will eventually come back
up and will be able to access the old disk on which it held its file channel
(this agent has to eventually come back up to write the event out - it does not
matter when it comes back, but it needs to - this brings up the question of
what happens if the agent comes back up and tries to write an event out because
it took it but never committed the transaction - we need to handle the
ownership case).
> Exactly once semantics for Flume
> --------------------------------
>
> Key: FLUME-2173
> URL: https://issues.apache.org/jira/browse/FLUME-2173
> Project: Flume
> Issue Type: Bug
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
>
> Currently Flume guarantees only at least once semantics. This jira is meant
> to track exactly once semantics for Flume. My initial idea is to include uuid
> event ids on events at the original source (use a config to mark a source an
> original source) and identify destination sinks. At the destination sinks,
> use a unique ZK Znode to track the events. If once seen (and configured),
> pull the duplicate out.
> This might need some refactoring, but my belief is we can do this in a
> backward compatible way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
