Hi Karim
On 10/13/2012 08:03 AM, Zamani, Karim wrote:
Hi,
Is Flume (or Flume-NG) suitable for collecting events such as alters
generated by various nodes in a distributed architecture to a central
location?
I read in the wiki FAQ that Flume was not designed to be a message
passing or eventing framework due to issues like message duplication,
ordering, etc.
Flume NG only guarrantees delivery(so long as you use durable channels
on every hop). Avoiding duplication is not a hard objective, and it is
possible. Same applies for message ordering.
That said, not all component will result in message order getting mixed
or duplicates. If you deliver messages in order to flume and sink them
without batching, along a single path, they should be in order and
without duplicates. However this puts some severe restrictions on
multiple delivery paths for redundancy and batching(not batching the
sink at all is going to hurt performance), so I couldn't claim flume is
ideal for a system where ordering/avoiding duplication is necessary.
Does Flume-NG have any enhancements to support message passing better?
Thanks!
Karim
