No not at all. Flume's transactional model guarantees delivery between hops. https://blogs.apache.org/flume/entry/flume_ng_architecture
On Mon, Apr 7, 2014 at 10:16 AM, Christopher Shannon <cshannon...@gmail.com>wrote: > So, this basically means that Flume's transactional model is also > unreliable. That would have to mean that the downstream agent is sending an > ack to the upstream agent before it actually persists the event. > > On Apr 7, 2014 10:48 AM, "Jeff Lord" <jl...@cloudera.com> wrote: > > > > No. If you need to guarantee delivery of events please use a file > channel. > > https://blogs.apache.org/flume/entry/apache_flume_filechannel > > > > > > On Mon, Apr 7, 2014 at 8:38 AM, Christopher Shannon < > cshannon...@gmail.com> wrote: > >> > >> > >> On Apr 7, 2014 9:35 AM, "Jeff Lord" <jl...@cloudera.com> wrote: > >> > > >> > > >> > > >> > > >> > On Thu, Apr 3, 2014 at 11:27 AM, Mohit Durgapal < > durgapalmo...@gmail.com> wrote: > >> >> > >> >> Hi Jeff, > >> >> > >> >> Yes, I am using the memory channel, and that's because I want it to > be more reliable and not miss any events/messages. > >> >> As I've read in flume documentation that the memory channel is fast > but there could be a chance of missing events if the in-memory buffer fills > up. > >> > > >> > > >> > Memory channel is not reliable, meaning if the flume agent goes down > or is restarted while there are events in the channel than this data will > be lost. > >> > For reliability please use the file channel. > >> > > >> > >> Jeff, > >> > >> I am using an upstream agent with a spooling directory source and a > memory channel, and the downstream agent uses a memory channel and an HDFS > sink. If my downstream agent goes down for any reason, are the entries lost > in the downstream agent's memory channel still preserved in the memory > channel / file directory of the upstream agent? > >> > >> All the best, Chris > > > > > No. If you need to guarantee delivery of events please use a file > channel. > https://blogs.apache.org/flume/entry/apache_flume_filechannel > > > On Mon, Apr 7, 2014 at 8:38 AM, Christopher Shannon <cshannon...@gmail.com > > wrote: > >> >> On Apr 7, 2014 9:35 AM, "Jeff Lord" <jl...@cloudera.com> wrote: >> > >> > >> > >> > >> > On Thu, Apr 3, 2014 at 11:27 AM, Mohit Durgapal < >> durgapalmo...@gmail.com> wrote: >> >> >> >> Hi Jeff, >> >> >> >> Yes, I am using the memory channel, and that's because I want it to be >> more reliable and not miss any events/messages. >> >> As I've read in flume documentation that the memory channel is fast >> but there could be a chance of missing events if the in-memory buffer fills >> up. >> > >> > >> > Memory channel is not reliable, meaning if the flume agent goes down or >> is restarted while there are events in the channel than this data will be >> lost. >> > For reliability please use the file channel. >> > >> >> Jeff, >> >> I am using an upstream agent with a spooling directory source and a >> memory channel, and the downstream agent uses a memory channel and an HDFS >> sink. If my downstream agent goes down for any reason, are the entries lost >> in the downstream agent's memory channel still preserved in the memory >> channel / file directory of the upstream agent? >> >> All the best, Chris >> > >
