I am using a AMQP Souce, so I don't know how changing to a JMS source would have any difference.
I am concerned about the volume of data and the file channel. Even if I switched to JMS, my question would be the same. On Fri, Oct 4, 2013 at 4:46 PM, Hari Shreedharan <hshreedha...@cloudera.com>wrote: > Have you tried the JMS Source? It can pick up data directly into Flume. > > > Thanks, > Hari > > On Friday, October 4, 2013 at 11:59 AM, David Sinclair wrote: > > Hi, > > I have a question regarding the RollingFileSink and > SpoolingDirectorySource. I was trying to write everything from an AMQP > source to a file sink, then have the spooling directory source pick up > these files. This won't work as the files aren't immutable. > > If I use a File Channel to store the events between my source and sink, is > there a concern about the number of events in the channel if the sink is > unable to deliver said events? For example, I will be getting around 5K > messages/sec and the size is about 2K. So roughly 10MB a second. If the > sink is unable to deliver the messages for 2 hours, that would be 36 > million events in the channel. > > Is the file channel designed to handle this? Or should I have a file sink > in between. > > thanks > > dave > > >
