Have you tried the JMS Source? It can pick up data directly into Flume.
Thanks, Hari On Friday, October 4, 2013 at 11:59 AM, David Sinclair wrote: > Hi, > > I have a question regarding the RollingFileSink and SpoolingDirectorySource. > I was trying to write everything from an AMQP source to a file sink, then > have the spooling directory source pick up these files. This won't work as > the files aren't immutable. > > If I use a File Channel to store the events between my source and sink, is > there a concern about the number of events in the channel if the sink is > unable to deliver said events? For example, I will be getting around 5K > messages/sec and the size is about 2K. So roughly 10MB a second. If the sink > is unable to deliver the messages for 2 hours, that would be 36 million > events in the channel. > > Is the file channel designed to handle this? Or should I have a file sink in > between. > > thanks > > dave
