Hi I was testing flume for log transfer in DFO mode. I was firing the single shot command as cron every 10th minute from the agent box. So the flume process starts every 10 minutes sends the logs to collector box and ends.
It was pushing the logs very well. To test the reliability I stopped the collector and brought it back after 1 hour, what I see is * The flume process (started by the agent cron) kept on running in the agent machine and not able to transfer the rest of the file to collector. * The process did not end even after 5/6 hours. (I had to manually kill it). As I know flume agent should continue sending the log once the collector is re-started. ??? Is there something I am missing ?? Thanks Chandrashekhar Shaw On Jul 24, 2012, at 6:50 PM, Harish Mandala wrote: Hello, Not too sure about Flume OG - I barely used it for a week before shifting to NG. Would there be any particular reason you are using OG? Yes, you can take a look at SyslogTcpSource (in NG), https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/apidocs/org/apache/flume/source/SyslogTcpSource.html This can be modified according to this Netty example (for asynchronous large file streaming) : http://static.netty.io/3.5/xref/org/jboss/netty/example/http/file/package-summary.html This is pretty much what I did when I needed a Flume HTTP source to match my requirements. Regards, Harish On Tue, Jul 24, 2012 at 3:02 AM, Chandrashekhar Shaw <[email protected]<mailto:[email protected]>> wrote: I am using Flume - OG https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz . It there an available version which implements netty?? Thanks Chandrashekhar Shaw On Jul 23, 2012, at 8:00 PM, Harish Mandala wrote: Hi, Which one of the sources are you using? Maybe a flume source that implements something like http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful. Regards, Harish On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw <[email protected]<mailto:[email protected]>>wrote: Hi, I am using flume for transferring logs from 20 agent boxes to 1 collector box. The file size which I am trying to transfer is of about 1 GB per hour. So every hour my agents run to transfer the log file generated during the last hour. I am using DFO sink for file transfer. But then I am facing following problems: 1) Some of the agents are not able to transfer the whole file even in one hour and the processes run continuously for hours. 2) Some of the agents stop within 5-6 minutes without sending the file completely. I want to know 1) if somebody else has faced the same issues? 2) What could be the problem? Thanks Chandrashekhar Shaw
