Can you take a thread dump, using kill -3 not jstack, when you hit that point?
On Monday, November 5, 2012 at 7:39 AM, Parag Hukeri wrote: > While starting my flume-agent I did use these options:- java -Xms512m > -Xmx2048m. But that did not help. What should I tweak in flume-env.sh? Not > sure? > I noticed that the source for fileChannel was included in flume 1.2. So I > tried compiling that & using that jar; but got the exact same results with > fileChannel - the process stopped after approx 9.5 hrs (after having written > approx 9.5 million records). > > Any help would be appreciated. Thanks. > > --- On Sat, 11/3/12, Alexander Lorenz <[email protected]> wrote: > > > > From: Alexander Lorenz <[email protected]> > > Subject: Re: ExecSource tail -F stops after a while > > To: [email protected] > > Date: Saturday, November 3, 2012, 1:55 AM > > > > Hi, > > > > http://flume.apache.org/FlumeUserGuide.html#exec-source > > Please see the WARNING ;) > > > > Anyhow, this error indicates that you've to less memory configured for too > > many events. Tune flume-env.sh and tweak JAVA_OPTS into "-Xms100m -Xmx200m" > > or something else > > > > FileChannel is available in from up to 1.3 > > http://flume.apache.org/FlumeUserGuide.html#recoverable-memory-channel > > > > cheers > > - Alex > > > > > > On Nov 3, 2012, at 1:21 AM, Parag Hukeri <[email protected] > > (http://us.mc1634.mail.yahoo.com/mc/[email protected])> > > wrote: > > > > > I tried to use the FileChannel; but am getting an exception - > > > > > > org.apache.flume.FlumeException: Unable to load channel type: file, > > > class: org.apache.flume.channel.file.FileChannel > > > Caused by: java.lang.ClassNotFoundException: > > > org.apache.flume.channel.file.FileChannel > > > > > > I am using apache-flume-1.2.0. What version of flume is needed for > > > FileChannel? > > > > > > Thanks > > > > > > --- On Fri, 11/2/12, Josh West <[email protected] > > > (http://us.mc1634.mail.yahoo.com/mc/[email protected])> wrote: > > > > > > > > > From: Josh West <[email protected] > > > (http://us.mc1634.mail.yahoo.com/mc/[email protected])> > > > Subject: Re: ExecSource tail -F stops after a while > > > To: [email protected] > > > (http://us.mc1634.mail.yahoo.com/mc/[email protected]) > > > Date: Friday, November 2, 2012, 1:48 PM > > > > > > > > > > > > This happened to me consistently when buffering tail -F contents into the > > > Memory Channel. Try switching to the File Channel. This solved the > > > issue for me. > > > > > > > > > On 11/02/2012 07:15 PM, Parag Hukeri wrote: > > > > > > > > > > > > > > > > > > > > > Hi > > > > > > I am using apache-flume-1.2.0. I have set up an ExecSource that tails a > > > log file & gets the entries from this log file. The log file size is > > > approx 4 GB (approx 12 million lines written over a period of 12 hrs). As > > > the process starts everything seems to work fine for approx 9.5 hrs. > > > (approx 9.5 million lines of log data is successfully retrieved by > > > ExecSource). It then stops reading the output from tail. I don't see any > > > exceptions being thrown. I don't think the problem is with the actual > > > tail command. I was able to successfully manually tail the log file & all > > > 12 million records were retrieved by tail. Running on a Linux system. Not > > > sure what the issue might be with ExecSource? Any help would be > > > appreciated. Thanks. > > > -- > > > Josh West > > > Lead Systems Administrator > > > One.com, [email protected] > > > (http://us.mc1634.mail.yahoo.com/mc/[email protected]) > > > > -- > > Alexander Alten-Lorenz > > http://mapredit.blogspot.com (http://mapredit.blogspot.com/) > > German Hadoop LinkedIn Group: http://goo.gl/N8pCF > >
