You haven't a control in such situations, since tailDir uses tail and holds the marker in memory.
We had few days ago a thread about: http://search-hadoop.com/m/JV0lh2RDXLX/flume+tail+source+problem+and+performance&subj=flume+tail+source+problem+and+performance - Alex On Feb 6, 2013, at 3:45 AM, 周梦想 <[email protected]> wrote: > Hello, > > I'm using tailDirs('mydir') source of the agent to gather logs to hadoop > hdfs. I notice some documents advise that if the agent collapsed, I have > to remove files in 'mydir' and clear flume.agent.logdir. Thus I will lose > some data or have duplicate data. And I don't know which line the agent > have sent to. > > I'm worrying about the agent failure and resend or miss-send the content to > collector. I want to know how to check which line of log file the agent > have sent if the agent exit suddenly. The files in flute log dir, such as > sending,sent can't be read. > > Please give some advise to process such situation. > Thanks. > > Andy Zhou -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF
