I see，there is no easy way or configure way to know the detail of what has sent and what haven't. I have to write my own source or sink code to do this. Thank you,Alex and all friends.
Andy 2013/2/6 Alexander Alten-Lorenz <[email protected]> > You haven't a control in such situations, since tailDir uses tail and > holds the marker in memory. > > We had few days ago a thread about: > > http://search-hadoop.com/m/JV0lh2RDXLX/flume+tail+source+problem+and+performance&subj=flume+tail+source+problem+and+performance > > - Alex > > On Feb 6, 2013, at 3:45 AM, 周梦想 <[email protected]> wrote: > > > Hello, > > > > I'm using tailDirs('mydir') source of the agent to gather logs to hadoop > > hdfs. I notice some documents advise that if the agent collapsed, I have > > to remove files in 'mydir' and clear flume.agent.logdir. Thus I will lose > > some data or have duplicate data. And I don't know which line the agent > > have sent to. > > > > I'm worrying about the agent failure and resend or miss-send the content > to > > collector. I want to know how to check which line of log file the agent > > have sent if the agent exit suddenly. The files in flute log dir, such as > > sending,sent can't be read. > > > > Please give some advise to process such situation. > > Thanks. > > > > Andy Zhou > > -- > Alexander Alten-Lorenz > http://mapredit.blogspot.com > German Hadoop LinkedIn Group: http://goo.gl/N8pCF > >
