small introducing: http://mapredit.blogspot.de/2012/03/flumeng-evolution.html
Exec-source: http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.0.0b2/FlumeUserGuide.html#h.8astmk8ul1ya with example ;) exec-agent.sources = THE_NAME exec-agent.channels = memoryChannel-1 exec-agent.sinks = logger exec-agent.sources.tail.type = exec exec-agent.sources.tail.command = tail -f /var/log/secure Instead tail -f you can use anything what you want, that command will be executed. - Alex -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF On May 18, 2012, at 7:45 PM, M@he$h wrote: > @Ychu, > > tailDir(("/usr/local/current/logs/","omq.log.%m%d%Y",false, 1) > > did not work :( flume is not producing any logs on the collector side and > there are no errors as well i.e. agent and client are running. > > > On Thu, May 17, 2012 at 7:08 PM, ychu <y...@isoftstone.com> wrote: > Sorry > > that I told you is invalid: > here is some valid example > > tailDir("/var/log/app") > // all files > > tailDir("/var/log/app",".*\.log") // all files with names that match the > ".*\.log" regex (in shell this > is *.log) > > tailDir("/var/log/app",".*\.log", false, 1) // all files with names that > match the ".*\.log" > regex, starting from beginning of file, with one level of recursion depth. > > tailDir("/var/log/app", delim="\n\n", delimMode="exclude") // all files with > names that match the ".*\.log" regex, starting from beginning of file, with > one level of recursion depth, that end with double new lines, excluding the > double new > lines > > tailDir("/var/log/app",".*\.log", false, 1, delim="\n\n", > delimMode="exclude") // all files with names that match the ".*\.log" regex, > starting from beginning of file, with one level of recursion depth, that end > with double new lines, excluding the double new lines > > > so in your condition you should try: > > tailDir(("/usr/local/current/logs/","omq.log.%m%d%Y",false, 1) > > > Best wishes > > 2012-05-18 > > 胡媛聪 > > 软通动力信息技术（集团）有限公司 > > 地址：深圳市南山区高新区高新南九道9号威新软件科技园8号楼3层 > > 邮编：518057 > > 电话：0755-26719999 > > 手机：13556898820 > > Email：y...@isoftstone.com > > 发件人： M@he$h > 发送时间： 2012-05-18 09:59:48 > 收件人： flume-user@incubator.apache.org > 抄送： > 主题： Re: tailDir > @Raji : > > Firstly , we cannot remove old logs from the directory by running cron job. > > and yes , I want flume to ignore previous day logs for the current day as you > said below. > > @Yui : > > let me try this and will let you know whether it worked or not. > > tailDir("/usr/local/current/logs/",fileregex="omq.log.%m%d%Y",startFromEnd=true,recurseDepth=3) > > > On Thu, May 17, 2012 at 6:46 PM, Rajalakshmi Ramesh <rrajalaks...@gatech.edu> > wrote: > you can try it out. I am not really sure. But, I am having trouble seeing > your reasoning here. Say some external application creates such log files for > you every night at 12 AM, then basically you want to ignore all the remaining > data until 11.59 without passing them over to flume? > > Thanks & Regards > Rajalakshmi (Raji) Ramesh > ------------------------------------------------------- > Graduate Student > Georgia Institute of Technology > Atlanta, GA > ------------------------------------------------------- > > > > On Thu, May 17, 2012 at 6:41 PM, M@he$h <mahesh...@gmail.com> wrote: > Hello Raji, > > Thanks for the reply. > > so using flume's tailDir , we cannot achieve what I want is it? > > does something like below will work or not? > > tailDir("/usr/local/current/logs/", "rls.log.%M%d%Y" , "true") > > > On Thu, May 17, 2012 at 6:35 PM, Rajalakshmi Ramesh <rrajalaks...@gatech.edu> > wrote: > I would suggest moving that intelligence to a cron job by removing log files > that ypu dont want to be tail at the end of the day. Will make your > configuration much easier. > > > alakshmi (Raji) Ramesh > ------------------------------------------------------- > Graduate Student > Georgia Institute of Technology > Atlanta, GA > ------------------------------------------------------- > > > > On Thu, May 17, 2012 at 6:31 PM, M@he$h <mahesh...@gmail.com> wrote: > Hello, > > We have a directory where there will be log files for each day being written. > for example : > > abc.log.05162012 > pqr.log.05162012 > abc.log.05172012 > pqr.log.05172012 > > > I want flume to tail files depending on that day's date. for example : as > today's date is 05/17/2012 , it should only tail file names which have > 05172012 at the end. > > Can we do it using tailDir? or can you suggest the best way to do this in > flume? > > Look forward for your response. > > > -- > Thanks and Regards, > > Mahesh > 619-816-7011. > > > > > -- > Thanks and Regards, > > Mahesh > 619-816-7011. > > > > > -- > Thanks and Regards, > > Mahesh > 619-816-7011. > > > > -- > Thanks and Regards, > > Mahesh > 619-816-7011.
