yes. The shellscript has to be execution rights, for sure. Alex
sent via my mobile device On May 18, 2012, at 8:28 PM, "M@he$h" <mahesh...@gmail.com> wrote: > Hello Alex, > > Thanks a lot for the reply and the links. > > so can I use something like this : > > exec-agent.sources.tail.command = filename.sh > > filename.sh > > #!/bin/bash > > budate=`date '+%m%d%Y'` > filenames="/usr/local/current/logs/omq.log.$budate" > > tail -f $filenames > > > On Fri, May 18, 2012 at 11:16 AM, alo alt <wget.n...@googlemail.com> wrote: > small introducing: > http://mapredit.blogspot.de/2012/03/flumeng-evolution.html > > Exec-source: > http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.0.0b2/FlumeUserGuide.html#h.8astmk8ul1ya > > with example ;) > > exec-agent.sources = THE_NAME > exec-agent.channels = memoryChannel-1 > exec-agent.sinks = logger > exec-agent.sources.tail.type = exec > exec-agent.sources.tail.command = tail -f /var/log/secure > > Instead tail -f you can use anything what you want, that command will be > executed. > > - Alex > > -- > Alexander Alten-Lorenz > http://mapredit.blogspot.com > German Hadoop LinkedIn Group: http://goo.gl/N8pCF > > On May 18, 2012, at 7:45 PM, M@he$h wrote: > > > @Ychu, > > > > tailDir(("/usr/local/current/logs/","omq.log.%m%d%Y",false, 1) > > > > did not work :( flume is not producing any logs on the collector side and > > there are no errors as well i.e. agent and client are running. > > > > > > On Thu, May 17, 2012 at 7:08 PM, ychu <y...@isoftstone.com> wrote: > > Sorry > > > > that I told you is invalid: > > here is some valid example > > > > tailDir("/var/log/app") > > // all files > > > > tailDir("/var/log/app",".*\.log") // all files with names that match the > > ".*\.log" regex (in shell this > > is *.log) > > > > tailDir("/var/log/app",".*\.log", false, 1) // all files with names that > > match the ".*\.log" > > regex, starting from beginning of file, with one level of recursion depth. > > > > tailDir("/var/log/app", delim="\n\n", delimMode="exclude") // all files > > with names that match the ".*\.log" regex, starting from beginning of file, > > with one level of recursion depth, that end with double new lines, > > excluding the double new > > lines > > > > tailDir("/var/log/app",".*\.log", false, 1, delim="\n\n", > > delimMode="exclude") // all files with names that match the ".*\.log" > > regex, starting from beginning of file, with one level of recursion depth, > > that end with double new lines, excluding the double new lines > > > > > > so in your condition you should try: > > > > tailDir(("/usr/local/current/logs/","omq.log.%m%d%Y",false, 1) > > > > > > Best wishes > > > > 2012-05-18 > > > > 胡媛聪 > > > > 软通动力信息技术（集团）有限公司 > > > > 地址：深圳市南山区高新区高新南九道9号威新软件科技园8号楼3层 > > > > 邮编：518057 > > > > 电话：0755-26719999 > > > > 手机：13556898820 > > > > Email：y...@isoftstone.com > > > > 发件人： M@he$h > > 发送时间： 2012-05-18 09:59:48 > > 收件人： flume-user@incubator.apache.org > > 抄送： > > 主题： Re: tailDir > > @Raji : > > > > Firstly , we cannot remove old logs from the directory by running cron job. > > > > and yes , I want flume to ignore previous day logs for the current day as > > you said below. > > > > @Yui : > > > > let me try this and will let you know whether it worked or not. > > > > tailDir("/usr/local/current/logs/",fileregex="omq.log.%m%d%Y",startFromEnd=true,recurseDepth=3) > > > > > > On Thu, May 17, 2012 at 6:46 PM, Rajalakshmi Ramesh > > <rrajalaks...@gatech.edu> wrote: > > you can try it out. I am not really sure. But, I am having trouble seeing > > your reasoning here. Say some external application creates such log files > > for you every night at 12 AM, then basically you want to ignore all the > > remaining data until 11.59 without passing them over to flume? > > > > Thanks & Regards > > Rajalakshmi (Raji) Ramesh > > ------------------------------------------------------- > > Graduate Student > > Georgia Institute of Technology > > Atlanta, GA > > ------------------------------------------------------- > > > > > > > > On Thu, May 17, 2012 at 6:41 PM, M@he$h <mahesh...@gmail.com> wrote: > > Hello Raji, > > > > Thanks for the reply. > > > > so using flume's tailDir , we cannot achieve what I want is it? > > > > does something like below will work or not? > > > > tailDir("/usr/local/current/logs/", "rls.log.%M%d%Y" , "true") > > > > > > On Thu, May 17, 2012 at 6:35 PM, Rajalakshmi Ramesh > > <rrajalaks...@gatech.edu> wrote: > > I would suggest moving that intelligence to a cron job by removing log > > files that ypu dont want to be tail at the end of the day. Will make your > > configuration much easier. > > > > > > alakshmi (Raji) Ramesh > > ------------------------------------------------------- > > Graduate Student > > Georgia Institute of Technology > > Atlanta, GA > > ------------------------------------------------------- > > > > > > > > On Thu, May 17, 2012 at 6:31 PM, M@he$h <mahesh...@gmail.com> wrote: > > Hello, > > > > We have a directory where there will be log files for each day being > > written. for example : > > > > abc.log.05162012 > > pqr.log.05162012 > > abc.log.05172012 > > pqr.log.05172012 > > > > > > I want flume to tail files depending on that day's date. for example : as > > today's date is 05/17/2012 , it should only tail file names which have > > 05172012 at the end. > > > > Can we do it using tailDir? or can you suggest the best way to do this in > > flume? > > > > Look forward for your response. > > > > > > -- > > Thanks and Regards, > > > > Mahesh > > 619-816-7011. > > > > > > > > > > -- > > Thanks and Regards, > > > > Mahesh > > 619-816-7011. > > > > > > > > > > -- > > Thanks and Regards, > > > > Mahesh > > 619-816-7011. > > > > > > > > -- > > Thanks and Regards, > > > > Mahesh > > 619-816-7011. > > > > > -- > Thanks and Regards, > > Mahesh > 619-816-7011.
