Hi Ashish , I used recursiveDirectorySearch in my conf and set it to true . Yet I am unable to read content under subdirectory .
Regards, PavaN -----Original Message----- From: Jakati, Pavan Sent: Monday, July 20, 2015 6:08 PM To: [email protected] Subject: RE: How to read content from files which are in nested directory Hi Ashish Thanks for the help. Can you pls help with procedure to update or pull the patch into my system . Thanks. Regards, PavaN -----Original Message----- From: Ashish [mailto:[email protected]] Sent: Saturday, July 18, 2015 12:44 AM To: [email protected] Subject: Re: How to read content from files which are in nested directory One more way to achieve the same is using this patch https://issues.apache.org/jira/browse/FLUME-1899 On Fri, Jul 17, 2015 at 7:14 AM, Johny Rufus <[email protected]> wrote: > The spooling directory source as of now supports only reading from a > flat directory and wont read files from subdirectories. > You could write an external script that transfers all the files in all > the date directories to a common directory which spooling source > points to. (If this fits your use case) > > Thanks, > Rufus > > On Fri, Jul 17, 2015 at 6:40 AM, Jakati, Pavan <[email protected]> wrote: >> >> Hi Folks, >> >> >> >> I am consuming Apache flume to read logs which are stored in >> following format , logs under date directory . How do I read the logs >> using spooldir source. >> >> >> >> >> >> Directory Structure : >> >> >> >> >> >> flume]# ll /ADAPTORS/MAIL_CONNECT/logs/ >> >> total 0 >> >> drwxr-xr-x 2 root root 29 Jun 3 18:04 06-03-15 >> >> drwxr-xr-x 2 root root 56 Jun 4 18:16 06-04-15 >> >> drwxr-xr-x 2 root root 29 Jun 5 14:03 06-05-15 >> >> drwxr-xr-x 2 root root 29 Jun 8 12:43 06-08-15 >> >> drwxr-xr-x 2 root root 29 Jun 9 14:47 06-09-15 >> >> drwxr-xr-x 2 root root 29 Jun 10 18:49 06-10-15 >> >> drwxr-xr-x 2 root root 29 Jun 11 17:22 06-11-15 >> >> drwxr-xr-x 2 root root 29 Jun 12 11:37 06-12-15 >> >> drwxr-xr-x 2 root root 29 Jun 15 11:39 06-15-15 >> >> drwxr-xr-x 2 root root 29 Jun 24 10:45 06-23-15 >> >> drwxr-xr-x 2 root root 29 Jun 24 10:35 06-24-15 >> >> drwxr-xr-x 2 root root 29 Jun 25 17:28 06-25-15 >> >> >> >> >> >> My current Configuration : >> >> agent1.sources.source1.type = spooldir >> >> agent1.sources.source1.channels = channel1 >> >> agent1.sources.source1.spoolDir = >> /ADAPTORS/MAIL_CONNECT/logs/07-17-15 >> >> agent1.sources.source1.fileHeader = true >> >> >> >> >> >> Pls suggest if i need to use some other source type . >> >> >> >> Can I use any script inside conf file to change the directory >> dynamically . Thanks >> >> >> >> >> >> Regards, >> >> PavaN >> >> > > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
