Hi Johny
Thanks for the update. Can you pls let me know if I can add the script into my flume conf and do the required actions. I do not wish to run script outside flume. Regards, PavaN From: Johny Rufus [mailto:[email protected]] Sent: Friday, July 17, 2015 7:45 PM To: [email protected] Subject: Re: How to read content from files which are in nested directory The spooling directory source as of now supports only reading from a flat directory and wont read files from subdirectories. You could write an external script that transfers all the files in all the date directories to a common directory which spooling source points to. (If this fits your use case) Thanks, Rufus On Fri, Jul 17, 2015 at 6:40 AM, Jakati, Pavan <[email protected]<mailto:[email protected]>> wrote: Hi Folks, I am consuming Apache flume to read logs which are stored in following format , logs under date directory . How do I read the logs using spooldir source. Directory Structure : flume]# ll /ADAPTORS/MAIL_CONNECT/logs/ total 0 drwxr-xr-x 2 root root 29 Jun 3 18:04 06-03-15 drwxr-xr-x 2 root root 56 Jun 4 18:16 06-04-15 drwxr-xr-x 2 root root 29 Jun 5 14:03 06-05-15 drwxr-xr-x 2 root root 29 Jun 8 12:43 06-08-15 drwxr-xr-x 2 root root 29 Jun 9 14:47 06-09-15 drwxr-xr-x 2 root root 29 Jun 10 18:49 06-10-15 drwxr-xr-x 2 root root 29 Jun 11 17:22 06-11-15 drwxr-xr-x 2 root root 29 Jun 12 11:37 06-12-15 drwxr-xr-x 2 root root 29 Jun 15 11:39 06-15-15 drwxr-xr-x 2 root root 29 Jun 24 10:45 06-23-15 drwxr-xr-x 2 root root 29 Jun 24 10:35 06-24-15 drwxr-xr-x 2 root root 29 Jun 25 17:28 06-25-15 My current Configuration : agent1.sources.source1.type = spooldir agent1.sources.source1.channels = channel1 agent1.sources.source1.spoolDir = /ADAPTORS/MAIL_CONNECT/logs/07-17-15 agent1.sources.source1.fileHeader = true Pls suggest if i need to use some other source type . Can I use any script inside conf file to change the directory dynamically . Thanks Regards, PavaN
