Hello, I have a scenario where in the client application is continuously pushing xml messages. Actually the application is writing these messages to files (new files; same directory). So we will be keep getting new files throughout the day. I am trying to configure Flume agents on these applcation servers (4 of them) to pick up the new data and transfer them to HDFS on a hadoop cluster. How should I configure my source to pick up new files (and exclude the files that have been processed already)? I don't think Exec source with tail -F will work in this scenario because data is not getting added to existing files; rather new files get created.
Thank you very much for your time and support. Sadu
