Thanks Jeff. This is useful. Can the spoolDir be on a different machine? We may have to setup a different process to copy files into 'spoolDir', right? Note: We have 'read only' access to these files. Any recommendations about this?
On Wed, Apr 16, 2014 at 5:16 PM, Jeff Lord <[email protected]> wrote: > http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source > > > On Wed, Apr 16, 2014 at 5:14 PM, Something Something < > [email protected]> wrote: > >> Hello, >> >> Needless to say I am newbie to Flume, but I've got a basic flow working >> in which I am importing a log file from my linux box to hdfs. I am using >> >> a1.sources.r1.command = tail -F /var/log/xyz.log >> >> which is working like a stream of messages. This is good! >> >> Now what I want to do is copy log files from a directory on a remote >> machine on a regular basis. For example: >> >> username@machinename:/var/log/logdir/<multiple files> >> >> One way to do it is to simply 'scp' files from the remote directory into >> my box on a regular basis, but what's the best way to do this in Flume? >> Please let me know. >> >> Thanks for the help. >> >> >> >
