Hi, I'm trying to use flumes Spooling Directory Source to move my logs from ec2 instances to aws s3.
I'm using rotatelogs to create log files of 5MB sizes. I have one doubt. If I specify the spooldir sources directory as the directory to which the apache creates log, will it work correctly. What I mean is, will flume wait till apache completes writing to the log file? What I observe is, only couple of lines of the log file is coming in the sink, but in the source side has more lines. I dont whether I put my question is correctly worder. My configruation files below. Apache configuration ----------------------------- CustomLog "|/opt/bitnami/apache2/bin/rotatelogs -l /mnt/je/logs/apache/jesites/epicenter/access/%Y-%m-%d-%H-%M-%S-access.log 5M" cookie env=!dontlog CustomLog "|/opt/bitnami/apache2/bin/rotatelogs -l /mnt/je/logs/apache/jesites/epicenter/access/%Y-%m-%d-%H-%M-%S-access_assets.log 5M" combined env=dontlog ErrorLog "|/opt/bitnami/apache2/bin/rotatelogs -l /mnt/je/logs/apache/jesites/epicenter/error/%Y-%m-%d-%H-%M-%S-errorlog.log 5M" Flume configuration --------------------------- #source is of type spooling directory - epicenter access logs agent1.sources.spooldir-epi-access.channels = ch1 agent1.sources.spooldir-epi-access.type = spooldir agent1.sources.spooldir-epi-access.spoolDir = /mnt/je/logs/apache/jesites/epicenter/access agent1.sources.spooldir-epi-access.interceptors = i1 hostname type agent1.sources.spooldir-epi-access.interceptors.i1.type = timestamp agent1.sources.spooldir-epi-access.interceptors.hostname.type = host agent1.sources.spooldir-epi-access.interceptors.hostname.useIP = false agent1.sources.spooldir-epi-access.interceptors.hostname.preserveExisting = true agent1.sources.spooldir-epi-access.interceptors.type.type = static agent1.sources.spooldir-epi-access.interceptors.type.key = type agent1.sources.spooldir-epi-access.interceptors.type.value = epi-access agent1.sources.spooldir-epi-access.fileHeader = true -- Regards, Robert George http://justEat.in | [email protected] | +919986442677
