[
https://issues.apache.org/jira/browse/FLUME-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033047#comment-14033047
]
Phil Scala commented on FLUME-2385:
-----------------------------------
Ed / Justin,
Looking at the code, there seems to be a code change that introduced this
behavior in the spool dir source,
Commit:
{noformat}
SHA-1: a345f611ba814c3689420edf4c2cc9b69ba3ac70
* FLUME-2283: Spool Dir source must check interrupt flag before writing to
channel
(Hari Shreedharan via Jarek Jarcec Cecho)
{noformat}
introduced this info logging statement and its logged all of the time,
irrespective if there is an error (you don't know how long I looked around for
an error today when I saw this being logged). The code in question is:
{code:java title=SpoolDirectorySource}
while (!Thread.interrupted()) {
List<Event> events = reader.readEvents(batchSize);
if (events.isEmpty()) {
break;
}
sourceCounter.addToEventReceivedCount(events.size());
sourceCounter.incrementAppendBatchReceivedCount();
try {
getChannelProcessor().processEventBatch(events);
reader.commit();
} catch (ChannelException ex) {
...
}
continue;
}
backoffInterval = 250;
sourceCounter.addToEventAcceptedCount(events.size());
sourceCounter.incrementAppendBatchAcceptedCount();
}
logger.info("Spooling Directory Source runner has shutdown.");
....
{code}
the issue is events.isEmpty() returns true when there is nothing new in the
spool dir, this calls the break and so the statement is logged. Since this is
called on a schedule, 500ms later the same thing happens over and over. I
believe the logging statement is not accurate because the source has not
shutdown, its just done checking for files for this iteration.
the fix to this issue is simply to remove this logging statement, which I will
submit a patch for.
> Flume spans log file with "Spooling Directory Source runner has shutdown"
> messages at INFO level
> ------------------------------------------------------------------------------------------------
>
> Key: FLUME-2385
> URL: https://issues.apache.org/jira/browse/FLUME-2385
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.4.0
> Reporter: Justin Hayes
> Priority: Minor
>
> When I start an agent with the following config, the spooling directory
> source emits "14/05/14 22:36:12 INFO source.SpoolDirectorySource: Spooling
> Directory Source runner has shutdown." messages twice a second. Pretty
> innocuous but it will fill up the file system needlessly and get in the way
> of other INFO messages.
> cis.sources = httpd
> cis.sinks = loggerSink
> cis.channels = mem2logger
> cis.sources.httpd.type = spooldir
> cis.sources.httpd.spoolDir = /var/log/httpd
> cis.sources.httpd.trackerDir = /var/lib/flume-ng/tracker/httpd
> cis.sources.httpd.channels = mem2logger
> cis.sinks.loggerSink.type = logger
> cis.sinks.loggerSink.channel = mem2logger
> cis.channels.mem2logger.type = memory
> cis.channels.mem2logger.capacity = 10000
> cis.channels.mem2logger.transactionCapacity = 1000
--
This message was sent by Atlassian JIRA
(v6.2#6252)
