Hi, I have a scenario where we are ingesting the log files(around 80MB each) in flume and flume does process these files and marks as completed but after processing few files it does not process any files further and moreover it does not mark the log file as completed.
We are using spooling directory source. I looked into the flume logs and found that when this scenario happens it shows the following line continuously in flume logs. DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/commerce-sense.conf for changes We have following configuration. agent.sources = spoolDir agent.channels = memoryChannel agent.sinks = sink agent.sources.spoolDir.interceptors = i1 #Channel Configuration agent.channels.memoryChannel.type = memory #Source configuration agent.sources.spoolDir.type = spooldir agent.sources.spoolDir.spoolDir = /opt/flume/spoolDir agent.sources.spoolDir.fileHeader = true agent.sources.spoolDir.basenameHeader = true agent.sources.spoolDir.deserializer = LINE agent.sources.spoolDir.inputCharset = ISO8859-1 agent.sources.spoolDir.deserializer.maxLineLength = 10000 agent.sources.spoolDir.interceptors.i1.type = org.apache.flume.sink.solr.morphline.UUIDInterceptor$Builder agent.sources.spoolDir.interceptors.i1.preserveExisting = true agent.sources.spoolDir.interceptors.i1.prefix = test agent.sources.spoolDir.channels = memoryChannel #Sink Configuration agent.sinks.sink.type = com. flume.sink.ExtendedKafkaSink agent.sinks.sink.topic = cdnLogsTopic agent.sinks.sink.brokerList = localhost:9092 agent.sinks.sink.batchSize = 100 agent.sinks.sink.sink.serializer = com. flume.serializer.ExtendedSerializer$Builder agent.sinks.sink.channel = memoryChannel Thanks, Saurabh
