Not sure that this helps, ... have you checked your operating system settings for the maximum number of open files?
Met vriendelijke groet, Ronald van de Kuil > Op 20 apr. 2016 om 08:48 heeft Saurabh Sharma <[email protected]> het > volgende geschreven: > > Hi, > > I have a scenario where we are ingesting the log files(around 80MB each) in > flume and flume does process these files and marks as completed but after > processing few files it does not process any files further and moreover it > does not mark the log file as completed. > > We are using spooling directory source. > > I looked into the flume logs and found that when this scenario happens it > shows the following line continuously in flume logs. > > DEBUG [conf-file-poller-0] > (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) > - Checking file:../conf/commerce-sense.conf for changes > > We have following configuration. > agent.sources = spoolDir > agent.channels = memoryChannel > agent.sinks = sink > agent.sources.spoolDir.interceptors = i1 > > > #Channel Configuration > agent.channels.memoryChannel.type = memory > > #Source configuration > agent.sources.spoolDir.type = spooldir > agent.sources.spoolDir.spoolDir = /opt/flume/spoolDir > agent.sources.spoolDir.fileHeader = true > agent.sources.spoolDir.basenameHeader = true > agent.sources.spoolDir.deserializer = LINE > agent.sources.spoolDir.inputCharset = ISO8859-1 > agent.sources.spoolDir.deserializer.maxLineLength = 10000 > agent.sources.spoolDir.interceptors.i1.type = > org.apache.flume.sink.solr.morphline.UUIDInterceptor$Builder > agent.sources.spoolDir.interceptors.i1.preserveExisting = true > agent.sources.spoolDir.interceptors.i1.prefix = test > agent.sources.spoolDir.channels = memoryChannel > > > #Sink Configuration > agent.sinks.sink.type = com. flume.sink.ExtendedKafkaSink > agent.sinks.sink.topic = cdnLogsTopic > agent.sinks.sink.brokerList = localhost:9092 > agent.sinks.sink.batchSize = 100 > agent.sinks.sink.sink.serializer = com. > flume.serializer.ExtendedSerializer$Builder > agent.sinks.sink.channel = memoryChannel > > Thanks, > Saurabh
