I increased it but no luck. I am using cp command instead of mv to place log file into the spooling directory. Will there be an issue?
Also when flume stops processing the file I could see the following line in logs continuously. DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume.conf for changes From: Ronald van de Kuil [mailto:[email protected]] Sent: 20 April 2016 14:12 To: [email protected] Subject: Re: Flume not marking log files as completed and do not process file further Could you set the value for the open files to 8000, reboot to OS, inspect that it is 8000 and see whether this resolves your problem? 2016-04-20 10:20 GMT+02:00 Chris Horrocks <[email protected]<mailto:[email protected]>>: Are the permissions on the files the same? Does the user running the flume agents have read permissions? Are the files still being written to/locked open by another process? Are there any logs being generated by the flume agent? -- Chris Horrocks On 20 April 2016 at 08:00:14, Saurabh Sharma ([email protected]<mailto:[email protected]>) wrote: Hi Ron, The maximum number of open files in our OS is 1024. Thanks From: Ronald Van De Kuil [mailto:[email protected]<mailto:[email protected]>] Sent: 20 April 2016 12:24 To: [email protected]<mailto:[email protected]> Subject: Re: Flume not marking log files as completed and do not process file further Not sure that this helps, ... have you checked your operating system settings for the maximum number of open files? Met vriendelijke groet, Ronald van de Kuil Op 20 apr. 2016 om 08:48 heeft Saurabh Sharma <[email protected]<mailto:[email protected]>> het volgende geschreven: Hi, I have a scenario where we are ingesting the log files(around 80MB each) in flume and flume does process these files and marks as completed but after processing few files it does not process any files further and moreover it does not mark the log file as completed. We are using spooling directory source. I looked into the flume logs and found that when this scenario happens it shows the following line continuously in flume logs. DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/commerce-sense.conf for changes We have following configuration. agent.sources = spoolDir agent.channels = memoryChannel agent.sinks = sink agent.sources.spoolDir.interceptors = i1 #Channel Configuration agent.channels.memoryChannel.type = memory #Source configuration agent.sources.spoolDir.type = spooldir agent.sources.spoolDir.spoolDir = /opt/flume/spoolDir agent.sources.spoolDir.fileHeader = true agent.sources.spoolDir.basenameHeader = true agent.sources.spoolDir.deserializer = LINE agent.sources.spoolDir.inputCharset = ISO8859-1 agent.sources.spoolDir.deserializer.maxLineLength = 10000 agent.sources.spoolDir.interceptors.i1.type = org.apache.flume.sink.solr.morphline.UUIDInterceptor$Builder agent.sources.spoolDir.interceptors.i1.preserveExisting = true agent.sources.spoolDir.interceptors.i1.prefix = test agent.sources.spoolDir.channels = memoryChannel #Sink Configuration agent.sinks.sink.type = com. flume.sink.ExtendedKafkaSink agent.sinks.sink.topic = cdnLogsTopic agent.sinks.sink.brokerList = localhost:9092 agent.sinks.sink.batchSize = 100 agent.sinks.sink.sink.serializer = com. flume.serializer.ExtendedSerializer$Builder agent.sinks.sink.channel = memoryChannel Thanks, Saurabh
