Hi All, I am using spooling directory source With Flume 1.4.0 (HDP) . Nginx is set up in a way that it rotates logs into spool directory every 15 minutes.
I am seeing this exception quite often with flume agents (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:173) - Uncaught exception in Runnable java.lang.IllegalStateException: Serializer has been closed at org.apache.flume.serialization.LineDeserializer.ensureOpen(LineDeserializer.java:124) at org.apache.flume.serialization.LineDeserializer.readEvents(LineDeserializer.java:88) at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:221) at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:160) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) and it just get stuck and needs Flume restart . I found these two tickets describing same issue, https://issues.apache.org/jira/browse/FLUME-2160 https://issues.apache.org/jira/browse/FLUME-1887 Has anyone faced similar issues and is there any workaround to avoid this ? Thanks, Snehal
