Hello, I have a question using spooldir source. If I have a large file such as more than 100MB, when I copy this file to spooldir, the flume agent will find it immediately and begin send data to another agent. If flume reading speed is faster than OS writing speed, it will reach end of file and begin change file name,but OS still write data to it. How to process this condition?
I encountered this situation in windows. a thread change file to .fin, report permission error. another thread change file name ok. and then this this thread compain "java.io.IOException: Stream closed" again and again. What should I do? below is the error log from windows. Thanks, Andy 27 Feb 2013 12:00:05,310 INFO [pool-5-thread-1] (org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile:229) - Preparing to move file D:\TKServer\HandResult\_BakLog\201302271200handresult_hllord.log to D:\TKServer\HandResult\_BakLog\201302271200handresult_hllord.log.fin 27 Feb 2013 12:00:05,310 ERROR [pool-5-thread-1] (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148) - Uncaught exception in Runnable org.apache.flume.FlumeException: Unable to move D:\TKServer\HandResult\_BakLog\201302271200handresult_hllord.log to D:\TKServer\HandResult\_BakLog\201302271200handresult_hllord.log.fin. This will likely cause duplicate events. Please verify that flume has sufficient permissions to perform these operations. at org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile(SpoolingFileLineReader.java:282) at org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:185) at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) 27 Feb 2013 12:00:05,825 ERROR [pool-5-thread-1] (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148) - Uncaught exception in Runnable java.io.IOException: Stream closed at java.io.BufferedReader.ensureOpen(BufferedReader.java:97) at java.io.BufferedReader.readLine(BufferedReader.java:292) at java.io.BufferedReader.readLine(BufferedReader.java:362) at org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:180) at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
