[
https://issues.apache.org/jira/browse/FLUME-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187712#comment-14187712
]
Patrick Dvorak commented on FLUME-2525:
---------------------------------------
When a partition containing the trackerDir (or spoolDir in the event trackerDir
is not specified) fills up, a new meta file can be created, but it will have a
zero byte size. This prevents flume from properly processing any files once
the partition full issue has been resolved, even across restarts. The error
thrown does not really indicate the root issue. Ideally flume should remove
any zero byte spooldir meta files when it starts up, as these are invalid.
> flume should handle a zero byte .flumespool-main.meta file for the spooldir
> source
> ----------------------------------------------------------------------------------
>
> Key: FLUME-2525
> URL: https://issues.apache.org/jira/browse/FLUME-2525
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.5.0.1
> Reporter: Patrick Dvorak
>
> When a zero byte .flumespool-main.meta file exists in the trackerDir (usually
> do to the partition filling up), flume will throw the following ambiguous
> error message when trying to read in new spool files:
> 2014-10-19 18:28:31,333 ERROR
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader: Exception
> opening file: /home/spooldir/input.log
> java.io.IOException: Not a data file.
> at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:102)
> at org.apache.avro.file.DataFileReader.<init>(DataFileReader.java:97)
> at org.apache.avro.file.DataFileWriter.appendTo(DataFileWriter.java:160)
> at org.apache.avro.file.DataFileWriter.appendTo(DataFileWriter.java:149)
> at
> org.apache.flume.serialization.DurablePositionTracker.<init>(DurablePositionTracker.java:141)
>
> at
> org.apache.flume.serialization.DurablePositionTracker.getInstance(DurablePositionTracker.java:76)
>
> at
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile(ReliableSpoolingFileEventReader.java:420)
>
> at
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:215)
>
> at
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:182)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
> Restarts of the flume agent do not resolve the issue. Only when the zero
> byte file is removed, will flume properly start processing files from the
> spooldir again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
