[
https://issues.apache.org/jira/browse/FLUME-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855332#comment-13855332
]
Roshan Naik commented on FLUME-2282:
------------------------------------
The exception was thrown because the channel was shut as part of the
reconfiguration (which changed manually)
.
Reproduced this by simply updating the sink batch size on the avro sink and the
spooldir sources in the config file of a running agent and tailed the logs,
thats it.
Also tried a quick patch in SpoolDir by explicitly catching the IllegalState
Exception and introducing a few seconds of sleep before retrying (similar to
the backof logic that was introduced recently). Once the file channel finished
replaying the spoolDir resume normal operations.
There are no other errors prior to this instance as such, but some general INFO
& WARN relating to what appears to be active source and sinks threads not
being able to make progress on their open transactions.
> File Channel throws IllegalStateException causing some sources to not be to
> recover from a reconfig
> ---------------------------------------------------------------------------------------------------
>
> Key: FLUME-2282
> URL: https://issues.apache.org/jira/browse/FLUME-2282
> Project: Flume
> Issue Type: Bug
> Components: Configuration, File Channel
> Reporter: Roshan Naik
> Labels: channel
>
> when the config file is updated and the file channel is shut, the Source
> receives a java.lang.IllegalState exception from FileChannel. Unfortunately
> unless sources are designed to catch all exception types, they will not know
> this is a channel exception and likely to go into fuzzy states.
> For instance the FLUME-2255 patch catches only ChannelException.
> The below stack trace can be seen in logs when this happens with FLUME-2255
> applied:
> {code}
> 21 Dec 2013 21:15:03,030 ERROR [pool-17-thread-1]
> (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:194)
> - Uncaught exception in SpoolDirSource thread
> java.lang.IllegalStateException: Channel closed [channel=channel_nontx]
> at
> org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:351)
> at
> org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:181)
> at
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:179)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
