Hi, we're not actually using flume nodes, but just the collector at the moment. We're listening on a syslog port and dumping straight to HDFS for now. After some digging, I'm pretty sure it's related to this:
https://issues.apache.org/jira/browse/FLUME-757 -Steve On Mon, Oct 17, 2011 at 11:53 AM, AD <[email protected]> wrote: > weird i have been seeing the same thing. Do you have the node and > collector on different hosts? Are you using hbase by chance? > > > On Mon, Oct 17, 2011 at 1:48 PM, Stephen Layland < > [email protected]> wrote: > >> Hi, after letting flume idle for the weekend listening in on a small >> stream of live data, we noticed several of our flume collector nodes failing >> with InterruptedException's being thrown. Logs have errors that look >> something like below. Any idea of what's going on here and how to fix it? >> >> 2011-10-17 08:02:02,107 INFO >>> com.cloudera.flume.handlers.debug.StubbornAppendSink: append Interrupted >>> event 'flume-node.lindenlab.com' [INFO Mon Oct 17 08:01:59 UTC 2011] { >>> syslogfacility : 16 } { syslogseverity : 6 } Oct 17 01:01:59 SOME MESSAGE' >>> with error: Blocked append interrupted by rotation event >>> 2011-10-17 08:02:02,107 INFO >>> com.cloudera.flume.handlers.rolling.RollSink: closing RollSink >>> 'escapedCustomDfs("hdfs://master-node:54310/logs/raw/%Y/%m/%d/%H00","test%{rolltag}" >>> )' >>> 2011-10-17 08:02:02,109 ERROR >>> com.cloudera.flume.core.connector.DirectDriver: Closing down due to >>> exception during append calls >>> java.lang.InterruptedException >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1223) >>> at >>> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:976) >>> at >>> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:296) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>> at >>> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>> at >>> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>> at >>> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>> at >>> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>> at >>> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>> at >>> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81) >>> at >>> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222) >>> at >>> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110) >>> 2011-10-17 08:02:02,109 INFO >>> com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode >>> node6-22 exited with error: nulljava.lang.InterruptedException >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1223) >>> >>> at >>> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:976) >>> at >>> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:296) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>> at >>> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>> at >>> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>> at >>> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>> at >>> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>> at >>> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>> at >>> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81) >>> at >>> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222) >>> at >>> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110) >>> 2011-10-17 08:02:02,109 INFO >>> com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode >>> node6-22 exited with error: null >>> java.lang.InterruptedException >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1223) >>> at >>> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:976) >>> at >>> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:296) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>> at >>> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>> at >>> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>> at >>> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>> at >>> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>> at >>> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62) >>> at >>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>> at >>> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81) >>> at >>> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222) >>> at >>> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110) >> >> >> Many thanks, >> >> -Steve >> > >
