weird i have been seeing the same thing. Do you have the node and collector on different hosts? Are you using hbase by chance?
On Mon, Oct 17, 2011 at 1:48 PM, Stephen Layland <[email protected]>wrote: > Hi, after letting flume idle for the weekend listening in on a small stream > of live data, we noticed several of our flume collector nodes failing with > InterruptedException's being thrown. Logs have errors that look something > like below. Any idea of what's going on here and how to fix it? > > 2011-10-17 08:02:02,107 INFO >> com.cloudera.flume.handlers.debug.StubbornAppendSink: append Interrupted >> event 'flume-node.lindenlab.com' [INFO Mon Oct 17 08:01:59 UTC 2011] { >> syslogfacility : 16 } { syslogseverity : 6 } Oct 17 01:01:59 SOME MESSAGE' >> with error: Blocked append interrupted by rotation event >> 2011-10-17 08:02:02,107 INFO com.cloudera.flume.handlers.rolling.RollSink: >> closing RollSink >> 'escapedCustomDfs("hdfs://master-node:54310/logs/raw/%Y/%m/%d/%H00","test%{rolltag}" >> )' >> 2011-10-17 08:02:02,109 ERROR >> com.cloudera.flume.core.connector.DirectDriver: Closing down due to >> exception during append calls >> java.lang.InterruptedException >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1223) >> at >> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:976) >> at >> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:296) >> at >> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >> at >> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >> at >> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175) >> at >> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >> at >> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78) >> at >> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >> at >> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110) >> at >> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >> at >> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113) >> at >> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >> at >> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62) >> at >> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >> at >> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81) >> at >> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222) >> at >> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110) >> 2011-10-17 08:02:02,109 INFO >> com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode >> node6-22 exited with error: nulljava.lang.InterruptedException >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1223) >> >> at >> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:976) >> at >> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:296) >> at >> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >> at >> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >> at >> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175) >> at >> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >> at >> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78) >> at >> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >> at >> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110) >> at >> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >> at >> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113) >> at >> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >> at >> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62) >> at >> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >> at >> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81) >> at >> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222) >> at >> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110) >> 2011-10-17 08:02:02,109 INFO >> com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode >> node6-22 exited with error: null >> java.lang.InterruptedException >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1223) >> at >> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:976) >> at >> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:296) >> at >> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >> at >> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >> at >> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175) >> at >> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >> at >> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78) >> at >> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >> at >> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110) >> at >> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >> at >> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113) >> at >> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >> at >> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62) >> at >> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >> at >> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81) >> at >> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222) >> at >> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110) > > > Many thanks, > > -Steve >
