Weird... It could have been the linux oom killer? You'd see something in /var/log/message if that was the case.
On Sat, Jun 1, 2013 at 2:47 AM, Lenin Raj <[email protected]> wrote: > Hello, > > I have a flume service which pulls twitter data and sinks to HDFS. I > started it last night at 8 PM. It was going well. When I checked the status > today morning, it seems flume service stopped at around 11:15 PM. > > I could not find any valid reason in the log file. Flume is doing "Flume > node stopping - TwitterAgent" automatically without throwing any error. > > > 31 May 2013 23:13:32,226 INFO [hdfs-HDFS-call-runner-7] > (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating > hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414624.tmp > 31 May 2013 23:14:02,247 INFO [hdfs-HDFS-roll-timer-0] > (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming > hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624.tmp > to > hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624 > 31 May 2013 23:14:26,260 INFO [hdfs-HDFS-call-runner-1] > (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating > hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414625.tmp > 31 May 2013 23:14:56,288 INFO [hdfs-HDFS-roll-timer-0] > (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming > hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625.tmp > to > hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625 > 31 May 2013 23:15:02,317 INFO [hdfs-HDFS-call-runner-5] > (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating > hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414626.tmp > 31 May 2013 23:15:32,340 INFO [hdfs-HDFS-roll-timer-0] > (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming > hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626.tmp > to > hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626 > 31 May 2013 23:15:43,632 INFO [hdfs-HDFS-call-runner-9] > (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating > hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414627.tmp > 31 May 2013 23:15:56,604 INFO [node-shutdownHook] > (org.apache.flume.node.FlumeNode.stop:67) - Flume node stopping - > TwitterAgent > 31 May 2013 23:15:56,604 INFO [node-shutdownHook] > (org.apache.flume.lifecycle.LifecycleSupervisor.stop:78) - Stopping > lifecycle supervisor 7 > 31 May 2013 23:15:56,606 INFO [node-shutdownHook] > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stop:215) - > Node manager stopping > 31 May 2013 23:15:56,606 INFO [node-shutdownHook] > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:68) > - Shutting down configuration: { > sourceRunners:{Twitter=EventDrivenSourceRunner: { > source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} }} > sinkRunners:{HDFS=SinkRunner: { > policy:org.apache.flume.sink.DefaultSinkProcessor@1d80e6d counterGroup:{ > name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=1252} > } }} channels:{MemChannel=org.apache.flume.channel.MemoryChannel{name: > MemChannel}} } > 31 May 2013 23:15:56,607 INFO [node-shutdownHook] > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:72) > - Stopping Source Twitter > 31 May 2013 23:15:56,607 INFO [node-shutdownHook] > (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:156) - > Stopping component: EventDrivenSourceRunner: { > source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} } > 31 May 2013 23:15:56,682 WARN [Twitter Stream consumer-1[Disposing > thread]] (twitter4j.internal.logging.SLF4JLogger.warn:99) - Stream already > closed. > 31 May 2013 23:15:56,682 INFO [node-shutdownHook] > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:82) > - Stopping Sink HDFS > 31 May 2013 23:15:56,683 INFO [node-shutdownHook] > (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:156) - > Stopping component: SinkRunner: { > policy:org.apache.flume.sink.DefaultSinkProcessor@1d80e6d counterGroup:{ > name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=1252} } } > 31 May 2013 23:15:56,754 WARN > [SinkRunner-PollingRunner-DefaultSinkProcessor] > (org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout:372) - > Unexpected Exception null > java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325) > at > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257) > at java.util.concurrent.FutureTask.get(FutureTask.java:119) > at > org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345) > at > org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741) > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443) > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at > org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:679) > 31 May 2013 23:15:56,797 ERROR > [SinkRunner-PollingRunner-DefaultSinkProcessor] > (org.apache.flume.sink.hdfs.HDFSEventSink.process:460) - process failed > java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325) > at > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257) > at java.util.concurrent.FutureTask.get(FutureTask.java:119) > at > org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345) > at > org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741) > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443) > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at > org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:679) > 31 May 2013 23:15:56,797 ERROR > [SinkRunner-PollingRunner-DefaultSinkProcessor] > (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver > event. Exception follows. > org.apache.flume.EventDeliveryException: java.lang.InterruptedException > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:464) > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at > org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:679) > Caused by: java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325) > at > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257) > at java.util.concurrent.FutureTask.get(FutureTask.java:119) > at > org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345) > at > org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741) > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443) > ... 3 more > 31 May 2013 23:16:01,798 INFO [node-shutdownHook] > (org.apache.flume.sink.hdfs.HDFSEventSink.stop:475) - Closing > hdfs://localhost:54310/user/flume/tweets/2013/05/31/20//FlumeData > 31 May 2013 23:16:01,811 INFO [hdfs-HDFS-call-runner-4] > (org.apache.flume.sink.hdfs.BucketWriter.doClose:283) - HDFSWriter is > already closed: > hdfs://localhost:54310/user/flume/tweets/2013/05/31/20//FlumeData.1370013399555.tmp > 31 May 2013 23:16:01,812 INFO [node-shutdownHook] > (org.apache.flume.sink.hdfs.HDFSEventSink.stop:475) - Closing > hdfs://localhost:54310/user/flume/tweets/2013/05/31/21//FlumeData > > Any reason Flume stopped the service? > > Thanks, > Lenin > -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
