On Linux: dmesg
or /var/log/messages - Alex On Jun 1, 2013, at 4:49 PM, Lenin Raj <[email protected]> wrote: > Hi Brock, > > Which log file should I check under /var/log? > > Following are the list of files under /var/log modified from yesterdat: > > -rw-r----- 1 syslog adm 1227 2013-05-31 06:25 syslog.2.gz > -rw-r----- 1 mysql adm 20 2013-05-31 06:25 mysql.log.1.gz > -rw-rw-r-- 1 root utmp 40704 2013-05-31 23:15 wtmp.1 > -rw-r----- 1 syslog adm 14296 2013-06-01 06:25 syslog.1 > -rw-r--r-- 1 root root 0 2013-06-01 06:25 dpkg.log > drwxr-xr-x 2 root root 4096 2013-06-01 06:25 apt > -rw-r--r-- 1 root root 0 2013-06-01 06:25 alternatives.log > -rw-r----- 1 mysql adm 0 2013-06-01 06:25 mysql.log > drwxr-s--- 2 Debian-exim adm 4096 2013-06-01 06:25 exim4 > -rw-rw---- 1 root utmp 0 2013-06-01 06:25 btmp > -rw-r----- 1 syslog adm 119666 2013-06-01 18:03 kern.log > -rw-r----- 1 syslog adm 8845 2013-06-01 21:09 syslog > -rw-r----- 1 syslog adm 4713281 2013-06-01 21:14 auth.log > -rw-rw-r-- 1 root utmp 3456 2013-06-01 21:14 wtmp > -rw-r--r-- 1 root root 293752 2013-06-01 21:14 lastlog > > > > Thanks, > Lenin > > > On Sat, Jun 1, 2013 at 9:12 PM, Brock Noland <[email protected]> wrote: > >> Weird... It could have been the linux oom killer? You'd see something in >> /var/log/message if that was the case. >> >> >> On Sat, Jun 1, 2013 at 2:47 AM, Lenin Raj <[email protected]> wrote: >> >>> Hello, >>> >>> I have a flume service which pulls twitter data and sinks to HDFS. I >>> started it last night at 8 PM. It was going well. When I checked the status >>> today morning, it seems flume service stopped at around 11:15 PM. >>> >>> I could not find any valid reason in the log file. Flume is doing "Flume >>> node stopping - TwitterAgent" automatically without throwing any error. >>> >>> >>> 31 May 2013 23:13:32,226 INFO [hdfs-HDFS-call-runner-7] >>> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating >>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414624.tmp >>> 31 May 2013 23:14:02,247 INFO [hdfs-HDFS-roll-timer-0] >>> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming >>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624.tmp >>> to >>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624 >>> 31 May 2013 23:14:26,260 INFO [hdfs-HDFS-call-runner-1] >>> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating >>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414625.tmp >>> 31 May 2013 23:14:56,288 INFO [hdfs-HDFS-roll-timer-0] >>> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming >>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625.tmp >>> to >>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625 >>> 31 May 2013 23:15:02,317 INFO [hdfs-HDFS-call-runner-5] >>> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating >>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414626.tmp >>> 31 May 2013 23:15:32,340 INFO [hdfs-HDFS-roll-timer-0] >>> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming >>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626.tmp >>> to >>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626 >>> 31 May 2013 23:15:43,632 INFO [hdfs-HDFS-call-runner-9] >>> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating >>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414627.tmp >>> 31 May 2013 23:15:56,604 INFO [node-shutdownHook] >>> (org.apache.flume.node.FlumeNode.stop:67) - Flume node stopping - >>> TwitterAgent >>> 31 May 2013 23:15:56,604 INFO [node-shutdownHook] >>> (org.apache.flume.lifecycle.LifecycleSupervisor.stop:78) - Stopping >>> lifecycle supervisor 7 >>> 31 May 2013 23:15:56,606 INFO [node-shutdownHook] >>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stop:215) - >>> Node manager stopping >>> 31 May 2013 23:15:56,606 INFO [node-shutdownHook] >>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:68) >>> - Shutting down configuration: { >>> sourceRunners:{Twitter=EventDrivenSourceRunner: { >>> source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} }} >>> sinkRunners:{HDFS=SinkRunner: { >>> policy:org.apache.flume.sink.DefaultSinkProcessor@1d80e6d counterGroup:{ >>> name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=1252} >>> } }} channels:{MemChannel=org.apache.flume.channel.MemoryChannel{name: >>> MemChannel}} } >>> 31 May 2013 23:15:56,607 INFO [node-shutdownHook] >>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:72) >>> - Stopping Source Twitter >>> 31 May 2013 23:15:56,607 INFO [node-shutdownHook] >>> (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:156) - >>> Stopping component: EventDrivenSourceRunner: { >>> source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} } >>> 31 May 2013 23:15:56,682 WARN [Twitter Stream consumer-1[Disposing >>> thread]] (twitter4j.internal.logging.SLF4JLogger.warn:99) - Stream already >>> closed. >>> 31 May 2013 23:15:56,682 INFO [node-shutdownHook] >>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:82) >>> - Stopping Sink HDFS >>> 31 May 2013 23:15:56,683 INFO [node-shutdownHook] >>> (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:156) - >>> Stopping component: SinkRunner: { >>> policy:org.apache.flume.sink.DefaultSinkProcessor@1d80e6d counterGroup:{ >>> name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=1252} } } >>> 31 May 2013 23:15:56,754 WARN >>> [SinkRunner-PollingRunner-DefaultSinkProcessor] >>> (org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout:372) - >>> Unexpected Exception null >>> java.lang.InterruptedException >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325) >>> at >>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257) >>> at java.util.concurrent.FutureTask.get(FutureTask.java:119) >>> at >>> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345) >>> at >>> org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741) >>> at >>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443) >>> at >>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) >>> at >>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) >>> at java.lang.Thread.run(Thread.java:679) >>> 31 May 2013 23:15:56,797 ERROR >>> [SinkRunner-PollingRunner-DefaultSinkProcessor] >>> (org.apache.flume.sink.hdfs.HDFSEventSink.process:460) - process failed >>> java.lang.InterruptedException >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325) >>> at >>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257) >>> at java.util.concurrent.FutureTask.get(FutureTask.java:119) >>> at >>> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345) >>> at >>> org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741) >>> at >>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443) >>> at >>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) >>> at >>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) >>> at java.lang.Thread.run(Thread.java:679) >>> 31 May 2013 23:15:56,797 ERROR >>> [SinkRunner-PollingRunner-DefaultSinkProcessor] >>> (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver >>> event. Exception follows. >>> org.apache.flume.EventDeliveryException: java.lang.InterruptedException >>> at >>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:464) >>> at >>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) >>> at >>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) >>> at java.lang.Thread.run(Thread.java:679) >>> Caused by: java.lang.InterruptedException >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325) >>> at >>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257) >>> at java.util.concurrent.FutureTask.get(FutureTask.java:119) >>> at >>> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345) >>> at >>> org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741) >>> at >>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443) >>> ... 3 more >>> 31 May 2013 23:16:01,798 INFO [node-shutdownHook] >>> (org.apache.flume.sink.hdfs.HDFSEventSink.stop:475) - Closing >>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/20//FlumeData >>> 31 May 2013 23:16:01,811 INFO [hdfs-HDFS-call-runner-4] >>> (org.apache.flume.sink.hdfs.BucketWriter.doClose:283) - HDFSWriter is >>> already closed: >>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/20//FlumeData.1370013399555.tmp >>> 31 May 2013 23:16:01,812 INFO [node-shutdownHook] >>> (org.apache.flume.sink.hdfs.HDFSEventSink.stop:475) - Closing >>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/21//FlumeData >>> >>> Any reason Flume stopped the service? >>> >>> Thanks, >>> Lenin >>> >> >> >> >> -- >> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org >>
