Hadoop is working fine Total files: 19125 (Files currently being written: 2) Total blocks (validated): 19740 (avg. block size 2303408 B) (Total open file blocks (not validated): 2) Minimally replicated blocks: 19740 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 4 Number of racks: 1 FSCK ended at Mon Aug 13 14:18:26 EDT 2012 in 498 milliseconds
The filesystem under path '/' is HEALTHY On Mon, Aug 13, 2012 at 2:13 PM, Alexander Lorenz <[email protected]>wrote: > Hi, > > looks like your HDFS are in an error state: > Exiting driver logicalNode > hadoop1.liaisondevqa.local-21 in error state > > check please that your cluster is running well. > > cheers, > Alex > > > On Aug 13, 2012, at 10:53 AM, Sandeep Reddy P <[email protected]> > wrote: > > > Hi, > > No, i followed apache flume docs. I'll try following cloudera flume. Here > > are my error logs > > 2012-08-13 11:22:49,059 INFO > > com.cloudera.flume.handlers.thrift.ThriftEventSource: Closed server on > port > > 35862... > > 2012-08-13 11:22:49,059 INFO > > com.cloudera.flume.handlers.thrift.ThriftEventSource: Queue still has 0 > > elements ... > > 2012-08-13 11:22:49,059 INFO > com.cloudera.flume.handlers.rolling.RollSink: > > closing RollSink > > 'escapedCustomDfs("hdfs://hadoop1.liaisondevqa.local/user/flume/ > > ","syslog%{rolltag}" )' > > 2012-08-13 11:22:49,059 ERROR > > com.cloudera.flume.core.connector.DirectDriver: Exiting driver > logicalNode > > hadoop1.liaisondevqa.local-21 in error state CollectorSource | Collector > > because Waiting for queue element was interrupted! null > > 2012-08-13 11:22:49,060 INFO > > com.cloudera.flume.handlers.thrift.ThriftEventSource: Starting blocking > > thread pool server on port 35862... > > 2012-08-13 11:22:49,061 ERROR > > org.apache.thrift.server.TSaneThreadPoolServer: Error occurred during > > listening. > > org.apache.thrift.transport.TTransportException: Could not create > > ServerSocket on address 0.0.0.0/0.0.0.0:35862. > > at > > > org.apache.thrift.transport.TSaneServerSocket.bind(TSaneServerSocket.java:110) > > at > > > org.apache.thrift.transport.TSaneServerSocket.listen(TSaneServerSocket.java:116) > > at > > > org.apache.thrift.server.TSaneThreadPoolServer.start(TSaneThreadPoolServer.java:162) > > at > > > com.cloudera.flume.handlers.thrift.ThriftEventSource.open(ThriftEventSource.java:151) > > at > > > com.cloudera.flume.collector.CollectorSource.open(CollectorSource.java:67) > > at > > > com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:87) > > 2012-08-13 11:22:49,061 INFO > com.cloudera.flume.handlers.rolling.RollSink: > > opening RollSink > > 'escapedCustomDfs("hdfs://hadoop1.liaisondevqa.local/user/hdfs/ > > ","syslog%{rolltag}" )' > > 2012-08-13 11:22:49,062 INFO > > com.cloudera.flume.handlers.debug.InsistentOpenDecorator: Opened > > MaskDecorator on try 0 > > 2012-08-13 11:30:10,056 INFO > com.cloudera.flume.handlers.rolling.RollSink: > > Created RollSink: trigger=[TimeTrigger: maxAge=30000 > > tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@10cb42cf] > > checkPeriodMs = 250 > > spec='escapedCustomDfs("hdfs://hadoop1.liaisondevqa.local/user/hdfs/ > > ","raw=%{rolltag}" )' > > 2012-08-13 11:30:20,058 WARN com.cloudera.flume.agent.LivenessManager: > > Heartbeats are backing up, currently behind by 1 heartbeats > > 2012-08-13 11:30:25,061 WARN com.cloudera.flume.agent.LivenessManager: > > Heartbeats are backing up, currently behind by 2 heartbeats > > 2012-08-13 11:30:30,063 WARN com.cloudera.flume.agent.LivenessManager: > > Heartbeats are backing up, currently behind by 3 heartbeats > > 2012-08-13 11:30:35,065 WARN com.cloudera.flume.agent.LivenessManager: > > Heartbeats are backing up, currently behind by 4 heartbeats > > 2012-08-13 11:30:40,056 ERROR com.cloudera.flume.agent.LogicalNode: > Forcing > > driver to exit uncleanly > > 2012-08-13 11:30:40,057 ERROR > > com.cloudera.flume.core.connector.DirectDriver: Closing down due to > > exception during append calls > > 2012-08-13 11:30:40,057 INFO > > com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode > > hadoop1.liaisondevqa.local-23 exited with error: Waiting for queue > element > > was interrupted! null > > java.io.IOException: Waiting for queue element was interrupted! null > > at > > > com.cloudera.flume.handlers.thrift.ThriftEventSource.next(ThriftEventSource.java:222) > > at > > > com.cloudera.flume.collector.CollectorSource.next(CollectorSource.java:72) > > at > > > com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:108) > > Caused by: java.lang.InterruptedException > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1961) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2038) > > at > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) > > at > > > com.cloudera.flume.handlers.thrift.ThriftEventSource.next(ThriftEventSource.java:209) > > ... 2 more > > > > > > On Mon, Aug 13, 2012 at 12:03 PM, alo alt <[email protected]> wrote: > > > >> Hi, > >> > >> Did you follow the userguide, archived here: > >> http://archive.cloudera.com/cdh/3/flume-0.9.1+1/UserGuide.html ? > >> > >> Without error messages or logfiles it's hard to say whats really happen. > >> > >> - Alex > >> > >> > >> On Aug 13, 2012, at 8:12 AM, Sandeep Reddy P < > [email protected]> > >> wrote: > >> > >>> Hi, > >>> I'm using flume version Flume 0.9.4-cdh3u4 > >>> I'm using flume master webpage to configure the following to move data > >> into > >>> hdfs. > >>> > >>> host : syslogTcp(5140) | agentSink("hadoop1.liaisondevqa.local",35862) > ; > >>> hadoop1.liaisondevqa.local : collectorSource(35862) | > >>> collectorSink("hdfs://hadoop1.liaisondevqa.local/user/flume/ > ","syslog"); > >>> > >>> Command history says its successful but i cant see any data in hdfs. > >>> > >>> Similarly how should i configure to move a log file from linux box to > >> hdfs > >>> using flume? I'm following apache flume cookbook. > >>> -- > >>> Thanks, > >>> sandeep > >>> > >>> > >>> > >>> > >>> -- > >>> Thanks, > >>> sandeep > >> > >> > >> -- > >> Alexander Alten-Lorenz > >> http://mapredit.blogspot.com > >> German Hadoop LinkedIn Group: http://goo.gl/N8pCF > >> > >> > > > > > > -- > > Thanks, > > sandeep > > > -- > Alexander Alten-Lorenz > http://mapredit.blogspot.com > German Hadoop LinkedIn Group: http://goo.gl/N8pCF > > -- Thanks, sandeep
