Hi, looks like your HDFS are in an error state: Exiting driver logicalNode hadoop1.liaisondevqa.local-21 in error state
check please that your cluster is running well. cheers, Alex On Aug 13, 2012, at 10:53 AM, Sandeep Reddy P <[email protected]> wrote: > Hi, > No, i followed apache flume docs. I'll try following cloudera flume. Here > are my error logs > 2012-08-13 11:22:49,059 INFO > com.cloudera.flume.handlers.thrift.ThriftEventSource: Closed server on port > 35862... > 2012-08-13 11:22:49,059 INFO > com.cloudera.flume.handlers.thrift.ThriftEventSource: Queue still has 0 > elements ... > 2012-08-13 11:22:49,059 INFO com.cloudera.flume.handlers.rolling.RollSink: > closing RollSink > 'escapedCustomDfs("hdfs://hadoop1.liaisondevqa.local/user/flume/ > ","syslog%{rolltag}" )' > 2012-08-13 11:22:49,059 ERROR > com.cloudera.flume.core.connector.DirectDriver: Exiting driver logicalNode > hadoop1.liaisondevqa.local-21 in error state CollectorSource | Collector > because Waiting for queue element was interrupted! null > 2012-08-13 11:22:49,060 INFO > com.cloudera.flume.handlers.thrift.ThriftEventSource: Starting blocking > thread pool server on port 35862... > 2012-08-13 11:22:49,061 ERROR > org.apache.thrift.server.TSaneThreadPoolServer: Error occurred during > listening. > org.apache.thrift.transport.TTransportException: Could not create > ServerSocket on address 0.0.0.0/0.0.0.0:35862. > at > org.apache.thrift.transport.TSaneServerSocket.bind(TSaneServerSocket.java:110) > at > org.apache.thrift.transport.TSaneServerSocket.listen(TSaneServerSocket.java:116) > at > org.apache.thrift.server.TSaneThreadPoolServer.start(TSaneThreadPoolServer.java:162) > at > com.cloudera.flume.handlers.thrift.ThriftEventSource.open(ThriftEventSource.java:151) > at > com.cloudera.flume.collector.CollectorSource.open(CollectorSource.java:67) > at > com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:87) > 2012-08-13 11:22:49,061 INFO com.cloudera.flume.handlers.rolling.RollSink: > opening RollSink > 'escapedCustomDfs("hdfs://hadoop1.liaisondevqa.local/user/hdfs/ > ","syslog%{rolltag}" )' > 2012-08-13 11:22:49,062 INFO > com.cloudera.flume.handlers.debug.InsistentOpenDecorator: Opened > MaskDecorator on try 0 > 2012-08-13 11:30:10,056 INFO com.cloudera.flume.handlers.rolling.RollSink: > Created RollSink: trigger=[TimeTrigger: maxAge=30000 > tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@10cb42cf] > checkPeriodMs = 250 > spec='escapedCustomDfs("hdfs://hadoop1.liaisondevqa.local/user/hdfs/ > ","raw=%{rolltag}" )' > 2012-08-13 11:30:20,058 WARN com.cloudera.flume.agent.LivenessManager: > Heartbeats are backing up, currently behind by 1 heartbeats > 2012-08-13 11:30:25,061 WARN com.cloudera.flume.agent.LivenessManager: > Heartbeats are backing up, currently behind by 2 heartbeats > 2012-08-13 11:30:30,063 WARN com.cloudera.flume.agent.LivenessManager: > Heartbeats are backing up, currently behind by 3 heartbeats > 2012-08-13 11:30:35,065 WARN com.cloudera.flume.agent.LivenessManager: > Heartbeats are backing up, currently behind by 4 heartbeats > 2012-08-13 11:30:40,056 ERROR com.cloudera.flume.agent.LogicalNode: Forcing > driver to exit uncleanly > 2012-08-13 11:30:40,057 ERROR > com.cloudera.flume.core.connector.DirectDriver: Closing down due to > exception during append calls > 2012-08-13 11:30:40,057 INFO > com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode > hadoop1.liaisondevqa.local-23 exited with error: Waiting for queue element > was interrupted! null > java.io.IOException: Waiting for queue element was interrupted! null > at > com.cloudera.flume.handlers.thrift.ThriftEventSource.next(ThriftEventSource.java:222) > at > com.cloudera.flume.collector.CollectorSource.next(CollectorSource.java:72) > at > com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:108) > Caused by: java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1961) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2038) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) > at > com.cloudera.flume.handlers.thrift.ThriftEventSource.next(ThriftEventSource.java:209) > ... 2 more > > > On Mon, Aug 13, 2012 at 12:03 PM, alo alt <[email protected]> wrote: > >> Hi, >> >> Did you follow the userguide, archived here: >> http://archive.cloudera.com/cdh/3/flume-0.9.1+1/UserGuide.html ? >> >> Without error messages or logfiles it's hard to say whats really happen. >> >> - Alex >> >> >> On Aug 13, 2012, at 8:12 AM, Sandeep Reddy P <[email protected]> >> wrote: >> >>> Hi, >>> I'm using flume version Flume 0.9.4-cdh3u4 >>> I'm using flume master webpage to configure the following to move data >> into >>> hdfs. >>> >>> host : syslogTcp(5140) | agentSink("hadoop1.liaisondevqa.local",35862) ; >>> hadoop1.liaisondevqa.local : collectorSource(35862) | >>> collectorSink("hdfs://hadoop1.liaisondevqa.local/user/flume/ ","syslog"); >>> >>> Command history says its successful but i cant see any data in hdfs. >>> >>> Similarly how should i configure to move a log file from linux box to >> hdfs >>> using flume? I'm following apache flume cookbook. >>> -- >>> Thanks, >>> sandeep >>> >>> >>> >>> >>> -- >>> Thanks, >>> sandeep >> >> >> -- >> Alexander Alten-Lorenz >> http://mapredit.blogspot.com >> German Hadoop LinkedIn Group: http://goo.gl/N8pCF >> >> > > > -- > Thanks, > sandeep -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF
