Hi every one,
this is an example of what I am getting :
2013-08-28 16:41:09,520 INFO org.apache.gora.mapreduce.GoraRecordWriter:
Flushing the datastore after 10000 records
2013-08-28 16:41:19,556 WARN
me.prettyprint.cassandra.connection.HConnectionManager: Could not
fullfill request on this host CassandraClient<crawler1:9160-2>
2013-08-28 16:41:19,557 WARN
me.prettyprint.cassandra.connection.HConnectionManager: Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
at
me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
at
me.prettyprint.cassandra.model.MutatorImpl.insert(MutatorImpl.java:69)
at
org.apache.gora.cassandra.store.HectorUtils.insertColumn(HectorUtils.java:52)
at
org.apache.gora.cassandra.store.CassandraClient.addColumn(CassandraClient.java:197)
at
org.apache.gora.cassandra.store.CassandraStore.addOrUpdateField(CassandraStore.java:353)
at
org.apache.gora.cassandra.store.CassandraStore.flush(CassandraStore.java:228)
at
org.apache.gora.mapreduce.GoraRecordWriter.write(GoraRecordWriter.java:70)
at
org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:586)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
org.apache.nutch.crawl.DbUpdateReducer.reduce(DbUpdateReducer.java:202)
at
org.apache.nutch.crawl.DbUpdateReducer.reduce(DbUpdateReducer.java:39)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: TimedOutException()
at
org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
at
org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
at
me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
at
me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
at
me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
... 20 more
this is happening in virtually all of my reduce tasks, and I think as a
result of this, I get this for about 15% of my reduce tasks:
Task attempt_201308151359_0006_r_000014_0 failed to report status for
1800 seconds. Killing!
is there anything that I can do to improve this situation? maybe
increasing the timeout time some where?
thanks,
--
Kaveh Minooie
