I think the key part is this:
java.io.IOException: All datanodes 10.33.100.74:50010 are bad. Aborting... > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2680) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1500(DFSClient.java:2172) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2371) > 2011-10-13 14:35:48,002 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: > requests=0, regions=4, stores=6, storefiles=5, storefileIndexSize=0, > memstoreSize=0, compactionQueueSize=0, flushQueueSize=0, usedHeap=31, > maxHeap=993, blockCacheSize=1011832, blockCacheFree=207366440, > blockCacheCount=9, blockCacheHitCount=10644, blockCacheMissCount=9, > blockCacheEvictedCount=0, blockCacheHitRatio=99, > blockCacheHitCachingRatio=99 > 2011-10-13 14:35:48,002 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: IOE in log > roller > 2011-10-13 14:35:48,002 INFO > org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting. > > This looks like the case that motivated HBASE-4222: https://issues.apache.org/jira/browse/HBASE-4222 Did you by any chance restart the cluster DataNodes while the RegionServers were running? This can easily cause this issue to occur. If not, look in the DataNode logs for other errors which would indicate the underlying problem. In either case, the next release should be a bit more resilient to this problem, as long as it is a transient error and you are not using deferred log flushing on your tables.
