Hi.
I'm running HBase in a small distributed setup (3 machines). When I
import "larger" sets of data (around 700,000 rows within 20 minutes),
HBase eventually breaks and the logs show messages like
2008-12-17 13:56:20,235 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/hbase/log_10.49.21.176_1229549686892_60020/hlog.dat.1229550980090
could only be replicated to 0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1270)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
(...)
and
2008-12-17 13:56:20,235 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/hbase/log_10.49.21.176_1229549686892_60020/hlog.dat.1229550980090
retries left 4
The Hadoop namenode logs show similar things, like
2008-12-17 13:56:20,098 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=hbase,users,dialout,video ip=/10.49.21.176 cmd=create
src=/hbase/log_10.49.21.176_1229549686892_60020/hlog.dat.1229550980090
dst=null perm=hbase:supergroup:rw-r--r--
2008-12-17 13:56:20,230 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place
enough replicas, still in need of 3
2008-12-17 13:56:20,232 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 54310, call
addBlock(/hbase/log_10.49.21.176_1229549686892_60020/hlog.dat.1229550980090,
DFSClient_-1955949534) from 10.49.21.176:39622: error:
java.io.IOException: File
/hbase/log_10.49.21.176_1229549686892_60020/hlog.dat.1229550980090
could only be replicated to 0 nodes, instead of 1
java.io.IOException: File
/hbase/log_10.49.21.176_1229549686892_60020/hlog.dat.1229550980090
could only be replicated to 0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1270)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
(...)
but the other Hbase logs do not have any suspicious entries.
Does anyone know what this could be? I'm still new to Hadoop/HBase so
I don't really have an idea about what could be wrong.
Thanks,
Max
