Hi
we used to run few (2-5) map reduce jobs which imports data (csv files on
hdfs) into one large hbase table (one family only). After few successful
runs we managed to import about 5-10 million records on each run. After few
last runs, we encounter strange behavior of hbase.
First, every task is killed because did not report status in 600 secs by
jobtracker, and we were unable to insert single row into table but there was
no error in any log (jobtracker logs, hdfs logs, hbase logs, ...), also what
is strange that every node (we are having 7 node cluster where each node
have 16MB of memory and 4 cores) did not report any extensive usage of cpu,
memory, HD or net (there was some activity, ~1% cpu, every java instance has
at least as 60% free heap, net was really low, i/o also, ...).
We noticed that job is waiting on table.put, but there was no errors in any
log.
After some investigation i noticed that this table is in disabled state (how
table can go in disable state?)
So, i create another table and start import, everything looks fine.
I also try to enable table from hbase shell, and i got error
"Unable to enable table ..."
Now in master log i'm getting following exception
2010-03-05 18:20:21,056 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 74 on 60000, call regionServerReport(address: 172.17.11.14:60020,
startcode: 1267806532298, load: (requests=0, regions=601, usedHeap=422,
maxHeap=3491), [Lorg.apache.hadoop.hbase.HMsg;@3a0975f2,
[Lorg.apache.hadoop.hbase.HRegionInfo;@431e80a2) from 172.17.11.14:34345:
error: java.io.IOException: java.lang.IllegalStateException: Cannot set a
region to be closed if it was not already marked as pending close, pending
open or closing. State:
name=profiles,c2f6e979-6d8a-4c7c-b717-b202c1a53c7f,1267777579663,
unassigned=true, pendingOpen=false, open=false, closing=false,
pendingClose=false, closed=false, offlined=false
java.io.IOException: java.lang.IllegalStateException: Cannot set a region to
be closed if it was not already marked as pending close, pending open or
closing. State:
name=profiles,c2f6e979-6d8a-4c7c-b717-b202c1a53c7f,1267777579663,
unassigned=true, pendingOpen=false, open=false, closing=false,
pendingClose=false, closed=false, offlined=false
at
org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1650)
at
org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:1093)
at
org.apache.hadoop.hbase.master.ServerManager.processRegionClose(ServerManager.java:659)
at
org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:454)
at
org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:414)
at
org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:323)
at
org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:724)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
We are using hadoop 0.20.1 and hbase 0.20.3
i found similar problem in this ticket
http://issues.apache.org/jira/browse/HBASE-2064
Any ideas?
Thanks
Zaharije Pasalic
--
View this message in context:
http://old.nabble.com/Unable-to-disable-table-...-tp27796959p27796959.html
Sent from the HBase User mailing list archive at Nabble.com.
