IllegalStateException: Cannot set a region to be closed it it was not already
marked as closing
-----------------------------------------------------------------------------------------------
Key: HBASE-1098
URL: https://issues.apache.org/jira/browse/HBASE-1098
Project: Hadoop HBase
Issue Type: Bug
Environment: apurtel cluster
Reporter: stack
content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322 is problem region.
below is snippet from master log. Looks to me like things are proceeding in
order.
{code}
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager:
Server XX.XX.XX.51:60020 is overloaded. Server load: 74 avg: 48.0, slop: 0.1
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager:
Choosing to reassign 26 regions. mostLoadedRegions has 10 regions in it.
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager:
Going to close region content,3ea6e20cf48a32fc063912ccae0e960e,1230416307426
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager:
Going to close region content,07730cecbb5b13364cbe17af5778f983,1230448573272
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager:
Going to close region content,e8c750a5983137306e6b3a84eb1fd6e6,1230424193882
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager:
Going to close region content,29be0a07dece4e9bb318c4d253807287,1230356680860
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager:
Going to close region content,23893c82d792ff5c9d3ff99123bc1773,1230390013767
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager:
Going to close region content,e76475012b59500d25000aa7658bc279,1230498762282
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager:
Going to close region result_domain,cn.com.sg.www,1230498785950
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager:
Going to close region content,c7b7e7a475765e16d0bbf1e51c95e85d,1230519854914
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager:
Going to close region content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322
2008-12-29 12:19:32,074 DEBUG org.apache.hadoop.hbase.master.RegionManager:
Going to close region content,e16b2ea9f6da8b32fa96ab3490e3d849,1230424086678
2008-12-29 12:19:32,074 INFO org.apache.hadoop.hbase.master.RegionManager:
Skipped 0 region(s) that are in transition states
2008-12-29 12:19:32,076 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: PendingOpenOperation from XX.XX.XX.51:60020
2008-12-29 12:19:32,076 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
content,42b19221f3a60945595b18017d6bacfa,1230357228381 open on XX.XX.XX.51:60020
2008-12-29 12:19:32,076 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
content,42b19221f3a60945595b18017d6bacfa,1230357228381 in region .META.,,1 with
startcode 1230571104777 and server XX.XX.XX.51:60020
2008-12-29 12:19:32,077 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: PendingOpenOperation from XX.XX.XX.51:60020
2008-12-29 12:19:32,077 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
content,3c619bc7a2c824fb3c4f5d747b0f2f12,1230354673241 open on XX.XX.XX.51:60020
2008-12-29 12:19:32,077 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
content,3c619bc7a2c824fb3c4f5d747b0f2f12,1230354673241 in region .META.,,1 with
startcode 1230571104777 and server XX.XX.XX.51:60020
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,36946541ed9a62f419cf7238d32a6a38,1230448587552, false
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,58221f5f3a79b7dd5e23c339c507a7c6,1230487059978, false
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,94b170cae87992c33c3913d74bb257e4,1230498360162, false
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,bb34d860d76c1eabbe2eb7d2ce8148d9,1230416266955, false
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,8ac5fbdd63b313ce252838a189c73438,1230530000480, false
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,f9f5b2d035349cf4292ed7dd7e9823f0,1230505507313, false
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,4b30e2a8b7949ed62e097824969f8bca,1230448501152, false
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,0c4b7c2825f674510a511233e9e0c539,1230530122456, false
2008-12-29 12:19:32,078 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,43f83c42e08bb0f4b1f6d1125f39f018,1230498357831, false
2008-12-29 12:19:32,299 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region content,43f83c42e08bb0f4b1f6d1125f39f018,1230498357831 to
server XX.XX.XX.34:60020
2008-12-29 12:19:32,300 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region content,bb34d860d76c1eabbe2eb7d2ce8148d9,1230416266955 to
server XX.XX.XX.34:60020
2008-12-29 12:19:32,301 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region content,94b170cae87992c33c3913d74bb257e4,1230498360162 to
server XX.XX.XX.34:60020
2008-12-29 12:19:32,301 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region content,8ac5fbdd63b313ce252838a189c73438,1230530000480 to
server XX.XX.XX.34:60020
2008-12-29 12:19:32,302 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region content,36946541ed9a62f419cf7238d32a6a38,1230448587552 to
server XX.XX.XX.34:60020
2008-12-29 12:19:32,302 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region content,58221f5f3a79b7dd5e23c339c507a7c6,1230487059978 to
server XX.XX.XX.34:60020
2008-12-29 12:19:32,303 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region content,f9f5b2d035349cf4292ed7dd7e9823f0,1230505507313 to
server XX.XX.XX.34:60020
2008-12-29 12:19:32,303 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region content,4b30e2a8b7949ed62e097824969f8bca,1230448501152 to
server XX.XX.XX.34:60020
2008-12-29 12:19:32,304 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region content,0c4b7c2825f674510a511233e9e0c539,1230530122456 to
server XX.XX.XX.34:60020
2008-12-29 12:19:35,080 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_CLOSE:
content,3ea6e20cf48a32fc063912ccae0e960e,1230416307426 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_CLOSE:
content,07730cecbb5b13364cbe17af5778f983,1230448573272 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,3ea6e20cf48a32fc063912ccae0e960e,1230416307426, false
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_CLOSE:
content,e8c750a5983137306e6b3a84eb1fd6e6,1230424193882 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,07730cecbb5b13364cbe17af5778f983,1230448573272, false
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_CLOSE:
content,29be0a07dece4e9bb318c4d253807287,1230356680860 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,e8c750a5983137306e6b3a84eb1fd6e6,1230424193882, false
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_CLOSE:
content,23893c82d792ff5c9d3ff99123bc1773,1230390013767 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,29be0a07dece4e9bb318c4d253807287,1230356680860, false
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_CLOSE:
content,e76475012b59500d25000aa7658bc279,1230498762282 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,23893c82d792ff5c9d3ff99123bc1773,1230390013767, false
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_CLOSE: result_domain,cn.com.sg.www,1230498785950 from
XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,e76475012b59500d25000aa7658bc279,1230498762282, false
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_CLOSE:
content,c7b7e7a475765e16d0bbf1e51c95e85d,1230519854914 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
result_domain,cn.com.sg.www,1230498785950, false
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_CLOSE:
content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,c7b7e7a475765e16d0bbf1e51c95e85d,1230519854914, false
2008-12-29 12:19:35,081 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_CLOSE:
content,e16b2ea9f6da8b32fa96ab3490e3d849,1230424086678 from XX.XX.XX.51:60020
2008-12-29 12:19:35,081 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322, false
2008-12-29 12:19:35,082 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: ProcessRegionClose of
content,e16b2ea9f6da8b32fa96ab3490e3d849,1230424086678, false
2008-12-29 12:19:35,083 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60000, call regionServerReport(address: XX.XX.XX.51:60020,
startcode: 1230571104777, load: (requests=0, regions=64, usedHeap=99,
maxHeap=1777), [Lorg.
apache.hadoop.hbase.HMsg;@3e2dce4e,
[Lorg.apache.hadoop.hbase.HRegionInfo;@314382c6) from XX.XX.XX.51:55371: error:
java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be
closed if it was not already marked as clo
sing. Region: content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322
java.io.IOException: java.lang.IllegalStateException: Cannot set a region to be
closed if it was not already marked as closing. Region:
content,ec0ec4d005ee7da14cbd1182339f8cf4,1230416299322
at
org.apache.hadoop.hbase.master.RegionManager$RegionState.setClosed(RegionManager.java:1108)
at
org.apache.hadoop.hbase.master.RegionManager.setClosed(RegionManager.java:816)
at
org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:377)
at
org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:327)
at
org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:240)
at
org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:570)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
