[hbase] Region split twice, once after region had closed
--------------------------------------------------------
Key: HADOOP-2728
URL: https://issues.apache.org/jira/browse/HADOOP-2728
Project: Hadoop Core
Issue Type: Bug
Components: contrib/hbase
Reporter: stack
Priority: Critical
Here are the log lines for the region. Was undergoing heavy load at the time.
{code}
...
2008-01-28 17:37:22,070 INFO org.apache.hadoop.hbase.HRegion: Unblocking
updates for region test_table,000000000844535,1201569706879 'IPC Server handler
7 on 60020'
2008-01-28 17:37:22,116 INFO org.apache.hadoop.hbase.HRegion: Unblocking
updates for region test_table,000000000844535,1201569706879 'IPC Server handler
8 on 60020'
2008-01-28 17:37:22,117 INFO org.apache.hadoop.hbase.HRegion: Unblocking
updates for region test_table,000000000844535,1201569706879 'IPC Server handler
0 on 60020'
2008-01-28 17:37:56,713 INFO org.apache.hadoop.hbase.HRegion: compaction
completed on region test_table,000000000844535,1201569706879. Took 9mins, 34sec
2008-01-28 17:37:56,713 INFO org.apache.hadoop.hbase.HRegion: starting
compaction on region test_table,000000000697863,1201569706879
2008-01-28 17:37:56,905 INFO org.apache.hadoop.hbase.HRegion: Splitting
test_table,000000000844535,1201569706879 because largest aggregate size is 1.3g
and desired size is 256.0m
2008-01-28 17:37:56,960 DEBUG org.apache.hadoop.hbase.HRegion: waiting for
cache flush to complete for region test_table,000000000844535,1201569706879
2008-01-28 17:38:26,498 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for region test_table,000000000844535,1201569706879 in 68738ms,
sequenceid=559406
2008-01-28 17:38:26,498 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region test_table,000000000697863,1201569706879. Size 128.0m
2008-01-28 17:38:26,499 DEBUG org.apache.hadoop.hbase.HRegion: compactions and
cache flushes disabled for region test_table,000000000844535,1201569706879
2008-01-28 17:38:26,500 DEBUG org.apache.hadoop.hbase.HRegion: new updates and
scanners for region test_table,000000000844535,1201569706879 disabled
2008-01-28 17:38:26,500 DEBUG org.apache.hadoop.hbase.HRegion: no more active
scanners for region test_table,000000000844535,1201569706879
2008-01-28 17:38:26,500 DEBUG org.apache.hadoop.hbase.HRegion: no more row
locks outstanding on region test_table,000000000844535,1201569706879
2008-01-28 17:38:26,647 DEBUG org.apache.hadoop.hbase.HRegionServer:
test_table,000000000844535,1201569706879 closing (Adding to retiringRegions)
2008-01-28 17:38:26,647 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region test_table,000000000844535,1201569706879. Size 77.7m
2008-01-28 17:38:36,316 INFO org.apache.hadoop.hbase.HRegion: Unblocking
updates for region test_table,000000000697863,1201569706879 'IPC Server handler
9 on 60020'
2008-01-28 17:38:36,317 INFO org.apache.hadoop.hbase.HRegion: Unblocking
updates for region test_table,000000000697863,1201569706879 'IPC Server handler
2 on 60020'
2008-01-28 17:38:36,321 INFO org.apache.hadoop.hbase.HRegion: Unblocking
updates for region test_table,000000000697863,1201569706879 'IPC Server handler
4 on 60020'
2008-01-28 17:39:21,023 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for region test_table,000000000697863,1201569706879 in 54525ms,
sequenceid=572619
2008-01-28 17:39:21,077 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache
flush for region test_table,000000000697863,1201569706879. Size 12.4m
2008-01-28 17:39:51,129 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for region test_table,000000000844535,1201569706879 in 84482ms,
sequenceid=573888
2008-01-28 17:39:51,130 DEBUG org.apache.hadoop.hbase.HRegionServer:
test_table,000000000844535,1201569706879 closed
2008-01-28 17:39:51,130 INFO org.apache.hadoop.hbase.HRegion: closed
test_table,000000000844535,1201569706879
2008-01-28 17:39:51,130 DEBUG org.apache.hadoop.hbase.HRegionServer:
test_table,000000000844535,1201569706879 closed
2008-01-28 17:39:55,664 DEBUG org.apache.hadoop.hbase.HRegion: Finished
memcache flush for region test_table,000000000697863,1201569706879 in 34587ms,
sequenceid=5738892008-01-28 17:40:06,756 INFO
org.apache.hadoop.hbase.HRegionServer: region split, META updated, and report
to master all successful. Old region=test_table,000000000844535,1201569706879,
new regions: test_table,000000000844535,1201570676957,
test_table,000000000920444,1201570676959. Split took 1mins, 40sec
2008-01-28 17:44:16,033 INFO org.apache.hadoop.hbase.HRegion: compaction
completed on region test_table,000000000697863,1201569706879. Took 6mins, 19sec
2008-01-28 17:44:16,262 DEBUG org.apache.hadoop.hbase.HRegion: region
test_table,000000000844535,1201569706879 does not need compaction
2008-01-28 17:44:16,262 INFO org.apache.hadoop.hbase.HRegion: starting
compaction on region test_table,000000000697863,1201569706879
2008-01-28 17:44:16,285 INFO org.apache.hadoop.hbase.HRegion: Splitting
test_table,000000000697863,1201569706879 because largest aggregate size is 1.4g
and desired size is 256.0m
2008-01-28 17:44:16,288 DEBUG org.apache.hadoop.hbase.HRegion: waiting for
compaction to complete for region test_table,000000000697863,1201569706879
2008-01-28 17:49:33,196 INFO org.apache.hadoop.hbase.HRegion: compaction
completed on region test_table,000000000697863,1201569706879. Took 5mins, 16sec
2008-01-28 17:49:33,196 DEBUG org.apache.hadoop.hbase.HRegion: compactions and
cache flushes disabled for region test_table,000000000697863,1201569706879
2008-01-28 17:49:33,196 DEBUG org.apache.hadoop.hbase.HRegion: region
test_table,000000000697863,1201569706879 does not need compaction
2008-01-28 17:49:33,196 DEBUG org.apache.hadoop.hbase.HRegion: new updates and
scanners for region test_table,000000000697863,1201569706879 disabled
2008-01-28 17:49:33,196 DEBUG org.apache.hadoop.hbase.HRegion: no more active
scanners for region test_table,000000000697863,1201569706879
2008-01-28 17:49:33,196 DEBUG org.apache.hadoop.hbase.HRegion: region
test_table,000000000697863,1201569706879 does not need compaction
2008-01-28 17:49:33,196 DEBUG org.apache.hadoop.hbase.HRegion: no more row
locks outstanding on region test_table,000000000697863,1201569706879
2008-01-28 17:49:33,196 DEBUG org.apache.hadoop.hbase.HRegionServer:
test_table,000000000697863,1201569706879 closing (Adding to retiringRegions)
2008-01-28 17:49:33,196 DEBUG org.apache.hadoop.hbase.HRegion: region
test_table,000000000844535,1201569706879 does not need compaction
2008-01-28 17:49:33,197 DEBUG org.apache.hadoop.hbase.HRegion: region
test_table,000000000844535,1201569706879 does not need compaction
2008-01-28 17:49:33,197 DEBUG org.apache.hadoop.hbase.HRegion: region
test_table,000000000844535,1201569706879 does not need compaction
2008-01-28 17:49:33,197 DEBUG org.apache.hadoop.hbase.HRegion: region
test_table,000000000697863,1201569706879 does not need compaction
2008-01-28 17:49:33,197 DEBUG org.apache.hadoop.hbase.HRegionServer:
test_table,000000000697863,1201569706879 closed
2008-01-28 17:49:33,197 INFO org.apache.hadoop.hbase.HRegion: closed
test_table,000000000697863,1201569706879
2008-01-28 17:49:33,197 DEBUG org.apache.hadoop.hbase.HRegion: region
test_table,000000000697863,1201569706879 does not need compaction
2008-01-28 17:49:33,197 DEBUG org.apache.hadoop.hbase.HRegionServer:
test_table,000000000697863,1201569706879 closed
2008-01-28 17:49:33,198 DEBUG org.apache.hadoop.hbase.HRegion: region
test_table,000000000844535,1201569706879 does not need compaction
2008-01-28 17:49:33,198 DEBUG org.apache.hadoop.hbase.HRegion: region
test_table,000000000697863,1201569706879 does not need compaction
2008-01-28 17:49:33,198 DEBUG org.apache.hadoop.hbase.HRegion: region
test_table,000000000697863,1201569706879 does not need compaction
2008-01-28 17:49:34,773 INFO org.apache.hadoop.hbase.HRegionServer: region
split, META updated, and report to master all successful. Old
region=test_table,000000000697863,1201569706879, new regions:
test_table,000000000697863,1201571056287,
test_table,000000000771750,1201571056288. Split took 1sec
2008-01-28 19:09:32,812 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6
on 60020, call batchUpdate(test_table,000000000844535,1201569706879,
9223372036854775807, [EMAIL PROTECTED]) from XX.XX.XX.58:59891: error:
org.apache.hadoop.hbase.NotServingRegionException:
test_table,000000000844535,1201569706879
org.apache.hadoop.hbase.NotServingRegionException:
test_table,000000000844535,1201569706879
2008-01-28 19:09:32,813 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8
on 60020, call batchUpdate(test_table,000000000844535,1201569706879,
9223372036854775807, [EMAIL PROTECTED]) from XX.XX.XX.58:59891: error:
org.apache.hadoop.hbase.NotServingRegionException:
test_table,000000000844535,1201569706879
org.apache.hadoop.hbase.NotServingRegionException:
test_table,000000000844535,1201569706879
2008-01-28 19:09:39,574 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0
on 60020, call batchUpdate(test_table,000000000844535,1201569706879,
9223372036854775807, [EMAIL PROTECTED]) from XX.XX.XX.32:50384: error:
org.apache.hadoop.hbase.NotServingRegionException:
test_table,000000000844535,1201569706879
org.apache.hadoop.hbase.NotServingRegionException:
test_table,000000000844535,1201569706879
2008-01-28 19:09:39,583 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9
on 60020, call batchUpdate(test_table,000000000844535,1201569706879,
9223372036854775807, [EMAIL PROTECTED]) from XX.XX.XX.32:50384: error:
org.apache.hadoop.hbase.NotServingRegionException:
test_table,000000000844535,1201569706879
org.apache.hadoop.hbase.NotServingRegionException:
test_table,000000000844535,1201569706879
...
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
