St.Ack:
Here are some logs snipts: the Master doesn't seem to be effected (no error
in the log files) , the RegiosServer is. both are pasted.
RegionServer: (I added some lines before and after the Error lines , for
reference. The Error is bolded)
2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
writeChunk packet full seqno 1
2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
writeChunk allocating new packet 2
2008-12-16 09:17:14,531 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer
block blk_4611894842462358439_1544 wrote packet seqno:1 size:65557
offsetInBlock:65024 lastPacketInBlock:false
2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
writeChunk packet full seqno 2
2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
writeChunk allocating new packet 3
2008-12-16 09:17:14,532 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer
block blk_4611894842462358439_1544 wrote packet seqno:2 size:65557
offsetInBlock:130048 lastPacketInBlock:false
2008-12-16 09:17:14,533 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer
block blk_4611894842462358439_1544 wrote packet seqno:3 size:49273
offsetInBlock:195072 lastPacketInBlock:true
2008-12-16 09:17:14,534 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
received ack for seqno 0
2008-12-16 09:17:14,535 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
received ack for seqno 1
2008-12-16 09:17:14,536 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
received ack for seqno 2
2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
received ack for seqno 3
2008-12-16 09:17:14,538 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old
block blk_4611894842462358439_1544
2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
writeChunk allocating new packet 0
2008-12-16 09:17:14,557 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
new block
2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
MY.IP.102:50010
2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
MY.IP.211:50010
2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to
MY.IP.102:50010
2008-12-16 09:17:14,558 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size
131071
2008-12-16 09:17:14,561 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer
block blk_-6152926948595840609_1544 wrote packet seqno:0 size:276
offsetInBlock:0 lastPacketInBlock:true
2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
received ack for seqno 0
2008-12-16 09:17:14,564 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old
block blk_-6152926948595840609_1544
2008-12-16 09:17:14,586 DEBUG org.apache.hadoop.dfs.DFSClient:
/hbase/my_table/1608250010/obde_content/info/5474929915463365960:
masked=rwxr-xr-x
2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
writeChunk allocating new packet 0
2008-12-16 09:17:14,609 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
new block
2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
MY.IP.102:50010
2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
MY.IP.213:50010
2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to
MY.IP.102:50010
2008-12-16 09:17:14,610 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size
131071
2008-12-16 09:17:14,613 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer
block blk_2767846628464404081_1545 wrote packet seqno:0 size:38
offsetInBlock:0 lastPacketInBlock:true
2008-12-16 09:17:14,616 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
received ack for seqno 0
2008-12-16 09:17:14,617 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old
block blk_2767846628464404081_1545
2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Added /hbase/my_table/1608250010/obde_content/mapfiles/5474929915463365960
with 56 entries, sequence id 235152, data size 236.3k, file size 238.2k
2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region
my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 in 182ms, sequence
id=235152, compaction requested=true
2008-12-16 09:17:14,644 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 1608250010/obde_content
2008-12-16 09:17:14,644 INFO org.apache.hadoop.hbase.regionserver.HRegion:
closed my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
*2008-12-16 09:17:14,649 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 4 on 60020, call batchUpdate([...@552a2a7d, row =>
f+C7Y24r+apSz+joQUhiQQ==, {column => obde_content:, value => '...'}, -1)
from MY.IP.212:46198: error:
org.apache.hadoop.hbase.NotServingRegionException: Region
my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region
my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at
org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at
org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)*
2008-12-16 09:17:14,651 DEBUG org.apache.hadoop.dfs.DFSClient:
/hbase/my_table/1608250010/splits/1010339458/obde_content/mapfiles/1405033884904780036.1608250010:
masked=rwxr-xr-x
2008-12-16 09:17:14,694 DEBUG org.apache.hadoop.dfs.DFSClient:
/hbase/my_table/1608250010/splits/1010339458/obde_content/info/1405033884904780036.1608250010:
masked=rwxr-xr-x
2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
writeChunk allocating new packet 0
2008-12-16 09:17:14,717 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
new block
2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
MY.IP.102:50010
2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
MY.IP.213:50010
2008-12-16 09:17:14,718 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to
MY.IP.102:50010
2008-12-16 09:17:14,719 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size
131071
2008-12-16 09:17:14,721 DEBUG org.apache.hadoop.dfs.DFSClient: DataStreamer
block blk_-4579777298287321197_1547 wrote packet seqno:0 size:84
offsetInBlock:0 lastPacketInBlock:true
2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
received ack for seqno 0
2008-12-16 09:17:14,724 DEBUG org.apache.hadoop.dfs.DFSClient: Closing old
block blk_-4579777298287321197_1547
*2008-12-16 09:17:14,745 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 9 on 60020, call batchUpdate([...@18084038, row =>
hflk16ESykcggRSkrq7vgQ==, {column => obde_content:, value => '...'}, -1)
from MY.IP.212:46198: error:
org.apache.hadoop.hbase.NotServingRegionException: Region
my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region
my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at
org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at
org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)*
2008-12-16 09:17:14,746 DEBUG org.apache.hadoop.dfs.DFSClient:
/hbase/my_table/1608250010/splits/2015194550/obde_content/mapfiles/215038473253378290.1608250010:
masked=rwxr-xr-x
2008-12-16 09:17:14,790 DEBUG org.apache.hadoop.dfs.DFSClient:
/hbase/my_table/1608250010/splits/2015194550/obde_content/info/215038473253378290.1608250010:
masked=rwxr-xr-x
2008-12-16 09:17:14,813 DEBUG org.apache.hadoop.dfs.DFSClient: DFSClient
writeChunk allocating new packet 0
*2008-12-16 09:17:14,837 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 8 on 60020, call batchUpdate([...@74da3b58, row =>
rG4yO2bs4Rw+JU4QKY7X2w==, {column => obde_content:, value => '...'}, -1)
from MY.IP.212:46198: error:
org.apache.hadoop.hbase.NotServingRegionException: Region
my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region
my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at
org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at
org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)*
2008-12-16 09:17:14,847 DEBUG org.apache.hadoop.dfs.DFSClient: Allocating
new block
2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
MY.IP.102:50010
2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: pipeline =
MY.IP.213:50010
2008-12-16 09:17:14,848 DEBUG org.apache.hadoop.dfs.DFSClient: Connecting to
MY.IP.102:50010
2008-12-16 09:17:14,849 DEBUG org.apache.hadoop.dfs.DFSClient: Send buf size
131071
*2008-12-16 09:17:14,864 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 60020, call batchUpdate([...@51575d48, row =>
aNq6QyMT+FePc7M78PaQMQ==, {column => obde_content:, value => '...'}, -1)
from MY.IP.212:46198: error:
org.apache.hadoop.hbase.NotServingRegionException: Region
my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region
my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at
org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at
org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-12-16 09:17:14,885 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 60020, call batchUpdate([...@63442ff5, row =>
iA4QenOmdZMbB8PTQPSnRw==, {column => obde_content:, value => '...'}, -1)
from MY.IP.212:46198: error:
org.apache.hadoop.hbase.NotServingRegionException: Region
my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
org.apache.hadoop.hbase.NotServingRegionException: Region
my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395 closed
at
org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1810)
at
org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1875)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1406)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1380)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1109)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)*
HBaseMaster: (from about the same time frame)
2008-12-16 09:16:37,827 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner REGION => {NAME =>
'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY =>
'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, TABLE =>
{{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES =>
[{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false',
LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1',
COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE =>
1229433746854
2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of meta region {regionname: .META.,,1,
startKey: <>, server: MY.IP.102:60020} complete
2008-12-16 09:16:37,827 INFO org.apache.hadoop.hbase.master.BaseScanner: all
meta regions scanned
2008-12-16 09:16:46,791 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 4, Num Servers: 2, Avg Load: 2.0
2008-12-16 09:17:01,807 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 4, Num Servers: 2, Avg Load: 2.0
2008-12-16 09:17:16,823 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 4, Num Servers: 2, Avg Load: 2.0
2008-12-16 09:17:16,902 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_SPLIT: my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395:
[...@674c5b37 from MY.IP.102:60020
2008-12-16 09:17:19,826 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 to server
MY.IP.213:60020
2008-12-16 09:17:19,828 INFO org.apache.hadoop.hbase.master.RegionManager:
assigning region my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 to server
MY.IP.213:60020
2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_PROCESS_OPEN:
my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 from MY.IP.213:60020
2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_OPEN: my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460
from MY.IP.213:60020
2008-12-16 09:17:22,833 INFO org.apache.hadoop.hbase.master.ServerManager:
Received MSG_REPORT_OPEN: my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460
from MY.IP.213:60020
2008-12-16 09:17:22,833 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: PendingOpenOperation from MY.IP.213:60020
2008-12-16 09:17:22,833 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 open on MY.IP.213:60020
2008-12-16 09:17:22,833 DEBUG
org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions:
1, onlineMetaRegions.size(): 1
2008-12-16 09:17:22,833 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 in region .META.,,1 with
startcode 1229433746880 and server MY.IP.213:60020
2008-12-16 09:17:22,835 DEBUG org.apache.hadoop.hbase.master.HMaster: Main
processing loop: PendingOpenOperation from MY.IP.213:60020
2008-12-16 09:17:22,835 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 open on MY.IP.213:60020
2008-12-16 09:17:22,835 DEBUG
org.apache.hadoop.hbase.master.RegionServerOperation: numberOfMetaRegions:
1, onlineMetaRegions.size(): 1
2008-12-16 09:17:22,835 INFO
org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460 in region .META.,,1 with
startcode 1229433746880 and server MY.IP.213:60020
2008-12-16 09:17:31,845 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Total Load: 5, Num Servers: 2, Avg Load: 3.0
2008-12-16 09:17:31,900 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
startKey: <>, server: MY.IP.213:60020}
2008-12-16 09:17:31,932 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '',
ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT =>
'false', IS_META => 'true', FAMILIES => [{NAME => 'historian', BLOOMFILTER
=> 'false', IN_MEMORY => 'false', VERSIONS => '2147483647', BLOCKCACHE =>
'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION => 'NONE'}, {NAME
=> 'info', BLOOMFILTER => 'false', IN_MEMORY => 'false', VERSIONS => '1',
BLOCKCACHE => 'false', LENGTH => '2147483647', TTL => '-1', COMPRESSION =>
'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:17:31,934 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0,
startKey: <>, server: MY.IP.213:60020} complete
2008-12-16 09:17:37,816 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>, server: MY.IP.102:60020}
2008-12-16 09:17:37,838 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner REGION => {NAME => 'my_table,,1229374459395',
STARTKEY => '', ENDKEY => 'RQue7uxNoe59vJxljcd1rQ==', ENCODED => 1847652518,
TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false',
FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY =>
'false', LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL
=> '-1', COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.102:60020', STARTCODE
=> 1229433746854
2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner REGION => {NAME =>
'my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395', STARTKEY =>
'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => '', ENCODED => 1608250010, OFFLINE =>
true, SPLIT => true, TABLE => {{NAME => 'my_table', IS_ROOT => 'false',
IS_META => 'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER =>
'false', IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE =>
'false', VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER =>
'MY.IP.102:60020', STARTCODE => 1229433746854
2008-12-16 09:17:37,839 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner REGION => {NAME =>
'my_table,RQue7uxNoe59vJxljcd1rQ==,1229437034460', STARTKEY =>
'RQue7uxNoe59vJxljcd1rQ==', ENDKEY => 'iGnmMuO6Z0jyDJoteHYqKA==', ENCODED =>
1010339458, TABLE => {{NAME => 'my_table', IS_ROOT => 'false', IS_META =>
'false', FAMILIES => [{NAME => 'obde_content', BLOOMFILTER => 'false',
IN_MEMORY => 'false', LENGTH => '2147483647', BLOCKCACHE => 'false',
VERSIONS => '1', TTL => '-1', COMPRESSION => 'NONE'}]}}}, SERVER =>
'MY.IP.213:60020', STARTCODE => 1229433746880
2008-12-16 09:17:37,840 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner REGION => {NAME =>
'my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460', STARTKEY =>
'iGnmMuO6Z0jyDJoteHYqKA==', ENDKEY => '', ENCODED => 2015194550, TABLE =>
{{NAME => 'my_table', IS_ROOT => 'false', IS_META => 'false', FAMILIES =>
[{NAME => 'obde_content', BLOOMFILTER => 'false', IN_MEMORY => 'false',
LENGTH => '2147483647', BLOCKCACHE => 'false', VERSIONS => '1', TTL => '-1',
COMPRESSION => 'NONE'}]}}}, SERVER => 'MY.IP.213:60020', STARTCODE =>
1229433746880
2008-12-16 09:17:37,856 DEBUG org.apache.hadoop.hbase.master.BaseScanner:
my_table,iGnmMuO6Z0jyDJoteHYqKA==,1229437034460 no longer has references to
my_table,RQue7uxNoe59vJxljcd1rQ==,1229374459395
2008-12-16 09:17:37,857 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of meta region {regionname: .META.,,1,
startKey: <>, server: MY.IP.102:60020} complete
Vale et me ama
Yossi
On Thu, Dec 18, 2008 at 10:53 PM, stack <[email protected]> wrote:
> Yossi:
>
> Something is broken. On split, new regions should be deployed. If that is
> not happening, you'll see the below NSRE. Can you look in master logs and
> try to figure whats happening such that the split is not being properly
> processed? Make sure you have DEBUG enabled. Paste snippets into the list
> here and we'll try and help.
>
> St.Ack
>
>
> Yossi Ittach wrote:
>
>> Hey
>>
>> I can't seem to resolve this problem - After a region split occurs I can't
>> insert more data to table unless I disable and re-enable it - I'm getting
>> get NotServingRegionException.
>>
>> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
>> contact
>> region server MY.IP.213:60020 for region
>> my_table,C74fV1ZrJT5JEeZwSniLSA==,1229510672251, row
>> 'EArVqQK7uAhJtdCI6SHIkg==', but failed after 1 attempts.
>> Exceptions:
>> org.apache.hadoop.hbase.NotServingRegionException:
>> org.apache.hadoop.hbase.NotServingRegionException:
>> my_table,C74fV1ZrJT5JEeZwSniLSA==,1229510672251
>> at
>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1519)
>> at
>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1105)
>> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
>> at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>>
>>
>> at
>>
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
>> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
>> at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
>>
>>
>>
>> Vale et me ama
>> Yossi
>>
>>
>> On Wed, Dec 17, 2008 at 1:53 AM, Yossi Ittach <[email protected]> wrote:
>>
>>
>>
>>> I'm using HBase 18.0
>>> I'm afraid that my table was affected - all the attempts to perform
>>> batchUpdates failed until I disabled and enabled the table
>>>
>>> Vale et me ama
>>> Yossi
>>>
>>>
>>> On Tue, Dec 16, 2008 at 6:36 PM, Jean-Daniel Cryans <[email protected]
>>> >wrote:
>>>
>>>
>>>
>>>> Yossi,
>>>>
>>>> Unfortunately this list does not accept attached files.
>>>>
>>>> Also the NotServingRegionException is normal, it just tells the client
>>>> library to refresh it's cache because that region either moved or was
>>>> split.
>>>> Your table should not be hosed by it, was it the case or you simply
>>>> stopped
>>>> when seeing this error?
>>>>
>>>> Also, which version of HBase?
>>>>
>>>> Thx,
>>>>
>>>> J-D
>>>>
>>>> On Tue, Dec 16, 2008 at 11:33 AM, Yossi Ittach <[email protected]>
>>>> wrote:
>>>>
>>>>
>>>>
>>>>> Hey All
>>>>>
>>>>> I'm running an HBase benchmark , and after ~400,000 files , I started
>>>>> to
>>>>> get NotServingRegionException.
>>>>> A quick view in the logs showed that it started write after a
>>>>>
>>>>>
>>>> RegionSplit
>>>>
>>>>
>>>>> was conducted - apparently , something is not connected right.
>>>>> The problem was resolved by disabling and re-enabling the table. Is
>>>>> this
>>>>>
>>>>>
>>>> a
>>>>
>>>>
>>>>> bug / misconfiguration ?
>>>>> Thanks!
>>>>>
>>>>> The (partial) log files are attached
>>>>>
>>>>>
>>>>> Vale et me ama
>>>>> Yossi
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>>
>
>
