Hi!
Just managed to gather the log files from one error run.
The error message changes between 3 different ones,
this time it was
Exception in thread "main" org.apache.hadoop.hbase.TableNotFoundException:
Table 'y' does not exist.
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:418)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:350)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:318)
at org.apache.hadoop.hbase.HTable.getRegionLocation(HTable.java:114)
at
org.apache.hadoop.hbase.HTable$ServerCallable.instantiateServer(HTable.java:1009)
at
org.apache.hadoop.hbase.HTable.getRegionServerWithRetries(HTable.java:1024)
at org.apache.hadoop.hbase.HTable.commit(HTable.java:763)
at org.apache.hadoop.hbase.HTable.commit(HTable.java:744)
at hbaseTest.PutValue(hbaseTest.java:297)
at hbaseTest.TesterPut(hbaseTest.java:108)
at hbaseTest.main(hbaseTest.java:35)
This happens exactly at the 10000 insert
Not really sure what is happening with the master.log, the only thing I get
now is the .out file
Regards Erik
2008-04-28 18:18:09,119 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=60020, port=60020
2008-04-28 18:18:09,373 INFO org.apache.hadoop.hbase.HLog: New log writer created at /hbase/log_192.168.249.101_1209431889164_60020/hlog.dat.000
2008-04-28 18:18:09,420 INFO org.mortbay.util.Credential: Checking Resource aliases
2008-04-28 18:18:09,449 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4
2008-04-28 18:18:09,449 INFO org.mortbay.util.Container: Started HttpContext[/static,/static]
2008-04-28 18:18:09,748 INFO org.mortbay.util.Container: Started [EMAIL PROTECTED]
2008-04-28 18:18:09,774 INFO org.mortbay.util.Container: Started WebApplicationContext[/,/]
2008-04-28 18:18:09,775 INFO org.mortbay.http.SocketListener: Started SocketListener on 0.0.0.0:60030
2008-04-28 18:18:09,775 INFO org.mortbay.util.Container: Started [EMAIL PROTECTED]
2008-04-28 18:18:09,775 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2008-04-28 18:18:09,775 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 60020: starting
2008-04-28 18:18:09,776 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020: starting
2008-04-28 18:18:09,776 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020: starting
2008-04-28 18:18:09,777 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020: starting
2008-04-28 18:18:09,777 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020: starting
2008-04-28 18:18:09,777 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020: starting
2008-04-28 18:18:09,777 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020: starting
2008-04-28 18:18:09,777 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020: starting
2008-04-28 18:18:09,777 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020: starting
2008-04-28 18:18:09,777 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020: starting
2008-04-28 18:18:09,777 INFO org.apache.hadoop.hbase.HRegionServer: HRegionServer started at: 192.168.249.101:60020
2008-04-28 18:18:09,777 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020: starting
2008-04-28 18:18:39,795 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : y,,1209431940536
2008-04-28 18:18:39,825 INFO org.apache.hadoop.hbase.HRegion: region y,,1209431940536/1302908521 available
2008-04-28 18:18:39,826 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,,1209431940536
2008-04-28 18:18:39,828 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,,1209431940536 in 0sec
2008-04-28 18:18:54,137 INFO org.apache.hadoop.hbase.HRegion: Blocking updates for 'IPC Server handler 4 on 60020': Memcache size 64.0m is >= than blocking 64.0m size
2008-04-28 18:18:54,167 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2008-04-28 18:18:54,168 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2008-04-28 18:18:55,876 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,,1209431940536
2008-04-28 18:18:55,876 INFO org.apache.hadoop.hbase.HRegion: Unblocking updates for region y,,1209431940536 'IPC Server handler 4 on 60020'
2008-04-28 18:18:55,877 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,,1209431940536 in 0sec
2008-04-28 18:19:00,363 INFO org.apache.hadoop.hbase.HRegion: Blocking updates for 'IPC Server handler 6 on 60020': Memcache size 64.0m is >= than blocking 64.0m size
2008-04-28 18:19:01,780 INFO org.apache.hadoop.hbase.HRegion: Unblocking updates for region y,,1209431940536 'IPC Server handler 6 on 60020'
2008-04-28 18:19:01,781 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,,1209431940536
2008-04-28 18:19:06,498 INFO org.apache.hadoop.hbase.HRegion: Blocking updates for 'IPC Server handler 8 on 60020': Memcache size 64.0m is >= than blocking 64.0m size
2008-04-28 18:19:10,740 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,,1209431940536 in 8sec
2008-04-28 18:19:11,084 INFO org.apache.hadoop.hbase.HRegion: Unblocking updates for region y,,1209431940536 'IPC Server handler 8 on 60020'
2008-04-28 18:19:11,084 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,,1209431940536
2008-04-28 18:19:16,078 INFO org.apache.hadoop.hbase.HRegion: Blocking updates for 'IPC Server handler 0 on 60020': Memcache size 64.0m is >= than blocking 64.0m size
2008-04-28 18:19:21,602 INFO org.apache.hadoop.hbase.HRegion: Unblocking updates for region y,,1209431940536 'IPC Server handler 0 on 60020'
2008-04-28 18:19:23,266 INFO org.apache.hadoop.hbase.HRegionServer: Rolling hlog. Number of entries: 30001
2008-04-28 18:19:23,276 INFO org.apache.hadoop.hbase.HLog: New log writer created at /hbase/log_192.168.249.101_1209431889164_60020/hlog.dat.001
2008-04-28 18:19:23,553 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,,1209431940536 in 12sec
2008-04-28 18:19:23,567 INFO org.apache.hadoop.hbase.HRegion: Splitting y,,1209431940536 because largest aggregate size is 256.9m and desired size is 256.0m
2008-04-28 18:19:23,948 INFO org.apache.hadoop.hbase.HRegion: closed y,,1209431940536
2008-04-28 18:19:23,950 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020, call batchUpdate(y,,1209431940536, 9223372036854775807, [EMAIL PROTECTED]) from 192.168.249.100:39028: error: org.apache.hadoop.hbase.NotServingRegionException: Region y,,1209431940536 closed
org.apache.hadoop.hbase.NotServingRegionException: Region y,,1209431940536 closed
at org.apache.hadoop.hbase.HRegion.obtainRowLock(HRegion.java:1690)
at org.apache.hadoop.hbase.HRegion.batchUpdate(HRegion.java:1372)
at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1544)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:901)
2008-04-28 18:19:24,225 INFO org.apache.hadoop.hbase.HRegion: region y,,1209431963571/1772946574 available
2008-04-28 18:19:24,225 INFO org.apache.hadoop.hbase.HRegion: closed y,,1209431963571
2008-04-28 18:19:24,266 INFO org.apache.hadoop.hbase.HRegion: region y,Row1967,1209431963571/549926246 available
2008-04-28 18:19:24,267 INFO org.apache.hadoop.hbase.HRegion: closed y,Row1967,1209431963571
2008-04-28 18:19:24,331 INFO org.apache.hadoop.hbase.HRegionServer: Updating .META. with region split info
2008-04-28 18:19:24,336 INFO org.apache.hadoop.hbase.HRegionServer: region split, META updated, and report to master all successful. Old region=regionname: y,,1209431940536, startKey: <>, endKey: <>, encodedName: 1302908521, offline: true, split: true, tableDesc: {name: y, families: {cF:={name: cF, max versions: 3, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}}}, new regions: y,,1209431963571, y,Row1967,1209431963571. Split took 0sec
2008-04-28 18:19:57,954 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : y,Row1967,1209432141650
2008-04-28 18:19:57,997 INFO org.apache.hadoop.hbase.HRegion: region y,Row1967,1209432141650/1602646343 available
2008-04-28 18:19:57,997 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,Row1967,1209432141650
2008-04-28 18:19:58,000 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : y,Row3164,1209432141650
2008-04-28 18:19:58,094 INFO org.apache.hadoop.hbase.HRegion: region y,Row3164,1209432141650/2034757443 available
2008-04-28 18:20:01,564 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,Row1967,1209432141650 in 3sec
2008-04-28 18:20:01,576 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,Row3164,1209432141650
2008-04-28 18:20:08,370 INFO org.apache.hadoop.hbase.HRegion: Blocking updates for 'IPC Server handler 4 on 60020': Memcache size 64.0m is >= than blocking 64.0m size
2008-04-28 18:20:10,241 INFO org.apache.hadoop.hbase.HRegion: Unblocking updates for region y,Row3164,1209432141650 'IPC Server handler 4 on 60020'
2008-04-28 18:20:12,217 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,Row3164,1209432141650 in 10sec
2008-04-28 18:20:12,228 INFO org.apache.hadoop.hbase.HRegion: Splitting y,Row3164,1209432141650 because largest aggregate size is 266.0m and desired size is 256.0m
2008-04-28 18:20:12,261 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate(y,Row3164,1209432141650, 9223372036854775807, [EMAIL PROTECTED]) from 192.168.249.100:39036: error: org.apache.hadoop.hbase.NotServingRegionException: y,Row3164,1209432141650
org.apache.hadoop.hbase.NotServingRegionException: y,Row3164,1209432141650
at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1734)
at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1706)
at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1541)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:901)
2008-04-28 18:20:12,963 INFO org.apache.hadoop.hbase.HRegion: closed y,Row3164,1209432141650
2008-04-28 18:20:13,444 INFO org.apache.hadoop.hbase.HRegion: region y,Row3164,1209432012259/462354321 available
2008-04-28 18:20:13,445 INFO org.apache.hadoop.hbase.HRegion: closed y,Row3164,1209432012259
2008-04-28 18:20:13,501 INFO org.apache.hadoop.hbase.HRegion: region y,Row4178,1209432012259/450023943 available
2008-04-28 18:20:13,501 INFO org.apache.hadoop.hbase.HRegion: closed y,Row4178,1209432012259
2008-04-28 18:20:13,502 INFO org.apache.hadoop.hbase.HRegionServer: Updating .META. with region split info
2008-04-28 18:20:13,506 INFO org.apache.hadoop.hbase.HRegionServer: region split, META updated, and report to master all successful. Old region=regionname: y,Row3164,1209432141650, startKey: <Row3164>, endKey: <>, encodedName: 2034757443, offline: true, split: true, tableDesc: {name: y, families: {cF:={name: cF, max versions: 3, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}}}, new regions: y,Row3164,1209432012259, y,Row4178,1209432012259. Split took 1sec
2008-04-28 18:20:15,980 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : y,Row4178,1209432012259
2008-04-28 18:20:16,018 INFO org.apache.hadoop.hbase.HRegion: region y,Row4178,1209432012259/450023943 available
2008-04-28 18:20:16,018 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : y,Row3164,1209432012259
2008-04-28 18:20:16,018 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,Row4178,1209432012259
2008-04-28 18:20:16,067 INFO org.apache.hadoop.hbase.HRegion: region y,Row3164,1209432012259/462354321 available
2008-04-28 18:20:25,914 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,Row4178,1209432012259 in 9sec
2008-04-28 18:20:25,997 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,Row3164,1209432012259
2008-04-28 18:20:26,895 INFO org.apache.hadoop.hbase.HRegion: Blocking updates for 'IPC Server handler 9 on 60020': Memcache size 64.0m is >= than blocking 64.0m size
2008-04-28 18:20:29,546 INFO org.apache.hadoop.hbase.HRegion: Unblocking updates for region y,Row4178,1209432012259 'IPC Server handler 9 on 60020'
2008-04-28 18:20:31,417 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,Row3164,1209432012259 in 5sec
2008-04-28 18:20:31,433 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,Row4178,1209432012259
2008-04-28 18:20:34,040 INFO org.apache.hadoop.hbase.HRegion: Blocking updates for 'IPC Server handler 1 on 60020': Memcache size 64.0m is >= than blocking 64.0m size
2008-04-28 18:20:35,859 INFO org.apache.hadoop.hbase.HRegion: Unblocking updates for region y,Row4178,1209432012259 'IPC Server handler 1 on 60020'
2008-04-28 18:20:38,442 INFO org.apache.hadoop.hbase.HRegionServer: Rolling hlog. Number of entries: 30001
2008-04-28 18:20:39,067 INFO org.apache.hadoop.hbase.HLog: New log writer created at /hbase/log_192.168.249.101_1209431889164_60020/hlog.dat.002
2008-04-28 18:20:39,067 INFO org.apache.hadoop.hbase.HLog: removing old log file /hbase/log_192.168.249.101_1209431889164_60020/hlog.dat.000 whose highest sequence/edit id is 30000
2008-04-28 18:20:42,099 INFO org.apache.hadoop.hbase.HRegion: Blocking updates for 'IPC Server handler 3 on 60020': Memcache size 64.0m is >= than blocking 64.0m size
2008-04-28 18:20:45,730 INFO org.apache.hadoop.hbase.HRegion: Unblocking updates for region y,Row4178,1209432012259 'IPC Server handler 3 on 60020'
2008-04-28 18:20:53,149 INFO org.apache.hadoop.hbase.HRegion: Blocking updates for 'IPC Server handler 5 on 60020': Memcache size 64.0m is >= than blocking 64.0m size
2008-04-28 18:20:56,288 INFO org.apache.hadoop.hbase.HRegion: Unblocking updates for region y,Row4178,1209432012259 'IPC Server handler 5 on 60020'
2008-04-28 18:20:56,448 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,Row4178,1209432012259 in 25sec
2008-04-28 18:20:56,465 INFO org.apache.hadoop.hbase.HRegion: Splitting y,Row4178,1209432012259 because largest aggregate size is 454.2m and desired size is 256.0m
2008-04-28 18:20:56,494 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020, call batchUpdate(y,Row4178,1209432012259, 9223372036854775807, [EMAIL PROTECTED]) from 192.168.249.100:39040: error: org.apache.hadoop.hbase.NotServingRegionException: y,Row4178,1209432012259
org.apache.hadoop.hbase.NotServingRegionException: y,Row4178,1209432012259
at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1734)
at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1706)
at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1541)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:901)
2008-04-28 18:20:56,629 INFO org.apache.hadoop.hbase.HRegion: closed y,Row4178,1209432012259
2008-04-28 18:20:57,439 INFO org.apache.hadoop.hbase.HRegion: region y,Row4178,1209432056492/651110401 available
2008-04-28 18:20:57,440 INFO org.apache.hadoop.hbase.HRegion: closed y,Row4178,1209432056492
2008-04-28 18:20:57,487 INFO org.apache.hadoop.hbase.HRegion: region y,Row5491,1209432056493/1024140885 available
2008-04-28 18:20:57,488 INFO org.apache.hadoop.hbase.HRegion: closed y,Row5491,1209432056493
2008-04-28 18:20:57,489 INFO org.apache.hadoop.hbase.HRegionServer: Updating .META. with region split info
2008-04-28 18:20:57,492 INFO org.apache.hadoop.hbase.HRegionServer: region split, META updated, and report to master all successful. Old region=regionname: y,Row4178,1209432012259, startKey: <Row4178>, endKey: <>, encodedName: 450023943, offline: true, split: true, tableDesc: {name: y, families: {cF:={name: cF, max versions: 3, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}}}, new regions: y,Row4178,1209432056492, y,Row5491,1209432056493. Split took 0sec
2008-04-28 18:21:28,027 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : y,Row7126,1209432219614
2008-04-28 18:21:28,062 INFO org.apache.hadoop.hbase.HRegion: region y,Row7126,1209432219614/137245780 available
2008-04-28 18:21:28,062 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,Row7126,1209432219614
2008-04-28 18:21:28,062 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : y,Row5491,1209432219613
2008-04-28 18:21:28,114 INFO org.apache.hadoop.hbase.HRegion: region y,Row5491,1209432219613/1638129649 available
2008-04-28 18:21:40,070 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,Row7126,1209432219614 in 12sec
2008-04-28 18:21:40,116 INFO org.apache.hadoop.hbase.HRegion: Splitting y,Row7126,1209432219614 because largest aggregate size is 285.0m and desired size is 256.0m
2008-04-28 18:21:40,119 INFO org.apache.hadoop.hbase.HRegion: closed y,Row7126,1209432219614
2008-04-28 18:21:40,222 INFO org.apache.hadoop.hbase.HRegion: region y,Row7126,1209432100118/578964660 available
2008-04-28 18:21:40,223 INFO org.apache.hadoop.hbase.HRegion: closed y,Row7126,1209432100118
2008-04-28 18:21:40,241 INFO org.apache.hadoop.hbase.HRegion: region y,Row8555,1209432100118/163743468 available
2008-04-28 18:21:40,241 INFO org.apache.hadoop.hbase.HRegion: closed y,Row8555,1209432100118
2008-04-28 18:21:40,243 INFO org.apache.hadoop.hbase.HRegionServer: Updating .META. with region split info
2008-04-28 18:21:40,245 INFO org.apache.hadoop.hbase.HRegionServer: region split, META updated, and report to master all successful. Old region=regionname: y,Row7126,1209432219614, startKey: <Row7126>, endKey: <>, encodedName: 137245780, offline: true, split: true, tableDesc: {name: y, families: {cF:={name: cF, max versions: 3, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}}}, new regions: y,Row7126,1209432100118, y,Row8555,1209432100118. Split took 0sec
2008-04-28 18:21:40,245 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,Row5491,1209432219613
2008-04-28 18:21:43,036 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : y,Row7126,1209432100118
2008-04-28 18:21:43,048 INFO org.apache.hadoop.hbase.HRegion: region y,Row7126,1209432100118/578964660 available
2008-04-28 18:21:47,845 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,Row5491,1209432219613 in 7sec
2008-04-28 18:21:47,857 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,Row7126,1209432100118
2008-04-28 18:21:55,595 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,Row7126,1209432100118 in 7sec
2008-04-28 18:20:38,075 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=60020, port=60020
2008-04-28 18:20:38,321 INFO org.apache.hadoop.hbase.HLog: New log writer created at /hbase/log_192.168.249.102_1209432038120_60020/hlog.dat.000
2008-04-28 18:20:38,368 INFO org.mortbay.util.Credential: Checking Resource aliases
2008-04-28 18:20:38,396 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4
2008-04-28 18:20:38,397 INFO org.mortbay.util.Container: Started HttpContext[/static,/static]
2008-04-28 18:20:38,650 INFO org.mortbay.util.Container: Started [EMAIL PROTECTED]
2008-04-28 18:20:38,689 INFO org.mortbay.util.Container: Started WebApplicationContext[/,/]
2008-04-28 18:20:38,691 INFO org.mortbay.http.SocketListener: Started SocketListener on 0.0.0.0:60030
2008-04-28 18:20:38,691 INFO org.mortbay.util.Container: Started [EMAIL PROTECTED]
2008-04-28 18:20:38,691 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2008-04-28 18:20:38,691 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 60020: starting
2008-04-28 18:20:38,692 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020: starting
2008-04-28 18:20:38,693 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020: starting
2008-04-28 18:20:38,693 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020: starting
2008-04-28 18:20:38,693 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020: starting
2008-04-28 18:20:38,693 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020: starting
2008-04-28 18:20:38,693 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020: starting
2008-04-28 18:20:38,693 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020: starting
2008-04-28 18:20:38,693 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020: starting
2008-04-28 18:20:38,693 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020: starting
2008-04-28 18:20:38,694 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020: starting
2008-04-28 18:20:38,694 INFO org.apache.hadoop.hbase.HRegionServer: HRegionServer started at: 192.168.249.102:60020
2008-04-28 18:20:38,700 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : -ROOT-,,0
2008-04-28 18:20:38,783 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2008-04-28 18:20:38,784 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2008-04-28 18:20:38,810 INFO org.apache.hadoop.hbase.HRegion: region -ROOT-,,0/70236052 available
2008-04-28 18:20:38,811 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region -ROOT-,,0
2008-04-28 18:20:39,034 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region -ROOT-,,0 in 0sec
2008-04-28 18:21:56,747 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : y,,1209431963571
2008-04-28 18:21:56,800 INFO org.apache.hadoop.hbase.HRegion: region y,,1209431963571/1772946574 available
2008-04-28 18:21:57,342 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,,1209431963571
2008-04-28 18:21:57,342 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : y,Row1967,1209431963571
2008-04-28 18:21:57,436 INFO org.apache.hadoop.hbase.HRegion: region y,Row1967,1209431963571/549926246 available
2008-04-28 18:22:00,154 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,,1209431963571 in 2sec
2008-04-28 18:22:00,167 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,Row1967,1209431963571
2008-04-28 18:22:06,476 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,Row1967,1209431963571 in 6sec
2008-04-28 18:22:09,362 INFO org.apache.hadoop.hbase.HRegion: Blocking updates for 'IPC Server handler 4 on 60020': Memcache size 64.0m is >= than blocking 64.0m size
2008-04-28 18:22:10,784 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,Row1967,1209431963571
2008-04-28 18:22:10,784 INFO org.apache.hadoop.hbase.HRegion: Unblocking updates for region y,Row1967,1209431963571 'IPC Server handler 4 on 60020'
2008-04-28 18:22:16,069 INFO org.apache.hadoop.hbase.HRegion: Blocking updates for 'IPC Server handler 6 on 60020': Memcache size 64.0m is >= than blocking 64.0m size
2008-04-28 18:22:20,355 INFO org.apache.hadoop.hbase.HRegion: Unblocking updates for region y,Row1967,1209431963571 'IPC Server handler 6 on 60020'
2008-04-28 18:22:21,629 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,Row1967,1209431963571 in 10sec
2008-04-28 18:22:21,647 INFO org.apache.hadoop.hbase.HRegion: Splitting y,Row1967,1209431963571 because largest aggregate size is 304.2m and desired size is 256.0m
2008-04-28 18:22:21,677 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020, call batchUpdate(y,Row1967,1209431963571, 9223372036854775807, [EMAIL PROTECTED]) from 192.168.249.100:44349: error: org.apache.hadoop.hbase.NotServingRegionException: y,Row1967,1209431963571
org.apache.hadoop.hbase.NotServingRegionException: y,Row1967,1209431963571
at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1734)
at org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1706)
at org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1541)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:901)
2008-04-28 18:22:23,975 INFO org.apache.hadoop.hbase.HRegion: closed y,Row1967,1209431963571
2008-04-28 18:22:24,278 INFO org.apache.hadoop.hbase.HRegion: region y,Row1967,1209432141650/1602646343 available
2008-04-28 18:22:24,279 INFO org.apache.hadoop.hbase.HRegion: closed y,Row1967,1209432141650
2008-04-28 18:22:24,314 INFO org.apache.hadoop.hbase.HRegion: region y,Row3164,1209432141650/2034757443 available
2008-04-28 18:22:24,315 INFO org.apache.hadoop.hbase.HRegion: closed y,Row3164,1209432141650
2008-04-28 18:22:24,402 INFO org.apache.hadoop.hbase.HRegionServer: Updating .META. with region split info
2008-04-28 18:22:24,409 INFO org.apache.hadoop.hbase.HRegionServer: region split, META updated, and report to master all successful. Old region=regionname: y,Row1967,1209431963571, startKey: <Row1967>, endKey: <>, encodedName: 549926246, offline: true, split: true, tableDesc: {name: y, families: {cF:={name: cF, max versions: 3, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}}}, new regions: y,Row1967,1209432141650, y,Row3164,1209432141650. Split took 2sec
2008-04-28 18:20:24,392 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=60020, port=60020
2008-04-28 18:20:24,611 INFO org.apache.hadoop.hbase.HLog: New log writer created at /hbase/log_192.168.249.103_1209432024437_60020/hlog.dat.000
2008-04-28 18:20:24,655 INFO org.mortbay.util.Credential: Checking Resource aliases
2008-04-28 18:20:24,684 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4
2008-04-28 18:20:24,684 INFO org.mortbay.util.Container: Started HttpContext[/static,/static]
2008-04-28 18:20:24,935 INFO org.mortbay.util.Container: Started [EMAIL PROTECTED]
2008-04-28 18:20:24,961 INFO org.mortbay.util.Container: Started WebApplicationContext[/,/]
2008-04-28 18:20:24,962 INFO org.mortbay.http.SocketListener: Started SocketListener on 0.0.0.0:60030
2008-04-28 18:20:24,962 INFO org.mortbay.util.Container: Started [EMAIL PROTECTED]
2008-04-28 18:20:24,962 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2008-04-28 18:20:24,962 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 60020: starting
2008-04-28 18:20:24,963 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 60020: starting
2008-04-28 18:20:24,963 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 60020: starting
2008-04-28 18:20:24,963 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 60020: starting
2008-04-28 18:20:24,964 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60020: starting
2008-04-28 18:20:24,964 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 60020: starting
2008-04-28 18:20:24,964 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 60020: starting
2008-04-28 18:20:24,964 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 60020: starting
2008-04-28 18:20:24,964 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 60020: starting
2008-04-28 18:20:24,964 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020: starting
2008-04-28 18:20:24,964 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 60020: starting
2008-04-28 18:20:24,964 INFO org.apache.hadoop.hbase.HRegionServer: HRegionServer started at: 192.168.249.103:60020
2008-04-28 18:20:30,973 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : .META.,,1
2008-04-28 18:20:31,041 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2008-04-28 18:20:31,042 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2008-04-28 18:20:31,047 INFO org.apache.hadoop.hbase.HRegion: region .META.,,1/1028785192 available
2008-04-28 18:20:31,048 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region .META.,,1
2008-04-28 18:20:31,050 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region .META.,,1 in 0sec
2008-04-28 18:23:16,071 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : y,Row4178,1209432056492
2008-04-28 18:23:16,157 INFO org.apache.hadoop.hbase.HRegion: region y,Row4178,1209432056492/651110401 available
2008-04-28 18:23:16,157 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : y,Row5491,1209432056493
2008-04-28 18:23:16,158 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,Row4178,1209432056492
2008-04-28 18:23:16,324 INFO org.apache.hadoop.hbase.HRegion: region y,Row5491,1209432056493/1024140885 available
2008-04-28 18:23:20,568 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,Row4178,1209432056492 in 4sec
2008-04-28 18:23:20,583 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,Row5491,1209432056493
2008-04-28 18:23:28,184 INFO org.apache.hadoop.hbase.HRegion: Blocking updates for 'IPC Server handler 2 on 60020': Memcache size 64.0m is >= than blocking 64.0m size
2008-04-28 18:23:30,109 INFO org.apache.hadoop.hbase.HRegion: Unblocking updates for region y,Row5491,1209432056493 'IPC Server handler 2 on 60020'
2008-04-28 18:23:39,570 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,Row5491,1209432056493 in 18sec
2008-04-28 18:23:39,581 INFO org.apache.hadoop.hbase.HRegion: Splitting y,Row5491,1209432056493 because largest aggregate size is 391.2m and desired size is 256.0m
2008-04-28 18:23:40,832 INFO org.apache.hadoop.hbase.HRegion: closed y,Row5491,1209432056493
2008-04-28 18:23:41,540 INFO org.apache.hadoop.hbase.HRegion: region y,Row5491,1209432219613/1638129649 available
2008-04-28 18:23:41,542 INFO org.apache.hadoop.hbase.HRegion: closed y,Row5491,1209432219613
2008-04-28 18:23:41,578 INFO org.apache.hadoop.hbase.HRegion: region y,Row7126,1209432219614/137245780 available
2008-04-28 18:23:41,579 INFO org.apache.hadoop.hbase.HRegion: closed y,Row7126,1209432219614
2008-04-28 18:23:41,968 INFO org.apache.hadoop.hbase.HRegionServer: Updating .META. with region split info
2008-04-28 18:23:41,972 INFO org.apache.hadoop.hbase.HRegionServer: region split, META updated, and report to master all successful. Old region=regionname: y,Row5491,1209432056493, startKey: <Row5491>, endKey: <>, encodedName: 1024140885, offline: true, split: true, tableDesc: {name: y, families: {cF:={name: cF, max versions: 3, compression: NONE, in memory: false, max length: 2147483647, bloom filter: none}}}, new regions: y,Row5491,1209432219613, y,Row7126,1209432219614. Split took 2sec
2008-04-28 18:24:01,114 INFO org.apache.hadoop.hbase.HRegionServer: MSG_REGION_OPEN : y,Row8555,1209432100118
2008-04-28 18:24:01,128 INFO org.apache.hadoop.hbase.HRegion: region y,Row8555,1209432100118/163743468 available
2008-04-28 18:24:01,475 INFO org.apache.hadoop.hbase.HRegion: starting compaction on region y,Row8555,1209432100118
2008-04-28 18:24:13,494 INFO org.apache.hadoop.hbase.HRegion: compaction completed on region y,Row8555,1209432100118 in 12sec
