Yea. DFS stays healthy. one of the nodes dies during massive load. I've
made the following changes:
1. upped FDs
2. block size to default. only properties set in hbase-site.xml are
'base.rootdir' and 'hbase.master'.
3. turned debugging on for hbase and hbase dfs.
4. set auth flushing to false
5. set 12MB client buffer
6. half second sleep in data load per row (1 second after 1000 batch
update).
restarted cluster & data load....
Hadoop DFS looks like this: =========================
Cluster Summary
34 files and directories, 8 blocks = 42 total. Heap Size is 28.88 MB / 2.6
GB (1%)
Configured Capacity : 550.01 GB
DFS Used : 158.55 KB
Non DFS Used : 28.49 GB
DFS Remaining : 521.53 GB
DFS Used% : 0 %
DFS Remaining% : 94.82 %
Live Nodes : 3
Dead Nodes : 0
========================================
1 region server went down again: tail of output:
========================================
ulimit -n 1024
2009-05-26 11:47:44,210 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer:
vmInputArguments=[-Xmx3000m, -XX:NewSize=6m, -XX:MaxNewSize=6m,
-XX:+UseConcMarkSweepGC, -verbose:gc, -XX:+PrintGCDetails,
-XX:+PrintGCTimeStamps, -XX:+CMSIncrementalMode,
-Xloggc:/home/hadoop/hbase19/logs/gc-hbase.log,
-XX:+HeapDumpOnOutOfMemoryError,
-Dhbase.log.dir=/home/hadoop/hbase19/bin/../logs,
-Dhbase.log.file=hbase-hadoop-regionserver-server175.apptechsys.com.log,
-Dhbase.home.dir=/home/hadoop/hbase19/bin/.., -Dhbase.id.str=hadoop,
-Dhbase.root.logger=INFO,DRFA,
-Djava.library.path=/home/hadoop/hbase19/bin/../lib/native/Linux-amd64-64]
2009-05-26 11:47:44,414 INFO
org.apache.hadoop.hbase.regionserver.MemcacheFlusher:
globalMemcacheLimit=1.2g, globalMemcacheLimitLowMark=749.9m, maxHeap=2.9g
2009-05-26 11:47:44,425 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 10000000ms
2009-05-26 11:47:44,482 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics:
Initializing RPC Metrics with hostName=HRegionServer, port=60020
2009-05-26 11:47:44,787 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at
server181:60000 that we are up
2009-05-26 11:47:45,919 INFO org.apache.hadoop.ipc.HBaseClass: Retrying
connect to server: server181/192.168.240.181:60000. Already tried 0 time(s).
2009-05-26 11:47:46,923 INFO org.apache.hadoop.ipc.HBaseClass: Retrying
connect to server: server181/192.168.240.181:60000. Already tried 1 time(s).
2009-05-26 11:47:47,819 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: sending initial server
load: requests=0, regions=0, usedHeap=22, maxHeap=-1096
2009-05-26 11:47:47,868 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master:
hbase.regionserver.address=192.168.240.175
2009-05-26 11:47:47,868 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master:
fs.default.name=hdfs://server181:54310/hbase
2009-05-26 11:47:47,868 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master:
hbase.rootdir=hdfs://server181:54310/hbase
2009-05-26 11:47:48,170 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Log dir
hdfs://server181:54310/hbase/log_192.168.240.175_1243363664487_60020
2009-05-26 11:47:48,275 INFO org.apache.hadoop.hbase.regionserver.HLog: New
log writer:
/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243363668204
2009-05-26 11:47:48,282 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=RegionServer,
sessionId=regionserver/0.0.0.0:60020
2009-05-26 11:47:48,284 INFO
org.apache.hadoop.hbase.regionserver.metrics.RegionServerMetrics:
Initialized
2009-05-26 11:47:48,493 INFO org.mortbay.util.Credential: Checking Resource
aliases
2009-05-26 11:47:48,502 INFO org.mortbay.http.HttpServer: Version
Jetty/5.1.4
2009-05-26 11:47:48,503 INFO org.mortbay.util.Container: Started
HttpContext[/logs,/logs]
2009-05-26 11:47:49,324 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.servlet.webapplicationhand...@6ecf829d
2009-05-26 11:47:49,393 INFO org.mortbay.util.Container: Started
WebApplicationContext[/static,/static]
2009-05-26 11:47:49,666 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.servlet.webapplicationhand...@608b8a47
2009-05-26 11:47:49,671 INFO org.mortbay.util.Container: Started
WebApplicationContext[/,/]
2009-05-26 11:47:49,676 INFO org.mortbay.http.SocketListener: Started
SocketListener on 0.0.0.0:60030
2009-05-26 11:47:49,676 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.ser...@49938039
2009-05-26 11:47:49,676 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting to exit safe
mode
2009-05-26 11:47:49,677 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder: starting
2009-05-26 11:47:49,679 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
listener on 60020: starting
2009-05-26 11:47:49,679 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: starting
2009-05-26 11:47:49,679 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: starting
2009-05-26 11:47:49,680 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: starting
2009-05-26 11:47:49,680 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: starting
2009-05-26 11:47:49,680 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: starting
2009-05-26 11:47:49,698 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: starting
2009-05-26 11:47:49,698 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer started
at: 192.168.240.175:60020
2009-05-26 11:47:49,699 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: starting
2009-05-26 11:47:49,699 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: starting
2009-05-26 11:47:49,699 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: starting
2009-05-26 11:47:49,699 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020: starting
2009-05-26 11:49:19,861 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
tableA,,1243363759453: safeMode=false
2009-05-26 11:49:19,864 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
tableA,,1243363759453: safeMode=false
2009-05-26 11:49:19,912 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT
at 192.168.240.179:60020
2009-05-26 11:49:19,918 DEBUG org.apache.hadoop.hbase.RegionHistorian:
Onlined
2009-05-26 11:49:19,930 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Opening region tableA,,1243363759453/1501852784
2009-05-26 11:49:20,025 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Next sequence id for region tableA,,1243363759453 is 0
2009-05-26 11:49:20,031 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region tableA,,1243363759453/1501852784 available
2009-05-26 11:49:20,031 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
requested for region tableA,,1243363759453/1501852784 because: Region open
check
2009-05-26 11:49:28,310 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region tableA,,1243363759453
2009-05-26 11:50:25,968 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
tableA,,1243363759453: safeMode=false
2009-05-26 11:50:25,969 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_CLOSE: tableA,,1243363759453: safeMode=false
2009-05-26 11:50:25,969 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing tableA,,1243363759453: compactions & flushes disabled
2009-05-26 11:50:25,969 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region, no outstanding scanners on
tableA,,1243363759453
2009-05-26 11:50:25,969 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
No more row locks outstanding on region tableA,,1243363759453
2009-05-26 11:50:25,969 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 1501852784/entity
2009-05-26 11:50:25,970 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 1501852784/link
2009-05-26 11:50:25,970 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed tableA,,1243363759453
2009-05-26 11:50:39,882 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: setting compaction limit
to 2
2009-05-26 11:50:50,014 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
tableA,,1243363849039: safeMode=false
2009-05-26 11:50:50,015 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
tableA,,1243363849039: safeMode=false
2009-05-26 11:50:50,016 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Opening region tableA,,1243363849039/407623107
2009-05-26 11:50:50,050 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Next sequence id for region tableA,,1243363849039 is 0
2009-05-26 11:50:50,054 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region tableA,,1243363849039/407623107 available
2009-05-26 11:50:50,054 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
requested for region tableA,,1243363849039/407623107 because: Region open
check
2009-05-26 11:50:50,054 INFO org.apache.hadoop.hbase.regionserver.HRegion:
starting compaction on region tableA,,1243363849039
2009-05-26 11:50:50,056 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
407623107/entity: no store files to compact
2009-05-26 11:50:50,056 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
407623107/link: no store files to compact
2009-05-26 11:50:50,058 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region tableA,,1243363849039 in 0sec
2009-05-26 11:52:39,913 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: setting compaction limit
to 3
2009-05-26 11:55:19,864 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: setting compaction limit
to 4
2009-05-26 11:58:16,316 INFO org.apache.hadoop.hbase.regionserver.HLog:
Closed
hdfs://server181:54310/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.0,
entries=100215. New log writer:
/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364296306
2009-05-26 11:58:16,317 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Found 0 logs to remove out of total 1; oldest outstanding seqnum is 0 from
region tableA,,1243363849039
2009-05-26 11:58:39,916 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: setting compaction limit
to -1
2009-05-26 11:58:59,921 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: compactions no longer
limited
2009-05-26 12:00:55,330 INFO org.apache.hadoop.hbase.regionserver.HLog:
Closed
hdfs://server181:54310/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243363668204,
entries=100788. New log writer:
/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364455324
2009-05-26 12:00:55,330 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Found 0 logs to remove out of total 2; oldest outstanding seqnum is 0 from
region tableA,,1243363849039
2009-05-26 12:00:57,273 INFO org.apache.hadoop.hbase.regionserver.HLog:
Closed
hdfs://server181:54310/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364296306,
entries=101000. New log writer:
/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364457266
2009-05-26 12:00:57,273 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Found 0 logs to remove out of total 3; oldest outstanding seqnum is 0 from
region tableA,,1243363849039
2009-05-26 12:03:33,816 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Flush requested on tableA,,1243363849039
2009-05-26 12:03:33,848 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memcache flush for region tableA,,1243363849039. Current region
memcache size 64.3m
2009-05-26 12:03:34,211 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded
the native-hadoop library
2009-05-26 12:03:34,214 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory:
Successfully loaded & initialized native-zlib library
2009-05-26 12:03:34,217 INFO org.apache.hadoop.io.compress.CodecPool: Got
brand-new compressor
2009-05-26 12:03:37,694 INFO org.apache.hadoop.io.compress.CodecPool: Got
brand-new decompressor
2009-05-26 12:03:37,695 INFO org.apache.hadoop.io.compress.CodecPool: Got
brand-new decompressor
2009-05-26 12:03:37,695 INFO org.apache.hadoop.io.compress.CodecPool: Got
brand-new decompressor
2009-05-26 12:03:37,695 INFO org.apache.hadoop.io.compress.CodecPool: Got
brand-new decompressor
2009-05-26 12:03:37,820 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Added /hbase/tableA/407623107/link/mapfiles/7688740598584924238 with 356003
entries, sequence id 356003, data size ~64.5m, file size 20.2m to
tableA,,1243363849039
2009-05-26 12:03:37,820 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush of ~64.5m for region tableA,,1243363849039 in
3972ms, sequence id=356003, compaction requested=false
2009-05-26 12:03:56,790 INFO org.apache.hadoop.hbase.regionserver.HLog:
Closed
hdfs://server181:54310/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364455324,
entries=154001. New log writer:
/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364636781
2009-05-26 12:03:56,790 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Last sequence written is empty. Deleting all old hlogs
2009-05-26 12:03:56,790 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old log file
/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.0 whose highest
sequence/edit id is 100214
2009-05-26 12:03:56,790 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old log file
/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.0 whose highest
sequence/edit id is 100214
2009-05-26 12:03:56,792 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old log file
/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243363668204 whose
highest sequence/edit id is 201002
2009-05-26 12:03:56,798 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old log file
/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364296306 whose
highest sequence/edit id is 302002
2009-05-26 12:03:56,806 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old log file
/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364455324 whose
highest sequence/edit id is 456003
2009-05-26 12:06:14,313 INFO org.apache.hadoop.hbase.regionserver.HLog:
Closed
hdfs://server181:54310/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364457266,
entries=101000. New log writer:
/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364774264
2009-05-26 12:06:14,314 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Found 0 logs to remove out of total 1; oldest outstanding seqnum is 456004
from region tableA,,1243363849039
2009-05-26 12:08:51,328 INFO org.apache.hadoop.hbase.regionserver.HLog:
Closed
hdfs://server181:54310/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364636781,
entries=101000. New log writer:
/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364931323
2009-05-26 12:08:51,328 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Found 0 logs to remove out of total 2; oldest outstanding seqnum is 456004
from region tableA,,1243363849039
2009-05-26 12:08:52,590 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memcache flush for region tableA,,1243363849039. Current region
memcache size 64.1m
2009-05-26 12:08:52,591 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Flush requested on tableA,,1243363849039
2009-05-26 12:09:04,876 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: java.net.SocketTimeoutException: 10000 millis timeout while
waiting for channel to be ready for write. ch :
java.nio.channels.SocketChannel[connected local=/192.168.240.175:56496
remote=/192.168.240.175:50010]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:162)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2209)
2009-05-26 12:09:04,877 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_-7325527218992385186_1022 bad datanode[0]
192.168.240.175:50010
2009-05-26 12:09:04,877 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_-7325527218992385186_1022 in pipeline
192.168.240.175:50010, 192.168.240.180:50010: bad datanode
192.168.240.175:50010
2009-05-26 12:09:10,084 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: java.net.SocketTimeoutException: 5000 millis timeout while
waiting for channel to be ready for write. ch :
java.nio.channels.SocketChannel[connected local=/192.168.240.175:56113
remote=/192.168.240.180:50010]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:162)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2209)
2009-05-26 12:09:10,085 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_-7325527218992385186_1023 bad datanode[0]
192.168.240.180:50010
2009-05-26 12:09:10,135 FATAL
org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog
required. Forcing server shutdown
org.apache.hadoop.hbase.DroppedSnapshotException: region:
tableA,,1243363849039
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:897)
at
org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:790)
at
org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:228)
at
org.apache.hadoop.hbase.regionserver.MemcacheFlusher.run(MemcacheFlusher.java:138)
Caused by: java.io.IOException: All datanodes 192.168.240.180:50010 are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2444)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1996)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1996)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160)
2009-05-26 12:09:10,138 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=1, stores=2, storefiles=1, storefileIndexSize=0,
memcacheSize=72, usedHeap=114, maxHeap=2999
2009-05-26 12:09:10,138 INFO
org.apache.hadoop.hbase.regionserver.MemcacheFlusher:
regionserver/0.0.0.0:60020.cacheFlusher exiting
2009-05-26 12:09:10,288 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/0.0.0.0:60020.compactor exiting
2009-05-26 12:09:10,512 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
2009-05-26 12:09:11,336 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
LogRoller exiting.
2009-05-26 12:09:12,744 DEBUG org.apache.hadoop.hbase.RegionHistorian:
Offlined
2009-05-26 12:09:12,745 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2009-05-26 12:09:12,746 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: exiting
2009-05-26 12:09:12,746 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server listener on 60020
2009-05-26 12:09:12,746 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: exiting
2009-05-26 12:09:12,746 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: exiting
2009-05-26 12:09:12,746 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: exiting
2009-05-26 12:09:12,748 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2009-05-26 12:09:12,748 INFO org.mortbay.util.ThreadedServer: Stopping
Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=60030]
2009-05-26 12:09:12,749 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: exiting
2009-05-26 12:09:12,749 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server Responder
2009-05-26 12:09:12,750 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: exiting
2009-05-26 12:09:12,749 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: exiting
2009-05-26 12:09:12,756 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: exiting
2009-05-26 12:09:12,756 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020: exiting
2009-05-26 12:09:12,756 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: exiting
2009-05-26 12:09:12,756 INFO org.mortbay.http.SocketListener: Stopped
SocketListener on 0.0.0.0:60030
2009-05-26 12:09:13,443 INFO org.mortbay.util.Container: Stopped
HttpContext[/logs,/logs]
2009-05-26 12:09:13,444 INFO org.mortbay.util.Container: Stopped
org.mortbay.jetty.servlet.webapplicationhand...@6ecf829d
2009-05-26 12:09:13,884 INFO org.mortbay.util.Container: Stopped
WebApplicationContext[/static,/static]
2009-05-26 12:09:13,885 INFO org.mortbay.util.Container: Stopped
org.mortbay.jetty.servlet.webapplicationhand...@608b8a47
2009-05-26 12:09:14,535 INFO org.mortbay.util.Container: Stopped
WebApplicationContext[/,/]
2009-05-26 12:09:14,536 INFO org.mortbay.util.Container: Stopped
org.mortbay.jetty.ser...@49938039
2009-05-26 12:09:14,536 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
closing log writer in
hdfs://server181:54310/hbase/log_192.168.240.175_1243363664487_60020
2009-05-26 12:09:14,537 INFO
org.apache.hadoop.hbase.regionserver.LogFlusher:
regionserver/0.0.0.0:60020.logFlusher exiting
2009-05-26 12:09:14,537 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
regionserver/0.0.0.0:60020.majorCompactionChecker exiting
2009-05-26 12:09:18,856 INFO org.apache.hadoop.hbase.Leases:
regionserver/0.0.0.0:60020.leaseChecker closing leases
2009-05-26 12:09:18,856 INFO org.apache.hadoop.hbase.Leases:
regionserver/0.0.0.0:60020.leaseChecker closed leases
2009-05-26 12:09:26,928 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: On abort, closed hlog
2009-05-26 12:09:26,929 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region
tableA,,1243363849039
2009-05-26 12:09:26,929 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing tableA,,1243363849039: compactions & flushes disabled
2009-05-26 12:09:26,929 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region, no outstanding scanners on
tableA,,1243363849039
2009-05-26 12:09:26,929 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
No more row locks outstanding on region tableA,,1243363849039
2009-05-26 12:09:26,929 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 407623107/entity
2009-05-26 12:09:26,929 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
closed 407623107/link
2009-05-26 12:09:26,929 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed tableA,,1243363849039
2009-05-26 12:09:26,929 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
192.168.240.175:60020
2009-05-26 12:09:27,033 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer:
regionserver/0.0.0.0:60020 exiting
2009-05-26 12:09:27,034 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
thread.
2009-05-26 12:09:27,034 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete
========================================
stack-3 wrote:
>
> That looks sick. Different log files can't close? Enable DEBUG in
> your logs. See FAQ for how. You sure your HDFS healthy? Is it even
> working?
> St.Ack
>
--
View this message in context:
http://www.nabble.com/HBase-looses-regions.-tp23657983p23730737.html
Sent from the HBase User mailing list archive at Nabble.com.
