On Tue, May 26, 2009 at 1:30 PM, llpind <[email protected]> wrote: > > Yea. DFS stays healthy. one of the nodes dies during massive load. I've > made the following changes: > > 1. upped FDs
Not according to the log below. It says: "> ulimit -n 1024" > 2. block size to default. only properties set in hbase-site.xml are > 'base.rootdir' and 'hbase.master'. Did you add the timeout here or symlink hadoop-site.xml into your $HBASE_HOME/conf? > 3. turned debugging on for hbase and hbase dfs. > 4. set auth flushing to false I believe this increases the load. > 5. set 12MB client buffer > 6. half second sleep in data load per row (1 second after 1000 batch > update). Until things are settled, why not wait a minute between 1000 row uploads. Of what do your rows consist? Are you inputting small or big payloads? How many columns at a time? How many column families? Reading your log below, the first time you try to flush we timeout on HDFS. Claims bad datanode. Can you check your datanode logs? You only have three. See what its complaining about? Might be the FD issue. Regionserver shuts itself down because it can't successfully flush. This is a controlled shutdown to minimize dataloss. St.Ack > > restarted cluster & data load.... > > Hadoop DFS looks like this: ========================= > > Cluster Summary > 34 files and directories, 8 blocks = 42 total. Heap Size is 28.88 MB / 2.6 > GB (1%) > > Configured Capacity : 550.01 GB > DFS Used : 158.55 KB > Non DFS Used : 28.49 GB > DFS Remaining : 521.53 GB > DFS Used% : 0 % > DFS Remaining% : 94.82 % > Live Nodes : 3 > Dead Nodes : 0 > > ======================================== > > 1 region server went down again: tail of output: > > ======================================== > > ulimit -n 1024 > 2009-05-26 11:47:44,210 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: > vmInputArguments=[-Xmx3000m, -XX:NewSize=6m, -XX:MaxNewSize=6m, > -XX:+UseConcMarkSweepGC, -verbose:gc, -XX:+PrintGCDetails, > -XX:+PrintGCTimeStamps, -XX:+CMSIncrementalMode, > -Xloggc:/home/hadoop/hbase19/logs/gc-hbase.log, > -XX:+HeapDumpOnOutOfMemoryError, > -Dhbase.log.dir=/home/hadoop/hbase19/bin/../logs, > -Dhbase.log.file=hbase-hadoop-regionserver-server175.apptechsys.com.log, > -Dhbase.home.dir=/home/hadoop/hbase19/bin/.., -Dhbase.id.str=hadoop, > -Dhbase.root.logger=INFO,DRFA, > -Djava.library.path=/home/hadoop/hbase19/bin/../lib/native/Linux-amd64-64] > 2009-05-26 11:47:44,414 INFO > org.apache.hadoop.hbase.regionserver.MemcacheFlusher: > globalMemcacheLimit=1.2g, globalMemcacheLimitLowMark=749.9m, maxHeap=2.9g > 2009-05-26 11:47:44,425 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 10000000ms > 2009-05-26 11:47:44,482 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics: > Initializing RPC Metrics with hostName=HRegionServer, port=60020 > 2009-05-26 11:47:44,787 DEBUG > org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at > server181:60000 that we are up > 2009-05-26 11:47:45,919 INFO org.apache.hadoop.ipc.HBaseClass: Retrying > connect to server: server181/192.168.240.181:60000. Already tried 0 time(s). > 2009-05-26 11:47:46,923 INFO org.apache.hadoop.ipc.HBaseClass: Retrying > connect to server: server181/192.168.240.181:60000. Already tried 1 time(s). > 2009-05-26 11:47:47,819 DEBUG > org.apache.hadoop.hbase.regionserver.HRegionServer: sending initial server > load: requests=0, regions=0, usedHeap=22, maxHeap=-1096 > 2009-05-26 11:47:47,868 DEBUG > org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: > hbase.regionserver.address=192.168.240.175 > 2009-05-26 11:47:47,868 DEBUG > org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: > fs.default.name=hdfs://server181:54310/hbase > 2009-05-26 11:47:47,868 DEBUG > org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: > hbase.rootdir=hdfs://server181:54310/hbase > 2009-05-26 11:47:48,170 DEBUG > org.apache.hadoop.hbase.regionserver.HRegionServer: Log dir > hdfs://server181:54310/hbase/log_192.168.240.175_1243363664487_60020 > 2009-05-26 11:47:48,275 INFO org.apache.hadoop.hbase.regionserver.HLog: New > log writer: > /hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243363668204 > 2009-05-26 11:47:48,282 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: > Initializing JVM Metrics with processName=RegionServer, > sessionId=regionserver/0.0.0.0:60020 > 2009-05-26 11:47:48,284 INFO > org.apache.hadoop.hbase.regionserver.metrics.RegionServerMetrics: > Initialized > 2009-05-26 11:47:48,493 INFO org.mortbay.util.Credential: Checking Resource > aliases > 2009-05-26 11:47:48,502 INFO org.mortbay.http.HttpServer: Version > Jetty/5.1.4 > 2009-05-26 11:47:48,503 INFO org.mortbay.util.Container: Started > HttpContext[/logs,/logs] > 2009-05-26 11:47:49,324 INFO org.mortbay.util.Container: Started > org.mortbay.jetty.servlet.webapplicationhand...@6ecf829d > 2009-05-26 11:47:49,393 INFO org.mortbay.util.Container: Started > WebApplicationContext[/static,/static] > 2009-05-26 11:47:49,666 INFO org.mortbay.util.Container: Started > org.mortbay.jetty.servlet.webapplicationhand...@608b8a47 > 2009-05-26 11:47:49,671 INFO org.mortbay.util.Container: Started > WebApplicationContext[/,/] > 2009-05-26 11:47:49,676 INFO org.mortbay.http.SocketListener: Started > SocketListener on 0.0.0.0:60030 > 2009-05-26 11:47:49,676 INFO org.mortbay.util.Container: Started > org.mortbay.jetty.ser...@49938039 > 2009-05-26 11:47:49,676 DEBUG > org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting to exit safe > mode > 2009-05-26 11:47:49,677 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > Responder: starting > 2009-05-26 11:47:49,679 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > listener on 60020: starting > 2009-05-26 11:47:49,679 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 0 on 60020: starting > 2009-05-26 11:47:49,679 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 1 on 60020: starting > 2009-05-26 11:47:49,680 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 2 on 60020: starting > 2009-05-26 11:47:49,680 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 3 on 60020: starting > 2009-05-26 11:47:49,680 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 4 on 60020: starting > 2009-05-26 11:47:49,698 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 5 on 60020: starting > 2009-05-26 11:47:49,698 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer started > at: 192.168.240.175:60020 > 2009-05-26 11:47:49,699 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 9 on 60020: starting > 2009-05-26 11:47:49,699 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 8 on 60020: starting > 2009-05-26 11:47:49,699 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 7 on 60020: starting > 2009-05-26 11:47:49,699 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 6 on 60020: starting > 2009-05-26 11:49:19,861 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > tableA,,1243363759453: safeMode=false > 2009-05-26 11:49:19,864 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: > tableA,,1243363759453: safeMode=false > 2009-05-26 11:49:19,912 DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT > at 192.168.240.179:60020 > 2009-05-26 11:49:19,918 DEBUG org.apache.hadoop.hbase.RegionHistorian: > Onlined > 2009-05-26 11:49:19,930 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Opening region tableA,,1243363759453/1501852784 > 2009-05-26 11:49:20,025 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Next sequence id for region tableA,,1243363759453 is 0 > 2009-05-26 11:49:20,031 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region tableA,,1243363759453/1501852784 available > 2009-05-26 11:49:20,031 DEBUG > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction > requested for region tableA,,1243363759453/1501852784 because: Region open > check > 2009-05-26 11:49:28,310 INFO org.apache.hadoop.hbase.regionserver.HRegion: > starting compaction on region tableA,,1243363759453 > > 2009-05-26 11:50:25,968 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE: > tableA,,1243363759453: safeMode=false > 2009-05-26 11:50:25,969 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: > MSG_REGION_CLOSE: tableA,,1243363759453: safeMode=false > 2009-05-26 11:50:25,969 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Closing tableA,,1243363759453: compactions & flushes disabled > 2009-05-26 11:50:25,969 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Updates disabled for region, no outstanding scanners on > tableA,,1243363759453 > 2009-05-26 11:50:25,969 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > No more row locks outstanding on region tableA,,1243363759453 > 2009-05-26 11:50:25,969 DEBUG org.apache.hadoop.hbase.regionserver.HStore: > closed 1501852784/entity > 2009-05-26 11:50:25,970 DEBUG org.apache.hadoop.hbase.regionserver.HStore: > closed 1501852784/link > 2009-05-26 11:50:25,970 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed tableA,,1243363759453 > 2009-05-26 11:50:39,882 DEBUG > org.apache.hadoop.hbase.regionserver.HRegionServer: setting compaction limit > to 2 > 2009-05-26 11:50:50,014 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > tableA,,1243363849039: safeMode=false > 2009-05-26 11:50:50,015 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: > tableA,,1243363849039: safeMode=false > 2009-05-26 11:50:50,016 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Opening region tableA,,1243363849039/407623107 > 2009-05-26 11:50:50,050 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Next sequence id for region tableA,,1243363849039 is 0 > 2009-05-26 11:50:50,054 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region tableA,,1243363849039/407623107 available > 2009-05-26 11:50:50,054 DEBUG > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction > requested for region tableA,,1243363849039/407623107 because: Region open > check > 2009-05-26 11:50:50,054 INFO org.apache.hadoop.hbase.regionserver.HRegion: > starting compaction on region tableA,,1243363849039 > 2009-05-26 11:50:50,056 DEBUG org.apache.hadoop.hbase.regionserver.HStore: > 407623107/entity: no store files to compact > 2009-05-26 11:50:50,056 DEBUG org.apache.hadoop.hbase.regionserver.HStore: > 407623107/link: no store files to compact > 2009-05-26 11:50:50,058 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region tableA,,1243363849039 in 0sec > 2009-05-26 11:52:39,913 DEBUG > org.apache.hadoop.hbase.regionserver.HRegionServer: setting compaction limit > to 3 > 2009-05-26 11:55:19,864 DEBUG > org.apache.hadoop.hbase.regionserver.HRegionServer: setting compaction limit > to 4 > 2009-05-26 11:58:16,316 INFO org.apache.hadoop.hbase.regionserver.HLog: > Closed > hdfs://server181:54310/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.0, > entries=100215. New log writer: > /hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364296306 > 2009-05-26 11:58:16,317 DEBUG org.apache.hadoop.hbase.regionserver.HLog: > Found 0 logs to remove out of total 1; oldest outstanding seqnum is 0 from > region tableA,,1243363849039 > 2009-05-26 11:58:39,916 DEBUG > org.apache.hadoop.hbase.regionserver.HRegionServer: setting compaction limit > to -1 > 2009-05-26 11:58:59,921 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: compactions no longer > limited > 2009-05-26 12:00:55,330 INFO org.apache.hadoop.hbase.regionserver.HLog: > Closed > hdfs://server181:54310/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243363668204, > entries=100788. New log writer: > /hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364455324 > 2009-05-26 12:00:55,330 DEBUG org.apache.hadoop.hbase.regionserver.HLog: > Found 0 logs to remove out of total 2; oldest outstanding seqnum is 0 from > region tableA,,1243363849039 > 2009-05-26 12:00:57,273 INFO org.apache.hadoop.hbase.regionserver.HLog: > Closed > hdfs://server181:54310/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364296306, > entries=101000. New log writer: > /hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364457266 > 2009-05-26 12:00:57,273 DEBUG org.apache.hadoop.hbase.regionserver.HLog: > Found 0 logs to remove out of total 3; oldest outstanding seqnum is 0 from > region tableA,,1243363849039 > 2009-05-26 12:03:33,816 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Flush requested on tableA,,1243363849039 > 2009-05-26 12:03:33,848 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Started memcache flush for region tableA,,1243363849039. Current region > memcache size 64.3m > 2009-05-26 12:03:34,211 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded > the native-hadoop library > 2009-05-26 12:03:34,214 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: > Successfully loaded & initialized native-zlib library > 2009-05-26 12:03:34,217 INFO org.apache.hadoop.io.compress.CodecPool: Got > brand-new compressor > 2009-05-26 12:03:37,694 INFO org.apache.hadoop.io.compress.CodecPool: Got > brand-new decompressor > 2009-05-26 12:03:37,695 INFO org.apache.hadoop.io.compress.CodecPool: Got > brand-new decompressor > 2009-05-26 12:03:37,695 INFO org.apache.hadoop.io.compress.CodecPool: Got > brand-new decompressor > 2009-05-26 12:03:37,695 INFO org.apache.hadoop.io.compress.CodecPool: Got > brand-new decompressor > 2009-05-26 12:03:37,820 DEBUG org.apache.hadoop.hbase.regionserver.HStore: > Added /hbase/tableA/407623107/link/mapfiles/7688740598584924238 with 356003 > entries, sequence id 356003, data size ~64.5m, file size 20.2m to > tableA,,1243363849039 > 2009-05-26 12:03:37,820 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Finished memcache flush of ~64.5m for region tableA,,1243363849039 in > 3972ms, sequence id=356003, compaction requested=false > 2009-05-26 12:03:56,790 INFO org.apache.hadoop.hbase.regionserver.HLog: > Closed > hdfs://server181:54310/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364455324, > entries=154001. New log writer: > /hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364636781 > 2009-05-26 12:03:56,790 DEBUG org.apache.hadoop.hbase.regionserver.HLog: > Last sequence written is empty. Deleting all old hlogs > 2009-05-26 12:03:56,790 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old log file > /hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.0 whose highest > sequence/edit id is 100214 > > 2009-05-26 12:03:56,790 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old log file > /hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.0 whose highest > sequence/edit id is 100214 > 2009-05-26 12:03:56,792 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old log file > /hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243363668204 whose > highest sequence/edit id is 201002 > 2009-05-26 12:03:56,798 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old log file > /hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364296306 whose > highest sequence/edit id is 302002 > 2009-05-26 12:03:56,806 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old log file > /hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364455324 whose > highest sequence/edit id is 456003 > 2009-05-26 12:06:14,313 INFO org.apache.hadoop.hbase.regionserver.HLog: > Closed > hdfs://server181:54310/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364457266, > entries=101000. New log writer: > /hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364774264 > 2009-05-26 12:06:14,314 DEBUG org.apache.hadoop.hbase.regionserver.HLog: > Found 0 logs to remove out of total 1; oldest outstanding seqnum is 456004 > from region tableA,,1243363849039 > 2009-05-26 12:08:51,328 INFO org.apache.hadoop.hbase.regionserver.HLog: > Closed > hdfs://server181:54310/hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364636781, > entries=101000. New log writer: > /hbase/log_192.168.240.175_1243363664487_60020/hlog.dat.1243364931323 > 2009-05-26 12:08:51,328 DEBUG org.apache.hadoop.hbase.regionserver.HLog: > Found 0 logs to remove out of total 2; oldest outstanding seqnum is 456004 > from region tableA,,1243363849039 > 2009-05-26 12:08:52,590 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Started memcache flush for region tableA,,1243363849039. Current region > memcache size 64.1m > 2009-05-26 12:08:52,591 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Flush requested on tableA,,1243363849039 > 2009-05-26 12:09:04,876 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer > Exception: java.net.SocketTimeoutException: 10000 millis timeout while > waiting for channel to be ready for write. ch : > java.nio.channels.SocketChannel[connected local=/192.168.240.175:56496 > remote=/192.168.240.175:50010] > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:162) > at > org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146) > at > org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2209) > > 2009-05-26 12:09:04,877 WARN org.apache.hadoop.hdfs.DFSClient: Error > Recovery for block blk_-7325527218992385186_1022 bad datanode[0] > 192.168.240.175:50010 > 2009-05-26 12:09:04,877 WARN org.apache.hadoop.hdfs.DFSClient: Error > Recovery for block blk_-7325527218992385186_1022 in pipeline > 192.168.240.175:50010, 192.168.240.180:50010: bad datanode > 192.168.240.175:50010 > 2009-05-26 12:09:10,084 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer > Exception: java.net.SocketTimeoutException: 5000 millis timeout while > waiting for channel to be ready for write. ch : > java.nio.channels.SocketChannel[connected local=/192.168.240.175:56113 > remote=/192.168.240.180:50010] > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:162) > at > org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146) > at > org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2209) > > 2009-05-26 12:09:10,085 WARN org.apache.hadoop.hdfs.DFSClient: Error > Recovery for block blk_-7325527218992385186_1023 bad datanode[0] > 192.168.240.180:50010 > 2009-05-26 12:09:10,135 FATAL > org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay of hlog > required. Forcing server shutdown > org.apache.hadoop.hbase.DroppedSnapshotException: region: > tableA,,1243363849039 > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:897) > at > org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:790) > at > org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:228) > at > org.apache.hadoop.hbase.regionserver.MemcacheFlusher.run(MemcacheFlusher.java:138) > Caused by: java.io.IOException: All datanodes 192.168.240.180:50010 are bad. > Aborting... > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2444) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1996) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160) > > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:1996) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2160) > 2009-05-26 12:09:10,138 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: > request=0.0, regions=1, stores=2, storefiles=1, storefileIndexSize=0, > memcacheSize=72, usedHeap=114, maxHeap=2999 > 2009-05-26 12:09:10,138 INFO > org.apache.hadoop.hbase.regionserver.MemcacheFlusher: > regionserver/0.0.0.0:60020.cacheFlusher exiting > 2009-05-26 12:09:10,288 INFO > org.apache.hadoop.hbase.regionserver.CompactSplitThread: > regionserver/0.0.0.0:60020.compactor exiting > 2009-05-26 12:09:10,512 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting > 2009-05-26 12:09:11,336 INFO org.apache.hadoop.hbase.regionserver.LogRoller: > LogRoller exiting. > 2009-05-26 12:09:12,744 DEBUG org.apache.hadoop.hbase.RegionHistorian: > Offlined > 2009-05-26 12:09:12,745 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > server on 60020 > 2009-05-26 12:09:12,746 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 1 on 60020: exiting > 2009-05-26 12:09:12,746 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC > Server listener on 60020 > 2009-05-26 12:09:12,746 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 3 on 60020: exiting > 2009-05-26 12:09:12,746 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 4 on 60020: exiting > 2009-05-26 12:09:12,746 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 2 on 60020: exiting > 2009-05-26 12:09:12,748 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer > 2009-05-26 12:09:12,748 INFO org.mortbay.util.ThreadedServer: Stopping > Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=60030] > 2009-05-26 12:09:12,749 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 5 on 60020: exiting > 2009-05-26 12:09:12,749 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC > Server Responder > 2009-05-26 12:09:12,750 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 8 on 60020: exiting > 2009-05-26 12:09:12,749 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 9 on 60020: exiting > 2009-05-26 12:09:12,756 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 7 on 60020: exiting > 2009-05-26 12:09:12,756 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 6 on 60020: exiting > 2009-05-26 12:09:12,756 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 0 on 60020: exiting > 2009-05-26 12:09:12,756 INFO org.mortbay.http.SocketListener: Stopped > SocketListener on 0.0.0.0:60030 > 2009-05-26 12:09:13,443 INFO org.mortbay.util.Container: Stopped > HttpContext[/logs,/logs] > 2009-05-26 12:09:13,444 INFO org.mortbay.util.Container: Stopped > org.mortbay.jetty.servlet.webapplicationhand...@6ecf829d > 2009-05-26 12:09:13,884 INFO org.mortbay.util.Container: Stopped > WebApplicationContext[/static,/static] > 2009-05-26 12:09:13,885 INFO org.mortbay.util.Container: Stopped > org.mortbay.jetty.servlet.webapplicationhand...@608b8a47 > 2009-05-26 12:09:14,535 INFO org.mortbay.util.Container: Stopped > WebApplicationContext[/,/] > 2009-05-26 12:09:14,536 INFO org.mortbay.util.Container: Stopped > org.mortbay.jetty.ser...@49938039 > 2009-05-26 12:09:14,536 DEBUG org.apache.hadoop.hbase.regionserver.HLog: > closing log writer in > hdfs://server181:54310/hbase/log_192.168.240.175_1243363664487_60020 > 2009-05-26 12:09:14,537 INFO > org.apache.hadoop.hbase.regionserver.LogFlusher: > regionserver/0.0.0.0:60020.logFlusher exiting > 2009-05-26 12:09:14,537 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker: > regionserver/0.0.0.0:60020.majorCompactionChecker exiting > 2009-05-26 12:09:18,856 INFO org.apache.hadoop.hbase.Leases: > regionserver/0.0.0.0:60020.leaseChecker closing leases > 2009-05-26 12:09:18,856 INFO org.apache.hadoop.hbase.Leases: > regionserver/0.0.0.0:60020.leaseChecker closed leases > 2009-05-26 12:09:26,928 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: On abort, closed hlog > 2009-05-26 12:09:26,929 DEBUG > org.apache.hadoop.hbase.regionserver.HRegionServer: closing region > tableA,,1243363849039 > 2009-05-26 12:09:26,929 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Closing tableA,,1243363849039: compactions & flushes disabled > 2009-05-26 12:09:26,929 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Updates disabled for region, no outstanding scanners on > tableA,,1243363849039 > 2009-05-26 12:09:26,929 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > No more row locks outstanding on region tableA,,1243363849039 > 2009-05-26 12:09:26,929 DEBUG org.apache.hadoop.hbase.regionserver.HStore: > closed 407623107/entity > 2009-05-26 12:09:26,929 DEBUG org.apache.hadoop.hbase.regionserver.HStore: > closed 407623107/link > 2009-05-26 12:09:26,929 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed tableA,,1243363849039 > 2009-05-26 12:09:26,929 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at: > 192.168.240.175:60020 > 2009-05-26 12:09:27,033 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: > regionserver/0.0.0.0:60020 exiting > 2009-05-26 12:09:27,034 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown > thread. > 2009-05-26 12:09:27,034 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete > > > ======================================== > > > stack-3 wrote: >> >> That looks sick. Different log files can't close? Enable DEBUG in >> your logs. See FAQ for how. You sure your HDFS healthy? Is it even >> working? >> St.Ack >> > > -- > View this message in context: > http://www.nabble.com/HBase-looses-regions.-tp23657983p23730737.html > Sent from the HBase User mailing list archive at Nabble.com. > >
