Can you check region server log on inspur253.deu.edu.cn,60020,1388053123213 ?
Cheers On Wed, Jan 8, 2014 at 5:34 PM, hzwangxx <[email protected]> wrote: > Hi, Ted > This is the master log: > > *2014-01-08 18:40:48,640 [IPC Server handler 37 on 60000] INFO > org.apache.hadoop.hbase.master.HMaster - Addeu.edu.cnded move plan > hri=test,|a1bd417af20749* > *110c98d37df4b4a4a8|1381022839579|2834679632306214,1382674684751.78c953d53f6498664d9a067701a7e7d7., > src=inspur251.deu.edu.cn <http://inspur251.deu.edu.cn>,60020,1388050383789, > d* > *est=inspur255.deu.edu.cn > <http://inspur255.deu.edu.cn>,60020,1388056934052, running balancer* > *2014-01-08 18:40:48,860 [main-EventThread] INFO > org.apache.hadoop.hbase.master.AssignmentManager - The master has opened > the region test,|a1bd4* > *17af20749110c98d37df4b4a4a8|1381022839579|2834679632306214,1382674684751.78c953d53f6498664d9a067701a7e7d7. > that was online on inspur255.deu.edu.cn <http://inspur255.deu.edu.cn>* > *,60020,1388056934052* > *2014-01-08 18:40:50,690 [IPC Server handler 44 on 60000] INFO > org.apache.hadoop.hbase.master.HMaster - Added move plan > hri=test,|a786b578f29c43* > *1374b62ca7df559277|1380621867798|7273081556605895,1382673452778.c621b3bf29262ca5248c03a8d6ebb41e., > src=inspur251.deu.edu.cn <http://inspur251.deu.edu.cn>,60020,1388050383789, > d* > *est=inspur253.deu.edu.cn > <http://inspur253.deu.edu.cn>,60020,1388053123213, running balancer* > *2014-01-08 18:40:51,078 [main-EventThread] INFO > org.apache.hadoop.hbase.master.AssignmentManager - The master has opened > the region test,|a786b* > *578f29c431374b62ca7df559277|1380621867798|7273081556605895,1382673452778.c621b3bf29262ca5248c03a8d6ebb41e. > that was online on inspur253.deu.edu.cn <http://inspur253.deu.edu.cn>* > *,60020,1388053123213* > *2014-01-08 18:40:53,944 [IPC Server handler 46 on 60000] INFO > org.apache.hadoop.hbase.master.HMaster - Added move plan > hri=test,|bucket-ynote-o* > *nline|4762b78b834d267a5ca71fadab88a9b9|1382995377500|4030050098946420,1386199418157.4dad873a6af4d3a9809339281c3cb34c., > src=inspur251.deu.edu.cn <http://inspur251.deu.edu.cn>,60* > *020,1388050383789, dest=inspur254.deu.edu.cn > <http://inspur254.deu.edu.cn>,60020,1388054917364, running balancer* > *2014-01-08 18:40:55,416 [main-EventThread] INFO > org.apache.hadoop.hbase.master.AssignmentManager - The master has opened > the region test,|bucke* > *t-ynote-online|4762b78b834d267a5ca71fadab88a9b9|1382995377500|4030050098946420,1386199418157.4dad873a6af4d3a9809339281c3cb34c. > that was online on ins* > *pur254.deu.edu.cn <http://pur254.deu.edu.cn>,60020,1388054917364* > *2014-01-08 18:40:57,067 [IPC Server handler 7 on 60000] INFO > org.apache.hadoop.hbase.master.HMaster - Added move plan > hri=test,|c4617a74baabdd0* > *6f3e69bf5b36fe8ec|1381090469015|2902310534369672,1382457919237.0e311941f5ff202bcefe57aa4079a188., > src=inspur251.deu.edu.cn <http://inspur251.deu.edu.cn>,60020,1388050383789, > de* > *st=inspur253.deu.edu.cn > <http://inspur253.deu.edu.cn>,60020,1388053123213, running balancer* > *2014-01-08 18:40:57,511 [main-EventThread] INFO > org.apache.hadoop.hbase.master.AssignmentManager - The master has opened > the region test,|c4617* > *a74baabdd06f3e69bf5b36fe8ec|1381090469015|2902310534369672,1382457919237.0e311941f5ff202bcefe57aa4079a188. > that was online on inspur253.deu.edu.cn <http://inspur253.deu.edu.cn>* > *,60020,1388053123213* > *2014-01-08 18:41:00,143 [IPC Server handler 23 on 60000] INFO > org.apache.hadoop.hbase.master.HMaster - Added move plan > hri=test,|coursera-video* > *|0dc2a6efeba02749d6187481d8f18357|1380039238151|41650362285832984,1380853043651.3a955559fb65caf32a05e18b8b6b93f8., > src=inspur251.deu.edu.cn <http://inspur251.deu.edu.cn>,60020,* > *1388050383789, dest=inspur308.deu.edu.cn > <http://inspur308.deu.edu.cn>,60020,1388059770705, running balancer* > *2014-01-08 18:41:00,989 [main-EventThread] INFO > org.apache.hadoop.hbase.master.AssignmentManager - The master has opened > the region test,|cours* > *era-video|0dc2a6efeba02749d6187481d8f18357|1380039238151|41650362285832984,1380853043651.3a955559fb65caf32a05e18b8b6b93f8. > that was online on inspur3* > *08.deu.edu.cn <http://08.deu.edu.cn>,60020,1388059770705* > *2014-01-08 18:41:01,904 [935521285@qtp-711761606-0] WARN > org.apache.hadoop.conf.Configuration - fs.default.name > <http://fs.default.name> is deprecated. Instead, use fs.defau* > *ltFS* > *2014-01-08 18:41:02,569 [IPC Server handler 24 on 60000] INFO > org.apache.hadoop.hbase.master.HMaster - Added move plan > hri=test,|dedb0fec255422* > *f3f9446a6abc1ac514|1379899817964|1711659483241241,1383445135018.34f3ef516fe6fba940eeb0902b9acd3d., > src=inspur251.deu.edu.cn <http://inspur251.deu.edu.cn>,60020,1388050383789, > d* > *est=inspur254.deu.edu.cn > <http://inspur254.deu.edu.cn>,60020,1388054917364, running balancer* > *2014-01-08 18:41:02,873 [main-EventThread] INFO > org.apache.hadoop.hbase.master.AssignmentManager - The master has opened > the region test,|dedb0* > *fec255422f3f9446a6abc1ac514|1379899817964|1711659483241241,1383445135018.34f3ef516fe6fba940eeb0902b9acd3d. > that was online on inspur254.deu.edu.cn <http://inspur254.deu.edu.cn>* > *,60020,1388054917364* > *2014-01-08 18:41:04,184 [IPC Server handler 44 on 60000] INFO > org.apache.hadoop.hbase.master.HMaster - Added move plan > hri=test,|e4e49102c1e6ea* > *97094a40c57420a628|1381085596723|42696720858381436,1382457119052.ac201a56d80f13ca5357d474578a91c2., > src=inspur251.deu.edu.cn > <http://inspur251.deu.edu.cn>,60020,1388050383789, * > *dest=inspur308.deu.edu.cn > <http://inspur308.deu.edu.cn>,60020,1388059770705, running balancer* > *2014-01-08 18:41:04,863 [main-EventThread] INFO > org.apache.hadoop.hbase.master.AssignmentManager - The master has opened > the region test,|e4e49* > *102c1e6ea97094a40c57420a628|1381085596723|42696720858381436,1382457119052.ac201a56d80f13ca5357d474578a91c2. > that was online on inspur308.photo.163.or* > *g,60020,1388059770705* > *2014-01-08 18:43:46,735 [935521285@qtp-711761606-0] WARN > org.apache.hadoop.conf.Configuration - fs.default.name > <http://fs.default.name> is deprecated. Instead, use fs.defau* > *ltFS* > *2014-01-08 18:46:51,704 [935521285@qtp-711761606-0] WARN > org.apache.hadoop.conf.Configuration - fs.default.name > <http://fs.default.name> is deprecated. Instead, use fs.defau* > *ltFS* > *2014-01-08 18:47:03,294 [935521285@qtp-711761606-0] INFO > org.apache.zookeeper.ZooKeeper - Initiating client connection, > connectString=inspur254.phot* > *o.163.org <http://o.163.org>:2181,inspur253.deu.edu.cn > <http://inspur253.deu.edu.cn>:2181,inspur252.deu.edu.cn > <http://inspur252.deu.edu.cn>:2181,inspur251.deu.edu.cn > <http://inspur251.deu.edu.cn>:2181,inspur255.deu.edu.cn > <http://inspur255.deu.edu.cn>:2181 sessionTimeout=120* > *0000 > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@172b29ed* > *2014-01-08 18:47:03,295 [935521285@qtp-711761606-0] INFO > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - The identifier of > this process is * > *[email protected] <[email protected]>* > *2014-01-08 18:47:03,295 > [935521285@qtp-711761606-0-SendThread(inspur251.deu.edu.cn > <http://inspur251.deu.edu.cn>:2181)] INFO org.apache.zookeeper.ClientCnxn > - Opening socket c* > *onnection to server inspur251.deu.edu.cn/172.17.7.1:2181 > <http://inspur251.deu.edu.cn/172.17.7.1:2181>. Will not attempt to > authenticate using SASL (Unable to locate a login configuration)* > *2014-01-08 18:47:03,297 > [935521285@qtp-711761606-0-SendThread(inspur251.deu.edu.cn > <http://inspur251.deu.edu.cn>:2181)] INFO org.apache.zookeeper.ClientCnxn > - Socket connectio* > *n established to inspur251.deu.edu.cn/172.17.7.1:2181 > <http://inspur251.deu.edu.cn/172.17.7.1:2181>, initiating session* > *2014-01-08 18:47:03,302 > [935521285@qtp-711761606-0-SendThread(inspur251.deu.edu.cn > <http://inspur251.deu.edu.cn>:2181)] INFO org.apache.zookeeper.ClientCnxn > - Session establishment complete on server > inspur251.deu.edu.cn/172.17.7.1:2181 > <http://inspur251.deu.edu.cn/172.17.7.1:2181>, sessionid = > 0x42b69781e0f11d, negotiated timeout = 300000* > *2014-01-08 18:47:24,670 [935521285@qtp-711761606-0] INFO > org.apache.zookeeper.ZooKeeper - Session: 0x42b69781e0f11d closed* > *2014-01-08 18:47:24,670 [935521285@qtp-711761606-0-EventThread] INFO > org.apache.zookeeper.ClientCnxn - EventThread shut down* > *2014-01-08 18:58:19,346 [IPC Reader 6 on port 60000] WARN > org.apache.hadoop.ipc.HBaseServer - Incorrect header or version mismatch > from 172.17.4.249:54719 <http://172.17.4.249:54719> got version 4 expected > version 3* > > *I killed the process around **2014-01-08 18:55.T**he hanging region (* > 812912be704946d24c5f1b5e3184b2f5*) has not any log.* > > *Thanks* > 在 2014年1月8日，23:58，Ted Yu <[email protected]> 写道： > > Can you pastebin master log around 2014-01-08 18:40 ? > > Thanks > > > On Wed, Jan 8, 2014 at 3:57 AM, hzwangxx <[email protected]> wrote: > >> Hi, all >> I restart a region server by using graceful_stop.sh >> (bin/graceful_stop.sh --restart --reload --debug hostname), when running a >> moment, the process hanging as follows: >> >> 2014-01-08 18:40:48,150 [main] INFO region_mover - Moving region >> 78c953d53f6498664d9a067701a7e7d7 (42 of 340) to server= >> inspur255.deu.edu.cn,60020,1388056934052 >> 2014-01-08 18:40:50,097 [main] INFO region_mover - Moving region >> c621b3bf29262ca5248c03a8d6ebb41e (43 of 340) to server= >> inspur253.deu.edu.cn,60020,1388053123213 >> 2014-01-08 18:40:51,652 [main] INFO region_mover - Moving region >> 4dad873a6af4d3a9809339281c3cb34c (44 of 340) to server= >> inspur254.deu.edu.cn,60020,1388054917364 >> 2014-01-08 18:40:56,701 [main] INFO region_mover - Moving region >> 0e311941f5ff202bcefe57aa4079a188 (45 of 340) to server= >> inspur253.deu.edu.cn,60020,1388053123213 >> 2014-01-08 18:40:58,632 [main] INFO region_mover - Moving region >> 3a955559fb65caf32a05e18b8b6b93f8 (46 of 340) to server= >> inspur308.deu.edu.cn,60020,1388059770705 >> 2014-01-08 18:41:02,127 [main] INFO region_mover - Moving region >> 34f3ef516fe6fba940eeb0902b9acd3d (47 of 340) to server= >> inspur254.deu.edu.cn,60020,1388054917364 >> 2014-01-08 18:41:03,689 [main] INFO region_mover - Moving region >> ac201a56d80f13ca5357d474578a91c2 (48 of 340) to server= >> inspur308.deu.edu.cn,60020,1388059770705 >> 2014-01-08 18:41:05,669 [main] INFO region_mover - Moving region >> 812912be704946d24c5f1b5e3184b2f5 (49 of 340) to server= >> inspur253.deu.edu.cn,60020,1388053123213 >> >> I run ‘du' command to check the last region , which has not any data. >> hadoop@inspur249:~/hbase$ hdfs dfs -du -s -h >> /hbase/test/812912be704946d24c5f1b5e3184b2f5/* >> 486 /hbase/test/812912be704946d24c5f1b5e3184b2f5/.regioninfo >> 0 /hbase/test/812912be704946d24c5f1b5e3184b2f5/body >> 0 /hbase/test/812912be704946d24c5f1b5e3184b2f5/meta >> >> hadoop version is cdh4.2.1 and hbase is 0.94 >> >> Thanks! >> Best Regards~ >> Xiyi >> >> > >
