Your nodes are too much out of sync with time. > Reported time is too far out of sync with master. Time difference of > 496371ms > max allowed of 30000ms You need to set up time synchronization service for your cluster.
Thank you! Sincerely, Leonid Fedotov Hortonworks support team On Dec 10, 2012, at 6:17 PM, tgh wrote: > Meanwhile , log is master , that is, blade1, is like this, there are some > ERRor like this, for > > 2012-09-01 06:31:05,558 INFO org.apache.hadoop.hbase.master.ServerManager: > Registering server=blade2,60020,1346452716636, regionCount=0, userLoad=false > 2012-09-01 06:31:05,569 WARN org.apache.hadoop.hbase.master.ServerManager: > Server blade4,60020,1346451768443 has been rejected; Reported time is too far > out of sync with master. Time difference of 496371ms > max allowed of 30000ms > 2012-09-01 06:31:05,581 WARN org.apache.hadoop.hbase.master.ServerManager: > Server blade5,60020,1346452001672 has been rejected; Reported time is too far > out of sync with master. Time difference of 263137ms > max allowed of 30000ms > 2012-09-01 06:31:05,583 ERROR org.apache.hadoop.hbase.master.HMaster: Region > server serverName=blade4,60020,1346451768443, load=(requests=0, regions=0, > usedHeap=142, maxHeap=966) reported a fatal error: > ABORTING region server serverName=blade4,60020,1346451768443, > load=(requests=0, regions=0, usedHeap=142, maxHeap=966): Unhandled exception: > org.apache.hadoop.hbase.ClockOutOfSyncException: Server > blade4,60020,1346451768443 has been rejected; Reported time is too far out of > sync with master. Time difference of 496371ms > max allowed of 30000ms > Cause: > org.apache.hadoop.hbase.ClockOutOfSyncException: > org.apache.hadoop.hbase.ClockOutOfSyncException: Server > blade4,60020,1346451768443 has been rejected; Reported time is too far out of > sync with master. Time difference of 496371ms > max allowed of 30000ms > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:525) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1574) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1531) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572) > at java.lang.Thread.run(Thread.java:722) > Caused by: org.apache.hadoop.ipc.RemoteException: > org.apache.hadoop.hbase.ClockOutOfSyncException: Server > blade4,60020,1346451768443 has been rejected; Reported time is too far out of > sync with master. Time difference of 496371ms > max allowed of 30000ms > at > org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:193) > at > org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:141) > at > org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:675) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) > > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) > at $Proxy5.regionServerStartup(Unknown Source) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1570) > ... 3 more > > > -----邮件原件----- > 发件人: [email protected] > [mailto:[email protected]] 代表 tgh > 发送时间: 2012年12月11日 9:59 > 收件人: [email protected] > 主题: 答复: how config multi regionserver, or what is wrong? > > And our hosts is follows > > [root@blade1 ~]# cat /etc/hosts > 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 > ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 > > 192.168.76.233 blade1 > 192.168.76.234 blade2 > 192.168.76.235 blade3 > 192.168.76.236 blade4 > 192.168.76.237 blade5 > 192.168.76.238 blade6 > 192.168.76.239 blade7 > 192.168.76.240 blade8 > > 192.168.76.245 fnode1 > 192.168.76.246 fnode2 > [root@blade1 ~]# > > -----邮件原件----- > 发件人: [email protected] > [mailto:[email protected]] 代表 tgh > 发送时间: 2012年12月11日 9:00 > 收件人: [email protected] > 主题: 答复: how config multi regionserver, or what is wrong? > > Thank you for your reply, > And the configuration file is here, > Could you help me, > > > Thank you > --------------------------- > Tian Guanhua > > > > [root@blade1 conf]# cat regionservers > blade1 > blade2 > blade3 > blade4 > blade5 > blade6 > blade7 > blade8 > [root@blade1 conf]# > [root@blade1 conf]# vim hbase-site.xml > <?xml version="1.0"?> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > <!-- > --> > <configuration> > <property> > <name>hbase.rootdir</name> > <value>hdfs://blade1:9000/hbase</value> > <description>The directory shared by RegionServers.</description> > </property> > <property> > <name>hbase.cluster.distributed</name> > <value>true</value> > </property> > <property> > <name>hbase.zookeeper.quorum</name> > <value>blade1,blade2,blade3</value> > </property> > <property> > <name>hbase.zookeeper.property.dataDir</name> > <value>/home/liuxin/zookeeper/data</value> > </property> > <property> > <name>dfs.support.append</name> > <value>true</value> > </property> > <property> > <name>dfs.datanode.max.xcievers</name> > <value>4096</value> > </property> > <property> > <name>hbase.master</name> > <value>blade1:60000</value> > </property> > </configuration> > > > > > > > -----邮件原件----- > 发件人: [email protected] > [mailto:[email protected]] 代表 > Jean-Marc Spaggiari > 发送时间: 2012年12月10日 20:54 > 收件人: [email protected] > 主题: Re: how config multi regionserver, or what is wrong? > > Hi Tian, > > Can you share you configuration files? > > Do you have something like that on your hbase-site.xml file? > > <property> > <name>hbase.cluster.distributed</name> > <value>true</value> > <description>The mode the cluster will be in. Possible values are > false: standalone and pseudo-distributed setups with managed Zookeeper > true: fully-distributed with unmanaged Zookeeper Quorum (see > hbase-env.sh) > </description> > </property> > > > JM > > 2012/12/10, tgh <[email protected]>: >> Hi >> I try to use hbase, and now ,I have a problem with hbase >> configuration, I use 8node for try, and it seems to work, hadoop, >> zookeeper, hbase all boot up, and it could insert into with putAPI , >> But when I try to use masterIP:60010 to manage it, I find there are >> only one regionserver there, and it is localhost:60030, why? >> I have set regionserver file , there are 8 nodes there, and all could >> ssh to each other without passwd, >> >> And I also use hdfsNamenodeIP:50070 to see HDFS, and it seems OK, the >> data have been balanced across 8node, but I wander hbase only have one >> regionserver to work, although when I star/stop regionserver , there >> are 8region server to start and stop, >> >> And I try to put data into hbase, it seems ok at first, but after >> 200million in hbase, it seems really hard to insert more into it, it >> is very slow, and I use masterIP:60010 to manage it, I find there are >> only one regionserver there, and it is localhost:60030, why? >> >> >> Could you help me, >> >> >> >> Thank you >> ------------------- >> Tian Guanhua >> >> >> >> >> >> >> > > > >
