I "somewhat" have HBase up and running in a distributed mode. It starts
fine, I can use "hbase shell" to create, disable, and drop tables;
however, after a short period of time HMaster and the HRegionalservers
terminate. Decoding the error messages is a bit bewildering and the
O'Reilly HBase book hasn't helped much with message decoding.
Here is a snippet of the messages from a regionalserver log:
~~~
U Stats: total=6.68 MB, free=807.12 MB, max=813.8 MB, blocks=2,
accesses=19, hits=17, hitRatio=89.47%%, cachingAccesses=17,
cachingHits=15, cachingHitsRatio=88.
23%%, evictions=0, evicted=0, evictedPerRun=NaN
2012-06-27 12:36:47,103 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.68
MB, free=807.12 MB, max=813.8 MB, blocks=2, accesses=19, hits=17,
hitRatio=89.47%%, cachingAccesses=17, cachingHits=15, cachingHitsRatio=88.
23%%, evictions=0, evicted=0, evictedPerRun=NaN
2012-06-27 12:40:02,106 INFO org.apache.zookeeper.ClientCnxn: Unable to
read additional data from server sessionid 0x382f6861690003, likely
server has closed socket, closing socket connection and attempting
reconnect
2012-06-27 12:40:02,112 INFO org.apache.zookeeper.ClientCnxn: Unable to
read additional data from server sessionid 0x382f6861690004, likely
server has closed socket, closing socket connection and attempting
reconnect
2012-06-27 12:40:02,245 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server devrackA-01/172.18.0.2:2181
2012-06-27 12:40:02,247 WARN org.apache.zookeeper.ClientCnxn: Session
0x382f6861690003 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.NoRouteToHostException: No route to host
~~~
No route to host would imply it can't reach one of my HQuorumpeers, but
it talks to them when I first run start-hase.sh. Also there is no DNS
involved, the /etc/hosts files are identical on all nodes, and it's
currently a closed cluster. All nodes are on the same subnet 172.18/16
Do I have something wrong in one of my xml files:
Core-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/var/hbase-hadoop/tmp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://devrackA-00:8020</value>
<final>true</final>
</property>
</configuration>
Hdfs-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/var/hbase-hadoop/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/var/hbase-hadoop/data</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/var/hbase-hadoop/namesecondary</value>
</property>
<property>
<name>dfs.datanode.max.xcievers</name>
<value>4096</value>
</property>
</configuration>
Hbase-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-->
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://devrackA-00:8020/var/hbase-hadoop/hbase</value>
</property>
<property>
<name>dfs.datanode.max.xcievers</name>
<value>4096</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.regionserver.handler.count</name>
<value>20</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>devrackA-00,devrackA-01,devrackA-25</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/var/hbase-hadoop/zookeeper</value>
</property>
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>500</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.zookeeper.property.initLimit</name>
<value>10</value>
</property>
<property>
<name>hbase.zookeeper.property.syncLimit</name>
<value>5</value>
</property>
<property>
<name>hbase.zookeeper.property.tickTime</name>
<value>2000000</value>
</property>
</configuration>
Hbase-env.sh:
#
#/**
# * Copyright 2007 The Apache Software Foundation
# *
# * Licensed to the Apache Software Foundation (ASF) under one
# * or more contributor license agreements. See the NOTICE file
# * distributed with this work for additional information
# * regarding copyright ownership. The ASF licenses this file
# * to you under the Apache License, Version 2.0 (the
# * "License"); you may not use this file except in compliance
# * with the License. You may obtain a copy of the License at
# *
# * http://www.apache.org/licenses/LICENSE-2.0
# *
# * Unless required by applicable law or agreed to in writing, software
# * distributed under the License is distributed on an "AS IS" BASIS,
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
# * See the License for the specific language governing permissions and
# * limitations under the License.
# */
export HBASE_HEAPSIZE=4096
export HBASE_MANAGES_ZK=true
regionalservers:
devrackA-03
devrackB-31
devrackA-25
master:
devrackA-01
slaves:
devrackA-01
devrackA-03
devrackA-04
devrackA-05
devrackA-06
devrackA-07
devrackA-08
devrackA-09
devrackA-10
devrackA-11
devrackA-12
devrackA-17
devrackA-18
devrackA-19
devrackA-20
devrackA-21
devrackA-22
devrackA-23
devrackA-24
devrackA-25
devrackB-07
devrackB-08
devrackB-09
devrackB-10
devrackB-11
devrackB-12
devrackB-13
devrackB-14
devrackB-15
devrackB-16
devrackB-17
devrackB-18
devrackB-19
devrackB-20
devrackB-21
devrackB-22
devrackB-23
devrackB-24
devrackB-25
devrackB-26
devrackB-27
devrackB-28
devrackB-29
devrackB-30
devrackB-31
Thanks for the help
Jay Wilson
