[
https://issues.apache.org/jira/browse/HBASE-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107551#comment-13107551
]
Eric Yang commented on HBASE-4415:
----------------------------------
*share/hbase/templates/conf* is an newly added template directory. People can
still use the default files in conf directory for backward compatibility until
hbase-setup-conf.sh become the standard method of configuring HBase cluster.
For now, there are two copies of hbase-env.sh, one in conf directory, and one
in src/packages/templates/conf. The copy in src/packages/templates/conf is
designed to be consumed by hbase-setup-conf.sh, and it mimics the changes that
happens to trunk/conf/hbase-env.sh. If the community agrees that we can use
one copy of base-env.sh and hbase-site.xml for both purpose of script
automation and hand edit config file. I am more than willing to unify them as
one copy and place in src/packages/templates/conf for clarity to indicate this
is for script automation.
Standard Maven practice uses the following structure:
{noformat}
src/main - application code
src/test - test code
{noformat}
In Hadoop related projects, release packaging has introduced new directory
structure:
{noformat}
src/packages/tarball - tarball manifest file or related meta data
src/packages/rpm - rpm related meta data
src/packages/deb - debian related meta data
src/packages/template/conf - Configuration template files
{noformat}
This practice is to improve the release packages. In the past, Hadoop related
projects are releasing a binary/source hybrid tar ball. This has proven
problematic to maintain for different architecture. The intention is to
improve the system so that we can create binary release for each architecture
and decouple source and binary from tar ball releases. There are some more
work to be done for packaging separation, but it is not in scope for this jira.
The script is designed to run as root. For setting up a hbase cluster with
hadoop 0.20.205. There are a few assumption made for kerberos key tabs. The
key tab file names should be:
namenode - nn.service.keytab
secondarynamenode - sn.service.keytab
datanode - dn.service.keytab
jobtracker - jt.service.keytab
tasktracker - tt.service.keytab
hbase - hbase.service.keytab
Compose a shell script that invokes each component's installation script and
run them through pdsh, or use Apache Ambari (HMS):
{noformat}
export namenode=nn1
export datanode=dn1,dn2,dn3,dn4
export secondary_namenode=sn1
export jobtracker=jt1
export tasktracker=tt1,tt2,tt3,tt4
export hbase_regionservers=rs1,rs2,rs3,rs4
export hbase_master=hbm
# Setup Hadoop configuration
/home/hms/apps/hadoop-0.20.205.0/sbin/hadoop-setup-conf.sh \
--java-home=/home/hms/apps/jdk1.6.0_27 \
--namenode-host=${namenode} \
--secondarynamenode-host=${secondary_namenode} \
--jobtracker-host=${jobtracker} \
--conf-dir=/etc/hadoop \
--hdfs-dir=/data/0/hadoop/var/hdfs,/data/1/hadoop/var/hdfs,/data/2/hadoop/var/hdfs,/data/3/hadoop/var/hdfs
\
--namenode-dir=/data/2/hadoop/var/hdfs/name \
--mapred-dir=/data/0/tmp/mapred-local,/data/1/tmp/mapred-local,/data/2/tmp/mapred-local,/data/3/tmp/mapred-local
\
--datanode-dir=/data/0/hadoop/var/hdfs/data,/data/1/hadoop/var/hdfs/data,/data/2/hadoop/var/hdfs/data,/data/3/hadoop/var/hdfs/data
\
--log-dir=/home/hms/apps/hadoop-0.20.205.0/logs \
--pid-dir=/home/hms/apps/hadoop-0.20.205.0/logs \
--hdfs-user=hdfs \
--mapreduce-user=mr \
--kerberos-realm=APACHE.ORG \
--hdfs-user-keytab=/home/hdfs/hdfs.headless.keytab \
--mapreduce-user-keytab=/home/mr/mr.headless.keytab \
--datanodes=${datanode} \
--tasktrackers=${tasktracker} \
--group=hadoop \
--auto
# Format Hadoop HDFS
/home/hms/apps/hadoop-0.20.205.0/sbin/hadoop-setup-hdfs.sh \
--config /etc/hadoop \
--hdfs-user=hrt_hdfs \
--mapreduce-user=hrt_mr \
--kerberos-realm=APACHE.ORG \
--hdfs-user-keytab=/home/hdfs/hdfs.headless.keytab \
--format
# Start up secondary name node, datanodes, job tracker, task tracker through
hadoop-daemon.sh
# Setup HBase
/home/hms/apps/hbase-0.90.3/sbin/hbase-setup-conf.sh \
--hadoop-conf=/etc/hadoop \
--hadoop-home=/home/hms/apps/hadoop-0.20.205.0 \
--hadoop-namenode=${namenode} \
--hbase-home=/home/hms/apps/hbase-0.90.3 \
--hbase-conf=/home/hms/apps/hbase-0.90.3/conf \
--hbase-log=/home/hms/apps/hbase-0.90.3/logs \
--hbase-pid=/home/hms/apps/hbase-0.90.3/var/run \
--java-home=/usr/java/default \
--kerberos-realm=APACHE.ORG \
--kerberos-principal-id=hbase/_HOST \
--keytab-dir=/etc/security/keytabs \
--regionservers=${hbase_regionservers} \
--zookeeper-quorum=${hbase_master} \
--zookeeper-snapshot=/home/hms/apps/var/lib/zookeeper
--hbase-user=hbase
--hadoop-replication=3
# Start HBase Master, region servers using hbase-daemon.sh.
{noformat}
Hope this helps.
> Add configuration script for setup HBase (hbase-setup-conf.sh)
> --------------------------------------------------------------
>
> Key: HBASE-4415
> URL: https://issues.apache.org/jira/browse/HBASE-4415
> Project: HBase
> Issue Type: New Feature
> Components: scripts
> Environment: Java 6, Linux
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HBASE-4415-1.patch, HBASE-4415.patch
>
>
> The goal of this jura is to provide a installation script for configuring
> HBase environment and configuration. By using the same pattern of
> *-setup-conf.sh for all Hadoop related projects. For HBase, the usage of the
> script looks like this:
> {noformat}
> usage: ./hbase-setup-conf.sh <parameters>
> Optional parameters:
> --hadoop-conf=/etc/hadoop Set Hadoop configuration directory
> location
> --hadoop-home=/usr Set Hadoop directory location
> --hadoop-namenode=localhost Set Hadoop namenode hostname
> --hadoop-replication=3 Set HDFS replication
> --hbase-home=/usr Set HBase directory location
> --hbase-conf=/etc/hbase Set HBase configuration
> directory location
> --hbase-log=/var/log/hbase Set HBase log directory location
> --hbase-pid=/var/run/hbase Set HBase pid directory location
> --hbase-user=hbase Set HBase user
> --java-home=/usr/java/default Set JAVA_HOME directory location
> --kerberos-realm=KERBEROS.EXAMPLE.COM Set Kerberos realm
> --kerberos-principal-id=_HOST Set Kerberos principal ID
> --keytab-dir=/etc/security/keytabs Set keytab directory
> --regionservers=localhost Set regionservers hostnames
> --zookeeper-home=/usr Set ZooKeeper directory location
> --zookeeper-quorum=localhost Set ZooKeeper Quorum
> --zookeeper-snapshot=/var/lib/zookeeper Set ZooKeeper snapshot location
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
