[
https://issues.apache.org/jira/browse/BIGTOP-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039891#comment-14039891
]
Evans Ye commented on BIGTOP-1336:
----------------------------------
OK, you might encounter an error like below when following the reproduce steps
to provision a kerberos enabled hadoop cluster on VMs.
{noformat}
err: /Stage[main]/Kerberos::Kdc/Exec[kdb5_util]/returns: change from notrun to
0 failed: Command exceeded timeout at
/tmp/vagrant-puppet-2/modules-0/kerberos/manifests/init.pp:113
{noformat}
Its actually raised by following shell command which was running exceed puppet
exec timeout during provisioning:
{noformat}
[root@bigtop1 ~]# kdb5_util -P cthulhu -r BIGTOP.ORG create -s
Loading random data
(hang for a long time)
{noformat}
If you go ahead look into the entropy of the VM, you'll get a pretty poor
number:
{noformat}
[root@bigtop1 ~]# cat /proc/sys/kernel/random/entropy_avail
4
{noformat}
So, the root cause of error we got is because the kerberos database
initialization step can not gather enough entropy on the VM.
The poor entropy performance often happened on virtual machines because its
hardware are simulated([Virtualbox Ticket
#11297|https://www.virtualbox.org/ticket/11297]).
A simple solution for this is to use
[rng-tools|http://my.itwnik.com/2012/04/how-to-increase-linux-kernel-entropy/].
Since this issue does not related to our puppet recipes but an environment
specific problem. I'll prefer to add rng-tools support in
[vagrant-puppet|https://github.com/apache/bigtop/tree/master/bigtop-deploy/vm/vagrant-puppet]'s
provision.sh.
Suggestions are welcome:)
> Puppet recipes failed to deploy kerberos enabled hadoop cluster
> ---------------------------------------------------------------
>
> Key: BIGTOP-1336
> URL: https://issues.apache.org/jira/browse/BIGTOP-1336
> Project: Bigtop
> Issue Type: Bug
> Components: Deployment
> Affects Versions: 0.7.0
> Reporter: Evans Ye
> Fix For: 0.8.0
>
> Attachments: BIGTOP-1336.1.patch
>
>
> Here are some missing dependency setting in our puppet recipes in order to
> get kerberos enabled on the hadoop cluster.
> The first one is that kerberos principal for hdfs user hasn't been created
> before formatting namenode, which cause the namenode formatting process
> failed.
> The second one is that {{/etc/default/hadoop-hdfs-datanode}} doesn't get
> ready before starting up datanodes and results in datanodes failed to startup.
> The datanode error log:
> {noformat}
> 2014-06-16 15:10:10,711 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.RuntimeException: Cannot start secure cluster without privileged
> resources.
> {noformat}
> Here's the reproduce steps using
> [vagrant-puppet|https://github.com/apache/bigtop/tree/master/bigtop-deploy/vm/vagrant-puppet]:
> 1.) Enable kerberos on the hadoop cluster.
> {noformat}
> $ vim bigtop-deploy/vm/vagrant-puppet/provision.sh
> {noformat}
> Add kerberos definitions.
> {noformat}
> cat > /bigtop-puppet/config/site.csv << EOF
> hadoop_head_node,$1
> hadoop_storage_dirs,/data/1,/data/2
> bigtop_yumrepo_uri,http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64
> jdk_package_name,java-1.7.0-openjdk-devel.x86_64
> components,hadoop,hbase,yarn,mapred-app
> hadoop_security,kerberos
> hadoop_kerberos_domain,vagrant
> hadoop_kerberos_realm,BIGTOP.ORG
> hadoop_kerberos_kdc_server,bigtop1.vagrant
> EOF
> {noformat}
> 2.) Spin up the cluster.
> {code}
> $ ./startup.sh --cluster
> {code}
> 3-1.) Get an error while formating namenode.
> {noformat}
> err: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Exec[namenode
> format]/returns: change from notrun to 0 failed: /bin/bash -c 'yes Y | hdfs
> namenode -format >> /var/lib/hadoop-hdfs/nn.format.log 2>&1' returned 1
> instead of one of [0] at
> /tmp/vagrant-puppet-2/modules-0/hadoop/manifests/init.pp:361
> {noformat}
> 3-2.) Get an error while starting up datanodes.
> {noformat}
> err:
> /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/Service[hadoop-hdfs-datanode]/ensure:
> change from stopped to running failed: Could not start
> Service[hadoop-hdfs-datanode]: Execution of '/sbin/service
> hadoop-hdfs-datanode start' returned 1: at
> /tmp/vagrant-puppet-2/modules-0/hadoop/manifests/init.pp:158
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
