[email protected] From: <Leidle>, "Leidle, Rob" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Friday, November 28, 2014 at 11:14 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Problem using puppet scripts to configure bigtop on AmazonLinux
Hello all, I am trying to configure & install Bigtop 0.8.0 using the puppet scripts on AmazonLinux on EC2. Thus far, almost everything has worked besides one minor change I have made to the site.pp manifest. However, I am running into a problem, it seems that the services such as proxy server or namenode are not immediately starting. You can see the error below in the purple text related to namenode. info: /Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]: Scheduling refresh of Service[hadoop-yarn-resourcemanager] info: /Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]: Scheduling refresh of Service[hadoop-mapreduce-historyserver] info: /Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]: Scheduling refresh of Service[hadoop-yarn-nodemanager] info: /Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]: Scheduling refresh of Service[hadoop-yarn-proxyserver] debug: /Stage[main]/Hadoop::Common-yarn/File[/etc/hadoop/conf/yarn-site.xml]: The container Class[Hadoop::Common-yarn] will propagate my refresh event debug: Service[hadoop-yarn-proxyserver](provider=redhat): Executing '/sbin/service hadoop-yarn-proxyserver status' debug: Service[hadoop-yarn-proxyserver](provider=redhat): Executing '/sbin/service hadoop-yarn-proxyserver start' err: /Stage[main]/Hadoop_head_node/Hadoop::Proxyserver[proxyserver]/Service[hadoop-yarn-proxyserver]/ensure: change from stopped to running failed: Could not start Service[hadoop-yarn-proxyserver]: Execution of '/sbin/service hadoop-yarn-proxyserver start' returned 3: at /mnt/var/lib/bootstrap-actions/1/bigtop-0.8.0/bigtop-deploy/puppet/modules/hadoop/manifests/init.pp:483 debug: Service[hadoop-yarn-proxyserver](provider=redhat): Executing '/sbin/service hadoop-yarn-proxyserver status' debug: /Stage[main]/Hadoop_head_node/Hadoop::Proxyserver[proxyserver]/Service[hadoop-yarn-proxyserver]: Skipping restart; service is not running notice: /Stage[main]/Hadoop_head_node/Hadoop::Proxyserver[proxyserver]/Service[hadoop-yarn-proxyserver]: Triggered 'refresh' from 4 events debug: /Stage[main]/Hadoop_head_node/Hadoop::Proxyserver[proxyserver]/Service[hadoop-yarn-proxyserver]: The container Hadoop::Proxyserver[proxyserver] will propagate my refresh event debug: Hadoop::Proxyserver[proxyserver]: The container Class[Hadoop_head_node] will propagate my refresh event debug: Class[Hadoop::Common-yarn]: The container Stage[main] will propagate my refresh event debug: Service[hadoop-hdfs-namenode](provider=redhat): Executing '/sbin/service hadoop-hdfs-namenode status' debug: Service[hadoop-hdfs-namenode](provider=redhat): Executing '/sbin/service hadoop-hdfs-namenode start' err: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[hadoop-hdfs-namenode]/ensure: change from stopped to running failed: Could not start Service[hadoop-hdfs-namenode]: Execution of '/sbin/service hadoop-hdfs-namenode start' returned 3: at /mnt/var/lib/bootstrap-actions/1/bigtop-0.8.0/bigtop-deploy/puppet/modules/hadoop/manifests/init.pp:335 debug: Service[hadoop-hdfs-namenode](provider=redhat): Executing '/sbin/service hadoop-hdfs-namenode status' debug: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[hadoop-hdfs-namenode]: Skipping restart; service is not running notice: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[hadoop-hdfs-namenode]: Triggered 'refresh' from 4 events debug: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Service[hadoop-hdfs-namenode]: The container Hadoop::Namenode[namenode] will propagate my refresh event debug: Hadoop::Namenode[namenode]: The container Class[Hadoop_head_node] will propagate my refresh event notice: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/Package[hadoop-hdfs-datanode]: Dependency Service[hadoop-hdfs-namenode] has failures: true warning: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/Package[hadoop-hdfs-datanode]: Skipping because of failed dependencies notice: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/File[/mnt1/hdfs]: Dependency Service[hadoop-hdfs-namenode] has failures: true warning: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/File[/mnt1/hdfs]: Skipping because of failed dependencies notice: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/File[/mnt/hdfs]: Dependency Service[hadoop-hdfs-namenode] has failures: true The interesting part is that if I query namenode status eventually the namenode will show up as started even though I have not taken any other actions: [hadoop@ip-10-168-87-216 1]$ sudo /sbin/service hadoop-hdfs-namenode status Hadoop namenode is not running [FAILED] [hadoop@ip-10-168-87-216 1]$ sudo /sbin/service hadoop-hdfs-namenode status Hadoop namenode is not running [FAILED] [hadoop@ip-10-168-87-216 1]$ sudo /sbin/service hadoop-hdfs-namenode status Hadoop namenode is running [ OK ] The same goes for proxy server. The problem here is that all other dependencies of namenode do not install (such as resource manager, etc). I am using the latest release of AmazonLinux (2014.09) and this has puppet 2.7.25-1. I am not sure what to do about this issue, has anyone else experienced something like this? Should I just move to puppet 3.x and only try to install out of the Bigtop trunk (0.9.0)?
