Konstantin, So I should be safer using CentOS 6?
Also - was I right to include "hadoop" in site.yaml components? It's not listed in the basic site.yaml, but I am guessing it is not installed automatically? -David On Thu, Mar 17, 2016 at 7:17 AM, David Starina <[email protected]> wrote: > That's right, I'm using CentOS 7. > > --David > > > On Thursday, 17 March 2016, Konstantin Boudnik <[email protected]> wrote: > >> Hmm... that's plain nuts... Looks like latest ignite is pulling in some >> crap >> like osgi... OMG, WTF? >> >> Thanks for discovering this David! Somehow it never got stepped upon in my >> tests. Are you on centos7? >> >> Cos >> >> On Wed, Mar 16, 2016 at 11:39PM, David Starina wrote: >> > Hi, >> > >> > Roman, Evans, thank you for the tips - that helped a lot. Still have >> some >> > problems though, an error when running puppet apply ... >> > >> > This is a sample of errors I got: >> > >> > >> > *Debug: Executing '/bin/yum -d 0 -e 0 -y install >> > ignite-hadoop-service'Error: Could not update: Execution of '/bin/yum >> -d 0 >> > -e 0 -y install ignite-hadoop-service' returned 1: Error: Package: >> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop) >> Requires: >> > osgi(org.apache.servicemix.bundles.spring-beans)Error: Package: >> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop) >> Requires: >> > osgi(org.apache.servicemix.bundles.spring-context-support)Error: >> Package: >> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop) >> Requires: >> > osgi(org.apache.servicemix.bundles.spring-jdbc)Error: Package: >> > ignite-hadoop-1.5.0.final-1.el7.centos.noarch (Bigtop) >> Requires: >> > osgi(org.apache.servicemix.bundles.spring-core)* >> > ..... >> > >> > Also, after that: >> > >> > >> > >> > >> > >> > >> > *Notice: >> > >> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/default/ignite-hadoop]: >> > Dependency Package[ignite-hadoop] has failures: trueWarning: >> > >> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/default/ignite-hadoop]: >> > Skipping because of failed dependenciesNotice: >> > /Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]: Dependency >> > Package[openjdk-7-jre-headless] has failures: trueWarning: >> > /Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]: Skipping >> > because of failed dependenciesNotice: >> > /Stage[main]/Hadoop::Nodemanager/Package[hadoop-yarn-nodemanager]: >> > Dependency Package[openjdk-7-jre-headless] has failures: trueWarning: >> > /Stage[main]/Hadoop::Nodemanager/Package[hadoop-yarn-nodemanager]: >> Skipping >> > because of failed dependenciesNotice: >> > >> /Stage[main]/Ignite_hadoop::Deploy/Ignite_hadoop::Server[ignite-hadoop-node]/File[/etc/ignite-hadoop/conf/default-config.xml]: >> > Dependency Package[ignite-hadoop] has failures: true* >> > ..... >> > >> > >> > This is the "site.yaml" I used ("hadoop" wasn't listed in components, >> but I >> > think it should be there?) >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > *bigtop::hadoop_head_node: "NameNode"hadoop::hadoop_storage_dirs: - >> > /data/1hadoop_cluster_node::cluster_components: - hadoop - >> ignite_hadoop >> > - hue - mahout - yarn - zookeeperbigtop::jdk_package_name: >> > "openjdk-7-jre-headless"bigtop::bigtop_repo_uri: >> > "http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64 >> > <http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64>"* >> > >> > >> > >> > >> > Any idea what I am doing wrong? >> > >> > --David >> > >> > On Tue, Mar 15, 2016 at 6:57 PM, Evans Ye <[email protected]> wrote: >> > >> > > David >> > > >> > > Most of the time you need to run same command on all nodes. "pdsh" >> would >> > > be a good tool to use. >> > > >> > > 1. You need to have puppet agent installed on every node. yum -y >> install >> > > puppet will do the trick, or you need to download repo, or use >> > > bigtop/bigtop_toolchain/bin/puppetize.sh directly. There's no deploy >> effort >> > > other than just install the package. >> > > >> > > 2. Masterless means each puppet agent takes commands and runs >> > > independently. We DONT use puppet master, which is a centralized >> master >> > > daemon to dispatch works to all the registered puppet agent. >> > > >> > > 3. Next, you need to copy bigtop puppet recipes and bigtop hiera >> > > configuration to all nodes, or at least make them available for all >> nodes >> > > via nfs. >> > > >> > > 4. Finally, execute the puppet apply command on all nodes and the >> puppet >> > > recipes will automatically figure out it's role. There no need to >> deploy >> > > head node first and then slaves. >> > > >> > > Ask if you still have questions. >> > > >> > > >> > > Thanks >> > > >> > > Evans >> > > 2016年3月15日 下午4:10於 "David Starina" <[email protected]>寫道： >> > > >> > >> Hi guys, >> > >> >> > >> me again. I haven't set up Puppet before, can you just point me in >> the >> > >> right direction? I know this is not a Puppet mailing list, but I bet >> you >> > >> have some tips for me :-) >> > >> >> > >> So what do I need to do? Install Puppet on every node? What does it >> mean >> > >> that the mode of Puppet is "*masterless"* - do I just have to do a >> basic >> > >> installation of Puppet and not care much about setting it up, and >> then >> > >> deploy bigtop on every node separately? And when setting the Hadoop >> head >> > >> node - does that mean that I need to deploy Bigtop on this node >> first, and >> > >> Bigtop will know this is the master node by the url or IP I provide >> to it? >> > >> And then deploy Bigtop via Puppet on all the other nodes and they >> will >> > >> connect to the head node (which is the NameNode, I suppose?). >> > >> >> > >> I can do this via trial and error, but if you guys can help me >> reduce the >> > >> "error" part, that will be great :) Sorry for such beginner >> questions, I >> > >> haven't used Puppet before ... The reason I'm asking is, I got >> access to a >> > >> "physical" (actually virtual, but it's the same) environment for a >> limited >> > >> amount of time (two weeks), and I need to start testing on it asap - >> and >> > >> since my day job takes a lot of my time, if I wanted to do this >> without >> > >> help it could take me two weeks just to set up the environment - but >> I need >> > >> to install it, which is not the central part of my project, and then >> run >> > >> some ML algorithms on a text corpus, which will be the main part of >> the >> > >> project ... >> > >> >> > >> --David >> > >> >> > >> On Thu, Mar 10, 2016 at 11:28 AM, David Starina < >> [email protected]> >> > >> wrote: >> > >> >> > >>> Thanks for the info! >> > >>> >> > >>> --David >> > >>> >> > >>> On Tue, Mar 8, 2016 at 7:15 PM, Jay Vyas < >> [email protected]> >> > >>> wrote: >> > >>> >> > >>>> I think there is enough info on youtube that you can cobble >> together a >> > >>>> syllabus if you need structured learning. >> > >>>> >> > >>>> See cos/Romans Bigtop talks which are on YouTube and then my Bigtop >> > >>>> internals hackathon video. That gives you both the high and low >> level >> > >>>> details to do whatever you want. >> > >>>> >> > >>>> > On Mar 8, 2016, at 1:11 PM, Konstantin Boudnik <[email protected]> >> > >>>> wrote: >> > >>>> > >> > >>>> > We also had this discussion to start producing "The Big Book of >> Apache >> > >>>> > Bigtop" like HBase does, but never got into the materialization >> part >> > >>>> of that >> > >>>> > project. >> > >>>> > >> > >>>> > Cos >> > >>>> > >> > >>>> >> On Tue, Mar 08, 2016 at 11:44PM, Evans Ye wrote: >> > >>>> >> Hi David, >> > >>>> >> >> > >>>> >> If you just want to take Bigtop binaries to deploy a cluster, >> you >> > >>>> can refer >> > >>>> >> to our puppet management recipes: >> > >>>> >> >> https://github.com/apache/bigtop/tree/master/bigtop-deploy/puppet >> > >>>> >> >> > >>>> >> BTW bigtop provisioner is the best source to reference for how >> bigtop >> > >>>> >> provision a cluster: >> > >>>> >> >> > >>>> >> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Provisioner+User+Guide >> > >>>> >> >> > >>>> >> Please ask us if you find anything confusing. >> > >>>> >> >> > >>>> >> >> > >>>> >> Evans >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> 2016-03-08 0:48 GMT+08:00 David Starina < >> [email protected]>: >> > >>>> >> >> > >>>> >>> Hi, >> > >>>> >>> >> > >>>> >>> Is there some version of Bigtop User Guide available? I know it >> > >>>> used to be >> > >>>> >>> on the be on http://bigtop.apache.org/ under the "User Guide" >> > >>>> link, but >> > >>>> >>> there's nothing there anymore ... >> > >>>> >>> >> > >>>> >>> I need some guide on how to install Bigtop on a physical >> cluster >> > >>>> ... It >> > >>>> >>> was easy to create a virtual cluster on my machine using >> Vagrant, >> > >>>> but I >> > >>>> >>> cannot find a manual to install on a physical cluster ... >> > >>>> >>> >> > >>>> >>> Cheers, >> > >>>> >>> David >> > >>>> >>> >> > >>>> >> > >>> >> > >>> >> > >> >> >
