also, not sure if i put this link up for you yet, but here : https://www.youtube.com/watch?v=4GfcKEjO6e8&feature=youtu.be . its a bigtop internals explanation which shows some info about how the vagrant and puppet stuff works. you will be ready to hack bigtop w/ the best of em after watching that :)
On Mon, Jun 8, 2015 at 8:02 PM, Demai Ni <[email protected]> wrote: > Evans, > > thank you very much for the detail instructions. > > I understood that the cluster need to be passwordless ssh or someway to > provide userid/password of all nodes. So that actions can be propogated to > other nodes. > > I was expecting some tool (pdsh, vagrant provisioner) as you mentioned, > would do such job for end-users. But the document didn't mention such > tool(or I missed them?), and I got the impression that that steps 1)~3) are > required by manual effort. > > Maybe those tools are common-sense for an experienced cluster administer, > so no document is necessary.:-) Anyway, now that with your help, I will try > them out. Thanks > > Demai > > On Sat, Jun 6, 2015 at 12:03 PM, Evans Ye <[email protected]> wrote: > >> The missing note in the document is that the cluster admin need to setup >> some general environment settings such as >> the password-less ssh. Definitely it would be great if we can provide some >> suggestions or ways to do that, but I think somehow you still need to setup >> that during machine provisioning(installing OS, setting up hostname, >> ip, password, ssh keys), right? >> >> If you have bunch of machines ready for the deployment, here's the step >> to setup a bigtop cluster: >> >> 1) install puppet on each node >> 2) install puppet modules on each node >> >> puppet apply --modulepath= -e "include bigtop_toolchain::puppet-modules" >> >> 2) propagate puppet recipes and configurations >> <https://github.com/apache/bigtop/tree/master/bigtop-deploy/puppet>on >> each node >> 3) run puppet apply on each node >> 4) done! >> >> The puppet and module installation can be done through cluster admin tool >> such as pdsh, which allows you to run same command across the whole >> cluster in parallel mode. The puppet recipes or bigtop repo can be put >> in a share storage such as NFS so that it is available for all nodes. After >> all of this are set, you can then run puppet apply on each node through >> pdsh as well. This should give you an up and running bigtop cluster. >> >> If you're not quite sure how to do that, you can also refer to bigtop's >> vagrant >> provisioner >> <https://github.com/apache/bigtop/tree/master/bigtop-deploy/vm/vagrant-puppet-vm>, >> which demonstrate the whole process to setup a bigtop cluster by a click of >> button. >> >> Of course, you can ask questions in mailing list when you encounter any >> issue. We're happy to help. :) >> >> 2015-06-06 6:30 GMT+08:00 Demai Ni <[email protected]>: >> >>> hi, folks, >>> >>> I am looking at BigTop as noncommercial hadoop distribution package, as >>> an alternative of CDH, HDP. And this is my first time to try out bigTop. >>> >>> My expectation was: >>> 1) setup my cluster with passwordless ssh >>> 2) download the necessary bigtop components on one of the nodes. >>> 3) run an installation/configure tool from the node on 2) >>> 4) BigTop will (through puppet) install and configure the whole cluster. >>> >>> the point is that no need of a lot of manual work on every node. >>> >>> But I am following instruction for 0.8.0 and 0.7.0, like >>> >>> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+BigTop+0.7.0+hadoop+on+CentOS+with+puppet >>> >>> there are quite a few steps, (almost all the steps in above link) are >>> required to be performed on each node? >>> >>> so it looks like the human work is significant higher between small(3 >>> node) and large(100+) cluster. Is that right? >>> >>> Thanks. >>> >>> Demai >>> >>> >> >> > -- jay vyas
