Hi Quynh Well, you can do the following
- Create a master node template and slave node template, make their configuration such that they keep that relationship, one way ssh key etc - Deploy the master node and configure the software on it then deploy as many slave nodes to connect back to the master node. - In our configuration, we deploy M identical nodes, then we pick one of the nodes as master and install the master node software (cloudera manager in our case). - Then we use cloudera manager to deploy the rest of the nodes, in our case this includes one hdfs name node, one job tracker, (M-3) hdfs data nodes and (M-3) map reduce task trackers. - We have deployed around 50 VMs within a 30 min period using this configuration. Shank On Fri, Jul 13, 2012 at 2:58 AM, Quynh Le <[email protected]> wrote: > Hello Shankhadeep, > > Thank you for your information. I am able to setup such a virtual cluster > using another cloud middleware like OpenNebula so I can understand the > situation. What I wanna make clear is: > - This is a kind of Master/Slave cluster: 1 head node and N worker nodes. > - We can launch a group of VMs to make N+1 VMs for the cluster. > - Then, do you have to setup hadoop master node and worker nodes manually, > OR are they (VMs) automatically configured to be "1 master + N workers". > - In this case, how many VM images you use? 1 VM image for master node, 1 > for worker nodes, or 1 for all? > > I'm looking forward to your sharing. > > Cheers, > Quynh > > ------------------------------ > *From:* Shankhadeep Shome <[email protected]> > *To:* Quynh Le <[email protected]> > *Cc:* "[email protected]" <[email protected]> > *Sent:* Friday, July 13, 2012 12:22 PM > *Subject:* Re: [one-users] MPI Cluster on OpenNebula > > Yes, clusters are particularly easy on this environment, I work with our > analytics group on hadoop clusters. The most important thing to do is > install good parallel computing utilities like pdsh and have ssh > equivalency setup, then creating a cluster of machine sis realyl easy > because you get N machines configured the same with full access to each > other. Then you use ssh based scripts to distribute your software. In our > environment we use Cloudera manager to distribute and start and manage our > hadoop cluster, there should be something similar for mpi clusters. The > nice thing about open nebula is that you can distribute your resource > allocation over generic cpu, memory and disk resources and hardware > optimization is easy because you can customize one template and redeploy > the entire cluster. > > On Thu, Jul 12, 2012 at 3:51 AM, Quynh Le <[email protected]> wrote: > > Hello, > > I intend to setup a ready-to-go MPI Cluster on OpenNebula from a VM image. > Is it possible? Can each VM know its role (master or slave), as well as > other VM, to fill in the mpdhost file? I don't know much about context yet. > > Thanks for any explanation. > > Best regards, > Quynh > > _______________________________________________ > Users mailing list > [email protected] > http://lists.opennebula.org/listinfo.cgi/users-opennebula.org > > > > >
_______________________________________________ Users mailing list [email protected] http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
