Ok, here we go... I start from an almost vanilla RHEL5.2 machine, except for the kernel. RHEL does not provide the 9p modules, so rather than trying to recompile their kernel I just got the 2.6.26 kernel from kernel.org. This allows me to build sxcpu right out of the box. I obtained it from the sourceforge svn repository:
svn co https://xcpu.svn.sourceforge.net/svnroot/xcpu/sxcpu/trunk sxcpu Here you simply do "make; make install" and it should all be available. It will be important to run the "statfs" daemon on the master, so that the commands you use later are aware of the status of the compute nodes in the cluster. These will need to be configured in the /etc/xcpu/statfs.conf file: [r...@dgk3 xcpu]# cat /etc/xcpu/statfs.conf n0000=tcp!10.10.0.10!6667 n0001=tcp!10.10.0.11!6667 See below for more details on the assignment of IP addresses to the nodes by perceus. Then the perceus side of things: I have perceus 1.4.4, downloaded directly from their site. To build it I just did the usual ./configure; make; make install with no special options. Now to the perceus configuration... My internal network to the compute nodes is eth0. Here is the ifconfig for my master node: [r...@dgk3 all]# ifconfig eth0 Link encap:Ethernet HWaddr 00:E0:81:2C:81:D0 inet addr:10.10.0.1 Bcast:10.10.0.255 Mask:255.255.255.0 inet6 addr: fe80::2e0:81ff:fe2c:81d0/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8044504 errors:0 dropped:0 overruns:0 frame:0 TX packets:10719515 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1770711038 (1.6 GiB) TX bytes:1542820770 (1.4 GiB) Interrupt:24 eth1 Link encap:Ethernet HWaddr 00:E0:81:2C:81:D1 inet addr:142.150.227.13 Bcast:142.150.227.255 Mask:255.255.252.0 inet6 addr: fec0::9:2e0:81ff:fe2c:81d1/64 Scope:Site inet6 addr: 2002:8e96:e1cc:9:2e0:81ff:fe2c:81d1/64 Scope:Global inet6 addr: fe80::2e0:81ff:fe2c:81d1/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:44696410 errors:0 dropped:0 overruns:0 frame:0 TX packets:1158903 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4665982604 (4.3 GiB) TX bytes:1197487595 (1.1 GiB) Interrupt:25 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:21339 errors:0 dropped:0 overruns:0 frame:0 TX packets:21339 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:66081636 (63.0 MiB) TX bytes:66081636 (63.0 MiB) In /etc/perceus there are several configuration files: ---defaults.conf--- [r...@dgk3 perceus]# cat defaults.conf # # Copyright (c) 2006-2008, Greg M. Kurtzer, Arthur A. Stevens and # Infiscale, Inc. All rights reserved # # This is the template name for all new nodes as they are configured. # Define the node name range. The '#' characters symbolize the node number # in the order of initalized. If you don't allocate enough number spaces # here for what you defined in 'Total Nodes' then it will be automatically # padded. Node Name = n#### # What is the default group for new nodes (this doesn't have to exist # anywhere before hand) Group Name = cluster # Define the default VNFS image that should be assigned to new nodes Vnfs Name = # Are new nodes automatically enabled and provisionined? Enabled = 1 # What is the first node number that we should count at? First Node = 0 # This is the total node count that Perceus would ever try and allocate a # node to. It is safe to make this big, so you should leave it big. Total Nodes = 10000 (I did not modify the defaults.conf file). ---dnsmasq.conf--- [r...@dgk3 perceus]# cat dnsmasq.conf interface=eth0 enable-tftp tftp-root=/usr/local/var/lib/perceus//tftp dhcp-option=vendor:Etherboot,60,"Etherboot" dhcp-boot=pxelinux.0 local=// domain=internal expand-hosts dhcp-range=10.10.0.128,10.10.0.254 dhcp-lease-max=21600 read-ethers ---perceus.conf--- [r...@dgk3 perceus]# cat perceus.conf # # Copyright (c) 2006-2008, Greg M. Kurtzer, Arthur A. Stevens and # Infiscale, Inc. All rights reserved # # This is the primary configuration file for Perceus # Define the network device on this system that is connected directly # and privately to the nodes. This device will be responding to DHCP # requests thus make sure you specify the proper device name! # note: This device must be configured for IP based communication. master network device = eth0 # What protocol should be used to retireve the VNFS information. Generally # Supported options in this version of Perceus are: 'xget', 'nfs', and 'http' # but others may also be available via specialized VNFS capsules or # feature enhancing Perceus Modules. vnfs transfer method = xget # Define the IP Address of the network file server. This address must be # set before Perceus can operate. If this option is left blank, the IP # address of the "master network device" defined above will be used. vnfs transfer master = # Define the VNFS transfer location if it is different from the default # ('statedir'). This gets used differently for different transfer methods # (e.g. NFS this replaces the path to statedir, while with http it is gets # prepended to the "/perceus" path). vnfs transfer prefix = # What is the default database that should be used. If this option is not # specified, then the default is "hash" to remain compatible with # previous versions of Perceus. Other options are 'btree' and 'mysql'. # note: btree is default as of version 1.4. database type = btree # If you selected an SQL database solution as your database type above, # then you will need to specify the SQL user login information here. # note: this will be ignored for non-SQL database types. database server = localhost database name = perceus database user = db user database pass = db pass # To allow for better scaling the Perceus daemon 'preforks' which creates # multiple subprocesses to better handle large number of simultaneous # connections. The default is 4 which on most systems can support # thousands of nodes per minute but for best tuning this number is highly # dependant on system configuration (both hardware and software). prefork = 4 # How long (in seconds) should we wait before considering a node as dead. # Note, that if you are not running node client daemons, then after # provisioning the node will never check in, and will no doubt expire. # Considering that the default node check in is 5 minutes, setting this # to double that should ensure that any living node would have checked in # by then (600). node timeout = 600 I only modified the master network device to point to eth0. Note that there are no VNFS images defined, as booting xcpu does not require them. Install the perceus startup script in /etc/rc.d/init.d, so that it will start on boot. I believe it gets installed by default (in the "make install" step), but it still needs to be configured with "chkconfig -a perceus". This will start the perceus daemons, including the dnsmasq which provides dhcp for the slave nodes. No other dhcp server can run, but this one can be configured to provide other network configuration for additional NICs. After rebooting, make sure perceus is running. Then run the command: perceus module activate xcpu perceus module activate ipaddr In order to get static addreses assigned to the compute nodes (desireable), their addresses must be added to the /etc/hosts file, e.g.: 10.10.0.1 master 10.10.0.10 n0000 10.10.0.11 n0001 You should be ready to start configuring the nodes at this stage. They must be set for pxe boot. All the necessary stuff for this is installed by perceus in /usr/local/var/lib/perceus. You boot your compute nodes in the order in which you want them named, starting, by default, as n0000. The first time they will be assigned an IP address from the dynamic range defined in the /etc/perceus/dnsmasq.conf file, but on reboot they will get the statically assigned address from the /etc/hosts file. By this stage you should have a useable xcpu cluster. You need to set up the groups and users using the xgroupset and xuserset commands. In order to get "proper" behaviour, in accordance with the version of sxcpu that you downloaded and built, you may need to update the xcpufs provided by perceus. This is done by statically linking the xcpufs daemon: In /usr/local/src/sxcpu/xcpufs (or wherever you installed the sxcpu sources) there is a script called LINKSTATIC. I am running on x86_64, so I modified it to read: [r...@dgk3 xcpufs]# cat LINKSTATIC #!/bin/sh echo This script is for linking statically on Linux. cc -static -o xcpufs.static -Wall -g -I ../include -DSYSNAME=Linux file.o pipe.o proc-Linux.o tspawn.o ufs.o xauth.o xcpufs.o -g -L../libstrutil -lstrutil -L../libspclient -lspclient -L../libspfs -lspfs -L../libxauth -lxauth -lcrypto /usr/lib64/libdl.a Note that it produces the "xcpufs.static" executable, and it looks for its libdl.a library in the /usr/lib64 directory. Then I copy the xcpufs.static executable to the location where perceus needs it: cp /usr/local/src/sxcpu/xcpufs/xcpufs.static /usr/local/var/lib/perceus/modules/xcpu/xcpufs and on reboot the nodes will pick up the latest and greatest xcpufs. After this you can do, for example: xgroupset add -a -u xuserset add -a -u in order to add all the groups and all the users to the permitted user list on the nodes. You can then run anything on the nodes, e.g. "xrx -a date". Needless to say, this requires that the "statfs" daemon be running. You can verify this with the "xstat" command (see the configuration instructions for this above). Now, I don't know anything about IB, mainly because I have never had access to an IB-connected cluster. I have no idea if perceus can manage pxe booting over IB, but I suspect that if you have IP over IB then it should, for all intents and purposes, look like just another network interface to it (I could be utterly wrong on this, of course...). However, if you need to configure a second interface, say for access to a fileserver on a separate network, then all you need to do is change the /etc/perceus/dnsmasq.conf and define the machines in there. Again, for static IP addresses on the second interface they need to be defined in /etc/hosts. Let me know if you would like details on how I did this. I then mounted my fileserver on the compute nodes by modifying the perceus xcpu startup script in /etc/perceus/nodescripts/init/all/05-xcpu.sh, so that the node gets a mount point and executes the nfs mount. Please let me know if/how this works for you. I hope it is complete... Best regards, Daniel p.s. Please feel free to modify this blurb and add it to the xcpu installation instructions. Greg from the perceus group is extremely helpful with any perceus issues. On Fri, Dec 12, 2008 at 3:41 PM, Chris Kinney <[email protected]> wrote: > Hey Daniel, > > My name is Chris Kinney, I'm Ron's intern. I was wondering if you could > show me how you're booting your perceus setup. We're in need of perceus > being able to work with IB and from what we've seen, the capsules just don't > work with it. What ever you got that can help that would be great! Thanks > again! > > -Chris > > ron minnich wrote: >> >> On Thu, Dec 11, 2008 at 5:48 PM, Daniel Gruner <[email protected]> wrote: >> >>> >>> Yeah, you don't even need a VNFS image in order to boot into xcpu! >>> All you need is the initial busybox provided by perceus and activating >>> the perceus xcpu module "perceus module activate xcpu". This will >>> give you a minimal xcpu node, and you can then add remote filesystems >>> for mounting if necessary. You only really need user files, if at >>> all, since the executables that you run with xrx take the necessary >>> libraries along automagically. >>> >>> >> >> daniel, this is an excellent point, and since we are having a terrible >> time getting our vnfs capsules to work with ib ... >> >> can you give us a quick writeup for how you set this up so we can use it >> too. >> >> thanks >> >> ron >> >> > >
