> On 2012-04-14 23:27:22, Charles Reiss wrote: > > I'd prefer we avoid this approach where we need to discover the VM IP and > > SSH into it. Can we instead pass boot options and have a startup script on > > the VM launch something that reads a protobuf contact address out of the > > boot options and retrieves a (protobuf-format) description of what to > > launch? [This should also avoid fragility from generating a shell script.] > > > > On a lower priority, there are some hard-coded paths and some use of > > mesos_home that won't work; we need to handle (a) out-of-source builds and > > (b) installation to "standard" paths ($(prefix)/bin > > $(prefix)/libexec/mesos, etc.). > > > >
Yes, I agree that hard coded paths should be dispensed with and will address those shortly. If I understand the main point, a better solution is to pass the VM a fixed IP address as a boot parameter. Using this address the VM can then provide its address and also receive launch configuration. This eliminates the need to scp the shell script and also eliminates need for find_addr.pl etc. - Charles ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4717/#review6925 ----------------------------------------------------------- On 2012-04-14 03:57:12, Charles Earl wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/4717/ > ----------------------------------------------------------- > > (Updated 2012-04-14 03:57:12) > > > Review request for mesos, Benjamin Hindman and Matei Zaharia. > > > Summary > ------- > > Earlier in the year I implemented a virtual machine isolation module. This > module uses lib-virt to launch and manage virtual machine containers. The > code is still rough and have done basic testing with the Spark example. > > This code works with the KVM (http://www.linux-kvm.org/page/Main_Page) > virtual machine manager. I've placed the relevant code in a branch called > mesos-vm, for now located at https://github.com/charlescearl/VirtualMesos. > The code is based upon the mesos lxc isolation module that is located in > src/slave/lxc_isolation_module.cpp/.hpp. My code based on the mesos master > branch dated Wed Nov 23 12:02:07 2011 -0800, commit > 059aabb2ec5bd7b20ed08ab9c439531a352ba3ec. I've included a patch for the > relevant code included for the review. Suggestions appreciated on whether > this is the appropriate branch/commit to patch against. > > Most of the implementation is contained in vm_isolation_module.cpp and > vm_isolation_module.hpp and there are some minor additions in launcher to > handle setup of the environment for the virtual machine. I use the libvirt > (http://libvirt.org/) library, to manage the virtual machine container in > which the jobs are executed. > > Dependencies > The code has been tested on Ubuntu 11.04 and 11.10 and depends on > libpython2.6 and libvirt0 > > Configuration of the virtual machine container > The virtual machine invocation depends upon a few configuration assumptions: > 1. ssh public keys installed on the container. I assume that the container > is setup to allow password-less secure access. > 2. Directory structure on the container matches the servant machine. For > example, in invoking a spark executor, assume that the paths match the setup > on the container host. > > Running it > In the $MESOS_HOME/conf/mesos.conf file add the line > isolation=vm > to use the virtual machine isolation. > > The Mesos slave is invoked with the isolation parameter set to vm. For > example > sudo bin/mesos-slave -m mesos://master@mesos-host:5050 -w 9839 > --isolation=vm > > Rough description of how it works > > The `vm_isolation_module` class forks a process that in turn launches a > virtual machine. A routine located in bin called find_addr.pl is responsible > for figuring out the IP address of the launched virtual machine. This is > probably not portable since it is explicitly looking for entry in the virbr0 > network. > > A script vmLauncherTemplate.sh located in bin assists the the vmLauncher > method to setup the environment for launching tasks inside of the virtual > machine. The vmLauncher method uses vmLauncherTemplate.sh to create a tasks > specific shell vmLauncherTemplate-<task_id>.sh, which is copied to the > running guest and used to run the executor inside the VM. This communicates > with the slave on the host. > > Comments and suggestions on improvements and next directions are appreciated! > > > Diffs > ----- > > bin/find_addr.pl PRE-CREATION > bin/killtree.sh PRE-CREATION > bin/vmLauncher.sh PRE-CREATION > bin/vmLauncherTemplate.sh PRE-CREATION > src/config/config.hpp PRE-CREATION > src/launcher/launcher.hpp b99b6d2 > src/launcher/launcher.cpp 4422224 > src/launcher/vm_mesos_launcher.cpp PRE-CREATION > src/slave/isolation_module.cpp 5b7b4a2 > src/slave/isolation_module_factory.cpp 6498945 > src/slave/lxc_isolation_module.cpp ab0843a > src/slave/main.cpp 9519ed2 > src/slave/slave.cpp 21fc9f2 > src/slave/vm_isolation_module.hpp PRE-CREATION > src/slave/vm_isolation_module.cpp PRE-CREATION > > Diff: https://reviews.apache.org/r/4717/diff > > > Testing > ------- > > This was run with the spark example on single KVM virtual machine. Not tested > extensively. > > > Thanks, > > Charles > >
