> On 2012-04-14 23:27:22, Charles Reiss wrote: > > I'd prefer we avoid this approach where we need to discover the VM IP and > > SSH into it. Can we instead pass boot options and have a startup script on > > the VM launch something that reads a protobuf contact address out of the > > boot options and retrieves a (protobuf-format) description of what to > > launch? [This should also avoid fragility from generating a shell script.] > > > > On a lower priority, there are some hard-coded paths and some use of > > mesos_home that won't work; we need to handle (a) out-of-source builds and > > (b) installation to "standard" paths ($(prefix)/bin > > $(prefix)/libexec/mesos, etc.). > > > > > > Charles Earl wrote: > Yes, I agree that hard coded paths should be dispensed with and will > address those shortly. If I understand the main point, a better solution is > to pass the VM a fixed IP address as a boot parameter. Using this address the > VM can then provide its address and also receive launch configuration. This > eliminates the need to scp the shell script and also eliminates need for > find_addr.pl etc. > > Charles Earl wrote: > Further, I would assume that mesos.conf is where configuration options > should (e.g. "phone home" address, options for the VM) placed.
What probably should be passed is the stringified form of a process::UPID for a process::Process<XXX> subclass managed by the isolation module along with the framework+executor ID (or an equivalent token). The VM's init scripts will then extract this from /proc/cmdline (for Linux VMs) and run some (new?) wrapper similar to mesos-launcher to phone home. - Charles ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4717/#review6925 ----------------------------------------------------------- On 2012-04-14 03:57:12, Charles Earl wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/4717/ > ----------------------------------------------------------- > > (Updated 2012-04-14 03:57:12) > > > Review request for mesos, Benjamin Hindman and Matei Zaharia. > > > Summary > ------- > > Earlier in the year I implemented a virtual machine isolation module. This > module uses lib-virt to launch and manage virtual machine containers. The > code is still rough and have done basic testing with the Spark example. > > This code works with the KVM (http://www.linux-kvm.org/page/Main_Page) > virtual machine manager. I've placed the relevant code in a branch called > mesos-vm, for now located at https://github.com/charlescearl/VirtualMesos. > The code is based upon the mesos lxc isolation module that is located in > src/slave/lxc_isolation_module.cpp/.hpp. My code based on the mesos master > branch dated Wed Nov 23 12:02:07 2011 -0800, commit > 059aabb2ec5bd7b20ed08ab9c439531a352ba3ec. I've included a patch for the > relevant code included for the review. Suggestions appreciated on whether > this is the appropriate branch/commit to patch against. > > Most of the implementation is contained in vm_isolation_module.cpp and > vm_isolation_module.hpp and there are some minor additions in launcher to > handle setup of the environment for the virtual machine. I use the libvirt > (http://libvirt.org/) library, to manage the virtual machine container in > which the jobs are executed. > > Dependencies > The code has been tested on Ubuntu 11.04 and 11.10 and depends on > libpython2.6 and libvirt0 > > Configuration of the virtual machine container > The virtual machine invocation depends upon a few configuration assumptions: > 1. ssh public keys installed on the container. I assume that the container > is setup to allow password-less secure access. > 2. Directory structure on the container matches the servant machine. For > example, in invoking a spark executor, assume that the paths match the setup > on the container host. > > Running it > In the $MESOS_HOME/conf/mesos.conf file add the line > isolation=vm > to use the virtual machine isolation. > > The Mesos slave is invoked with the isolation parameter set to vm. For > example > sudo bin/mesos-slave -m mesos://master@mesos-host:5050 -w 9839 > --isolation=vm > > Rough description of how it works > > The `vm_isolation_module` class forks a process that in turn launches a > virtual machine. A routine located in bin called find_addr.pl is responsible > for figuring out the IP address of the launched virtual machine. This is > probably not portable since it is explicitly looking for entry in the virbr0 > network. > > A script vmLauncherTemplate.sh located in bin assists the the vmLauncher > method to setup the environment for launching tasks inside of the virtual > machine. The vmLauncher method uses vmLauncherTemplate.sh to create a tasks > specific shell vmLauncherTemplate-<task_id>.sh, which is copied to the > running guest and used to run the executor inside the VM. This communicates > with the slave on the host. > > Comments and suggestions on improvements and next directions are appreciated! > > > Diffs > ----- > > bin/find_addr.pl PRE-CREATION > bin/killtree.sh PRE-CREATION > bin/vmLauncher.sh PRE-CREATION > bin/vmLauncherTemplate.sh PRE-CREATION > src/config/config.hpp PRE-CREATION > src/launcher/launcher.hpp b99b6d2 > src/launcher/launcher.cpp 4422224 > src/launcher/vm_mesos_launcher.cpp PRE-CREATION > src/slave/isolation_module.cpp 5b7b4a2 > src/slave/isolation_module_factory.cpp 6498945 > src/slave/lxc_isolation_module.cpp ab0843a > src/slave/main.cpp 9519ed2 > src/slave/slave.cpp 21fc9f2 > src/slave/vm_isolation_module.hpp PRE-CREATION > src/slave/vm_isolation_module.cpp PRE-CREATION > > Diff: https://reviews.apache.org/r/4717/diff > > > Testing > ------- > > This was run with the spark example on single KVM virtual machine. Not tested > extensively. > > > Thanks, > > Charles > >
