On 3 August 2015 at 12:37, Andrew Stuart <[email protected]> wrote: > Any feedback from anyone? > > If not, what should I do from here - add it to the wiki, put it somewhere on > github? There’s been a few updates but largely its intact.
github please... I am just looking at this today and copying scripts out of emails is not very easy, and I dont really know which bits need customising. There do appear to be ways to create images without using an EC2 machine to do it. I am doing a tutorial in a couple of weeks and thinking I will use EC2 as its perhaps the most accessible, although havent decided yet. Justin > as > > > On 26 Jul 2015, at 6:02 pm, Andrew Stuart <[email protected]> > wrote: > > I’m interested to get feedback please on this. Any thoughts and opinions > valued. I’m not precious about it so no need to tiptoe around - rip it apart > if you want - it’s my first shell script larger than 3 lines so any > improvement ideas welcome. > > This is a script for getting a rump kernel to work on EC2. Start point was > the MiraegOS script for doing a similar thing but this is substantially > modified. > > You must modify rumprun/platform/xen/callmain.c before building rumprun > (which is done with ./build-rr.sh xen) and include hardcoded_jsoncfg. Here > is an example: > > static char hardcoded_jsoncfg[] = "{\"cmdline\": \"hiawatha_baked.xen -d -c > /data/conf\", \"net\": {\"cloner\": \"true\", \"type\": \"inet\", \"method\": > \"dhcp\", \"if\": \"xenif0\"}, \"blk\": {\"fstype\": \"ext2fs\", \"source\": > \"etfs\", \"path\": \"blk0\", \"mountpoint\": \"/\"}}”; > > After rumprun is built, you must then do all the normal steps for building a > rump kernel and bake it with something like: > > rumpbake xen_pv bin/hiawatha_baked.xen bin/hiawatha > > (Incidentally that syntax seems reversed where output target filename comes > first?) > > To run the script below you must > 1: launch an EC2 instance > 2: copy the script onto that instance > 3: ensure you have set up the ec2 cli > http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/set-up-ec2-cli-linux.html > On Ubuntu its pretty easy: > sudo apt-add-repository ppa:awstools-dev/awstools > sudo apt-get update > sudo apt-get install ec2-api-tools > > ensure your environment has the AWS keys set up > export AWS_ACCESS_KEY=XXXXXXXXXXX > export AWS_SECRET_KEY=YYYYYYYYYYYYY > > > In addition to creating an EC2 AMI for the unikernel, this script creates and > attaches an ext2 formatted block device. The script copies files onto that > block device so that your rump kernel is able to access them. You tell the > script which directory to copy onto the block device via the -p > /filesystempath parameter. > > The script must be run from an EC2 instance and cannot run outside EC2. > > For now, it creates a unikernel in the EC2 availability zone that the > instance is on that executes the script. I’ll change that at some point sp > it can built for other availability zones. > > A final important note is that right now the launched unikernel doesn’t work > as there’s an issues currently being resolved I suspect with vbd mapping but > waiting to hear ideas on how to resolve it from this list. > > #!/bin/bash > > # todo need to give snapshots a name > # todo need to finalise block mapping for AMI when its working > > # this script is copied from the Mirage-OS project and modified for rump > kernels. > # IMPORTANT: script must be run from an EC2 instance - it won't work outside > EC2 > #set -x > SUDO=sudo > UNIKERNELMOUNTPOINT=/mnt/unikernel > FILESYSTEMMOUNTPOINT=/mnt/filesystem > > # the commands in this script must be run in the same availability zone and > region that the EC2 instance is running on > THISINSTANCEID=`wget -q -O - > http://instance-data/latest/meta-data/instance-id` > THISREGION=`wget -q -O - > http://instance-data/latest/dynamic/instance-identity/document | awk > '/region/ {gsub(/[",]/, "", $3); print $3}'` > THISAVAILABILITYZONE=`wget -q -O - > http://instance-data/latest/dynamic/instance-identity/document | awk > '/availabilityZone/ {gsub(/[",]/, "", $3); print $3}'` > > # http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/UserProvidedKernels.html > case "${THISREGION}" in > ap-northeast-1) > KERNELID=aki-176bf516 > ;; > > ap-southeast-1) > KERNELID=aki-503e7402 > ;; > > ap-southeast-2) > KERNELID=aki-c362fff9 > ;; > > eu-central-1) > KERNELID=aki-184c7a05 > ;; > > eu-west-1) > KERNELID=aki-52a34525 > ;; > > sa-east-1) > KERNELID=aki-5553f448 > ;; > > us-east-1) > KERNELID=aki-919dcaf8 > ;; > > us-gov-west-1) > KERNELID=aki-1de98d3e > ;; > > us-west-1) > KERNELID=aki-880531cd > ;; > > us-west-2) > KERNELID=aki-fc8f11cc > ;; > > *) > echo $"Error selecting pvgrub kernel for region" > exit 1 > esac > > ## TODO - how to handle when the target region is not the current region? > Needs thinking. > > NAME=unikernelfilename > while getopts "h:p:n:r:k:" arg; do > case $arg in > h) > echo "usage: $0 [-h] [-n <name>] [-r <region>] -k <unikernel> -p > <filesystempath> " > echo "usage: $0 [-h] [-n <name>] [-r <region>] -k <unikernel> -p > <filesystempath> " > echo "" > echo "<filesystempath>: Directory path to copy to block file system and > attach to unikernel" > echo "<unikernel>: Name of the kernel file (e.g. rump.xen)" > echo "<name>: the application name to use (default: ${NAME})" > echo "<region>: the EC2 region to register AMI in (default: ${THISREGION})" > > echo To run this script you will need the Amazon command line tools > installed from here: > echo > http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/set-up-ec2-cli-linux.html > echo "" > echo Remember to set each of the following environment variables in your > echo environment before running this script: > echo AWS_ACCESS_KEY > echo AWS_SECRET_KEY > exit 1 ;; > p) FILESYSTEMPATH=$OPTARG ;; > n) NAME=$OPTARG ;; > r) THISREGION=$OPTARG ;; > k) UNIKERNELFILE=$OPTARG ;; > esac > done > > if [ ! -e "${FILESYSTEMPATH}" ]; then > echo "Must specify a file system path file with the [-p] flag." > echo "Run '$0 -h' for full option list." > exit 1 > fi > > if [ ! -e "${UNIKERNELFILE}" ]; then > echo "Must specify a unikernel file with the [-k] flag." > echo "Run '$0 -h' for full option list." > exit 1 > fi > > # Make name unique to avoid registration clashes > NAME=${NAME}-`date +"%d-%b-%Y-%s"` > > echo Name : ${NAME} > echo THISREGION: ${THISREGION} > echo THISINSTANCEID: ${THISINSTANCEID} > echo THISAVAILABILITYZONE: ${THISAVAILABILITYZONE} > echo UNIKERNELMOUNTPOINT: ${UNIKERNELMOUNTPOINT} > echo FILESYSTEMMOUNTPOINT: ${FILESYSTEMMOUNTPOINT} > echo UNIKERNELFILE: ${UNIKERNELFILE} > echo FILESYSTEMPATH: ${FILESYSTEMPATH} > > ########################################################################################## > ########################################################################################## > ###### ask EC2 what block devices are attached to this instance. > ###### unmount and detach the block devices in case they are still attached > for some reason > ########################################################################################## > ########################################################################################## > > BLKDEVICEXVDK=`ec2-describe-instance-attribute ${THISINSTANCEID} > --block-device-mapping --region ${THISREGION} | awk '/xvdk/ {print $3}'` > if [ -e "${BLKDEVICEXVDK}" ]; then > echo BLKDEVICEXVDK: ${BLKDEVICEXVDK} > ${SUDO} umount ${UNIKERNELMOUNTPOINT} > echo Waiting for unmount to complete...... > sleep 5 > ec2-detach-volume ${BLKDEVICEXVDK} --region ${THISREGION} > fi > > BLKDEVICEXVDK=`ec2-describe-instance-attribute ${THISINSTANCEID} > --block-device-mapping --region ${THISREGION} | awk '/xvdl/ {print $3}'` > if [ -e "${BLKDEVICEXVDL}" ]; then > echo BLKDEVICEXVDK: ${BLKDEVICEXVDL} > ${SUDO} umount ${UNIKERNELMOUNTPOINT} > echo Waiting for unmount to complete...... > sleep 5 > ec2-detach-volume ${BLKDEVICEXVDL} --region ${THISREGION} > fi > > ########################################################################################## > ########################################################################################## > ###### prepare block device > ###### this will be the root file system for the unikernel > ########################################################################################## > ########################################################################################## > > # create a 1 GB EBS volume using the AWS console > # > http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-CreateVolume.html > UNIKERNELVOLUMEID=`ec2-create-volume --availability-zone > ${THISAVAILABILITYZONE} --region ${THISREGION} -s 1 | awk '{print $2}'` > > # wait for EC2 to get its act together > echo Waiting for create volume to complete...... > sleep 10 > > # attach the EBS volume > # > http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-AttachVolume.html > ec2-attach-volume ${UNIKERNELVOLUMEID} --region ${THISREGION} --instance > ${THISINSTANCEID} --device /dev/xvdk > > echo Waiting for attach volume to complete...... > sleep 10 > > # unmount any existing volume at the UNIKERNELMOUNTPOINT > ${SUDO} umount ${UNIKERNELMOUNTPOINT} > > # create the UNIKERNELMOUNTPOINT > ${SUDO} mkdir -p ${UNIKERNELMOUNTPOINT} > > # format the EBS volume as ext2 > ${SUDO} mkfs -t ext4 /dev/xvdk > > #Label the disk. AWS has an unofficial tutorial that does not include this > step. > ${SUDO} tune2fs -L '/' /dev/xvdk > > # mount the device > ${SUDO} mount /dev/xvdk ${UNIKERNELMOUNTPOINT} > > # set permissions > ${SUDO} chmod -R ug+rwx ${UNIKERNELMOUNTPOINT} > > > ${SUDO} mkdir -p ${UNIKERNELMOUNTPOINT}/boot/grub > echo default 0 > menu.lst > echo timeout 1 >> menu.lst > echo title Rump >> menu.lst > echo " root (hd0)" >> menu.lst > echo " kernel /boot/rump-os.gz" >> menu.lst > ${SUDO} mv menu.lst ${UNIKERNELMOUNTPOINT}/boot/grub/menu.lst > ${SUDO} gzip -c ${UNIKERNELFILE} > ./rump-os.gz > ${SUDO} mv ./rump-os.gz ${UNIKERNELMOUNTPOINT}/boot/. > > # show what is in the target > ${SUDO} find ${UNIKERNELMOUNTPOINT} > > # unmount any existing volume at the UNIKERNELMOUNTPOINT > ${SUDO} umount ${UNIKERNELMOUNTPOINT} > > # detach the EBS volume > # > http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DetachVolume.html > ec2-detach-volume ${UNIKERNELVOLUMEID} --region ${THISREGION} > > > ########################################################################################## > ########################################################################################## > ###### prepare block device > ###### this block device will contain the unikernel code, and is booted by EC2 > ########################################################################################## > ########################################################################################## > > # create a 1 GB EBS volume using the AWS console > # > http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-CreateVolume.html > FILESYSTEMVOLUMEID=`ec2-create-volume --availability-zone > ${THISAVAILABILITYZONE} \ > --region ${THISREGION} -s 1 | awk > '{print $2}'` > echo Waiting for create volume to complete...... > sleep 5 > > # attach the EBS volume > # > http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-AttachVolume.html > ec2-attach-volume ${FILESYSTEMVOLUMEID} --region ${THISREGION} --instance > ${THISINSTANCEID} --device /dev/xvdl > echo Waiting for attach volume to complete...... > sleep 5 > > # unmount any existing volume at the UNIKERNELMOUNTPOINT > ${SUDO} umount ${FILESYSTEMMOUNTPOINT} > > # create the UNIKERNELMOUNTPOINT > ${SUDO} mkdir -p ${FILESYSTEMMOUNTPOINT} > > # format the EBS volume as ext2 > ${SUDO} mke2fs /dev/xvdl > > # mount the device > ${SUDO} mount /dev/xvdl ${FILESYSTEMMOUNTPOINT} > > # set permissions > ${SUDO} chmod -R ugo+rwx ${FILESYSTEMMOUNTPOINT} > > # copy into the EBS volume the files that will live on the block device > ${SUDO} cp -r ${FILESYSTEMPATH} ${FILESYSTEMMOUNTPOINT} > > # show what is in the target > ${SUDO} find ${FILESYSTEMMOUNTPOINT} > > # unmount any existing volume at the mount point > ${SUDO} umount ${FILESYSTEMMOUNTPOINT} > > # detach the EBS volume > # > http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DetachVolume.html > ec2-detach-volume ${FILESYSTEMVOLUMEID} --region ${THISREGION} > > # make a snapshot of the filesystem block volume > # -- block devices must be attached to AMI’s from snapshots > FILESYSTEMSNAPSHOTID=`ec2-create-snapshot --description 'unikernel filesystem > volume' \ > --region ${THISREGION} ${FILESYSTEMVOLUMEID} | > awk '{print $2}'` > > > > ########################################################################################## > ###### prepare the unikernel for booting on EC2 > ########################################################################################## > > # make a snapshot of the unikernel root block volume > # -- AMI’s cannot be created from volumes, only from snapshots > UNIKERNELSNAPSHOTID=`ec2-create-snapshot --description 'unikernel boot > volume' \ > --region ${THISREGION} ${UNIKERNELVOLUMEID} | > awk '{print $2}'` > > # Create image/AMI from the snapshot > # > http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-CreateImage.html > ## HAVING TROUBLE? COULD IT BE [--root-device-name name] > echo Waiting for snapshot to complete...... > sleep 20 > > # -b "/dev/sdf=${FILESYSTEMSNAPSHOTID}" \ > > > AMIID=`ec2-register --name "${NAME}" \ > --description "${NAME}" \ > -a x86_64 \ > -s ${UNIKERNELSNAPSHOTID} \ > --region ${THISREGION} \ > --kernel ${KERNELID} \ > --virtualization-type paravirtual \ > | awk '{print $2}'` > > > ########################################################################################## > ###### unmount and detach the block devices in case they are still attached > for some reason > ########################################################################################## > > BLKDEVICEXVDK=`ec2-describe-instance-attribute ${THISINSTANCEID} > --block-device-mapping --region ${THISREGION} | awk '/xvdk/ {print $3}'` > if [ -e "${BLKDEVICEXVDK}" ]; then > echo BLKDEVICEXVDK: ${BLKDEVICEXVDK} > ${SUDO} umount ${UNIKERNELMOUNTPOINT} > echo Waiting for unmount to complete...... > sleep 5 > ec2-detach-volume ${BLKDEVICEXVDK} --region ${THISREGION} > fi > > BLKDEVICEXVDK=`ec2-describe-instance-attribute ${THISINSTANCEID} > --block-device-mapping --region ${THISREGION} | awk '/xvdl/ {print $3}'` > if [ -e "${BLKDEVICEXVDL}" ]; then > echo BLKDEVICEXVDK: ${BLKDEVICEXVDL} > ${SUDO} umount ${UNIKERNELMOUNTPOINT} > echo Waiting for unmount to complete...... > sleep 5 > ec2-detach-volume ${BLKDEVICEXVDL} --region ${THISREGION} > fi > > > ########################################################################################## > ###### finish > ########################################################################################## > > echo You can now start this instance via: > echo ec2-run-instances --region ${THISREGION} ${AMIID} > echo "" > echo Don\'t forget to customise this with a security group, as the > echo default one won\'t let any inbound traffic in. > > > > >
