Title:  HOWTO-FreeBSD-on-XCP
Author: John D. "Trix" Farrar
Date:   2012-02-09

* The Challenge - ParaVirtualized FreeBSD 9.0-RELEASE under XCP 1.1.0

The idea here is to create a PV FreeBSD VM under Xen Cloud Platform
(XCP) without starting with an HVM first.  The documentation I''ve
been able to find on-line is at least a year old and is mostly written
for Xen (under Linux or NetBSD) rather than XCP.  To make things more
interesting, the hardware on which I have XCP installed doesn''t
support Hardware Virtualization -- so I can''t just install an HVM and
convert it over. A PV under XCP means using pygrub as a bootloader to
load the FreeBSD kernel.  It seems that pygrub doesn''t know enough
about BSD disk slices and partitions to be able to find the kernel on
its own, so it will need a patch

This solution isn''t perfect, but I think it''s a good start.

* The Equipment

Yes, my hardware is old/cheap.

  * Xen Cloud Platform server - XCP v1.1.0

  * Build Server - FreeBSD 9.0-RELEASE (32-bit)

  * NFS Server - anything w/ enough space.  Can be the Build
    Server. (should NOT be the XCP Server)

* The Process

  * Patching pygrub

    Adrian Chadd (http://wiki.freebsd.org/AdrianChadd/XenHackery) did
    a great job laying the groundwork for a Xen (not XCP)
    installation.  I also got a lot of help from Adian''s patch to
    pygrub at http://people.freebsd.org/~adrian/xen/bsd_pygrub/.  The
    problem is that Adrian''s patch "is against the pygrub shipped with
    the xen-3.0.3-80.el5_3.2 CentOS 5.3 package."  Using Adrian''s
    patch, I wsa able to create a patch against the pygrub shipped
    with XCP v1.1.0:

Patch attached.
  * Creating a disk image

    I mounted an exported filesystem from the NFS server so that I
    would have a large, sharable space in which to place the disk
    image.  This space is also mounted on the XCP server.  In both
    cases, the NFS filesystem is mounted under /mnt.

    The following steps are taken on the BUILD host where /usr/src has
    already been compiled with buiidworld and buildkernel targets.

export FSIMAGE="FreeBSD-PV.img"

truncate -s 10G /mnt/${FSIMAGE}              # Create the file where the image 
will reside.
mdconfig -a -t vnode -f /mnt/${FSIMAGE} -u 0 # Create the image itself
fdisk -BI md0                                # Write the DOS partition table

# Create an inital BSD Label 
# For testing, we'll assume one large partition
bsdlabel -w -B md0s1
bsdlabel -e md0s1 # Change partition a from 'unused' to '4.2BSD'
newfs -O1 /dev/md0s1a # UFS, because pygrub doesn't grok UFS2

# Mount up the image and do the installation
mount /dev/md0s1a /media

# Perform the installation to the disk image.
cd /usr/src
make -s DESTDIR=/media KERNCONF=XEN installworld
make -s DESTDIR=/media KERNCONF=XEN installkernel
make -s DESTDIR=/media KERNCONF=XEN distribution

# Set up the ttys(5) file so the VM will use Xen's console.
cat >>/media/etc/ttys <<EOF

# Xen Console
xc0     "/usr/libexec/getty Pc"         vt100   on  secure


# Create fstab(5) so the root (only) filesystem will mount.
cat >>/media/etc/fstab <<EOF
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/xbd0s1a            /               ufs     rw              1       1


# Start and rc.conf(5) file
# NOTE: You won't be able to login via SSH until users are added.
cat >>/media/etc/rc.conf <<EOF

# BUILD Server - Unmount the filesystem, the image is done.
umount /media
mdconfig -d -u 0

The following section is the script I actually used to create a VM
that is more-or-less realistic as a template.


truncate -s 20G /mnt/${FSIMAGE}
mdconfig -a -t vnode -f /mnt/${FSIMAGE} -u 0
fdisk -BI md0
bsdlabel -w -B md0s1
cat >>xcp-optionb-bsdlabel.txt <<EOF
# /dev/md0s1:
8 partitions:
#          size     offset    fstype   [fsize bsize bps/cpg]
  a:      512MB         63    4.2BSD        0     0     0
  b:      512MB          *      swap                    
  c:   41929587          0    unused        0     0     # "raw" part, don't edit
  d:      512MB          *    4.2BSD        0     0     0
  e:    2097152    3145791    4.2BSD        0     0     0
  f:    6291456    5242943    4.2BSD        0     0     0
  g:   12582912   11534399    4.2BSD        0     0     0
  h:   17812276   24117311    4.2BSD        0     0     0


bsdlabel -R md0s1 xcp-optionb-bsdlabel.txt
rm xcp-optionb-bsdlabel.txt

# root partition has to be UFS1.  PyGRUB chokes on UFS2 (fsimage.ufs, actually)
newfs -O1 /dev/md0s1a && mount -v /dev/md0s1a /media
newfs     /dev/md0s1d && mkdir /media/tmp       && mount -v /dev/md0s1d 
newfs     /dev/md0s1e && mkdir /media/var       && mount -v /dev/md0s1e 
newfs     /dev/md0s1f && mkdir /media/usr       && mount -v /dev/md0s1f 
newfs     /dev/md0s1g && mkdir /media/usr/local && mount -v /dev/md0s1g 
newfs     /dev/md0s1h && mkdir /media/home      && mount -v /dev/md0s1h 
pushd /usr/src
make -s DESTDIR=/media KERNCONF=XEN installworld installkernel distribution
cat >>/media/etc/ttys <<EOF

# Xen Console
xc0     "/usr/libexec/getty Pc"         vt100   on  secure


cat >>/media/etc/fstab <<EOF
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/xbd0s1a            /               ufs     rw              1       1
/dev/xbd0s1f            /usr            ufs     rw              1       1
/dev/xbd0s1g            /usr/local      ufs     rw              1       1
/dev/xbd0s1d            /tmp            ufs     rw              1       1
/dev/xbd0s1e            /var            ufs     rw              1       1
/dev/xbd0s1h            /home           ufs     rw              1       1


cat >>/media/etc/rc.conf <<EOF

umount -v /media/home /media/usr/local /media/usr /media/var /media/tmp /media


  * Create the Virtual Host

    I got the bulk of this information from Grant McWilliams'' HOWTO
    for installing CentOS on an XCP host 

    All of the next steps are to be carried on at the Local Command
    Shell prompt on the XCP host.

VMNAME="FreeBSD-9.0"     # Edit this value, each VM must be different

# XCP Server - Create the Virtual Disk Image and copy .img file into it.
export SR_UUID=$(xe sr-list name-label="Local storage" --minimal)
export VDIUUID=$(xe vdi-create \
                    sr-uuid=${SR_UUID} \
                    name-label=${VMNAME}-vdi \
                    type=system \
                    virtual-size=11GiB )

# Pull the built image into a VDI (Virtual Disk Image)
# This operatoin will take a while. (like an hour and a half)
xe vdi-import \
    uuid=${VDIUUID} \

# Create a VM from the "Other install media" template
# Then customize to our taste
export TPLUUID=$(xe template-list \
                    name-label="Other install media" \
                    params=uuid --minimal)

# Create the VM itself
export VM_UUID=$(xe vm-install \
                    new-name-label=${VMNAME} \

# Customize
# Defaults:
# Desc.  Value    VM Parameter(s)
# RAM  = 256 MB = memory-static-max, memory-dynamic-max, memory-dynamic-min
#        128 MB = memory-static-min
# CPUs =   1    = VCPUs-at-startup, VCPUs-max
xe vm-param-set   uuid=${VM_UUID} name-description="New FreeBSD host"
xe vm-param-clear uuid=${VM_UUID} param-name=HVM-boot-policy
xe vm-param-clear uuid=${VM_UUID} param-name=HVM-boot-params
xe vm-param-set   uuid=${VM_UUID} PV-bootloader=pygrub
xe vm-param-set   uuid=${VM_UUID} 
xe vm-param-set   uuid=${VM_UUID} other-config:disable_pv_vnc=1
## xe vm-param-set   uuid=${VM_UUID} other-config:mac_seed=''

# Create the Network Interface

SWITCH="xenbr0"          # 
# The vendor prefix 00:16:3e is assigned to Xen in oid.txt
MAC="00:16:3e:12:34:56"  # Edit this value, each VIF must be different
## MAC="random"          # if you prefer randomly assigned addresses

# Get an ID for the virtual switch.  
# It is assumed that this is on the first bridge.
export NETUUID=$(xe network-list bridge=${SWITCH} --minimal)

export VIFUUID=$(xe vif-create \
                    vm-uuid=${VM_UUID} \
                    network-uuid=${NETUUID} \
                    mac=${MAC} \

# Create the Virtual Block Device that links the
# VDI to the VM
export VBDUUID=$(xe vbd-create \
                    vm-uuid=${VM_UUID} \
                    device=0 \
                    vdi-uuid=${VDIUUID} \
                    type=Disk \
                    mode=RW \
                    bootable=true )


  * Booting it up

# Starting the first time takes a while
xe vm-start uuid=${VM_UUID}
DOMID=$(xe vm-list uuid=${VM_UUID} params=dom-id --minimal)
/usr/lib/xen/bin/xenconsole ${DOMID}

# The main difference on the XCP side is that the disk image is
bigger.  The other steps are the same.  I threw in a condenced line to
start the VM and fork the console just because it's easier to paste
into a terminal window.

VDIUUID=$(xe vdi-create \
             sr-uuid=${SR_UUID} \
             name-label=${VMNAME}-vdi \
             type=system \
             virtual-size=21474836480 )

xe vm-start uuid=${VM_UUID} && \
    /usr/lib/xen/bin/xenconsole $(xe vm-list uuid=${VM_UUID} params=dom-id 

* Open Issues

I'm probably missing something really obvious here, but here are a
couple of issues that keep this from being a complete win.

  * Kernel ignoring/not-getting arguments from the domain builder.
    This means that the kernel can''t find its root device.

  * The pygrub loader won''t execute /boot/loader, so the kernel has to
    be called directly.  No access to loader.conf(5) and the
    customization it affords.

John D. "Trix" Farrar               __\\|//__               Basement.NET
t...@basement.net                   (` o-o ')   http://www.basement.net/
   GPG Key Fprint: 525F DBA7 1A62 E4C4 E642  DF95 384B B851 3CEF C10A
--- pygrub.xcp-orig	2012-02-09 10:39:29.000000000 -0600
+++ pygrub.xcp-ufs-debug	2012-02-09 10:41:44.000000000 -0600
@@ -12,8 +12,13 @@
 # along with this program; if not, write to the Free Software
 # Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+# 2012-02-08 - John D. "Trix" Farrar - 0.6.1-bbkt
+#  Using XCF version of pygrub, incorporated patches from:
+#    http://people.freebsd.org/~adrian/xen/bsd_pygrub/
+#  which seem to be based on an older version of "0.6"
+#  Also included a patch to add a --debug switch for easier troubleshooting.
-import os, sys, atexit, string, struct, tempfile, re
+import os, sys, atexit, string, struct, tempfile, re, traceback
 import copy
 import logging
 import platform
@@ -26,7 +31,7 @@
 import grub.LiloConf
 import grub.ExtLinuxConf
+PYGRUB_VER = "0.6.1bbkt"
 def enable_cursor(ison):
     if ison:
@@ -74,6 +79,105 @@
     raise RuntimeError, "No root slice found"      
+# Find the root slice in a FreeBSD disklabel.
+# Some magic constants for the BSD disklabel
+# Magic offsets in the label
+BSD_LABEL_OFFSET_MAGIC=0                        # u_int32_t
+BSD_LABEL_OFFSET_SECTORSIZE=40                  # u_int32_t - number of bytes per sector
+BSD_LABEL_OFFSET_NPARTITIONS=138                # u_int16_t
+# Magic offsets in the -partition- section
+BSD_LABEL_PARTITION_SIZE=0                      # u_int32_t - size in sectors
+BSD_LABEL_PARTITION_OFFSET=4                    # u_int32_t - offset in sectors
+BSD_LABEL_PARTITION_FSTYPE=12                   # u_int8_t - partition type
+# How big is the partition structure?
+# BSD label partition types that we care about
+# Get the partition size from the given freebsd disklabel
+# partbuf contains the beginning of the disklabel
+def get_freebsd_partition_size(partbuf, partid):
+    return struct.unpack("<I", partbuf[i+BSD_LABEL_PARTITION_SIZE:i+BSD_LABEL_PARTITION_SIZE+4])[0]
+def get_freebsd_partition_offset(partbuf, partid):
+    return struct.unpack("<I", partbuf[i+BSD_LABEL_PARTITION_OFFSET:i+BSD_LABEL_PARTITION_OFFSET+4])[0]
+def get_freebsd_partition_fstype(partbuf, partid):
+    return struct.unpack("<B", partbuf[i+BSD_LABEL_PARTITION_FSTYPE:i+BSD_LABEL_PARTITION_FSTYPE+1])[0]
+# The FreeBSD disklabel can either be dangerously dedicated or inside a DOS
+# partition. This function takes "offset" to mean the beginning of the FreeBSD
+# disk label and searches for a root partition "a" in the given slice.
+# The slice must have the FS type set to something that we know about - this
+# is currently "BSDFFS" (type 7). Anything else will fail.
+# The filesystem must be at the beginning of that returned offset for the FS library
+# to discover and handle.
+# The returned offset is relative to the passed in offset - ie the beginning of
+# the "disklabel". This is so this code can be used regardless of the underlying
+# partitioning scheme (BSD FFS direct on disk, BSD FFS in a DOS primary partition,
+# etc.)
+def get_freebsd_slice(file, offset):
+    """Find the root slice in a FreeBSD disklabel."""
+    print >>sys.stderr, "get_freebsd_slice: file  :  %s" % file
+    print >>sys.stderr, "get_freebsd_slice: offset:  %s" % offset
+    fd = os.open(file, os.O_RDONLY)
+    buf = os.read(fd, 1024)
+    # Check the magic - is it a valid disklabel? if not, barf
+    bsd_d_magic = struct.unpack("<I", buf[BSD_LABEL_OFFSET_MAGIC:BSD_LABEL_OFFSET_MAGIC+4])[0]
+    print >>sys.stderr, "bsd magic:    %s" % bsd_d_magic
+    print >>sys.stderr, "needed magic: %s" % BSD_LABEL_DISKMAGIC
+    if bsd_d_magic <> BSD_LABEL_DISKMAGIC:
+        raise RuntimeError, "FreeBSD label diskmagic incorrect"
+    # Get sectorsize - does it make sense? If not, barf
+    bsd_d_secsize = struct.unpack("<I", buf[BSD_LABEL_OFFSET_SECTORSIZE:BSD_LABEL_OFFSET_SECTORSIZE+4])[0];
+    print >>sys.stderr, "bsd sector size:  %s" % bsd_d_secsize
+    # Get the first partition - is it the right type? If not, barf
+    bsd_d_npartitions = struct.unpack("<H", buf[BSD_LABEL_OFFSET_NPARTITIONS:BSD_LABEL_OFFSET_NPARTITIONS+2])[0];
+    print >>sys.stderr, "bsd number of partitions:  %s" % bsd_d_npartitions
+    # Grab the filesystem offset in sectors
+    bsd_d_part_offset = get_freebsd_partition_offset(buf, 0)
+    bsd_d_part_size = get_freebsd_partition_size(buf, 0)
+    bsd_d_part_fstype = get_freebsd_partition_fstype(buf, 0)
+    print >>sys.stderr, "bsd partition 0 offset:  %s" % bsd_d_part_offset
+    print >>sys.stderr, "bsd partition 0 size  :  %s" % bsd_d_part_size
+    print >>sys.stderr, "bsd partition 0 type  :  %s" % bsd_d_part_fstype
+    if bsd_d_part_fstype <> BSD_FS_BSDFFS:
+        raise RuntimeError, "First BSD partition is not type BSDFFS (7)"
+    # XXX one should now verify that the given partition size and offset fall within the disk size..
+    # Calculate the filesystem offset, return that
+    return (bsd_d_part_offset * bsd_d_secsize)
 def get_fs_offset_gpt(file):
     fd = os.open(file, os.O_RDONLY)
     # assume the first partition is an EFI system partition.
@@ -81,9 +185,11 @@
     buf = os.read(fd, 512)
     return struct.unpack("<Q", buf[32:40])[0] * SECTOR_SIZE
+# Partition Types
 def get_partition_offsets(file):
     if not is_disk_image(file):
@@ -112,6 +218,13 @@
             except RuntimeError:
                 continue # no solaris magic at that offset, ignore partition
+        if type == FDISK_PART_FREEBSD:
+            try:
+                offset += get_freebsd_slice(file, offset)
+            except RuntimeError:
+                print >>sys.stderr, "BAIL!  I don't know what to do!"
+                sys.exit(1)
         if type == FDISK_PART_GPT:
             offset = get_fs_offset_gpt(file)
@@ -620,7 +733,7 @@
                                    ["quiet", "interactive", "not-really", 
                                     "help", "output=", "entry=", "kernel=", 
                                     "ramdisk=", "args=", "default_args=", 
-                                    "extra_args=", "vm="])
+                                    "extra_args=", "vm=", "debug"])
     except getopt.GetoptError:
@@ -633,6 +746,7 @@
     output = None
     entry = None
     interactive = True
+    debug = False
     not_really = False
     default_args = ""
     extra_args = ""
@@ -654,6 +768,8 @@
         elif o in ("-n", "--not-really"):
             not_really = True
+        elif o in ("--debug"):
+            debug = True
         elif o in ("--output",):
             output = a
         elif o in ("--kernel",):
@@ -671,6 +787,9 @@
         elif o in ("--extra_args",):
             extra_args = a
+    if debug:
+        logging.basicConfig(level=logging.DEBUG)
     if output is None or output == "-":
         fd = sys.stdout.fileno()
@@ -700,6 +819,11 @@
                 # IOErrors raised by fsimage.open
                 # RuntimeErrors raised by run_grub if no menu.lst present
+                if debug:
+                    traceback.print_exc() 
+                    print >>sys.stderr, "Filename: %s" % file
+                    print >>sys.stderr, "Offset  : %s" % offset
+                    print >>sys.stderr, "Options : %s" % bootfsoptions
                 fs = None
@@ -738,6 +862,8 @@
                 # IOErrors raised by fsimage.open
                 # RuntimeErrors raised by run_grub if no menu.lst present
+                if debug:
+                    traceback.print_exc() 
                 fs = None
Version: GnuPG v2.0.18 (FreeBSD)

freebsd-xen@freebsd.org mailing list
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Reply via email to