Hi,
while I was porting D01 platform
(https://wiki.linaro.org/Boards/D01) to Xen I wrote a small document
trying to describe the step I made and problems I encountered hoping
it could useful to other people. The idea is to put such documentation
in a wiki page or in the docs directory.
Let me know what do you think.
Regards,
Frediano
Porting a new ARM platform to Xen
=================================
ARM platforms are quite different one from each another. This can affect Xen
porting:
- interrupt controller (GIC on ARM) have different version;
- interrupts numbers does not have many standards (beside CPU ones called SGIs);
- every platforms have different addresses for devices;
- devices list can be quite different.
As a start you should read pages [1] and [2] from the wiki.
One concept you have to learn is (at least currently) the device tree (DT for
short, which can be found as source in .dts files in arch/arm/boot/dts Linux
directory, .fdt or .dtb for compiled versions).
Device tree is a specification that allow to describe device configurations. As
device list is not standard DT contains description of all devices from memory
to CPUs, system board, IO devices and so on. Information can be I/O ranges (in
ARM I/O is memory mapped so there are not the port concept you can find on
Intel), interrupts, device type, clock attached and many more. DT specification
are quite extensible and allow for instance to have information like kernel
command line too. You can find quite a lot of information on DT in
Documentation/devicetree/bindings Linux directory (specifically there is an arm
sub-directory).
Another big difference is the boot process. Unfortunately is not still well
defined as Intel one. For instance Xen use multiboot but multiboot is not still
standard on ARM so can be quite painful.
Happily Xen require very few devices in order to work, as stated in [2]:
- GIC;
- generic timers;
- SMMU;
- one UART.
Now, fortunately timers are now quite standard and the interface to use them
use CPU registers.
UART
----
UART is the first device you should implement. The reason is simple, it allows
you to see something from the beginning! Fortunately a lot of boards implement
the standard (or compatible) 8250. At the beginning you should try to make the
"early printk" work. Early printk is initialised very soon (just after few
assembly lines). If you have a driver all you need is to setup memory ranges and
device type and boot Xen.
To make our UART work I just did this change to xen/arch/arm/Rules.mk
(changeset 5ad71a6eefc9b09fe7458ded5b86b69fcc57436c):
EARLY_PRINTK_BAUD := 115200
EARLY_UART_BASE_ADDRESS := 0x7ff80000
endif
+ifeq ($(CONFIG_EARLY_PRINTK), hip04-d01)
+EARLY_PRINTK_INC := 8250
+EARLY_PRINTK_BAUD := 115200
+EARLY_UART_BASE_ADDRESS := 0xE4007000
+EARLY_UART_REG_SHIFT := 2
+endif
ifneq ($(EARLY_PRINTK_INC),)
EARLY_PRINTK := y
To enable early printk (after implementing the right code), you can have
something like this in your .config:
debug := y
CONFIG_EARLY_PRINTK := hip04-d01
(hip04-d01 is the string decided for our platform).
Early printk devices are implemented in xen/arch/arm/arm32 and
xen/arch/arm/arm64 directories in files like debug-<NAME>.inc.
More documentation on docs/misc/arm/early-printk.txt file.
Xen booting
-----------
As said before starting Xen can be quite tricky. However to start you can focus
on just the hypervisor. If you can boot Linux (and you really should make it
work before trying Xen) you can replace the kernel with the Xen hypervisor
(xen/xen file) and Xen will start. So you can test if serial and CPUs are
working.
Now that boot loader is configured and UART is working you should see Xen
booting and try to do something! But without the FDT Xen won't do much.
DTB file
--------
Xen on ARM require a initial FDT that describe your hardware in order to handle
your platform. If you were able to load the FDT while booting the Linux kernel
you can use the same way and same FDT. Another way (which I used) is to embed
the FDT file into Xen. In my .config file I have a line like
CONFIG_DTB_FILE := $(XEN_ROOT)/hip04-d01.dtb
this will embed hip04-d01.dtb file into Xen (in our case the DTB file was
generated compiling Linux while on other cases is already provided by the
firmware).
Now with FDT sorted out you should see Xen doing some more progresses.
Note: if you are using early printk after a while Xen will try to setup the
serial again using the command line. In my case for some reasons at the
beginning this setup caused serial to go silent, the temporary solution if to
not pass a command line to Xen (which the default DT does not provide). Using
this trick I was able to progress some more.
This is the change to add the Xen command line to the DTS I made:
chosen {
linux,initrd-start = <0 0x10d00000>;
linux,initrd-end = <0 0x12500000>;
+ xen,xen-bootargs = "console=dtuart dtuart=serial0";
};
See docs/misc/arm/booting.txt and docs/misc/arm/device-tree/booting.txt for
more information.
Platform
--------
The next thing that probably will fail is the platform setting up. To initialise
SMP you need some code specific to your platform un less your board support
PSCI standard (you should find some psci string on dts files). Happily on
xen/arch/arm/platforms there are many examples. Mainly the code require some
tweak for setting up the additional CPUs. You can skip at the beginning the
restart code but you should sort out processors initialisation. The other CPUs
are started with a SGI (Software Generated Interrupt) but you also need to set
the start entry of the CPUs. This is not standard and require specific code.
To implement mine I looked at Linux sources and changesets for the platform.
GIC
---
Now if your GIC is compatible to standard ones (GICv2 or GICv3) you can just add
your compatible string (it's a string you find in the DT to say the
compatibility of each device) to the list and Xen should initialise and try to
load the kernel.
To add our GIC to the GICv2 list we did this change to
xen/include/asm-arm/gic.h:
#define DT_COMPAT_GIC_400 "arm,gic-400"
#define DT_COMPAT_GIC_CORTEX_A15 "arm,cortex-a15-gic"
#define DT_COMPAT_GIC_CORTEX_A7 "arm,cortex-a7-gic"
+#define DT_COMPAT_GIC_HIP04 "hisilicon,hip04-intc"
#define DT_MATCH_GIC_V2 DT_MATCH_COMPATIBLE(DT_COMPAT_GIC_CORTEX_A15), \
DT_MATCH_COMPATIBLE(DT_COMPAT_GIC_CORTEX_A7), \
- DT_MATCH_COMPATIBLE(DT_COMPAT_GIC_400)
+ DT_MATCH_COMPATIBLE(DT_COMPAT_GIC_400), \
+ DT_MATCH_COMPATIBLE(DT_COMPAT_GIC_HIP04)
#define DT_COMPAT_GIC_V3 "arm,gic-v3"
Unfortunately our GIC was not fully compatible so it required changes to source
(xen/arch/arm/gic-v2.c).
Dom0 kernel
-----------
If you reach this point you put the hypervisor in place of the kernel so where
should Xen take the kernel to setup dom0? Remember: XEN IS NOT A BOOT LOADER. It
has no driver for disk or network so the kernel must be already in memory,
usually loaded by the boot loader. In my case I used another trick replacing the
initrd with the kernel. As the boot loader was not working for me the firmware
take kernel and initrd from specific part of board flash memory. So the trick
was:
- load kernel into initrd section of flash;
- change FDT to point to initrd memory portion (same location used to provide
initrd to kernel are used to provide kernel to xen).
This is the change I made in DTS file to put the Linux kernel in the initrd
partition:
chosen {
- linux,initrd-start = <0 0x10d00000>;
- linux,initrd-end = <0 0x12500000>;
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ xen,xen-bootargs = "console=dtuart dtuart=serial0";
+ xen,dom0-bootargs = "console=hvc0 earlyprintk root=/dev/sda2 rw
loglevel=7";
+ module@0x10d00000 {
+ compatible = "xen,linux-kernel", "xen,multiboot-module";
+ reg = <0x10d00000 0x300000>;
+ };
};
As you can see the start of module@0x10d00000 is the same of old initrd. Also
note that the size of the area is smaller. This as Xen would try to load
everything failing. Just put some value bigger than your kernel but not too
much (0x300000 is 3 MB).
Note that the kernel you boot inside Xen can be different from the kernel
booted on bare metal. As [1] state ("Dom0 kernel" section):
- you must enable Xen in the kernel configuration. See [3] for options to
enable;
- you must disable DTB inclusion if enabled. Xen do some tweak on the DTB in
order to avoid conflicts. If kernel use just bare metal configuration bad
things are going to happen.
Debugging the kernel
--------------------
Checking interrupts. Unfortunately I lost a lot of time (2/3 days) with
interrupts. Linux was not getting interrupts due to a change in GICH registers.
If you have a standard GIC probably you won't have this problem but is useful to
understand the behaviour as interrupts can be very difficult to debug and sort
out. Linux kernel was booting quite fine but suddenly it stopped. Using Xen
console I was able to see register state and CPUs was idle. Before
understanding that interrupts was not working I took a lot of debugging code
and restarts.
In ARM interrupts got mainly dispatched by a single function (code for my
platform was in ./arch/arm/kernel/entry-armv.S). Basically a function registered
with set_handle_irq is called (in my case by gic_init_bases in
drivers/irqchip/irq-gic.c). Putting some debugging code here was the final prove
for interrupts problems.
Once the GIC was fixed dom0 start booting quite fine beside some small issues:
- a small crash of some drivers as not included in the initial DTB, solved
disabling it;
- ethernet driver was not working, solved adding a call to
dma_coerce_mask_and_coherent to set DMA bitmask correctly.
I won't go into details on these part as they are more related to Linux kernel
than to Xen.
References
----------
[1] http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions
[2] http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions_whitepaper
[3] http://wiki.xen.org/wiki/Mainline_Linux_Kernel_Configs
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
