Hi Simon,
On Fri, Oct 7, 2016 at 4:59 AM, Simon Glass <s...@chromium.org> wrote:
> This adds the basic code for binman, including command parsing, processing
> of entries and generation of images.
>
> So far no entry types are supported. These will be added in future commits
> as examples of how to add new types.
>
> See the README for documentation.
>
> Signed-off-by: Simon Glass <s...@chromium.org>
> ---
>
> Changes in v4: None
> Changes in v3: None
> Changes in v2:
> - Add test for code coverage
> - Fix the -b option
> - drop the unused __len__() method
>
> tools/binman/.gitignore | 1 +
> tools/binman/README | 491
> ++++++++++++++++++++++++++++++++++++++++++++
> tools/binman/binman | 1 +
> tools/binman/binman.py | 116 +++++++++++
> tools/binman/cmdline.py | 53 +++++
> tools/binman/control.py | 118 +++++++++++
> tools/binman/etype/entry.py | 190 +++++++++++++++++
> tools/binman/fdt_test.py | 48 +++++
> tools/binman/image.py | 229 +++++++++++++++++++++
> 9 files changed, 1247 insertions(+)
> create mode 100644 tools/binman/.gitignore
> create mode 100644 tools/binman/README
> create mode 120000 tools/binman/binman
> create mode 100755 tools/binman/binman.py
> create mode 100644 tools/binman/cmdline.py
> create mode 100644 tools/binman/control.py
> create mode 100644 tools/binman/etype/entry.py
> create mode 100644 tools/binman/fdt_test.py
> create mode 100644 tools/binman/image.py
>
> diff --git a/tools/binman/.gitignore b/tools/binman/.gitignore
> new file mode 100644
> index 0000000..0d20b64
> --- /dev/null
> +++ b/tools/binman/.gitignore
> @@ -0,0 +1 @@
> +*.pyc
> diff --git a/tools/binman/README b/tools/binman/README
> new file mode 100644
> index 0000000..c73fb3c
> --- /dev/null
> +++ b/tools/binman/README
> @@ -0,0 +1,491 @@
> +# Copyright (c) 2016 Google, Inc
> +#
> +# SPDX-License-Identifier: GPL-2.0+
> +#
> +
> +Introduction
> +------------
> +
> +Firmware often consists of several components which must be packaged
> together.
> +For example, we may have SPL, U-Boot, a device tree and an environment area
> +grouped together and placed in MMC flash. When the system starts, it must be
> +able to find these pieces.
> +
> +So far U-Boot has not provided a way to handle creating such images in a
> +general way. Each SoC does what it needs to build an image, often packing or
> +concatenating images in the U-Boot build system.
> +
> +Binman aims to provide a mechanism for building images, from simple
> +SPL + U-Boot combinations, to more complex arrangements with many parts.
> +
> +
> +What it does
> +------------
> +
> +Binman reads your board's device tree and finds a node which describes the
> +required image layout. It uses this to work out what to place where. The
> +output file normally contains the device tree, so it is in principle possible
> +to read an image and extract its constituent parts.
> +
> +
> +Features
> +--------
> +
> +So far binman is pretty simple. It supports binary blobs, such as 'u-boot',
> +'spl' and 'fdt'. It supports empty entries (such as setting to 0xff). It can
> +place entries at a fixed location in the image, or fit them together with
> +suitable padding and alignment. It provides a way to process binaries before
> +they are included, by adding a Python plug-in. The device tree is available
> +to U-Boot at run-time so that the images can be interpreted.
> +
> +Binman does not yet update the device tree with the final location of
> +everything when it is done. A simple C structure could be generated for
> +constrained environments like SPL (using dtoc) but this is also not
> +implemented.
> +
> +Binman can also support incorporating filesystems in the image if required.
> +For example x86 platforms may use CBFS in some cases.
> +
> +Binman is intended for use with U-Boot but is designed to be general enough
> +to be useful in other image-packaging situations.
> +
> +
> +Motivation
> +----------
> +
> +Packaging of firmware is quite a different task from building the various
> +parts. In many cases the various binaries which go into the image come from
> +separate build systems. For example, ARM Trusted Firmware is used on ARMv8
> +devices but is not built in the U-Boot tree. If a Linux kernel is included
> +in the firmware image, it is built elsewhere.
> +
> +It is of course possible to add more and more build rules to the U-Boot
> +build system to cover these cases. It can shell out to other Makefiles and
> +build scripts. But it seems better to create a clear divide between building
> +software and packaging it.
> +
> +At present this is handled by manual instructions, different for each board,
> +on how to create images that will boot. By turning these instructions into a
> +standard format, we can support making valid images for any board without
> +manual effort, lots of READMEs, etc.
> +
> +Benefits:
> +- Each binary can have its own build system and tool chain without creating
> +any dependencies between them
> +- Avoids the need for a single-shot build: individual parts can be updated
> +and brought in as needed
> +- Provides for a standard image description available in the build and at
> +run-time
> +- SoC-specific image-signing tools can be accomodated
> +- Avoids cluttering the U-Boot build system with image-building code
> +- The image description is automatically available at run-time in U-Boot,
> +SPL. It can be made available to other software also
> +- The image description is easily readable (it's a text file in device-tree
> +format) and permits flexible packing of binaries
> +
> +
> +Terminology
> +-----------
> +
> +Binman uses the following terms:
> +
> +- image - an output file containing a firmware image
> +- binary - an input binary that goes into the image
> +
> +
> +Relationship to FIT
> +-------------------
> +
> +FIT is U-Boot's official image format. It supports multiple binaries with
> +load / execution addresses, compression. It also supports verification
> +through hashing and RSA signatures.
> +
> +FIT was originally designed to support booting a Linux kernel (with an
> +optional ramdisk) and device tree chosen from various options in the FIT.
> +Now that U-Boot supports configuration via device tree, it is possible to
> +load U-Boot from a FIT, with the device tree chosen by SPL.
> +
> +Binman considers FIT to be one of the binaries it can place in the image.
> +
> +Where possible it is best to put as much as possible in the FIT, with binman
> +used to deal with cases not covered by FIT. Examples include initial
> +execution (since FIT itself does not have an executable header) and dealing
> +with device boundaries, such as the read-only/read-write separation in SPI
> +flash.
> +
> +For U-Boot, binman should not be used to create ad-hoc images in place of
> +FIT.
> +
> +
> +Relationship to mkimage
> +-----------------------
> +
> +The mkimage tool provides a means to create a FIT. Traditionally it has
> +needed an image description file: a device tree, like binman, but in a
> +different format. More recently it has started to support a '-f auto' mode
> +which can generate that automatically.
> +
> +More relevant to binman, mkimage also permits creation of many SoC-specific
> +image types. These can be listed by running 'mkimage -T list'. Examples
> +include 'rksd', the Rockchip SD/MMC boot format. The mkimage tool is often
> +called from the U-Boot build system for this reason.
> +
> +Binman considers the output files created by mkimage to be binary blobs
> +which it can place in an image. Binman does not replace the mkimage tool or
> +this purpose. It would be possible in some situtions to create a new entry
> +type for the images in mkimage, but this would not add functionality. It
> +seems better to use the mkiamge tool to generate binaries and avoid blurring
> +the boundaries between building input files (mkimage) and packaging then
then -> them?
> +into a final image (binman).
> +
> +
> +Example use of binman in U-Boot
> +-------------------------------
> +
> +Binman aims to replace some of the ad-hoc image creation in the U-Boot
> +build system.
> +
> +Consider sunxi. It has the following steps:
> +
> +1. It uses a custom mksunxiboot tool to build an SPL image called
> +sunxi-spl.bin. This should probably move into mkimage.
> +
> +2. It uses mkimage to package U-Boot into a legacy image file (so that it can
> +hold the load and execution address) called u-boot.img.
> +
> +3. It builds a final output image called u-boot-sunxi-with-spl.bin which
> +consists of sunxi-spl.bin, some padding and u-boot.img.
> +
> +Binman is intended to replace the last step. The U-Boot build system builds
> +u-boot.bin and sunxi-spl.bin. Binman can then take over creation of
> +sunxi-spl.bin (by calling mksunxiboot, or hopefully one day mkimage). In any
> +case, it would then create the image from the component parts.
> +
> +This simplifies the U-Boot Makefile somewhat, since various pieces of logic
> +can be replaced by a call to binman.
> +
> +
> +Example use of binman for x86
> +-----------------------------
> +
> +In most cases x86 images have a lot of binary blobs, 'black-box' code
> +provided by Intel which must be run for the platform to work. Typically
> +these blobs are not relocatable and must be placed at fixed areas in the
> +firmare image.
> +
> +Currently this is handled by ifdtool, which places microcode, FSP, MRC, VGA
> +BIOS, reference code and Intel ME binaries into a u-boot.rom file.
> +
> +Binman is intended to replace all of this, with ifdtool left to handle only
> +the configuration of the Intel-format descriptor.
> +
> +
> +Running binman
> +--------------
> +
> +Type:
> +
> + binman -b <board_name>
> +
> +to build an image for a board. The board name is the same name used when
> +configuring U-Boot (e.g. for sandbox_defconfig the board name is 'sandbox').
> +Binman assumes that the input files for the build are in ../b/<board_name>.
> +
> +Or you can specify this explicitly:
> +
> + binman -I <build_path>
> +
> +where <build_path> is the build directory containing the output of the U-Boot
> +build.
> +
> +(Future work will make this more configurable)
> +
> +In either case, binman picks up the device tree file (u-boot.dtb) and looks
> +for its instructions in the 'binman' node.
> +
> +Binman has a few other options which you can see by running 'binman -h'.
> +
> +
> +Image description format
> +------------------------
> +
> +The binman node is called 'binman'. An example image description is shown
> +below:
> +
> + binman {
> + filename = "u-boot-sunxi-with-spl.bin";
> + pad-byte = <0xff>;
> + blob {
> + filename = "spl/sunxi-spl.bin";
> + };
> + u-boot {
> + pos = <CONFIG_SPL_PAD_TO>;
> + };
> + };
> +
> +
> +This requests binman to create an image file called u-boot-sunxi-with-spl.bin
> +consisting of a specially formatted SPL (spl/sunxi-spl.bin, built by the
> +normal U-Boot Makefile), some 0xff padding, and a U-Boot legacy image. The
> +padding comes from the fact that the second binary is placed at
> +CONFIG_SPL_PAD_TO. If that line were omitted then the U-Boot binary would
> +immediately follow the SPL binary.
> +
> +The binman node describes an image. The sub-nodes describe entries in the
> +image. Each entry represents a region within the overall image. The name of
> +the entry (blob, u-boot) tells binman what to put there. For 'blob' we must
> +provide a filename. For 'u-boot', binman knows that this means 'u-boot.bin'.
> +
> +Entries are normally placed into the image sequentially, one after the other.
> +The image size is the total size of all entries. As you can see, you can
> +specify the start position of an entry using the 'pos' property.
> +
> +Note that due to a device tree requirement, all entries must have a unique
> +name. If you want to put the same binary in the image multiple times, you can
> +use any unique name, with the 'type' property providing the type.
> +
> +The attributes supported for entries are described below.
> +
> +pos:
> + This sets the position of an entry within the image. The first byte
> + of the image is normally at position 0. If 'pos' is not provided,
> + binman sets it to the end of the previous region, or the start of
> + the image's entry area (normally 0) if there is no previous region.
> +
> +align:
> + This sets the alignment of the entry. The entry position is adjusted
> + so that the entry starts on an aligned boundary within the image. For
> + example 'align = <16>' means that the entry will start on a 16-byte
> + boundary. Alignment shold be a power of 2. If 'align' is not
> + provided, no alignment is performed.
> +
> +size:
> + This sets the size of the entry. The contents will be padded out to
> + this size. If this is not provided, it will be set to the size of the
> + contents.
> +
> +pad-before:
> + Padding before the contents of the entry. Normally this is 0, meaning
> + that the contents start at the beginning of the entry. This can be
> + offset the entry contents a little. Defaults to 0.
> +
> +pad-after:
> + Padding after the contents of the entry. Normally this is 0, meaning
> + that the entry ends at the last byte of content (unless adjusted by
> + other properties). This allows room to be created in the image for
> + this entry to expand later. Defaults to 0.
> +
> +align-size:
> + This sets the alignment of the entry size. For example, to ensure
> + that the size of an entry is a multiple of 64 bytes, set this to 64.
> + If 'align-size' is not provided, no alignment is performed.
> +
> +align-end:
> + This sets the alignment of the end of an entry. Some entries require
> + that they end on an alignment boundary, regardless of where they
> + start. If 'align-end' is not provided, no alignment is performed.
> +
> + Note: This is not yet implemented in binman.
> +
> +filename:
> + For 'blob' types this provides the filename containing the binary to
> + put into the entry. If binman knows about the entry type (like
> + u-boot-bin), then there is no need to specify this.
u-boot.bin
> +
> +type:
> + Sets the type of an entry. This defaults to the entry name, but it is
> + possible to use any name, and then add (for example) 'type = "u-boot"'
> + to specify the type.
> +
> +
> +The attributes supported for images are described below. Several are similar
> +to those for entries.
> +
> +size:
> + Sets the image size in bytes, for example 'size = <0x100000>' for a
> + 1MB image.
> +
> +align-size:
> + This sets the alignment of the image size. For example, to ensure
> + that the image ends on a 512-byte boundary, use 'align-size = <512>'.
> + If 'align-size' is not provided, no alignment is performed.
> +
> +pad-before:
> + This sets the padding before the image entries. The first entry will
> + be positionad after the padding. This defaults to 0.
> +
> +pad-after:
> + This sets the padding after the image entries. The padding will be
> + placed after the last entry. This defaults to 0.
> +
> +pad-byte:
> + This specifies the pad byte to use when padding in the image. It
> + defaults to 0. To use 0xff, you would add 'pad-byte = <0xff>'.
> +
> +filename:
> + This specifies the image filename. It defaults to 'image.bin'.
> +
> +sort-by-pos:
> + This causes binman to reorder the entries as needed to make sure they
> + are in increasing positional order. This can be used when your entry
> + order may not match the positional order. A common situation is where
> + the 'pos' properties are set by CONFIG options, so their ordering is
> + not known a priori.
> +
> + This is a boolean property so needs no value. To enable it, add a
> + line 'sort-by-pos;' to your description.
> +
> +multiple-images:
> + Normally only a single image is generated. To create more than one
> + image, put this property in the binman node. For example, this will
> + create image1.bin containing u-boot.bin, and image2.bin containing
> + both spl/u-boot-spl.bin and u-boot.bin:
> +
> + binman {
> + multiple-images;
> + image1 {
> + u-boot {
> + };
> + };
> +
> + image2 {
> + spl {
> + };
> + u-boot {
> + };
> + };
> + };
> +
> +end-at-4gb:
> + For x86 machines the ROM positions start just before 4GB and extend
> + up so that the image finished at the 4GB boundary. This boolean
> + option can be enabled to support this. The image size must be
> + provided so that binman knows when the image should start. For an
> + 8MB ROM, the position of the first entry would be 0xfff80000 with
> + this option, instead of 0 without this option.
> +
> +
> +Examples of the above options can be found in the tests. See the
> +tools/binman/test directory.
> +
> +
> +Order of image creation
> +-----------------------
> +
> +Image creation proceeds in the following order, for each entry in the image.
> +
> +1. GetEntryContents() - the contents of each entry are obtained, normally by
> +reading from a file. This calls the Entry.ObtainContents() to read the
> +contents. The default version of Entry.ObtainContents() calls
> +Entry.GetDefaultFilename() and then reads that file. So a common mechanism
> +to select a file to read is to override that function in the subclass. The
> +functions must return True when they have read the contents. Binman will
> +retry calling the functions a few times if False is returned, allowing
> +dependencies between the contents of different entries.
> +
> +2. GetEntryPositions() - calls Entry.GetPositions() for each entry. This can
> +return a dict containing entries that need updating. The key should be the
> +entry name and the value is a tuple (pos, size). This allows an entry to
> +provide the position and size for other entries. The default implementation
> +of GetEntryPositions() returns {}.
> +
> +3. PackEntries() - calls Entry.Pack() which figures out the position and
> +size of an entry. The 'current' image position is passed in, and the function
> +returns the position immediately after the entry being packed. The default
> +implementation of Pack() is usually sufficient.
> +
> +4. CheckSize() - checks that the contents of all the entries fits within
> +the image size. If the image does not have a defined size, the size is set
> +large enough to hold all the entries.
> +
> +5. CheckEntries() - checks that the entries do not overlap, nor extend
> +outside the image.
> +
> +6. ProcessEntryContents() - this calls Entry.ProcessContents() on each entry.
> +The default implementatoin does nothing. This can be overriden to adjust the
> +contents of an entry in some way. For example, it would be possible to create
> +an entry containing a hash of the contents of some other entries. At this
> +stage the position and size of entries should not be adjusted.
> +
> +7. BuildImage() - builds the image and writes it to a file. This is the final
> +step.
> +
> +
> +Advanced Features / Technical docs
> +----------------------------------
> +
> +The behaviour of entries is defined by the Entry class. All other entries are
> +a subclass of this. An important subclass is Entry_blob which takes binary
> +data from a file and places it in the entry. In fact most entry types are
> +subclasses of Entry_blob.
> +
> +Each entry type is a separate file in the tools/binman/etype directory. Each
> +file contains a class called Entry_<type> where <type> is the entry type.
> +New entry types can be supported by adding new files in that directory.
> +These will automatically be detected by binman when needed.
> +
> +Entry properties are documented in entry.py. The entry subclasses are free
> +to change the values of properties to support special behaviour. For example,
> +when Entry_blob loads a file, it sets content_size to the size of the file.
> +Entry classes can adjust other entries. For example, an entry that knows
> +where other entries should be positioned can set up those entries' positions
> +so they don't need to be set in the binman decription. It can also adjust
> +entry contents.
> +
> +Most of the time such essoteric behaviour is not needed, but it can be
typo: esoteric
> +essential for complex images.
> +
> +
> +History / Credits
> +-----------------
> +
> +Binman takes a lot of inspiration from a Chrome OS tool called
> +'cros_bundle_firmware', which I wrote some years ago. That tool was based on
> +a reasonably simple and sound design but has expanded greatly over the
> +years. In particular its handling of x86 images is convoluted.
> +
> +Quite a few lessons have been learned which are hopefully be applied here.
> +
> +
> +Design notes
> +------------
> +
> +On the face of it, a tool to create firmware images should be fairly simple:
> +just find all the input binaries and place them at the right place in the
> +image. The difficulty comes from the wide variety of input types (simple
> +flat binaries containing code, packaged data with various headers), packing
> +requirments (alignment, spacing, device boundaries) and other required
> +features such as hierarchical images.
> +
> +The design challenge is to make it easy to create simple images, while
> +allowing the more complex cases to be supported. For example, for most
> +images we don't much care exactly where each binary ends up, so we should
> +not have to specify that unnecessarily.
> +
> +New entry types should aim to provide simple usage where possible. If new
> +core features are needed, they can be added in the Entry base class.
> +
> +
> +To do
> +-----
> +
> +Some ideas:
> +- Fill out the device tree to include the final position and size of each
> + entry (since the input file may not always specify these)
> +- Use of-platdata to make the information available to code that is unable
> + to use device tree (such as a very small SPL image)
> +- Write an image map to a text file
> +- Allow easy building of images by specifying just the board name
> +- Produce a full Python binding for libfdt (for upstream)
> +- Add an option to decode an image into the constituent binaries
> +- Suppoort hierarchical images (packing of binaries into another binary
> + which is then placed in the image)
> +- Support building an image for a board (-b) more completely, with a
> + configurable build directory
> +- Consider making binman work with buildman, although if it is used in the
> + Makefile, this will be automatic
> +- Implement align-end
> +
> +--
> +Simon Glass <s...@chromium.org>
> +7/7/2016
Only reviewed the doc. Will do some testing.
[snip]
Regards,
Bin
_______________________________________________
U-Boot mailing list
U-Boot@lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot
