Howdy,
This is a long mail meant to describe the rationale behind a number of
the changes in newt in the develop branch. Feel free to read at your
leisure and comment on it with improvements or likes/dislikes.
It's turned into somewhat more of a description of the newt tool, which
I think was needed anyway. Description of changes at the end in the
section "Changes from Previous Newt."
Cheers,
Sterling
Introduction
Newt is a combination of build and package management system for
embedded contexts. The idea was to build a single tool that did both
source package management and build, debug and install.
There are a couple of reasons for this:
- In order to architect an operating system that works well for
constrained environments across the many different types of
microcontroller applications(from doorbells to medical devices to power
grids), you need a system that lets you select which packages to install
and which packages to build.
- The build systems for embedded devices are often fairly complicated
and not well served. Autoconf is more designed for detecting system
compatibility issues, but not well suited when it comes to tasks like:
- Building for multiple targets
- Deciding what to build in and what not to build in
- Managing dependencies between components
When looking at solving this problem, what we realized is that the
embedded problem is one very similar to what's been solved with source
package management systems in higher level languages such as Javascript
(Node), Go, PHP and Ruby. We decided to fuse their source management
systems with a make system built for embedded systems.
Thus begat newt.
Build System
For the past mos the majority of focus on newt has been on making the
build system support a lot of the common things you might want to do
when developing embedded applications. This includes:
- Generating full flash images
- Downloading debug images to a target board using a debugger
- Conditionally compiling libraries & code based upon build settings
In order to accomplish this, newt has a fairly smart package manager
that can read a directory tree, build a dependency tree, and emit the
right build artifacts. An example newt source tree is in
incubator-mynewt-blinky/develop:
$ tree -L 3
.
├── DISCLAIMER
├── LICENSE
├── NOTICE
├── README.md
├── apps
│ └── blinky
│ ├── pkg.yml
│ └── src
├── project.yml
└── targets
├── my_blinky_sim
│ ├── pkg.yml
│ └── target.yml
└── unittest
├── pkg.yml
└── target.yml
6 directories, 10 files
When newt sees a directory tree that contains a "project.yml" file.
Newt knows that its in the base directory of a project, and
automatically builds a package tree.
Here, you can see that there are two package directories, "apps" and
"targets." Apps is where applications are stored, and applications are
where the main() function is contained. They represent the top-level of
the build tree, and define the dependencies and features for the rest of
the system.
An example of blinky's app.yml file is:
$ more apps/blinky/pkg.yml
<snip>
pkg.name: apps/blinky
pkg.vers: 0.8.0
pkg.description: Basic example application which blinks an LED.
pkg.author: "Apache Mynewt <[email protected]>"
pkg.homepage: "http://mynewt.apache.org/";
pkg.repository:
pkg.keywords:
pkg.deps:
- "@apache-mynewt-core/libs/os"
- "@apache-mynewt-core/hw/hal"
- "@apache-mynewt-core/libs/console/full"
This file says that the name of the package is apps/blinky, and it
depends on libs/os, hw/hal and libs/console/full packages.
NOTE: @apache-mynewt-core is a repository descriptor, and this will be
covered in the "repository" section.
Now, when newt is told to build this project, it will:
- Find the top-level project.yml file
- Recurse the packages in the package tree, and build a list of all
source packages
Newt then looks at the target that the user set, for example, blinky_sim:
$ more targets/my_blinky_sim/
pkg.yml target.yml
$ more targets/my_blinky_sim/target.yml
### Target: targets/my_blinky_sim
target.app: "apps/blinky"
target.bsp: "@apache-mynewt-core/hw/bsp/native"
target.build_profile: "debug"
The target specifies two major things:
- Application (target.app): The application to build
- Board Support Package (target.bsp): The board support package to build
along with that application.
These two packages represent the top of the build dependency tree. Newt
then goes and builds the dependency tree specified by all the packages.
While building this tree, it does a few other things:
- Any package that depends on another package, automatically gets the
includes from the package it includes. Include directories in the
newt structure must always be prefix by the package name:
i.e. libs/os has the following include tree:
$ tree
.
├── include
│ └── shell
│ └── shell.h
├── pkg.yml
└── src
├── shell.c
├── shell_os.c
└── shell_priv.h
include contains the package name "shell" before any header files. This
is in order to avoid any header file conflicts.
- API requirements are validated. Packages can export APIs they
implement, (i.e. pkg.api: hw-hal-impl), and other packages can require
those APIs (i.e. pkg.req_api: hw-hal-impl).
- Features options are supported. Packages can change what dependencies
they have, or what Cflags they are using based upon what features are
enabled in the system. As an example, many packages will add additional
software, based on whether the shell package is present. To do this,
they can overwrite cflags or deps based upon the shell "feature."
pkg.cflags.SHELL: -DSHELL_PRESENT
In order to properly resolve all dependencies in the build system, newt
recursively processes the package dependencies until there are no new
dependencies or features (because features can add dependencies.) And
it builds a big list of all the packages that need to be build.
Newt then goes through this package list, and builds every package into
an archive file.
NOTE: The newt tool generates compiler dependencies for all of these
packages, and only rebuilds the packages whose dependencies have not
changed. Changes in package & project dependencies are also taken into
account.
Once newt has built all the archive files, it then links the archive
files together. The linkerscript to use is specified by the board
support package (BSP.)
NOTE: One common use of the "features" option above is to overwrite
which linkerscript is used, based upon whether or not the BSP is being
build for a raw image, bootable image or bootloader itself.
The newt tool places all of it's artifacts into the bin/ directory at
the top-level of the project, prefixed by the target name being built,
for example:
$ tree -L 4 bin/
bin/
└── my_blinky_sim
├── apps
│ └── blinky
│ ├── blinky.a
│ ├── blinky.a.cmd
│ ├── blinky.elf
│ ├── blinky.elf.cmd
│ ├── blinky.elf.dSYM
│ ├── blinky.elf.lst
│ ├── main.d
│ ├── main.o
│ └── main.o.cmd
├── hw
│ ├── bsp
│ │ └── native
│ ├── hal
│ │ ├── flash_map.d
│ │ ├── flash_map.o
<snip>
As you can see, a number of files are generated:
- Archive File
- *.cmd: The command use to generate the object or archive file
- *.lst: The list file where symbols are located
- *.o The object files that get put into the archive file
Download/Debug Support
Once a target has been build, there are a number of helper functions
that work on the target. These are:
download Download built target to board
debug Open debugger session to target
size Size of target components
create-image Add image header to target binary
Download and debug handles driving GDB and the system debugger. These
commands call out to scripts that are defined by the BSP.
$ more repos/apache-mynewt-core/hw/bsp/nrf52pdk/nrf52pdk_debug.sh
<snip>
#
if [ $# -lt 1 ]; then
echo "Need binary to download"
exit 1
fi
FILE_NAME=$2.elf
GDB_CMD_FILE=.gdb_cmds
echo "Debugging" $FILE_NAME
# Monitor mode. Background process gets it's own process group.
set -m
JLinkGDBServer -device nRF52 -speed 4000 -if SWD -port 3333 -singlerun &
set +m
echo "target remote localhost:3333" > $GDB_CMD_FILE
arm-none-eabi-gdb -x $GDB_CMD_FILE $FILE_NAME
rm $GDB_CMD_FILE
The idea is that every BSP will add support for the debugger environment
for that board. That way common tools can be used across various
development boards and kits.
NOTE: Both for compiler definitions and debugger scripts, the plan is to
create Dockerizable containers of these toolchains. This should make
things much easier to support across Mac OS X, Linux and Windows. Newt
will know how to call out to Docker to perform these processes.
Source Management
First, congrats for making it here. You are a dedicated reader, I wrote
this mail for you.
The other major element of the newt tool is the ability to create
reusable source distributions from a collection of code. The first
question we had to answer is what is a reusable container of source
code: a package or a project.
In our development process, we have a number of packages that we really
think should (if for nothing else than convenience) should be released
together. It makes sense to release the RTOS core, and filesystem APIs
and networking stack together -- so that releases of the Mynewt OS have
some cohesion to them and are not drastically different.
Therefore, the decision was made to provide versioning and
redistribution of the project, and not the individual packages within
those projects.
A project that has been made redistributable is known as a repository.
Repositories can be added to your local project by adding them into your
project.yml file. Here is an example of the blinky project's yml file,
which relies on apache-mynewt-core:
$ more project.yml
<snip>
project.repositories:
- apache-mynewt-core
# Use github's distribution mechanism for core ASF libraries.
# This provides mirroring automatically for us.
#
repository.apache-mynewt-core:
type: github
vers: 0-latest
user: apache
repo: incubator-mynewt-core
When you specify this repository in the blinky's project file, you can
then use newt to install dependencies:
$ newt install
Downloading repository description for apache-mynewt-core... success!
Downloading repository incubator-mynewt-core (branch: develop) at
https://github.com/apache/incubator-mynewt-core.git
Cloning into
'/var/folders/7l/7b3w9m4n2mg3sqmgw2q1b9p80000gn/T/newt-repo814721459'...
remote: Counting objects: 17601, done.
remote: Compressing objects: 100% (300/300), done.
remote: Total 17601 (delta 142), reused 0 (delta 0), pack-reused 17284
Receiving objects: 100% (17601/17601), 6.09 MiB | 3.17 MiB/s, done.
Resolving deltas: 100% (10347/10347), done.
Checking connectivity... done.
Repos successfully installed
Newt will install this repository in the <project>/repos directory. In
the case of blinky, the directory structure ends up looking like:
$ tree -L 2
.
├── DISCLAIMER
├── LICENSE
├── NOTICE
├── README.md
├── apps
│ └── blinky
├── project.state
├── project.yml
├── repos
│ └── apache-mynewt-core
└── targets
├── my_blinky_sim
└── unittest
In order to reference the installed repositories in packages, the "@"
notation should be specified in the repository specifier. As an
example, the apps/blinky application has the following dependencies in
it's pkg.yml file:
$ more apps/blinky/pkg.yml
<snip>
pkg.deps:
- "@apache-mynewt-core/libs/os"
- "@apache-mynewt-core/hw/hal"
- "@apache-mynewt-core/libs/console/full"
This tells the build system to look in the base directory of
repos/apache-mynewt-core for the "libs/os" package.
In order to create a repository out of a project, all you need to do is
create a repository.yml file, and check it into the master branch of
your project.
NOTE: Currently only github is supported by our package management
system, but straight git will be added soon.
The repository.yml defines all versions of this repository and the
corresponding source control tags that these versions correspond to. As
an example, the repository.yml file has the following contents:
$ more repository.yml
repo.name: apache-mynewt-core
repo.versions:
"0.0.0": "develop"
"0-latest": "0.0.0"
There is one version of the apache-mynewt-core operating system
available, which is 0.0.0 (we haven't released yet! :-) This version
corresponds to the "develop" branch in this repository.
In addition to the 0.0.0 branch, there is a holding version 0-latest,
which specifies the latest version of the 0.0.0 release (which is 0.)
In many cases, most people who are maintaining dependencies to the
repository will likely provide some form of major/minor and the holding
branch (e.g. 0.8-stable, 0.8-latest). These map to specific versions in
the repository.yml file, and get finally resolved into the branch name.
Repositories can also have dependencies on other repositories. These
dependencies should be listed out on a per-tag basis. So, for example,
if apache-mynewt-core were to depend on sterlys-little-repo, you might
have the following directives in the repository.yml:
develop.repositories:
sterlys-little-repo:
type: github
vers: 0.8-latest
user: sterlinghughes
repo: sterlys-little-repo
This would tell newt that for anything that resolves to the develop
branch, this repository requires the sterlys-little-repo repository.
Dependencies are resolved circularly by the newt tool, and every
dependent repository is placed as a sibling in the repos directory.
Currently, if two repositories have the same name, they will conflict
and bad things will happen.
When a repository is installed to the repos/ directory, the current
version of that repository is written to the "project.state" file. The
project state file contains the currently installed version of any given
repository. This way, the current set of repositories can be recreated
from the project.state file reliably, whereas the project.yml file can
have higher level directives (i.e. include 0.8-stable.)
In order to upgrade a previously installed repository, the "newt
upgrade" command should be issued:
$ newt upgrade
Newt upgrade will look at the current desired version in project.yml,
and compare it to the version in project.state. If these two differ, it
will upgrade the dependency. Upgrade works not just for the dependency
in project.yml, but for all the sub-dependencies that they might have.
A NOTE ON DEPENDENCY RESOLUTION:
At the moment, all dependencies must match, otherwise newt will provide
an error. As an example, if you have a set of dependencies such that:
apache-mynewt-core depends on sterlys-little-repo 0.6-stable
apache-mynewt-core depends on sterlys-big-repo 0.5.1
sterlys-big-repo-0.5.1 depends on sterlys-little-repo 0.6.2
Where 0.6-stable is 0.6.3
The newt tool will try and resolve the dependency to
sterlys-little-repo. It will notice that there are two conflicting
versions of the repository, and not perform installation.
In the future newt will be smarter about loading in all dependencies,
and then looking to satisfy those dependencies to the best match of all
potential options.
Changes from Previous Newt
The original newt was always written as "write one to throw it away."
We wanted to get familiar with and test the concepts behind the tool.
As such, the code wasn't very pretty. Combine that with our naming
packages, "eggs" and our base directory "larva" and multiple stages of
renaming without refactoring.
Oh, and we learned Go while writing it. Eek.
Very little has changed on the build side of the newt tool in terms of
functionality. Identities have been renamed to features, capabilities
to APIs, and helper commands have been added, but the cleanups have been
mostly internal here.
The repository management has been designed and written from the ground
up, and is therefore a bit rawer, and more susceptible to change. I
believe we have the right approach here now, and so it will stabilize
much quicker.
For reference, the previous package installation and search was built on
the concept that newt would search for individual packages within git
repositories and place them in the local directory without any prefixing
or version history. In addition to the difficulty of maintaining a
branching strategy for this, it ended up being confusing as to what were
system packages and what were application packages.
To add pain, the package installation and upgrade system used the build
system's notion of package trees. The requirements for the two are
somewhat at odds (build tree very specific, and is a true tree, whereas
install & upgrade is more of a DAG.)
This has all been cleaned up in the new implementation.
Known Issues & Potential Future Changes
- Not 100% sure if packages/repositories shouldn't be rename to
libraries/packages.
- Dependency resolution is not as smart as it should be. It needs to
build a graph of all supported dependencies and then resolve the best
match from that.
- Missing helper commands to generate repository.yml and project.yml files
- Missing templates for common use cases (applications, drivers, etc.)
- Missing any ability to search for 3rd party packages
