Thanks Sterling I see this as really great. It's much clearer how to build my own project around this model.
Also, it's great to support the repos so naturally. It's going to take trying this out tomorrow to really solidify this in my mind. Two comments after reading 1) when I think of package management I think of the repository as a place to hold many different versioned packages (brew, maven, yum etc). But in newt each repository there is only one version more like source code repos. So it's not clear if repository is the right name. Wonder if we look at source packages from these other systems for some name ideas (they distribute buildable bundles of source as well). 2) it's going to help the community if there are a few common ways to find repositories and share their info with the community. I really hope newt can focus on solving this sooner than later. It's a good time to build this foundation to encourage sharing and community. > On Mar 10, 2016, at 5:15 PM, Sterling Hughes <[email protected]> wrote: > > Howdy, > > This is a long mail meant to describe the rationale behind a number of the > changes in newt in the develop branch. Feel free to read at your leisure and > comment on it with improvements or likes/dislikes. > > It's turned into somewhat more of a description of the newt tool, which I > think was needed anyway. Description of changes at the end in the section > "Changes from Previous Newt." > > Cheers, > Sterling > > > Introduction > > Newt is a combination of build and package management system for embedded > contexts. The idea was to build a single tool that did both source package > management and build, debug and install. > > There are a couple of reasons for this: > > - In order to architect an operating system that works well for constrained > environments across the many different types of microcontroller > applications(from doorbells to medical devices to power grids), you need a > system that lets you select which packages to install and which packages to > build. > > - The build systems for embedded devices are often fairly complicated and not > well served. Autoconf is more designed for detecting system compatibility > issues, but not well suited when it comes to tasks like: > - Building for multiple targets > - Deciding what to build in and what not to build in > - Managing dependencies between components > > When looking at solving this problem, what we realized is that the embedded > problem is one very similar to what's been solved with source package > management systems in higher level languages such as Javascript (Node), Go, > PHP and Ruby. We decided to fuse their source management systems with a make > system built for embedded systems. > > Thus begat newt. > > Build System > > For the past mos the majority of focus on newt has been on making the build > system support a lot of the common things you might want to do when > developing embedded applications. This includes: > > - Generating full flash images > - Downloading debug images to a target board using a debugger > - Conditionally compiling libraries & code based upon build settings > > In order to accomplish this, newt has a fairly smart package manager that can > read a directory tree, build a dependency tree, and emit the right build > artifacts. An example newt source tree is in incubator-mynewt-blinky/develop: > > $ tree -L 3 > . > ├── DISCLAIMER > ├── LICENSE > ├── NOTICE > ├── README.md > ├── apps > │ └── blinky > │ ├── pkg.yml > │ └── src > ├── project.yml > └── targets > ├── my_blinky_sim > │ ├── pkg.yml > │ └── target.yml > └── unittest > ├── pkg.yml > └── target.yml > > 6 directories, 10 files > > When newt sees a directory tree that contains a "project.yml" file. Newt > knows that its in the base directory of a project, and automatically builds a > package tree. > > Here, you can see that there are two package directories, "apps" and > "targets." Apps is where applications are stored, and applications are where > the main() function is contained. They represent the top-level of the build > tree, and define the dependencies and features for the rest of the system. > > An example of blinky's app.yml file is: > > $ more apps/blinky/pkg.yml > <snip> > pkg.name: apps/blinky > pkg.vers: 0.8.0 > pkg.description: Basic example application which blinks an LED. > pkg.author: "Apache Mynewt <[email protected]>" > pkg.homepage: "http://mynewt.apache.org/"; > pkg.repository: > pkg.keywords: > > pkg.deps: > - "@apache-mynewt-core/libs/os" > - "@apache-mynewt-core/hw/hal" > - "@apache-mynewt-core/libs/console/full" > > This file says that the name of the package is apps/blinky, and it depends on > libs/os, hw/hal and libs/console/full packages. > > NOTE: @apache-mynewt-core is a repository descriptor, and this will be > covered in the "repository" section. > > Now, when newt is told to build this project, it will: > > - Find the top-level project.yml file > > - Recurse the packages in the package tree, and build a list of all source > packages > > Newt then looks at the target that the user set, for example, blinky_sim: > > $ more targets/my_blinky_sim/ > pkg.yml target.yml > $ more targets/my_blinky_sim/target.yml > ### Target: targets/my_blinky_sim > target.app: "apps/blinky" > target.bsp: "@apache-mynewt-core/hw/bsp/native" > target.build_profile: "debug" > > The target specifies two major things: > > - Application (target.app): The application to build > - Board Support Package (target.bsp): The board support package to build > along with that application. > > These two packages represent the top of the build dependency tree. Newt then > goes and builds the dependency tree specified by all the packages. While > building this tree, it does a few other things: > > - Any package that depends on another package, automatically gets the > includes from the package it includes. Include directories in the > newt structure must always be prefix by the package name: > > i.e. libs/os has the following include tree: > > $ tree > . > ├── include > │ └── shell > │ └── shell.h > ├── pkg.yml > └── src > ├── shell.c > ├── shell_os.c > └── shell_priv.h > > include contains the package name "shell" before any header files. This is > in order to avoid any header file conflicts. > > - API requirements are validated. Packages can export APIs they implement, > (i.e. pkg.api: hw-hal-impl), and other packages can require those APIs (i.e. > pkg.req_api: hw-hal-impl). > > - Features options are supported. Packages can change what dependencies they > have, or what Cflags they are using based upon what features are enabled in > the system. As an example, many packages will add additional software, based > on whether the shell package is present. To do this, they can overwrite > cflags or deps based upon the shell "feature." > > pkg.cflags.SHELL: -DSHELL_PRESENT > > In order to properly resolve all dependencies in the build system, newt > recursively processes the package dependencies until there are no new > dependencies or features (because features can add dependencies.) And it > builds a big list of all the packages that need to be build. > > Newt then goes through this package list, and builds every package into an > archive file. > > NOTE: The newt tool generates compiler dependencies for all of these > packages, and only rebuilds the packages whose dependencies have not changed. > Changes in package & project dependencies are also taken into account. > > Once newt has built all the archive files, it then links the archive files > together. The linkerscript to use is specified by the board support package > (BSP.) > > NOTE: One common use of the "features" option above is to overwrite which > linkerscript is used, based upon whether or not the BSP is being build for a > raw image, bootable image or bootloader itself. > > The newt tool places all of it's artifacts into the bin/ directory at the > top-level of the project, prefixed by the target name being built, for > example: > > $ tree -L 4 bin/ > bin/ > └── my_blinky_sim > ├── apps > │ └── blinky > │ ├── blinky.a > │ ├── blinky.a.cmd > │ ├── blinky.elf > │ ├── blinky.elf.cmd > │ ├── blinky.elf.dSYM > │ ├── blinky.elf.lst > │ ├── main.d > │ ├── main.o > │ └── main.o.cmd > ├── hw > │ ├── bsp > │ │ └── native > │ ├── hal > │ │ ├── flash_map.d > │ │ ├── flash_map.o > <snip> > > As you can see, a number of files are generated: > > - Archive File > - *.cmd: The command use to generate the object or archive file > - *.lst: The list file where symbols are located > - *.o The object files that get put into the archive file > > Download/Debug Support > > Once a target has been build, there are a number of helper functions that > work on the target. These are: > > download Download built target to board > debug Open debugger session to target > size Size of target components > create-image Add image header to target binary > > Download and debug handles driving GDB and the system debugger. These > commands call out to scripts that are defined by the BSP. > > $ more repos/apache-mynewt-core/hw/bsp/nrf52pdk/nrf52pdk_debug.sh > <snip> > # > if [ $# -lt 1 ]; then > echo "Need binary to download" > exit 1 > fi > > FILE_NAME=$2.elf > GDB_CMD_FILE=.gdb_cmds > > echo "Debugging" $FILE_NAME > > # Monitor mode. Background process gets it's own process group. > set -m > JLinkGDBServer -device nRF52 -speed 4000 -if SWD -port 3333 -singlerun & > set +m > > echo "target remote localhost:3333" > $GDB_CMD_FILE > > arm-none-eabi-gdb -x $GDB_CMD_FILE $FILE_NAME > > rm $GDB_CMD_FILE > > The idea is that every BSP will add support for the debugger environment for > that board. That way common tools can be used across various development > boards and kits. > > NOTE: Both for compiler definitions and debugger scripts, the plan is to > create Dockerizable containers of these toolchains. This should make things > much easier to support across Mac OS X, Linux and Windows. Newt will know > how to call out to Docker to perform these processes. > > Source Management > > First, congrats for making it here. You are a dedicated reader, I wrote this > mail for you. > > The other major element of the newt tool is the ability to create reusable > source distributions from a collection of code. The first question we had to > answer is what is a reusable container of source code: a package or a project. > > In our development process, we have a number of packages that we really think > should (if for nothing else than convenience) should be released together. > It makes sense to release the RTOS core, and filesystem APIs and networking > stack together -- so that releases of the Mynewt OS have some cohesion to > them and are not drastically different. > > Therefore, the decision was made to provide versioning and redistribution of > the project, and not the individual packages within those projects. > > A project that has been made redistributable is known as a repository. > Repositories can be added to your local project by adding them into your > project.yml file. Here is an example of the blinky project's yml file, which > relies on apache-mynewt-core: > > $ more project.yml > <snip> > project.repositories: > - apache-mynewt-core > > # Use github's distribution mechanism for core ASF libraries. > # This provides mirroring automatically for us. > # > repository.apache-mynewt-core: > type: github > vers: 0-latest > user: apache > repo: incubator-mynewt-core > > > When you specify this repository in the blinky's project file, you can then > use newt to install dependencies: > > $ newt install > Downloading repository description for apache-mynewt-core... success! > Downloading repository incubator-mynewt-core (branch: develop) at > https://github.com/apache/incubator-mynewt-core.git > Cloning into > '/var/folders/7l/7b3w9m4n2mg3sqmgw2q1b9p80000gn/T/newt-repo814721459'... > remote: Counting objects: 17601, done. > remote: Compressing objects: 100% (300/300), done. > remote: Total 17601 (delta 142), reused 0 (delta 0), pack-reused 17284 > Receiving objects: 100% (17601/17601), 6.09 MiB | 3.17 MiB/s, done. > Resolving deltas: 100% (10347/10347), done. > Checking connectivity... done. > Repos successfully installed > > Newt will install this repository in the <project>/repos directory. In the > case of blinky, the directory structure ends up looking like: > > $ tree -L 2 > . > ├── DISCLAIMER > ├── LICENSE > ├── NOTICE > ├── README.md > ├── apps > │ └── blinky > ├── project.state > ├── project.yml > ├── repos > │ └── apache-mynewt-core > └── targets > ├── my_blinky_sim > └── unittest > > In order to reference the installed repositories in packages, the "@" > notation should be specified in the repository specifier. As an example, the > apps/blinky application has the following dependencies in it's pkg.yml file: > > $ more apps/blinky/pkg.yml > <snip> > pkg.deps: > - "@apache-mynewt-core/libs/os" > - "@apache-mynewt-core/hw/hal" > - "@apache-mynewt-core/libs/console/full" > > > This tells the build system to look in the base directory of > repos/apache-mynewt-core for the "libs/os" package. > > In order to create a repository out of a project, all you need to do is > create a repository.yml file, and check it into the master branch of your > project. > > NOTE: Currently only github is supported by our package management system, > but straight git will be added soon. > > The repository.yml defines all versions of this repository and the > corresponding source control tags that these versions correspond to. As an > example, the repository.yml file has the following contents: > > $ more repository.yml > repo.name: apache-mynewt-core > repo.versions: > "0.0.0": "develop" > "0-latest": "0.0.0" > > There is one version of the apache-mynewt-core operating system available, > which is 0.0.0 (we haven't released yet! :-) This version corresponds to > the "develop" branch in this repository. > > In addition to the 0.0.0 branch, there is a holding version 0-latest, which > specifies the latest version of the 0.0.0 release (which is 0.) In many > cases, most people who are maintaining dependencies to the repository will > likely provide some form of major/minor and the holding branch (e.g. > 0.8-stable, 0.8-latest). These map to specific versions in the > repository.yml file, and get finally resolved into the branch name. > > Repositories can also have dependencies on other repositories. These > dependencies should be listed out on a per-tag basis. So, for example, if > apache-mynewt-core were to depend on sterlys-little-repo, you might have the > following directives in the repository.yml: > > develop.repositories: > sterlys-little-repo: > type: github > vers: 0.8-latest > user: sterlinghughes > repo: sterlys-little-repo > > > This would tell newt that for anything that resolves to the develop branch, > this repository requires the sterlys-little-repo repository. > > Dependencies are resolved circularly by the newt tool, and every dependent > repository is placed as a sibling in the repos directory. Currently, if two > repositories have the same name, they will conflict and bad things will > happen. > > When a repository is installed to the repos/ directory, the current version > of that repository is written to the "project.state" file. The project state > file contains the currently installed version of any given repository. This > way, the current set of repositories can be recreated from the project.state > file reliably, whereas the project.yml file can have higher level directives > (i.e. include 0.8-stable.) > > In order to upgrade a previously installed repository, the "newt upgrade" > command should be issued: > > $ newt upgrade > > Newt upgrade will look at the current desired version in project.yml, and > compare it to the version in project.state. If these two differ, it will > upgrade the dependency. Upgrade works not just for the dependency in > project.yml, but for all the sub-dependencies that they might have. > > A NOTE ON DEPENDENCY RESOLUTION: > > At the moment, all dependencies must match, otherwise newt will provide an > error. As an example, if you have a set of dependencies such that: > > apache-mynewt-core depends on sterlys-little-repo 0.6-stable > apache-mynewt-core depends on sterlys-big-repo 0.5.1 > sterlys-big-repo-0.5.1 depends on sterlys-little-repo 0.6.2 > > Where 0.6-stable is 0.6.3 > > The newt tool will try and resolve the dependency to sterlys-little-repo. It > will notice that there are two conflicting versions of the repository, and > not perform installation. > > In the future newt will be smarter about loading in all dependencies, and > then looking to satisfy those dependencies to the best match of all potential > options. > > Changes from Previous Newt > > The original newt was always written as "write one to throw it away." We > wanted to get familiar with and test the concepts behind the tool. As such, > the code wasn't very pretty. Combine that with our naming packages, "eggs" > and our base directory "larva" and multiple stages of renaming without > refactoring. > > Oh, and we learned Go while writing it. Eek. > > Very little has changed on the build side of the newt tool in terms of > functionality. Identities have been renamed to features, capabilities to > APIs, and helper commands have been added, but the cleanups have been mostly > internal here. > > The repository management has been designed and written from the ground up, > and is therefore a bit rawer, and more susceptible to change. I believe we > have the right approach here now, and so it will stabilize much quicker. > > For reference, the previous package installation and search was built on the > concept that newt would search for individual packages within git > repositories and place them in the local directory without any prefixing or > version history. In addition to the difficulty of maintaining a branching > strategy for this, it ended up being confusing as to what were system > packages and what were application packages. > > To add pain, the package installation and upgrade system used the build > system's notion of package trees. The requirements for the two are somewhat > at odds (build tree very specific, and is a true tree, whereas install & > upgrade is more of a DAG.) > > This has all been cleaned up in the new implementation. > > Known Issues & Potential Future Changes > > - Not 100% sure if packages/repositories shouldn't be rename to > libraries/packages. > > - Dependency resolution is not as smart as it should be. It needs to build a > graph of all supported dependencies and then resolve the best match from that. > > - Missing helper commands to generate repository.yml and project.yml files > > - Missing templates for common use cases (applications, drivers, etc.) > > - Missing any ability to search for 3rd party packages
