Re: [yocto] RFC: Yocto BSP and kernel usability tools
Hi Tom, This is really late in coming, my apologies. This looks to be the beginnings of a very powerful set of tools, and the detail you started with should help ensure a successful initial version. Nice work. Some general questions that don't belong in any one particular place inline: o You mention the use of json files. Is there a web UI planned for this? Or did you just decide that the requirements for the UI were the same as a web UI so you opted to use an established format rather than create a new one? o As a follow-on to the previous, it is always nice if the input files to tools such as this can be easily human readable (and IMO this excludes things like XML and serialized python objects). A simple INI type config seems like it might be more accessible and python has a module for this. I imagine you've considered this, I'm curious as to your rationale for the method outlined below. o I have some concern about yet another description format of BSPs and essentially kernel configuration. We already ask users to think in terms of SCC and CFG fragments, rather than defconfigs. This effectively asks users to start thinking in terms of json name value pairs to map to bitbake syntax to describe scc and cfg fragments so we can assemble a .config. Having the tools and structure in place eliminates a lot of the guesswork, so in the end I think it is probably worth it ... I just worry about so many levels of abstraction - and I do not have a better alternative in mind. o I see the tools for KMACHINE but nothing for KBRANCH. The distinction here is subtle and not described in any of our documentations (BSP guide, Kernel Arch and Usage, or Yocto Developer guide). Ideally we use standard/default/base for new BSPs and only add the meta-data to the linux-yocto kernel. Is this workflow supported? I ask because I'm failing to accomplish this with the sys940x and seem to have to add the sys940x branch in order for kernel_configme to pick up the sys940x meta data. -- Darren On 12/19/2011 08:55 AM, Tom Zanussi wrote: Hi, This is what I hope to implement in the M2 timeframe. Any and all comments welcome... Thanks, Tom --- Getting Yocto running on a new board is often a prerequisite task to the real task of interest for new users, which is creating new applications designed to run on the board, or creating new board-specific embedded systems. In other words, most users just want to get the hardware up and running and not have to think too much about it before moving on to what they're really interested in, which is the stuff running on top of it. While there really can't be any such thing as a magic-bullet application that would do that automatically i.e. generate and maintain a guaranteed-working Yocto BSP for any given piece of hardware, it should at least be possible to provide a tool that would generate a base BSP usable as a starting point for that, and which would additionally allow the user to tweak various BSP settings, including kernel configuration options and simple kernel patches. Such a tool should be usable with only a minimal understanding of the Yocto build system and metadata, and with no knowledge at all of kernel internals or the Yocto kernel recipe bindings, or of the details of the internals of yocto kernel repositories, or in fact any knowledge of git or any other specific build-related tool. This document provides the design for such a tool. At a high level, the goal of the tool is to provide a user the means of creating a new Yocto BSP from which it should be possible (assuming a perfect run-time outcome, admittedly unlikely on a first pass) to submit a 'pull request' for inclusion into any repo(s) that would accept an 'official' Yocto BSP. For example, the output of the tool should produce metadata and kernel configuration that would be directly mergeable into the meta-intel and the linux-yocto kernel repos. The tool should also allow the BSP maintainer to afterwards at any time make changes to the kernel configuration and/or submit kernel patches for the BSP. Note that although knowledge of the Yocto kernel or kernel internals is not expected, the user is expected to know the basics of how to deal with the linux kernel at the 'user' level, specifically understanding and determining the specific kernel config options needed for the user's BSP. Specifically, the tool will allow for the following high-level tasks, which are discussed in more detail below: - create a new BSP from scratch from a set of user-settable parameters - manage individual kernel config settings - create and use groups of kernel config settings as KERNEL_FEATUREs - apply patches to a BSP's machine branch The following capabilities are specifically not provided by the tool: - help in determining a 'correct' set of kernel config options to set. The tool assumes the user already knows that, either through diffs of .config files before and
Re: [yocto] RFC: Yocto BSP and kernel usability tools
On Tue, 2012-01-31 at 14:52 -0800, Darren Hart wrote: Hi Tom, This is really late in coming, my apologies. This looks to be the beginnings of a very powerful set of tools, and the detail you started with should help ensure a successful initial version. Nice work. Some general questions that don't belong in any one particular place inline: o You mention the use of json files. Is there a web UI planned for this? Or did you just decide that the requirements for the UI were the same as a web UI so you opted to use an established format rather than create a new one? o As a follow-on to the previous, it is always nice if the input files to tools such as this can be easily human readable (and IMO this excludes things like XML and serialized python objects). A simple INI type config seems like it might be more accessible and python has a module for this. I imagine you've considered this, I'm curious as to your rationale for the method outlined below. As a design point, I've tried to completely decouple the user interface from the back-end, so that yes, in the future a web UI could easily replace the current command-line user interface. I decided to use json because it's pretty standard for web applications, and Python also has json parsers available. And it's human-readable, unlike XML or serialized Python. But it really doesn't need to be any of those, if something else makes sense. o I have some concern about yet another description format of BSPs and essentially kernel configuration. We already ask users to think in terms of SCC and CFG fragments, rather than defconfigs. This effectively asks users to start thinking in terms of json name value pairs to map to bitbake syntax to describe scc and cfg fragments so we can assemble a .config. Having the tools and structure in place eliminates a lot of the guesswork, so in the end I think it is probably worth it ... I just worry about so many levels of abstraction - and I do not have a better alternative in mind. No, users never see json or any description format (UI tool or webapp developers do though). The json format (or whatever it ends up being) is only used as a way of feeding user input values into the back-end in the absence of a user interface (e.g. sent from a browser, or specified as a group on the command-line). Regardless of where the input comes from, in any case it's finally translated it into the .scc and .cfg fragments we all know and love. Actually, the point of all this is that the end user doesn't even need to know about .scc files or .cfg fragments, and deals only at a high-level with 'features' or config items or patches; this whole scheme allows UI developers to creat UIs that do that. o I see the tools for KMACHINE but nothing for KBRANCH. The distinction here is subtle and not described in any of our documentations (BSP guide, Kernel Arch and Usage, or Yocto Developer guide). Ideally we use standard/default/base for new BSPs and only add the meta-data to the linux-yocto kernel. Is this workflow supported? I ask because I'm failing to accomplish this with the sys940x and seem to have to add the sys940x branch in order for kernel_configme to pick up the sys940x meta data. In the current implementation of the cmd-line UI, the user can specify any branch ending with base e.g. standard/base standard/default/base standard/default/common-pc-64/base standard/default/common-pc/base standard/preempt-rt/base 'base' is removed from the branch name and replaced with the machine name e.g. standard/default/foo The 'publish' function will end up creating that branch in the repo and adding the .scc/.cfg files as a bsp/ in the meta branch. Also, new features (.cfg and .scc contained in subdirs in the SRC_URI) will be created under features/ and referenced in the bsp .scc. I've also run into the same thing you describe - the current tools seem to require a 1-1 mapping between KBRANCH/KMACHINE and the machine's metadata, but I don't see a problem with that and it seems more intuitive to the user (and allows a place to commit bsp-specific branches immediately without explicitly having to force a new branch first). Tom -- Darren On 12/19/2011 08:55 AM, Tom Zanussi wrote: Hi, This is what I hope to implement in the M2 timeframe. Any and all comments welcome... Thanks, Tom --- Getting Yocto running on a new board is often a prerequisite task to the real task of interest for new users, which is creating new applications designed to run on the board, or creating new board-specific embedded systems. In other words, most users just want to get the hardware up and running and not have to think too much about it before moving on to what they're really interested in, which is the stuff running on top of it. While there really can't be any such thing as a magic-bullet application that would do that automatically i.e. generate and maintain
Re: [yocto] RFC: Yocto BSP and kernel usability tools
On Wed, 2011-12-28 at 00:36 +, McClintock Matthew-B29882 wrote: (This probably belongs somewhere inline but I got lost on the long email) A lot of times users would already have a git tree containing linux and/or u-boot already working for their boards. Perhaps you could let the user point at external linux and/or u-boot trees with defconfigs, u-boot configs, device trees, etc for their target BSP? Yes, definitely, I think that makes sense and should be easily doable using the templating mechanism e.g. using the 'choicelist' input method the user could be presented with a choice of kernel recipes to use similarly to how xserver-choice is defined in the examples. Choosing for example the Yocto 3.0 kernel would select what's already been described, but additional choices could select alternate kernels and allow the user to point those recipes at specific git repos or tarballs, attach defconfigs, .dts files or whatever to those kernels' SRC_URIs. There would need to be a way for the user to specify the files to include, etc, but that could also be easily done by adding some python code for that purpose (or maybe that's already covered by the code that adds patches to a SRC_URI). In any case, I haven't thought about all the details yet but I think the problem can be generalized even further and those details will come out in the implementation. But as a general answer to your question, I think that yes, it is possible and desirable, and I'll at least try to have a simple example of it implemented in the first pass. Tom -M On Mon, Dec 19, 2011 at 10:55 AM, Tom Zanussi tom.zanu...@intel.com wrote: Hi, This is what I hope to implement in the M2 timeframe. Any and all comments welcome... Thanks, Tom --- [snip] ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] RFC: Yocto BSP and kernel usability tools
(This probably belongs somewhere inline but I got lost on the long email) A lot of times users would already have a git tree containing linux and/or u-boot already working for their boards. Perhaps you could let the user point at external linux and/or u-boot trees with defconfigs, u-boot configs, device trees, etc for their target BSP? -M On Mon, Dec 19, 2011 at 10:55 AM, Tom Zanussi tom.zanu...@intel.com wrote: Hi, This is what I hope to implement in the M2 timeframe. Any and all comments welcome... Thanks, Tom --- Getting Yocto running on a new board is often a prerequisite task to the real task of interest for new users, which is creating new applications designed to run on the board, or creating new board-specific embedded systems. In other words, most users just want to get the hardware up and running and not have to think too much about it before moving on to what they're really interested in, which is the stuff running on top of it. While there really can't be any such thing as a magic-bullet application that would do that automatically i.e. generate and maintain a guaranteed-working Yocto BSP for any given piece of hardware, it should at least be possible to provide a tool that would generate a base BSP usable as a starting point for that, and which would additionally allow the user to tweak various BSP settings, including kernel configuration options and simple kernel patches. Such a tool should be usable with only a minimal understanding of the Yocto build system and metadata, and with no knowledge at all of kernel internals or the Yocto kernel recipe bindings, or of the details of the internals of yocto kernel repositories, or in fact any knowledge of git or any other specific build-related tool. This document provides the design for such a tool. At a high level, the goal of the tool is to provide a user the means of creating a new Yocto BSP from which it should be possible (assuming a perfect run-time outcome, admittedly unlikely on a first pass) to submit a 'pull request' for inclusion into any repo(s) that would accept an 'official' Yocto BSP. For example, the output of the tool should produce metadata and kernel configuration that would be directly mergeable into the meta-intel and the linux-yocto kernel repos. The tool should also allow the BSP maintainer to afterwards at any time make changes to the kernel configuration and/or submit kernel patches for the BSP. Note that although knowledge of the Yocto kernel or kernel internals is not expected, the user is expected to know the basics of how to deal with the linux kernel at the 'user' level, specifically understanding and determining the specific kernel config options needed for the user's BSP. Specifically, the tool will allow for the following high-level tasks, which are discussed in more detail below: - create a new BSP from scratch from a set of user-settable parameters - manage individual kernel config settings - create and use groups of kernel config settings as KERNEL_FEATUREs - apply patches to a BSP's machine branch The following capabilities are specifically not provided by the tool: - help in determining a 'correct' set of kernel config options to set. The tool assumes the user already knows that, either through diffs of .config files before and after using 'bitbake -c menuconfig', or some other means. - modification of the BSP metadata after BSP creation. The tool provides an initial 'write-once' BSP-generation capability, but doesn't allow it to be read in and modified after the initial pass. The user will have to make further modifications manually. The one exception to this is kernel features, which can be added to an already existing kernel bbappend file. - a guarantee or even expectation that the generated BSP will work on the actual hardware it's targeted for - it's highly unlikely the BSP will work 'out-of-the-box' and it's the developer's reponsibility to do the often hard work of figuring out what settings and/or code are actually needed to get the hardware to work as expected. The goal of the tool is to make that job easier, not to actually do that job. Also, though it's not explicitly a requirement of this tool, the design should be sufficiently modular to allow for its participation in generic logical pipelines. For instance, while the tool may present to the user a text-based UI for gathering information, it should also be able to operate without any kind of user interaction and retrieve the data it needs from a file, for instance e.g.: $ yocto-bsp create x86-input.json Another example that the design should not preclude would be its use at the end of a bsp-import pipeline e.g.: $ cat 3rdparty.bsp | 3rdparty2yocto | yocto-bsp create The tool will initially be implemented as a set of command-line tools which will essentially be thin layers on top of a modular Python API; future tools may be
