[yocto] [RFC] Binary Configuration support design
Hi, I have intiated few discussions earlier to support binary configuration tool, which allows configuration of post built binaries to generate custom image, without a need to rebuild from source. It is being tracked under bug 3252. Design to realize the same is updated in wiki. https://wiki.yoctoproject.org/wiki/Binary_configuration_support Please go through it and provide comments, so that I can proceed to submit patches. Regards, Ramana ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] RFC: Post build configuration
I agree that each product might have different connectivity and different configuration needs on the rootfs. However current thoughts are based on 1) Some of the common configurations of YP are currently at build time in yocto project, but appying them doesnot require a rebuild. Such configuration can be allowed(or moved) at post build, So that they are applied before flashing or installing the image on different devices, without rebuilding. 2) YP can provide a configuration platform, so that various new configuration options/methods ranging from OS to applications can be added easily, inlining with package addition/modification in various OE layers. It can be a place holder of various configurations. It can strengthen YP post build stage, saving rebuild effort, allowing it easily configurable and usable. Regards, Ramana ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] RFC: Post build configuration
This is in reference to my previous post regarding post build configuration, with few points re-iterating to set the ground. https://lists.yoctoproject.org/pipermail/yocto/2012-November/012867.html Post build configuration and need for configuration platform/model: There have been several discussions regarding need for image format, file system and disk configuration, network, users addition etc. Most are of these features are configurable in other distributions like opensuse, fedora during the image installation on the target. Considering the Embedded Linux distribution scenario, these features can be configured offline and the generated image can be directly deployed on the target. Example configuration options that can be supported include Configure the users/passwords Configure the network Configure the host name Select the services to be started by default Security related configuration Configure image formats ramfs/ext3/... paritions,disk management etc Serial port configuration etc.. Also a userscenario include Offline Binary Configuration, which allows a binary image customization before making a final target image. This case will be more evident in larger companies, where platform teams, product teams, application teams are distributed and Linux build from source will be owned and lab tested by a single team, like platform team. Other teams just configure to use it for product variants from same platform build. As configuration needs and options vary based on packages/features selected during for the distribution, so there is a need for a configuration platform/model 1) which allows new configuration options can be added easily and each package/distribution knows how it can be configured, so that selection of package or feature will expose those configuration options to the user. 2) Tools like HOB and Webhob also need a clean configuration interface to configure the image. 3) This can be extended to support Generic configuration UI like kernel menuconfig. Considering the above Post build configuration in mind, I like to put forward a model (high level flow) detailing 1. How the configuration functions/logic are maintained in poky? 2. How is configuration functions shipped along with image? 3. How configuration functions are used by configuration tool(s) to apply them over binary image, based on user configuration inputs? 1. How the configuration functions/logic are maintained in poky? --- Various configuration functions supporting the distribution configuration needs are maintained in current poky layers. a) Configuration functions include Image level and package level configuration functions. Where package level configuration functions kept along with package recipe(say pkgname_version.cfg). b) Package level configuration functions can override the image level configuration functions. Example: Useradd logic need to be different, if busybox is selected or if pwdutils package is selected. c) Each layer can override configuration functions, just like bb files. d) Menu Definition Information contains the metadata about configuration functions and their parameters allowing it to represent it in Generic UI, is also kept along with configuration functions. Menu definition Information can be represented in Menu Definition Language (MDL, say xml based). 2. How is configuration functions shipped along with image? - Configuration Library (ConfigLib) is created from configuration functions and MDL (from step1) during the package build time. ConfigLib is based on build time packages and features selected. This configuration Library is added to package feed. 3. How configuration functions are used by configuration tool(s) to apply them over binary image, based on user input/need? --- New tool can be created or existing tool(s) can be extended to support Post build package selection, image generation and configuration. Its functionality includes a) Selection of packages from package feed. b) Install the selected packages to relocatable rootfs. c) Copy configuration functions corresponding to selected packages from ConfigLib to SelectedConfigLib. d) User can configure the image by - Manually modifying offline configuration file (oct.conf) - Using a generic configuration UI which generates offline configuration file (oct.conf) e) Tool reads the offline configuration file(oct.conf) and maps the user configuration inputs to configuration functions in SelectedConfigLib. It applies the configuration options on the
Re: [yocto] Need for offline binary configuration
Added few more thoughts Looking for possible options to implement Offline Configuration Tool requirement. please check and comment on it. Configuration flow: a) Task do_createconfig will generate data to configure. This serves as input for configuration UI in OCT. b) OCT UI will read that config data(from step(a)) to show it in GUI. User configures using GUI inputs. Stores the user OCT configuration to OCT config file. (Note user can directly modify the OCT config file, if User is familier with it) c) preinst, postinst, prerm, postrm, or a do_applyconfig(without bb files reparse), can read OCT config file content (using a library) and apply necessary changes. Now we can look into possibility of using a specific configuration UI implementation or a generic configuration model or combination of both. 1) Using a specific configuration GUI implementation: do_createconfig generates the configuration information, in format like datafile/xml and OCT reads this config data and provides a GUI to modify or configure the features. Pros: Detailed and complex GUIs can be supported which will be user friendly. Ex: User management, network management etc. Cons: As UI is static, it has to be modified to support new configuration requirements and UI modifications are generally complex. 2) A generic configuration model do_createconfig generates configuration information with presentation information included (like kconfig) and OCT reads this config information and provides a generic UI implementation to modify or config the data like menuconfig. Pros: - Generic and dynamic UI allowing the config menus to be added from poky layers providing platform for configuration. - New features can add new menus configuration options. Feature/package enable and disable can control the configurtion menus. - So configuration is shown only if feature/package is present/selected. Cons: - Generic method of implementing can be complex, specifically for features where configuration is complex ex: User management, network configuration. - Custom or specific UI will be more user friendly and gives more control to UI designer. 3) Combination of both Complex features requiring more UI control can implement those features using the method(1) and also providing the plaform for generic UI implementation. OCT displays both UI configurations. Ex: Static Image level features like User management, network config can be implemented using method(1) Service configuration, file system type, hist size configure etc can be implemented using method(2). Pros: Gives control to project to select the Complex and simple UIs. Cons: Still needs a OCT UI change if method(1) feature need to be changed. Drawing a line between complex and simple UI features may not very straight forward. Adding dynamic Complex UI is not possible to be added from poky layers. Decision to provide a generic configuration model is mainly based on how much we expect the configuration area to grow along with layers. Image level features like user management, network management generally are not dynamic features, so having a specific/custom UI implementation looks fine for such features. Regards, Ramana ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Need for offline binary configuration
Looking for comments. Thanks From: Bruce Ashfield [bruce.ashfi...@windriver.com] Sent: Thursday, November 22, 2012 12:59 AM To: Venkata ramana gollamudi Cc: yocto@yoctoproject.org; Sanil kumar; Hatle, Mark; 'dvh...@linux.intel.com' Subject: Re: [yocto] Need for offline binary configuration On 12-11-21 11:29 AM, Venkata ramana gollamudi wrote: Reply inline -Original Message- From: Bruce Ashfield [mailto:bruce.ashfi...@windriver.com] Sent: Tuesday, November 20, 2012 8:53 PM To: Venkata ramana gollamudi Cc: yocto@yoctoproject.org; Sanil kumar; Hatle, Mark Subject: Re: [yocto] Need for offline binary configuration On 12-11-20 10:09 AM, Venkata ramana gollamudi wrote: Poky allows to build custom Linux for you, but we have cases where the post build customization is required, like user-addition, network configuration, service control. Even selecting the required packages can be a post build activity. The current model requires the image to be rebuilt to support these configuration. Offline Configuration tool (OCT), which allows a binary image customization before making a final target image. This case will be more evident in larger companies, where platform teams, product teams , application teams are distributed and Linux build from source will be owned and lab tested by a single team, like platform team. Other teams just configure to use it for product variants from same platform build. Detailed use cases can be found in enhancement bug:3252 OCT should work on the binary pool of compiled packages generated from poky. The basic operations that can be supported includes a) Select/deselect required packages from pool of binary packages into final target image. b) Provision to select external binary packages like ADT compiled applications as input and add them to final target image. c) Binary level Offline configuration can includes Configure the users/passwords Configure the network Configure the host name Select the services to be started by default Security related configuration Generate initrd in ramfs/ext3/... format etc.. Considering the methods to support these in our current yocto model, following changes can be done. 1) HOB can be the tool which can be extended to support these Poky can generate a binary package pool as one if its output and Hob can work on this package pool to select packages, configure and generate image. So HOB can support opening HOB in Binary(or OCT) mode i.e., without build options but only with binary package selection. Configuration GUI needs to be added to HOB. Note:HOB+OCT is together or separate, needs a bit more thought and overall organization as they will be intended for different users. Is there some overlap between this point and the other ongoing discussions about image construction, deployment and package management ? i.e. this thread: [OE-core] RFC: OE-Core image creation and deployment http://lists.linuxtogo.org/pipermail/openembedded-core/2012- July/026938.html These may already be coordinated, but I do see some common threads and it would be nice to make sure everything will work together and that we aren't duplicating effort! Cheers, Bruce Bruce, Thanks for the information. After your reply, I have gone through the discussions and agree that binary pool is in similar lines. Great to see that the realization happening in yocto1.4. Understood that package-feed can be used to generate the target image. Is there any RFC/mail/wiki available explaining how the configuration(like fstype) during image generation will be realized? Not that I know of. It is still under design last I heard, but MarkH is the person to provide the details. He's out of the office at the moment, but I'm sure that when he is back he can provide plenty of information. I am looking for post build configuration tool, which allows the product team users also to configure users, network, services etc . Agreed. I see this as something to start with, since it doesn't overlap with the other efforts (that I know of), and when I first read your email I thought it was the main focus. When you continued into image creation and package selection, that's when I noted the overlap. Image type, file system and Partition configuration can be one of them. I expect the product team users who configures image and generates target image, will have a little or no knowledge of bitbake, also needs easy installation and less dependencies. Can look in this context, how HOB will fit into this scenario or needs a new tool. Keeping the number of tools low is a good thing, so hopefully it can fit within the existing options. 2) Binary package pool can be a minimal/partial sstate-cache, as complete sstate-cache is quite big and not required for product teams as they are not expected to build but just need
Re: [yocto] Need for offline binary configuration
Reply inline -Original Message- From: Bruce Ashfield [mailto:bruce.ashfi...@windriver.com] Sent: Tuesday, November 20, 2012 8:53 PM To: Venkata ramana gollamudi Cc: yocto@yoctoproject.org; Sanil kumar; Hatle, Mark Subject: Re: [yocto] Need for offline binary configuration On 12-11-20 10:09 AM, Venkata ramana gollamudi wrote: Poky allows to build custom Linux for you, but we have cases where the post build customization is required, like user-addition, network configuration, service control. Even selecting the required packages can be a post build activity. The current model requires the image to be rebuilt to support these configuration. Offline Configuration tool (OCT), which allows a binary image customization before making a final target image. This case will be more evident in larger companies, where platform teams, product teams , application teams are distributed and Linux build from source will be owned and lab tested by a single team, like platform team. Other teams just configure to use it for product variants from same platform build. Detailed use cases can be found in enhancement bug:3252 OCT should work on the binary pool of compiled packages generated from poky. The basic operations that can be supported includes a) Select/deselect required packages from pool of binary packages into final target image. b) Provision to select external binary packages like ADT compiled applications as input and add them to final target image. c) Binary level Offline configuration can includes Configure the users/passwords Configure the network Configure the host name Select the services to be started by default Security related configuration Generate initrd in ramfs/ext3/... format etc.. Considering the methods to support these in our current yocto model, following changes can be done. 1) HOB can be the tool which can be extended to support these Poky can generate a binary package pool as one if its output and Hob can work on this package pool to select packages, configure and generate image. So HOB can support opening HOB in Binary(or OCT) mode i.e., without build options but only with binary package selection. Configuration GUI needs to be added to HOB. Note:HOB+OCT is together or separate, needs a bit more thought and overall organization as they will be intended for different users. Is there some overlap between this point and the other ongoing discussions about image construction, deployment and package management ? i.e. this thread: [OE-core] RFC: OE-Core image creation and deployment http://lists.linuxtogo.org/pipermail/openembedded-core/2012- July/026938.html These may already be coordinated, but I do see some common threads and it would be nice to make sure everything will work together and that we aren't duplicating effort! Cheers, Bruce Bruce, Thanks for the information. After your reply, I have gone through the discussions and agree that binary pool is in similar lines. Great to see that the realization happening in yocto1.4. Understood that package-feed can be used to generate the target image. Is there any RFC/mail/wiki available explaining how the configuration(like fstype) during image generation will be realized? I am looking for post build configuration tool, which allows the product team users also to configure users, network, services etc . Image type, file system and Partition configuration can be one of them. I expect the product team users who configures image and generates target image, will have a little or no knowledge of bitbake, also needs easy installation and less dependencies. Can look in this context, how HOB will fit into this scenario or needs a new tool. 2) Binary package pool can be a minimal/partial sstate-cache, as complete sstate-cache is quite big and not required for product teams as they are not expected to build but just need to select and configure. I think it is sufficient to keep the minimal binaries from sstate-cache which are required to execute image.bbclass, do_rootfs task to generate image. Point 2, No longer applicable as package-feed is a binary pool. 3) Along with specific configuration UI implementation, a generic configuration model similar to kernel kconfig and menuconfig can be considered, in cases where more detailed offline configurations is required like detailed security configuration. Point 3, still can be thought about. Regards, Ramana ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] Need for offline binary configuration
Poky allows to build custom Linux for you, but we have cases where the post build customization is required, like user-addition, network configuration, service control. Even selecting the required packages can be a post build activity. The current model requires the image to be rebuilt to support these configuration. Offline Configuration tool (OCT), which allows a binary image customization before making a final target image. This case will be more evident in larger companies, where platform teams, product teams , application teams are distributed and Linux build from source will be owned and lab tested by a single team, like platform team. Other teams just configure to use it for product variants from same platform build. Detailed use cases can be found in enhancement bug:3252 OCT should work on the binary pool of compiled packages generated from poky. The basic operations that can be supported includes a) Select/deselect required packages from pool of binary packages into final target image. b) Provision to select external binary packages like ADT compiled applications as input and add them to final target image. c) Binary level Offline configuration can includes Configure the users/passwords Configure the network Configure the host name Select the services to be started by default Security related configuration Generate initrd in ramfs/ext3/... format etc.. Considering the methods to support these in our current yocto model, following changes can be done. 1) HOB can be the tool which can be extended to support these Poky can generate a binary package pool as one if its output and Hob can work on this package pool to select packages, configure and generate image. So HOB can support opening HOB in Binary(or OCT) mode i.e., without build options but only with binary package selection. Configuration GUI needs to be added to HOB. Note:HOB+OCT is together or separate, needs a bit more thought and overall organization as they will be intended for different users. 2) Binary package pool can be a minimal/partial sstate-cache, as complete sstate-cache is quite big and not required for product teams as they are not expected to build but just need to select and configure. I think it is sufficient to keep the minimal binaries from sstate-cache which are required to execute image.bbclass, do_rootfs task to generate image. 3) Along with specific configuration UI implementation, a generic configuration model similar to kernel kconfig and menuconfig can be considered, in cases where more detailed offline configurations is required like detailed security configuration. Regards, Ramana ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] How do you release distros produced with Yocto? How should I?
On Tue, Oct 02, 2012 at 12:43:27PM -0400, Jerrod Peach wrote:
I was thinking about doing something very close to that in actual Yocto,
except I'd store only the revisions/branches that were different from what
the bb file prescribed in local.conf. I ran that idea by a couple of
colleagues separately and they both immediately pointed out a concern: they
didn't want to have two different places that could manage the revision of
the package, i.e. local.conf and the package bb file. They don't like how
that works in the system we have today. Instead, they wanted to branch all
the metadata and have people update individual bb files with different
revisions as necessary. That leaves only one place to manage the revision:
the package bb file. Doing all that branching concerns me (it seems
heavier-weight than it needs to be, at the very least), but I can't say
specifically why branching the bb files is harmful. If you guys can think
of a reason why, let me know. If you think it's reasonable, let me know
that too.
We have solved this issue by adding OS_VERSION variable to local.conf
Lets say your product (OS) should be released in versions, Ex: 1.0 , 1.1 or
1.1.1 etc.
Which is built from packages of different versions. So each package contains
pkgname.inc and package_version.bb file.
revision number and patches are maintained in package_version.bb and all
common part under package.inc.
Now the problem comes to mapping the package versions to your product(OS)
version.
This is done having a file included into disto file listing out all the
PREFERRED_VERSIONS and PREFERRED_PROVIDERS for all packages.
ex: conf/disto/include/osversion_1.0.inc
And this is included to disto file like
require conf/disto/include/osversion_${OS_VERSION}.inc
The file osversion_1.0.inc is freezed after your one version release.
Next version will add a new file osversion_1.1.inc overiding to pick all the
required package version changes to make your product version.
This will allow you to release multiple product(OS) versions and clearly maps
the package version, creating individual distributions.
You can even build older version product by just giving OS_VERSION=1.0 in
local.conf
This eliminates branching all together.
Regards,
Ramana
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto
[yocto] mechanism to disable package management on target
Hi, Is there any mechanism to disable package management, so that my rootfs should not have any package management related packages like opkg etc. I found that one of the methods ipk/rpm/deb need to be mandatorily selected. Please suggest if any such method exists. Regards, Ramana ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] Problem in downloading linux-yocto source behind proxy
Hi, We tried to build yocto release downloaded from autobuilder http://autobuilder.yoctoproject.org/pub/nightly/20120621-3/yocto.tar.bz2 We work behind a http proxy and firewall setup from office network. Previously we used to build by setting PREMIRRORS to git://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \ as all the source is available in the above mentioned mirror path for every RC release. 1) But now we couldnot find linux-yocto source at (for 1.3M1RC1 or RC2 release) i.e http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.linux-yocto-3.4.git.tar.gz 2) I remember that there was a discussion to enable readonly way of accessing git through http. Is it currently available? Please suggest some alternative mechanism. Regards, Ramana ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
