Re: [OE-core] [RFC][PATCH 0/6] development vs. production builds
On 05/16/2017 05:25 PM, Patrick Ohly wrote: It is done like that already, it's just not very consistent from what I can see. For example, core-image-sato-dev.bb: = require core-image-sato.bb DESCRIPTION = "Image with Sato for development work. It includes everything \ within core-image-sato plus a native toolchain, application development and \ testing libraries, profiling and debug symbols." IMAGE_FEATURES += "dev-pkgs" = That's different. Here an image recipe specifies what it is meant to *contain*, not how it is meant to *behave*. You can totally specify behavioral IMAGE_FEATURES in image recipes. The distinction is not formally specified or enforced anywhere, as far as I know. One would need core-image-sato-dev-production.bb (no debug-tweaks, dev-pkgs), core-image-sato-dev-debug.bb (debug-tweaks, dev-pkgs), core-image-sato-production.bb (no debug-tweaks, no dev-pkgs), core-image-sato-debug.bb (debug-tweaks, no dev-pkgs). Allowing EXTRA_IMAGE_FEATURES in local.conf.sample avoids that. Consider that the suffix in image recipe name is essentially same as your IMAGE_MODE idea. It specifies who is the target audience for the image. Then you can avoid combinatorial explosion, if you define who the image is meant for: someimage-developers.bb (debug tweaks, dev-pkgs) someimage-endusers.bb (no debug tweaks, no dev-pkgs) I'm not a big fan of placing INHERIT into local.conf either, by the way. I believe in functional programming principles, and this goes directly against them. It makes sense to me when the functionality is orthogonal to the content, like enabling buildhistory. Unfortunately, there is no way to enforce orthogonality. The door is wide open for awful hacks here. Alex -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [RFC][PATCH 0/6] development vs. production builds
On Tue, 2017-05-16 at 17:02 +0300, Alexander Kanavin wrote: > On 05/16/2017 04:47 PM, Patrick Ohly wrote: > > > Then why is not already done like that in practice? Is it just because > > OE-core and Poky set such a bad precedence with teaching developers to > > add EXTRA_IMAGE_FEATURES ?= "debug-tweaks" to make the images usable, > > and then that approach gets copied? > > It is done like that already, it's just not very consistent from what I > can see. For example, core-image-sato-dev.bb: > = > require core-image-sato.bb > > DESCRIPTION = "Image with Sato for development work. It includes > everything \ > within core-image-sato plus a native toolchain, application development > and \ > testing libraries, profiling and debug symbols." > > IMAGE_FEATURES += "dev-pkgs" > = That's different. Here an image recipe specifies what it is meant to *contain*, not how it is meant to *behave*. One would need core-image-sato-dev-production.bb (no debug-tweaks, dev-pkgs), core-image-sato-dev-debug.bb (debug-tweaks, dev-pkgs), core-image-sato-production.bb (no debug-tweaks, no dev-pkgs), core-image-sato-debug.bb (debug-tweaks, no dev-pkgs). Allowing EXTRA_IMAGE_FEATURES in local.conf.sample avoids that. > I'm not a big fan of placing INHERIT into local.conf either, by the way. > I believe in functional programming principles, and this goes directly > against them. It makes sense to me when the functionality is orthogonal to the content, like enabling buildhistory. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [RFC][PATCH 0/6] development vs. production builds
On 05/16/2017 04:47 PM, Patrick Ohly wrote: Then why is not already done like that in practice? Is it just because OE-core and Poky set such a bad precedence with teaching developers to add EXTRA_IMAGE_FEATURES ?= "debug-tweaks" to make the images usable, and then that approach gets copied? It is done like that already, it's just not very consistent from what I can see. For example, core-image-sato-dev.bb: = require core-image-sato.bb DESCRIPTION = "Image with Sato for development work. It includes everything \ within core-image-sato plus a native toolchain, application development and \ testing libraries, profiling and debug symbols." IMAGE_FEATURES += "dev-pkgs" = I think everyone agrees that removing "debug-tweaks" would be a good idea. But completely removing the global (sic!) EXTRA_IMAGE_FEATURES in local.conf.sample would go even further, and I am not sure how the reactions to that would be. I suspect there are people who find it useful to have just one image recipe that gets build in different configurations (dangerous and not so dangerous). I'm not sure either, but I think that's not actually a bad idea - dropping support for it altogether. :) I'm not a big fan of placing INHERIT into local.conf either, by the way. I believe in functional programming principles, and this goes directly against them. Alex -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [RFC][PATCH 0/6] development vs. production builds
On Tue, 2017-05-16 at 14:49 +0300, Alexander Kanavin wrote: > On 05/16/2017 11:21 AM, Patrick Ohly wrote: > > >> While the "development/production" switch may be great for some projects, > >> it'll make things only more complicated for others while gaining nothing > >> above > >> what we have now. > > > > What about the approach I outlined in my reponse to Richard, where we > > just introduce the IMAGE_MODE mechanism in OE-core without defining > > specific modes? Would you find that useful? > > Please no. Global variables are a headache, and the less we have of them > the better. > > Here in particularly, what is wrong with defining: > > patricks-awesome-image-developers.bb > (containing IMAGE_FEATURE = "no-password-for-anything") > > patricks-awesome-image-production.bb > (containing IMAGE_FEATURE = "serious-security") > > and the common bits can go to patricks-awesome-image.inc. Then why is not already done like that in practice? Is it just because OE-core and Poky set such a bad precedence with teaching developers to add EXTRA_IMAGE_FEATURES ?= "debug-tweaks" to make the images usable, and then that approach gets copied? I think everyone agrees that removing "debug-tweaks" would be a good idea. But completely removing the global (sic!) EXTRA_IMAGE_FEATURES in local.conf.sample would go even further, and I am not sure how the reactions to that would be. I suspect there are people who find it useful to have just one image recipe that gets build in different configurations (dangerous and not so dangerous). Feel free to prepare and propose a patch that implements the idea above for OE-core; I personally won't, I've had enough negative feedback for this week already ;-} -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [RFC][PATCH 0/6] development vs. production builds
On 05/16/2017 11:21 AM, Patrick Ohly wrote: While the "development/production" switch may be great for some projects, it'll make things only more complicated for others while gaining nothing above what we have now. What about the approach I outlined in my reponse to Richard, where we just introduce the IMAGE_MODE mechanism in OE-core without defining specific modes? Would you find that useful? Please no. Global variables are a headache, and the less we have of them the better. Here in particularly, what is wrong with defining: patricks-awesome-image-developers.bb (containing IMAGE_FEATURE = "no-password-for-anything") patricks-awesome-image-production.bb (containing IMAGE_FEATURE = "serious-security") and the common bits can go to patricks-awesome-image.inc. It's simple, direct and self-explanatory. At the end of the day, you *always* need to drill down to the code that does the work, and learn the details, to truly understand what is happening behind a configuration setting, and adding abstraction layers and scattering functionality over many different files does not help - quite the opposite. Alex -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [RFC][PATCH 0/6] development vs. production builds
This seems to assume that every project is similar and that develop vs production is something well defined. Well, it's not. I for one would welcome the demise of debug-tweaks. It's an option that enforces one of two policies that may both be wrong. For example, if the target is a settop box where users tend to port-forward the SSH port to the internet, and which is open so that they're expected to log in and make changes, the policy we want is that there is initially no root password, and that SSH access requires a valid password. This makes the system secure against outside attacks. Setting debug-tweaks will allow blank SSH passwords, while not setting it would invalidate the root login. Neither is acceptable for this project, so one has to resort to overriding the image class methods zap_root_password and ssh_allow_empty_password to do nothing. Instead of some obscure switch with implications hidden somewhere in various recipes, classes, distros and local configs, can't we just define options that actually do what they say, like for example a boolean option "ssh_allow_empty_password"? While the "development/production" switch may be great for some projects, it'll make things only more complicated for others while gaining nothing above what we have now. On 15-05-17 15:26, Patrick Ohly wrote: At OEDAM [1] I took the AR to flesh out some of my ideas for introducing global and per-image settings for switching between development and production builds. The goal is partly to establish common configure options that then can be used by different layers, partly to have some actual useful functionality attached to them already in OE-core. "development" builds are what a developer does when trying out a distro or working on his own personal device. "production" is what device manufacturer put onto the actual end-user hardware. At OEDAM we already concluded that per-image settings are more useful. However, sometimes a component also has compile-time choices that cannot be changed later on in an image, and indeed I found one example for that (kmod) in OE-core. Therefore I have included "debug-build" DISTRO_FEATURES support. It's a bit similar to manpages.bbclass, but in contrast to that (currently) is meant to be inherited globally - that's partly due to misunderstanding how manpages.bbclass was meant to be used. This can be changed, for now I just want to demonstrate that such a distro feature is not entirely useless, and what effect it could have already in OE-core. In refkit, we also switch globally between development and production builds via .inc files. However, I am in the process of replacing that with the more flexible per-image IMAGE_MODE check. So from my perspective, the IMAGE_MODE is more important and useful than the "debug-build" DISTRO_FEATURE. From a design perspective, the approach taken here is to let a developer or image define what mode it wants, and default features can be configured accordingly. That alternative would be to continue defining features as before and use the mode to configure QA warnings or errors. But I find that less flexible, and I suspect it would be harder to keep track of what is meant to be usable in which mode. Developers also would have a harder time overriding the defaults. [1] https://www.openembedded.org/wiki/OEDAM_2017 Patrick Ohly (6): build-mode.bbclass: distro-wide debug-build mode basefiles: warn about non-production DISTRO_FEATURES in motd defaultsetup.conf: enable special "debug-build" DISTRO_FEATURES support image-mode.bbclass: per-image production/development/debug mode image.bbclass: include IMAGE_MODE support local.conf.sample: make debug-tweaks depend on IMAGE_MODE meta/classes/build-mode.bbclass | 16 - meta/classes/image-mode.bbclass | 62 - meta/classes/image.bbclass| 3 +- meta/conf/distro/defaultsetup.conf| 2 +- meta/conf/local.conf.sample | 5 +- meta/recipes-core/base-files/base-files_3.0.14.bb | 11 +++- 6 files changed, 96 insertions(+), 3 deletions(-) create mode 100644 meta/classes/build-mode.bbclass create mode 100644 meta/classes/image-mode.bbclass base-commit: 9f9ebf2e1ba6eda48fb5e3f20d4ca5bbabe3dad4 Kind regards, Mike Looijmans System Expert TOPIC Products Materiaalweg 4, NL-5681 RJ Best Postbus 440, NL-5680 AK Best Telefoon: +31 (0) 499 33 69 79 E-mail: mike.looijm...@topicproducts.com Website: www.topicproducts.com Please consider the environment before printing this e-mail Join our presentation at Electronics & Applications 2017: FPGA for real-time data processing, subject “Hardware platform for industrial ultrasound steel plate Inspection” Topic Embedded Systems - Herman Kuster, 1st June 10 AM Visit http://eabeurs.nl/author/612884/ for more information -- ___ Openembedded-core mailing list
Re: [OE-core] [RFC][PATCH 0/6] development vs. production builds
On Tue, 2017-05-16 at 09:35 +0200, Mike Looijmans wrote: > This seems to assume that every project is similar and that develop vs > production is something well defined. Well, it's not. > > I for one would welcome the demise of debug-tweaks. It's an option that > enforces one of two policies that may both be wrong. If that's an option, then I'm also for it, and I don't mind dropping the "debug-build" distro feature. > While the "development/production" switch may be great for some projects, > it'll make things only more complicated for others while gaining nothing > above > what we have now. What about the approach I outlined in my reponse to Richard, where we just introduce the IMAGE_MODE mechanism in OE-core without defining specific modes? Would you find that useful? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [RFC][PATCH 0/6] development vs. production builds
On Tue, 2017-05-16 at 08:29 +0100, Richard Purdie wrote: > I'm not really sure I like this change, it seems fairly invasive and > complex for gains which don't seem to add up to me. > > For example after your patchset, we end up with two different places > motd may be generated based on various flags. I think it would be > confusing for the user to find out where a message as coming from. Fair enough. I'm not a big fan of the "debug-build" DISTRO_FEATURES either, and just included it as reference for what could be done at the distro level. What about just patch 4, the one which introduces image-mode.bbclass? Is that of enough interest to carry it in OE-core? With or without enabling it by default for images (patch 5)? That classes currently defines three modes in IMAGE_MODE_VALID, but intentionally with the weakest default possible. Distros can override that to match their intended usage modes, or (better, now that I think about it) we make the default empty and when users try to set IMAGE_MODE (because they might have seen it in some other distro) they get an error (assuming that we inherit the class in image.bbclass). My concern with not having this class in OE-core is that each distro will have to come up with their own way of doing something like this, with no consistency between distros. That is going to confuse some users. If a distro does not want this feature, they can just ignore the class, once we make IMAGE_MODE_VALID empty by default. > One of the reasons I have grown to hate "debug-tweaks" is that when you > enable it, you have no idea what you're really changing as the name > doesn't tell you what it does. I'm fine with eliminating "debug-tweaks" entirely. In that case, local.conf.sample should just list the individual features directly? > I worry "debug-build" will become the > same problem, it doesn't do one thing well but many things badly. If it > really just changes debug and logging PACKAGECONFIG (if present), why > doesn't the distro just do that? The idea was that the distro or user can set a global flag without having to know all these details. For example, consider the case where a user adds a custom recipe to a distro. If the distro has a "development.inc" and "production.inc" with PACKAGECONFIGs that tune recipes to match the intended usage, then those lists won't configure the custom recipe. > I also worry about the number of classes this introduces. Whilst > classes are cheap and easy, I have heard concerns we have too many of > them. In this case image-mode gets hardcoded into image.bbclass. We could merge the content of image-mode.bbclass also into image.bbclass, if there is consensus that this code should be active by default. I've done it differently here because I wanted to have the ability to use the standalone image-mode.bbclass in refkit. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [RFC][PATCH 0/6] development vs. production builds
On Mon, 2017-05-15 at 15:26 +0200, Patrick Ohly wrote: > At OEDAM [1] I took the AR to flesh out some of my ideas for > introducing global and per-image settings for switching between > development and production builds. The goal is partly to establish > common configure options that then can be used by different layers, > partly to have some actual useful functionality attached to them > already in OE-core. > > "development" builds are what a developer does when trying out a > distro or working on his own personal device. "production" is what > device manufacturer put onto the actual end-user hardware. > > At OEDAM we already concluded that per-image settings are more > useful. However, sometimes a component also has compile-time choices > that cannot be changed later on in an image, and indeed I found one > example for that (kmod) in OE-core. > > Therefore I have included "debug-build" DISTRO_FEATURES support. It's > a > bit similar to manpages.bbclass, but in contrast to that (currently) > is meant to be inherited globally - that's partly due to > misunderstanding how manpages.bbclass was meant to be used. This can > be changed, for now I just want to demonstrate that such a distro > feature is not entirely useless, and what effect it could have > already > in OE-core. > > In refkit, we also switch globally between development and production > builds via .inc files. However, I am in the process of replacing that > with the more flexible per-image IMAGE_MODE check. So from my > perspective, the IMAGE_MODE is more important and useful than the > "debug-build" DISTRO_FEATURE. > > From a design perspective, the approach taken here is to let a > developer or image define what mode it wants, and default features > can > be configured accordingly. That alternative would be to continue > defining > features as before and use the mode to configure QA warnings or > errors. > > But I find that less flexible, and I suspect it would be harder to > keep track of what is meant to be usable in which mode. Developers > also would have a harder time overriding the defaults. > > [1] https://www.openembedded.org/wiki/OEDAM_2017 I'm not really sure I like this change, it seems fairly invasive and complex for gains which don't seem to add up to me. For example after your patchset, we end up with two different places motd may be generated based on various flags. I think it would be confusing for the user to find out where a message as coming from. One of the reasons I have grown to hate "debug-tweaks" is that when you enable it, you have no idea what you're really changing as the name doesn't tell you what it does. I worry "debug-build" will become the same problem, it doesn't do one thing well but many things badly. If it really just changes debug and logging PACKAGECONFIG (if present), why doesn't the distro just do that? I also worry about the number of classes this introduces. Whilst classes are cheap and easy, I have heard concerns we have too many of them. In this case image-mode gets hardcoded into image.bbclass. Also, have you seen bb.utils.filter()? It might be useful instead of some of your .intersection() code? I do think you're right that we need to make some changes in this area but overall this patchset seems too complex to me somehow and I think we probably can achieve something similar in a simpler way... Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
[OE-core] [RFC][PATCH 0/6] development vs. production builds
At OEDAM [1] I took the AR to flesh out some of my ideas for introducing global and per-image settings for switching between development and production builds. The goal is partly to establish common configure options that then can be used by different layers, partly to have some actual useful functionality attached to them already in OE-core. "development" builds are what a developer does when trying out a distro or working on his own personal device. "production" is what device manufacturer put onto the actual end-user hardware. At OEDAM we already concluded that per-image settings are more useful. However, sometimes a component also has compile-time choices that cannot be changed later on in an image, and indeed I found one example for that (kmod) in OE-core. Therefore I have included "debug-build" DISTRO_FEATURES support. It's a bit similar to manpages.bbclass, but in contrast to that (currently) is meant to be inherited globally - that's partly due to misunderstanding how manpages.bbclass was meant to be used. This can be changed, for now I just want to demonstrate that such a distro feature is not entirely useless, and what effect it could have already in OE-core. In refkit, we also switch globally between development and production builds via .inc files. However, I am in the process of replacing that with the more flexible per-image IMAGE_MODE check. So from my perspective, the IMAGE_MODE is more important and useful than the "debug-build" DISTRO_FEATURE. >From a design perspective, the approach taken here is to let a developer or image define what mode it wants, and default features can be configured accordingly. That alternative would be to continue defining features as before and use the mode to configure QA warnings or errors. But I find that less flexible, and I suspect it would be harder to keep track of what is meant to be usable in which mode. Developers also would have a harder time overriding the defaults. [1] https://www.openembedded.org/wiki/OEDAM_2017 Patrick Ohly (6): build-mode.bbclass: distro-wide debug-build mode basefiles: warn about non-production DISTRO_FEATURES in motd defaultsetup.conf: enable special "debug-build" DISTRO_FEATURES support image-mode.bbclass: per-image production/development/debug mode image.bbclass: include IMAGE_MODE support local.conf.sample: make debug-tweaks depend on IMAGE_MODE meta/classes/build-mode.bbclass | 16 - meta/classes/image-mode.bbclass | 62 - meta/classes/image.bbclass| 3 +- meta/conf/distro/defaultsetup.conf| 2 +- meta/conf/local.conf.sample | 5 +- meta/recipes-core/base-files/base-files_3.0.14.bb | 11 +++- 6 files changed, 96 insertions(+), 3 deletions(-) create mode 100644 meta/classes/build-mode.bbclass create mode 100644 meta/classes/image-mode.bbclass base-commit: 9f9ebf2e1ba6eda48fb5e3f20d4ca5bbabe3dad4 -- git-series 0.9.1 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
