On Fri, 2021-07-16 at 14:35 +0000, Peter Kjellerstedt wrote: > > This is a long answer, and responding to multiple subjects that have been > brought up in the different threads, so please bear with me. > > I would love to see improvements to the bitbake syntax when it comes to > overrides. Unfortunately, I do not see how I would be able to ever update > to said version if it is done as a flag day release. The way we work is > to have our layers compatible with the version of bitbake we are using > (currently Hardknott). Then I (who basically do all work to build our > layers with master of Poky) have a special adaptation layer where I add > bbappends and some occasional backported bbclass to make our layers work > with Poky master. This work runs in parallel with the normal recipe updates > that the developers do so that when a new major release of Poky is ready, > so are we. However, if we would have to use a whole different version of > all our recipes, there is no way this process would work, and I can't stop > the normal development for a flag day upgrade of Poky. :( > > The way I see it can be done would involve supporting both syntaxes in > one LTS-release. In the next non-LTS release, the support for the old > syntax can be dropped. That way there is always an intermediate release > that one can update to, update one's own recipes and then proceed to the > latest release. This of course means that the Yocto Project cannot reap > the benefits of the new and improved syntax immediately, but would have > to hold off for one release. On the other hand, this would mean the > transitioning path for all users of the Yocto Project would be trivial > (for some definition of trivial), compared to a major PITA. > > Now, I don't know if supporting both syntaxes at once is feasible given > some of the comments given in the thread. However, most of them seem > related to being able to drop support for the old syntax and thereby > gaining the expected improvements to the internal data, which would not > be a goal for the release that actually has to support both syntaxes. > Its goal is compatibility and a way forward.
I'm not sure it is going to be feasible to take two years to make this kind of transition. I've been looking for alternatives and one possibility may be semi-automated translation. By that, I mean that if you have a script with some knowledge of the layer it is translating, it appears from some quick tests locally that you can automate translation of a layer. My test was with poky (so bitbake, oe-core, meta-yocto and even the docs). The kind of knowledge a script needs is the names of the overrides in use and also function names or named parameter which contain keywords like "prepend", "append" and "remove". Once you prime it with that infomation, which a layer maintainer should have a good idea of, the script seems to be able to handle the bulk of translation. Code speaks volumes so: http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rpurdie/overrides-convert which has a script to translate poky, a commit of the translated content and a patch to bitbake to make it use ":" instead of "_" for overrides. I appreciate that translation like this isn't ideal, but it might just work well enough for people to work with to allow us to move forward. > As to the actual syntax changes, one thing I would like to se is the > _append/_prepend/_remove operators actually turn into real operators. > I definitely do not want to see the same syntax for the operators as > for overrides (that is one of the most confusing things with the > current solution). I very much want to improve things. The trouble is if we try and do too much at once, we'll lose the support of our users. There are several things we're effectively trying to do: a) Add more information to the metadata (which text *is* an override?) b) Change the syntax to be more readable c) Remove certain syntax/operations d) Potentially change the behaviour of some of the syntax In the original thread, I was mixing up several of these things. I've tried to step back and try and just solve a) in a way we can agree on as a first step. If we can't even do that... > Here is one suggestion: > > PACKAGECONFIG_append = "foo" => PACKAGECONFIG{} .= "foo" > PACKAGECONFIG_append = " foo" => PACKAGECONFIG{} += "foo" > PACKAGECONFIG_prepend = "foo " => PACKAGECONFIG{} =. "foo" > PACKAGECONFIG_prepend = " foo " => PACKAGECONFIG{} =+ "foo" > PACKAGECONFIG_remove = "foo " => PACKAGECONFIG{} -= "foo" How do you explain PACKAGECONFIG{} += "foo" vs PACKAGECONFIG += "foo" to a user? I suspect that is the kind of thing which will cause us problems. [Just to be clear to everyone reading this, the problem is that .= and _append are not equivalent since the former applies to the default value, the latter to all possible values including overrides.] > > PACKAGECONFIG_append_bar = "foo" => PACKAGECONFIG{bar} .= "foo" > PACKAGECONFIG_append_bar = " foo" => PACKAGECONFIG{bar} += "foo" > PACKAGECONFIG_prepend_bar = "foo " => PACKAGECONFIG{bar} =. "foo" > PACKAGECONFIG_prepend_bar = " foo " => PACKAGECONFIG{bar} =+ "foo" > PACKAGECONFIG_remove_bar = "foo " => PACKAGECONFIG{bar} -= "foo" > > PACKAGECONFIG_append_bar_foo = "foo" => PACKAGECONFIG{bar&foo} .= "foo" > PACKAGECONFIG_append_bar_foo = " foo" => PACKAGECONFIG{bar&foo} += "foo" > PACKAGECONFIG_prepend_bar_foo = "foo " => PACKAGECONFIG{bar&foo} =. "foo" > PACKAGECONFIG_prepend_bar_foo = " foo " => PACKAGECONFIG{bar&foo} =+ "foo" > PACKAGECONFIG_remove_bar_foo = "foo " => PACKAGECONFIG{bar&foo} -= "foo" > > This would not have a corresponding solution for, e.g., > PACKAGECONFIG_bar_foo_append, but I actually think that is good. Right, I think elsewhere I said this form was usually error prone and not correct. I was considering dropping it, *if* we can get to a point where the parser can reliably detect it. > This would also allow for clearing out the pending appends for a > given override by doing: > > PACKAGECONFIG{} = "" I like this idea. > or: > > PACKAGECONFIG{bar} = "" However: PACKAGECONFIG = "X" PACKAGECONFIG_bar = "" is a common usage. > And a totally different solution (and probably more controversial) would > be to add a more programmatical way to handle overrides, e.g.: > > PACKAGECONFIG += "foo" if overrides("foo bar") > > or: > > if override("foo") && override("bar"): PACKAGECONFIG += "foo" > > It doesn't help with whether the update to the variable is immediate or > deferred though. I think these are getting too clever and steps too far beyond what we can cope with both in bitbake and with the developers we have. I can't begin to think how these would work in the code :(. > As to RDEPENDS_${PN} & co, wouldn't it make sense to use RDEPENDS_pn-${PN}, > since "pn-<package name>" already exists as an override? PN is really badly named since it is effectively the recipe name in many contexts. We're specifying one single thing in overrides with it rather than a suite of packages a recipe generates. I can't see how using pn- like that would work for the use cases we have for the package name override. > On that note, > I would like to see overrides being prefixed more (maybe even as a > requirement). Some suggestions: > > pn- - Package name > df- - Distro feature > mf- - Machine feature > pf- - Package feature (can we rename/alias PACKAGECONFIG to > PACKAGE_FEATURES while at it, for consistency?) > arch- - Architecture > dist- - Distribution > > That way it would be more obvious what an override does. Especially the > use of the distro name as an override can have some unforeseen consequences > if the name is somewhat common). We have tried to do this with newer overrides where it makes sense. It is a tradeoff between longer names and usability. I don't think adding something for the package name would help. Having "distro-" and "machine-" could help in some cases but it comes at the price of more complicated conversions and it becomes a question of whether things become more readable. Worth thinking about but the above list doesn't really make sense since features are in overrides. (PACKAGECONFIG should really be RECIPE_FEATURES if we're going to talk names!) Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#1272): https://lists.openembedded.org/g/openembedded-architecture/message/1272 Mute This Topic: https://lists.openembedded.org/mt/84225642/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
