Re: [Openembedded-architecture] create-spdx support in dunfell
Hi Richard On Wednesday, 1 March 2023 23:33:22 NZDT Richard Purdie wrote: > After reading an email about encouraging best practises when supporting > older software releases, it got me thinking about our LTS and spdx/sbom > support. We don't have create-spdx there and our official policy says > no feature backports or breaking changes. > > If you understand LTS as "support" instead of "stable", there is a > compelling argument that we should be supporting people of older > releases to meet things like the new legislation around SBoMs. I think > the world is learning that difference and being more open to it too. > > I discussed this with the YP TSC and we agreed we probably should > evaluate the options for adding create-spdx to dunfell. There are a lot > of questions about it and it is something a lot of users would like. I > think it would better reflect on the project to support it. > > There is some risk to this. We need the bb.compress zstd code and some > changes to package.bbclass, specially: > > https://git.yoctoproject.org/poky/commit/?id=7ec54b174304e940ec66f21ac512f7b > 50fa637b3 > > The class itself isn't problematic as it is standalone. > > This doesn't mean any policy will change, it would be a one off > exception granted by the TSC for an initiative to support older > releases in a changing world. > > I think Joshua said he'd be willing to have a look at what was needed, > I wanted to put the idea out there and say the TSC is open minded to > the idea assuming testing works out ok and so on. We need to look > carefully at the zstd requirement in particular. I believe our LTS > maintainer is open to the idea too if there is support from the > community in working out the details. FWIW we have been running with this feature backported on top of dunfell for a while and haven't noticed any ill effects. I can't recall at this stage what changes were needed but if there were any I'd think they were minimal. Assuming there are no objections to having this in dunfell, I'll tentatively put up my (personal) hand to support it given that it seems to be fairly straightforward. If desired I could send the patches we have. Cheers Paul -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#1707): https://lists.openembedded.org/g/openembedded-architecture/message/1707 Mute This Topic: https://lists.openembedded.org/mt/97311397/21656 Group Owner: openembedded-architecture+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [Openembedded-architecture] Disruptive changes and the next LTS 3.5 - what to aim for?
On Friday, 10 September 2021 19:54:11 NZST Richard Purdie wrote: > > > Do we change the master branch to something else? I personally have a > > > preference for "devel" over "main" regardless of what others are doing > > > as it matches what it is in our case. Changing that alone is days of > > > work for me trying to get all our automation to deal with it. > > > > If the master branch gets renamed, what about yocto-buildhistory, > > yocto-buildstats and yocto-testresults? Would master branches or tags > > get renamed as well? Is this part of the day's worth of work you > > mentioned? > > > > Would the Yocto Project enforce a renaming in all the repos they host? > > > > What about layer-index? > > All good questions. The days of work I was mentioning would cover some of > this but there is a lot to do, more than people realise. It's not readily visible in the UI, but the layer index already has an internal means of specifying an alternative branch name that can be set by admins on a per layer-branch basis. I deliberately haven't exposed it as I thought it was better that people keep things simple and just use the same scheme as OE-Core for their branches, but the mechanism exists. We could also fairly easily add automatic mapping in the layer index code if we choose a standard name to replace master, as well as changing the name shown in the UI (and change the assumptions that "master" is the main branch, there are a few sprinkled through the code). I think even if we do nothing else we'll have to address those assumptions as there will probably be layer maintainers who want to change (if they have not done so already). If it's not clear, I can volunteer to take care of this ;) > I remembered what I was missing too: > > j) Convert to SPDX license names > > We should really switch to using SPDX license names in the LICENSE field > directly and be able to drop the current SPDX mapping code. This I'm not sure of. Are these mappings costing us much? AFAICT we'd only gain a bit of code simplification at the expense of making things a little bit harder for our users. Cheers Paul -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#1316): https://lists.openembedded.org/g/openembedded-architecture/message/1316 Mute This Topic: https://lists.openembedded.org/mt/85488159/21656 Group Owner: openembedded-architecture+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [Openembedded-architecture] Backporting python module move to OE-Core in dunfell
FWIW whilst this does fall outside normal policies, I think this is a reasonable course of action under the (fairly exceptional) circumstances. We would of course have to review future situations on a case-by-case basis. Paul On Wednesday, 3 June 2020 21:27:00 NZST Richard Purdie wrote: > Hi All, > > We'd like to get the first 3.1 point release Dunfell LTS out soon. > There is one issue we need to resolve first. > > meta-arm has recently been established and the hope is to avoid > duplication of a set of recipes in multiple BSP layers, a goal which I > believe is in everyone's interests. As with any new layer there are a > few growing pains like the removal of python2 dependencies and its made > good progress but there is an issue we didn't foresee. > > Two of the key components that BSPs need from there (tf-a and optee) > which are needed for handling boot crypto have dependencies on > pyelftools and pycryptodome which were in meta-python in meta- > openembedded. In master we moved them to core: > > http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=d52929ee35d043211cb012e > 1b86409599a8b53c6 > http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=1e327abcc92d6575df5c18 > 71e6e1284987b74b87 > > and upgraded them to the latest versions: > > http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=cb94d9019b4d1ce32ad9196 > cb9c22236631e8781 > http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=7ebe299c00ec309cd46905 > ad40036d937fe609d7 > > For master that is easy, problem solved. > > For Dunfell, I've already seen complaints that using any of the BSPs > means adding meta-arm which means adding meta-python which pulls in > other chunks of meta-oe. Using a BSP shouldn't require pulling in large > numbers of other layers and this defeats what we're trying to achieve > with a granular layer model. > > Had we realised sooner, we likely would have taken this change into > dunfell before release but sadly we didn't. > > The LTS policy is quite clear we now shouldn't make a change like this. > Equally, we're early in Dunfell's development, we have an unforeseen > issue which is causing users real world pain. > > Our options are: > > a) Do nothing > b) Move the recipes > c) Move the recipes and take the upgrades > > Given the impact on end users and the desire to have good user > experience with our first LTS, as the layer maintainer and having > discussed this with others and on the tech call yesterday, I am > strongly leaning to b or c. I think this issue isn't one Steve as LTS > maintainer or I as OE-Core maintainer can make alone and I'm therefore > referring to the TSC. Which TSC is a good question. LTS is owned by YP, > the layers by OE. As such I've cc'd both sets of TSC members. I've put > it to the architecture list so everyone can see why we're considering > this and any discussion on it. > > Considering the potential regressions, if a user upgrades both layers > with these changes things are fine. If the user upgrades meta-python > but not OE-Core, they get a build error which is also fine and > straightforward to debug. If the user upgrades OE-Core but not meta- > python, there is the possibility they could see old versions being used > due to layer priority (assuming we upgrade the versions) but this isn't > known to cause real world issues. > > We have the option of the patch to increase the OE-Core version and > then bumping the version requirement in meta-python to further improve > the user experience. > > I'm conscious there has already been talk of "favouritism". I would > stress this issue has actually been raised by end users and that is the > driving factor here. > > The fact there are interested member companies of YP does actually help > in some ways too since it means there are people available to help > handle any issues should there be unexpected fallout from this change. > > If we do this, I would say as a TSC member that we'd do it without > setting any precedent, this is being discussed based on the situation > at hand, in the future other situations would have to be looked at on a > case by case basis. > > I did quickly sound out the YP TSC at a meeting yesterday and I believe > on balance they're in favour of b/c, two of the OE TSC members on the > tech call yesterday were also in favour in principle. Whilst that is a > majority, that does leave a couple of TSC members who haven't seen the > discussion and everyone probably benefits from this having been written > down. > > Does anyone object to doing this? If we do it, do we upgrade the > recipes or not? I'm probably leaning towards upgrading too as up to > date crypto components are probably better than not updated and some > kind of parity with master now will probably catch more potential > issues. > > Cheers, > > Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#1088): https://lists.openembedded.org/g/openembedded-architecture/message/1088 Mute
Re: [Openembedded-architecture] layers.openembedded.org - Layer Policy Proposal (revised)
(see the previous > item), > has the layer’s “master related” branch been updated within the last 18 > months? > > If the layer is not compatible, and the branch has not been updated within the > last 18 months the layer will be removed from the “master branch” index as > inactive. Note: release branches are not expected to be affected by this > change! > > The above items will processed via automation. (See the attachment as to what > changes would occur if the above policy proposal is approved.) > > > Additional Request > -- > > During the initial proposal that was sent, another item was identified. Below > covers this suggested change to the layer index itself. > > It was request that we should add a delete or ‘unpublish’ button accessible to > the owners of the layer. It is my opinion that actually deleting the layer > should be reserved for layer index administrators to prevent bad actors from > removing layers that are otherwise still valid. Unpublishing a layer could > still cause temporary inconvenience, but would be very easy to revert. > > > Attachment > -- > > The attachments are tables per branch. A few of the indications are > explained here: > > !URL - The URL is invalid > > !layer - the layer itself isn't valid, but the URL worked > > !Compat - The layer does not indicate it is compatible > > Note - A note needs to be added to this layer in the index > > Email - The maintainer of the layer would be contacted prior (not yet slated > for > removal) > > Remove - the layer should be removed from this branch's index > > (master only) > > LAST - number of days since the last commit, if !COMPAT is detected. > > > Stats: > > (in all cases the number of layers with bad urls exceeds the 90 days.) > > Sumo - Remove 3 layers > - Add compat notes to 11 layers > > Thud - Remove 1 layer > - Add compat notes to 4 layers > > Warrior - Remove 5 layers > - Add compat notes to 2 layers > > Zeus - (no layers to remove) > - Add compat notes to 3 layers > > Master - Remove 129 layers >- Add compat notes to 103 layers > -- Paul Eggleton Intel System Software Products ___ Openembedded-architecture mailing list Openembedded-architecture@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
Re: [Openembedded-architecture] OE-Core python minimum version requirement
On Thursday, 5 December 2019 7:48:11 PM NZDT Nicolas Dechesne wrote: > On Thu, Dec 5, 2019 at 3:08 AM Paul Eggleton > wrote: > > FYI I came across repology.org which tells you the version of various > > packages in each distro, though it's not ideal: > > > > https://repology.org/project/python/versions > > Nice! How do we get OE plugged into that? Have you looked at that? No, I haven't - only discovered it today. I agree we should look into it though. Cheers Paul -- Paul Eggleton Intel System Software Products ___ Openembedded-architecture mailing list Openembedded-architecture@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
Re: [Openembedded-architecture] OE-Core python minimum version requirement
On Thursday, 5 December 2019 6:45:03 AM NZDT Richard Purdie wrote: > I just enabled hashequiv's server in local mode by default in poky. > > This causes an unintended side effect of requiring python 3.5 as the > minimum version. > > We had thought that the servers would be 'rare' and a 3.5 version > requirement for that was fine. It turns out a local server is also > extremely useful. > > The code needed the python 3.5 async support and trying to write it any > other way is a nightmare, we need that performance for the server. > > At this point I think we just give in and require python 3.5 as a > minimum. Any objections? Seems reasonable to me. Theoretically this should allow us to complete the subprocess cleanups as well (time permitting of course). FYI I came across repology.org which tells you the version of various packages in each distro, though it's not ideal: https://repology.org/project/python/versions That seems to be missing a few distros, here's another entry: https://repology.org/project/python3-defaults/versions For Ubuntu you have to go back to 14.04 to have 3.4; any others other than CentOS 7 that are as old we don't really support. There is also pkgs.org that I could have sworn used to support comparing package versions across distros but it doesn't seem to want to do it anymore. Distrowatch also tracks this per distro but only allows side-by-side comparisons across versions within a distro or only between two distros. I think in this instance we have the information we need though. Cheers, Paul -- Paul Eggleton Intel System Software Products ___ Openembedded-architecture mailing list Openembedded-architecture@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
Re: [Openembedded-architecture] layers.openembedded.org - suggestion to remove obsolete layers
On Thursday, 5 December 2019 2:09:50 AM NZDT Tom Rini wrote: > My concerns here are: > - There's a lot of non-public projects out there not on "last 3" and > having older information is still useful. > - As was already stated (but on my mind right now as I hop over to see > if anyone made recipes for X/Y/Z) being able to answer the question > of "a long time ago someone made a recipe for X" is useful. Indeed, I can certainly see this as being useful. One thing that has been discussed a long time ago (I think it was Martin's suggestion) is to make recipes that only appear in older branches easier to find by showing them when you do a recipe search in master - perhaps at the end of the normal search results. > So I would ask for more visual indicators that something is old rather > than removing old things. Yep, as long as we define what the criteria is (for setting flags either manually or automatically) I think we can do that. Cheers Paul -- Paul Eggleton Intel System Software Products ___ Openembedded-architecture mailing list Openembedded-architecture@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
Re: [Openembedded-architecture] layers.openembedded.org - suggestion to remove obsolete layersHow
On Wednesday, 4 December 2019 4:23:18 AM NZDT akuster808 wrote: > On 12/2/19 6:28 AM, Mark Hatle wrote: > > On 12/1/19 10:52 PM, akuster808 wrote: > >> On 12/1/19 1:47 PM, Mark Hatle wrote: > >>> I've been looking through the layer index (master primarly), and I think > >>> we need > >>> to figure out a policy of when to remove a layer from master indexing. > >>> > >>> So I'd like to suggest the following policy: > >>> > >>> - For released branches, we do not remove layers from the index unless > >>> the layer > >>> URL is no longer valid. If it is not valid for more then 90 days, we > >>> should > >>> remove it. > >>> > >>> - For master branch, we use a series of tests to determine if the layer > >>> is still > >>> actively maintained and useful to users: > >>> > >>>1. Is the layer URL valid? If it has not been valid for more then 90 > >>> days, > >>> it should be removed. > >>> > >>>2. Does the layer claim to support any of the last three releases: the > >>> current (or planned release) or prior two releases? > >>> i.e. LAYERSERIES_COMPAT does not have thud, warrior or zeus in it. > >>> > >>> This means the layer has not been updated within the last 18 > >>> months, and > >>> should be removed. > >>> > >>> > >>> So why the suggestion above? > >>> > >>> The URL one should be obvious, if the layer can no longer be downloaded > >>> then it > >>> should be removed. We should be able to detect when the index hasn't been > >>> updated in 90 days. (We could consider dropping this to 45 or even 30 > >>> days.) > >>> > >>> Looking for the last 3 releases give people the opportunity to update > >>> their > >>> layers, or other people to find layers that may be of interest to them and > >>> submit updates. After 18 months, the layer is clearly no longer being > >>> maintained and will cause more confusion then actually helping people find > >>> something useful. (Also after 18 months of inactivity, there is a much > >>> higher > >>> likelihood of security vulnerabilities in the code.) > >> Can we get a "Delete " button so I take that action myself for my Layers? > >> > >> Why put that burden on you and Paul? > > I thought if you were the owner of the layer you could already remove it > > yourself. > > Nope. All I can see I can do is change its status to "inactive". I don't > know what that does. Are there doc's ? Right, there is no way for maintainers without admin rights to delete their own layers. Beyond the FAQ there aren't any docs, but if you're referring to the "Active/Inactive" field next to each maintainer that is just a flag that prevents that maintainer from showing up in the index entry (to anyone but admins and maintainers who can edit the layer). The idea was to make it easy to enable and disable maintainers who came and went and keep a record of who had been a maintainer in the past, but it's not really all that useful in practice. > > (You just have to login, otherwise contact Paul or I and we can do it > > pretty quickly.) > > This is what i want to avoid. I don't think this is so common an action that we need to allow maintainers to delete their own entries - we can take care of that. I'd rather there were some controls on layers getting deleted - if the original maintainer is dropping a layer, it might be decided in some cases that it's in everyone's best interest to have someone else pick it up immediately - depends on the layer and the situation. FWIW though I'm happy to grant yourself and others rights on the layer index as appropriate - in fact there are several others already have admin access (including Martin, Michael H, Nicolas, and Randy M) if we aren't around. One thing maintainers can do at any time is add a note on their layer to mark it as obsolete (with whatever text they want in the note). It might also be useful if there was a button there to send a message to the admins so people don't have to figure out who those are - that function could be implemented pretty easily. Cheers Paul -- Paul Eggleton Intel System Software Products ___ Openembedded-architecture mailing list Openembedded-architecture@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
Re: [Openembedded-architecture] layers.openembedded.org - suggestion to remove obsolete layers
On Tuesday, 3 December 2019 3:46:00 AM NZDT Mark Hatle wrote: > On 12/2/19 8:30 AM, Martin Jansa wrote: > > Even if the original URL isn't available anymore there might be some > > existing > > forks on github or elsewhere. > > But how would someone find it, and if they did find a fork how would they know > it's 'authentic', i.e. not maliciously tampered with? > > It's one thing if someone forks it and maintains it, I'd hope/expect in that > case they would try to take ownership on the layerindex. (This has certainly > happened in the past.) Indeed, and I'd like to encourage more of this in future. If unmaintained layers could be marked in some way, perhaps with a link to suggested next steps for those interested in picking up the reins, I think that process would be facilitated. > Yes, I'm only saying it is removed from master, not all of the index. > > In the current layer index version there is a switch. One is simply to stop > indexing a particular layer. This is not what I am suggesting. (As an aside, in case anyone wonders, we have not used this switch in the public index - it was added to the code for use in private instances.) > Instead what we could do is stop indexing/publishing new layer branches when > the > LAYERCOMPAT doesn't match oe-core. > > This would allow a layer that still exists but is no longer functional to be > disabled from master.. but if master and/or a new release branch is created it > can validate and then publish. (Once published it wouldn't be 'unpublished' > from a release branch. Also it wouldn't be unpublished/removed from master > unless the suggested policy happens, i.e. URL goes away or it hasn't been > updated in in 3 releases.) I think this seems reasonable. Additionally we can easily add flags with comments to layers that are considered unmaintained, that would show up in the web interface but could also be shown by Toaster, bitbake-layers and other places. (We do already have layer notes that can be used for this in a free-form manner, but an explicit flag would allow for easier filtering, possibly excluding such layers by default in certain contexts.) FYI, relating to this thread at Mark's suggestion I am already working on adding some fields to record when a layer was last successfully fetched, and when we last attempted to fetch it, so we can produce report on and/or automatically flag layers where the repo is gone for a long time (if we can't fetch a layer for e.g. 30 days, assuming it wasn't prevented from happening by some local issue I think we can safely assume it's gone). Cheers, Paul -- Paul Eggleton Intel System Software Products ___ Openembedded-architecture mailing list Openembedded-architecture@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
Re: [Openembedded-architecture] Heterogeneous System Proposal
Taking just this part of the discussion for now (since it is a small familiar aspect to me) On Tuesday, 3 December 2019 1:08:54 PM NZDT Mark Hatle wrote: > > You may want to consider this now. Adding system with the "nomachine" > > mode you mention is a nightmare scenario for parsing. Bitbake has a > > problem with knowing when to parse or not to parse since to 'skip' > > parsing a recipe, it needs to parse enough to know it should skip it. > > COMPATIBLE_MACHINE is implemented with the skip mechanism for example. > > If parsing is a concern (and I agree it is), you could probably get > > much further with a BBMASK approach to just the system-images recipe as > > then bitbake would know what it needs to parse without needing to read > > the recipes. > > The problem I have with BBMASK is that when a user tries to do something with > it, they don't get a reasonable error message. > > For instance if I BBMASK out bash, and the user does bitbake bash they get an > error. Instead of a 'bash is unavailable because ...'. The only practical way for that to work is the existing skip mechanism, since PN isn't always completely defined by the .bb filename, we need to actually parse to determine its value. gcc-cross and other parts of the toolchain are examples. BBMASK is such a usability pain - both in your scenario but also in the reverse where you have to hack it until it only masks exactly what you do want masked - that I think we would be best advised not to make any more use of it than we currently do, especially if it's only to save a bit of time parsing. Cheers Paul -- Paul Eggleton Intel System Software Products ___ Openembedded-architecture mailing list Openembedded-architecture@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
Re: [Openembedded-architecture] Interactive password prompts and the git fetcher
On Tuesday, 4 September 2018 7:54:55 PM NZST richard.pur...@linuxfoundation.org wrote: > On Tue, 2018-09-04 at 16:18 +1200, Paul Eggleton wrote: > > In the layer index / RRS code I have found that if it tries to use > > bitbake's git fetcher to access a git repository on github that has > > been renamed or gone private via http/https, I get prompted for a > > password interactively, presumably on the assumption that maybe if I > > authenticate I might be able to see the repo. Here's an example git > > command that will trigger it (trimmed from the actual command issued > > by the fetcher): > > > > git ls-remote http://github.com/symless/synergy.git > > > > This can be disabled by setting the environment variables GIT_ASKPASS > > to an empty string and GIT_TERMINAL_PROMPT to 0, which allows the > > fetch command to fail immediately. I could conceivably do this in the > > layer index / RRS scripts, but it occurred to me that perhaps this > > should be disabled in bitbake's git fetcher itself. We need to decide > > whether or not interactive password prompts should be allowed during > > a fetch operation (I'm guessing not, although I have a feeling this > > might be a "spacebar heating" situation for some). > > > > Thoughts? In particular, does anyone object to disabling such > > interactive prompts during fetching? > > I think this should be disabled in the fetcher. We don't support input > from stdin during builds and this isn't good behaviour. > > stdin tends to be redirected to /dev/null which would cause read errors > but we should also perhaps revisit and check that happens in all the > codepaths. It may be the way the layer index is using the parser code. > Could you check this also happens with that url in a normal build? So, you're right - if you run bitbake -c checkuri synergy (taking care to ensure PREMIRRORS and MIRRORS are cleared out first) it does fail immediately as it should. As you say, stdin being redirected likely negates the console input (based on the log it doesn't even try), and SSH_ASKPASS not being allowed through into the fetch environment prevents the graphical askpass that is set up in my environment from being triggered, so a bit of a false alarm. I guess I need to do something similar in the two places I call the fetcher within the layer index code. Apologies for the noise! Thanks, Paul -- Paul Eggleton Intel Open Source Technology Centre ___ Openembedded-architecture mailing list Openembedded-architecture@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
