On Thu, 2023-09-14 at 20:51 +0200, Alexander Kanavin wrote: > On Thu, 14 Sept 2023 at 14:56, Richard Purdie > <richard.pur...@linuxfoundation.org> wrote: > > For the task signatures, we need to think about some questions. If I > > make a change locally, can I query how much will rebuild and how much > > will be reused? There is bitbake --dry-run but perhaps it is time for a > > an option (or dedicated separate command?) to give some statistics > > about what bitbake would do? How much sstate would be reused? > > > > That then logically leads into the questions, can we tell what has > > changed? Why isn't my sstate being reused? For that we perhaps should > > define some existing scenarios where it is currently very difficult to > > work this out and then work out how we can report that information to > > the user. These could become test cases? > > So I think there are two questions here that the tools should answer: > > 1. If I would run a build, what would be missing in the cache and need > to be built? The missing cache objects are in a dependency hierarchy, > so only those missing objects with no dependecies on other missing > objects would be printed. That should be comparatively easy to add as > bitbake already does those checks all the time.
Right, what we lack is a way for the user to ask this and see the result easily. As you say, bitbake can do this already. > Is there something else that's easily done and useful to print? I think there is also the scenario of: "I've run a build and have an existing TMPDIR and stamp info. I've now pulled in a change. How much is going to rebuild and more importantly *why*?" This is different to a remote sstate situation as you have the stamp info of the previous build already there to compare against. > 2. Then there's the question of *why* they are missing, which is > harder to answer. If, say, curl:do_package is not in the cache, then > the tool would have to walk the cache tree (I/O heavy operation as > there is no index), make a list of all curl:do_package objects that > are there, and do a recursive bitbake-diffsig (going up the task tree) > on them vs the one we want. Then print them starting with the newest. > Something like: > > Existing cache objects are not suitable because: > <object id 1> was built on <date> and has a mismatching SRCREV > <object id 2> was built on <earlier date> and has a different do_compile() In theory you can do an: ls sstate/*/*/sstate:curl:corei7-64-poky-linux:<PV>:<PR>:corei7-64:10:*_package.tar.zst and get a list of possible objects. Some key information was put into the cache file names for this reason. Obviously this becomes much trickier when the sstate is remote over http though. "newest" is problematic in this context as you really want the closest match. There is no concept of build date in the cache as it often isn't relevant (building an old release for example). The only point a date/time is used is for cleaning out the cache for files which haven't been accessed in a long time. The tools are already supposed to support doing this with local file sstate sources, they just do a bad job at getting the diffs right. One intent of this work item was to try and understand why they don't work and address that so at least for filesystem sstate mirrors, you can get better results. I don't know how we solve the remote http issue as yet. > > One of the big problems in the past was that we lost much of the hash > > information after parsing completed. This meant that if the hashes then > > didn't match, we couldn't tell why as the original computation was > > lost. I did some work on allowing us to retain more of the information > > so that we didn't have to recompute it every time to be able to do > > processing with it. I have to admit I've totally lost track of where I > > got to with that. > > Here's an idea I can't get out of my head. Right now, the cache is > simply an amorphous mass of objects, with no information regarding how > they were created. How about storing complete build confgurations as > well into the same directory? There would be a dedicated, separate > area for each configuration that placed objects into the cache, > containing: > - list of layers and revisions > - config template used > - complete content of build/conf > - bitbake invocation (e.g. targets and prefixed variables like MACHINE etc.) > - complete list of sstate objects that were produced as a result, so > they can be checked for existence > > This would be written into the cache dir at the very end of the build > when everything else is already there. I'm not sure this helps as much as you'd like. For example I build core-image-sato-sdk on the autobuilder and populate this but you want to build core-image-sato locally. There would be no info here that would help with information about what core-image-sato would need, despite the fact we know one is a derivative of the other and there would be a near full cache for it. Taking your example above and curl:do_package, how would the list of information above tell us why it wasn't and what the configuration difference was? It can tell us a set of different configs which result in hashes but not how an unknown hash may differ from the existing configs. The best you could do is iterate the configs and see which one got you closest to what you have but I suspect that may be even more expensive that the sstate search mentioned above, indexing of http aside. I can see it helping for some limited debug scenarios but the information you want is really mostly what is already being encoded into the sstate filename. Running with that thought, if that information is what we really need, should be we indexing it and sharing through some kind of hash equivalence like service to provide the index? > Right now, everyone sets up their own builds first, then points > local.conf or site.conf to the cache, and hopes for the best regarding > hit rates. Having stored build configs would allow inverting the > workflow, so that you first ask from the cache what it can provide > (e.g. it can provide mickledore or kirkstone core-image-minimal for > qemux86, and that's exactly what you want as a starting point), then > you use the build config stored in the cache to set up a build, and > run it - and that would guarantee complete sstate reuse and getting to > a functional image as soon as possible. Kind of like binary distro, > but implemented with sstate. Effectively those "configs" are what the eSDK is, you're just proposing a server side set of them rather than a built copy. In many ways that could be useful way of possibly rethinking the way the eSDK works. So in that sense I like the idea, we should just be clear the problem it solves as it doesn't solve the "why is my build not using the cache" question specifically but it potentially solves other issues. Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#1754): https://lists.openembedded.org/g/openembedded-architecture/message/1754 Mute This Topic: https://lists.openembedded.org/mt/101356419/21656 Group Owner: openembedded-architecture+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
