Hi, I've a few things I've been pondering for a while and given the summit and OEDVM this week, I thought I'd sent out this email and some RFC patches.
Export variable flag (and unexport/network) ------------------------------------------- The patch sent as an RFC to the bitbake list has the details on this one, do we make the export and network flags more useful/useable at the cost of a 4% parsing speed hit? I'm torn, I just don't know. addpylib directive ------------------ The easier one is probably the "addpylib" directive. This would move the python module import code into a usable ABI within conf files such as layer.conf. There are patches to OE-Core and Bitbake on the mailing list as RFCs. The reasons for doing this are: * establish a clean API for adding python modules * drop what was originally a workaround from base.bbclass * allow potential to extend existing functionality to cover such modules accesses to the datastore in task hashes I chose to make it append to sys.path in the new implementation and stack on a first come, first in the queue approach so it will follow bblayers.conf only. We have too many ways for things to break and don't need that level of potential craziness IMO. I added BBIMPORT as a way to list the namespaces to import into bitbake. I did allow other namespaces to be imported as this is how time, os and sys are accessed by our anonymous code. I suspect we should perhaps only allow that from a different API but we can discuss that or improve in due course. There is a second patch, admittedly a bit thrown together which implements variable dependency information from the python functions. It basically runs the imported functions through our codeparser and then merges that dependency information with the datastore task signature/hash generation code. There are a number of things that need to be improved with it but it does prove the concept can work. I don't like the global variable, some of the functionality needs exporting to a function and there are probably data lifecycle issues there but it is good enough to start a discussion of the idea. The big win if we do this is allowing large chunks of package.bbclass and friends to be moved into lib/oe without having to worry about variable accesses. This should then lead to better memory usage and lower cache sizes. Right now it doesn't invalidate the cache on changes to the functions themselves, only the variable dependencies changing. That can be changed easily enough if desired. cache data changes ------------------ We now have the potential to add all of the data that makes up basehash (and hence taskhash) to the bitbake cache file. The upside would be much nicer information about why basehashes mismatch. The downsides are: * increased cache file size * slower parsing speed * slower load of the cache file back into memory I have patches which pickle the cache all in one go at the expense of the cache load progress bar. I did also develop a patch to support efficient incremental pickling of the data and patches which compress the cache file and add the basehash data. There is a tradeoff against various speeds to be made and I'm torn on what to do. We could make the data an option you could enable for debugging but often the situations are hard to reproduce so that may not be as useful as it first seems. These patches are in my poky-contrib branches. There are further ideas I have floating around but these ones are the ones there are patches out there for and that are probably ready/worthy of discussion. Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#1662): https://lists.openembedded.org/g/openembedded-architecture/message/1662 Mute This Topic: https://lists.openembedded.org/mt/95298341/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
