-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi Richard,
> Even ten recipes using this will show a degradation in parsing speed > and I do > get a lot of complaints when parsing slows down for any reason. The > user doesn't > expect this and it won't be visible what bitbake is doing (sitting at > 99% parsed > for a period). > > Also, the "tend to be small" implies someone will create a huge one > at some > point even if that is a bad idea for whatever reasons, I just know > how these > things end up going :(. I believe this is something that could be addressed in the documentation of the repo fetcher? Something along the lines of "Using the repo fetcher can increase the time spent in the parsing phase, as we need to clone and inspect the manifest repository. To avoid this, keep the size of your manifest repository at a minimum." When this is done, the dalay is barely noticable. We have a testsetup with 24 recipes and reasonable parsing speeds. > * Does it only clone a repo in the AUTOREV case? No, it also clones with fixed refspecs, as we want to ensure that each of the git repos referenced in the manifest also uses fixed refspecs. Otherwise, the sources in the background might change, while the recipe still (correctly) references the same revision of the manifest repo. > * Could it only obtain the manifest file somehow without a clone of > the repo? Unfortunately not really. git does not offer a way to list the content of a file without cloning it beforehand. The only other way I could imagine this to work would require us to keep the manifest file within the meta layer and do a somewhat "hacky" workaround: It is possible to run `repo init` on a local git folder. As such, something like this could work: 1) create temp folder (e.g. /tmp/repo-XXXXXXXXX) 2) run git init in temp folder 3) install manifest file into temp folder 4) add & commit 5) run repo init -u file://$temp-folder The repo manifest could even still be used outside of yocto, by using the metalayer repo as basis for the repo init: `repo init -u $metalayer-repo -m $path-to-manifest-file`. Not pretty, but it works. So if cloning within the parser is a dealbreaker for you, this option might be worth looking into. - -- With best regards Jasper Orschulko DevOps Engineer Tel. +49 30 58 58 14 265 Fax +49 30 58 58 14 999 [email protected] • • • • • • • • • • • • • • • • • • • • • • • • • • iris-GmbH infrared & intelligent sensors Schnellerstraße 1-5 | 12439 Berlin https://iris-sensing.com/ On Wed, 2021-11-10 at 16:33 +0000, Richard Purdie wrote: > On Wed, 2021-11-10 at 13:52 +0000, Jasper Orschulko wrote: > > Hi Richard, > > > > > When you say "fixed refspec", will that be a definitive sha > > > revision > > > or a tag? > > > We always force resolution of tags as they tend to cause problems > > > and > > > can change even if it is bad form. > > > > that's a good point. Actually, Martin and I have just been > > discussing > > this, as we noticed that this point actually got "lost" during our > > implementation. We are currently working on fixing this. Good to > > know > > how you handle this. I will keep you posted. > > Ok, it is good to be clear on that one. I know the fact we hit the > network for > tags does concern some but it really is the only way to handle them. > > > > This is potentially a big issue. Cloning operations during > > > parsing is > > > pretty > > > horrible. We'd not expect any thing being written out like that > > > during a parse. > > > It would probably work "ok" for one recipe but if you start > > > getting > > > the hundreds > > > of git recipes we have in some layers, it wouldn't scale if we > > > allowed that :(. > > > > > > Not sure what to recommend here but it is definitely problematic. > > > > Just to make sure that we are on the same page: This ONLY affects > > recipes which use the repo fetcher. And it ONLY clones the > > repository > > containing the repo manifest (which tend to be small in size). > > Correct, we are on the same page. This is still quite problematic as > the recipes > are meant to parse quickly and a repository clone is definitely not > expected. > > > So unless developers start using hundreds of repo-based recipes, > > which I > > find a very unlikely scenario, this should not be an issue. > > Even ten recipes using this will show a degradation in parsing speed > and I do > get a lot of complaints when parsing slows down for any reason. The > user doesn't > expect this and it won't be visible what bitbake is doing (sitting at > 99% parsed > for a period). > > Also, the "tend to be small" implies someone will create a huge one > at some > point even if that is a bad idea for whatever reasons, I just know > how these > things end up going :(. > > > Unfortunately, I don't see any other way to access the repo > > manifest > > file, as we need to calculate the commit hashes of the git repos > > referenced in the repo manifest file. Otherwise, it is impossible > > for > > us to determinate the necessity of an update when SRCREV = > > "${AUTOREV}". > > Some further questions: > > * Does it only clone a repo in the AUTOREV case? > * Could it only obtain the manifest file somehow without a clone of > the repo? > > > However, I see one potential improvement here. Currently the > > cloning of > > the manifest repo is done on a per-recipe basis. E.g. this means if > > we > > have 10 recipes inheriting a bbclass containing a repo fetcher, we > > will > > clone 10 identical manifest repos. We'll work on improving this. > > At least for wget or git, it is assumed that for a given url, there > would be one > tarball/clone and that there is locking in place to share it between > them. This > means you'll see do_fetch tasks for binutils, binutils-cross-XXX, > nativesdk- > binutils and binutils-native and one will block the others but the > fetch will > happen once and be shared between them. I guess with repo it may not > be as > simple as that but we should try and share what we can if possible. > > Cheers, > > Richard > > -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGNAZwACgkQYgqew07V MNUYpAf/foJRPcTJZTmDixuz5ZmI/4x9RB47XWupJrwy//1hHNJDPyc3iHqj79te 6T8EuqIYUtYGIE2FHzkbVwAiyNQYCODhh3Uc/LmNoo5/59XypLNVgf+UH7KowytX 1YptbB477Dl/hR39ul7oDxdC9D+j0gHjClzs8KmHt0iMzzc1nLvih9G8yXkYNYVH +0ovvlnpjHsC+WFYackEJe8CphUTFEn/EgffYtr/vl1SefRCzqs8aFB/gnhXKMjU 3zXH3DoeUZBdNahyEP9lAlbceM1AVoh9kNCXcVp/lUu1m7M3MVyxREMoeVfo22up J1kZetA1LqX02PwDacU4Fftoc502Dw== =BQ5W -----END PGP SIGNATURE-----
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#158170): https://lists.openembedded.org/g/openembedded-core/message/158170 Mute This Topic: https://lists.openembedded.org/mt/86840389/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
