[Re: [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos] On 02/04/2021 (Fri 23:14) Richard Purdie wrote:
> On Fri, 2021-04-02 at 13:15 -0400, Paul Gortmaker wrote: > > Next Steps: > > ----------- > > With this being a functional implementation, it seems like a good time to > > get other people looking at it. Ideally step #1 will be getting general > > agreement that this is something we need, something that is overdue, > > and that the implementation as shown here makes sense in the absence of > > any similar effort from anyone that does the same but in a better way. > > > > From there, we'll want more people not just looking at it, but testing it > > as well. I know I want to write a commit (script?) that will avoid any > > "transition tax" by prepopulating new repos with "old" already downloaded > > git objects where we can. And to add/do tests with my own popluated mirror > > and NO_NETWORK, and also try to ensure nothing in BB_SHALLOW gets upset, > > but I wasn't going to hold up starting a review of this any longer. > > > > I suspect I can get some co-workers using/testing it too, but Yocto gets > > used in a bunch of different ways by different groups, so we'll no doubt > > have to do some additional fixups to ensure everybody gets the benefits of > > this sharing. But I'm hopeful that when people see the benefits above, > > they'll pitch in to help take this the final mile by ensuring it works for > > their use case as well. > > > > I'm not too worried about pontificating out beyond that until we get past > > the acceptance/testing hurdles outlined above. So, please do have a read > > of the commits, kick the tires, put on your bikeshedding clothes and grab > > a brush, and lets see where it goes from here... > > > > https://github.com/paulgortmaker/poky/compare/reference-RC1 > > I've only briefly looked through this but the more I look, the more worried > I'm getting which is never a good sign. Well, firstly thanks for the quick response, and I hope you'll give it another look when you have more time, because I'm not quite convinced I've conveyed how it is to be used in a way that got through as intended. > I totally agree with the use case and agree we need to do something in this > area. I'm not sure the implementation looks right though, its clever but I > don't think its very usable to end users. Not sure who you consider "end users". I don't really consider myself or Bruce or anyone else crafting up a kernel recipe from scratch as an end user. But even if an "end user" wants to do that, everything here is opt-in. If you want to go clone 3G of stuff from mysuperkernel.org in your new recipe, then feel free. Nothing stops them. > The warning signs to me are: > > a) Needing new/confusing "fetch only" recipes Not sure how they are confusing - and not really new either; I stole the idea of zeroing out all the tasks from kernel-devsrc if I recall... > b) A large number of new options and variables to the fetcher I'd disagree with the characterization of "large" - but if you do have the time, please go look over the commit logs for each again. They all clearly indicate a necessary use case we don't handle well (or at all) -- and we'll have to solve them one way or another. I'm open to considering alternate solutions for any of them. > c) Needing recipes to change and people to migrate, potentially with scripts > between old and new It feels like we are looking at two different things. The users or "people" don't really migrate and the only people changing things would be a small subset of people who maintain kernel recipes - which is a small group to begin with. Recycling old git objects from existing downloads would most likey be done in a transparent and hands free way. > d) Variable namespacing needs work OK - if given specifics it can and will be addressed. > I'm very worried this confuses up the git fetcher code and nobody will be able > to tell what is going on any more :/. That kind of surprises me, to hear that after having looked at the shallow clone changes to the fetcher, but again - I'm listening - what can we change in the fetcher? What is off limits? The "pool" idea below doesn't address the fact that "--mirror" is simply not viable for some repos, and that is hard-coded into the fetcher. Similar for the other fetcher changes. And the "pool" idea is still going to have to teach the fetcher about some kind of "--reference" because that *has* to be added at fetch/clone time and not as some later afterthought post-fetch. I could simply take over do_fetch for the kernel and leave the fetcher untouched, but when the problems being solved weren't really kernel specific, that didn't seem like the right approach. > What I'd envisaged was git urls having something like a > "mainline-linux-kernel" > tag added in the url as a parameter and a table somewhere which meant the > checkout > for this would share git objects in a common pool. There would be a variable > mapping that name to git.kernel.org/pub/scm/linux/kernel/git somewhere but > that > should be all that is needed. I saw this idea floated in the earlier thread that Randy pointed me at, but I just don't see it being viable or the right solution even if one could make it work. How do you decide what lives in the pool? If it is just mainline, then it really isn't a pool. Are we only solving for the kernel, or are we making a solution for any repo that is unweildy? What triggers populating the pool? Do you stuff all mainline and -rt and all of stable in the pool? Or just chunks of rt and stable? If you aren't tracking what is in there via encapsulating chunks somehow, how do you expire stuff from the pool? If it is one giant blob covering mainline and linux-yocto objects, with no fetch-on-demand properties, then we are back to the same old 3G-downloads-from-the-mirror problem. How do you split that monster? The devil is in the details. > No, this wouldn't cover 100% of artefacts but it should cover a majority of > them > and be much simpler for users to comprehend. I haven't gone into this in > detail > and perhaps I'm missing some problem that prevents it? :/ Well, I've thought about this a bunch, and I didn't come up with a whole bunch of options to choose from that deliver what this delivers. A mainline-only kernel-specific non-pool "pool" would be better than nothing, but I'd still want to hear you flesh out how you thought it would look and operate before I went out and tried to guess what you had in mind. Also, I'm still confused by this use of "users" - what is here all happens transparently in the background and users aren't aware of it any more than they are aware of what the fetcher is currently doing today. > The other issue here is there are no tests. The bitbake fetcher code is one of > the few pieces of the project where we do have a fairly complete test suite > and > we don't add things there without tests (see bitbake-selftest and > lib/bb/tests/). Yes, of course; and I've been told that by several people in advance. But as you can imagine, I wasn't going to write tests without 1st floating the general concept as an RFC and seeing how that went. Anyway, thanks for the initial scan and I do hope you have a chance to revisit it in more detail and consider the various problems it solves. Paul. -- > > Cheers, > > Richard >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#9661): https://lists.yoctoproject.org/g/linux-yocto/message/9661 Mute This Topic: https://lists.yoctoproject.org/mt/81808149/21656 Group Owner: [email protected] Unsubscribe: https://lists.yoctoproject.org/g/linux-yocto/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
