Hi, Maxim Cournoyer <[email protected]> skribis:
> While attempting to bisect against the Linux kernel tree, the > performance of libgit2 quickly became problematic, to the point where > simply cloning the repo became a multiple hours affair, using upward to > 3 GiB of RAM for the clone and indexing of the objects (!) Did you confirm with a pure Guile-Git snippet that calls ‘clone’ that this is the behavior observed? > Given that: > > * the git CLI doesn't suffer from such poor performance; > * This kind of performance problem has been known for years in libgit2 > [0] with no fix in sight; This reports talks about 5x wall-clock time, which is obviously not great, but it doesn’t talk about memory usage, does it? It talks about SHAttered though; that’s a key consideration to make sure we’re doing an apples-to-apples comparison. > * other projects such as Cargo support using the git CLI and that > projects are using it for that reason [1]; Should we follow Cargo’s lead for packaging as well? :-) > Would it make sense to switch to use the git command directly instead of > calling into this libgit2 C library that ends up being slower? It would > provide a hefty speed-up when using 'guix refresh' or building new > packages fetched from git without substitutes, or using 'git-checkout', > etc. > > What do you think? I think that’s not an option. The level of integration we have in (guix git), (guix channels), etc. is not achievable by shelling out to ‘git’. "Philip McGrath" <[email protected]> skribis: > Along those lines, there’s an implementation of clone/checkout in pure Racket > (for the package manager) that could probably be ported to Guile relatively > easily. I’d expect libgit2 to be faster for the things that it supports, but > the Racket implementation does support shallow checkout, so it might pay off > if that skips a lot of work. > > Code: > https://github.com/racket/racket/blob/master/racket/collects/net/git-checkout.rkt > Docs: https://docs.racket-lang.org/net/git-checkout.html That sounds like a worthy avenue; support for shallow clones would already be an improvement. > (More broadly, I haven’t investigated performance issues, but my basic > inclination would be toward improving libgit2 over running the git > executable.) Same here. The way I see it, we could gradually move bits of Guile-Git to being pure Scheme. So perhaps the first step would be to provide a pure Scheme ‘clone’ based on the Racket code above? Thanks, Ludo’.
