Re: hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[Simon Tournier]

Hi Ludo,

On Mon, 18 Sept 2023 at 15:56, Ludovic Courtès  wrote:

> Anyhow, how about this plan:
>
>   1. Merge  with the hard Git
>  dependency.

Is #65866 fixing bug#63331 (Guile-GnuTLS/Git circular dependency) [1]?

Does the merge of #65866 lead to close #63331?  Because you wrote,

This patch series is a first step towards getting Git out of
derivation graphs when it’s only used to fetch source code
(origins with ‘git-fetch’), with the goal of fixing:

so...

>   2. When libgit2 1.7 with shallow clones is available in Guix, work on
>  a patch to use Guile-Git for clones and evaluate it.

...we could also suggest to continue and have a complete fix of #63331
before merging #65866.  It avoids to introduce a hard dependency which
will be difficult to remove and let the time for this evaluation of
libgit-2.1.7, no?

For what my opinion is worth, I have nothing for introducing a hard
dependency to Git but we have to be clear that 1. once introduced it
will hard to remove, 2. we will merge patches using faster Git
plumbing equivalent implementation and so 3. it will push out
Guile-Git.


> As I wrote, as an example, I don’t think that there could be a practical
> implementation of (guix git-authenticate) shelling out to ‘git’.

[...]

> PS: I don’t buy the “libgit2 will disappear from Guix” argument because
> it’s not a natural phenomenon that we’re observing but a willful
> construction.

As I wrote elsewhere, Git-Annex (or Magit) are shelling out to 'git',
IIRC.  Well, personally I do not consider that Git-Annex is slow or
that Git-Annex does not implement features as complex as (guix
git-authenticate).

After reading [2],

 I cannot imagine a viable implementation of things like ‘commit-closure’
 and ‘commit-relation’ from (guix git) done by shelling out to ‘git’.
 I’m quite confident this would be slow and brittle.

wolf came 3 days later [3] with a first rough implementation for
'commit-relation' using Git plumbing which is much more faster than
the one implemented with Guile-Git.  Even, speaking specifically about
'commit-relation', I challenge whoever to beat "git merge-base
--is-ancestor"; life is risky: I bet my round of beers in Brussels in
the next Guix Days. :-)  Reading the C implementation of
"merge-base.c" [4] and following the various procedures, it appears to
me impossible to beat it; bah using Guile-Git cumulates various
penalties from talking to libgit2 to the Garbage Collector of Guile.

The question does not appear to me if you buy it or not. :-)  The
question is instead: do we merge code that uses Git plumbing shelling
out that is faster than the current implementation using Guile-Git?

Cheers,
simon

1: https://issues.guix.gnu.org/issue/63331

2: bug#65720: Guile-Git-managed checkouts grow way too much
Ludovic Courtès 
Fri, 08 Sep 2023 19:08:05 +0200
id:87pm2s385m@gnu.org
https://issues.guix.gnu.org//65720
https://issues.guix.gnu.org/msgid/87pm2s385m@gnu.org
https://yhetil.org/guix/87pm2s385m@gnu.org

3: bug#65720: Guile-Git-managed checkouts grow way too much
wolf 
Mon, 11 Sep 2023 16:42:59 +0200
id:ZP8nc1m8rN_34XV-@ws
https://issues.guix.gnu.org//65720
https://issues.guix.gnu.org/msgid/ZP8nc1m8rN_34XV-@ws
https://yhetil.org/guix/ZP8nc1m8rN_34XV-@ws

4: 
https://github.com/git/git/blob/bda494f4043963b9ec9a1ecd4b19b7d1cd9a0518/builtin/merge-base.c#L103-L115

Re: hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[Ludovic Courtès]

Hello!

Maxim Cournoyer  skribis:

> Yeah, 'git gc' is lacking from libgit2.  I'm not against adding
> dependency on the real 'git' CLI, but at that point, as Simon mentioned,
> I see little reason to keep libgit2 around for much longer, given it
> performs worst than git CLI in every aspect.  I doubt forking processes
> on GNU/Linux would cause a performance hit compared to using libgit2,
> especially given how optimized git appears to be (at least compared to
> libgit2).

As I wrote, as an example, I don’t think that there could be a practical
implementation of (guix git-authenticate) shelling out to ‘git’.

Anyhow, how about this plan:

  1. Merge  with the hard Git
 dependency.

  2. When libgit2 1.7 with shallow clones is available in Guix, work on
 a patch to use Guile-Git for clones and evaluate it.

  3. Brainstorm on ways to address lack of GC support based on a closer
 analysis of disk usage for Guix’s cached checkouts.

Deal?

Ludo’.

PS: I don’t buy the “libgit2 will disappear from Guix” argument because
it’s not a natural phenomenon that we’re observing but a willful
construction.

Re: hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[Maxim Cournoyer]

Hi Ludovic,

Ludovic Courtès  writes:

> Hi!
>
> Maxim Cournoyer  skribis:
>
>> So given there's no technical reasons not to use libgit2, I'd use that
>> and keep the closure size down.
>
> For the record, that’s a 6% increase:
>
> $ guix size guix | tail -1
> total: 633.0 MiB
> $ guix size guix git-minimal | tail -1
> total: 675.7 MiB
>
> (Of course it all adds up; I’m not saying we can dismiss it.)

As Simon pointed out, it'd be more after wrapping 'git' with coreutils
and possible util-linux on its PATH.

> In the context of  plus the lack of
> GC in libgit2 discussed in , my
> inclination is to include that hard dependency on Git.
>
> That’s not a happy choice for me, but it has the advantage of solving
> two immediate problems.
>
> I would revisit it as soon as libgit2 supports shallow clones (which is
> coming, as you write)

This isn't "coming", it's already been released :-).

> and GC (or a workaround to that effect).  SHA256 may also soon be a
> requirement: we’ll need to be able to clone repos that use it.
>
> How does that sound?

Yeah, 'git gc' is lacking from libgit2.  I'm not against adding
dependency on the real 'git' CLI, but at that point, as Simon mentioned,
I see little reason to keep libgit2 around for much longer, given it
performs worst than git CLI in every aspect.  I doubt forking processes
on GNU/Linux would cause a performance hit compared to using libgit2,
especially given how optimized git appears to be (at least compared to
libgit2).

So, I think we need to agree on the future of libgit2 in the big picture
and decide to invest in it or let it in favor of just using git.

-- 
Thanks,
Maxim

Re: hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[Simon Tournier]

Hi,

On Thu, 14 Sept 2023 at 18:51, Ludovic Courtès  wrote:

> For the record, that’s a 6% increase:
>
> --8<---cut here---start->8---
> $ guix size guix | tail -1
> total: 633.0 MiB
> $ guix size guix git-minimal | tail -1
> total: 675.7 MiB
> --8<---cut here---end--->8---
>
> (Of course it all adds up; I’m not saying we can dismiss it.)
>
> In the context of  plus the lack of
> GC in libgit2 discussed in , my
> inclination is to include that hard dependency on Git.

And considering bug#65924 [1], it is not 6% but more.  Because
currently git-minimal is broken and coreutils and potentially
util-linux would also be part of the closure.

--8<---cut here---start->8---
$ guix size guix | tail -1
total: 633.0 MiB
$ guix size guix git-minimal coreutils | tail -1
total: 692.7 MiB
$ guix size guix git-minimal coreutils util-linux | tail -1
total: 706.6 MiB
--8<---cut here---end--->8---

Therefore. it is 9.4% or worse 11.6%.

1: bug#65924: git searches coreutils and util-linux commands in PATH
Maxim Cournoyer 
Wed, 13 Sep 2023 14:00:09 -0400
id:87fs3iuf6e@gmail.com
https://yhetil.org/guix/87fs3iuf6e@gmail.com
https://issues.guix.gnu.org/msgid/87fs3iuf6e@gmail.com


> That’s not a happy choice for me, but it has the advantage of solving
> two immediate problems.

Three. :-)

Once git-minimal and plumbing Git commands around, some slow
procedures using libgit2 will be replaced by faster ones.  And I also
have in mind some Git repository normalization that differs from SWH;
having plain Git commands would also easy that part.

Just to point that the introduction of git-minimal as hard dependency
of Guix means that libgit2 will be slowly removed from the picture,
IMHO.

Cheers,
simon

Re: hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[Ludovic Courtès]

Hi!

Maxim Cournoyer  skribis:

> So given there's no technical reasons not to use libgit2, I'd use that
> and keep the closure size down.

For the record, that’s a 6% increase:

--8<---cut here---start->8---
$ guix size guix | tail -1
total: 633.0 MiB
$ guix size guix git-minimal | tail -1
total: 675.7 MiB
--8<---cut here---end--->8---

(Of course it all adds up; I’m not saying we can dismiss it.)

In the context of  plus the lack of
GC in libgit2 discussed in , my
inclination is to include that hard dependency on Git.

That’s not a happy choice for me, but it has the advantage of solving
two immediate problems.

I would revisit it as soon as libgit2 supports shallow clones (which is
coming, as you write) and GC (or a workaround to that effect).  SHA256
may also soon be a requirement: we’ll need to be able to clone repos
that use it.

How does that sound?

Ludo’.

Re: hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[Ludovic Courtès]

Hi,

Josselin Poiret  skribis:

> My opinion is that the preferred API for Git is still the UNIX one via
> plumbing commands.  Anything else is trying to catch up to it, and then
> we get into this conundrum that we want to do everything in Scheme, but
> we're unable to do it as well as Git itself.  If I had to choose, a
> Guile library wrapping the Git commands would be the best, especially
> since we're managing long-living checkouts, something libgit2 doen't
> seem too interested in.

I have mixed feelings here.  Clearly, I don’t think a Unix command can
ever be as rich and efficient as a “proper library”.

Are alternative Git implementations doomed to always try to catch up?
My intuition would be “no”, because not so much changes in Git as an
on-disk format and protocol.

There is one big change coming up though: SHA256 support (now officially
supported in Git).  Is it being discussed in libgit2?

Maxim Cournoyer  skribis:

> Josselin Poiret  writes:

[...]

>> There's still the `git gc` problem though.
>
> It's klunky, but a workaround is to locally clone the checkout anew
> using libgit2, as suggested here [0].
>
> [0]  https://github.com/libgit2/libgit2/issues/3247#issuecomment-486585944

That doesn’t work, at least with libgit2 1.3.2:

  https://issues.guix.gnu.org/65720#7

> We could also try to contribute to libgit2 toward adding proper
> support for a 'gc' action.

I share this sentiment: if we’re gonna depend on it, we’ve gotta invest
in it.  We’re benefiting from it so we shouldn’t be mere consumers.

I have to admit I don’t see myself doing it right now, but I would
definitely encourage others to do so.

Now, as a corollary to what I wrote above: if we don’t invest in it, we
should be prepared to drop it.

Ludo’.

Re: hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[wolf]

On 2023-09-12 08:56:15 -0400, Maxim Cournoyer wrote:
> Hi,
> 
> Josselin Poiret  writes:
> 
> > Hi,
> >
> > Maxim Cournoyer  writes:
> >
> >> That's no longer true, as of libgit2 v1.7.0 [0].  I already have it
> >> packaged in a branch, but I need to iron out dependent breakages.
> >>
> >> [0]  https://github.com/libgit2/libgit2/releases/tag/v1.7.0
> >>
> >> So given there's no technical reasons not to use libgit2, I'd use that
> >> and keep the closure size down.
> >
> > There's still the `git gc` problem though.
> 
> It's klunky, but a workaround is to locally clone the checkout anew using
> libgit2, as suggested here [0].

While the comment does suggest a workaround, I would recommend to read the
comment right below it before using it.  I am not sure if Guix would be affected
by those mentioned concurrency issues, but sounds like something that needs to
be understood in detail.

> We could also try to contribute to libgit2 toward adding proper support for a
> 'gc' action.
> 
> [0]  https://github.com/libgit2/libgit2/issues/3247#issuecomment-486585944
> 
> -- 
> Thanks,
> Maxim
> 

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.


signature.asc
Description: PGP signature

Re: hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[Maxim Cournoyer]

Hi,

Josselin Poiret  writes:

> Hi,
>
> Maxim Cournoyer  writes:
>
>> That's no longer true, as of libgit2 v1.7.0 [0].  I already have it
>> packaged in a branch, but I need to iron out dependent breakages.
>>
>> [0]  https://github.com/libgit2/libgit2/releases/tag/v1.7.0
>>
>> So given there's no technical reasons not to use libgit2, I'd use that
>> and keep the closure size down.
>
> There's still the `git gc` problem though.

It's klunky, but a workaround is to locally clone the checkout anew
using libgit2, as suggested here [0].  We could also try to contribute
to libgit2 toward adding proper support for a 'gc' action.

[0]  https://github.com/libgit2/libgit2/issues/3247#issuecomment-486585944

-- 
Thanks,
Maxim

Re: hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[Josselin Poiret]

Hi,

Maxim Cournoyer  writes:

> That's no longer true, as of libgit2 v1.7.0 [0].  I already have it
> packaged in a branch, but I need to iron out dependent breakages.
>
> [0]  https://github.com/libgit2/libgit2/releases/tag/v1.7.0
>
> So given there's no technical reasons not to use libgit2, I'd use that
> and keep the closure size down.

There's still the `git gc` problem though.

My opinion is that the preferred API for Git is still the UNIX one via
plumbing commands.  Anything else is trying to catch up to it, and then
we get into this conundrum that we want to do everything in Scheme, but
we're unable to do it as well as Git itself.  If I had to choose, a
Guile library wrapping the Git commands would be the best, especially
since we're managing long-living checkouts, something libgit2 doen't
seem too interested in.

Best,
-- 
Josselin Poiret


signature.asc
Description: PGP signature

Re: hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[Simon Tournier]

Hi Vagrant,

On Mon, 11 Sep 2023 at 12:35, Vagrant Cascadian  wrote:

> What about making git an optional dependency, and only calling out to
> "git gc" if git is available in PATH?

Somehow, that’s more or less the case, IIUC,

--8<---cut here---start->8---
15 candidates:
./build/android-repo.scm:57:  (invoke git-repo-command "init" "-u" 
manifest-url "-b" manifest-revision
./build/android-repo.scm:59:  (invoke git-repo-command "sync" "-c" 
"--fail-fast" "-v" "-j"
./build/git.scm:55:  (invoke git-command "init" "--initial-branch=main")
./build/git.scm:56:  (invoke git-command "remote" "add" "origin" url)
./build/git.scm:58:  (invoke git-command "checkout" "FETCH_HEAD")
./build/git.scm:62:(invoke git-command "fetch" "origin")
./build/git.scm:63:(invoke git-command "checkout" commit)))
./build/git.scm:66:(invoke git-command "submodule" "update" "--init" 
"--recursive")
./git-download.scm:175: (invoke "git" "-C" #$output 
"init")
./git-download.scm:176: (invoke "git" "-C" #$output 
"config" "--local"
./git-download.scm:178: (invoke "git" "-C" #$output 
"config" "--local"
./git-download.scm:180: (invoke "git" "-C" #$output 
"add" ".")
./git-download.scm:181: (invoke "git" "-C" #$output 
"commit" "-am" "init")
./git-download.scm:182: (invoke "git" "-C" #$output 
"read-tree" "--empty")
./git-download.scm:183: (invoke "git" "-C" #$output 
"reset" "--hard")
--8<---cut here---end--->8---

An “optional dependency”, is it not already the case?

I read hard-dependency as the idea behind a change like [1].  For
instance, something like:

--8<---cut here---start->8---
diff --git a/guix/self.scm b/guix/self.scm
index 81a36e007f..41c5f40786 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -68,6 +68,7 @@ (define %packages
   ("gzip"   . ,(ref 'compression 'gzip))
   ("bzip2"  . ,(ref 'compression 'bzip2))
   ("xz" . ,(ref 'compression 'xz))
+  ("git-minimal". ,(ref 'version-control 'git-minimal))
   ("po4a"   . ,(ref 'gettext 'po4a))
   ("gettext-minimal". ,(ref 'gettext 'gettext-minimal))
   ("gcc-toolchain"  . ,(ref 'commencement 'gcc-toolchain))
@@ -825,6 +826,9 @@ (define* (compiled-guix source #:key
   (define guile-lzma
 (specification->package "guile-lzma"))

+  (define git
+(specification->package "git-minimal"))
+
--8<---cut here---end--->8---

In the context of the proposal patch#65866 [1], this hard-dependency
makes sense.  From my point of view, once we have git-minimal as a
hard-dependency, I do not see the point to keep slower Git operations
using libgit2; as ’commit-relation’ for one example.

But maybe I am missing something.

Cheers,
simon


1: [bug#65866] [PATCH 5/8] build: Add dependency on Git.
Ludovic Courtès 
Mon, 11 Sep 2023 16:25:23 +0200
id:4eca94501c2c1e9986e1f718eeccb3eb9276dcd4.1694441831.git.l...@gnu.org
https://issues.guix.gnu.org//65866
https://issues.guix.gnu.org/msgid/4eca94501c2c1e9986e1f718eeccb3eb9276dcd4.1694441831.git.l...@gnu.org
https://yhetil.org/guix/4eca94501c2c1e9986e1f718eeccb3eb9276dcd4.1694441831.git.l...@gnu.org

Re: hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[Csepp]

Vagrant Cascadian  writes:

> [[PGP Signed Part:Undecided]]
> On 2023-09-11, Simon Tournier wrote:
>> On Mon, 11 Sep 2023 at 16:23, Ludovic Courtès  wrote:
>>> Note that the patch series adds a hard dependency on Git.
>>> This is because the existing ‘git-fetch’ code depends on Git,
>>> which is itself motivated by the fact that Git supports
>>> shallow clones and libgit2/Guile-Git doesn’t.
> ...
>> Personally, I do not have a strong opinion about the Big Plan™.  I note
>> that the introduction of Git as a hard dependency is a slippery slope
>> considering the current state of libgit2.  Here, it starts with “git
>> clone”, then “git gc” (unsupported by libgit2) is also in the pipes
>> (#65720 [1]).
>
> What about making git an optional dependency, and only calling out to
> "git gc" if git is available in PATH? Maybe possible also with shallow
> clones?
>
> Then you have the best/worst of both worlds! Speaking to the worst, you
> have at least two disparate codepaths for a seemingly similar operation,
> and that might be annoying...
>
> live well,
>   vagrant
>
> [[End of PGP Signed Part]]

For what it's worth, I wrote a small (incomplete) tool for some commit
analysis that used specific --format arguments that were easy to parse.
It's not especially difficult.

Re: hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[Vagrant Cascadian]

On 2023-09-11, Simon Tournier wrote:
> On Mon, 11 Sep 2023 at 16:23, Ludovic Courtès  wrote:
>> Note that the patch series adds a hard dependency on Git.
>> This is because the existing ‘git-fetch’ code depends on Git,
>> which is itself motivated by the fact that Git supports
>> shallow clones and libgit2/Guile-Git doesn’t.
...
> Personally, I do not have a strong opinion about the Big Plan™.  I note
> that the introduction of Git as a hard dependency is a slippery slope
> considering the current state of libgit2.  Here, it starts with “git
> clone”, then “git gc” (unsupported by libgit2) is also in the pipes
> (#65720 [1]).

What about making git an optional dependency, and only calling out to
"git gc" if git is available in PATH? Maybe possible also with shallow
clones?

Then you have the best/worst of both worlds! Speaking to the worst, you
have at least two disparate codepaths for a seemingly similar operation,
and that might be annoying...

live well,
  vagrant


signature.asc
Description: PGP signature

Re: hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[Maxim Cournoyer]

Hi wolf,

wolf  writes:

[...]

>> Personally, I do not have a strong opinion about the Big Plan™.  I note
>> that the introduction of Git as a hard dependency is a slippery slope
>> considering the current state of libgit2.  Here, it starts with “git
>> clone”, then “git gc” (unsupported by libgit2) is also in the pipes
>> (#65720 [1]).  And after timing, I am almost sure that many operations
>> using Guile-Git will be slower than their plain Git counter-parts.  And
>> we will start to parse the output of ’git’ plumbing commands.
>
> If you don't mind me asking, why is that so problematic approach?  Git's
> plumbing commands are intended to be used in scripts, so I am unsure what the
> problem is.

In the grand scheme of things (pun intended), we'd like every
programming to be feasible via nice Scheme APIs, which is what Guile-Git
provides to work with git repositories.  The appeal is to have a single
language to rule them all, reducing friction among Guix contributors.
The alternative here is to have an API reduced to invoking system
commands with string arguments, which is less expressive and lacks
elegance.

-- 
Thanks,
Maxim

Re: hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[Maxim Cournoyer]

Hi,

Simon Tournier  writes:

> Hi,
>
> On Mon, 11 Sep 2023 at 16:23, Ludovic Courtès  wrote:
>
>> Note that the patch series adds a hard dependency on Git.
>> This is because the existing ‘git-fetch’ code depends on Git,
>> which is itself motivated by the fact that Git supports
>> shallow clones and libgit2/Guile-Git doesn’t.

That's no longer true, as of libgit2 v1.7.0 [0].  I already have it
packaged in a branch, but I need to iron out dependent breakages.

[0]  https://github.com/libgit2/libgit2/releases/tag/v1.7.0

So given there's no technical reasons not to use libgit2, I'd use that
and keep the closure size down.

-- 
Thanks,
Maxim

Re: hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[Simon Tournier]

Re,

On Mon, 11 Sep 2023 at 17:17, Simon Tournier  wrote:
> On Mon, 11 Sep 2023 at 16:23, Ludovic Courtès  wrote:
>
>> Note that the patch series adds a hard dependency on Git.
>> This is because the existing ‘git-fetch’ code depends on Git,
>> which is itself motivated by the fact that Git supports
>> shallow clones and libgit2/Guile-Git doesn’t.

[...]

> Personally, I do not have a strong opinion about the Big Plan™.  I note
> that the introduction of Git as a hard dependency is a slippery slope
> considering the current state of libgit2.  Here, it starts with “git
> clone”, then “git gc” (unsupported by libgit2) is also in the pipes
> (#65720 [1]).  And after timing, I am almost sure that many operations
> using Guile-Git will be slower than their plain Git counter-parts.  And
> we will start to parse the output of ’git’ plumbing commands.  Slippery
> slope for pushing Guile-Git out, no?

For example, having plain Git command with shallow clone would allow to
save resource when cloning the first time the Guix checkout cache.

Compare,

--8<---cut here---start->8---
$ git clone https://git.savannah.gnu.org/git/guix.git guix-full
Cloning into 'guix-full'... 

   
remote: Counting objects: 696917, done. 

   
remote: Compressing objects: 100% (143179/143179), done.

   
remote: Total 696917 (delta 552872), reused 696909 (delta 552867)   

   
Receiving objects: 100% (696917/696917), 347.14 MiB | 29.31 MiB/s, done.

   
Resolving deltas: 100% (552872/552872), done.
--8<---cut here---end--->8---

and,

--8<---cut here---start->8---
$ git clone --shallow-since=2019-04-30 
https://git.savannah.gnu.org/git/guix.git guix-oldest
Cloning into 'guix-oldest'...
remote: Counting objects: 426879, done.
remote: Compressing objects: 100% (87246/87246), done.
remote: Total 426879 (delta 342111), reused 423970 (delta 339518)
Receiving objects: 100% (426879/426879), 259.75 MiB | 11.26 MiB/s, done.
Resolving deltas: 100% (342111/342111), done.
Checking connectivity: 426863, done.
--8<---cut here---end--->8---

(Here, 2019-04-30 is the date that contains the %oldest-possible-commit
"6298c3ffd9654d3231a6f25390b056483e8f407c" v1.0.)

Well, ’shallow’ probably implies an overload on server side.  But that
is far less than the bits it saves: 87.39 MiB (= 347.14 - 259.75).  It
saves something like 25% to download, if I read correctly.  I let you do
some maths for the improvement you will get. :-)

And there is no integration with Guile-Git, if I read correctly.  The
job is done by the procedure ’clone*’ from the module (guix git).  This
procedure call the Guile-Git procedure ’clone’ which could drop-in
replaced by (invoke git-command "clone --shallow-since=2019-04-30" …).


Cheers,
simon

Re: hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[wolf]

On 2023-09-11 17:17:24 +0200, Simon Tournier wrote:
> Hi,
> 
> On Mon, 11 Sep 2023 at 16:23, Ludovic Courtès  wrote:
> 
> > Note that the patch series adds a hard dependency on Git.
> > This is because the existing ‘git-fetch’ code depends on Git,
> > which is itself motivated by the fact that Git supports
> > shallow clones and libgit2/Guile-Git doesn’t.
> 
> Going this path, I appears to me worth to revisit the proposal:
> 
> RFC: libgit2 is slow/inefficient; switch to git command?
> Maxim Cournoyer 
> Mon, 21 Nov 2022 21:21:02 -0500
> id:87cz9fpw4x@gmail.com
> https://yhetil.org/guix/87cz9fpw4x@gmail.com
> https://lists.gnu.org/archive/html/guix-devel/2022-11
> 
> I know it is not an option for now to parse the output of ’git’ commands
> in order to keep the features of (guix git), (guix channels), etc.
> 
> However, this discussion was mentioning an implementation of
> clone/checkout in pure Racket supporting shallow checkout.  Considering
> the current level of integration, I thought the next Big Plan™ was to
> gradually move bits of Guile-Git to being pure Scheme, maybe based on
> the Racket implementation of ’clone’ as a starting point.
> 
> Personally, I do not have a strong opinion about the Big Plan™.  I note
> that the introduction of Git as a hard dependency is a slippery slope
> considering the current state of libgit2.  Here, it starts with “git
> clone”, then “git gc” (unsupported by libgit2) is also in the pipes
> (#65720 [1]).  And after timing, I am almost sure that many operations
> using Guile-Git will be slower than their plain Git counter-parts.  And
> we will start to parse the output of ’git’ plumbing commands.

If you don't mind me asking, why is that so problematic approach?  Git's
plumbing commands are intended to be used in scripts, so I am unsure what the
problem is.

I cannot recall a single instance when some tooling I have at home or wrote for
work stopped working due to git changing.  I guess there likely are such
instances, but would be interested in examples if someone has a list.

> Slippery slope for pushing Guile-Git out, no?

Guile-git does not seem to be in the best place currently, when I was putting
together the script I sent to #65720, I tried to use the info manual, and the
best way to describe it is "incomplete".  Some behaviors are... surprising.

Arguably that is the impression I got based on one morning of trying to use it,
so it is probably inaccurate description and lacks some details.

> 
> And I do not speak about the closure.  Is it possible to extract the
> command ’git-clone’ from git-minimal?  It would reduce the size, no?
> 
> 
> Cheers,
> simon
> 
> 1: https://issues.guix.gnu.org/65720
> 

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.


signature.asc
Description: PGP signature

hard dependency on Git? (was bug#65866: [PATCH 0/8] Add built-in builder for Git checkouts)

[Simon Tournier]

Hi,

On Mon, 11 Sep 2023 at 16:23, Ludovic Courtès  wrote:

> Note that the patch series adds a hard dependency on Git.
> This is because the existing ‘git-fetch’ code depends on Git,
> which is itself motivated by the fact that Git supports
> shallow clones and libgit2/Guile-Git doesn’t.

Going this path, I appears to me worth to revisit the proposal:

RFC: libgit2 is slow/inefficient; switch to git command?
Maxim Cournoyer 
Mon, 21 Nov 2022 21:21:02 -0500
id:87cz9fpw4x@gmail.com
https://yhetil.org/guix/87cz9fpw4x@gmail.com
https://lists.gnu.org/archive/html/guix-devel/2022-11

I know it is not an option for now to parse the output of ’git’ commands
in order to keep the features of (guix git), (guix channels), etc.

However, this discussion was mentioning an implementation of
clone/checkout in pure Racket supporting shallow checkout.  Considering
the current level of integration, I thought the next Big Plan™ was to
gradually move bits of Guile-Git to being pure Scheme, maybe based on
the Racket implementation of ’clone’ as a starting point.

Personally, I do not have a strong opinion about the Big Plan™.  I note
that the introduction of Git as a hard dependency is a slippery slope
considering the current state of libgit2.  Here, it starts with “git
clone”, then “git gc” (unsupported by libgit2) is also in the pipes
(#65720 [1]).  And after timing, I am almost sure that many operations
using Guile-Git will be slower than their plain Git counter-parts.  And
we will start to parse the output of ’git’ plumbing commands.  Slippery
slope for pushing Guile-Git out, no?

And I do not speak about the closure.  Is it possible to extract the
command ’git-clone’ from git-minimal?  It would reduce the size, no?


Cheers,
simon

1: https://issues.guix.gnu.org/65720