Re: [PATCH] t5310-pack-bitmaps: fix bogus 'pack-objects to file can use bitmap' test

[Kirill Smelkov]

On Mon, Aug 27, 2018 at 07:04:52PM -0400, Jeff King wrote:
> On Mon, Aug 27, 2018 at 10:22:46AM +0000, Kirill Smelkov wrote:
> 
> > A minor comment from outside observer: running tests under something
> > like
> > 
> > -e and -o pipefail
> > 
> > would automatically catch the mistake in the first place. Maybe `-o
> > pipefail` is bashism (I had not checked), but `git grep " | " t/` shows
> > there are a lot of pipelines being used, and thus similar errors might be
> > silently resting there. Something like -o pipefail would catch all such
> > problems automatically.
> 
> Yes, "pipefail" is a bash-ism that we can't rely on.
> 
> I will say that I have been bitten before by "set -e -o pipefail" and
> its subtle handling of SIGPIPE. Try this:
> 
>   set -e -o pipefail
>   yes | head

Thanks for the information. Oh well...

Re: [PATCH] t5310-pack-bitmaps: fix bogus 'pack-objects to file can use bitmap' test

[Kirill Smelkov]

On Tue, Aug 14, 2018 at 01:47:21PM +0200, SZEDER Gábor wrote:
> The test 'pack-objects to file can use bitmap' added in 645c432d61
> (pack-objects: use reachability bitmap index when generating
> non-stdout pack, 2016-09-10) is silently buggy and doesn't check what
> it's supposed to.
>
> In 't5310-pack-bitmaps.sh', the 'list_packed_objects' helper function
> does what its name implies by running:
>
>   git show-index <"$1" | cut -d' ' -f2
>
> The test in question invokes this function like this:
>
>   list_packed_objects packa.objects &&
>   list_packed_objects packb.objects &&
>   test_cmp packa.objects packb.objects
>
> Note how these two callsites don't specify the name of the pack index
> file as the function's parameter, but redirect the function's standard
> input from it.  This triggers an error message from the shell, as it
> has no filename to redirect from in the function, but this error is
> ignored, because it happens upstream of a pipe.  Consequently, both
> invocations produce empty 'pack{a,b}.objects' files, and the
> subsequent 'test_cmp' happily finds those two empty files identical.
>
> Fix these two 'list_packed_objects' invocations by specifying the pack
> index files as parameters.  Furthermore, eliminate the pipe in that
> function by replacing it with an &&-chained pair of commands using an
> intermediate file, so a failure of 'git show-index' or the shell
> redirection will fail the test.
>
> Signed-off-by: SZEDER Gábor 
> ---
>  t/t5310-pack-bitmaps.sh | 7 ---
>  1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh
> index 6ee4d3f2d9..557bd0d0c0 100755
> --- a/t/t5310-pack-bitmaps.sh
> +++ b/t/t5310-pack-bitmaps.sh
> @@ -9,7 +9,8 @@ objpath () {
>
>  # show objects present in pack ($1 should be associated *.idx)
>  list_packed_objects () {
> - git show-index <"$1" | cut -d' ' -f2
> + git show-index <"$1" >object-list &&
> + cut -d' ' -f2 object-list
>  }
>
>  # has_any pattern-file content-file
> @@ -204,8 +205,8 @@ test_expect_success 'pack-objects to file can use bitmap' 
> '
>   # verify equivalent packs are generated with/without using bitmap index
>   packasha1=$(git pack-objects --no-use-bitmap-index --all packa 
>packbsha1=$(git pack-objects --use-bitmap-index --all packb  &&
> - list_packed_objects packa.objects &&
> - list_packed_objects packb.objects &&
> + list_packed_objects packa-$packasha1.idx >packa.objects &&
> + list_packed_objects packb-$packbsha1.idx >packb.objects &&
>   test_cmp packa.objects packb.objects
>  '

Thanks for catching and correcting this.

A minor comment from outside observer: running tests under something
like

-e and -o pipefail

would automatically catch the mistake in the first place. Maybe `-o
pipefail` is bashism (I had not checked), but `git grep " | " t/` shows
there are a lot of pipelines being used, and thus similar errors might be
silently resting there. Something like -o pipefail would catch all such
problems automatically.

Kirill

Re: [PATCH] t5500: prettify non-commit tag tests

[Kirill Smelkov]

On Tue, Jul 03, 2018 at 12:55:19PM -0400, Jeff King wrote:
> I had prepared this as a squash-in for what became c12c9df527, but since
> that's now in master, it can go on top (or get dropped, but I think it
> is worth it as a style fixup).

I'm ok with the patch. I thought it was already squashed in into my
version, but maybe it got lost.

Thanks for this prettification.

Kirill

Re: [PATCH] fetch-pack: test explicitly that --all can fetch tag references pointing to non-commits

[Kirill Smelkov]

On Thu, Jun 14, 2018 at 09:07:26AM -0700, Junio C Hamano wrote:
> Kirill Smelkov  writes:
> 
> > Jeff, thanks for corrections. I originally tried to look into invoking
> > "git tag" two times, but since git tag always creates a reference it
> > would not be semantically the same as having one referenced tag object
> > pointing to another tag object which does not have anything in refs/
> > pointing to it directly.
> 
> Well, then you could remove it after you are done, no?  I.e.
> 
>   git tag -a -m "tag to commit" tag-to-commit HEAD
>   git tag -a -m "tag to commit (1)" temp-tag HEAD~1
>   git tag -a -m "tag to tag" tag-to-tag temp-tag
>   git tag -d temp-tag
> 
> would make the temp-tag object reachable only via refs/tags/tag-to-tag
> I think.

Yes, I could remove, and I though about it originally, but to me it is
less clean compared to just creating new tag object without any
reference to it in the first place.

Kirill

Re: [PATCH v2] fetch-pack: don't try to fetch peel values with --all

[Kirill Smelkov]

On Wed, Jun 13, 2018 at 05:13:02PM -0400, Jeff King wrote:
> On Tue, Jun 12, 2018 at 06:54:17PM +0000, Kirill Smelkov wrote:
> 
> > > If an extra connection isn't a problem, you might be better off with
> > > "git ls-remote", and then picking through the results for refs of
> > > interest, and then "git fetch-pack" to actually get the pack. That's how
> > > git-fetch worked when it was a shell script (e.g., see c3a200120d, the
> > > last shell version).
> > 
> > Yes, this is what I ended up doing:
> > 
> > https://lab.nexedi.com/kirr/git-backup/commit/899103bf
> > 
> > but for another reason - to avoid repeating for every fetched repository
> > slow (in case of my "big" destination backup repository) quickfetch()
> > checking in every spawned `git fetch`: git-backup can build index of
> > objects we already have ourselves only once at startup, and then in
> > fetch, after checking lsremote output, consult that index, and if we see
> > we already have everything for an advertised reference - just avoid
> > giving it to fetch-pack to process. It turns out for many pulled
> > repositories there is usually no references changed at all and this way
> > fetch-pack can be skipped completely:
> > 
> > https://lab.nexedi.com/kirr/git-backup/commit/3efed898
> 
> Thanks for sharing that, it's an interesting case. I'd hope that
> git-fetch is smart enough not to even bother with quickfetch() if there
> are no refs to update. But if we have even one change to fetch, then
> yeah, in the general case it makes sense to me that you could do better
> by amortizing the scan of local objects across many operations.

Thanks for feedback. For the reference in case of git-backup `git fetch`
or `git fetch-pack` would have to always do quickfetch scan or equivalent,
because in case of backup repo there is only one reference in it - its
master - and references of backed up repositories do not have anything
representing them in backup.git/refs/ .

Kirill

Re: [PATCH] fetch-pack: test explicitly that --all can fetch tag references pointing to non-commits

[Kirill Smelkov]

On Wed, Jun 13, 2018 at 07:11:47PM -0400, Jeff King wrote:
> On Wed, Jun 13, 2018 at 05:05:09PM -0400, Jeff King wrote:
> 
> > > In order to be sure fetching funky tags will never break, let's
> > > explicitly test all relevant cases with 4 tag objects pointing to 1) a
> > > blob, 2) a tree, 3) a commit, and 4) another tag objects. The referenced
> > > tag objects themselves are referenced from under regular refs/tags/*
> > > namespace. Before e9502c0a7f `fetch-pack --all` was failing e.g. this way:
> > > 
> > > .../git/t/trash directory.t5500-fetch-pack/fetchall$ git 
> > > ls-remote ..
> > > 44085874...HEAD
> > > ...
> > > bc4e9e1f...refs/tags/tag-to-blob
> > > 038f48ad...refs/tags/tag-to-blob^{}   # peeled
> > > 520db1f5...refs/tags/tag-to-tree
> > > 7395c100...refs/tags/tag-to-tree^{}   # peeled
> > > 
> > > .../git/t/trash directory.t5500-fetch-pack/fetchall$ git 
> > > fetch-pack --all ..
> > > fatal: A git upload-pack: not our ref 038f48ad...
> > > fatal: The remote end hung up unexpectedly
> > 
> > TBH, I do not find this snippet all that compelling. We know that
> > e9502c0a7f already fixed the bug, and that it had nothing to do with
> > non-commits at all.
> > 
> > The primary reason to add these tests is that in general we do not cover
> > fetch-pack over tags to non-commits. And I think the reason to use
> > otherwise unreferenced objects is that it they are more likely to have
> > detectable symptoms if they tickle a bug.
> > 
> > So why don't we say that, instead of re-hashing output from the earlier
> > fix?
> 
> Hmm, it looks like this already hit 'next', so it is too late to change
> the commit message (although 'next' will get rewound after the release,
> so we _could_ do it then).
> 
> I also was going to suggest these style fixes, which could be applied on
> top (or squashed if we end up going that route). I actually wonder if
> the final tag one could just use two invocations of "git tag -m", but
> it's probably not worth spending too much time on polishing.

Jeff, thanks for corrections. I originally tried to look into invoking
"git tag" two times, but since git tag always creates a reference it
would not be semantically the same as having one referenced tag object
pointing to another tag object which does not have anything in refs/
pointing to it directly.

Maybe the commit text is not good but I tried to explain the reason you
are talking about with "In order to be sure fetching funky tags will
never break ..."

Kirill

> -- >8 --
> Subject: [PATCH] t5500: prettify non-commit tag tests
> 
> We don't need to use backslash continuation, as the "&&"
> already provides continuation (and happily soaks up empty
> lines between commands).
> 
> We can also expand the multi-line printf into a
> here-document, which lets us use line breaks more naturally
> (and avoids another continuation that required us to break
> the natural indentation).
> 
> Signed-off-by: Jeff King 
> ---
>  t/t5500-fetch-pack.sh | 19 +--
>  1 file changed, 13 insertions(+), 6 deletions(-)
> 
> diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
> index ea6570e819..3d33ab3875 100755
> --- a/t/t5500-fetch-pack.sh
> +++ b/t/t5500-fetch-pack.sh
> @@ -533,19 +533,26 @@ test_expect_success 'test --all wrt tag to non-commits' 
> '
>   # are reachable only via created tag references.
>   blob=$(echo "hello blob" | git hash-object -t blob -w --stdin) &&
>   git tag -a -m "tag -> blob" tag-to-blob $blob &&
> - \
> +
>   tree=$(printf "100644 blob $blob\tfile" | git mktree) &&
>   git tag -a -m "tag -> tree" tag-to-tree $tree &&
> - \
> +
>   tree2=$(printf "100644 blob $blob\tfile2" | git mktree) &&
>   commit=$(git commit-tree -m "hello commit" $tree) &&
>   git tag -a -m "tag -> commit" tag-to-commit $commit &&
> - \
> +
>   blob2=$(echo "hello blob2" | git hash-object -t blob -w --stdin) &&
> - tag=$(printf "object $blob2\ntype blob\ntag tag-to-blob2\n\
> -tagger author A U Thor  0 +\n\nhello tag" | git 
> mktag) &&
> + tag=$(git mktag <<-EOF
> + object $blob2
> + type blob
> + tag tag-to-blob2
> + tagger author A U Thor  0 +
> +
> + hello tag
> + EOF
> + ) &&
>   git tag -a -m "tag -> tag" tag-to-tag $tag &&
> - \
> +
>   # `fetch-pack --all` should succeed fetching all those objects.
>   mkdir fetchall &&
>   (
> -- 
> 2.18.0.rc2.519.gb87ed92113

Re: [PATCH] fetch-pack: test explicitly that --all can fetch tag references pointing to non-commits

[Kirill Smelkov]

On Wed, Jun 13, 2018 at 10:42:33AM -0700, Junio C Hamano wrote:
> Kirill Smelkov  writes:
> 
> > Fetch-pack --all became broken with respect to unusual tags in
> > 5f0fc64513 (fetch-pack: eliminate spurious error messages, 2012-09-09),
> > and was fixed only recently in e9502c0a7f (fetch-pack: don't try to fetch
> > peel values with --all, 2018-06-11). However the test added in
> > e9502c0a7f does not explicitly cover all funky cases.
> 
> Thanks.  Very much appreciated

Thanks.


> > In order to be sure fetching funky tags will never break, let's
> > explicitly test all relevant cases with 4 tag objects pointing to 1) a
> > blob, 2) a tree, 3) a commit, and 4) another tag objects. The referenced
> > tag objects themselves are referenced from under regular refs/tags/*
> > namespace. Before e9502c0a7f `fetch-pack --all` was failing e.g. this way:
> >
> > .../git/t/trash directory.t5500-fetch-pack/fetchall$ git ls-remote 
> > ..
> > 440858748ae905d48259d4fb67a12a7aa1520cf7HEAD
> > ...
> > bc4e9e1fa80662b449805b1ac29fc9b1e4c49187
> > refs/tags/tag-to-blob   # <-- NOTE
> > 038f48ad0beaffbea71d186a05084b79e3870cbf
> > refs/tags/tag-to-blob^{}
> > 520db1f5e1afeaa12b1a8d73ce82db72ca036ee1
> > refs/tags/tag-to-tree   # <-- NOTE
> > 7395c100223b7cd760f58ccfa0d3f3d2dd539bb6
> > refs/tags/tag-to-tree^{}
> > .../git/t/trash directory.t5500-fetch-pack/fetchall$ git fetch-pack 
> > --all ..
> > fatal: A git upload-pack: not our ref 
> > 038f48ad0beaffbea71d186a05084b79e3870cbf
> > fatal: The remote end hung up unexpectedly
> 
> ... except for this bit.  I am not sure readers understand what
> these two overlong lines want to say.  Would it be easier to
> understand if you wrote the above like this?
> 
>  .../git/t/trash directory.t5500-fetch-pack/fetchall$ git ls-remote ..
>  44085874...HEAD
>  ...
>  bc4e9e1f...refs/tags/tag-to-blob
>  038f48ad...refs/tags/tag-to-blob^{}  # peeled
>  520db1f5...refs/tags/tag-to-tree
>  7395c100...refs/tags/tag-to-tree^{}  # peeled
>  .../git/t/trash directory.t5500-fetch-pack/fetchall$ git fetch-pack 
> --all ..
>  fatal: A git upload-pack: not our ref 038f48ad...
>  fatal: The remote end hung up unexpectedly
> 
> Instead of marking the tag-to-blob and tag-to-tree entries (which
> are not where the 'fatal' breakage comes from), I think it makes
> more sense to mark the entries that show peeled version (which also
> matches the object name in the error message), and by shortening the
> line, readers can see more easily which lines are highlighted.

Makes sense. I've amended the patch description with your suggestion.

> > +test_expect_success 'test --all wrt tag to non-commits' '
> > +   blob=$(echo "hello blob" | git hash-object -t blob -w --stdin) &&
> > +   git tag -a -m "tag -> blob" tag-to-blob $blob &&
> > + \
> > +   tree=$(printf "100644 blob $blob\tfile" | git mktree) &&
> > +   git tag -a -m "tag -> tree" tag-to-tree $tree &&
> > + \
> > +   tree2=$(printf "100644 blob $blob\tfile2" | git mktree) &&
> > +   commit=$(git commit-tree -m "hello commit" $tree) &&
> > +   git tag -a -m "tag -> commit" tag-to-commit $commit &&
> > + \
> > +   blob2=$(echo "hello blob2" | git hash-object -t blob -w --stdin) &&
> > +   tag=$(printf "object $blob2\ntype blob\ntag tag-to-blob2\n\
> > +tagger author A U Thor  0 +\n\nhello tag" | git 
> > mktag) &&
> > +   git tag -a -m "tag -> tag" tag-to-tag $tag &&
> 
> All of the above, while may not be techincallly wrong per-se, look
> unnecessarily complex.
> 
> I guess the reason why you do the above is because you do not want
> to use any object that is reachable via other existing refs and
> instead want to ensure these objects are reachable only via the tags
> you are creating in this test.  Otherwise using HEAD~4:test.txt and
> HEAD^{tree} instead of $blob and $tree constructed via hash-object
> and mktree would be sufficient and more readable.  Oh well.

Yes, it is exactly the reason here. I've added corresponding comment
explaining this to the test case.

Kirill

 8< 
From: Kirill Smelkov 
Date: Wed, 13 Jun 2018 12:28:21 +0300
Subject: [PATCH v2] fetch-pack: test explicitly that --all can fetch tag
 r

Re: [PATCH] fetch-pack: demonstrate --all failure when remote is empty

[Kirill Smelkov]

On Wed, Jun 13, 2018 at 10:13:07AM -0700, Junio C Hamano wrote:
> Kirill Smelkov  writes:
> 
> > ( Junio, please pick up the patch provided in the end )
> >
> > On Tue, Jun 12, 2018 at 06:54:17PM +, Kirill Smelkov wrote:
> >> On Tue, Jun 12, 2018 at 05:48:49AM -0400, Jeff King wrote:
> >> > On Mon, Jun 11, 2018 at 09:43:02AM +, Kirill Smelkov wrote:
> > [...]
> >
> >> > > I'm not sure, but I would say that `fetch-pack --all` from an empty
> >> > > repository should not fail and should just give empty output as fetch
> >> > > does.
> >> > 
> >> > Yeah, that seems reasonable to me. The die() that catches this dates
> >> > back to 2005-era, and we later taught the "fetch" porcelain to handle
> >> > this. I don't _think_ anybody would be upset that the plumbing learned
> >> > to treat this as a noop. It's probably a one-liner change in
> >> > fetch_pack() to return early instead of dying.
> 
> I actually have a slight preference to the current "attempting to
> fetch from a total emptiness is so rare that it is worth grabbing
> attention of whoever does so" behaviour, to be honest.

I see.

> Oh, wait, is this specific to "fetch-pack" and the behaviour of
> end-user-facing "git fetch" is kept same as before?  If then, I'd be
> somewhat sympathetic to the cause---it would be more convenient for
> the calling Porcelain script if this turned into a silent noop (even
> though it would probably make it harder to diagnose when such a
> Porcelain is set up incorrectly e.g. pointing at an empty repository
> that is not the one the Porcelain writer intended to fetch from).

Yes, it is only for fetch-pack, and behaviour of porcelain fetch is kept
as it was before.

> > However with transport.c being there too, since I'm no longer using
> > `fetch-pack --all`, now it is best for me to not delve into this story
> > and just stop with attached patch.
> 
> If we do not plan to change the behaviour later ourselves, I do not
> think it makes sense, nor it is fair to those future developers who
> inherit this project, to declare that the established behaviour is
> wrong with an 'expect-failure' test like this, to be honest.

I see. Let's please cancel this patch then.


> > +test_expect_failure 'test --all wrt empty.git' '
> > +   git init --bare empty.git &&
> > +   (
> > +   cd client &&
> > +   git fetch-pack --all ../empty.git
> > +   )
> > +'

[PATCH] fetch-pack: demonstrate --all failure when remote is empty

[Kirill Smelkov]

( Junio, please pick up the patch provided in the end )

On Tue, Jun 12, 2018 at 06:54:17PM +, Kirill Smelkov wrote:
> On Tue, Jun 12, 2018 at 05:48:49AM -0400, Jeff King wrote:
> > On Mon, Jun 11, 2018 at 09:43:02AM +0000, Kirill Smelkov wrote:
[...]

> > > I'm not sure, but I would say that `fetch-pack --all` from an empty
> > > repository should not fail and should just give empty output as fetch
> > > does.
> > 
> > Yeah, that seems reasonable to me. The die() that catches this dates
> > back to 2005-era, and we later taught the "fetch" porcelain to handle
> > this. I don't _think_ anybody would be upset that the plumbing learned
> > to treat this as a noop. It's probably a one-liner change in
> > fetch_pack() to return early instead of dying.
> 
> Ok, I will try to send related testcase, and it is indeed easy to find
> - the fix itself.

I started doing it in full with the following

--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -1581,6 +1581,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,

if (!ref) {
packet_flush(fd[1]);
+   if (nr_sought == 0) // XXX or better args->fetch_all
+   return NULL; /* nothing to fetch */
die(_("no matching remote head"));
}
prepare_shallow_info(, shallow);


but then came to the fact that !ref fetch_pack() return is analyzed in 2
places:

- in builtin/fetch-pack.c itself:

ref = fetch_pack(, fd, conn, ref, dest, sought, nr_sought,
 , pack_lockfile_ptr, protocol_v0);

...

ret = !ref;

- and in transport.c in fetch_refs_via_pack():

case protocol_v1:
case protocol_v0:
refs = fetch_pack(, data->fd, data->conn,
  refs_tmp ? refs_tmp : transport->remote_refs,
  dest, to_fetch, nr_heads, >shallow,
  >pack_lockfile, data->version);
break;

...

if (refs == NULL)
ret = -1;


As I don't know git codebase well-enough I don't see offhand how to
distinguish empty result from a real error when something was requested
and not fetched. If it would be only builtin/fetch-pack I could start to
play ugly games with analyzing too in the calling site args.fetch_all
and nr_though and if at that level we also know we requested nothing,
don't treat !refs as an error.

However with transport.c being there too, since I'm no longer using
`fetch-pack --all`, now it is best for me to not delve into this story
and just stop with attached patch.

Thanks,
Kirill

 8< 
>From 76d80ffcfd4574715545c62413d64d40af063d09 Mon Sep 17 00:00:00 2001
From: Kirill Smelkov 
Date: Wed, 13 Jun 2018 15:46:00 +0300
Subject: [PATCH] fetch-pack: demonstrate --all failure when remote is empty

Currently `fetch-pack --all` from an empty repository gives:

fatal: no matching remote head

However it would be logical for this fetch operation to succeed with
empty result. Add test showing the failure.

Signed-off-by: Kirill Smelkov 

---
 t/t5500-fetch-pack.sh | 8 
 1 file changed, 8 insertions(+)

diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
index 82aee1c2d8..2234bad411 100755
--- a/t/t5500-fetch-pack.sh
+++ b/t/t5500-fetch-pack.sh
@@ -528,6 +528,14 @@ test_expect_success 'test --all with tag to non-tip' '
)
 '
 
+test_expect_failure 'test --all wrt empty.git' '
+   git init --bare empty.git &&
+   (
+   cd client &&
+   git fetch-pack --all ../empty.git
+   )
+'
+
 test_expect_success 'shallow fetch with tags does not break the repository' '
mkdir repo1 &&
(
-- 
2.18.0.rc1.253.gf85a566b11.dirty

[PATCH] fetch-pack: test explicitly that --all can fetch tag references pointing to non-commits

[Kirill Smelkov]

Fetch-pack --all became broken with respect to unusual tags in
5f0fc64513 (fetch-pack: eliminate spurious error messages, 2012-09-09),
and was fixed only recently in e9502c0a7f (fetch-pack: don't try to fetch
peel values with --all, 2018-06-11). However the test added in
e9502c0a7f does not explicitly cover all funky cases.

In order to be sure fetching funky tags will never break, let's
explicitly test all relevant cases with 4 tag objects pointing to 1) a
blob, 2) a tree, 3) a commit, and 4) another tag objects. The referenced
tag objects themselves are referenced from under regular refs/tags/*
namespace. Before e9502c0a7f `fetch-pack --all` was failing e.g. this way:

.../git/t/trash directory.t5500-fetch-pack/fetchall$ git ls-remote ..
440858748ae905d48259d4fb67a12a7aa1520cf7HEAD
...
bc4e9e1fa80662b449805b1ac29fc9b1e4c49187refs/tags/tag-to-blob   
# <-- NOTE
038f48ad0beaffbea71d186a05084b79e3870cbfrefs/tags/tag-to-blob^{}
520db1f5e1afeaa12b1a8d73ce82db72ca036ee1refs/tags/tag-to-tree   
# <-- NOTE
7395c100223b7cd760f58ccfa0d3f3d2dd539bb6refs/tags/tag-to-tree^{}

.../git/t/trash directory.t5500-fetch-pack/fetchall$ git fetch-pack 
--all ..
fatal: A git upload-pack: not our ref 
038f48ad0beaffbea71d186a05084b79e3870cbf
fatal: The remote end hung up unexpectedly

Signed-off-by: Kirill Smelkov 
---
 t/t5500-fetch-pack.sh | 28 
 1 file changed, 28 insertions(+)

diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
index f20bb59d22..b560d90c7b 100755
--- a/t/t5500-fetch-pack.sh
+++ b/t/t5500-fetch-pack.sh
@@ -528,6 +528,34 @@ test_expect_success 'test --all with tag to non-tip' '
)
 '
 
+test_expect_success 'test --all wrt tag to non-commits' '
+   blob=$(echo "hello blob" | git hash-object -t blob -w --stdin) &&
+   git tag -a -m "tag -> blob" tag-to-blob $blob &&
+ \
+   tree=$(printf "100644 blob $blob\tfile" | git mktree) &&
+   git tag -a -m "tag -> tree" tag-to-tree $tree &&
+ \
+   tree2=$(printf "100644 blob $blob\tfile2" | git mktree) &&
+   commit=$(git commit-tree -m "hello commit" $tree) &&
+   git tag -a -m "tag -> commit" tag-to-commit $commit &&
+ \
+   blob2=$(echo "hello blob2" | git hash-object -t blob -w --stdin) &&
+   tag=$(printf "object $blob2\ntype blob\ntag tag-to-blob2\n\
+tagger author A U Thor  0 +\n\nhello tag" | git mktag) 
&&
+   git tag -a -m "tag -> tag" tag-to-tag $tag &&
+ \
+   mkdir fetchall &&
+   (
+   cd fetchall &&
+   git init &&
+   git fetch-pack --all .. &&
+   git cat-file blob $blob >/dev/null &&
+   git cat-file tree $tree >/dev/null &&
+   git cat-file commit $commit >/dev/null &&
+   git cat-file tag $tag >/dev/null
+   )
+'
+
 test_expect_success 'shallow fetch with tags does not break the repository' '
mkdir repo1 &&
(
-- 
2.18.0.rc1.253.gf85a566b11.dirty

Re: [PATCH v2] fetch-pack: don't try to fetch peel values with --all

[Kirill Smelkov]

On Tue, Jun 12, 2018 at 05:48:49AM -0400, Jeff King wrote:
> On Mon, Jun 11, 2018 at 09:43:02AM +0000, Kirill Smelkov wrote:
> 
> > > Looking deeper, we do not need these trees and blobs at all. The problem
> > > is really just a tag that peels to an object that is not otherwise a ref
> > > tip, regardless of its type.
> > 
> > Thanks for feedback and for coming up with the fix. Sure, I'm ok with
> > moving the test into your patch. However, even if a test becomes
> > different - narrowing down root of _current_ problem, I suggest to also
> > keep explicitly testing tag-to-blob and tag-to-tree (and if we really
> > also want tag-to-commit and tag-to-tag) behaviour. Reason is: if we skip
> > those now, they can potentially break in the future.
> 
> Yeah, I have no problem testing these cases separately. There's no bug
> with them now, but it is a slightly uncommon case. My suggestion would
> be to submit a patch that goes on top of mine that covers these cases.

Ok, I will try to do it.


> > I would also suggest to fix upload-pack, as it is just not consistent to
> > reject sending objects that were advertised, and so can strike again
> > some way in the future. After all git.git's fetch-pack is not the only
> > git client that should be possible to interact with git.git's
> > upload-pack on remote side, right?
> 
> No, it's not the only client. At the same time, I am on the fence over
> whether upload-pack's behavior is wrong or not. It depends what you take
> a peeled advertisement line to mean. Does it mean: this object has been
> advertised and clients should be able to fetch it? Or does it mean: by
> the way, you may be interested to know the peeled value of this tag in
> case you want to do tag-following?
> 
> So far I think it has only meant the latter. I could see an argument for
> the former, but any client depending on that would never have worked,
> AFAICT. We could _make_ it work, but how would a client know which
> server version it's talking to (and therefore whether it is safe to make
> the request?). I think you'd have to add a capability to negotiate.

I see. I don't know the details of the exchange, just it was surprising
for outside observer that fetching what was advertised is rejected. For
the reference there is no strong need for me for this to work anymore
(please see below).


> > I'm not sure, but I would say that `fetch-pack --all` from an empty
> > repository should not fail and should just give empty output as fetch
> > does.
> 
> Yeah, that seems reasonable to me. The die() that catches this dates
> back to 2005-era, and we later taught the "fetch" porcelain to handle
> this. I don't _think_ anybody would be upset that the plumbing learned
> to treat this as a noop. It's probably a one-liner change in
> fetch_pack() to return early instead of dying.

Ok, I will try to send related testcase, and it is indeed easy to find
- the fix itself.


> > For the reference all the cases presented here are real - they appear in
> > our repositories on lab.nexedi.com for which I maintain the backup, and
> > I've noticed them in the process of switching git-backup from using
> > fetch to fetch-pack here:
> > 
> > https://lab.nexedi.com/kirr/git-backup/blob/0ab7bbb6/git-backup.go#L436
> 
> I applaud you using the porcelain for your scripts, but I suspect that
> fetch-pack by itself is not at all well-used or well-tested these days
> (certainly this --all bug has been around for almost 6 years and is not
> very hard to trigger in practice).

I see; thanks for the warning.


> If an extra connection isn't a problem, you might be better off with
> "git ls-remote", and then picking through the results for refs of
> interest, and then "git fetch-pack" to actually get the pack. That's how
> git-fetch worked when it was a shell script (e.g., see c3a200120d, the
> last shell version).

Yes, this is what I ended up doing:

https://lab.nexedi.com/kirr/git-backup/commit/899103bf

but for another reason - to avoid repeating for every fetched repository
slow (in case of my "big" destination backup repository) quickfetch()
checking in every spawned `git fetch`: git-backup can build index of
objects we already have ourselves only once at startup, and then in
fetch, after checking lsremote output, consult that index, and if we see
we already have everything for an advertised reference - just avoid
giving it to fetch-pack to process. It turns out for many pulled
repositories there is usually no references changed at all and this way
fetch-pack can be skipped completely:

https://lab.nexedi.com/kirr/git-backup/commit/3efed898

> It may also be sane to just use "git fetch", which I'd say is _

Re: [PATCH v2] fetch-pack: don't try to fetch peel values with --all

[Kirill Smelkov]

Jeff,

On Mon, Jun 11, 2018 at 01:53:57AM -0400, Jeff King wrote:
> On Mon, Jun 11, 2018 at 01:28:23AM -0400, Eric Sunshine wrote:
> 
> > On Mon, Jun 11, 2018 at 12:47 AM, Jeff King  wrote:
> > > Subject: fetch-pack: don't try to fetch peeled values with --all
> > > [...]
> > > Original report and test from Kirill Smelkov.
> > >
> > > Signed-off-by: Kirill Smelkov 
> > > Signed-off-by: Jeff King 
> > > ---
> > > diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
> > > @@ -506,30 +506,45 @@ test_expect_success 'test missing ref before 
> > > existing' '
> > > +test_expect_success 'test --all wrt tag to non-commits' '
> > > +   blob_sha1=$(echo "hello blob" | git hash-object -t blob -w 
> > > --stdin) &&
> > > +   git tag -a -m "tag -> blob" tag-to-blob $blob_sha1 &&
> > > +   tree_sha1=$(printf "100644 blob $blob_sha1\tfile\n" | git mktree) 
> > > &&
> > 
> > Perhaps modernize these names to 'blob_oid' and 'tree_oid', or even
> > simpler, just 'blob' and 'tree'.
> 
> Looking deeper, we do not need these trees and blobs at all. The problem
> is really just a tag that peels to an object that is not otherwise a ref
> tip, regardless of its type.

Thanks for feedback and for coming up with the fix. Sure, I'm ok with
moving the test into your patch. However, even if a test becomes
different - narrowing down root of _current_ problem, I suggest to also
keep explicitly testing tag-to-blob and tag-to-tree (and if we really
also want tag-to-commit and tag-to-tag) behaviour. Reason is: if we skip
those now, they can potentially break in the future.

I would also suggest to fix upload-pack, as it is just not consistent to
reject sending objects that were advertised, and so can strike again
some way in the future. After all git.git's fetch-pack is not the only
git client that should be possible to interact with git.git's
upload-pack on remote side, right?

By the way, another problem I noticed with fetch-pack is that fetching
with --all from completely empty repository also fails:

.../r$ git init --bare repo.git
Initialized empty Git repository in /home/kirr/tmp/trashme/r/repo.git/
.../r$ mkdir clone
.../r$ cd clone/
.../r/clone$ git init
Initialized empty Git repository in /home/kirr/tmp/trashme/r/clone/.git/
.../r/clone$ git fetch-pack --all ../repo.git/
fatal: no matching remote head
.../r/clone$ echo $?
128
.../r/clone$ git ls-remote ../repo.git/
.../r/clone$ echo $?
0
.../r/clone$ git fetch ../repo.git/ 'refs/*:refs/repo/*'
.../r/clone$ echo $?
0

I'm not sure, but I would say that `fetch-pack --all` from an empty
repository should not fail and should just give empty output as fetch
does.

For the reference all the cases presented here are real - they appear in
our repositories on lab.nexedi.com for which I maintain the backup, and
I've noticed them in the process of switching git-backup from using
fetch to fetch-pack here:

https://lab.nexedi.com/kirr/git-backup/blob/0ab7bbb6/git-backup.go#L436

Kirill



> So below is a patch that simplifies the test even further (the actual
> code change is the same).
> 
> > > +   git tag -a -m "tag -> tree" tag-to-tree $tree_sha1 &&
> > > +   mkdir fetchall &&
> > > +   (
> > > +   cd fetchall &&
> > > +   git init &&
> > > +   git fetch-pack --all .. &&
> > 
> > Simpler:
> > 
> > git init fetchall &&
> > (
> > cd fetchall &&
> > git fetch-pack --all .. &&
> > 
> > Although, I see that this script already has a mix of the two styles
> > (simpler and not-so-simple), so...
> 
> The nearby tests actually reuse the "client" directory. We can do that,
> too, if we simply create new objects for our test, to make sure they
> still need fetching. See below (we could also use "git -C" there, but
> the subshell matches the other tests).
> 
> -- >8 --
> Subject: fetch-pack: don't try to fetch peel values with --all
> 
> When "fetch-pack --all" sees a tag-to-blob on the remote, it
> tries to fetch both the tag itself ("refs/tags/foo") and the
> peeled value that the remote advertises ("refs/tags/foo^{}").
> Asking for the object pointed to by the latter can cause
> upload-pack to complain with "not our ref", since it does
> not mark the peeled objects with the OUR_REF (unless they
> were at the tip of some other ref).

[PATCH] fetch-pack: demonstrate --all breakage when remote have tags to non-commit objects

[Kirill Smelkov]

  -> refs/origin/heads/C
 * [new branch]  D   -> refs/origin/heads/D
 * [new branch]  E   -> refs/origin/heads/E
 * [new branch]  F   -> refs/origin/heads/F
 * [new tag] A   -> refs/origin/tags/A
 * [new tag] B   -> refs/origin/tags/B
 * [new tag] C   -> refs/origin/tags/C
 * [new tag] D   -> refs/origin/tags/D
 * [new tag] E   -> refs/origin/tags/E
 * [new tag] F   -> refs/origin/tags/F
 * [new tag] OLDTAG  -> refs/origin/tags/OLDTAG
 * [new tag] TAGA1   -> refs/origin/tags/TAGA1
 * [new tag] TAGA2   -> refs/origin/tags/TAGA2
 * [new tag] TAGB1   -> refs/origin/tags/TAGB1
 * [new tag] TAGB2   -> refs/origin/tags/TAGB2
 * [new tag] tag-to-blob -> refs/origin/tags/tag-to-blob
 * [new tag] tag-to-tree -> refs/origin/tags/tag-to-tree
 * [new tag] A   -> A
 * [new tag] B   -> B
 * [new tag] C   -> C
 * [new tag] D   -> D
 * [new tag] E   -> E
 * [new tag] F   -> F
 * [new tag] OLDTAG  -> OLDTAG
 * [new tag] TAGA1   -> TAGA1
 * [new tag] TAGA2   -> TAGA2
 * [new tag] TAGB1   -> TAGB1
 * [new tag]     TAGB2   -> TAGB2
 * [new tag] tag-to-blob -> tag-to-blob 
# <-- NOTE
 * [new tag] tag-to-tree -> tag-to-tree 
# <-- NOTE

Signed-off-by: Kirill Smelkov 
---
 t/t5500-fetch-pack.sh | 15 +++
 1 file changed, 15 insertions(+)

diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
index d4f435155f..000f338172 100755
--- a/t/t5500-fetch-pack.sh
+++ b/t/t5500-fetch-pack.sh
@@ -518,6 +518,21 @@ test_expect_success 'test --all, --depth, and explicit 
tag' '
) >out-adt 2>error-adt
 '
 
+test_expect_failure 'test --all wrt tag to non-commits' '
+   blob_sha1=$(echo "hello blob" | git hash-object -t blob -w --stdin) &&
+   git tag -a -m "tag -> blob" tag-to-blob $blob_sha1 &&
+   tree_sha1=$(echo -e "100644 blob $blob_sha1\tfile" | git mktree) &&
+   git tag -a -m "tag -> tree" tag-to-tree $tree_sha1 &&
+   mkdir fetchall &&
+   (
+   cd fetchall &&
+   git init &&
+   git fetch-pack --all .. &&
+   git cat-file blob $blob_sha1 >/dev/null &&
+   git cat-file tree $tree_sha1 >/dev/null
+   )
+'
+
 test_expect_success 'shallow fetch with tags does not break the repository' '
mkdir repo1 &&
(
-- 
2.18.0.rc1.242.g61856ae69a.dirty

Re: [PATCH 1/2 v8] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use

[Kirill Smelkov]

On Mon, Sep 12, 2016 at 11:23:18PM -0700, Junio C Hamano wrote:
> Kirill Smelkov <k...@nexedi.com> writes:
> 
> > +static int want_found_object(int exclude, struct packed_git *p)
> > +{
> > +   if (exclude)
> > +   return 1;
> > +   if (incremental)
> > +   return 0;
> > +
> > +   /*
> > +* When asked to do --local (do not include an object that appears in a
> > +* pack we borrow from elsewhere) or --honor-pack-keep (do not include
> > +* an object that appears in a pack marked with .keep), finding a pack
> > +* that matches the criteria is sufficient for us to decide to omit it.
> > +* However, even if this pack does not satisfy the criteria, we need to
> > +* make sure no copy of this object appears in _any_ pack that makes us
> > +* to omit the object, so we need to check all the packs.
> > +*
> > +* We can however first check whether these options can possible matter;
> > +* if they do not matter we know we want the object in generated pack.
> > +* Otherwise, we signal "-1" at the end to tell the caller that we do
> > +* not know either way, and it needs to check more packs.
> > +*/
> > +   if (!ignore_packed_keep &&
> > +   (!local || !have_non_local_packs))
> > +   return 1;
> > +
> > +   if (local && !p->pack_local)
> > +   return 0;
> > +   if (ignore_packed_keep && p->pack_local && p->pack_keep)
> > +   return 0;
> > +
> > +   /* we don't know yet; keep looking for more packs */
> > +   return -1;
> > +}
> 
> Moving this logic out to this helper made the main logic in the
> caller easier to grasp.
> 
> > @@ -958,15 +993,30 @@ static int want_object_in_pack(const unsigned char 
> > *sha1,
> >off_t *found_offset)
> >  {
> > struct packed_git *p;
> > +   int want;
> >  
> > if (!exclude && local && has_loose_object_nonlocal(sha1))
> > return 0;
> >  
> > +   /*
> > +* If we already know the pack object lives in, start checks from that
> > +* pack - in the usual case when neither --local was given nor .keep 
> > files
> > +* are present we will determine the answer right now.
> > +*/
> > +   if (*found_pack) {
> > +   want = want_found_object(exclude, *found_pack);
> > +   if (want != -1)
> > +   return want;
> > +   }
> >  
> > for (p = packed_git; p; p = p->next) {
> > +   off_t offset;
> > +
> > +   if (p == *found_pack)
> > +   offset = *found_offset;
> > +   else
> > +   offset = find_pack_entry_one(sha1, p);
> > +
> > if (offset) {
> > if (!*found_pack) {
> > if (!is_pack_valid(p))
> > @@ -974,31 +1024,9 @@ static int want_object_in_pack(const unsigned char 
> > *sha1,
> > *found_offset = offset;
> > *found_pack = p;
> > }
> > +   want = want_found_object(exclude, p);
> > +   if (want != -1)
> > +   return want;
> > }
> > }
> 
> As Peff noted in his earlier review, however, MRU code needed to be
> grafted in to the caller (an update to the MRU list was done in the
> code that was moved to the want_found_object() helper).  I think I
> did it correctly, which ended up looking like this:
> 
> want = want_found_object(exclude, p);
> if (!exclude && want > 0)
> mru_mark(packed_git_mru, entry);
> if (want != -1)
> return want;
> 
> I somewhat feel that it is ugly that the helper knows about exclude
> (i.e. in the original code, we immediately returned 1 without
> futzing with the MRU when we find an entry that is to be excluded,
> which now is done in the helper), and the caller also knows about
> exclude (i.e. the caller knows that the helper may return positive
> in two cases, it knows that MRU marking needs to happen only one of
> the two cases, and it also knows that "exclude" is what
> differentiates between the two cases) at the same time.
> 
> But probably the reason why I feel it ugly is only because I knew
> how the original looked like.  I dunno.

Junio, the code above is correct semantic merge of pack-mru and my
topic, because in pack-mru if found and exclude=1, 1 was returned
without marking found pack.

But I wonder: even if we exclude an object, we were still looking for it
in packs, and when we found it, we found the corresponding pack too. So,
that pack _was_ most-recently-used, and it is correct to mark it as MRU.

We can do the simplification in the follow-up patch after the merge, so
merge does not change semantics and it is all bisectable, etc.

Jeff?

Re: [PATCH] t/perf/run: Don't forget to copy config.mak.autogen & friends to build area

[Kirill Smelkov]

On Mon, Sep 12, 2016 at 04:10:09PM -0700, Junio C Hamano wrote:
> Junio C Hamano <gits...@pobox.com> writes:
> 
> > In other words, something along this line, perhaps.
> > ...
> 
> Not quite.  There is no guanratee that the user is using autoconf at
> all.  It should be more like this, I think.
> 
>  t/perf/run | 8 +++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/t/perf/run b/t/perf/run
> index aa383c2..7ec3734 100755
> --- a/t/perf/run
> +++ b/t/perf/run
> @@ -30,7 +30,13 @@ unpack_git_rev () {
>  }
>  build_git_rev () {
>   rev=$1
> - cp -t build/$rev ../../{config.mak,config.mak.autogen,config.status}
> + for config in config.mak config.mak.autogen config.status
> + do
> + if test -f "../../$config"
> + then
> + cp "../../$config" "build/$rev/"
> + fi
> + done
>   (cd build/$rev && make $GIT_PERF_MAKE_OPTS) ||
>   die "failed to build revision '$mydir'"
>  }

Junio, thanks for encouraging feedback and for catching the *-isms. What
you propose is good (and we also automatically fix error when there was
no config.mak - it was working but cp was giving an error to stderr but
script was continuing normally).

I would amend your squash the following way:

* `test -f` -> `test -e`, because -f tests whether a file exists _and_
  is regular file. Some people might have config.mak as a symlink for
  example. We don't want to miss them too.

Please find updated patch below:

 8< 
From: Kirill Smelkov <k...@nexedi.com>
Subject: [PATCH] t/perf/run: Don't forget to copy config.mak.autogen & friends
 to build area

Otherwise for people who use autotools-based configure in main worktree,
the performance testing results will be inconsistent as work and build
trees could be using e.g. different optimization levels.

See e.g.


http://public-inbox.org/git/20160818175222.bmm3ivjheokf2...@sigill.intra.peff.net/

for example.

NOTE config.status has to be copied because otherwise without it the build
would want to run reconfigure this way loosing just copied config.mak.autogen.

Signed-off-by: Kirill Smelkov <k...@nexedi.com>
---
 t/perf/run | 8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/t/perf/run b/t/perf/run
index cfd7012..e8adeda 100755
--- a/t/perf/run
+++ b/t/perf/run
@@ -30,7 +30,13 @@ unpack_git_rev () {
 }
 build_git_rev () {
rev=$1
-   cp ../../config.mak build/$rev/config.mak
+   for config in config.mak config.mak.autogen config.status
+   do
+   if test -e "../../$config"
+   then
+   cp "../../$config" "build/$rev/"
+   fi
+   done
(cd build/$rev && make $GIT_PERF_MAKE_OPTS) ||
die "failed to build revision '$mydir'"
 }
-- 
2.9.2.701.gf965a18.dirty

[PATCH] t/perf/run: Don't forget to copy config.mak.autogen & friends to build area

[Kirill Smelkov]

Otherwise for people who use autotools-based configure in main worktree,
the performance testing results will be inconsistent as work and build
trees could be using e.g. different optimization levels.

See e.g.


http://public-inbox.org/git/20160818175222.bmm3ivjheokf2...@sigill.intra.peff.net/

for example.

NOTE config.status has to be copied because otherwise without it the build
would want to run reconfigure this way loosing just copied config.mak.autogen.

Signed-off-by: Kirill Smelkov <k...@nexedi.com>
---
 ( Resending as separate patch-mail, just in case )

 t/perf/run | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/perf/run b/t/perf/run
index cfd7012..aa383c2 100755
--- a/t/perf/run
+++ b/t/perf/run
@@ -30,7 +30,7 @@ unpack_git_rev () {
 }
 build_git_rev () {
rev=$1
-   cp ../../config.mak build/$rev/config.mak
+   cp -t build/$rev ../../{config.mak,config.mak.autogen,config.status}
(cd build/$rev && make $GIT_PERF_MAKE_OPTS) ||
die "failed to build revision '$mydir'"
 }
-- 
2.9.2.701.gf965a18.dirty

[PATCH 2/2 v8] pack-objects: use reachability bitmap index when generating non-stdout pack

[Kirill Smelkov]

$ time git index-pack erp5pack-stdout.pack

real0m50.873s   <-- more than 2 times slower than time to generate pack 
itself!
user0m49.300s
sys 0m1.360s

So the time for

`pack-object --stdout >file.pack` + `index-pack file.pack`  is  72s,

while

`pack-objects file.pack` which does both pack and index is  27s.

And even

`pack-objects --no-use-bitmap-index file.pack`  is  37s.

Jeff explains:

The packfile does not carry the sha1 of the objects. A receiving
index-pack has to compute them itself, including inflating and applying
all of the deltas.

that's why for `git-backup restore` we want to teach `git pack-objects
file.pack` to use bitmaps instead of using `git pack-objects --stdout
>file.pack` + `git index-pack file.pack`.

NOTE3

The speedup is now tracked via t/perf/p5310-pack-bitmaps.sh

Test56dfeb62  this tree


5310.2: repack to disk  8.98(8.05+0.29)   9.05(8.08+0.33) 
+0.8%
5310.3: simulated clone 2.02(2.27+0.09)   2.01(2.25+0.08) 
-0.5%
5310.4: simulated fetch 0.81(1.07+0.02)   0.81(1.05+0.04) 
+0.0%
5310.5: pack to file7.58(7.04+0.28)   7.60(7.04+0.30) 
+0.3%
5310.6: pack to file (bitmap)   7.55(7.02+0.28)   3.25(2.82+0.18) 
-57.0%
5310.8: clone (partial bitmap)  1.83(2.26+0.12)   1.82(2.22+0.14) 
-0.5%
5310.9: pack to file (partial bitmap)   6.86(6.58+0.30)   2.87(2.74+0.20) 
-58.2%

More context:

http://marc.info/?t=14679210141=1=2
http://public-inbox.org/git/20160707190917.20011-1-k...@nexedi.com/T/#t

Cc: Vicent Marti <tan...@gmail.com>
Helped-by: Jeff King <p...@peff.net>
Signed-off-by: Kirill Smelkov <k...@nexedi.com>
Signed-off-by: Junio C Hamano <gits...@pobox.com>
---
 builtin/pack-objects.c   | 31 ---
 t/perf/p5310-pack-bitmaps.sh | 14 +-
 t/t5310-pack-bitmaps.sh  | 12 
 3 files changed, 49 insertions(+), 8 deletions(-)

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 19668d3..d48c290 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -67,7 +67,8 @@ static struct packed_git *reuse_packfile;
 static uint32_t reuse_packfile_objects;
 static off_t reuse_packfile_offset;
 
-static int use_bitmap_index = 1;
+static int use_bitmap_index_default = 1;
+static int use_bitmap_index = -1;
 static int write_bitmap_index;
 static uint16_t write_bitmap_options;
 
@@ -2274,7 +2275,7 @@ static int git_pack_config(const char *k, const char *v, 
void *cb)
write_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;
}
if (!strcmp(k, "pack.usebitmaps")) {
-   use_bitmap_index = git_config_bool(k, v);
+   use_bitmap_index_default = git_config_bool(k, v);
return 0;
}
if (!strcmp(k, "pack.threads")) {
@@ -2523,13 +2524,13 @@ static void loosen_unused_packed_objects(struct 
rev_info *revs)
 }
 
 /*
- * This tracks any options which a reader of the pack might
- * not understand, and which would therefore prevent blind reuse
- * of what we have on disk.
+ * This tracks any options which pack-reuse code expects to be on, or which a
+ * reader of the pack might not understand, and which would therefore prevent
+ * blind reuse of what we have on disk.
  */
 static int pack_options_allow_reuse(void)
 {
-   return allow_ofs_delta;
+   return pack_to_stdout && allow_ofs_delta;
 }
 
 static int get_object_list_from_bitmap(struct rev_info *revs)
@@ -2822,7 +2823,23 @@ int cmd_pack_objects(int argc, const char **argv, const 
char *prefix)
if (!rev_list_all || !rev_list_reflog || !rev_list_index)
unpack_unreachable_expiration = 0;
 
-   if (!use_internal_rev_list || !pack_to_stdout || 
is_repository_shallow())
+   /*
+* "soft" reasons not to use bitmaps - for on-disk repack by default we 
want
+*
+* - to produce good pack (with bitmap index not-yet-packed objects are
+*   packed in suboptimal order).
+*
+* - to use more robust pack-generation codepath (avoiding possible
+*   bugs in bitmap code and possible bitmap index corruption).
+*/
+   if (!pack_to_stdout)
+   use_bitmap_index_default = 0;
+
+   if (use_bitmap_index < 0)
+   use_bitmap_index = use_bitmap_index_default;
+
+   /* "hard" reasons not to use bitmaps; these just won't work at all */
+   if (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) 
|| is_repository_shallow())
use_bitmap_index = 0;
 
if (pack_to_stdout || !rev_list_all)
diff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-

[PATCH 1/2 v8] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use

[Kirill Smelkov]

Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there
are two codepaths in pack-objects: with & without using bitmap
reachability index.

However add_object_entry_from_bitmap(), despite its non-bitmapped
counterpart add_object_entry(), in no way does check for whether --local
or --honor-pack-keep or --incremental should be respected. In
non-bitmapped codepath this is handled in want_object_in_pack(), but
bitmapped codepath has simply no such checking at all.

The bitmapped codepath however was allowing to pass in all those options
and with bitmap indices still being used under such conditions -
potentially giving wrong output (e.g. including objects from non-local or
.keep'ed pack).

We can easily fix this by noting the following: when an object comes to
add_object_entry_from_bitmap() it can come for two reasons:

1. entries coming from main pack covered by bitmap index, and
2. object coming from, possibly alternate, loose or other packs.

"2" can be already handled by want_object_in_pack() and to cover
"1" we can teach want_object_in_pack() to expect that *found_pack can be
non-NULL, meaning calling client already found object's pack entry.

In want_object_in_pack() we care to start the checks from already found
pack, if we have one, this way determining the answer right away
in case neither --local nor --honour-pack-keep are active. In
particular, as p5310-pack-bitmaps.sh shows (3 consecutive runs), we do
not do harm to served-with-bitmap clones performance-wise:

Test  56dfeb62  this tree
-
5310.2: repack to disk9.08(8.20+0.25)   9.09(8.14+0.32) +0.1%
5310.3: simulated clone   1.92(2.12+0.08)   1.93(2.12+0.09) +0.5%
5310.4: simulated fetch   0.82(1.07+0.04)   0.82(1.06+0.04) +0.0%
5310.6: partial bitmap1.96(2.42+0.13)   1.95(2.40+0.15) -0.5%

Test  56dfeb62  this tree
-
5310.2: repack to disk9.11(8.16+0.32)   9.11(8.19+0.28) +0.0%
5310.3: simulated clone   1.93(2.14+0.07)   1.92(2.11+0.10) -0.5%
5310.4: simulated fetch   0.82(1.06+0.04)   0.82(1.04+0.05) +0.0%
5310.6: partial bitmap1.95(2.38+0.16)   1.94(2.39+0.14) -0.5%

Test  56dfeb62  this tree
-
5310.2: repack to disk9.13(8.17+0.31)   9.07(8.13+0.28) -0.7%
5310.3: simulated clone   1.92(2.13+0.07)   1.91(2.12+0.06) -0.5%
5310.4: simulated fetch   0.82(1.08+0.03)   0.82(1.08+0.03) +0.0%
5310.6: partial bitmap1.96(2.43+0.14)   1.96(2.42+0.14) +0.0%

with delta timings showing they are all within noise from run to run.

In the general case we do not want to call find_pack_entry_one() more than
once, because it is expensive. This patch splits the loop in
want_object_in_pack() into two parts: finding the object and seeing if it
impacts our choice to include it in the pack. We may call the inexpensive
want_found_object() twice, but we will never call find_pack_entry_one() if we
do not need to.

I appreciate help and discussing this change with Junio C Hamano and
Jeff King.

Signed-off-by: Kirill Smelkov <k...@nexedi.com>
Signed-off-by: Junio C Hamano <gits...@pobox.com>
---
 builtin/pack-objects.c  | 97 -
 t/t5310-pack-bitmaps.sh | 92 ++
 2 files changed, 156 insertions(+), 33 deletions(-)

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index c4c2a3c..19668d3 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -944,13 +944,48 @@ static int have_duplicate_entry(const unsigned char *sha1,
return 1;
 }
 
+static int want_found_object(int exclude, struct packed_git *p)
+{
+   if (exclude)
+   return 1;
+   if (incremental)
+   return 0;
+
+   /*
+* When asked to do --local (do not include an object that appears in a
+* pack we borrow from elsewhere) or --honor-pack-keep (do not include
+* an object that appears in a pack marked with .keep), finding a pack
+* that matches the criteria is sufficient for us to decide to omit it.
+* However, even if this pack does not satisfy the criteria, we need to
+* make sure no copy of this object appears in _any_ pack that makes us
+* to omit the object, so we need to check all the packs.
+*
+* We can however first check whether these options can possible matter;
+* if they do not matter we know we want the object in generated pack.
+* Otherwise, we signal "-1" at the end to tell the caller that we do
+* not know either way, and it needs to check more packs.
+*/
+   if (!ignore_packed_keep &&
+   (!loc

Re: [PATCH 2/2 v7] pack-objects: use reachability bitmap index when generating non-stdout pack

[Kirill Smelkov]

On Thu, Aug 18, 2016 at 02:06:15PM -0400, Jeff King wrote:
> On Tue, Aug 09, 2016 at 10:32:17PM +0300, Kirill Smelkov wrote:
> 
> > Subject: Re: [PATCH 2/2 v7] pack-objects: use reachability bitmap index when
> >generating non-stdout pack
> 
> This is v7, but as I understand your numbering, it goes with v5 of patch
> 1/2 that I just reviewed (usually we just increment the version number
> on the whole series and treat it as a unit, even if some patches didn't
> change from version to version).

The reason those patches are having their own numbers is that they are
orthogonal to each other and can be applied / rejected independently.
Since I though Junio might want to pick them up as separate topics they
were versioned separately.

But ok, since now we have them considered both together, their next
versions posted will be uniform v8.


> > So we can teach pack-objects to use bitmap index for initial object
> > counting phase when generating resultant pack file too:
> > 
> > - if we care it is not activated under git-repack:
> 
> Do you mean "if we take care that it is not..." here?
> 
> (I think you might just be getting tripped up in the English idioms;
> "care" means that we have a preference; "to take care" means that we are
> being careful).

Ok, I've might have been tripped and thanks for the catch up. I've changed to

"if we take care to not let it be activated under git-repack"

> 
> > - if we know bitmap index generation is not enabled for resultant pack:
> > 
> >   Current code has singleton bitmap_git so cannot work simultaneously
> >   with two bitmap indices.
> 
> Minor English fixes:
> 
>   The current code has a singleton bitmap_git, so it cannot work
>   simultaneously with two bitmap indices.

ok.

> > - if we keep pack reuse enabled still only for "send-to-stdout" case:
> > 
> >   Because on pack reuse raw entries are directly written out to destination
> >   pack by write_reused_pack() bypassing needed for pack index generation
> >   bookkeeping done by regular codepath in write_one() and friends.
> 
> Ditto on English:
> 
>   On pack reuse raw entries are directly written out to the destination
>   pack by write_reused_pack(), bypassing the need for pack index
>   generation bookkeeping done by the regular code path in write_one()
>   and friends.
> 
> I think this is missing the implication. Why wouldn't we want to reuse
> in this case? Certainly we don't when doing a "careful" on-disk repack.
> I suspect the answer is that we cannot write a ".idx" off of the result
> of write_reused_pack(), and write-to-disk always includes the .idx.

Yes, mentioning pack-to-file needs to generate .idx makes it more clear
and thanks for pointing this out. I've changed this item to the
following (picking some of your English corrections):

- if we keep pack reuse enabled still only for "send-to-stdout" case:

  Because pack-to-file needs to generate index for destination pack, and
  currently on pack reuse raw entries are directly written out to the
  destination pack by write_reused_pack(), bypassing needed for pack index
  generation bookkeeping done by regular codepath in write_one() and
  friends.

  ( In the future we might teach pack-reuse code about cases when index
also needs to be generated for resultant pack and remove
pack-reuse-only-for-stdout limitation )

Hope it is ok.

> > More context:
> > 
> > http://marc.info/?t=14679210141=1=2
> 
> Can we turn this into a link to public-inbox? We have just been bit by
> all of our old links to gmane dying, and they cannot easily be replaced
> because they use a gmane-specific article number. public-inbox URLs use
> message-ids, which should be usable for other archives if public-inbox
> goes away.

Yes, makes sense to put msgid here. I've added

http://public-inbox.org/git/20160707190917.20011-1-k...@nexedi.com/T/#t


> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> > index b1007f2..c92d7fc 100644
> > --- a/builtin/pack-objects.c
> > +++ b/builtin/pack-objects.c
> 
> The code here looks fine.

Thanks.

> > diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh
> > index a278d30..9602e9a 100755
> > --- a/t/t5310-pack-bitmaps.sh
> > +++ b/t/t5310-pack-bitmaps.sh
> > @@ -196,6 +196,18 @@ test_expect_success 'pack-objects respects --local 
> > (non-local bitmapped pack)' '
> > ! has_any packbitmap.objects 3b.objects
> >  '
> >  
> > +test_expect_success 'pack-objects to file can use bitmap' '
> > +   # make sure we still have 1 bitmap index from previo

Re: [PATCH 1/2 v5] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use

[Kirill Smelkov]

On Thu, Aug 18, 2016 at 01:52:22PM -0400, Jeff King wrote:
> On Tue, Aug 09, 2016 at 10:31:43PM +0300, Kirill Smelkov wrote:
> 
> > Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there
> > are two codepaths in pack-objects: with & without using bitmap
> > reachability index.
> 
> Sorry, I got distracted from reviewing these patches. I'll give them a
> detailed look now and hopefully we can finalize the topic.

Jeff, thanks for feedback. On my side I'm sorry for the delay because I
was travelling and only recently got back to work.

> > In want_object_in_pack() we care to start the checks from already found
> > pack, if we have one, this way determining the answer right away
> > in case neither --local nor --honour-pack-keep are active. In
> > particular, as p5310-pack-bitmaps.sh shows, we do not do harm to
> > served-with-bitmap clones performance-wise:
> > 
> > Test  56dfeb62  this tree
> > -
> > 5310.2: repack to disk9.63(8.67+0.33)   9.47(8.55+0.28) -1.7%
> > 5310.3: simulated clone   2.07(2.17+0.12)   2.03(2.14+0.12) -1.9%
> > 5310.4: simulated fetch   0.78(1.03+0.02)   0.76(1.00+0.03) -2.6%
> > 5310.6: partial bitmap1.97(2.43+0.15)   1.92(2.36+0.14) -2.5%
> > 
> > with all differences strangely showing we are a bit faster now, but
> > probably all being within noise.
> 
> Good to know there is no regression. It is curious that there is a
> slight _improvement_ across the board. Do we have an explanation for
> that? It seems odd that noise would be so consistent.

Yes, I too thought it and it turned out to be t/perf/run does not copy
config.mak.autogen & friends to build/ and I'm using autoconf with
CFLAGS="-march=native -O3 ..."

Junio, I could not resist to the following:

 8< 
From: Kirill Smelkov <k...@nexedi.com>
Subject: [PATCH] t/perf/run: Don't forget to copy config.mak.autogen & friends
 to build area

Otherwise for people who use autotools-based configure in main worktree,
the performance testing results will be inconsistent as work and build
trees could be using e.g. different optimization levels.

See e.g.


http://public-inbox.org/git/20160818175222.bmm3ivjheokf2...@sigill.intra.peff.net/

for example.

NOTE config.status has to be copied because otherwise without it the build
would want to run reconfigure this way loosing just copied config.mak.autogen.

Signed-off-by: Kirill Smelkov <k...@nexedi.com>
---
 t/perf/run | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/perf/run b/t/perf/run
index cfd7012..aa383c2 100755
--- a/t/perf/run
+++ b/t/perf/run
@@ -30,7 +30,7 @@ unpack_git_rev () {
 }
 build_git_rev () {
rev=$1
-   cp ../../config.mak build/$rev/config.mak
+   cp -t build/$rev ../../{config.mak,config.mak.autogen,config.status}
(cd build/$rev && make $GIT_PERF_MAKE_OPTS) ||
die "failed to build revision '$mydir'"
 }
-- 
2.9.2.701.gf965a18.dirty
 8< 

With corrected t/perf/run the timings are more realistic - e.g. 3
consecutive runs of `./run 56dfeb62 . ./p5310-pack-bitmaps.sh`:

Test  56dfeb62  this tree
-
5310.2: repack to disk9.08(8.20+0.25)   9.09(8.14+0.32) +0.1%
5310.3: simulated clone   1.92(2.12+0.08)   1.93(2.12+0.09) +0.5%
5310.4: simulated fetch   0.82(1.07+0.04)   0.82(1.06+0.04) +0.0%
5310.6: partial bitmap1.96(2.42+0.13)   1.95(2.40+0.15) -0.5%

Test  56dfeb62  this tree
-
5310.2: repack to disk9.11(8.16+0.32)   9.11(8.19+0.28) +0.0%
5310.3: simulated clone   1.93(2.14+0.07)   1.92(2.11+0.10) -0.5%
5310.4: simulated fetch   0.82(1.06+0.04)   0.82(1.04+0.05) +0.0%
5310.6: partial bitmap1.95(2.38+0.16)   1.94(2.39+0.14) -0.5%

Test  56dfeb62  this tree
-
5310.2: repack to disk9.13(8.17+0.31)   9.07(8.13+0.28) -0.7%
5310.3: simulated clone   1.92(2.13+0.07)   1.91(2.12+0.06) -0.5%
5310.4: simulated fetch   0.82(1.08+0.03)   0.82(1.08+0.03) +0.0%
5310.6: partial bitmap1.96(2.43+0.14)   1.96(2.42+0.14) +0.0%



> > And in the general case we care not to have duplicate
> > find_pack_entry_one(*found_pack) calls. Worst what can happen is we can
> > call want_found_object(*found_pack) -- newly introduced helper for
> > checking whether we want object -- twice, but since want_found_object()
> > is very lightweight it does not make any difference.
> 
> I had trouble parsing this. I think maybe:
> 
>   In the general case we do not want to c

[PATCH 2/2 v7] pack-objects: use reachability bitmap index when generating non-stdout pack

[Kirill Smelkov]

72s,

while

`pack-objects file.pack` which does both pack and index is  27s.

And even

`pack-objects --no-use-bitmap-index file.pack`  is  37s.

Jeff explains:

The packfile does not carry the sha1 of the objects. A receiving
index-pack has to compute them itself, including inflating and applying
all of the deltas.

that's why for `git-backup restore` we want to teach `git pack-objects
file.pack` to use bitmaps instead of using `git pack-objects --stdout
>file.pack` + `git index-pack file.pack`.

More context:

http://marc.info/?t=146792101400001=1=2

Cc: Vicent Marti <tan...@gmail.com>
Helped-by: Jeff King <p...@peff.net>
Signed-off-by: Kirill Smelkov <k...@nexedi.com>
---
 builtin/pack-objects.c  | 31 ---
 t/t5310-pack-bitmaps.sh | 12 
 2 files changed, 36 insertions(+), 7 deletions(-)

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index b1007f2..c92d7fc 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -67,7 +67,8 @@ static struct packed_git *reuse_packfile;
 static uint32_t reuse_packfile_objects;
 static off_t reuse_packfile_offset;
 
-static int use_bitmap_index = 1;
+static int use_bitmap_index_default = 1;
+static int use_bitmap_index = -1;
 static int write_bitmap_index;
 static uint16_t write_bitmap_options;
 
@@ -2270,7 +2271,7 @@ static int git_pack_config(const char *k, const char *v, 
void *cb)
write_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;
}
if (!strcmp(k, "pack.usebitmaps")) {
-   use_bitmap_index = git_config_bool(k, v);
+   use_bitmap_index_default = git_config_bool(k, v);
return 0;
}
if (!strcmp(k, "pack.threads")) {
@@ -2519,13 +2520,13 @@ static void loosen_unused_packed_objects(struct 
rev_info *revs)
 }
 
 /*
- * This tracks any options which a reader of the pack might
- * not understand, and which would therefore prevent blind reuse
- * of what we have on disk.
+ * This tracks any options which pack-reuse code expects to be on, or which a
+ * reader of the pack might not understand, and which would therefore prevent
+ * blind reuse of what we have on disk.
  */
 static int pack_options_allow_reuse(void)
 {
-   return allow_ofs_delta;
+   return pack_to_stdout && allow_ofs_delta;
 }
 
 static int get_object_list_from_bitmap(struct rev_info *revs)
@@ -2818,7 +2819,23 @@ int cmd_pack_objects(int argc, const char **argv, const 
char *prefix)
if (!rev_list_all || !rev_list_reflog || !rev_list_index)
unpack_unreachable_expiration = 0;
 
-   if (!use_internal_rev_list || !pack_to_stdout || 
is_repository_shallow())
+   /*
+* "soft" reasons not to use bitmaps - for on-disk repack by default we 
want
+*
+* - to produce good pack (with bitmap index not-yet-packed objects are
+*   packed in suboptimal order).
+*
+* - to use more robust pack-generation codepath (avoiding possible
+*   bugs in bitmap code and possible bitmap index corruption).
+*/
+   if (!pack_to_stdout)
+   use_bitmap_index_default = 0;
+
+   if (use_bitmap_index < 0)
+   use_bitmap_index = use_bitmap_index_default;
+
+   /* "hard" reasons not to use bitmaps; these just won't work at all */
+   if (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) 
|| is_repository_shallow())
use_bitmap_index = 0;
 
if (pack_to_stdout || !rev_list_all)
diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh
index a278d30..9602e9a 100755
--- a/t/t5310-pack-bitmaps.sh
+++ b/t/t5310-pack-bitmaps.sh
@@ -196,6 +196,18 @@ test_expect_success 'pack-objects respects --local 
(non-local bitmapped pack)' '
! has_any packbitmap.objects 3b.objects
 '
 
+test_expect_success 'pack-objects to file can use bitmap' '
+   # make sure we still have 1 bitmap index from previous tests
+   ls .git/objects/pack/ | grep bitmap >output &&
+   test_line_count = 1 output &&
+   # verify equivalent packs are generated with/without using bitmap index
+   packasha1=$(git pack-objects --no-use-bitmap-index --all packa 
packa.objects &&
+   list_packed_objects <packb-$packbsha1.idx >packb.objects &&
+   test_cmp packa.objects packb.objects
+'
+
 test_expect_success 'full repack, reusing previous bitmaps' '
git repack -ad &&
ls .git/objects/pack/ | grep bitmap >output &&
-- 
2.9.2.701.gf965a18.dirty
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 1/2 v5] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use

[Kirill Smelkov]

Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there
are two codepaths in pack-objects: with & without using bitmap
reachability index.

However add_object_entry_from_bitmap(), despite its non-bitmapped
counterpart add_object_entry(), in no way does check for whether --local
or --honor-pack-keep or --incremental should be respected. In
non-bitmapped codepath this is handled in want_object_in_pack(), but
bitmapped codepath has simply no such checking at all.

The bitmapped codepath however was allowing to pass in all those options
and with bitmap indices still being used under such conditions -
potentially giving wrong output (e.g. including objects from non-local or
.keep'ed pack).

We can easily fix this by noting the following: when an object comes to
add_object_entry_from_bitmap() it can come for two reasons:

1. entries coming from main pack covered by bitmap index, and
2. object coming from, possibly alternate, loose or other packs.

"2" can be already handled by want_object_in_pack() and to cover
"1" we can teach want_object_in_pack() to expect that *found_pack can be
non-NULL, meaning calling client already found object's pack entry.

In want_object_in_pack() we care to start the checks from already found
pack, if we have one, this way determining the answer right away
in case neither --local nor --honour-pack-keep are active. In
particular, as p5310-pack-bitmaps.sh shows, we do not do harm to
served-with-bitmap clones performance-wise:

Test  56dfeb62  this tree
-
5310.2: repack to disk9.63(8.67+0.33)   9.47(8.55+0.28) -1.7%
5310.3: simulated clone   2.07(2.17+0.12)   2.03(2.14+0.12) -1.9%
5310.4: simulated fetch   0.78(1.03+0.02)   0.76(1.00+0.03) -2.6%
5310.6: partial bitmap1.97(2.43+0.15)   1.92(2.36+0.14) -2.5%

with all differences strangely showing we are a bit faster now, but
probably all being within noise.

And in the general case we care not to have duplicate
find_pack_entry_one(*found_pack) calls. Worst what can happen is we can
call want_found_object(*found_pack) -- newly introduced helper for
checking whether we want object -- twice, but since want_found_object()
is very lightweight it does not make any difference.

I appreciate help and discussing this change with Junio C Hamano and
Jeff King.

Signed-off-by: Kirill Smelkov <k...@nexedi.com>
---
 builtin/pack-objects.c  | 93 +++--
 t/t5310-pack-bitmaps.sh | 92 
 2 files changed, 152 insertions(+), 33 deletions(-)

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index c4c2a3c..b1007f2 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -944,13 +944,44 @@ static int have_duplicate_entry(const unsigned char *sha1,
return 1;
 }
 
+static int want_found_object(int exclude, struct packed_git *p)
+{
+   if (exclude)
+   return 1;
+   if (incremental)
+   return 0;
+
+   /*
+* When asked to do --local (do not include an object that appears in a
+* pack we borrow from elsewhere) or --honor-pack-keep (do not include
+* an object that appears in a pack marked with .keep), finding a pack
+* that matches the criteria is sufficient for us to decide to omit it.
+* However, even if this pack does not satisfy the criteria, we need to
+* make sure no copy of this object appears in _any_ pack that makes us
+* to omit the object, so we need to check all the packs. Signal that by
+* returning -1 to the caller.
+*/
+   if (!ignore_packed_keep &&
+   (!local || !have_non_local_packs))
+   return 1;
+
+   if (local && !p->pack_local)
+   return 0;
+   if (ignore_packed_keep && p->pack_local && p->pack_keep)
+   return 0;
+
+   /* we don't know yet; keep looking for more packs */
+   return -1;
+}
+
 /*
  * Check whether we want the object in the pack (e.g., we do not want
  * objects found in non-local stores if the "--local" option was used).
  *
- * As a side effect of this check, we will find the packed version of this
- * object, if any. We therefore pass out the pack information to avoid having
- * to look it up again later.
+ * If the caller already knows an existing pack it wants to take the object
+ * from, that is passed in *found_pack and *found_offset; otherwise this
+ * function finds if there is any pack that has the object and returns the pack
+ * and its offset in these variables.
  */
 static int want_object_in_pack(const unsigned char *sha1,
   int exclude,
@@ -958,15 +989,30 @@ static int want_object_in_pack(const unsigned char *sha1,
   off_t *found_offset)
 

Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental

[Kirill Smelkov]

On Tue, Aug 09, 2016 at 09:52:18AM -0700, Junio C Hamano wrote:
> Kirill Smelkov <k...@nexedi.com> writes:
> 
> > Would you please explain why we should not use touch if we do not care
> > about timestamps? Simply style?
> 
> To help readers.
> 
> "touch A" forcess the readers wonder "does the timestamp of A
> matter, and if so in what way?" and "does any later test care what
> is _in_ A, and if so in what way?"  Both of them is wasting their
> time when there is no reason why "touch" should have been used. 

I see, thanks for explaining. I used to read it a bit the other way;
maybe it is just an environment difference.


> > diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh
> > index cce95d8..44914ac 100755
> > --- a/t/t5310-pack-bitmaps.sh
> > +++ b/t/t5310-pack-bitmaps.sh
> > @@ -8,16 +8,15 @@ objpath () {
> >  }
> >  
> >  # show objects present in pack ($1 should be associated *.idx)
> > -packobjects () {
> > -   git show-index <$1 | cut -d' ' -f2
> > +pack_list_objects () {
> > +   git show-index <"$1" | cut -d' ' -f2
> >  }
> 
> pack-list-objects still sounds as if you are packing "list objects",
> though.  If you are listing packed objects (or objects in a pack),
> list-packed-objects (or list-objects-in-pack) reads clearer and more
> to the point, at least to me.

Ok, let it be list_packed_objects().


> > -# hasany pattern-file content-file
> > +# has_any pattern-file content-file
> >  # tests whether content-file has any entry from pattern-file with entries 
> > being
> >  # whole lines.
> > -hasany () {
> > -   # NOTE `grep -f` is not portable
> > -   git grep --no-index -qFf $1 $2
> > +has_any () {
> > +   grep -qFf "$1" "$2"
> 
> Omitting "-q" would help those who have to debug breakage in this
> test or the code that this test checks.  What test_expect_success
> outputs is not shown by default, and running the test script with
> "-v" would show them as a debugging aid.

Ok, makes sense. Both patches adjusted and will be reposted.

Thanks,
Kirill
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 2/2 v6] pack-objects: use reachability bitmap index when generating non-stdout pack

[Kirill Smelkov]

72s,

while

`pack-objects file.pack` which does both pack and index is  27s.

And even

`pack-objects --no-use-bitmap-index file.pack`  is  37s.

Jeff explains:

The packfile does not carry the sha1 of the objects. A receiving
index-pack has to compute them itself, including inflating and applying
all of the deltas.

that's why for `git-backup restore` we want to teach `git pack-objects
file.pack` to use bitmaps instead of using `git pack-objects --stdout
>file.pack` + `git index-pack file.pack`.

More context:

http://marc.info/?t=146792101400001=1=2

Cc: Vicent Marti <tan...@gmail.com>
Helped-by: Jeff King <p...@peff.net>
Signed-off-by: Kirill Smelkov <k...@nexedi.com>
---
 builtin/pack-objects.c  | 31 ---
 t/t5310-pack-bitmaps.sh | 12 
 2 files changed, 36 insertions(+), 7 deletions(-)

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index b1007f2..c92d7fc 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -67,7 +67,8 @@ static struct packed_git *reuse_packfile;
 static uint32_t reuse_packfile_objects;
 static off_t reuse_packfile_offset;
 
-static int use_bitmap_index = 1;
+static int use_bitmap_index_default = 1;
+static int use_bitmap_index = -1;
 static int write_bitmap_index;
 static uint16_t write_bitmap_options;
 
@@ -2270,7 +2271,7 @@ static int git_pack_config(const char *k, const char *v, 
void *cb)
write_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;
}
if (!strcmp(k, "pack.usebitmaps")) {
-   use_bitmap_index = git_config_bool(k, v);
+   use_bitmap_index_default = git_config_bool(k, v);
return 0;
}
if (!strcmp(k, "pack.threads")) {
@@ -2519,13 +2520,13 @@ static void loosen_unused_packed_objects(struct 
rev_info *revs)
 }
 
 /*
- * This tracks any options which a reader of the pack might
- * not understand, and which would therefore prevent blind reuse
- * of what we have on disk.
+ * This tracks any options which pack-reuse code expects to be on, or which a
+ * reader of the pack might not understand, and which would therefore prevent
+ * blind reuse of what we have on disk.
  */
 static int pack_options_allow_reuse(void)
 {
-   return allow_ofs_delta;
+   return pack_to_stdout && allow_ofs_delta;
 }
 
 static int get_object_list_from_bitmap(struct rev_info *revs)
@@ -2818,7 +2819,23 @@ int cmd_pack_objects(int argc, const char **argv, const 
char *prefix)
if (!rev_list_all || !rev_list_reflog || !rev_list_index)
unpack_unreachable_expiration = 0;
 
-   if (!use_internal_rev_list || !pack_to_stdout || 
is_repository_shallow())
+   /*
+* "soft" reasons not to use bitmaps - for on-disk repack by default we 
want
+*
+* - to produce good pack (with bitmap index not-yet-packed objects are
+*   packed in suboptimal order).
+*
+* - to use more robust pack-generation codepath (avoiding possible
+*   bugs in bitmap code and possible bitmap index corruption).
+*/
+   if (!pack_to_stdout)
+   use_bitmap_index_default = 0;
+
+   if (use_bitmap_index < 0)
+   use_bitmap_index = use_bitmap_index_default;
+
+   /* "hard" reasons not to use bitmaps; these just won't work at all */
+   if (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) 
|| is_repository_shallow())
use_bitmap_index = 0;
 
if (pack_to_stdout || !rev_list_all)
diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh
index a50d867..44914ac 100755
--- a/t/t5310-pack-bitmaps.sh
+++ b/t/t5310-pack-bitmaps.sh
@@ -196,6 +196,18 @@ test_expect_success 'pack-objects respects --local 
(non-local bitmapped pack)' '
! has_any packbitmap.objects 3b.objects
 '
 
+test_expect_success 'pack-objects to file can use bitmap' '
+   # make sure we still have 1 bitmap index from previous tests
+   ls .git/objects/pack/ | grep bitmap >output &&
+   test_line_count = 1 output &&
+   # verify equivalent packs are generated with/without using bitmap index
+   packasha1=$(git pack-objects --no-use-bitmap-index --all packa 
packa.objects &&
+   pack_list_objects <packb-$packbsha1.idx >packb.objects &&
+   test_cmp packa.objects packb.objects
+'
+
 test_expect_success 'full repack, reusing previous bitmaps' '
git repack -ad &&
ls .git/objects/pack/ | grep bitmap >output &&
-- 
2.9.2.701.gf965a18.dirty
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 1/2 v4] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use

[Kirill Smelkov]

Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there
are two codepaths in pack-objects: with & without using bitmap
reachability index.

However add_object_entry_from_bitmap(), despite its non-bitmapped
counterpart add_object_entry(), in no way does check for whether --local
or --honor-pack-keep or --incremental should be respected. In
non-bitmapped codepath this is handled in want_object_in_pack(), but
bitmapped codepath has simply no such checking at all.

The bitmapped codepath however was allowing to pass in all those options
and with bitmap indices still being used under such conditions -
potentially giving wrong output (e.g. including objects from non-local or
.keep'ed pack).

We can easily fix this by noting the following: when an object comes to
add_object_entry_from_bitmap() it can come for two reasons:

1. entries coming from main pack covered by bitmap index, and
2. object coming from, possibly alternate, loose or other packs.

"2" can be already handled by want_object_in_pack() and to cover
"1" we can teach want_object_in_pack() to expect that *found_pack can be
non-NULL, meaning calling client already found object's pack entry.

In want_object_in_pack() we care to start the checks from already found
pack, if we have one, this way determining the answer right away
in case neither --local nor --honour-pack-keep are active. In
particular, as p5310-pack-bitmaps.sh shows, we do not do harm to
served-with-bitmap clones performance-wise:

Test  56dfeb62  this tree
-
5310.2: repack to disk9.63(8.67+0.33)   9.47(8.55+0.28) -1.7%
5310.3: simulated clone   2.07(2.17+0.12)   2.03(2.14+0.12) -1.9%
5310.4: simulated fetch   0.78(1.03+0.02)   0.76(1.00+0.03) -2.6%
5310.6: partial bitmap1.97(2.43+0.15)   1.92(2.36+0.14) -2.5%

with all differences strangely showing we are a bit faster now, but
probably all being within noise.

And in the general case we care not to have duplicate
find_pack_entry_one(*found_pack) calls. Worst what can happen is we can
call want_found_object(*found_pack) -- newly introduced helper for
checking whether we want object -- twice, but since want_found_object()
is very lightweight it does not make any difference.

I appreciate help and discussing this change with Junio C Hamano and
Jeff King.

Signed-off-by: Kirill Smelkov <k...@nexedi.com>
---
 builtin/pack-objects.c  | 93 +++--
 t/t5310-pack-bitmaps.sh | 92 
 2 files changed, 152 insertions(+), 33 deletions(-)

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index c4c2a3c..b1007f2 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -944,13 +944,44 @@ static int have_duplicate_entry(const unsigned char *sha1,
return 1;
 }
 
+static int want_found_object(int exclude, struct packed_git *p)
+{
+   if (exclude)
+   return 1;
+   if (incremental)
+   return 0;
+
+   /*
+* When asked to do --local (do not include an object that appears in a
+* pack we borrow from elsewhere) or --honor-pack-keep (do not include
+* an object that appears in a pack marked with .keep), finding a pack
+* that matches the criteria is sufficient for us to decide to omit it.
+* However, even if this pack does not satisfy the criteria, we need to
+* make sure no copy of this object appears in _any_ pack that makes us
+* to omit the object, so we need to check all the packs. Signal that by
+* returning -1 to the caller.
+*/
+   if (!ignore_packed_keep &&
+   (!local || !have_non_local_packs))
+   return 1;
+
+   if (local && !p->pack_local)
+   return 0;
+   if (ignore_packed_keep && p->pack_local && p->pack_keep)
+   return 0;
+
+   /* we don't know yet; keep looking for more packs */
+   return -1;
+}
+
 /*
  * Check whether we want the object in the pack (e.g., we do not want
  * objects found in non-local stores if the "--local" option was used).
  *
- * As a side effect of this check, we will find the packed version of this
- * object, if any. We therefore pass out the pack information to avoid having
- * to look it up again later.
+ * If the caller already knows an existing pack it wants to take the object
+ * from, that is passed in *found_pack and *found_offset; otherwise this
+ * function finds if there is any pack that has the object and returns the pack
+ * and its offset in these variables.
  */
 static int want_object_in_pack(const unsigned char *sha1,
   int exclude,
@@ -958,15 +989,30 @@ static int want_object_in_pack(const unsigned char *sha1,
   off_t *found_offset)
 

Re: [PATCH v5] pack-objects: teach it to use reachability bitmap index when generating non-stdout pack too

[Kirill Smelkov]

On Mon, Aug 08, 2016 at 01:53:20PM -0700, Junio C Hamano wrote:
> Kirill Smelkov <k...@nexedi.com> writes:
> 
> > diff --git a/Documentation/config.txt b/Documentation/config.txt
> > index bc1c433..4ba0c4a 100644
> > --- a/Documentation/config.txt
> > +++ b/Documentation/config.txt
> > @@ -2244,6 +2244,9 @@ pack.useBitmaps::
> > to stdout (e.g., during the server side of a fetch). Defaults to
> > true. You should not generally need to turn this off unless
> > you are debugging pack bitmaps.
> > ++
> > +*NOTE*: when packing to file (e.g., on repack) the default is always not 
> > to use
> > +   pack bitmaps.
> 
> This is a bit hard to read and understand.
> 
> The patched result starts with "When true, git will use bitmap when
> packing to stdout", i.e. when packing to file, git will not.  So
> this *NOTE* is repeating the same thing.  The reader is made to
> wonder "Why does it need to repeat the same thing?  Does this mean
> when the variable is set, a pack sent to a disk uses the bitmap?"
> 
> I think what you actually do in the code is to make the variable
> affect _only_ the standard-output case, and users need a command
> line option if they want to use bitmap when writing to a file (the
> code to do so looks correctly done).

Yes it is this way how it is programmed. But I've added the note because
it is very implicit to me that "When true, git will use bitmap when
packing to stdout" means 1) the default for packing-to-file is different
and 2) there is no way to set the default for packing-to-file. That's
why I added the explicit info.

And especially since the config name "pack.useBitmaps" does not contain
"stdout" at all it can be very confusing to people looking at this the
first time (at least it was so this way for me). Also please recall you
wondering why 6b8fda2d added bitmap support only for to-stdout case not
even mentioning about why it is done only for that case and not for
to-file case).

I do not insist on the note however - I only thought it is better to
have it - so if you prefer we go without it - let us drop this note.

Will send v6 as reply to this mail with below interdiff.

Thanks,
Kirill

 8<  (interdiff)

--- b/Documentation/config.txt
+++ a/Documentation/config.txt
@@ -2246,9 +2246,6 @@
to stdout (e.g., during the server side of a fetch). Defaults to
true. You should not generally need to turn this off unless
you are debugging pack bitmaps.
-+
-*NOTE*: when packing to file (e.g., on repack) the default is always not to use
-   pack bitmaps.
 
 pack.writeBitmaps (deprecated)::
This is a deprecated synonym for `repack.writeBitmaps`.
diff -u b/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh
--- b/t/t5310-pack-bitmaps.sh
+++ b/t/t5310-pack-bitmaps.sh
@@ -219,8 +219,8 @@
# verify equivalent packs are generated with/without using bitmap index
packasha1=$(git pack-objects --no-use-bitmap-index --all packa 
packa.objects &&
-   git show-index <packb-$packbsha1.idx | cut -d" " -f2 >packb.objects &&
+   pack_list_objects <packa-$packasha1.idx >packa.objects &&
+   pack_list_objects <packb-$packbsha1.idx >packb.objects &&
test_cmp packa.objects packb.objects
 '
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental

[Kirill Smelkov]

Junio, first of all thanks for feedback,

On Mon, Aug 08, 2016 at 12:26:33PM -0700, Junio C Hamano wrote:
> Kirill Smelkov <k...@nexedi.com> writes:
[...]
> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> > index c4c2a3c..e06c1bf 100644
> > --- a/builtin/pack-objects.c
> > +++ b/builtin/pack-objects.c
> > @@ -944,13 +944,45 @@ static int have_duplicate_entry(const unsigned char 
> > *sha1,
> > return 1;
> >  }
> >  
> > +static int want_found_object(int exclude, struct packed_git *p)
> > +{
> > +   if (exclude)
> > +   return 1;
> > +   if (incremental)
> > +   return 0;
> > +
> > +   /*
> > +* When asked to do --local (do not include an
> > +* object that appears in a pack we borrow
> > +* from elsewhere) or --honor-pack-keep (do not
> > +* include an object that appears in a pack marked
> > +* with .keep), we need to make sure no copy of this
> > +* object come from in _any_ pack that causes us to
> > +* omit it, and need to complete this loop.  When
> > +* neither option is in effect, we know the object
> > +* we just found is going to be packed, so break
> > +* out of the search loop now.
> > +*/
> 
> The blame is mine, but "no copy of this object appears in _any_ pack"
> would be more correct and easier to read.
> 
> This code is no longer in a search loop; its caller is.  Further
> rephrasing is needed.  "When asked to do ...these things..., finding
> a pack that matches the criteria is sufficient for us to decide to
> omit it.  However, even if this pack does not satisify the criteria,
> we need to make sure no copy of this object appears in _any_ pack
> that makes us to omit the object, so we need to check all the packs.
> Signal that by returning -1 to the caller." or something along that
> line.

Ok, I've rephrased it your way. Thanks for advising.

> >  /*
> >   * Check whether we want the object in the pack (e.g., we do not want
> >   * objects found in non-local stores if the "--local" option was used).
> >   *
> > - * As a side effect of this check, we will find the packed version of this
> > - * object, if any. We therefore pass out the pack information to avoid 
> > having
> > - * to look it up again later.
> > + * As a side effect of this check, if object's pack entry was not already 
> > found,
> > + * we will find the packed version of this object, if any. We therefore 
> > pass
> > + * out the pack information to avoid having to look it up again later.
> 
> The reasoning leading to "We therefore" is understandable, but "pass
> out the pack information" is not quite.  Is this meant to explain
> the fact that *found_pack and *found_offset are in-out parameters?
> 
> The explanation to justify why *found_pack and *found_offset that
> used to be out parameters are made in-out parameters belongs to the
> log message.  We do not want this in-code comment to explain the
> updated code relative to what the code used to do; that is not
> useful to those who read the code for the first time in the context
> of the committed state.
> 
> /* 
>  * Check whether we want to pack the object in the pack (e.g. ...).
>  *
>  * If the caller already knows an existing pack it wants to
>  * take the object from, that is passed in *found_pack and
>  * *found_offset; otherwise this function finds if there is
>  * any pack that has the object and returns the pack and its
>  * offset in these variables.
>  */

The "pass out the pack information ..." is not my text - I only added
"if object's pack entry was not already found" in the middle of the
sentence and rewrapped this paragraph. The "pass out the pack
information ..." comes from ce2bc424 (pack-objects: split
add_object_entry; 2013-12-21)

I agree your text is more clear and it is better to adjust the comments.

> > diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh
> > index 3893afd..e71caa4 100755
> > --- a/t/t5310-pack-bitmaps.sh
> > +++ b/t/t5310-pack-bitmaps.sh
> > @@ -7,6 +7,19 @@ objpath () {
> > echo ".git/objects/$(echo "$1" | sed -e 's|\(..\)|\1/|')"
> >  }
> >  
> > +# show objects present in pack ($1 should be associated *.idx)
> > +packobjects () {
> > +   git show-index <$1 | cut -d' ' -f2
> > +}
> 
> That is a misleading name for a helper function that produces a list
> of objects that were packed.  "list_packed_objects", perhaps.

I a

Re: [PATCH v4 2/2] pack-objects: Teach it to use reachability bitmap index when generating non-stdout pack too

[Kirill Smelkov]

On Mon, Aug 08, 2016 at 11:28:02AM -0700, Junio C Hamano wrote:
> Kirill Smelkov <k...@nexedi.com> writes:
> 
> > Another question: I'm preparing another version of "pack-objects: Teach
> > --use-bitmap-index codepath to  respect --local ..." and was going to
> > put
> >
> > ( updated patch is in the end of this mail )
> >
> > in the top of the message. Is it ok or better not to do so and just respin
> > the patch in its own separate mail?
> 
> That would force those who pick leftover bits to _open_ and read a
> first few lines.
> 
> Definitely it is better than burying a patch after 60+ lines, but a
> separate patch with incremented "[PATCH v6 1/2]" on the subject line
> beats it hands-down from discoverability's point of view.

Thanks, I see. I've resent both patches as separate mails.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v3] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental

[Kirill Smelkov]

Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there
are two codepaths in pack-objects: with & without using bitmap
reachability index.

However add_object_entry_from_bitmap(), despite its non-bitmapped
counterpart add_object_entry(), in no way does check for whether --local
or --honor-pack-keep or --incremental should be respected. In
non-bitmapped codepath this is handled in want_object_in_pack(), but
bitmapped codepath has simply no such checking at all.

The bitmapped codepath however was allowing to pass in all those options
and with bitmap indices still being used under such conditions -
potentially giving wrong output (e.g. including objects from non-local or
.keep'ed pack).

We can easily fix this by noting the following: when an object comes to
add_object_entry_from_bitmap() it can come for two reasons:

1. entries coming from main pack covered by bitmap index, and
2. object coming from, possibly alternate, loose or other packs.

"2" can be already handled by want_object_in_pack() and to cover
"1" we can teach want_object_in_pack() to expect that *found_pack can be
non-NULL, meaning calling client already found object's pack entry.

In want_object_in_pack() we care to start the checks from already found
pack, if we have one, this way determining the answer right away
in case neither --local nor --honour-pack-keep are active. In
particular, as p5310-pack-bitmaps.sh shows, we do not do harm to
served-with-bitmap clones performance-wise:

Test  56dfeb62  this tree
-
5310.2: repack to disk9.63(8.67+0.33)   9.47(8.55+0.28) -1.7%
5310.3: simulated clone   2.07(2.17+0.12)   2.03(2.14+0.12) -1.9%
5310.4: simulated fetch   0.78(1.03+0.02)   0.76(1.00+0.03) -2.6%
5310.6: partial bitmap1.97(2.43+0.15)   1.92(2.36+0.14) -2.5%

with all differences strangely showing we are a bit faster now, but
probably all being within noise.

And in the general case we care not to have duplicate
find_pack_entry_one(*found_pack) calls. Worst what can happen is we can
call want_found_object(*found_pack) -- newly introduced helper for
checking whether we want object -- twice, but since want_found_object()
is very lightweight it does not make any difference.

I appreciate help and discussing this change with Junio C Hamano and
Jeff King.

Signed-off-by: Kirill Smelkov <k...@nexedi.com>
---
 builtin/pack-objects.c  |  94 ++--
 t/t5310-pack-bitmaps.sh | 111 
 2 files changed, 172 insertions(+), 33 deletions(-)

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index c4c2a3c..e06c1bf 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -944,13 +944,45 @@ static int have_duplicate_entry(const unsigned char *sha1,
return 1;
 }
 
+static int want_found_object(int exclude, struct packed_git *p)
+{
+   if (exclude)
+   return 1;
+   if (incremental)
+   return 0;
+
+   /*
+* When asked to do --local (do not include an
+* object that appears in a pack we borrow
+* from elsewhere) or --honor-pack-keep (do not
+* include an object that appears in a pack marked
+* with .keep), we need to make sure no copy of this
+* object come from in _any_ pack that causes us to
+* omit it, and need to complete this loop.  When
+* neither option is in effect, we know the object
+* we just found is going to be packed, so break
+* out of the search loop now.
+*/
+   if (!ignore_packed_keep &&
+   (!local || !have_non_local_packs))
+   return 1;
+
+   if (local && !p->pack_local)
+   return 0;
+   if (ignore_packed_keep && p->pack_local && p->pack_keep)
+   return 0;
+
+   /* we don't know yet; keep looking for more packs */
+   return -1;
+}
+
 /*
  * Check whether we want the object in the pack (e.g., we do not want
  * objects found in non-local stores if the "--local" option was used).
  *
- * As a side effect of this check, we will find the packed version of this
- * object, if any. We therefore pass out the pack information to avoid having
- * to look it up again later.
+ * As a side effect of this check, if object's pack entry was not already 
found,
+ * we will find the packed version of this object, if any. We therefore pass
+ * out the pack information to avoid having to look it up again later.
  */
 static int want_object_in_pack(const unsigned char *sha1,
   int exclude,
@@ -958,15 +990,30 @@ static int want_object_in_pack(const unsigned char *sha1,
   off_t *found_offset)
 {
struct packed_git *p;
+   int want;
 
if (!exclude && local &

[PATCH v5] pack-objects: teach it to use reachability bitmap index when generating non-stdout pack too

[Kirill Smelkov]

72s,

while

`pack-objects file.pack` which does both pack and index is  27s.

And even

`pack-objects --no-use-bitmap-index file.pack`  is  37s.

Jeff explains:

The packfile does not carry the sha1 of the objects. A receiving
index-pack has to compute them itself, including inflating and applying
all of the deltas.

that's why for `git-backup restore` we want to teach `git pack-objects
file.pack` to use bitmaps instead of using `git pack-objects --stdout
>file.pack` + `git index-pack file.pack`.

More context:

http://marc.info/?t=146792101400001=1=2

Cc: Vicent Marti <tan...@gmail.com>
Helped-by: Jeff King <p...@peff.net>
Signed-off-by: Kirill Smelkov <k...@nexedi.com>
---
 Documentation/config.txt |  3 +++
 builtin/pack-objects.c   | 31 ---
 t/t5310-pack-bitmaps.sh  | 12 
 3 files changed, 39 insertions(+), 7 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index bc1c433..4ba0c4a 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -2244,6 +2244,9 @@ pack.useBitmaps::
to stdout (e.g., during the server side of a fetch). Defaults to
true. You should not generally need to turn this off unless
you are debugging pack bitmaps.
++
+*NOTE*: when packing to file (e.g., on repack) the default is always not to use
+   pack bitmaps.
 
 pack.writeBitmaps (deprecated)::
This is a deprecated synonym for `repack.writeBitmaps`.
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 92e2e5f..0a89e8d 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -66,7 +66,8 @@ static struct packed_git *reuse_packfile;
 static uint32_t reuse_packfile_objects;
 static off_t reuse_packfile_offset;
 
-static int use_bitmap_index = 1;
+static int use_bitmap_index_default = 1;
+static int use_bitmap_index = -1;
 static int write_bitmap_index;
 static uint16_t write_bitmap_options;
 
@@ -2226,7 +2227,7 @@ static int git_pack_config(const char *k, const char *v, 
void *cb)
write_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;
}
if (!strcmp(k, "pack.usebitmaps")) {
-   use_bitmap_index = git_config_bool(k, v);
+   use_bitmap_index_default = git_config_bool(k, v);
return 0;
}
if (!strcmp(k, "pack.threads")) {
@@ -2475,13 +2476,13 @@ static void loosen_unused_packed_objects(struct 
rev_info *revs)
 }
 
 /*
- * This tracks any options which a reader of the pack might
- * not understand, and which would therefore prevent blind reuse
- * of what we have on disk.
+ * This tracks any options which pack-reuse code expects to be on, or which a
+ * reader of the pack might not understand, and which would therefore prevent
+ * blind reuse of what we have on disk.
  */
 static int pack_options_allow_reuse(void)
 {
-   return allow_ofs_delta;
+   return pack_to_stdout && allow_ofs_delta;
 }
 
 static int get_object_list_from_bitmap(struct rev_info *revs)
@@ -2774,7 +2775,23 @@ int cmd_pack_objects(int argc, const char **argv, const 
char *prefix)
if (!rev_list_all || !rev_list_reflog || !rev_list_index)
unpack_unreachable_expiration = 0;
 
-   if (!use_internal_rev_list || !pack_to_stdout || 
is_repository_shallow())
+   /*
+* "soft" reasons not to use bitmaps - for on-disk repack by default we 
want
+*
+* - to produce good pack (with bitmap index not-yet-packed objects are
+*   packed in suboptimal order).
+*
+* - to use more robust pack-generation codepath (avoiding possible
+*   bugs in bitmap code and possible bitmap index corruption).
+*/
+   if (!pack_to_stdout)
+   use_bitmap_index_default = 0;
+
+   if (use_bitmap_index < 0)
+   use_bitmap_index = use_bitmap_index_default;
+
+   /* "hard" reasons not to use bitmaps; these just won't work at all */
+   if (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) 
|| is_repository_shallow())
use_bitmap_index = 0;
 
if (pack_to_stdout || !rev_list_all)
diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh
index 3893afd..ffecc6a 100755
--- a/t/t5310-pack-bitmaps.sh
+++ b/t/t5310-pack-bitmaps.sh
@@ -118,6 +118,18 @@ test_expect_success 'incremental repack can disable 
bitmaps' '
git repack -d --no-write-bitmap-index
 '
 
+test_expect_success 'pack-objects to file can use bitmap' '
+   # make sure we still have 1 bitmap index from previous tests
+   ls .git/objects/pack/ | grep bitmap >output &&
+   test_line_count = 1 output &&
+   # verify equivalent packs are generated with/without using bitmap index
+   packasha1=$(git pack-objects --no-use-bitmap-index --all packa 
packa.objects &&
+   git show-index <

Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental

[Kirill Smelkov]

( updated patch is in the end of this mail )

Jeff, first of all thanks for commenting,

On Mon, Aug 08, 2016 at 09:50:20AM -0400, Jeff King wrote:
> On Mon, Aug 08, 2016 at 03:37:35PM +0300, Kirill Smelkov wrote:
> 
> > @@ -958,15 +958,30 @@ static int want_object_in_pack(const unsigned char 
> > *sha1,
> >off_t *found_offset)
> >  {
> > struct packed_git *p;
> > +   struct packed_git *pack1 = *found_pack;
> > +   int pack1_seen = !pack1;
> >  
> > if (!exclude && local && has_loose_object_nonlocal(sha1))
> > return 0;
> >  
> > -   *found_pack = NULL;
> > -   *found_offset = 0;
> > +   /*
> > +* If we already know the pack object lives in, start checks from that
> > +* pack - in the usual case when neither --local was given nor .keep 
> > files
> > +* are present the loop will degenerate to have only 1 iteration.
> > +*/
> > +   for (p = (pack1 ? pack1 : packed_git); p;
> > +p = (pack1_seen ? p->next : packed_git), pack1_seen = 1) {
> > +   off_t offset;
> 
> Hmm. So this is basically sticking the found-pack at the front of the
> loop.
> 
> We either need to look at zero packs here (we already know where the
> object is, and we don't need to bother with --local or .keep lookups),
> or we need to look at all of them (to check for local/keep).
> 
> I guess you structured it this way to try to reuse the "can we break out
> early" logic from the middle of the loop. So we go through the loop one
> time, and then break out. And then this:
> 
> > +   if (p == pack1) {
> > +   if (pack1_seen)
> > +   continue;
> > +   offset = *found_offset;
> > +   }
> > +   else {
> > +   offset = find_pack_entry_one(sha1, p);
> > +   }
> 
> is meant to make that one-time through the loop cheaper. So I don't
> think it's wrong, but it's very confusing to me.
> 
> Would it be simpler to stick that logic in a function like:
> 
>   static int want_found_object(int exclude, struct packed_git *pack)
>   {
>   if (exclude)
>   return 1;
>   if (incremental)
>   return 0;
> 
>   /* if we can break early, then do so */
>   if (!ignore_packed_keep &&
>   (!local || !have_non_local_packs))
>   return 1;
> 
>   if (local && !p->pack_local)
>   return 0;
>   if (ignore_packed_keep && p->pack_local && p->pack_keep)
>   return 0;
> 
>   /* indeterminate; keep looking for more packs */
>   return -1;
>   }
> 
>   static int want_object_in_pack(...)
>   {
>   ...
>   if (!exclude && local && has_loose_object_nonlocal(sha1))
>   return 0;
> 
>   if (*found_pack) {
>   int ret = want_found_object(exclude, *found_pack);
>   if (ret != -1)
>   return ret;
>   }
> 
>   for (p = packed_git; p; p = p->next) {
>   off_t offset;
> 
>   if (p == *found_pack)
>   offset = *found_offset;
>   else
>   offset = find_pack_entry(sha1, p);
>   if (offset) {
>   ... fill in *found_pack ...
>   int ret = want_found_object(exclude, p);
>   if (ret != -1)
>   return ret;
>   }
>   }
>   return 1;
>   }
> 
> That's a little more verbose, but IMHO the flow is a lot easier to
> follow (especially as the later re-rolls of that series actually muck
> with the loop order more, but with this approach there's no conflict).

On Mon, Aug 08, 2016 at 09:08:51AM -0700, Junio C Hamano wrote:
> I agree; Kirill's version was so confusing that I couldn't see what
> it was trying to do with "pack1_seen" flag that is reset every time
> loop repeats (at least, before got my coffee ;-).  A helper function
> like the above makes the logic a lot easier to grasp.

Ok, at least I put today's record for the most confusing code. I agree
with your comments - it is better to simplify control-flow logic. Somehow
my head was refusing doing that and insisted on keeping the loop inside
intact. Maybe I should have a bit of rest... Scratch all that in favour
of want_found_object() and thanks for heads-up.



> >  static int add_object_entry(const unsigned char *sha1, enum object_type 
> > type,
> >   

Re: [PATCH v4 2/2] pack-objects: Teach it to use reachability bitmap index when generating non-stdout pack too

[Kirill Smelkov]

On Mon, Aug 08, 2016 at 11:08:34AM -0700, Junio C Hamano wrote:
> Kirill Smelkov <k...@nexedi.com> writes:
> 
> > Thanks for the info. I did not knew about show-index when I was starting
> > to work on this and later it just came out of sight. Please find
> > corrected patch below.
> >
> >  8< 
> > From: Kirill Smelkov <k...@nexedi.com>
> > Date: Fri, 29 Jul 2016 10:47:46 +0300
> > Subject: [PATCH v5] pack-objects: Teach it to use reachability bitmap index 
> > when
> >  generating non-stdout pack too
> 
> Please don't do this (not the patch text itself, but saying "Please
> find ..." and attaching the patch AFTER 60+ lines of response).
> When going through old/read messages to see if there are patches
> that fell through the cracks, if it is not immediately clear in the
> top part of the message that it contains an updated patch, such a
> patch will certainly be missed.
> 
> Please say "I'll follow up with a corrected patch" instead of
> "Please find ..." and respond to that message with just the patch.

Ok, I see. Should I resend this v5 as separated one or only starting
from next time?

Another question: I'm preparing another version of "pack-objects: Teach
--use-bitmap-index codepath to  respect --local ..." and was going to
put

( updated patch is in the end of this mail )

in the top of the message. Is it ok or better not to do so and just respin
the patch in its own separate mail?

Thanks beforehand for clarifying,
Kirill

P.S. I put updated patches in the same mail not because I'm trying to
make maintainer's life harder, but because this is the way I would
expect and prefer them to be coming to me...
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4 2/2] pack-objects: Teach it to use reachability bitmap index when generating non-stdout pack too

[Kirill Smelkov]

On Mon, Aug 08, 2016 at 09:56:00AM -0400, Jeff King wrote:
> On Fri, Jul 29, 2016 at 10:47:46AM +0300, Kirill Smelkov wrote:
> 
> > @@ -2527,7 +2528,7 @@ static int get_object_list_from_bitmap(struct 
> > rev_info *revs)
> > if (prepare_bitmap_walk(revs) < 0)
> > return -1;
> >  
> > -   if (pack_options_allow_reuse() &&
> > +   if (pack_options_allow_reuse() && pack_to_stdout &&
> > !reuse_partial_packfile_from_bitmap(
> 
> Should pack_to_stdout just be part of pack_options_allow_reuse()?

Yes, makes sense; thanks for catching this.


> > @@ -2812,7 +2813,23 @@ int cmd_pack_objects(int argc, const char **argv, 
> > const char *prefix)
> > if (!rev_list_all || !rev_list_reflog || !rev_list_index)
> > unpack_unreachable_expiration = 0;
> >  
> > -   if (!use_internal_rev_list || !pack_to_stdout || 
> > is_repository_shallow())
> > +   /*
> > +* "soft" reasons not to use bitmaps - for on-disk repack by default we 
> > want
> > +*
> > +* - to produce good pack (with bitmap index not-yet-packed objects are
> > +*   packed in suboptimal order).
> > +*
> > +* - to use more robust pack-generation codepath (avoiding possible
> > +*   bugs in bitmap code and possible bitmap index corruption).
> > +*/
> > +   if (!pack_to_stdout)
> > +   use_bitmap_index_default = 0;
> > +
> > +   if (use_bitmap_index < 0)
> > +   use_bitmap_index = use_bitmap_index_default;
> > +
> > +   /* "hard" reasons not to use bitmaps; these just won't work at all */
> > +   if (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) 
> > || is_repository_shallow())
> > use_bitmap_index = 0;
> 
> This all makes sense and looks good.

Thanks.


> > +test_expect_success 'pack-objects to file can use bitmap' '
> > +   # make sure we still have 1 bitmap index from previous tests
> > +   ls .git/objects/pack/ | grep bitmap >output &&
> > +   test_line_count = 1 output &&
> > +   # verify equivalent packs are generated with/without using bitmap index
> > +   packasha1=$(git pack-objects --no-use-bitmap-index --all packa 
> >  > +   packbsha1=$(git pack-objects --use-bitmap-index --all packb  > &&
> > +   git verify-pack -v packa-$packasha1.pack >packa.verify &&
> > +   git verify-pack -v packb-$packbsha1.pack >packb.verify &&
> > +   grep -o "^$_x40" packa.verify |sort >packa.objects &&
> > +   grep -o "^$_x40" packb.verify |sort >packb.objects &&
> > +   test_cmp packa.objects packb.objects
> > +'
> 
> I don't think "grep -o" is portable. However, an easier way to do this
> is probably:
> 
>   # these are already in sorted order
>   git show-index <packa-$packasha1.pack | cut -d' ' -f2 >packa.objects &&
>   git show-index <packb-$packbsha1.pack | cut -d' ' -f2 >packb.objects &&
>   test_cmp packa.objects packb.objects

Thanks for the info. I did not knew about show-index when I was starting
to work on this and later it just came out of sight. Please find
corrected patch below.

 8< 
From: Kirill Smelkov <k...@nexedi.com>
Date: Fri, 29 Jul 2016 10:47:46 +0300
Subject: [PATCH v5] pack-objects: Teach it to use reachability bitmap index when
 generating non-stdout pack too

Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects)
if a repository has bitmap index, pack-objects can nicely speedup
"Counting objects" graph traversal phase. That however was done only for
case when resultant pack is sent to stdout, not written into a file.

The reason here is for on-disk repack by default we want:

- to produce good pack (with bitmap index not-yet-packed objects are
  emitted to pack in suboptimal order).

- to use more robust pack-generation codepath (avoiding possible
  bugs in bitmap code and possible bitmap index corruption).

Jeff Kind further explains:

The reason for this split is that pack-objects tries to determine how
"careful" it should be based on whether we are packing to disk or to
stdout. Packing to disk implies "git repack", and that we will likely
delete the old packs after finishing. We want to be more careful (so
as not to carry forward a corruption, and to generate a more optimal
pack), and we presumably run less frequently and can afford extra CPU.
Whereas packing to stdout implies serving a remote via "git fetch" or
"git push". This happens more frequently (e.g., a server handling many
fetching

Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental

[Kirill Smelkov]

On Mon, Aug 01, 2016 at 11:17:30AM -0700, Junio C Hamano wrote:
> Kirill Smelkov <k...@nexedi.com> writes:
> 
> > Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there
> > are two codepaths in pack-objects: with & without using bitmap
> > reachability index.
> >
> > However add_object_entry_from_bitmap(), despite its non-bitmapped
> > counterpart add_object_entry(), in no way does check for whether --local
> > or --honor-pack-keep or --incremental should be respected. In
> > non-bitmapped codepath this is handled in want_object_in_pack(), but
> > bitmapped codepath has simply no such checking at all.
> >
> > The bitmapped codepath however was allowing to pass in all those options
> > and with bitmap indices still being used under such conditions -
> > potentially giving wrong output (e.g. including objects from non-local or
> > .keep'ed pack).
> >
> > We can easily fix this by noting the following: when an object comes to
> > add_object_entry_from_bitmap() it can come for two reasons:
> >
> > 1. entries coming from main pack covered by bitmap index, and
> > 2. object coming from, possibly alternate, loose or other packs.
> >
> > For "2" we always have pack not yet found by bitmap traversal code, and
> > thus we can simply reuse non-bitmapped want_object_in_pack() to find in
> > which pack an object lives and also for taking omitting decision.
> >
> > For "1" we always have pack already found by bitmap traversal code and we
> > only need to check that pack for same criteria used in
> > want_object_in_pack() for found_pack.
> >
> > Suggested-by: Junio C Hamano <gits...@pobox.com>
> > Discussed-with: Jeff King <p...@peff.net>
> > ---
> 
> I do not think I suggested much of this to deserve credit like this,
> though, as I certainly haven't thought about the pros-and-cons
> between adding the same "some object in pack may not want to be in
> the output" logic to the bitmap side, or punting the bitmap codepath
> when local/keep are involved.

I understand. Still for me it was you who convinced me to add proper
support for e.g. --local vs bitmap instead of special-casing it.
I think we also can avoid punting the bitmap codepath - please see
below.


> > +/* Like want_object_in_pack() but for objects coming from-under bitmapped 
> > traversal */
> > +static int want_object_in_pack_bitmap(const unsigned char *sha1,
> > + struct packed_git **found_pack,
> > + off_t *found_offset)
> > +{
> > +   struct packed_git *p = *found_pack;
> > +
> > +   /*
> > +* There are two types of requests coming here:
> > +* 1. entries coming from main pack covered by bitmap index, and
> > +* 2. object coming from, possibly alternate, loose or other packs.
> > +*
> > +* For "1" we always have *found_pack != NULL passed here from
> > +* traverse_bitmap_commit_list(). (*found_pack is bitmap_git.pack
> > +* actually).
> > +*
> > +* For "2" we always have *found_pack == NULL passed here from
> > +* traverse_bitmap_commit_list() - since this is the way bitmap
> > +* traversal passes here "extended" bitmap entries.
> > +*/
> > +
> > +   /* objects not covered by bitmap */
> > +   if (!p)
> > +   return want_object_in_pack(sha1, 0, found_pack, found_offset);
> > +   /* objects covered by bitmap - we only have to check p wrt local and 
> > .keep */
> 
> I am assuming that p != NULL only means "this object exists in THIS
> pack", without saying anything about "this object may also exist in
> other places", but "we only have to check" implies that "p != NULL"
> means "this object exists *ONLY* in this pack and nowhere else".
> 
> Puzzled.

You are right. Being new to --local and .keep I've missed this. I've
added tests to cover cases like "object lives in both bitmapped pack and
non-local loose or .keep'ed pack" and made the adjustments. The checks
are now live unified in want_object_in_pack() for both bitmapped and
non-bitmapped codepaths. Please apply the following corrected patch on
top of 56dfeb62 (jk/pack-objects-optim).

Thanks,
Kirill

 8< 
From: Kirill Smelkov <k...@nexedi.com>
Date: Fri, 29 Jul 2016 10:46:56 +0300
Subject: [PATCH v2] pack-objects: Teach --use-bitmap-index codepath to respect
 --local, --honor-pack-keep and --incremental

Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there
are two codepaths in pack-objects:

[PATCH v4 2/2] pack-objects: Teach it to use reachability bitmap index when generating non-stdout pack too

[Kirill Smelkov]

x-pack file.pack`  is  72s,

while

`pack-objects file.pack` which does both pack and index is  27s.

And even

`pack-objects --no-use-bitmap-index file.pack`  is  37s.

Jeff explains:

The packfile does not carry the sha1 of the objects. A receiving
index-pack has to compute them itself, including inflating and applying
all of the deltas.

that's why for `git-backup restore` we want to teach `git pack-objects
file.pack` to use bitmaps instead of using `git pack-objects --stdout
>file.pack` + `git index-pack file.pack`.

More context:

http://article.gmane.org/gmane.comp.version-control.git/299063
http://article.gmane.org/gmane.comp.version-control.git/299107
http://article.gmane.org/gmane.comp.version-control.git/299420
http://article.gmane.org/gmane.comp.version-control.git/300217

Cc: Vicent Marti <tan...@gmail.com>
Helped-by: Jeff King <p...@peff.net>
Signed-off-by: Kirill Smelkov <k...@nexedi.com>
---
 Documentation/config.txt |  3 +++
 builtin/pack-objects.c   | 25 +
 t/t5310-pack-bitmaps.sh  | 14 ++
 3 files changed, 38 insertions(+), 4 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 8b1aee4..6a903c0 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -2244,6 +2244,9 @@ pack.useBitmaps::
to stdout (e.g., during the server side of a fetch). Defaults to
true. You should not generally need to turn this off unless
you are debugging pack bitmaps.
++
+*NOTE*: when packing to file (e.g., on repack) the default is always not to use
+   pack bitmaps.
 
 pack.writeBitmaps (deprecated)::
This is a deprecated synonym for `repack.writeBitmaps`.
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 34b3019..2b2e74a 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -66,7 +66,8 @@ static struct packed_git *reuse_packfile;
 static uint32_t reuse_packfile_objects;
 static off_t reuse_packfile_offset;
 
-static int use_bitmap_index = 1;
+static int use_bitmap_index_default = 1;
+static int use_bitmap_index = -1;
 static int write_bitmap_index;
 static uint16_t write_bitmap_options;
 
@@ -2264,7 +2265,7 @@ static int git_pack_config(const char *k, const char *v, 
void *cb)
write_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;
}
if (!strcmp(k, "pack.usebitmaps")) {
-   use_bitmap_index = git_config_bool(k, v);
+   use_bitmap_index_default = git_config_bool(k, v);
return 0;
}
if (!strcmp(k, "pack.threads")) {
@@ -2527,7 +2528,7 @@ static int get_object_list_from_bitmap(struct rev_info 
*revs)
if (prepare_bitmap_walk(revs) < 0)
return -1;
 
-   if (pack_options_allow_reuse() &&
+   if (pack_options_allow_reuse() && pack_to_stdout &&
!reuse_partial_packfile_from_bitmap(
_packfile,
_packfile_objects,
@@ -2812,7 +2813,23 @@ int cmd_pack_objects(int argc, const char **argv, const 
char *prefix)
if (!rev_list_all || !rev_list_reflog || !rev_list_index)
unpack_unreachable_expiration = 0;
 
-   if (!use_internal_rev_list || !pack_to_stdout || 
is_repository_shallow())
+   /*
+* "soft" reasons not to use bitmaps - for on-disk repack by default we 
want
+*
+* - to produce good pack (with bitmap index not-yet-packed objects are
+*   packed in suboptimal order).
+*
+* - to use more robust pack-generation codepath (avoiding possible
+*   bugs in bitmap code and possible bitmap index corruption).
+*/
+   if (!pack_to_stdout)
+   use_bitmap_index_default = 0;
+
+   if (use_bitmap_index < 0)
+   use_bitmap_index = use_bitmap_index_default;
+
+   /* "hard" reasons not to use bitmaps; these just won't work at all */
+   if (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) 
|| is_repository_shallow())
use_bitmap_index = 0;
 
if (pack_to_stdout || !rev_list_all)
diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh
index a76f6ca..58c3b29 100755
--- a/t/t5310-pack-bitmaps.sh
+++ b/t/t5310-pack-bitmaps.sh
@@ -200,6 +200,20 @@ test_expect_success 'pack-objects respects --local 
(non-local bitmapped pack)' '
mv alt_objects/pack/$packbitmap.* .git/objects/pack/
 '
 
+test_expect_success 'pack-objects to file can use bitmap' '
+   # make sure we still have 1 bitmap index from previous tests
+   ls .git/objects/pack/ | grep bitmap >output &&
+   test_line_count = 1 output &&
+   # verify equivalent packs are generated with/without using bitmap index
+   packasha1=$(git pack-objects --no-use-bitmap-index --all packa 
packa.verify &&

[PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental

[Kirill Smelkov]

Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there
are two codepaths in pack-objects: with & without using bitmap
reachability index.

However add_object_entry_from_bitmap(), despite its non-bitmapped
counterpart add_object_entry(), in no way does check for whether --local
or --honor-pack-keep or --incremental should be respected. In
non-bitmapped codepath this is handled in want_object_in_pack(), but
bitmapped codepath has simply no such checking at all.

The bitmapped codepath however was allowing to pass in all those options
and with bitmap indices still being used under such conditions -
potentially giving wrong output (e.g. including objects from non-local or
.keep'ed pack).

We can easily fix this by noting the following: when an object comes to
add_object_entry_from_bitmap() it can come for two reasons:

1. entries coming from main pack covered by bitmap index, and
2. object coming from, possibly alternate, loose or other packs.

For "2" we always have pack not yet found by bitmap traversal code, and
thus we can simply reuse non-bitmapped want_object_in_pack() to find in
which pack an object lives and also for taking omitting decision.

For "1" we always have pack already found by bitmap traversal code and we
only need to check that pack for same criteria used in
want_object_in_pack() for found_pack.

Suggested-by: Junio C Hamano 
Discussed-with: Jeff King 
---
 builtin/pack-objects.c  |  39 +++
 t/t5310-pack-bitmaps.sh | 100 
 2 files changed, 139 insertions(+)

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index a2f8cfd..34b3019 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -987,6 +987,42 @@ static int want_object_in_pack(const unsigned char *sha1,
return 1;
 }
 
+/* Like want_object_in_pack() but for objects coming from-under bitmapped 
traversal */
+static int want_object_in_pack_bitmap(const unsigned char *sha1,
+ struct packed_git **found_pack,
+ off_t *found_offset)
+{
+   struct packed_git *p = *found_pack;
+
+   /*
+* There are two types of requests coming here:
+* 1. entries coming from main pack covered by bitmap index, and
+* 2. object coming from, possibly alternate, loose or other packs.
+*
+* For "1" we always have *found_pack != NULL passed here from
+* traverse_bitmap_commit_list(). (*found_pack is bitmap_git.pack
+* actually).
+*
+* For "2" we always have *found_pack == NULL passed here from
+* traverse_bitmap_commit_list() - since this is the way bitmap
+* traversal passes here "extended" bitmap entries.
+*/
+
+   /* objects not covered by bitmap */
+   if (!p)
+   return want_object_in_pack(sha1, 0, found_pack, found_offset);
+
+   /* objects covered by bitmap - we only have to check p wrt local and 
.keep */
+   if (incremental)
+   return 0;
+   if (local && !p->pack_local)
+   return 0;
+   if (ignore_packed_keep && p->pack_local && p->pack_keep)
+   return 0;
+
+   return 1;
+}
+
 static void create_object_entry(const unsigned char *sha1,
enum object_type type,
uint32_t hash,
@@ -1055,6 +1091,9 @@ static int add_object_entry_from_bitmap(const unsigned 
char *sha1,
if (have_duplicate_entry(sha1, 0, _pos))
return 0;
 
+   if (!want_object_in_pack_bitmap(sha1, , ))
+   return 0;
+
create_object_entry(sha1, type, name_hash, 0, 0, index_pos, pack, 
offset);
 
display_progress(progress_state, nr_result);
diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh
index 3893afd..a76f6ca 100755
--- a/t/t5310-pack-bitmaps.sh
+++ b/t/t5310-pack-bitmaps.sh
@@ -118,6 +118,88 @@ test_expect_success 'incremental repack can disable 
bitmaps' '
git repack -d --no-write-bitmap-index
 '
 
+test_expect_success 'pack-objects respects --local (non-local loose)' '
+   mkdir -p alt_objects/pack &&
+   echo $(pwd)/alt_objects > .git/objects/info/alternates &&
+   echo content1 > file1 &&
+   objsha1=$(GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w file1) &&
+   git add file1 &&
+   test_tick &&
+   git commit -m commit_file1 &&
+   echo HEAD | \
+   git pack-objects --local --stdout --revs >1.pack &&
+   git index-pack 1.pack &&
+   git verify-pack -v 1.pack >1.objects &&
+   if egrep "^$objsha1" 1.objects; then
+   echo "Non-local object present in pack generated with --local: 
$objsha1"
+   return 1
+   fi
+'
+
+test_expect_success 'pack-objects respects --honor-pack-keep (local 
non-bitmapped pack)' '
+   echo content2 > file2 &&
+   objsha2=$(git hash-object -w 

Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

[Kirill Smelkov]

On Thu, Jul 28, 2016 at 02:18:29PM -0700, Junio C Hamano wrote:
> Kirill Smelkov <k...@nexedi.com> writes:
> 
> > I'm waiting so long for main patch to be at least queued to pu, that I'm
> > now a bit frustrated and ready to do something not related to main goal :)
> 
> Perhaps the first step would be to stop putting multiple patches in
> a single e-mail buried after a few pages of discussion.  I will not
> even find that there _are_ multiple patches in the message if I am
> not involved directly in the discussion, and the discussion is still
> ongoing, because it is likely that I'd skim just a few paragraphs at
> the top before going on to other messages.
> 
> I won't touch the message I am responding to, as your -- 8< -- cut
> mark does not even seem to be a reliable marker between patches
> (i.e.  I see something like this that is clearly not a message
> boundary:
> 
> than `git pack-objects file.pack`. Extracting erp5.git pack from
> lab.nexedi.com backup repository:
> 
>  8< 
> $ time echo 0186ac99 | git pack-objects --stdout --revs >erp5pack-stdout.pack
> 
> real0m22.309s
> ...
> )

Ok, makes sense and my fault. I'm resending each patch as separate
message in reply to this mail.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

[Kirill Smelkov]

Junio, first of all thanks for feedback,

On Wed, Jul 27, 2016 at 01:40:36PM -0700, Junio C Hamano wrote:
> Kirill Smelkov <k...@nexedi.com> writes:
> 
> > > From: Kirill Smelkov <k...@nexedi.com>
> > Subject: [PATCH 1/2] pack-objects: Make sure use_bitmap_index is not active 
> > under
> >  --local or --honor-pack-keep
> >
> > Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there
> > are two codepaths in pack-objects: with & without using bitmap
> > reachability index.
> >
> > However add_object_entry_from_bitmap(), despite its non-bitmapped
> > counterpart add_object_entry(), in no way does check for whether --local
> > or --honor-pack-keep should be respected. In non-bitmapped codepath this
> > is handled in want_object_in_pack(), but bitmapped codepath has simply
> > no such checking at all.
> >
> > The bitmapped codepath however was allowing to pass --local and
> > --honor-pack-keep and bitmap indices were still used under such
> > conditions - potentially giving wrong output (including objects from
> > non-local or .keep'ed pack).
> >
> > Instead of fixing bitmapped codepath to respect those options, since
> > currently no one actually need or use them in combination with bitmaps,
> > let's just force use_bitmap_index=0 when any of --local or
> > --honor-pack-keep are used and add appropriate comment about
> > not-checking for those in add_object_entry_from_bitmap()
> >
> > Suggested-by: Jeff King <p...@peff.net>
> > ---
> >  builtin/pack-objects.c | 15 +++
> >  1 file changed, 15 insertions(+)
> >
> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> > index 15866d7..d7cf782 100644
> > --- a/builtin/pack-objects.c
> > +++ b/builtin/pack-objects.c
> > @@ -1055,6 +1055,12 @@ static int add_object_entry_from_bitmap(const 
> > unsigned char *sha1,
> > if (have_duplicate_entry(sha1, 0, _pos))
> > return 0;
> >  
> > +   /*
> > +* for simplicity we always want object to be in pack, as
> > +* use_bitmap_index codepath assumes neither --local nor 
> > --honor-pack-keep
> > +* is active.
> > +*/
> 
> I am not sure this comment is useful to readers.
> 
> Unless the readers are comparing add_object_entry() and this
> function and wondering why this side lacks a check here, iow, when
> they are merely following from a caller of this function through
> this function down to its callee to understand what goes on, this
> comment would not help them and only confuse them.
> 
> If we were to say something to help those who are comparing these
> two functions, I think we should be more explicit, i.e.
> 
> The caller disables use-bitmap-index when --local or
> --honor-pack-keep options are in effect because bitmap code is
> not prepared to handle them.  Because the control does not reach
> here if these options are in effect, the check with
> want_object_in_pack() to skip objects is not done.
> 
> or something like that.

You are probably right.


> Or is the rest of the bitmap codepath prepared to handle these
> options and it is just the matter of adding the missing check with
> want_object_in_pack() here to make it work correctly?

I'm waiting so long for main patch to be at least queued to pu, that I'm
now a bit frustrated and ready to do something not related to main goal :)
(they say every joke contains part of a joke). Here is something from
sleepy me:

 8< 
From: Kirill Smelkov <k...@nexedi.com>
Date: Wed, 27 Jul 2016 22:18:04 +0300
Subject: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to
 respect --local, --honor-pack-keep and --incremental

Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there
are two codepaths in pack-objects: with & without using bitmap
reachability index.

However add_object_entry_from_bitmap(), despite its non-bitmapped
counterpart add_object_entry(), in no way does check for whether --local
or --honor-pack-keep or --incremental should be respected. In
non-bitmapped codepath this is handled in want_object_in_pack(), but
bitmapped codepath has simply no such checking at all.

The bitmapped codepath however was allowing to pass in all those options
and with bitmap indices still being used under such conditions -
potentially giving wrong output (e.g. including objects from non-local or
.keep'ed pack).

We can easily fix this by noting the following: when an object comes to
add_object_entry_from_bitmap() it can come for two reasons:

1. entries coming from main pack covered by bitmap index, and
2. object coming from, possibly alternate, loose or other packs.

For "2

Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

[Kirill Smelkov]

On Mon, Jul 25, 2016 at 02:40:25PM -0400, Jeff King wrote:
> On Wed, Jul 13, 2016 at 01:52:17PM +0300, Kirill Smelkov wrote:
> 
> > > So I think if you were to repeatedly "git repack -adb" over time, you
> > > would get worse and worse ordering as objects are added to the
> > > repository.
> > 
> > Jeff, first of all thanks for clarifying.
> > 
> > So it is not-yet-packed-objects which make packing with bitmap less
> > efficient. I was originally keeping in mind fresh repacked repository
> > with just built bitmap index and for that case extracting pack with
> > bitmap index seems to be just ok, but the more not-yet-packed objects we
> > have the worse the result can be.
> 
> Right. So I think your scheme is fine as long as you are doing your
> regular "pack all into one" repacks with a real walk, and then
> "branching" off of that with one-off bitmap-computed packs into files
> (even if you later take a bunch of those files and pull them into a
> single bitmapped, as long as that final "all into one" does the walk).
> 
> Or I guess another way to think about it would be that if you're
> computing bitmaps, you'd want to do the actual traversal.

Yes, exactly, and thanks for stating it clearly. We are doing repacks
and recomputing bitmaps doing the real walk. As you say this should be
fine.


> > Yes, it also make sense. I saw write_reused_pack() in upstream git just
> > copy raw bytes from original to destination pack. You mentioned you have
> > something better for pack reuse - in your patch queue, in two words, is
> > it now reusing pack based on object, not raw bytes, or is it something
> > else?
> > 
> > In other words in which way it works better? (I'm just curious here as
> > it is interesting to know)
> 
> The problem with the existing pack-reuse code is that it doesn't kick in
> often enough. I think it looks to see that the client wants some
> percentage of the pack (e.g., 90%), and then just sends the whole
> beginning. This works especially badly if you have a bunch of related
> repositories packed together (e.g., all of the forks of torvalds/linux
> on GitHub), because you'll never hit 90% of that big pack; it has too
> much unrelated cruft, even if most of the stuff you want _is_ at the
> beginning. And "percent of pack" is not really a useful metric anyway.
> 
> So the better scheme is more like:
> 
>   1. Generate the bitmap of objects to send using reachability bitmaps.
> 
>   2. Do a quick scan of their content in the packfile to see which can
>  be reused verbatim. If they're base objects, we can send them
>  as-is. If they're deltas, we can send them if their base is going
>  to be sent. This fills in another bitmap of "reusable" objects.
> 
>  After a long string of unusable objects, you can give up and set
>  the rest of the bitmap to zeroes.
> 
>   3. Walk the "reuse" bitmap and send out the objects more-or-less
>  verbatim. You do have make adjustments to delta-base-offsets for
>  any "holes" (so if an object's entry says "my base is 500 bytes
>  back", but you omitted some objects in between, you have to adjust
>  that offset).
> 
> The upside is that you can send out those objects without even making a
> "struct object_entry" for them, which drastically reduces the memory
> requirements for serving a clone. Any objects which didn't get marked
> for reuse just get handled in the usual way (so stuff that was not close
> by in the pack, or stuff that was pushed since your last big repack).

Thanks for clarifying. Yes, you are right, current upstream code checks
to see whether >= 90% of pack is what destination wants and only reuse
in such case. (I forgot about it, initially putting reuse at side in my
head as "not applicable to git-backup" because of that >= 90% reason).

So with the scheme you are drawing above it can be indeed more
efficient, and applicable to both torvalds/linux+forks and git-backup
case (extracting packs from big pack of all repos).

I'm looking forward to your patches on this topic. Please cc me on those
if you find it convenient.

> The downside is that because those objects aren't in our normal packing
> list, they're not available as delta bases for the new objects we _do_
> send. So it can make the resulting pack a little bit bigger.

So once again, the badness effect is the more, the more we have such
"new" objects not in original main pack - i.e. as loose objects or
objects living in other smaller packs. The badness comes to zero in
ideal case of freshly repacked repo with only one big pack.

Also: after sending reused object, with more

Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

[Kirill Smelkov]

On Tue, Jul 19, 2016 at 05:29:07AM -0600, Jeff King wrote:
> On Sun, Jul 17, 2016 at 08:06:49PM +0300, Kirill Smelkov wrote:
> 
> > > Anyway, please find below updated patch according to your suggestion.
> > > Hope it is ok now.
> > 
> > Ping. Is the patch ok or something needs to be improved still?
> 
> Sorry, I'm traveling and haven't carefully reviewed it yet. It's still
> on my list, but it may be a few days.

Jeff thanks for feedback. Have a good traveling and good to know the patch
was not forgotten. I will be waiting for the time while you are on trip.

Thanks again for feedback,
Kirill
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

[Kirill Smelkov]

On Wed, Jul 13, 2016 at 01:52:16PM +0300, Kirill Smelkov wrote:
> On Wed, Jul 13, 2016 at 04:26:53AM -0400, Jeff King wrote:
> > On Fri, Jul 08, 2016 at 01:38:55PM +0300, Kirill Smelkov wrote:
> > 
> > > >   - we will not compute the same write order (which is based on
> > > > traversal order), leading to packs that have less efficient cache
> > > > characteristics
> > > 
> > > I agree the order can be not exactly the same. Still if original pack is
> > > packed well (with good recency order), while using bitmap we will tend
> > > to traverse it in close to original order.
> > > 
> > > Maybe I'm not completely right on this, but to me it looks to be the
> > > case because if objects in original pack are put there linearly sorted
> > > by recency order, and we use bitmap index to set of all reachable
> > > objects from a root, and then just _linearly_ gather all those objects
> > > from original pack by 1s in bitmap and put them in the same order into
> > > destination pack, the recency order won't be broken.
> > > 
> > > Or am I maybe misunderstanding something?
> > 
> > Yeah, I think you can go some of the way by reusing the order from the
> > old pack. But keep in mind that the bitmap result may also contain
> > objects that are not yet packed. Those will just come in a big lump at
> > the end of the bitmap (these are the "extended entries" in the bitmap
> > code).
> > 
> > So I think if you were to repeatedly "git repack -adb" over time, you
> > would get worse and worse ordering as objects are added to the
> > repository.
> 
> Jeff, first of all thanks for clarifying.
> 
> So it is not-yet-packed-objects which make packing with bitmap less
> efficient. I was originally keeping in mind fresh repacked repository
> with just built bitmap index and for that case extracting pack with
> bitmap index seems to be just ok, but the more not-yet-packed objects we
> have the worse the result can be.
> 
> > As an aside, two other things that pack order matters for: it makes the
> > bitmaps themselves compress better (because it increases locality of
> > reachability, so you get nice runs of "1" or "0" bits).
> 
> Yes I agree and thanks for bringing this up - putting objects in recency
> order in pack also makes bitmap index to have larger runs of same 1 or 0.
> 
> > It also makes
> > the pack-reuse code more efficient (since in an ideal case, you can just
> > dump a big block of data from the front of the pack). Note that the
> > pack-reuse code that's in upstream git isn't that great; I have a better
> > system on my big pile of patches to send upstream (that never seems to
> > get smaller; ).
> 
> Yes, it also make sense. I saw write_reused_pack() in upstream git just
> copy raw bytes from original to destination pack. You mentioned you have
> something better for pack reuse - in your patch queue, in two words, is
> it now reusing pack based on object, not raw bytes, or is it something
> else?
> 
> In other words in which way it works better? (I'm just curious here as
> it is interesting to know)
> 
> 
> > > >   - we don't learn about the filename of trees and blobs, which is going
> > > > to make the delta step much less efficient. This might be mitigated
> > > > by turning on the bitmap name-hash cache; I don't recall how much
> > > > detail pack-objects needs on the name (i.e., the full name versus
> > > > just the hash).
> > > 
> > > If I understand it right, it uses only uint32_t name hash while 
> > > searching. From
> > > pack-objects.{h,c} :
> > 
> > Yeah, I think you are right. Not having the real names is a problem for
> > doing rev-list output, but I think pack-objects doesn't care (though do
> > note that the name-hash cache is not enabled by default).
> 
> Yes, for packing it is only hash which is used. And I assume name-hash
> for bitmap is not enabled by default for compatibility with JGit code.
> 
> It would make sense to me to eventually enable name-hash bitmap
> extension by default, as packing result is much better with it. And
> those who care about compatibility with JGit can just turn it off in
> their git config.
> 
> Just my thoughts.
> 
> > > > There may be other subtle things, too. The general idea of tying the
> > > > bitmap use to pack_to_stdout is that you _do_ want to use it for
> > > > serving fetches and pushes, but for a full on-disk repack via gc, it's
> > > > more

Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

[Kirill Smelkov]

On Wed, Jul 13, 2016 at 04:26:53AM -0400, Jeff King wrote:
> On Fri, Jul 08, 2016 at 01:38:55PM +0300, Kirill Smelkov wrote:
> 
> > >   - we will not compute the same write order (which is based on
> > > traversal order), leading to packs that have less efficient cache
> > > characteristics
> > 
> > I agree the order can be not exactly the same. Still if original pack is
> > packed well (with good recency order), while using bitmap we will tend
> > to traverse it in close to original order.
> > 
> > Maybe I'm not completely right on this, but to me it looks to be the
> > case because if objects in original pack are put there linearly sorted
> > by recency order, and we use bitmap index to set of all reachable
> > objects from a root, and then just _linearly_ gather all those objects
> > from original pack by 1s in bitmap and put them in the same order into
> > destination pack, the recency order won't be broken.
> > 
> > Or am I maybe misunderstanding something?
> 
> Yeah, I think you can go some of the way by reusing the order from the
> old pack. But keep in mind that the bitmap result may also contain
> objects that are not yet packed. Those will just come in a big lump at
> the end of the bitmap (these are the "extended entries" in the bitmap
> code).
> 
> So I think if you were to repeatedly "git repack -adb" over time, you
> would get worse and worse ordering as objects are added to the
> repository.

Jeff, first of all thanks for clarifying.

So it is not-yet-packed-objects which make packing with bitmap less
efficient. I was originally keeping in mind fresh repacked repository
with just built bitmap index and for that case extracting pack with
bitmap index seems to be just ok, but the more not-yet-packed objects we
have the worse the result can be.

> As an aside, two other things that pack order matters for: it makes the
> bitmaps themselves compress better (because it increases locality of
> reachability, so you get nice runs of "1" or "0" bits).

Yes I agree and thanks for bringing this up - putting objects in recency
order in pack also makes bitmap index to have larger runs of same 1 or 0.

> It also makes
> the pack-reuse code more efficient (since in an ideal case, you can just
> dump a big block of data from the front of the pack). Note that the
> pack-reuse code that's in upstream git isn't that great; I have a better
> system on my big pile of patches to send upstream (that never seems to
> get smaller; ).

Yes, it also make sense. I saw write_reused_pack() in upstream git just
copy raw bytes from original to destination pack. You mentioned you have
something better for pack reuse - in your patch queue, in two words, is
it now reusing pack based on object, not raw bytes, or is it something
else?

In other words in which way it works better? (I'm just curious here as
it is interesting to know)


> > >   - we don't learn about the filename of trees and blobs, which is going
> > > to make the delta step much less efficient. This might be mitigated
> > > by turning on the bitmap name-hash cache; I don't recall how much
> > > detail pack-objects needs on the name (i.e., the full name versus
> > > just the hash).
> > 
> > If I understand it right, it uses only uint32_t name hash while searching. 
> > From
> > pack-objects.{h,c} :
> 
> Yeah, I think you are right. Not having the real names is a problem for
> doing rev-list output, but I think pack-objects doesn't care (though do
> note that the name-hash cache is not enabled by default).

Yes, for packing it is only hash which is used. And I assume name-hash
for bitmap is not enabled by default for compatibility with JGit code.

It would make sense to me to eventually enable name-hash bitmap
extension by default, as packing result is much better with it. And
those who care about compatibility with JGit can just turn it off in
their git config.

Just my thoughts.

> > > There may be other subtle things, too. The general idea of tying the
> > > bitmap use to pack_to_stdout is that you _do_ want to use it for
> > > serving fetches and pushes, but for a full on-disk repack via gc, it's
> > > more important to generate a good pack.
> > 
> > It is better we send good packs to clients too, right? And with
> > pack.writeBitmapHashCache=true and retaining recency order (please see
> > above, but again maybe I'm not completely right) to me we should be still
> > generating a good pack while using bitmap reachability index for object
> > graph traversal.
> 
> We do want to send the client a good pack, but it's always a tradeoff.
> We could spend much more time searching

Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

[Kirill Smelkov]

On Fri, Jul 08, 2016 at 01:38:55PM +0300, Kirill Smelkov wrote:
> Peff first of all thanks for feedback,
> 
> On Thu, Jul 07, 2016 at 04:52:23PM -0400, Jeff King wrote:
> > On Thu, Jul 07, 2016 at 10:09:17PM +0300, Kirill Smelkov wrote:
> > 
> > > Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects)
> > > if a repository has bitmap index, pack-objects can nicely speedup
> > > "Counting objects" graph traversal phase. That however was done only for
> > > case when resultant pack is sent to stdout, not written into a file.
> > > 
> > > We can teach pack-objects to use bitmap index for initial object
> > > counting phase when generating resultant pack file too:
> > 
> > I'm not sure this is a good idea in general. When bitmaps are in use, we
> > cannot fill out the details in the object-packing list as thoroughly. In
> > particular:
> > 
> >   - we will not compute the same write order (which is based on
> > traversal order), leading to packs that have less efficient cache
> > characteristics
> 
> I agree the order can be not exactly the same. Still if original pack is
> packed well (with good recency order), while using bitmap we will tend
> to traverse it in close to original order.
> 
> Maybe I'm not completely right on this, but to me it looks to be the
> case because if objects in original pack are put there linearly sorted
> by recency order, and we use bitmap index to set of all reachable
> objects from a root, and then just _linearly_ gather all those objects
> from original pack by 1s in bitmap and put them in the same order into
> destination pack, the recency order won't be broken.
> 
> Or am I maybe misunderstanding something?
> 
> Please also see below:
> 
> >   - we don't learn about the filename of trees and blobs, which is going
> > to make the delta step much less efficient. This might be mitigated
> > by turning on the bitmap name-hash cache; I don't recall how much
> > detail pack-objects needs on the name (i.e., the full name versus
> > just the hash).
> 
> If I understand it right, it uses only uint32_t name hash while searching. 
> From
> pack-objects.{h,c} :
> 
>  8< 
> struct object_entry {
>   ...
>   uint32_t hash;  /* name hint hash */
> 
> 
> /*
>  * We search for deltas in a list sorted by type, by filename hash, and then
>  * by size, so that we see progressively smaller and smaller files.
>  * That's because we prefer deltas to be from the bigger file
>  * to the smaller -- deletes are potentially cheaper, but perhaps
>  * more importantly, the bigger file is likely the more recent
>  * one.  The deepest deltas are therefore the oldest objects which are
>  * less susceptible to be accessed often.
>  */
> static int type_size_sort(const void *_a, const void *_b)
> {
> const struct object_entry *a = *(struct object_entry **)_a;
> const struct object_entry *b = *(struct object_entry **)_b;
> 
> if (a->type > b->type)
> return -1;
> if (a->type < b->type) 
> return 1;
> if (a->hash > b->hash)
> return -1;
> if (a->hash < b->hash)
> return 1;
>   ...
>  8< 
> 
> Documentation/technical/pack-heuristics.txt also confirms this:
> 
>  8< 
> ...
>  The quote from the above linus should be rewritten a
> bit (wait for it):
> - first sort by type.  Different objects never delta with
>   each other.
> - then sort by filename/dirname.  hash of the basename
>   occupies the top BITS_PER_INT-DIR_BITS bits, and bottom
>   DIR_BITS are for the hash of leading path elements.
> 
> ...
> 
> If I might add, the trick is to make files that _might_ be similar be
> located close to each other in the hash buckets based on their file
> names.  It used to be that "foo/Makefile", "bar/baz/quux/Makefile" and
> "Makefile" all landed in the same bucket due to their common basename,
> "Makefile". However, now they land in "close" buckets.
> 
> The algorithm allows not just for the _same_ bucket, but for _close_
> buckets to be considered delta candidates.  The rationale is
> essentially that files, like Makefiles, often have very similar
> content no matter what directory they live in.
>  8< 
> 
> 
> So yes, exactly as you say with pack.writeBitmapHashCache=true (ae4f07fb) the
> delta-search heu

Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

[Kirill Smelkov]

Peff first of all thanks for feedback,

On Thu, Jul 07, 2016 at 04:52:23PM -0400, Jeff King wrote:
> On Thu, Jul 07, 2016 at 10:09:17PM +0300, Kirill Smelkov wrote:
> 
> > Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects)
> > if a repository has bitmap index, pack-objects can nicely speedup
> > "Counting objects" graph traversal phase. That however was done only for
> > case when resultant pack is sent to stdout, not written into a file.
> > 
> > We can teach pack-objects to use bitmap index for initial object
> > counting phase when generating resultant pack file too:
> 
> I'm not sure this is a good idea in general. When bitmaps are in use, we
> cannot fill out the details in the object-packing list as thoroughly. In
> particular:
> 
>   - we will not compute the same write order (which is based on
> traversal order), leading to packs that have less efficient cache
> characteristics

I agree the order can be not exactly the same. Still if original pack is
packed well (with good recency order), while using bitmap we will tend
to traverse it in close to original order.

Maybe I'm not completely right on this, but to me it looks to be the
case because if objects in original pack are put there linearly sorted
by recency order, and we use bitmap index to set of all reachable
objects from a root, and then just _linearly_ gather all those objects
from original pack by 1s in bitmap and put them in the same order into
destination pack, the recency order won't be broken.

Or am I maybe misunderstanding something?

Please also see below:

>   - we don't learn about the filename of trees and blobs, which is going
> to make the delta step much less efficient. This might be mitigated
> by turning on the bitmap name-hash cache; I don't recall how much
> detail pack-objects needs on the name (i.e., the full name versus
> just the hash).

If I understand it right, it uses only uint32_t name hash while searching. From
pack-objects.{h,c} :

 8< 
struct object_entry {
...
uint32_t hash;  /* name hint hash */


/*
 * We search for deltas in a list sorted by type, by filename hash, and then
 * by size, so that we see progressively smaller and smaller files.
 * That's because we prefer deltas to be from the bigger file
 * to the smaller -- deletes are potentially cheaper, but perhaps
 * more importantly, the bigger file is likely the more recent
 * one.  The deepest deltas are therefore the oldest objects which are
 * less susceptible to be accessed often.
 */
static int type_size_sort(const void *_a, const void *_b)
{
const struct object_entry *a = *(struct object_entry **)_a;
const struct object_entry *b = *(struct object_entry **)_b;

if (a->type > b->type)
return -1;
if (a->type < b->type) 
return 1;
if (a->hash > b->hash)
return -1;
if (a->hash < b->hash)
return 1;
...
 8< 

Documentation/technical/pack-heuristics.txt also confirms this:

 8< 
...
 The quote from the above linus should be rewritten a
bit (wait for it):
- first sort by type.  Different objects never delta with
  each other.
- then sort by filename/dirname.  hash of the basename
  occupies the top BITS_PER_INT-DIR_BITS bits, and bottom
  DIR_BITS are for the hash of leading path elements.

...

If I might add, the trick is to make files that _might_ be similar be
located close to each other in the hash buckets based on their file
names.  It used to be that "foo/Makefile", "bar/baz/quux/Makefile" and
"Makefile" all landed in the same bucket due to their common basename,
"Makefile". However, now they land in "close" buckets.

The algorithm allows not just for the _same_ bucket, but for _close_
buckets to be considered delta candidates.  The rationale is
essentially that files, like Makefiles, often have very similar
content no matter what directory they live in.
 8< 


So yes, exactly as you say with pack.writeBitmapHashCache=true (ae4f07fb) the
delta-search heuristics is almost as efficient as with just raw filenames.

I can confirm this also via e.g. (with my patch applied) :

 8< 
$ time echo 0186ac99 | git pack-objects --no-use-bitmap-index --revs 
erp5pack-plain
Counting objects: 627171, done.
Compressing objects: 100% (176949/176949), done.
50570987560d481742af4a8083028c2322a0534a
Writing objects: 100% (627171/627171), done.
Total 627171 (delta 439404), reused 594820 (delta 410210)

real0m37.272s
user0m33.648s
sys 0m1.580s

$ time echo 0186ac99 | git pack-objects --revs erp5pack-bitmap
Counting objects: 627171,

[PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too

[Kirill Smelkov]

Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects)
if a repository has bitmap index, pack-objects can nicely speedup
"Counting objects" graph traversal phase. That however was done only for
case when resultant pack is sent to stdout, not written into a file.

We can teach pack-objects to use bitmap index for initial object
counting phase when generating resultant pack file too:

- if we know bitmap index generation is not enabled for resultant pack:

  Current code has singleton bitmap_git so cannot work simultaneously
  with two bitmap indices.

- if we keep pack reuse enabled still only for "send-to-stdout" case:

  Because on pack reuse raw entries are directly written out to destination
  pack by write_reused_pack() bypassing needed for pack index generation
  bookkeeping done by regular codepath in write_one() and friends.

  (at least that's my understanding after briefly looking at the code)

We also need to care and teach add_object_entry_from_bitmap() to respect
--local via not adding nonlocal loose object to resultant pack (this
is bitmap-codepath counterpart of daae0625 (pack-objects: extend --local
to mean ignore non-local loose objects too) -- not to break 'loose
objects in alternate ODB are not repacked' in t7700-repack.sh .

Otherwise all git tests pass, and for pack-objects -> file we get nice
speedup:

erp5.git[1] (~230MB) extracted from ~ 5GB lab.nexedi.com backup
repository managed by git-backup[2] via

time echo 0186ac99 | git pack-objects --revs erp5pack

before:  37.2s
after:   26.2s

And for `git repack -adb` packed git.git

time echo 5c589a73 | git pack-objects --revs gitpack

before:   7.1s
after:3.6s

i.e. it can be 30% - 50% speedup for pack extraction.

git-backup extracts many packs on repositories restoration. That was my
initial motivation for the patch.

[1] https://lab.nexedi.com/nexedi/erp5
[2] https://lab.nexedi.com/kirr/git-backup

Cc: Vicent Marti <tan...@gmail.com>
Cc: Jeff King <p...@peff.net>
Signed-off-by: Kirill Smelkov <k...@nexedi.com>
---
 builtin/pack-objects.c  | 7 +--
 t/t5310-pack-bitmaps.sh | 9 +
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index a2f8cfd..be0ebe8 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1052,6 +1052,9 @@ static int add_object_entry_from_bitmap(const unsigned 
char *sha1,
 {
uint32_t index_pos;
 
+   if (local && has_loose_object_nonlocal(sha1))
+   return 0;
+
if (have_duplicate_entry(sha1, 0, _pos))
return 0;
 
@@ -2488,7 +2491,7 @@ static int get_object_list_from_bitmap(struct rev_info 
*revs)
if (prepare_bitmap_walk(revs) < 0)
return -1;
 
-   if (pack_options_allow_reuse() &&
+   if (pack_options_allow_reuse() && pack_to_stdout &&
!reuse_partial_packfile_from_bitmap(
_packfile,
_packfile_objects,
@@ -2773,7 +2776,7 @@ int cmd_pack_objects(int argc, const char **argv, const 
char *prefix)
if (!rev_list_all || !rev_list_reflog || !rev_list_index)
unpack_unreachable_expiration = 0;
 
-   if (!use_internal_rev_list || !pack_to_stdout || 
is_repository_shallow())
+   if (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) 
|| is_repository_shallow())
use_bitmap_index = 0;
 
if (pack_to_stdout || !rev_list_all)
diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh
index 3893afd..533fc31 100755
--- a/t/t5310-pack-bitmaps.sh
+++ b/t/t5310-pack-bitmaps.sh
@@ -118,6 +118,15 @@ test_expect_success 'incremental repack can disable 
bitmaps' '
git repack -d --no-write-bitmap-index
 '
 
+test_expect_success 'pack-objects to file can use bitmap' '
+   # make sure we still have 1 bitmap index from previous tests
+   ls .git/objects/pack/ | grep bitmap >output &&
+   test_line_count = 1 output &&
+   # pack-objects uses bitmap index by default, when it is available
+   packsha1=$(git pack-objects --all mypack output &&
-- 
2.9.0.431.gb11dac7.dirty
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[ANNOUNCE] Git-backup - Backup set of Git repositories & just files; efficiently

[Kirill Smelkov]

https://lab.nexedi.com/kirr/git-backup.git

This program backups files and set of bare Git repositories into one Git 
repository.
Files are copied to blobs and then added to tree under certain place, and for
Git repositories, all reachable objects are pulled in with maintaining index
which remembers reference -> sha1 for all pulled repositories.

This allows to leverage Git's good data deduplication ability, especially for
cases when there are many hosted repositories which are forks of each other,
and for backup to have history and be otherwise managed as a usual Git
repository.  In particular it is possible to use standard git pull/push to
synchronize backups in several places.

Backup workflow is:

1. create backup repository::

 $ mkdir backup
 $ cd backup
 $ git init # both bare and non-bare possible

2. pull files and Git repositories into backup repository::

 $ git-backup pull dir1:prefix1 dir2:prefix2 ...

   This will pull bare Git repositories & just files from `dir1` into backup
   under `prefix1`, from `dir2` into backup prefix `prefix2`, etc...

3. restore files and Git repositories from backup::

 $ git-backup restore  prefix1:dir1

   Restore Git repositories & just files from backup `prefix1` into `dir1`,
   from backup `prefix2` into `dir2`, etc...

   Backup state to restore is taken from  which is sha1 or
   ref pointing to backup repository state.

4. backup repository itself can be managed with Git. In particular it can be
   synchronized between several places with standard git pull/push, be
   repacked, etc::

 $ git push ...
 $ git pull ...


Technical overview of how it works
--

After objects from backuped Git repositories are pulled in, we create new
commit which references tree with changed backup index and files, and also has
all head objects from pulled-in repositories in its parents(*). This way backup
has history and all pulled objects become reachable from single head commit in
backup repository. In particular that means that the whole state of backup can
be described with only single sha1, and that backup repository itself could be
synchronized via standard git pull/push, be repacked, etc.

Restoration process is the opposite - from a particular backup state, files are
extracted at a proper place, and for Git repositories a pack with all objects
reachable from that repository heads is prepared and extracted from backup
repository object database.

Data for all backuped files and repositories can be accessed if one has access
to backup repository, so either they all should be in the same security domain,
or extra care has to be taken to protect access to backup repository.

File permissions are not managed with strict details due to inherent
nature of Git. This aspect can be improved with e.g. etckeeper-like
(http://etckeeper.branchable.com/) approach if needed.

NOTE the idea of pulling all refs together is similar to git-namespaces
 http://git-scm.com/docs/gitnamespaces

(*) Tag objects are handled specially - because in a lot of places Git insists 
and
assumes commit parents can only be commit objects. We encode tag objects in
specially-crafted commit object on pull, and decode back on backup restore.

We do likewise if a ref points to tree or blob, which is valid in Git.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 17/19] Portable alloca for Git

[Kirill Smelkov]

On Thu, Mar 27, 2014 at 06:22:50PM +0400, Kirill Smelkov wrote:
 On Mon, Mar 24, 2014 at 02:47:24PM -0700, Junio C Hamano wrote:
  Kirill Smelkov k...@mns.spb.ru writes:
  
   On Fri, Feb 28, 2014 at 06:19:58PM +0100, Erik Faye-Lund wrote:
   On Fri, Feb 28, 2014 at 6:00 PM, Kirill Smelkov k...@mns.spb.ru wrote:
   ...
In fact that would be maybe preferred, for maintainers to enable alloca
with knowledge and testing, as one person can't have them all at hand.
   
   Yeah, you're probably right.
  
   Erik, the patch has been merged into pu today. Would you please
   follow-up with tested MINGW change?
  
  Sooo I lost track but this discussion seems to have petered out
  around here.  I think the copy we have had for a while on 'pu' is
  basically sound, and can easily built on by platform folks by adding
  or removing the -DHAVE_ALLOCA_H from the Makefile.
 
 Yes, that is all correct - that version works and we can improve it in
 the future with platform-specific follow-up patches, if needed.

Junio, thanks for merging this and other diff-tree patches to next.  It
so happened that I'm wrestling with MSysGit today, so please also find
alloca-for-mingw patch attached below.

Thanks,
Kirill

 8 
Subject: [PATCH] mingw: activate alloca

Both MSVC and MINGW have alloca(3) definitions in malloc.h, so by moving
win32-compat alloca.h from compat/vcbuild/include/ to compat/win32/ ,
which is included by both MSVC and MINGW CFLAGS, we can make alloca()
work on both those Windows environments.

In MINGW, malloc.h has explicit check for GNUC and if it is so, defines
alloca to __builtin_alloca, so it looks like we don't need to add any
code to here-shipped alloca.h to get optimum performance.

Compile-tested on Windows in MSysGit.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---
 compat/{vcbuild/include = win32}/alloca.h | 0
 config.mak.uname   | 1 +
 2 files changed, 1 insertion(+)
 rename compat/{vcbuild/include = win32}/alloca.h (100%)

diff --git a/compat/vcbuild/include/alloca.h b/compat/win32/alloca.h
similarity index 100%
rename from compat/vcbuild/include/alloca.h
rename to compat/win32/alloca.h
diff --git a/config.mak.uname b/config.mak.uname
index 17ef893..67bc054 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -480,6 +480,7 @@ ifeq ($(uname_S),NONSTOP_KERNEL)
 endif
 ifneq (,$(findstring MINGW,$(uname_S)))
pathsep = ;
+   HAVE_ALLOCA_H = YesPlease
NO_PREAD = YesPlease
NEEDS_CRYPTO_WITH_SSL = YesPlease
NO_LIBGEN_H = YesPlease
-- 
1.9.0.msysgit.0.31.g74d1b9a.dirty

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 18/19] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

[Kirill Smelkov]

On Mon, Apr 07, 2014 at 10:29:46AM -0700, Junio C Hamano wrote:
 Kirill Smelkov k...@navytux.spb.ru writes:
 
  The following
  ...
  maybe looks a bit simpler, but calls tree_entry_pathcmp twice more times.
 
  Besides for important nparent=1 case we were not calling
  tree_entry_pathcmp at all and here we'll call it once, which would slow
  execution down a bit, as base_name_compare shows measurable enough in 
  profile.
  To avoid that we'll need to add 'if (i==imin) continue' and this won't
  be so simple then. And for general nparent case, as I've said, we'll be
  calling tree_entry_pathcmp twice more times...
 
  Because of all that I'd suggest to go with my original version.
 
 OK.

Thanks.

  ... After some break on the topic,
  with a fresh eye I see a lot of confusion goes from the notation I've
  chosen initially (because of how I was reasoning about it on paper, when
  it was in flux) - i.e. xi for x[imin] and also using i as looping
  variable. And also because xi was already used for x[imin] I've used
  another letter 'k' denoting all other x'es, which leads to confusion...
 
 
  I propose we do the following renaming to clarify things:
 
  A/a -  T/t (to match resulting tree t name in the code)
  X/x -  P/p (to match parents trees tp in the code)
  i   -  imin(so that i would be free for other tasks)
 
  then the above (with a prologue) would look like
 
   8 
   *   T P1   Pn
   *   - --
   *  |t|   |p1| |pn|
   *  |-|   |--| ... |--|  imin = argmin(p1...pn)
   *  | |   |  | |  |
   *  |-|   |--| |--|
   *  |.|   |. | |. |
   *   . ..
   *   . ..
   *
   * at any time there could be 3 cases:
   *
   *  1)  t  p[imin];
   *  2)  t  p[imin];
   *  3)  t = p[imin].
   *
   * Schematic deduction of what every case means, and what to do, follows:
   *
   * 1)  t  p[imin]  -  ∀j t ∉ Pj  -  +t ∈ D(T,Pj)  -  D += +t;  t↓
   *
   * 2)  t  p[imin]
   *
   * 2.1) ∃j: pj  p[imin]  -  -p[imin] ∉ D(T,Pj)  -  D += ø;  ∀ 
  pi=p[imin]  pi↓
   * 2.2) ∀i  pi = p[imin]  -  pi ∉ T  -  -pi ∈ D(T,Pi)  -  D += 
  -p[imin];  ∀i pi↓
   *
   * 3)  t = p[imin]
   *
   * 3.1) ∃j: pj  p[imin]  -  +t ∈ D(T,Pj)  -  only pi=p[imin] 
  remains to investigate
   * 3.2) pi = p[imin]  -  investigate δ(t,pi)
   *  |
   *  |
   *  v
   *
   * 3.1+3.2) looking at δ(t,pi) ∀i: pi=p[imin] - if all != ø  -
   *
   *   ⎧δ(t,pi)  - if pi=p[imin]
   *  -  D += ⎨
   *   ⎩+t - if pip[imin]
   *
   *
   * in any case t↓  ∀ pi=p[imin]  pi↓
  ...
  now xk is gone and i matches p[i] (= pi) etc so variable names correlate
  to algorithm description better.
 
  Does that maybe clarify things?
 
 That sounds more consistent (modulo perhaps s/argmin/min/ at the
 beginning?).

Thanks. argmin is there on purpose - min(p1...pn) is the minimal p, and
argmin(p1...pn) is imin such that p[imin] is minimal. As we are finding
the index of the minimal element we should use argmin.


  P.S. Sorry for maybe some crept-in mistakes - I've tried to verify it
  thoroughly, but am too sleepy to be completely sure. On the other hand I
  think and hope the patch should be ok.
 
 Thanks and do not be sorry for mistakes---we have the review
 process exactly for catching them.

Thanks, I appreciate that.


On Mon, Apr 07, 2014 at 11:07:47AM -0700, Junio C Hamano wrote:
 Kirill Smelkov k...@navytux.spb.ru writes:
 
   +if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)) {
   +for (i = 0; i  nparent; ++i)
   +if (tp[i].entry.mode  
   S_IFXMIN_NEQ)
   +goto skip_emit_tp;
   +}
   +
   +p = emit_path(p, base, opt, nparent,
   +/*t=*/NULL, tp, imin);
   +
   +skip_emit_tp:
   +/* ∀ xk=ximin  xk↓ */
   +update_tp_entries(tp, nparent);
  
  There are parents whose path sort earlier than what is in 't'
  (i.e. they were lost in the result---we would want to show
  removal).  What makes us jump to the skip label?
  
  We are looking at path in 't', and some parents have paths that
  sort earlier than that path.  We will not go to skip label if
  any one of the parent's entry sorts after some other parent (or
  the parent in question has ran out its entries), which means we
  show the entry from the parents only when all the parents have
  that same path, which is missing from 't'.
  
  I am not sure if I am reading this correctly, though.
  
  For the two-way diff, the above degenerates to show all parent
  entries that come before the first entry in 't', which is correct.
  For the combined diff, the current

Re: [PATCH v2 18/19] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

[Kirill Smelkov]

Junio,

First of all thanks a lot for reviewing this patch. I'll reply inline
with corrected version attached in the end.

On Fri, Apr 04, 2014 at 11:42:39AM -0700, Junio C Hamano wrote:
 Kirill Smelkov k...@navytux.spb.ru writes:
 
  +extern
  +struct combine_diff_path *diff_tree_paths(
 
 These two on the same line, please.

Ok

  +   struct combine_diff_path *p, const unsigned char *sha1,
  +   const unsigned char **parent_sha1, int nparent,
  +   struct strbuf *base, struct diff_options *opt);
   extern int diff_tree_sha1(const unsigned char *old, const unsigned char 
  *new,
const char *base, struct diff_options *opt);
  ...
  +/*
  + * convert path - opt-diff_*() callbacks
  + *
  + * emits diff to first parent only, and tells diff tree-walker that we are 
  done
  + * with p and it can be freed.
  + */
  +static int emit_diff_first_parent_only(struct diff_options *opt, struct 
  combine_diff_path *p)
   {
 
 Very straight-forward; good.

Thanks

  +static struct combine_diff_path *path_appendnew(struct combine_diff_path 
  *last,
  +   int nparent, const struct strbuf *base, const char *path, int pathlen,
  +   unsigned mode, const unsigned char *sha1)
  +{
  +   struct combine_diff_path *p;
  +   int len = base-len + pathlen;
  +   int alloclen = combine_diff_path_size(nparent, len);
  +
  +   /* if last-next is !NULL - it is a pre-allocated memory, we can reuse 
  */
  +   p = last-next;
  +   if (p  (alloclen  (intptr_t)p-next)) {
  +   free(p);
  +   p = NULL;
  +   }
  +
  +   if (!p) {
  +   p = xmalloc(alloclen);
  +
  +   /*
  +* until we go to it next round, .next holds how many bytes we
  +* allocated (for faster realloc - we don't need copying old 
  data).
  +*/
  +   p-next = (struct combine_diff_path *)(intptr_t)alloclen;
 
 This reuse of the .next field is somewhat yucky, but it is very
 localized inside a function that has a single callsite to this
 function, so let's let it pass.

I agree it is not pretty, but it was the best approach I could find
for avoiding memory re-allocation without introducing new fields into
`struct combine_diff_path`. And yes, the trick is localized, so let's
let it live.


  +static struct combine_diff_path *emit_path(struct combine_diff_path *p,
  +   struct strbuf *base, struct diff_options *opt, int nparent,
  +   struct tree_desc *t, struct tree_desc *tp,
  +   int imin)
   {
 
 Again, fairly straight-forward and good.

Thanks again.


  +/*
  + * generate paths for combined diff D(sha1,parents_sha1[])
  + ...
  +static struct combine_diff_path *ll_diff_tree_paths(
  +   struct combine_diff_path *p, const unsigned char *sha1,
  +   const unsigned char **parents_sha1, int nparent,
  +   struct strbuf *base, struct diff_options *opt)
  +{
  +   struct tree_desc t, *tp;
  +   void *ttree, **tptree;
  +   int i;
  +
  +   tp = xalloca(nparent * sizeof(tp[0]));
  +   tptree = xalloca(nparent * sizeof(tptree[0]));
  +
  +   /*
  +* load parents first, as they are probably already cached.
  +*
  +* ( log_tree_diff() parses commit-parent before calling here via
  +*   diff_tree_sha1(parent, commit) )
  +*/
  +   for (i = 0; i  nparent; ++i)
  +   tptree[i] = fill_tree_descriptor(tp[i], parents_sha1[i]);
  +   ttree = fill_tree_descriptor(t, sha1);
   
  /* Enable recursion indefinitely */
  opt-pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
   
  for (;;) {
  -   int cmp;
  +   int imin, cmp;
   
  if (diff_can_quit_early(opt))
  break;
  +
  if (opt-pathspec.nr) {
  -   skip_uninteresting(t1, base, opt);
  -   skip_uninteresting(t2, base, opt);
  +   skip_uninteresting(t, base, opt);
  +   for (i = 0; i  nparent; i++)
  +   skip_uninteresting(tp[i], base, opt);
  }
  -   if (!t1.size  !t2.size)
  -   break;
   
  -   cmp = tree_entry_pathcmp(t1, t2);
  +   /* comparing is finished when all trees are done */
  +   if (!t.size) {
  +   int done = 1;
  +   for (i = 0; i  nparent; ++i)
  +   if (tp[i].size) {
  +   done = 0;
  +   break;
  +   }
  +   if (done)
  +   break;
  +   }
  +
  +   /*
  +* lookup imin = argmin(x1...xn),
  +* mark entries whether they =tp[imin] along the way
  +*/
  +   imin = 0;
  +   tp[0].entry.mode = ~S_IFXMIN_NEQ;
  +
  +   for (i = 1; i  nparent; ++i) {
  +   cmp = tree_entry_pathcmp(tp[i], tp[imin]);
  +   if (cmp  0) {
  +   imin = i;
  +   tp[i

Re: [PATCH 17/19] Portable alloca for Git

[Kirill Smelkov]

On Mon, Mar 24, 2014 at 02:47:24PM -0700, Junio C Hamano wrote:
 Kirill Smelkov k...@mns.spb.ru writes:
 
  On Fri, Feb 28, 2014 at 06:19:58PM +0100, Erik Faye-Lund wrote:
  On Fri, Feb 28, 2014 at 6:00 PM, Kirill Smelkov k...@mns.spb.ru wrote:
  ...
   In fact that would be maybe preferred, for maintainers to enable alloca
   with knowledge and testing, as one person can't have them all at hand.
  
  Yeah, you're probably right.
 
  Erik, the patch has been merged into pu today. Would you please
  follow-up with tested MINGW change?
 
 Sooo I lost track but this discussion seems to have petered out
 around here.  I think the copy we have had for a while on 'pu' is
 basically sound, and can easily built on by platform folks by adding
 or removing the -DHAVE_ALLOCA_H from the Makefile.

Yes, that is all correct - that version works and we can improve it in
the future with platform-specific follow-up patches, if needed.

Please pick up the patch with ack from Thomas Schwinge.

Thanks,
Kirill

(please keep author email)
 8 
From: Kirill Smelkov k...@mns.spb.ru
Date: Mon, 24 Feb 2014 20:21:49 +0400
Subject: [PATCH v1a] Portable alloca for Git

In the next patch we'll have to use alloca() for performance reasons,
but since alloca is non-standardized and is not portable, let's have a
trick with compatibility wrappers:

1. at configure time, determine, do we have working alloca() through
   alloca.h, and define

#define HAVE_ALLOCA_H

   if yes.

2. in code

#ifdef HAVE_ALLOCA_H
# include alloca.h
# define xalloca(size)  (alloca(size))
# define xalloca_free(p)do {} while(0)
#else
# define xalloca(size)  (xmalloc(size))
# define xalloca_free(p)(free(p))
#endif

   and use it like

   func() {
   p = xalloca(size);
   ...

   xalloca_free(p);
   }

This way, for systems, where alloca is available, we'll have optimal
on-stack allocations with fast executions. On the other hand, on
systems, where alloca is not available, this gracefully fallbacks to
xmalloc/free.

Both autoconf and config.mak.uname configurations were updated. For
autoconf, we are not bothering considering cases, when no alloca.h is
available, but alloca() works some other way - its simply alloca.h is
available and works or not, everything else is deep legacy.

For config.mak.uname, I've tried to make my almost-sure guess for where
alloca() is available, but since I only have access to Linux it is the
only change I can be sure about myself, with relevant to other changed
systems people Cc'ed.

NOTE

SunOS and Windows had explicit -DHAVE_ALLOCA_H in their configurations.
I've changed that to now-common HAVE_ALLOCA_H=YesPlease which should be
correct.

Cc: Brandon Casey draf...@gmail.com
Cc: Marius Storm-Olsen msto...@gmail.com
Cc: Johannes Sixt j...@kdbg.org
Cc: Johannes Schindelin johannes.schinde...@gmx.de
Cc: Ramsay Jones ram...@ramsay1.demon.co.uk
Cc: Gerrit Pape p...@smarden.org
Cc: Petr Salinger petr.salin...@seznam.cz
Cc: Jonathan Nieder jrnie...@gmail.com
Cc: Thomas Schwinge tschwi...@gnu.org
Acked-by: Thomas Schwinge tho...@codesourcery.com (GNU Hurd changes)
Signed-off-by: Kirill Smelkov k...@mns.spb.ru
Signed-off-by: Junio C Hamano gits...@pobox.com
---

Changes since v1:

 - added ack for GNU/Hurd.

 Makefile  |  6 ++
 config.mak.uname  | 10 --
 configure.ac  |  8 
 git-compat-util.h |  8 
 4 files changed, 30 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index dddaf4f..0334806 100644
--- a/Makefile
+++ b/Makefile
@@ -30,6 +30,8 @@ all::
 # Define LIBPCREDIR=/foo/bar if your libpcre header and library files are in
 # /foo/bar/include and /foo/bar/lib directories.
 #
+# Define HAVE_ALLOCA_H if you have working alloca(3) defined in that header.
+#
 # Define NO_CURL if you do not have libcurl installed.  git-http-fetch and
 # git-http-push are not built, and you cannot use http:// and https://
 # transports (neither smart nor dumb).
@@ -1099,6 +1101,10 @@ ifdef USE_LIBPCRE
EXTLIBS += -lpcre
 endif
 
+ifdef HAVE_ALLOCA_H
+   BASIC_CFLAGS += -DHAVE_ALLOCA_H
+endif
+
 ifdef NO_CURL
BASIC_CFLAGS += -DNO_CURL
REMOTE_CURL_PRIMARY =
diff --git a/config.mak.uname b/config.mak.uname
index 7d31fad..71602ee 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -28,6 +28,7 @@ ifeq ($(uname_S),OSF1)
NO_NSEC = YesPlease
 endif
 ifeq ($(uname_S),Linux)
+   HAVE_ALLOCA_H = YesPlease
NO_STRLCPY = YesPlease
NO_MKSTEMPS = YesPlease
HAVE_PATHS_H = YesPlease
@@ -35,6 +36,7 @@ ifeq ($(uname_S),Linux)
HAVE_DEV_TTY = YesPlease
 endif
 ifeq ($(uname_S),GNU/kFreeBSD)
+   HAVE_ALLOCA_H = YesPlease
NO_STRLCPY = YesPlease
NO_MKSTEMPS = YesPlease
HAVE_PATHS_H = YesPlease
@@ -103,6 +105,7 @@ ifeq ($(uname_S),SunOS)
NEEDS_NSL = YesPlease
SHELL_PATH = /bin/bash
SANE_TOOL_PATH = /usr/xpg6/bin:/usr/xpg4/bin

Re: [PATCH v2 18/19] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

[Kirill Smelkov]

On Mon, Feb 24, 2014 at 08:21:50PM +0400, Kirill Smelkov wrote:
[...]
 not changed:
 
 - low-level helpers are still named with __ prefix as, imho, that is the 
 best
   convention to name such helpers, without sacrificing signal/noise ratio. All
   of them are now static though.

Please find attached corrected version of this patch with
__diff_tree_sha1() renamed to ll_diff_tree_sha1() and other identifiers
corrected similarly for consistency with Git codebase style.

Thanks,
Kirill

(please keep author email)
 8 
From: Kirill Smelkov k...@mns.spb.ru
Date: Mon, 24 Feb 2014 20:21:50 +0400
Subject: [PATCH v3] tree-diff: rework diff_tree() to generate diffs for 
multiparent cases as well

Previously diff_tree(), which is now named ll_diff_tree_sha1(), was
generating diff_filepair(s) for two trees t1 and t2, and that was
usually used for a commit as t1=HEAD~, and t2=HEAD - i.e. to see changes
a commit introduces.

In Git, however, we have fundamentally built flexibility in that a
commit can have many parents - 1 for a plain commit, 2 for a simple merge,
but also more than 2 for merging several heads at once.

For merges there is a so called combine-diff, which shows diff, a merge
introduces by itself, omitting changes done by any parent. That works
through first finding paths, that are different to all parents, and then
showing generalized diff, with separate columns for +/- for each parent.
The code lives in combine-diff.c .

There is an impedance mismatch, however, in that a commit could
generally have any number of parents, and that while diffing trees, we
divide cases for 2-tree diffs and more-than-2-tree diffs. I mean there
is no special casing for multiple parents commits in e.g.
revision-walker .

That impedance mismatch *hurts* *performance* *badly* for generating
combined diffs - in combine-diff: optimize combine_diff_path
sets intersection I've already removed some slowness from it, but from
the timings provided there, it could be seen, that combined diffs still
cost more than an order of magnitude more cpu time, compared to diff for
usual commits, and that would only be an optimistic estimate, if we take
into account that for e.g. linux.git there is only one merge for several
dozens of plain commits.

That slowness comes from the fact that currently, while generating
combined diff, a lot of time is spent computing diff(commit,commit^2)
just to only then intersect that huge diff to almost small set of files
from diff(commit,commit^1).

That's because at present, to compute combine-diff, for first finding
paths, that every parent touches, we use the following combine-diff
property/definition:

D(A,P1...Pn) = D(A,P1) ^ ... ^ D(A,Pn)  (w.r.t. paths)

where

D(A,P1...Pn) is combined diff between commit A, and parents Pi

and

D(A,Pi) is usual two-tree diff Pi..A

So if any of that D(A,Pi) is huge, tracting 1 n-parent combine-diff as n
1-parent diffs and intersecting results will be slow.

And usually, for linux.git and other topic-based workflows, that
D(A,P2) is huge, because, if merge-base of A and P2, is several dozens
of merges (from A, via first parent) below, that D(A,P2) will be diffing
sum of merges from several subsystems to 1 subsystem.

The solution is to avoid computing n 1-parent diffs, and to find
changed-to-all-parents paths via scanning A's and all Pi's trees
simultaneously, at each step comparing their entries, and based on that
comparison, populate paths result, and deduce we could *skip*
*recursing* into subdirectories, if at least for 1 parent, sha1 of that
dir tree is the same as in A. That would save us from doing significant
amount of needless work.

Such approach is very similar to what diff_tree() does, only there we
deal with scanning only 2 trees simultaneously, and for n+1 tree, the
logic is a bit more complex:

D(A,X1...Xn) calculation scheme
---

D(A,X1...Xn) = D(A,X1) ^ ... ^ D(A,Xn)   (regarding resulting paths set)

 D(A,Xj) - diff between A..Xj
 D(A,X1...Xn)- combined diff from A to parents X1,...,Xn

We start from all trees, which are sorted, and compare their entries in
lock-step:

  A X1   Xn
  - --
 |a|   |x1| |xn|
 |-|   |--| ... |--|  i = argmin(x1...xn)
 | |   |  | |  |
 |-|   |--| |--|
 |.|   |. | |. |
  . ..
  . ..

at any time there could be 3 cases:

 1)  a  xi;
 2)  a  xi;
 3)  a = xi.

Schematic deduction of what every case means, and what to do, follows:

1)  a  xi  -  ∀j a ∉ Xj  -  +a ∈ D(A,Xj)  -  D += +a;  a↓

2)  a  xi

2.1) ∃j: xj  xi  -  -xi ∉ D(A,Xj)  -  D += ø;  ∀ xk=xi  xk↓
2.2) ∀j  xj = xi  -  xj ∉ A  -  -xj ∈ D(A,Xj)  -  D += -xi;  ∀j xj↓

3)  a = xi

3.1) ∃j: xj  xi  -  +a ∈ D(A,Xj)  -  only xk=xi remains to investigate
3.2) xj = xi  -  investigate δ(a,xj)
 |
 |
 v

3.1+3.2) looking at δ(a,xk) ∀k: xk=xi - if all != ø

Re: [PATCH v2 16/19] tree-diff: reuse base str(buf) memory on sub-tree recursion

[Kirill Smelkov]

On Tue, Mar 25, 2014 at 01:23:20PM +0400, Kirill Smelkov wrote:
 On Mon, Mar 24, 2014 at 02:43:36PM -0700, Junio C Hamano wrote:
  Kirill Smelkov k...@mns.spb.ru writes:
  
   instead of allocating it all the time for every subtree in
   __diff_tree_sha1, let's allocate it once in diff_tree_sha1, and then all
   callee just use it in stacking style, without memory allocations.
  
   This should be faster, and for me this change gives the following
   slight speedups for
  
   git log --raw --no-abbrev --no-renames --format='%H'
  
   navy.gitlinux.git v3.10..v3.11
  
   before  0.618s  1.903s
   after   0.611s  1.889s
   speedup 1.1%0.7%
  
   Signed-off-by: Kirill Smelkov k...@mns.spb.ru
   ---
  
   Changes since v1:
  
- don't need to touch diff.h, as the function we are changing became 
   static.
  
tree-diff.c | 36 ++--
1 file changed, 18 insertions(+), 18 deletions(-)
  
   diff --git a/tree-diff.c b/tree-diff.c
   index aea0297..c76821d 100644
   --- a/tree-diff.c
   +++ b/tree-diff.c
   @@ -115,7 +115,7 @@ static void show_path(struct strbuf *base, struct 
   diff_options *opt,
 if (recurse) {
 strbuf_addch(base, '/');
 __diff_tree_sha1(t1 ? t1-entry.sha1 : NULL,
   -  t2 ? t2-entry.sha1 : NULL, base-buf, opt);
   +  t2 ? t2-entry.sha1 : NULL, base, opt);
 }

 strbuf_setlen(base, old_baselen);
  
  I was scratching my head for a while, after seeing that there does
  not seem to be any *new* code added by this patch in order to
  store-away the original length and restore the singleton base buffer
  to the original length after using addch/addstr to extend it.
  
  But I see that the code has already been prepared to do this
  conversion.  I wonder why we didn't do this earlier ;-)
 
 The conversion to reusing memory started in 48932677 diff-tree: convert
 base+baselen to writable strbuf which allowed to avoid quite a bit of
 malloc() and memcpy(), but for this to work allocation at diff_tree()
 entry had to be there.
 
 In particular it had to be there, because diff_tree() accepted base as C
 string, not strbuf, and since diff_tree() was calling itself
 recursively - oops - new allocation on every subtree.
 
 I've opened the door for avoiding allocations via splitting diff_tree
 into high-level and low-level parts. The high-level part still accepts
 `char *base`, but low-level function operates on strbuf and recurses
 into low-level self.
 
 The high-level diff_tree_sha1() still allocates memory for every
 diff(tree1,tree2), but that is significantly lower compared to
 allocating memory on every subtree...
 
 The lesson here is: better use strbuf for api unless there is a reason
 not to.
 
 
  Looks good.  Thanks.
 
 Thanks.

Thanks again. Here it goes adjusted to __diff_tree_sha1 - ll_diff_tree_sha1 
renaming:

(please keep author email)
 8 
From: Kirill Smelkov k...@mns.spb.ru
Date: Mon, 24 Feb 2014 20:21:48 +0400
Subject: [PATCH v3] tree-diff: reuse base str(buf) memory on sub-tree recursion

instead of allocating it all the time for every subtree in
ll_diff_tree_sha1, let's allocate it once in diff_tree_sha1, and then all
callee just use it in stacking style, without memory allocations.

This should be faster, and for me this change gives the following
slight speedups for

git log --raw --no-abbrev --no-renames --format='%H'

navy.gitlinux.git v3.10..v3.11

before  0.618s  1.903s
after   0.611s  1.889s
speedup 1.1%0.7%

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---

Changes since v2:

 - adjust to __diff_tree_sha1 - ll_diff_tree_sha1 renaming.

Changes since v1:

 - don't need to touch diff.h, as the function we are changing became
   static.

 tree-diff.c | 38 +++---
 1 file changed, 19 insertions(+), 19 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 7fbb022..8c8bde6 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -8,7 +8,7 @@
 
 
 static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char 
*new,
-const char *base_str, struct diff_options *opt);
+struct strbuf *base, struct diff_options *opt);
 
 /*
  * Compare two tree entries, taking into account only path/S_ISDIR(mode),
@@ -123,7 +123,7 @@ static void show_path(struct strbuf *base, struct 
diff_options *opt,
if (recurse) {
strbuf_addch(base, '/');
ll_diff_tree_sha1(t1 ? t1-entry.sha1 : NULL,
- t2 ? t2-entry.sha1 : NULL, base-buf, opt);
+ t2 ? t2-entry.sha1 : NULL, base, opt);
}
 
strbuf_setlen(base, old_baselen);
@@ -146,12 +146,10 @@ static void skip_uninteresting(struct tree_desc *t, 
struct strbuf *base,
 }
 
 static int ll_diff_tree_sha1

Re: [PATCH 15/19] tree-diff: no need to call full diff_tree_sha1 from show_path()

[Kirill Smelkov]

On Mon, Feb 24, 2014 at 08:21:47PM +0400, Kirill Smelkov wrote:
 As described in previous commit, when recursing into sub-trees, we can
 use lower-level tree walker, since its interface is now sha1 based.
 
 The change is ok, because diff_tree_sha1() only invokes
 __diff_tree_sha1(), and also, if base is empty, try_to_follow_renames().
 But base is not empty here, as we have added a path and '/' before
 recursing.
 
 Signed-off-by: Kirill Smelkov k...@mns.spb.ru
 Signed-off-by: Junio C Hamano gits...@pobox.com
 ---
 
 ( re-posting without change )
 
  tree-diff.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
 
 diff --git a/tree-diff.c b/tree-diff.c
 index f90acf5..aea0297 100644
 --- a/tree-diff.c
 +++ b/tree-diff.c
 @@ -114,8 +114,8 @@ static void show_path(struct strbuf *base, struct 
 diff_options *opt,
  
   if (recurse) {
   strbuf_addch(base, '/');
 - diff_tree_sha1(t1 ? t1-entry.sha1 : NULL,
 -t2 ? t2-entry.sha1 : NULL, base-buf, opt);
 + __diff_tree_sha1(t1 ? t1-entry.sha1 : NULL,
 +  t2 ? t2-entry.sha1 : NULL, base-buf, opt);
   }
  
   strbuf_setlen(base, old_baselen);

I've found this does not compile as I've forgot to add __diff_tree_sha1
prototype, and also we are changing naming for __diff_tree_sha1() to
ll_diff_tree_sha1() to follow Git coding style for consistency and
corrections to previous patch, so here goes v2:

(please keep author email)
 8 
From: Kirill Smelkov k...@mns.spb.ru
Date: Mon, 24 Feb 2014 20:21:47 +0400
Subject: [PATCH v2] tree-diff: no need to call full diff_tree_sha1 from 
show_path()

As described in previous commit, when recursing into sub-trees, we can
use lower-level tree walker, since its interface is now sha1 based.

The change is ok, because diff_tree_sha1() only invokes
ll_diff_tree_sha1(), and also, if base is empty, try_to_follow_renames().
But base is not empty here, as we have added a path and '/' before
recursing.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---

Changes since v1:

 - adjust to renaming __diff_tree_sha1 - ll_diff_tree_sha1;
 - added ll_diff_tree_sha1 prototype as the function is defined below
   here-introduced call-site.

 tree-diff.c | 8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 1d02e43..7fbb022 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -6,6 +6,10 @@
 #include diffcore.h
 #include tree.h
 
+
+static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char 
*new,
+const char *base_str, struct diff_options *opt);
+
 /*
  * Compare two tree entries, taking into account only path/S_ISDIR(mode),
  * but not their sha1's.
@@ -118,8 +122,8 @@ static void show_path(struct strbuf *base, struct 
diff_options *opt,
 
if (recurse) {
strbuf_addch(base, '/');
-   diff_tree_sha1(t1 ? t1-entry.sha1 : NULL,
-  t2 ? t2-entry.sha1 : NULL, base-buf, opt);
+   ll_diff_tree_sha1(t1 ? t1-entry.sha1 : NULL,
+ t2 ? t2-entry.sha1 : NULL, base-buf, opt);
}
 
strbuf_setlen(base, old_baselen);
-- 
1.9.rc0.143.g6fd479e
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Kirill Smelkov]

On Wed, Mar 26, 2014 at 02:34:24PM -0700, Junio C Hamano wrote:
 Kirill Smelkov k...@navytux.spb.ru writes:
 
  On Tue, Mar 25, 2014 at 10:46:32AM -0700, Junio C Hamano wrote:
  Kirill Smelkov k...@navytux.spb.ru writes:
  
   What are the downsides of __ prefix by the way?
  
  Aren't these names reserved for compiler/runtime implementations?
 
  Yes, but there are precedents when people don't obey it widely and
  in practice everything works :)
 
 I think you are alluding to the practice in the Linux kernel, but
 their requirement is vastly different---their product do not even
 link with libc and they always compile with specific selected
 versions of gcc, no?

Yes, that is correct. Only __ was so visually appealing that there was
a temptation to break the rules, but...


  Let it be something portable anyway -
  how about diff_tree_sha1_low() ?
 
 Sure.
 
 As this is a file-scope static, I do not think the exact naming
 matters that much.  Just FYI, we seem to use ll_ prefix (standing
 for low-level) in some places.

... let's then use this ll_ prefix scheme for consistency.

Corrected patch is below, and I've sent corrections to follow-up
patches as well.

Thanks,
Kirill

(please keep author email)
 8 
From: Kirill Smelkov k...@mns.spb.ru
Date: Mon, 24 Feb 2014 20:21:46 +0400
Subject: [PATCH v3a] tree-diff: rework diff_tree interface to be sha1 based

In the next commit this will allow to reduce intermediate calls, when
recursing into subtrees - at that stage we know only subtree sha1, and
it is natural for tree walker to start from that phase. For now we do

diff_tree
show_path
diff_tree_sha1
diff_tree
...

and the change will allow to reduce it to

diff_tree
show_path
diff_tree

Also, it will allow to omit allocating strbuf for each subtree, and just
reuse the common strbuf via playing with its len.

The above-mentioned improvements go in the next 2 patches.

The downside is that try_to_follow_renames(), if active, we cause
re-reading of 2 initial trees, which was negligible based on my timings,
and which is outweighed cogently by the upsides.

NOTE To keep with the current interface and semantics, I needed to
rename the function from diff_tree() to diff_tree_sha1(). As
diff_tree_sha1() was already used, and the function we are talking here
is its more low-level helper, let's use convention for prefixing
such helpers with ll_. So the final renaming is

diff_tree() - ll_diff_tree_sha1()

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---

Changes since v3:

 - further rename diff_tree_sha1_low() - ll_diff_tree_sha1() to follow Git
   style for naming low-level helpers.

Changes since v2:

 - renamed __diff_tree_sha1() - diff_tree_sha1_low() as the former
   overlaps with reserved-for-implementation identifiers namespace.

Changes since v1:

 - don't need to touch diff.h, as diff_tree() became static.


 tree-diff.c | 60 
 1 file changed, 28 insertions(+), 32 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index f137f39..1d02e43 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -141,12 +141,17 @@ static void skip_uninteresting(struct tree_desc *t, 
struct strbuf *base,
}
 }
 
-static int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
-const char *base_str, struct diff_options *opt)
+static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char 
*new,
+const char *base_str, struct diff_options *opt)
 {
+   struct tree_desc t1, t2;
+   void *t1tree, *t2tree;
struct strbuf base;
int baselen = strlen(base_str);
 
+   t1tree = fill_tree_descriptor(t1, old);
+   t2tree = fill_tree_descriptor(t2, new);
+
/* Enable recursion indefinitely */
opt-pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
 
@@ -159,39 +164,41 @@ static int diff_tree(struct tree_desc *t1, struct 
tree_desc *t2,
if (diff_can_quit_early(opt))
break;
if (opt-pathspec.nr) {
-   skip_uninteresting(t1, base, opt);
-   skip_uninteresting(t2, base, opt);
+   skip_uninteresting(t1, base, opt);
+   skip_uninteresting(t2, base, opt);
}
-   if (!t1-size  !t2-size)
+   if (!t1.size  !t2.size)
break;
 
-   cmp = tree_entry_pathcmp(t1, t2);
+   cmp = tree_entry_pathcmp(t1, t2);
 
/* t1 = t2 */
if (cmp == 0) {
if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
-   hashcmp(t1-entry.sha1, t2-entry.sha1) ||
-   (t1-entry.mode != t2-entry.mode))
-   show_path(base, opt, t1, t2);
+   hashcmp(t1.entry.sha1, t2

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Kirill Smelkov]

+stefanbeller

On Thu, Mar 27, 2014 at 11:48:11AM -0700, Junio C Hamano wrote:
 Kirill Smelkov k...@navytux.spb.ru writes:
 
  (please keep author email)
   8 
  From: Kirill Smelkov k...@mns.spb.ru
  Date: Mon, 24 Feb 2014 20:21:46 +0400
  Subject: [PATCH v3a] tree-diff: rework diff_tree interface to be sha1 based
 
 git am -c will discard everything above the scissors and then
 start parsing the in-body headers from there, so the above From:
 will be used.

Thanks.

 But you have a few entries in .mailmap; do you want to update them
 as well?

When Stefan Beller was contacting me on emails, if I recall correctly, I
told him all those kirr@... entries are mine, but the one this patch is
authored with indicates that something was done at work, and I'd prefer to
acknowledge that. So maybe

 8 
From: Kirill Smelkov k...@navytux.spb.ru
Date: Thu, 27 Mar 2014 23:32:14 +0400
Subject: [PATCH] .mailmap: Separate Kirill Smelkov personal and work addresses

The address k...@mns.spb.ru indicates that a patch was done at work and
I'd like to acknowledge that.

The address k...@navytux.spb.ru is my personal email and indicates that
a contribution is done completely on my own time and resources.

k...@landau.phys.spbu.ru is old university account which no longer works
(sigh, to much spam because of me on the server) and maps to
k...@navytux.spb.ru which should be considered as primary.

Signed-off-by: Kirill Smelkov k...@navytux.spb.ru
---
 .mailmap | 1 -
 1 file changed, 1 deletion(-)

diff --git a/.mailmap b/.mailmap
index 11057cb..0be5e02 100644
--- a/.mailmap
+++ b/.mailmap
@@ -117,7 +117,6 @@ Keith Cascio ke...@cs.ucla.edu ke...@cs.ucla.edu
 Kent Engstrom k...@lysator.liu.se
 Kevin Leung kevin...@gmail.com
 Kirill Smelkov k...@navytux.spb.ru k...@landau.phys.spbu.ru
-Kirill Smelkov k...@navytux.spb.ru k...@mns.spb.ru
 Knut Franke knut.fra...@gmx.de k.fra...@science-computing.de
 Lars Doelle lars.doelle@on-line ! de
 Lars Doelle lars.doe...@on-line.de
-- 
1.9.rc0.143.g6fd479e
 8 

On the other hand, it is still all me, and the main address (navytux) is
indicated correctly, so I dunno...

 By the way, in general I do not appreciate people lying on the Date:
 with an in-body header in their patches, either in the original or
 in rerolls.
 
 Thanks.

I see. Somehow it is pity that the date of original work is lost via
this approach, as now we are only changing cosmetics etc, and the bulk
of the work was done earlier.

Anyway, we can drop the date, but please keep the email, as it is used
for the acknowledgment.

Thanks,
Kirill
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Kirill Smelkov]

On Tue, Mar 25, 2014 at 10:46:32AM -0700, Junio C Hamano wrote:
 Kirill Smelkov k...@navytux.spb.ru writes:
 
  What are the downsides of __ prefix by the way?
 
 Aren't these names reserved for compiler/runtime implementations?

Yes, but there are precedents when people don't obey it widely and
in practice everything works :) Let it be something portable anyway -
how about diff_tree_sha1_low() ?

So corrected patch is below. If such suffixing will be accepted, I will
send follow-up patches corrected similiary.

  ( or please pull them from
git://repo.or.cz/git/kirr.git y6/tree-diff-walk-multitree )

Thanks,
Kirill

 8 
From: Kirill Smelkov k...@mns.spb.ru
Date: Mon, 24 Feb 2014 20:21:46 +0400
Subject: [PATCH v3] tree-diff: rework diff_tree interface to be sha1 based

In the next commit this will allow to reduce intermediate calls, when
recursing into subtrees - at that stage we know only subtree sha1, and
it is natural for tree walker to start from that phase. For now we do

diff_tree
show_path
diff_tree_sha1
diff_tree
...

and the change will allow to reduce it to

diff_tree
show_path
diff_tree

Also, it will allow to omit allocating strbuf for each subtree, and just
reuse the common strbuf via playing with its len.

The above-mentioned improvements go in the next 2 patches.

The downside is that try_to_follow_renames(), if active, we cause
re-reading of 2 initial trees, which was negligible based on my timings,
and which is outweighed cogently by the upsides.

NOTE To keep with the current interface and semantics, I needed to
rename the function from diff_tree() to diff_tree_sha1(). As
diff_tree_sha1() was already used, and the function we are talking here
is its more low-level helper, let's use convention for suffixing
such helpers with _low. So the final renaming is

diff_tree() - diff_tree_sha1_low()

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---

Changes since v2:

 - renamed __diff_tree_sha1() - diff_tree_sha1_low() as the former
   overlaps with reserved-for-implementation identifiers namespace.


 tree-diff.c | 60 
 1 file changed, 28 insertions(+), 32 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index f137f39..0277c5c 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -141,12 +141,17 @@ static void skip_uninteresting(struct tree_desc *t, 
struct strbuf *base,
}
 }
 
-static int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
-const char *base_str, struct diff_options *opt)
+static int diff_tree_sha1_low(const unsigned char *old, const unsigned char 
*new,
+ const char *base_str, struct diff_options *opt)
 {
+   struct tree_desc t1, t2;
+   void *t1tree, *t2tree;
struct strbuf base;
int baselen = strlen(base_str);
 
+   t1tree = fill_tree_descriptor(t1, old);
+   t2tree = fill_tree_descriptor(t2, new);
+
/* Enable recursion indefinitely */
opt-pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
 
@@ -159,39 +164,41 @@ static int diff_tree(struct tree_desc *t1, struct 
tree_desc *t2,
if (diff_can_quit_early(opt))
break;
if (opt-pathspec.nr) {
-   skip_uninteresting(t1, base, opt);
-   skip_uninteresting(t2, base, opt);
+   skip_uninteresting(t1, base, opt);
+   skip_uninteresting(t2, base, opt);
}
-   if (!t1-size  !t2-size)
+   if (!t1.size  !t2.size)
break;
 
-   cmp = tree_entry_pathcmp(t1, t2);
+   cmp = tree_entry_pathcmp(t1, t2);
 
/* t1 = t2 */
if (cmp == 0) {
if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
-   hashcmp(t1-entry.sha1, t2-entry.sha1) ||
-   (t1-entry.mode != t2-entry.mode))
-   show_path(base, opt, t1, t2);
+   hashcmp(t1.entry.sha1, t2.entry.sha1) ||
+   (t1.entry.mode != t2.entry.mode))
+   show_path(base, opt, t1, t2);
 
-   update_tree_entry(t1);
-   update_tree_entry(t2);
+   update_tree_entry(t1);
+   update_tree_entry(t2);
}
 
/* t1  t2 */
else if (cmp  0) {
-   show_path(base, opt, t1, /*t2=*/NULL);
-   update_tree_entry(t1);
+   show_path(base, opt, t1, /*t2=*/NULL);
+   update_tree_entry(t1);
}
 
/* t1  t2 */
else {
-   show_path(base, opt, /*t1=*/NULL, t2);
-   update_tree_entry(t2

Re: [PATCH 12/19] tree-diff: remove special-case diff-emitting code for empty-tree cases

[Kirill Smelkov]

On Mon, Mar 24, 2014 at 02:18:10PM -0700, Junio C Hamano wrote:
 Kirill Smelkov k...@mns.spb.ru writes:
 
  via teaching tree_entry_pathcmp() how to compare empty tree descriptors:
 
 Drop this line, as you explain the pretend empty compares bigger
 than anything else idea later anyway?  This early part of the
 proposed log message made me hiccup while reading it.

Hmm, I was trying to show the big picture first and only then details...


  While walking trees, we iterate their entries from lowest to highest in
  sort order, so empty tree means all entries were already went over.
 
  If we artificially assign +infinity value to such tree entry, it will
  go after all usual entries, and through the usual driver loop we will be
  taking the same actions, which were hand-coded for special cases, i.e.
 
  t1 empty, t2 non-empty
  pathcmp(+∞, t2) - +1
  show_path(/*t1=*/NULL, t2); /* = t1  t2 case in main loop */
 
  t1 non-empty, t2-empty
  pathcmp(t1, +∞) - -1
  show_path(t1, /*t2=*/NULL); /* = t1  t2 case in main loop */
 
 Sounds good.  I would have phrased a bit differently, though:
 
 When we have T1 and T2, we return a sign that tells the caller
 to indicate the earlier one to be emitted, and by returning
 the sign that causes the non-empty side to be emitted, we will
 automatically cause the entries from the remaining side to be
 emitted, without attempting to touch the empty side at all.  We
 can teach tree_entry_pathcmp() to pretend that an empty tree has
 an element that sorts after anything else to achieve this.
 
 without saying infinity.

Doesn't your description, especially an element that sorts after
anything else match what infinity is pretty exactly? :)

I agree it could read more clearly to those new to the concept, but we
are basically talking about the same thing and once someone is familiar
with infinity and its friends the second description imho is less
obvious.

Let's maybe as a compromise add your text as In other words textual
description ... ?

This way, it will hopefully be good both ways...


  Right now we never go to when compared tree descriptors are infinity,...
 
 Sorry, but I cannot parse this.

Sorry, I've omitted one word here. It should read

Right now we never go to when compared tree descriptors are _both_ 
infinity,...

i.e. right now we never call tree_entry_pathcmp with both t1 and t2
being empty.


  as
  this condition is checked in the loop beginning as finishing criteria,
 
 What condition and which loop?  The loop that immediately surrounds
 the callsite of tree_entry_pathcmp() is the infinite for (;;) { loop,
 and after it prepares t1 and t2 by skipping paths outside pathspec,
 we check if both are empty (i.e. we ran out).  Is that the condition
 you are referring to?

Yes exactly. Modulo diff_can_quit_early() logic, we break from loop in
diff_tree (the loop in which special-case diff-tree emitting code was)
when both trees were scanned to the end, i.e.

if (!t1-size  !t2-size)
break;

in other words when both t1 and t2 are +∞.

Because of that, at this stage we will never go into tree_entry_pathcmp
with (+∞,+∞) arguments, which could mean (!t1-size  !t2-size) case
could be unnecessary in tree_entry_pathcmp and should not be coded at
all...

  but will do in the future, when there will be several parents iterated
  simultaneously, and some pair of them would run to the end.

... I was trying to say this case will probably be needed later, and that
it is better to have it for generality.

I hope this should be more clear once that prologue with both included
is not confusing.


  Signed-off-by: Kirill Smelkov k...@mns.spb.ru
  Signed-off-by: Junio C Hamano gits...@pobox.com
  ---
 
  ( re-posting without change )
 
   tree-diff.c | 21 +
   1 file changed, 9 insertions(+), 12 deletions(-)
 
  diff --git a/tree-diff.c b/tree-diff.c
  index cf96ad7..2fd6d0e 100644
  --- a/tree-diff.c
  +++ b/tree-diff.c
  @@ -12,12 +12,19 @@
*
* NOTE files and directories *always* compare differently, even when 
  having
*  the same name - thanks to base_name_compare().
  + *
  + * NOTE empty (=invalid) descriptor(s) take part in comparison as +infty.
 
 The basic idea is very sane.  It is a nice (and obvious---once you
 are told about the trick) and clean restructuring of the code.

Thanks. I was surprised it is seen as a trick, as infinity is very handy
and common concept in many areas and in sorting too.

 
*/
   static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
   {
  struct name_entry *e1, *e2;
  int cmp;
   
  +   if (!t1-size)
  +   return t2-size ? +1 /* +∞  c */  : 0 /* +∞ = +∞ */;
  +   else if (!t2-size)
  +   return -1;  /* c  +∞ */
 
 Where do these c come from?  I somehow feel that these comments
 are making it harder to understand what is going on.

c means some finite

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Kirill Smelkov]

On Mon, Mar 24, 2014 at 02:36:22PM -0700, Junio C Hamano wrote:
 Kirill Smelkov k...@mns.spb.ru writes:
 
  The downside is that try_to_follow_renames(), if active, we cause
  re-reading of 2 initial trees, which was negligible based on my timings,
 
 That would depend on how often the codepath triggered in your test
 case, but is totally understandable.  It fires only when the path we
 have been following disappears from the parent, and the processing
 of try-to-follow itself is very compute-intensive (it needs to run
 find-copies-harder logic) that will end up reading many subtrees of
 the two initial trees; two more reading of tree objects will be
 dwarfed by the actual processing.

I agree and thanks for the explanation.

  and which is outweighed cogently by the upsides.
 
  Changes since v1:
 
   - don't need to touch diff.h, as diff_tree() became static.
 
 Nice.  I wonder if it is an option to let the function keep its name
 diff_tree() without renaming it to __diff_tree_whatever(), though.

As I see it, in Git for functions operating on trees, there is convention
to accept either `struct tree_desc *` and be named simply, or sha1 and
be named with _sha1 suffix. From this point of view for new diff_tree()
accepting sha1's and staying with its old name would be confusing.

Besides, in the end we'll have two function with high-level wrapper, and
lower-lever worker:

- diff_tree_sha1(), and
- diff_tree_paths().

So it's not about this only particular case.  Both do some simple
preparation, call worker, and perform some cleanup.

So the question is how to name the worker?

In Linux they use __ prefix. We could also use some other prefix or
suffix, e.g. _bh (for bottom-half), _worker, _low, _raw, etc...


To me, personally, the cleanest is __ prefix, but maybe I'm too used
to Linux etc... I'm open to other naming scheme, only it should be
consistent.

What are the downsides of __ prefix by the way?


   tree-diff.c | 60 
  
   1 file changed, 28 insertions(+), 32 deletions(-)
 
  diff --git a/tree-diff.c b/tree-diff.c
  index b99622c..f90acf5 100644
  --- a/tree-diff.c
  +++ b/tree-diff.c
  @@ -137,12 +137,17 @@ static void skip_uninteresting(struct tree_desc *t, 
  struct strbuf *base,
  }
   }
   
  -static int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
  -const char *base_str, struct diff_options *opt)
  +static int __diff_tree_sha1(const unsigned char *old, const unsigned char 
  *new,
  +   const char *base_str, struct diff_options *opt)
   {
  +   struct tree_desc t1, t2;
  +   void *t1tree, *t2tree;
  struct strbuf base;
  int baselen = strlen(base_str);
   
  +   t1tree = fill_tree_descriptor(t1, old);
  +   t2tree = fill_tree_descriptor(t2, new);
  +
  /* Enable recursion indefinitely */
  opt-pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
   
  @@ -155,39 +160,41 @@ static int diff_tree(struct tree_desc *t1, struct 
  tree_desc *t2,
  if (diff_can_quit_early(opt))
  break;
  if (opt-pathspec.nr) {
  -   skip_uninteresting(t1, base, opt);
  -   skip_uninteresting(t2, base, opt);
  +   skip_uninteresting(t1, base, opt);
  +   skip_uninteresting(t2, base, opt);
  }
  -   if (!t1-size  !t2-size)
  +   if (!t1.size  !t2.size)
  break;
   
  -   cmp = tree_entry_pathcmp(t1, t2);
  +   cmp = tree_entry_pathcmp(t1, t2);
   
  /* t1 = t2 */
  if (cmp == 0) {
  if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
  -   hashcmp(t1-entry.sha1, t2-entry.sha1) ||
  -   (t1-entry.mode != t2-entry.mode))
  -   show_path(base, opt, t1, t2);
  +   hashcmp(t1.entry.sha1, t2.entry.sha1) ||
  +   (t1.entry.mode != t2.entry.mode))
  +   show_path(base, opt, t1, t2);
   
  -   update_tree_entry(t1);
  -   update_tree_entry(t2);
  +   update_tree_entry(t1);
  +   update_tree_entry(t2);
  }
   
  /* t1  t2 */
  else if (cmp  0) {
  -   show_path(base, opt, t1, /*t2=*/NULL);
  -   update_tree_entry(t1);
  +   show_path(base, opt, t1, /*t2=*/NULL);
  +   update_tree_entry(t1);
  }
   
  /* t1  t2 */
  else {
  -   show_path(base, opt, /*t1=*/NULL, t2);
  -   update_tree_entry(t2);
  +   show_path(base, opt, /*t1=*/NULL, t2);
  +   update_tree_entry(t2);
  }
  }
   
  strbuf_release(base);
  +   free(t2tree);
  +   free(t1tree);
  return 0;
   }
   
  @@ -202,7 +209,7

Re: [PATCH 11/19] tree-diff: simplify tree_entry_pathcmp

[Kirill Smelkov]

On Mon, Mar 24, 2014 at 02:25:04PM -0700, Junio C Hamano wrote:
 Kirill Smelkov k...@mns.spb.ru writes:
 
  Since an earlier Finally switch over tree descriptors to contain a
  pre-parsed entry, we can safely access all tree_desc-entry fields
  directly instead of first extracting them through
  tree_entry_extract.
 
  Use it. The code generated stays the same - only it now visually looks
  cleaner.
 
  Signed-off-by: Kirill Smelkov k...@mns.spb.ru
  Signed-off-by: Junio C Hamano gits...@pobox.com
  ---
 
  ( re-posting without change )
 
 Thanks.
 
 Hopefully I'll be merging the series up to this point to 'next'
 soonish.

Thanks a lot!


   tree-diff.c | 17 ++---
   1 file changed, 6 insertions(+), 11 deletions(-)
 
  diff --git a/tree-diff.c b/tree-diff.c
  index 20a4fda..cf96ad7 100644
  --- a/tree-diff.c
  +++ b/tree-diff.c
  @@ -15,18 +15,13 @@
*/
   static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
   {
  -   unsigned mode1, mode2;
  -   const char *path1, *path2;
  -   const unsigned char *sha1, *sha2;
  -   int cmp, pathlen1, pathlen2;
  +   struct name_entry *e1, *e2;
  +   int cmp;
   
  -   sha1 = tree_entry_extract(t1, path1, mode1);
  -   sha2 = tree_entry_extract(t2, path2, mode2);
  -
  -   pathlen1 = tree_entry_len(t1-entry);
  -   pathlen2 = tree_entry_len(t2-entry);
  -
  -   cmp = base_name_compare(path1, pathlen1, mode1, path2, pathlen2, mode2);
  +   e1 = t1-entry;
  +   e2 = t2-entry;
  +   cmp = base_name_compare(e1-path, tree_entry_len(e1), e1-mode,
  +   e2-path, tree_entry_len(e2), e2-mode);
  return cmp;
   }
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 16/19] tree-diff: reuse base str(buf) memory on sub-tree recursion

[Kirill Smelkov]

On Mon, Mar 24, 2014 at 02:43:36PM -0700, Junio C Hamano wrote:
 Kirill Smelkov k...@mns.spb.ru writes:
 
  instead of allocating it all the time for every subtree in
  __diff_tree_sha1, let's allocate it once in diff_tree_sha1, and then all
  callee just use it in stacking style, without memory allocations.
 
  This should be faster, and for me this change gives the following
  slight speedups for
 
  git log --raw --no-abbrev --no-renames --format='%H'
 
  navy.gitlinux.git v3.10..v3.11
 
  before  0.618s  1.903s
  after   0.611s  1.889s
  speedup 1.1%0.7%
 
  Signed-off-by: Kirill Smelkov k...@mns.spb.ru
  ---
 
  Changes since v1:
 
   - don't need to touch diff.h, as the function we are changing became 
  static.
 
   tree-diff.c | 36 ++--
   1 file changed, 18 insertions(+), 18 deletions(-)
 
  diff --git a/tree-diff.c b/tree-diff.c
  index aea0297..c76821d 100644
  --- a/tree-diff.c
  +++ b/tree-diff.c
  @@ -115,7 +115,7 @@ static void show_path(struct strbuf *base, struct 
  diff_options *opt,
  if (recurse) {
  strbuf_addch(base, '/');
  __diff_tree_sha1(t1 ? t1-entry.sha1 : NULL,
  -t2 ? t2-entry.sha1 : NULL, base-buf, opt);
  +t2 ? t2-entry.sha1 : NULL, base, opt);
  }
   
  strbuf_setlen(base, old_baselen);
 
 I was scratching my head for a while, after seeing that there does
 not seem to be any *new* code added by this patch in order to
 store-away the original length and restore the singleton base buffer
 to the original length after using addch/addstr to extend it.
 
 But I see that the code has already been prepared to do this
 conversion.  I wonder why we didn't do this earlier ;-)

The conversion to reusing memory started in 48932677 diff-tree: convert
base+baselen to writable strbuf which allowed to avoid quite a bit of
malloc() and memcpy(), but for this to work allocation at diff_tree()
entry had to be there.

In particular it had to be there, because diff_tree() accepted base as C
string, not strbuf, and since diff_tree() was calling itself
recursively - oops - new allocation on every subtree.

I've opened the door for avoiding allocations via splitting diff_tree
into high-level and low-level parts. The high-level part still accepts
`char *base`, but low-level function operates on strbuf and recurses
into low-level self.

The high-level diff_tree_sha1() still allocates memory for every
diff(tree1,tree2), but that is significantly lower compared to
allocating memory on every subtree...

The lesson here is: better use strbuf for api unless there is a reason
not to.


 Looks good.  Thanks.

Thanks.

  @@ -138,12 +138,10 @@ static void skip_uninteresting(struct tree_desc *t, 
  struct strbuf *base,
   }
   
   static int __diff_tree_sha1(const unsigned char *old, const unsigned char 
  *new,
  -   const char *base_str, struct diff_options *opt)
  +   struct strbuf *base, struct diff_options *opt)
   {
  struct tree_desc t1, t2;
  void *t1tree, *t2tree;
  -   struct strbuf base;
  -   int baselen = strlen(base_str);
   
  t1tree = fill_tree_descriptor(t1, old);
  t2tree = fill_tree_descriptor(t2, new);
  @@ -151,17 +149,14 @@ static int __diff_tree_sha1(const unsigned char *old, 
  const unsigned char *new,
  /* Enable recursion indefinitely */
  opt-pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
   
  -   strbuf_init(base, PATH_MAX);
  -   strbuf_add(base, base_str, baselen);
  -
  for (;;) {
  int cmp;
   
  if (diff_can_quit_early(opt))
  break;
  if (opt-pathspec.nr) {
  -   skip_uninteresting(t1, base, opt);
  -   skip_uninteresting(t2, base, opt);
  +   skip_uninteresting(t1, base, opt);
  +   skip_uninteresting(t2, base, opt);
  }
  if (!t1.size  !t2.size)
  break;
  @@ -173,7 +168,7 @@ static int __diff_tree_sha1(const unsigned char *old, 
  const unsigned char *new,
  if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
  hashcmp(t1.entry.sha1, t2.entry.sha1) ||
  (t1.entry.mode != t2.entry.mode))
  -   show_path(base, opt, t1, t2);
  +   show_path(base, opt, t1, t2);
   
  update_tree_entry(t1);
  update_tree_entry(t2);
  @@ -181,18 +176,17 @@ static int __diff_tree_sha1(const unsigned char *old, 
  const unsigned char *new,
   
  /* t1  t2 */
  else if (cmp  0) {
  -   show_path(base, opt, t1, /*t2=*/NULL);
  +   show_path(base, opt, t1, /*t2=*/NULL);
  update_tree_entry(t1);
  }
   
  /* t1  t2 */
  else

Re: [PATCH 17/19] Portable alloca for Git

[Kirill Smelkov]

On Fri, Feb 28, 2014 at 06:19:58PM +0100, Erik Faye-Lund wrote:
 On Fri, Feb 28, 2014 at 6:00 PM, Kirill Smelkov k...@mns.spb.ru wrote:
  On Fri, Feb 28, 2014 at 02:50:04PM +0100, Erik Faye-Lund wrote:
  On Fri, Feb 28, 2014 at 2:44 PM, Erik Faye-Lund kusmab...@gmail.com 
  wrote:
   On Mon, Feb 24, 2014 at 5:21 PM, Kirill Smelkov k...@mns.spb.ru wrote:
   diff --git a/Makefile b/Makefile
   index dddaf4f..0334806 100644
   --- a/Makefile
   +++ b/Makefile
   @@ -316,6 +321,7 @@ endif
ifeq ($(uname_S),Windows)
   GIT_VERSION := $(GIT_VERSION).MSVC
   pathsep = ;
   +   HAVE_ALLOCA_H = YesPlease
   NO_PREAD = YesPlease
   NEEDS_CRYPTO_WITH_SSL = YesPlease
   NO_LIBGEN_H = YesPlease
  
   In MSVC, alloca is defined in in malloc.h, not alloca.h:
  
   http://msdn.microsoft.com/en-us/library/wb1s57t5.aspx
  
   In fact, it has no alloca.h at all. But we don't have malloca.h in
   mingw either, so creating a compat/win32/alloca.h that includes
   malloc.h is probably sufficient.
 
  But we don't have alloca.h in mingw either, sorry.
 
  Don't we have that for MSVC already in
 
  compat/vcbuild/include/alloca.h
 
  and
 
  ifeq ($(uname_S),Windows)
  ...
  BASIC_CFLAGS = ... -Icompat/vcbuild/include ...
 
 
  in config.mak.uname ?
 
 Ah, of course. Thanks for setting me straight!
 
  And as I've not touched MINGW part in config.mak.uname the patch stays
  valid as it is :) and we can incrementally update what platforms have
  working alloca with follow-up patches.
 
  In fact that would be maybe preferred, for maintainers to enable alloca
  with knowledge and testing, as one person can't have them all at hand.
 
 Yeah, you're probably right.

Erik, the patch has been merged into pu today. Would you please
follow-up with tested MINGW change?

Thanks beforehand,
Kirill
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 17/19] Portable alloca for Git

[Kirill Smelkov]

On Fri, Feb 28, 2014 at 02:50:04PM +0100, Erik Faye-Lund wrote:
 On Fri, Feb 28, 2014 at 2:44 PM, Erik Faye-Lund kusmab...@gmail.com wrote:
  On Mon, Feb 24, 2014 at 5:21 PM, Kirill Smelkov k...@mns.spb.ru wrote:
  diff --git a/Makefile b/Makefile
  index dddaf4f..0334806 100644
  --- a/Makefile
  +++ b/Makefile
  @@ -316,6 +321,7 @@ endif
   ifeq ($(uname_S),Windows)
  GIT_VERSION := $(GIT_VERSION).MSVC
  pathsep = ;
  +   HAVE_ALLOCA_H = YesPlease
  NO_PREAD = YesPlease
  NEEDS_CRYPTO_WITH_SSL = YesPlease
  NO_LIBGEN_H = YesPlease
 
  In MSVC, alloca is defined in in malloc.h, not alloca.h:
 
  http://msdn.microsoft.com/en-us/library/wb1s57t5.aspx
 
  In fact, it has no alloca.h at all. But we don't have malloca.h in
  mingw either, so creating a compat/win32/alloca.h that includes
  malloc.h is probably sufficient.
 
 But we don't have alloca.h in mingw either, sorry.

Don't we have that for MSVC already in

compat/vcbuild/include/alloca.h

and

ifeq ($(uname_S),Windows)
...
BASIC_CFLAGS = ... -Icompat/vcbuild/include ...


in config.mak.uname ?

And as I've not touched MINGW part in config.mak.uname the patch stays
valid as it is :) and we can incrementally update what platforms have
working alloca with follow-up patches.

In fact that would be maybe preferred, for maintainers to enable alloca
with knowledge and testing, as one person can't have them all at hand.

Thanks,
Kirill
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 00/19] Multiparent diff tree-walker + combine-diff speedup

[Kirill Smelkov]

On Tue, Feb 25, 2014 at 06:43:24AM +0700, Duy Nguyen wrote:
 On Mon, Feb 24, 2014 at 11:21 PM, Kirill Smelkov k...@mns.spb.ru wrote:
  Hello up there.
 
  Here go combine-diff speedup patches in form of first reworking diff
  tree-walker to work in general case - when a commit have several parents, 
  not
  only one - we are traversing all 1+nparent trees in parallel.
 
  Then we are taking advantage of the new diff tree-walker for speeding up
  combine-diff, which for linux.git results in ~14 times speedup.
 
 I think there is another use case for this n-tree walker (but I'm not
 entirely sure yet as I haven't really read the series). In git-log
 (either with pathspec or --patch) we basically do this
 
 diff HEAD^ HEAD
 diff HEAD^^ HEAD^
 diff HEAD^^^ HEAD^^
 diff HEAD HEAD^^^
 ...
 
 so except HEAD (and the last commit), all commits' tree will be
 read/diff'd twice. With n-tree walker I think we may be able to diff
 them in batch to reduce extra processing: commit lists are split into
 16-commit blocks where 16 trees are fed to the new tree walker at the
 same time. I hope it would make git-log a bit faster (especially for
 -S). Maybe not much.

Thanks for commenting.

Unfortunately, as it is now, no, and I doubt savings will be
significant. The real speedup comes from the fact that for combined
diff, we can omit recursing into subdirectories, if we know some diff
D(commit,parent_i) is empty. Let me quote myself from

http://article.gmane.org/gmane.comp.version-control.git/242217

On Sun, Feb 16, 2014 at 12:08:29PM +0400, Kirill Smelkov wrote:
 On Fri, Feb 14, 2014 at 09:37:00AM -0800, Junio C Hamano wrote:
  I wonder if this machinery can be reused for log -m as well (or
  perhaps you do that already?).  After all, by performing a single
  parallel scan, you are gathering all the necessary information to
  let you pretend that you did N pairwise diff-tree.
 
 Unfortunately, as it is now, no, and let me explain why:
 
 The reason that is not true, is that we omit recursing into directories,
 if we know D(A,some-parent) for that path is empty. That means we don't
 calculate D(A,any-other-parents) for that path and subpaths.
 
 More structured description is that combined diff and log -m, which
 could be though as all diffs D(A,Pi) are different things:
 
 - the combined diff is D(A,B) generalization based on ^ (sets
   intersection) operator, and
 
 - log -m, aka all diffs is D(A,B) generalization based on v
   (sets union) operator.
 
 Intersection means, we can omit calculating parts from other sets, if we
 know some set does not have an element (remember don't recurse into
 subdirectories?), and unioning does not have this property.
 
 It does so happen, that ^ case (combine-diff) is more interesting,
 because in the end it allows to see new information - the diff a merge
 itself introduces. log -m does not have this property and is no more
 interesting to what plain diff(HEAD,HEAD^n) can provide - in other words
 it's just a convenience.
 
 Now, the diff tree-walker could be generalized once more, to allow
 clients specify, which diffs combination operator to use - intersection
 or unioning, but I doubt that for unioning case that would add
 significant speedup - we can't reduce any diff generation based on
 another diff and the only saving is that we traverse resulting commit
 tree once, but for some cases that could be maybe slower, say if result
 and some parents don't have a path and some parent does, we'll be
 recursing into that path and do more work compared to plain D(A,Pi) for
 Pi that lacks the path.
 
 In short: it could be generalized more, if needed, but I propose we
 first establish the ground with generalizing to just combine-diff.

besides

D(HEAD~,  HEAD)
D(HEAD~2, HEAD~)
...
D(HEAD~{n}, HEAD~{n-1})

is different even from log -m case as now there is no single commit
with several parents.

On a related note, while developing this n-tree walker, I've learned
that it is important to load trees in correct order. Quoting patch 18:

-   t1tree = fill_tree_descriptor(t1, old);
-   t2tree = fill_tree_descriptor(t2, new);
+   /*
+* load parents first, as they are probably already cached.
+*
+* ( log_tree_diff() parses commit-parent before calling here via
+*   diff_tree_sha1(parent, commit) )
+*/
+   for (i = 0; i  nparent; ++i)
+   tptree[i] = fill_tree_descriptor(tp[i], parents_sha1[i]);
+   ttree = fill_tree_descriptor(t, sha1);

so it loads parent's tree first. If we change this to be the other way,
i.e. load commit's tree first, and then parent's tree, there will be up
to 4% slowdown for whole plain `git log` (without -c).

So maybe what could be done to speedup plain log is for diff tree-walker
to populate some form of recently-loaded trees while walking, and drop
trees from will not-be used anymore commits - e.g. after doing
HEAD~..HEAD for next diff for HEAD~~..HEAD~ HEAD

[PATCH 02/19] combine-diff: move changed-paths scanning logic into its own function

[Kirill Smelkov]

Move code for finding paths for which diff(commit,parent_i) is not-empty
for all parents to separate function - at present we have generic (and
slow) code for this job, which translates 1 n-parent problem to n
1-parent problems and then intersect results, and will be adding another
limited, but faster, paths scanning implementation in the next patch.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
Signed-off-by: Junio C Hamano gits...@pobox.com
---

( re-posting without change )

 combine-diff.c | 80 ++
 1 file changed, 53 insertions(+), 27 deletions(-)

diff --git a/combine-diff.c b/combine-diff.c
index 68d2e53..1732dfd 100644
--- a/combine-diff.c
+++ b/combine-diff.c
@@ -1301,6 +1301,51 @@ static const char *path_path(void *obj)
return path-path;
 }
 
+
+/* find set of paths that every parent touches */
+static struct combine_diff_path *find_paths(const unsigned char *sha1,
+   const struct sha1_array *parents, struct diff_options *opt)
+{
+   struct combine_diff_path *paths = NULL;
+   int i, num_parent = parents-nr;
+
+   int output_format = opt-output_format;
+   const char *orderfile = opt-orderfile;
+
+   opt-output_format = DIFF_FORMAT_NO_OUTPUT;
+   /* tell diff_tree to emit paths in sorted (=tree) order */
+   opt-orderfile = NULL;
+
+   for (i = 0; i  num_parent; i++) {
+   /*
+* show stat against the first parent even when doing
+* combined diff.
+*/
+   int stat_opt = (output_format 
+   (DIFF_FORMAT_NUMSTAT|DIFF_FORMAT_DIFFSTAT));
+   if (i == 0  stat_opt)
+   opt-output_format = stat_opt;
+   else
+   opt-output_format = DIFF_FORMAT_NO_OUTPUT;
+   diff_tree_sha1(parents-sha1[i], sha1, , opt);
+   diffcore_std(opt);
+   paths = intersect_paths(paths, i, num_parent);
+
+   /* if showing diff, show it in requested order */
+   if (opt-output_format != DIFF_FORMAT_NO_OUTPUT 
+   orderfile) {
+   diffcore_order(orderfile);
+   }
+
+   diff_flush(opt);
+   }
+
+   opt-output_format = output_format;
+   opt-orderfile = orderfile;
+   return paths;
+}
+
+
 void diff_tree_combined(const unsigned char *sha1,
const struct sha1_array *parents,
int dense,
@@ -1308,7 +1353,7 @@ void diff_tree_combined(const unsigned char *sha1,
 {
struct diff_options *opt = rev-diffopt;
struct diff_options diffopts;
-   struct combine_diff_path *p, *paths = NULL;
+   struct combine_diff_path *p, *paths;
int i, num_paths, needsep, show_log_first, num_parent = parents-nr;
 
/* nothing to do, if no parents */
@@ -1327,35 +1372,16 @@ void diff_tree_combined(const unsigned char *sha1,
 
diffopts = *opt;
copy_pathspec(diffopts.pathspec, opt-pathspec);
-   diffopts.output_format = DIFF_FORMAT_NO_OUTPUT;
DIFF_OPT_SET(diffopts, RECURSIVE);
DIFF_OPT_CLR(diffopts, ALLOW_EXTERNAL);
-   /* tell diff_tree to emit paths in sorted (=tree) order */
-   diffopts.orderfile = NULL;
 
-   /* find set of paths that everybody touches */
-   for (i = 0; i  num_parent; i++) {
-   /* show stat against the first parent even
-* when doing combined diff.
-*/
-   int stat_opt = (opt-output_format 
-   (DIFF_FORMAT_NUMSTAT|DIFF_FORMAT_DIFFSTAT));
-   if (i == 0  stat_opt)
-   diffopts.output_format = stat_opt;
-   else
-   diffopts.output_format = DIFF_FORMAT_NO_OUTPUT;
-   diff_tree_sha1(parents-sha1[i], sha1, , diffopts);
-   diffcore_std(diffopts);
-   paths = intersect_paths(paths, i, num_parent);
-
-   /* if showing diff, show it in requested order */
-   if (diffopts.output_format != DIFF_FORMAT_NO_OUTPUT 
-   opt-orderfile) {
-   diffcore_order(opt-orderfile);
-   }
-
-   diff_flush(diffopts);
-   }
+   /* find set of paths that everybody touches
+*
+* NOTE find_paths() also handles --stat, as it computes
+* diff(sha1,parent_i) for all i to do the job, specifically
+* for parent0.
+*/
+   paths = find_paths(sha1, parents, diffopts);
 
/* find out number of surviving paths */
for (num_paths = 0, p = paths; p; p = p-next)
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 00/19] Multiparent diff tree-walker + combine-diff speedup

[Kirill Smelkov]

Hello up there.

Here go combine-diff speedup patches in form of first reworking diff
tree-walker to work in general case - when a commit have several parents, not
only one - we are traversing all 1+nparent trees in parallel.

Then we are taking advantage of the new diff tree-walker for speeding up
combine-diff, which for linux.git results in ~14 times speedup.

This is the second posting for the whole series - sent here patches should go
instead of already-in-pu ks/diff-tree-more and ks/tree-diff-nway into
ks/tree-diff-nway - patches are related and seeing them all at once is more
logical to me.

I've tried to do my homework based on review feedback and the changes compared
to v1 are:

- fixed last-minute thinko/bug last time introduced on my side (sorry) with
  opt-pathchange manipulation in __diff_tree_sha1() - we were forgetting to
  restore opt-pathchange, which led to incorrect log -c (merges _and_ plain
  diff-tree) output;

  This time, I've verified several times, log output stays really the same.

- direct use of alloca() changed to portability wrappers xalloca/xalloca_free
  which gracefully degrade to xmalloc/free on systems, where alloca is not
  available (see new patch 17).

- i = 0; do { ... } while (++i  nparent) is back to usual looping
  for (i = 0; i  nparent; ++), as I've re-measured timings and the
  difference is negligible.

  ( Initially, when I was fighting for every cycle it made sense, but real
no-slowdown turned out to be related to avoiding mallocs, load trees in 
correct
order and reducing register pressure. )

- S_IFXMIN_NEQ definition moved out to cache.h, to have all modes registry in 
one place;


- diff_tree() becomes static (new patch 13), as nobody is using it outside
  tree-diff.c (and is later renamed to __diff_tree_sha1);

- p0 - first_parent; corrected comments about how emit_diff_first_parent_only
  behaves;


not changed:

- low-level helpers are still named with __ prefix as, imho, that is the best
  convention to name such helpers, without sacrificing signal/noise ratio. All
  of them are now static though.


Signoffs were left intact, if a patch was already applied to pu with one, and
had not changed.

Please apply and thanks,
Kirill

P.S. Sorry for the delay - I was very busy.


Kirill Smelkov (19):
  combine-diff: move show_log_first logic/action out of paths scanning
  combine-diff: move changed-paths scanning logic into its own function
  tree-diff: no need to manually verify that there is no mode change for a path
  tree-diff: no need to pass match to skip_uninteresting()
  tree-diff: show_tree() is not needed
  tree-diff: consolidate code for emitting diffs and recursion in one place
  tree-diff: don't assume compare_tree_entry() returns -1,0,1
  tree-diff: move all action-taking code out of compare_tree_entry()
  tree-diff: rename compare_tree_entry - tree_entry_pathcmp
  tree-diff: show_path prototype is not needed anymore
  tree-diff: simplify tree_entry_pathcmp
  tree-diff: remove special-case diff-emitting code for empty-tree cases
  tree-diff: diff_tree() should now be static
  tree-diff: rework diff_tree interface to be sha1 based
  tree-diff: no need to call full diff_tree_sha1 from show_path()
  tree-diff: reuse base str(buf) memory on sub-tree recursion
  Portable alloca for Git
  tree-diff: rework diff_tree() to generate diffs for multiparent cases as well
  combine-diff: speed it up, by using multiparent diff tree-walker directly

 Makefile  |   6 +
 cache.h   |  15 ++
 combine-diff.c| 170 +++---
 config.mak.uname  |  10 +-
 configure.ac  |   8 +
 diff.c|   2 +
 diff.h|  12 +-
 git-compat-util.h |   8 +
 tree-diff.c   | 666 +++---
 9 files changed, 724 insertions(+), 173 deletions(-)

-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 01/19] combine-diff: move show_log_first logic/action out of paths scanning

[Kirill Smelkov]

Judging from sample outputs and tests nothing changes in diff -c output,
and this change will help later patches, when we'll be refactoring paths
scanning into its own function with several variants - the
show_log_first logic / code will stay common to all of them.

NOTE: only now we have to take care to explicitly not show anything if
parents array is empty, as in fact there are some clients in Git code,
which calls diff_tree_combined() in such a way.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
Signed-off-by: Junio C Hamano gits...@pobox.com
---

( re-posting without change )

 combine-diff.c | 24 ++--
 1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/combine-diff.c b/combine-diff.c
index 24ca7e2..68d2e53 100644
--- a/combine-diff.c
+++ b/combine-diff.c
@@ -1311,6 +1311,20 @@ void diff_tree_combined(const unsigned char *sha1,
struct combine_diff_path *p, *paths = NULL;
int i, num_paths, needsep, show_log_first, num_parent = parents-nr;
 
+   /* nothing to do, if no parents */
+   if (!num_parent)
+   return;
+
+   show_log_first = !!rev-loginfo  !rev-no_commit_id;
+   needsep = 0;
+   if (show_log_first) {
+   show_log(rev);
+
+   if (rev-verbose_header  opt-output_format)
+   printf(%s%c, diff_line_prefix(opt),
+  opt-line_termination);
+   }
+
diffopts = *opt;
copy_pathspec(diffopts.pathspec, opt-pathspec);
diffopts.output_format = DIFF_FORMAT_NO_OUTPUT;
@@ -1319,8 +1333,6 @@ void diff_tree_combined(const unsigned char *sha1,
/* tell diff_tree to emit paths in sorted (=tree) order */
diffopts.orderfile = NULL;
 
-   show_log_first = !!rev-loginfo  !rev-no_commit_id;
-   needsep = 0;
/* find set of paths that everybody touches */
for (i = 0; i  num_parent; i++) {
/* show stat against the first parent even
@@ -1336,14 +1348,6 @@ void diff_tree_combined(const unsigned char *sha1,
diffcore_std(diffopts);
paths = intersect_paths(paths, i, num_parent);
 
-   if (show_log_first  i == 0) {
-   show_log(rev);
-
-   if (rev-verbose_header  opt-output_format)
-   printf(%s%c, diff_line_prefix(opt),
-  opt-line_termination);
-   }
-
/* if showing diff, show it in requested order */
if (diffopts.output_format != DIFF_FORMAT_NO_OUTPUT 
opt-orderfile) {
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 13/19] tree-diff: diff_tree() should now be static

[Kirill Smelkov]

We reworked all its users to use the functionality through
diff_tree_sha1 variant in recent patches (see tree-diff: allow
diff_tree_sha1 to accept NULL sha1 and what comes next).

diff_tree() is now not used outside tree-diff.c - make it static.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---

 ( new patch )

 diff.h  | 2 --
 tree-diff.c | 4 ++--
 2 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/diff.h b/diff.h
index e79f3b3..5d7b9f7 100644
--- a/diff.h
+++ b/diff.h
@@ -189,8 +189,6 @@ const char *diff_line_prefix(struct diff_options *);
 
 extern const char mime_boundary_leader[];
 
-extern int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
-const char *base, struct diff_options *opt);
 extern int diff_tree_sha1(const unsigned char *old, const unsigned char *new,
  const char *base, struct diff_options *opt);
 extern int diff_root_tree_sha1(const unsigned char *new, const char *base,
diff --git a/tree-diff.c b/tree-diff.c
index 2fd6d0e..b99622c 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -137,8 +137,8 @@ static void skip_uninteresting(struct tree_desc *t, struct 
strbuf *base,
}
 }
 
-int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
- const char *base_str, struct diff_options *opt)
+static int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
+const char *base_str, struct diff_options *opt)
 {
struct strbuf base;
int baselen = strlen(base_str);
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 09/19] tree-diff: rename compare_tree_entry - tree_entry_pathcmp

[Kirill Smelkov]

Since previous commit, this function does not compare entry hashes, and
mode are compared fully outside of it. So what it does is compare entry
names and DIR bit in modes. Reflect this in its name.

Add documentation stating the semantics, and move the note about
files/dirs comparison to it.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
Signed-off-by: Junio C Hamano gits...@pobox.com
---

( re-posting without change )

 tree-diff.c | 15 +--
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 6207372..3345534 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -9,7 +9,14 @@
 static void show_path(struct strbuf *base, struct diff_options *opt,
  struct tree_desc *t1, struct tree_desc *t2);
 
-static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2)
+/*
+ * Compare two tree entries, taking into account only path/S_ISDIR(mode),
+ * but not their sha1's.
+ *
+ * NOTE files and directories *always* compare differently, even when having
+ *  the same name - thanks to base_name_compare().
+ */
+static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
 {
unsigned mode1, mode2;
const char *path1, *path2;
@@ -22,10 +29,6 @@ static int compare_tree_entry(struct tree_desc *t1, struct 
tree_desc *t2)
pathlen1 = tree_entry_len(t1-entry);
pathlen2 = tree_entry_len(t2-entry);
 
-   /*
-* NOTE files and directories *always* compare differently,
-* even when having the same name.
-*/
cmp = base_name_compare(path1, pathlen1, mode1, path2, pathlen2, mode2);
return cmp;
 }
@@ -169,7 +172,7 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
continue;
}
 
-   cmp = compare_tree_entry(t1, t2);
+   cmp = tree_entry_pathcmp(t1, t2);
 
/* t1 = t2 */
if (cmp == 0) {
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 11/19] tree-diff: simplify tree_entry_pathcmp

[Kirill Smelkov]

Since an earlier Finally switch over tree descriptors to contain a
pre-parsed entry, we can safely access all tree_desc-entry fields
directly instead of first extracting them through
tree_entry_extract.

Use it. The code generated stays the same - only it now visually looks
cleaner.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
Signed-off-by: Junio C Hamano gits...@pobox.com
---

( re-posting without change )

 tree-diff.c | 17 ++---
 1 file changed, 6 insertions(+), 11 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 20a4fda..cf96ad7 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -15,18 +15,13 @@
  */
 static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
 {
-   unsigned mode1, mode2;
-   const char *path1, *path2;
-   const unsigned char *sha1, *sha2;
-   int cmp, pathlen1, pathlen2;
+   struct name_entry *e1, *e2;
+   int cmp;
 
-   sha1 = tree_entry_extract(t1, path1, mode1);
-   sha2 = tree_entry_extract(t2, path2, mode2);
-
-   pathlen1 = tree_entry_len(t1-entry);
-   pathlen2 = tree_entry_len(t2-entry);
-
-   cmp = base_name_compare(path1, pathlen1, mode1, path2, pathlen2, mode2);
+   e1 = t1-entry;
+   e2 = t2-entry;
+   cmp = base_name_compare(e1-path, tree_entry_len(e1), e1-mode,
+   e2-path, tree_entry_len(e2), e2-mode);
return cmp;
 }
 
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 06/19] tree-diff: consolidate code for emitting diffs and recursion in one place

[Kirill Smelkov]

Currently both compare_tree_entry() and show_path() invoke opt diff
callbacks (opt-add_remove() and opt-change()), and also they both have
code which decides whether to recurse into sub-tree, and whether to emit
a tree as separate entry if DIFF_OPT_TREE_IN_RECURSIVE is set.

I.e. we have code duplication and logic scattered on two places.

Let's consolidate it - all diff emmiting code and recurion logic moves
to show_entry, which is now named as show_path, because it shows diff
for a path, based on up to two tree entries, with actual diff emitting
code being kept in new helper emit_diff() for clarity.

What we have as the result, is that compare_tree_entry is now free from
code with logic for diff generation, and also performance is not
affected as timings for

`git log --raw --no-abbrev --no-renames`

for navy.git and `linux.git v3.10..v3.11`, just like in previous patch,
stay the same.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
Signed-off-by: Junio C Hamano gits...@pobox.com
---

( re-posting without change )

 tree-diff.c | 115 
 1 file changed, 84 insertions(+), 31 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 2ad7788..a5b9ff9 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -6,8 +6,8 @@
 #include diffcore.h
 #include tree.h
 
-static void show_entry(struct diff_options *opt, const char *prefix,
-  struct tree_desc *desc, struct strbuf *base);
+static void show_path(struct strbuf *base, struct diff_options *opt,
+ struct tree_desc *t1, struct tree_desc *t2);
 
 static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2,
  struct strbuf *base, struct diff_options *opt)
@@ -16,7 +16,6 @@ static int compare_tree_entry(struct tree_desc *t1, struct 
tree_desc *t2,
const char *path1, *path2;
const unsigned char *sha1, *sha2;
int cmp, pathlen1, pathlen2;
-   int old_baselen = base-len;
 
sha1 = tree_entry_extract(t1, path1, mode1);
sha2 = tree_entry_extract(t2, path2, mode2);
@@ -30,51 +29,105 @@ static int compare_tree_entry(struct tree_desc *t1, struct 
tree_desc *t2,
 */
cmp = base_name_compare(path1, pathlen1, mode1, path2, pathlen2, mode2);
if (cmp  0) {
-   show_entry(opt, -, t1, base);
+   show_path(base, opt, t1, /*t2=*/NULL);
return -1;
}
if (cmp  0) {
-   show_entry(opt, +, t2, base);
+   show_path(base, opt, /*t1=*/NULL, t2);
return 1;
}
if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)  !hashcmp(sha1, sha2)  
mode1 == mode2)
return 0;
 
-   strbuf_add(base, path1, pathlen1);
-   if (DIFF_OPT_TST(opt, RECURSIVE)  S_ISDIR(mode1)) {
-   if (DIFF_OPT_TST(opt, TREE_IN_RECURSIVE)) {
-   opt-change(opt, mode1, mode2,
-   sha1, sha2, 1, 1, base-buf, 0, 0);
-   }
-   strbuf_addch(base, '/');
-   diff_tree_sha1(sha1, sha2, base-buf, opt);
-   } else {
-   opt-change(opt, mode1, mode2, sha1, sha2, 1, 1, base-buf, 0, 
0);
-   }
-   strbuf_setlen(base, old_baselen);
+   show_path(base, opt, t1, t2);
return 0;
 }
 
-/* An entry went away or appeared */
-static void show_entry(struct diff_options *opt, const char *prefix,
-  struct tree_desc *desc, struct strbuf *base)
+
+/* convert path, t1/t2 - opt-diff_*() callbacks */
+static void emit_diff(struct diff_options *opt, struct strbuf *path,
+ struct tree_desc *t1, struct tree_desc *t2)
+{
+   unsigned int mode1 = t1 ? t1-entry.mode : 0;
+   unsigned int mode2 = t2 ? t2-entry.mode : 0;
+
+   if (mode1  mode2) {
+   opt-change(opt, mode1, mode2, t1-entry.sha1, t2-entry.sha1,
+   1, 1, path-buf, 0, 0);
+   }
+   else {
+   const unsigned char *sha1;
+   unsigned int mode;
+   int addremove;
+
+   if (mode2) {
+   addremove = '+';
+   sha1 = t2-entry.sha1;
+   mode = mode2;
+   }
+   else {
+   addremove = '-';
+   sha1 = t1-entry.sha1;
+   mode = mode1;
+   }
+
+   opt-add_remove(opt, addremove, mode, sha1, 1, path-buf, 0);
+   }
+}
+
+
+/* new path should be added to diff
+ *
+ * 3 cases on how/when it should be called and behaves:
+ *
+ * !t1,  t2- path added, parent lacks it
+ *  t1, !t2- path removed from parent
+ *  t1,  t2- path modified
+ */
+static void show_path(struct strbuf *base, struct diff_options *opt,
+ struct tree_desc *t1, struct tree_desc *t2)
 {
unsigned mode;
const char *path

[PATCH 12/19] tree-diff: remove special-case diff-emitting code for empty-tree cases

[Kirill Smelkov]

via teaching tree_entry_pathcmp() how to compare empty tree descriptors:

While walking trees, we iterate their entries from lowest to highest in
sort order, so empty tree means all entries were already went over.

If we artificially assign +infinity value to such tree entry, it will
go after all usual entries, and through the usual driver loop we will be
taking the same actions, which were hand-coded for special cases, i.e.

t1 empty, t2 non-empty
pathcmp(+∞, t2) - +1
show_path(/*t1=*/NULL, t2); /* = t1  t2 case in main loop */

t1 non-empty, t2-empty
pathcmp(t1, +∞) - -1
show_path(t1, /*t2=*/NULL); /* = t1  t2 case in main loop */

Right now we never go to when compared tree descriptors are infinity, as
this condition is checked in the loop beginning as finishing criteria,
but will do in the future, when there will be several parents iterated
simultaneously, and some pair of them would run to the end.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
Signed-off-by: Junio C Hamano gits...@pobox.com
---

( re-posting without change )

 tree-diff.c | 21 +
 1 file changed, 9 insertions(+), 12 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index cf96ad7..2fd6d0e 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -12,12 +12,19 @@
  *
  * NOTE files and directories *always* compare differently, even when having
  *  the same name - thanks to base_name_compare().
+ *
+ * NOTE empty (=invalid) descriptor(s) take part in comparison as +infty.
  */
 static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
 {
struct name_entry *e1, *e2;
int cmp;
 
+   if (!t1-size)
+   return t2-size ? +1 /* +∞  c */  : 0 /* +∞ = +∞ */;
+   else if (!t2-size)
+   return -1;  /* c  +∞ */
+
e1 = t1-entry;
e2 = t2-entry;
cmp = base_name_compare(e1-path, tree_entry_len(e1), e1-mode,
@@ -151,18 +158,8 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
skip_uninteresting(t1, base, opt);
skip_uninteresting(t2, base, opt);
}
-   if (!t1-size) {
-   if (!t2-size)
-   break;
-   show_path(base, opt, /*t1=*/NULL, t2);
-   update_tree_entry(t2);
-   continue;
-   }
-   if (!t2-size) {
-   show_path(base, opt, t1, /*t2=*/NULL);
-   update_tree_entry(t1);
-   continue;
-   }
+   if (!t1-size  !t2-size)
+   break;
 
cmp = tree_entry_pathcmp(t1, t2);
 
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 04/19] tree-diff: no need to pass match to skip_uninteresting()

[Kirill Smelkov]

It is neither used there as input, nor the output written through it, is
used outside.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
Signed-off-by: Junio C Hamano gits...@pobox.com
---

( re-posting without change )

 tree-diff.c | 17 -
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 5810b00..a8c2aec 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -109,13 +109,14 @@ static void show_entry(struct diff_options *opt, const 
char *prefix,
 }
 
 static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
-  struct diff_options *opt,
-  enum interesting *match)
+  struct diff_options *opt)
 {
+   enum interesting match;
+
while (t-size) {
-   *match = tree_entry_interesting(t-entry, base, 0, 
opt-pathspec);
-   if (*match) {
-   if (*match == all_entries_not_interesting)
+   match = tree_entry_interesting(t-entry, base, 0, 
opt-pathspec);
+   if (match) {
+   if (match == all_entries_not_interesting)
t-size = 0;
break;
}
@@ -128,8 +129,6 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
 {
struct strbuf base;
int baselen = strlen(base_str);
-   enum interesting t1_match = entry_not_interesting;
-   enum interesting t2_match = entry_not_interesting;
 
/* Enable recursion indefinitely */
opt-pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
@@ -141,8 +140,8 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
if (diff_can_quit_early(opt))
break;
if (opt-pathspec.nr) {
-   skip_uninteresting(t1, base, opt, t1_match);
-   skip_uninteresting(t2, base, opt, t2_match);
+   skip_uninteresting(t1, base, opt);
+   skip_uninteresting(t2, base, opt);
}
if (!t1-size) {
if (!t2-size)
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 19/19] combine-diff: speed it up, by using multiparent diff tree-walker directly

[Kirill Smelkov]

As was recently shown in combine-diff: optimize
combine_diff_path sets intersection, combine-diff runs very slowly. In
that commit we optimized paths sets intersection, but that accounted
only for ~ 25% of the slowness, and as my tracing showed, for linux.git
v3.10..v3.11, for merges a lot of time is spent computing
diff(commit,commit^2) just to only then intersect that huge diff to
almost small set of files from diff(commit,commit^1).

In previous commit, we described the problem in more details, and
reworked the diff tree-walker to be general one - i.e. to work in
multiple parent case too. Now is the time to take advantage of it for
finding paths for combine diff.

The implementation is straightforward - if we know, we can get generated
diff paths directly, and at present that means no diff filtering or
rename/copy detection was requested(*), we can call multiparent tree-walker
directly and get ready paths.

(*) because e.g. at present, all diffcore transformations work on
diff_filepair queues, but in the future, that limitation can be
lifted, if filters would operate directly on combine_diff_paths.

Timings for `git log --raw --no-abbrev --no-renames` without `-c` (git log)
and with `-c` (git log -c) and with `-c --merges` (git log -c --merges)
before and after the patch are as follows:

linux.git v3.10..v3.11

log log -c log -c --merges

before  1.9s16.4s  15.2s
after   1.9s 2.4s   1.1s

The result stayed the same.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
Signed-off-by: Junio C Hamano gits...@pobox.com
---

( re-posting without change )

 combine-diff.c | 88 ++
 diff.c |  1 +
 2 files changed, 84 insertions(+), 5 deletions(-)

diff --git a/combine-diff.c b/combine-diff.c
index 1732dfd..12764fb 100644
--- a/combine-diff.c
+++ b/combine-diff.c
@@ -1303,7 +1303,7 @@ static const char *path_path(void *obj)
 
 
 /* find set of paths that every parent touches */
-static struct combine_diff_path *find_paths(const unsigned char *sha1,
+static struct combine_diff_path *find_paths_generic(const unsigned char *sha1,
const struct sha1_array *parents, struct diff_options *opt)
 {
struct combine_diff_path *paths = NULL;
@@ -1316,6 +1316,7 @@ static struct combine_diff_path *find_paths(const 
unsigned char *sha1,
/* tell diff_tree to emit paths in sorted (=tree) order */
opt-orderfile = NULL;
 
+   /* D(A,P1...Pn) = D(A,P1) ^ ... ^ D(A,Pn)  (wrt paths) */
for (i = 0; i  num_parent; i++) {
/*
 * show stat against the first parent even when doing
@@ -1346,6 +1347,35 @@ static struct combine_diff_path *find_paths(const 
unsigned char *sha1,
 }
 
 
+/*
+ * find set of paths that everybody touches, assuming diff is run without
+ * rename/copy detection, etc, comparing all trees simultaneously (= faster).
+ */
+static struct combine_diff_path *find_paths_multitree(
+   const unsigned char *sha1, const struct sha1_array *parents,
+   struct diff_options *opt)
+{
+   int i, nparent = parents-nr;
+   const unsigned char **parents_sha1;
+   struct combine_diff_path paths_head;
+   struct strbuf base;
+
+   parents_sha1 = xmalloc(nparent * sizeof(parents_sha1[0]));
+   for (i = 0; i  nparent; i++)
+   parents_sha1[i] = parents-sha1[i];
+
+   /* fake list head, so worker can assume it is non-NULL */
+   paths_head.next = NULL;
+
+   strbuf_init(base, PATH_MAX);
+   diff_tree_paths(paths_head, sha1, parents_sha1, nparent, base, opt);
+
+   strbuf_release(base);
+   free(parents_sha1);
+   return paths_head.next;
+}
+
+
 void diff_tree_combined(const unsigned char *sha1,
const struct sha1_array *parents,
int dense,
@@ -1355,6 +1385,7 @@ void diff_tree_combined(const unsigned char *sha1,
struct diff_options diffopts;
struct combine_diff_path *p, *paths;
int i, num_paths, needsep, show_log_first, num_parent = parents-nr;
+   int need_generic_pathscan;
 
/* nothing to do, if no parents */
if (!num_parent)
@@ -1377,11 +1408,58 @@ void diff_tree_combined(const unsigned char *sha1,
 
/* find set of paths that everybody touches
 *
-* NOTE find_paths() also handles --stat, as it computes
-* diff(sha1,parent_i) for all i to do the job, specifically
-* for parent0.
+* NOTE
+*
+* Diffcore transformations are bound to diff_filespec and logic
+* comparing two entries - i.e. they do not apply directly to combine
+* diff.
+*
+* If some of such transformations is requested - we launch generic
+* path scanning, which works significantly slower compared to
+* simultaneous all-trees-in-one-go scan in find_paths_multitree().
+*
+* TODO some

[PATCH 15/19] tree-diff: no need to call full diff_tree_sha1 from show_path()

[Kirill Smelkov]

As described in previous commit, when recursing into sub-trees, we can
use lower-level tree walker, since its interface is now sha1 based.

The change is ok, because diff_tree_sha1() only invokes
__diff_tree_sha1(), and also, if base is empty, try_to_follow_renames().
But base is not empty here, as we have added a path and '/' before
recursing.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
Signed-off-by: Junio C Hamano gits...@pobox.com
---

( re-posting without change )

 tree-diff.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index f90acf5..aea0297 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -114,8 +114,8 @@ static void show_path(struct strbuf *base, struct 
diff_options *opt,
 
if (recurse) {
strbuf_addch(base, '/');
-   diff_tree_sha1(t1 ? t1-entry.sha1 : NULL,
-  t2 ? t2-entry.sha1 : NULL, base-buf, opt);
+   __diff_tree_sha1(t1 ? t1-entry.sha1 : NULL,
+t2 ? t2-entry.sha1 : NULL, base-buf, opt);
}
 
strbuf_setlen(base, old_baselen);
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 17/19] Portable alloca for Git

[Kirill Smelkov]

In the next patch we'll have to use alloca() for performance reasons,
but since alloca is non-standardized and is not portable, let's have a
trick with compatibility wrappers:

1. at configure time, determine, do we have working alloca() through
   alloca.h, and define

#define HAVE_ALLOCA_H

   if yes.

2. in code

#ifdef HAVE_ALLOCA_H
# include alloca.h
# define xalloca(size)  (alloca(size))
# define xalloca_free(p)do {} while(0)
#else
# define xalloca(size)  (xmalloc(size))
# define xalloca_free(p)(free(p))
#endif

   and use it like

   func() {
   p = xalloca(size);
   ...

   xalloca_free(p);
   }

This way, for systems, where alloca is available, we'll have optimal
on-stack allocations with fast executions. On the other hand, on
systems, where alloca is not available, this gracefully fallbacks to
xmalloc/free.

Both autoconf and config.mak.uname configurations were updated. For
autoconf, we are not bothering considering cases, when no alloca.h is
available, but alloca() works some other way - its simply alloca.h is
available and works or not, everything else is deep legacy.

For config.mak.uname, I've tried to make my almost-sure guess for where
alloca() is available, but since I only have access to Linux it is the
only change I can be sure about myself, with relevant to other changed
systems people Cc'ed.

NOTE

SunOS and Windows had explicit -DHAVE_ALLOCA_H in their configurations.
I've changed that to now-common HAVE_ALLOCA_H=YesPlease which should be
correct.

Cc: Brandon Casey draf...@gmail.com
Cc: Marius Storm-Olsen msto...@gmail.com
Cc: Johannes Sixt j...@kdbg.org
Cc: Johannes Schindelin johannes.schinde...@gmx.de
Cc: Ramsay Jones ram...@ramsay1.demon.co.uk
Cc: Gerrit Pape p...@smarden.org
Cc: Petr Salinger petr.salin...@seznam.cz
Cc: Jonathan Nieder jrnie...@gmail.com
Cc: Thomas Schwinge tschwi...@gnu.org
Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---

( new patch )

 Makefile  |  6 ++
 config.mak.uname  | 10 --
 configure.ac  |  8 
 git-compat-util.h |  8 
 4 files changed, 30 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index dddaf4f..0334806 100644
--- a/Makefile
+++ b/Makefile
@@ -30,6 +30,8 @@ all::
 # Define LIBPCREDIR=/foo/bar if your libpcre header and library files are in
 # /foo/bar/include and /foo/bar/lib directories.
 #
+# Define HAVE_ALLOCA_H if you have working alloca(3) defined in that header.
+#
 # Define NO_CURL if you do not have libcurl installed.  git-http-fetch and
 # git-http-push are not built, and you cannot use http:// and https://
 # transports (neither smart nor dumb).
@@ -1099,6 +1101,10 @@ ifdef USE_LIBPCRE
EXTLIBS += -lpcre
 endif
 
+ifdef HAVE_ALLOCA_H
+   BASIC_CFLAGS += -DHAVE_ALLOCA_H
+endif
+
 ifdef NO_CURL
BASIC_CFLAGS += -DNO_CURL
REMOTE_CURL_PRIMARY =
diff --git a/config.mak.uname b/config.mak.uname
index 7d31fad..71602ee 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -28,6 +28,7 @@ ifeq ($(uname_S),OSF1)
NO_NSEC = YesPlease
 endif
 ifeq ($(uname_S),Linux)
+   HAVE_ALLOCA_H = YesPlease
NO_STRLCPY = YesPlease
NO_MKSTEMPS = YesPlease
HAVE_PATHS_H = YesPlease
@@ -35,6 +36,7 @@ ifeq ($(uname_S),Linux)
HAVE_DEV_TTY = YesPlease
 endif
 ifeq ($(uname_S),GNU/kFreeBSD)
+   HAVE_ALLOCA_H = YesPlease
NO_STRLCPY = YesPlease
NO_MKSTEMPS = YesPlease
HAVE_PATHS_H = YesPlease
@@ -103,6 +105,7 @@ ifeq ($(uname_S),SunOS)
NEEDS_NSL = YesPlease
SHELL_PATH = /bin/bash
SANE_TOOL_PATH = /usr/xpg6/bin:/usr/xpg4/bin
+   HAVE_ALLOCA_H = YesPlease
NO_STRCASESTR = YesPlease
NO_MEMMEM = YesPlease
NO_MKDTEMP = YesPlease
@@ -146,7 +149,7 @@ ifeq ($(uname_S),SunOS)
endif
INSTALL = /usr/ucb/install
TAR = gtar
-   BASIC_CFLAGS += -D__EXTENSIONS__ -D__sun__ -DHAVE_ALLOCA_H
+   BASIC_CFLAGS += -D__EXTENSIONS__ -D__sun__
 endif
 ifeq ($(uname_O),Cygwin)
ifeq ($(shell expr $(uname_R) : '1\.[1-6]\.'),4)
@@ -166,6 +169,7 @@ ifeq ($(uname_O),Cygwin)
else
NO_REGEX = UnfortunatelyYes
endif
+   HAVE_ALLOCA_H = YesPlease
NEEDS_LIBICONV = YesPlease
NO_FAST_WORKING_DIRECTORY = UnfortunatelyYes
NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease
@@ -239,6 +243,7 @@ ifeq ($(uname_S),AIX)
 endif
 ifeq ($(uname_S),GNU)
# GNU/Hurd
+   HAVE_ALLOCA_H = YesPlease
NO_STRLCPY = YesPlease
NO_MKSTEMPS = YesPlease
HAVE_PATHS_H = YesPlease
@@ -316,6 +321,7 @@ endif
 ifeq ($(uname_S),Windows)
GIT_VERSION := $(GIT_VERSION).MSVC
pathsep = ;
+   HAVE_ALLOCA_H = YesPlease
NO_PREAD = YesPlease
NEEDS_CRYPTO_WITH_SSL = YesPlease
NO_LIBGEN_H = YesPlease
@@ -363,7 +369,7 @@ ifeq ($(uname_S),Windows)
COMPAT_OBJS = compat/msvc.o compat/winansi.o \
compat

[PATCH 03/19] tree-diff: no need to manually verify that there is no mode change for a path

[Kirill Smelkov]

Because if there is, such two tree entries would never be compared as
equal - the code in base_name_compare() explicitly compares modes, if
there is a change for dir bit, even for equal paths, entries would
compare as different.

The code I'm removing here is from 2005 April 262e82b4 (Fix diff-tree
recursion), which pre-dates base_name_compare() introduction in 958ba6c9
(Introduce base_name_compare() helper function) by a month.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
Signed-off-by: Junio C Hamano gits...@pobox.com
---

( re-posting without change )

 tree-diff.c | 15 +--
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 11c3550..5810b00 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -23,6 +23,11 @@ static int compare_tree_entry(struct tree_desc *t1, struct 
tree_desc *t2,
 
pathlen1 = tree_entry_len(t1-entry);
pathlen2 = tree_entry_len(t2-entry);
+
+   /*
+* NOTE files and directories *always* compare differently,
+* even when having the same name.
+*/
cmp = base_name_compare(path1, pathlen1, mode1, path2, pathlen2, mode2);
if (cmp  0) {
show_entry(opt, -, t1, base);
@@ -35,16 +40,6 @@ static int compare_tree_entry(struct tree_desc *t1, struct 
tree_desc *t2,
if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)  !hashcmp(sha1, sha2)  
mode1 == mode2)
return 0;
 
-   /*
-* If the filemode has changed to/from a directory from/to a regular
-* file, we need to consider it a remove and an add.
-*/
-   if (S_ISDIR(mode1) != S_ISDIR(mode2)) {
-   show_entry(opt, -, t1, base);
-   show_entry(opt, +, t2, base);
-   return 0;
-   }
-
strbuf_add(base, path1, pathlen1);
if (DIFF_OPT_TST(opt, RECURSIVE)  S_ISDIR(mode1)) {
if (DIFF_OPT_TST(opt, TREE_IN_RECURSIVE)) {
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 05/19] tree-diff: show_tree() is not needed

[Kirill Smelkov]

We don't need special code for showing added/removed subtree, because we
can do the same via diff_tree_sha1, just passing NULL for absent tree.

And compared to show_tree(), which was calling show_entry() for every
tree entry, that would lead to the same show_entry() callings:

show_tree(t):
for e in t.entries:
show_entry(e)

diff_tree_sha1(NULL, new):  /* the same applies to (old, NULL) */
diff_tree(t1=NULL, t2)
...
if (!t1-size)
show_entry(t2)
...

and possible overhead is negligible, since after the patch, timing for

`git log --raw --no-abbrev --no-renames`

for navy.git and `linux.git v3.10..v3.11` is practically the same.

So let's say goodbye to show_tree() - it removes some code, but also,
and what is important, consolidates more code for showing/recursing into
trees into one place.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---

( re-posting without change )

 tree-diff.c | 35 +++
 1 file changed, 3 insertions(+), 32 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index a8c2aec..2ad7788 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -55,25 +55,7 @@ static int compare_tree_entry(struct tree_desc *t1, struct 
tree_desc *t2,
return 0;
 }
 
-/* A whole sub-tree went away or appeared */
-static void show_tree(struct diff_options *opt, const char *prefix,
- struct tree_desc *desc, struct strbuf *base)
-{
-   enum interesting match = entry_not_interesting;
-   for (; desc-size; update_tree_entry(desc)) {
-   if (match != all_entries_interesting) {
-   match = tree_entry_interesting(desc-entry, base, 0,
-  opt-pathspec);
-   if (match == all_entries_not_interesting)
-   break;
-   if (match == entry_not_interesting)
-   continue;
-   }
-   show_entry(opt, prefix, desc, base);
-   }
-}
-
-/* A file entry went away or appeared */
+/* An entry went away or appeared */
 static void show_entry(struct diff_options *opt, const char *prefix,
   struct tree_desc *desc, struct strbuf *base)
 {
@@ -85,23 +67,12 @@ static void show_entry(struct diff_options *opt, const char 
*prefix,
 
strbuf_add(base, path, pathlen);
if (DIFF_OPT_TST(opt, RECURSIVE)  S_ISDIR(mode)) {
-   enum object_type type;
-   struct tree_desc inner;
-   void *tree;
-   unsigned long size;
-
-   tree = read_sha1_file(sha1, type, size);
-   if (!tree || type != OBJ_TREE)
-   die(corrupt tree sha %s, sha1_to_hex(sha1));
-
if (DIFF_OPT_TST(opt, TREE_IN_RECURSIVE))
opt-add_remove(opt, *prefix, mode, sha1, 1, base-buf, 
0);
 
strbuf_addch(base, '/');
-
-   init_tree_desc(inner, tree, size);
-   show_tree(opt, prefix, inner, base);
-   free(tree);
+   diff_tree_sha1(*prefix == '-' ? sha1 : NULL,
+  *prefix == '+' ? sha1 : NULL, base-buf, opt);
} else
opt-add_remove(opt, prefix[0], mode, sha1, 1, base-buf, 0);
 
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 18/19] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

[Kirill Smelkov]

 be absent from it.

That case was my initial motivation for combined diffs speedup.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---

Changes since v1:

- fixed last-minute thinko/bug last time introduced on my side (sorry) with
  opt-pathchange manipulation in __diff_tree_sha1() - we were forgetting to
  restore opt-pathchange, which led to incorrect log -c (merges _and_ plain
  diff-tree) output;

  This time, I've verified several times, log output stays really the same.

- direct use of alloca() changed to portability wrappers xalloca/xalloca_free
  which gracefully degrade to xmalloc/free on systems, where alloca is not
  available (see new patch 17).

- i = 0; do { ... } while (++i  nparent) is back to usual looping
  for (i = 0; i  nparent; ++), as I've re-measured timings and the
  difference is negligible.

  ( Initially, when I was fighting for every cycle it made sense, but real
no-slowdown turned out to be related to avoiding mallocs, load trees in 
correct
order and reducing register pressure. )

- S_IFXMIN_NEQ definition moved out to cache.h, to have all modes registry in 
one place;


- p0 - first_parent; corrected comments about how emit_diff_first_parent_only
  behaves;


not changed:

- low-level helpers are still named with __ prefix as, imho, that is the best
  convention to name such helpers, without sacrificing signal/noise ratio. All
  of them are now static though.

 cache.h |  15 ++
 diff.c  |   1 +
 diff.h  |  10 ++
 tree-diff.c | 508 
 4 files changed, 471 insertions(+), 63 deletions(-)

diff --git a/cache.h b/cache.h
index dc040fb..e7f5a0c 100644
--- a/cache.h
+++ b/cache.h
@@ -75,6 +75,21 @@ unsigned long git_deflate_bound(git_zstream *, unsigned 
long);
 #define S_ISGITLINK(m) (((m)  S_IFMT) == S_IFGITLINK)
 
 /*
+ * Some mode bits are also used internally for computations.
+ *
+ * They *must* not overlap with any valid modes, and they *must* not be emitted
+ * to outside world - i.e. appear on disk or network. In other words, it's just
+ * temporary fields, which we internally use, but they have to stay in-house.
+ *
+ * ( such approach is valid, as standard S_IF* fits into 16 bits, and in Git
+ *   codebase mode is `unsigned int` which is assumed to be at least 32 bits )
+ */
+
+/* used internally in tree-diff */
+#define S_DIFFTREE_IFXMIN_NEQ  0x8000
+
+
+/*
  * Intensive research over the course of many years has shown that
  * port 9418 is totally unused by anything else. Or
  *
diff --git a/diff.c b/diff.c
index 8e4a6a9..cda4aa8 100644
--- a/diff.c
+++ b/diff.c
@@ -3216,6 +3216,7 @@ void diff_setup(struct diff_options *options)
options-context = diff_context_default;
DIFF_OPT_SET(options, RENAME_EMPTY);
 
+   /* pathchange left =NULL by default */
options-change = diff_change;
options-add_remove = diff_addremove;
options-use_color = diff_use_color_default;
diff --git a/diff.h b/diff.h
index 5d7b9f7..732dca7 100644
--- a/diff.h
+++ b/diff.h
@@ -15,6 +15,10 @@ struct diff_filespec;
 struct userdiff_driver;
 struct sha1_array;
 struct commit;
+struct combine_diff_path;
+
+typedef int (*pathchange_fn_t)(struct diff_options *options,
+struct combine_diff_path *path);
 
 typedef void (*change_fn_t)(struct diff_options *options,
 unsigned old_mode, unsigned new_mode,
@@ -157,6 +161,7 @@ struct diff_options {
int close_file;
 
struct pathspec pathspec;
+   pathchange_fn_t pathchange;
change_fn_t change;
add_remove_fn_t add_remove;
diff_format_fn_t format_callback;
@@ -189,6 +194,11 @@ const char *diff_line_prefix(struct diff_options *);
 
 extern const char mime_boundary_leader[];
 
+extern
+struct combine_diff_path *diff_tree_paths(
+   struct combine_diff_path *p, const unsigned char *sha1,
+   const unsigned char **parent_sha1, int nparent,
+   struct strbuf *base, struct diff_options *opt);
 extern int diff_tree_sha1(const unsigned char *old, const unsigned char *new,
  const char *base, struct diff_options *opt);
 extern int diff_root_tree_sha1(const unsigned char *new, const char *base,
diff --git a/tree-diff.c b/tree-diff.c
index c76821d..b682d77 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -7,6 +7,22 @@
 #include tree.h
 
 /*
+ * internal mode marker, saying a tree entry != entry of tp[imin]
+ * (see __diff_tree_paths for what it means there)
+ *
+ * we will update/use/emit entry for diff only with it unset.
+ */
+#define S_IFXMIN_NEQ   S_DIFFTREE_IFXMIN_NEQ
+
+
+static struct combine_diff_path *__diff_tree_paths(
+   struct combine_diff_path *p, const unsigned char *sha1,
+   const unsigned char **parents_sha1, int nparent,
+   struct strbuf *base, struct diff_options *opt);
+static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
+   struct strbuf *base, struct diff_options

[PATCH 10/19] tree-diff: show_path prototype is not needed anymore

[Kirill Smelkov]

We moved all action-taking code below show_path() in recent HEAD~~
(tree-diff: move all action-taking code out of compare_tree_entry).

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
Signed-off-by: Junio C Hamano gits...@pobox.com
---

( re-posting without change )

 tree-diff.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 3345534..20a4fda 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -6,9 +6,6 @@
 #include diffcore.h
 #include tree.h
 
-static void show_path(struct strbuf *base, struct diff_options *opt,
- struct tree_desc *t1, struct tree_desc *t2);
-
 /*
  * Compare two tree entries, taking into account only path/S_ISDIR(mode),
  * but not their sha1's.
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 08/19] tree-diff: move all action-taking code out of compare_tree_entry()

[Kirill Smelkov]

- let it do only comparison.

This way the code is cleaner and more structured - cmp function only
compares, and the driver takes action based on comparison result.

There should be no change in performance, as effectively, we just move
if series from on place into another, and merge it to was-already-there
same switch/if, so the result is maybe a little bit faster.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
Signed-off-by: Junio C Hamano gits...@pobox.com
---

( re-posting without change )

 tree-diff.c | 28 
 1 file changed, 12 insertions(+), 16 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 5f7dbbf..6207372 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -9,8 +9,7 @@
 static void show_path(struct strbuf *base, struct diff_options *opt,
  struct tree_desc *t1, struct tree_desc *t2);
 
-static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2,
- struct strbuf *base, struct diff_options *opt)
+static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2)
 {
unsigned mode1, mode2;
const char *path1, *path2;
@@ -28,19 +27,7 @@ static int compare_tree_entry(struct tree_desc *t1, struct 
tree_desc *t2,
 * even when having the same name.
 */
cmp = base_name_compare(path1, pathlen1, mode1, path2, pathlen2, mode2);
-   if (cmp  0) {
-   show_path(base, opt, t1, /*t2=*/NULL);
-   return -1;
-   }
-   if (cmp  0) {
-   show_path(base, opt, /*t1=*/NULL, t2);
-   return 1;
-   }
-   if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)  !hashcmp(sha1, sha2)  
mode1 == mode2)
-   return 0;
-
-   show_path(base, opt, t1, t2);
-   return 0;
+   return cmp;
 }
 
 
@@ -161,6 +148,8 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
strbuf_add(base, base_str, baselen);
 
for (;;) {
+   int cmp;
+
if (diff_can_quit_early(opt))
break;
if (opt-pathspec.nr) {
@@ -180,21 +169,28 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
continue;
}
 
-   cmp = compare_tree_entry(t1, t2, base, opt);
+   cmp = compare_tree_entry(t1, t2);
 
/* t1 = t2 */
if (cmp == 0) {
+   if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
+   hashcmp(t1-entry.sha1, t2-entry.sha1) ||
+   (t1-entry.mode != t2-entry.mode))
+   show_path(base, opt, t1, t2);
+
update_tree_entry(t1);
update_tree_entry(t2);
}
 
/* t1  t2 */
else if (cmp  0) {
+   show_path(base, opt, t1, /*t2=*/NULL);
update_tree_entry(t1);
}
 
/* t1  t2 */
else {
+   show_path(base, opt, /*t1=*/NULL, t2);
update_tree_entry(t2);
}
}
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 07/19] tree-diff: don't assume compare_tree_entry() returns -1,0,1

[Kirill Smelkov]

It does, but we'll be reworking it in the next patch after it won't, and
besides it is better to stick to standard
strcmp/memcmp/base_name_compare/etc... convention, where comparison
function returns 0, =0, 0

Regarding performance, comparing for 0, =0, 0 should be a little bit
faster, than switch, because it is just 1 test-without-immediate
instruction and then up to 3 conditional branches, and in switch you
have up to 3 tests with immediate and up to 3 conditional branches.

No worry, that update_tree_entry(t2) is duplicated for =0 and 0 - it
will be good after we'll be adding support for multiparent walker and
will stay that way.

=0 case goes first, because it happens more often in real diffs - i.e.
paths are the same.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
Signed-off-by: Junio C Hamano gits...@pobox.com
---

( re-posting without change )

 tree-diff.c | 22 ++
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index a5b9ff9..5f7dbbf 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -179,18 +179,24 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
update_tree_entry(t1);
continue;
}
-   switch (compare_tree_entry(t1, t2, base, opt)) {
-   case -1:
+
+   cmp = compare_tree_entry(t1, t2, base, opt);
+
+   /* t1 = t2 */
+   if (cmp == 0) {
update_tree_entry(t1);
-   continue;
-   case 0:
+   update_tree_entry(t2);
+   }
+
+   /* t1  t2 */
+   else if (cmp  0) {
update_tree_entry(t1);
-   /* Fallthrough */
-   case 1:
+   }
+
+   /* t1  t2 */
+   else {
update_tree_entry(t2);
-   continue;
}
-   die(git diff-tree: internal error);
}
 
strbuf_release(base);
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 16/19] tree-diff: reuse base str(buf) memory on sub-tree recursion

[Kirill Smelkov]

instead of allocating it all the time for every subtree in
__diff_tree_sha1, let's allocate it once in diff_tree_sha1, and then all
callee just use it in stacking style, without memory allocations.

This should be faster, and for me this change gives the following
slight speedups for

git log --raw --no-abbrev --no-renames --format='%H'

navy.gitlinux.git v3.10..v3.11

before  0.618s  1.903s
after   0.611s  1.889s
speedup 1.1%0.7%

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---

Changes since v1:

 - don't need to touch diff.h, as the function we are changing became static.

 tree-diff.c | 36 ++--
 1 file changed, 18 insertions(+), 18 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index aea0297..c76821d 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -115,7 +115,7 @@ static void show_path(struct strbuf *base, struct 
diff_options *opt,
if (recurse) {
strbuf_addch(base, '/');
__diff_tree_sha1(t1 ? t1-entry.sha1 : NULL,
-t2 ? t2-entry.sha1 : NULL, base-buf, opt);
+t2 ? t2-entry.sha1 : NULL, base, opt);
}
 
strbuf_setlen(base, old_baselen);
@@ -138,12 +138,10 @@ static void skip_uninteresting(struct tree_desc *t, 
struct strbuf *base,
 }
 
 static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
-   const char *base_str, struct diff_options *opt)
+   struct strbuf *base, struct diff_options *opt)
 {
struct tree_desc t1, t2;
void *t1tree, *t2tree;
-   struct strbuf base;
-   int baselen = strlen(base_str);
 
t1tree = fill_tree_descriptor(t1, old);
t2tree = fill_tree_descriptor(t2, new);
@@ -151,17 +149,14 @@ static int __diff_tree_sha1(const unsigned char *old, 
const unsigned char *new,
/* Enable recursion indefinitely */
opt-pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
 
-   strbuf_init(base, PATH_MAX);
-   strbuf_add(base, base_str, baselen);
-
for (;;) {
int cmp;
 
if (diff_can_quit_early(opt))
break;
if (opt-pathspec.nr) {
-   skip_uninteresting(t1, base, opt);
-   skip_uninteresting(t2, base, opt);
+   skip_uninteresting(t1, base, opt);
+   skip_uninteresting(t2, base, opt);
}
if (!t1.size  !t2.size)
break;
@@ -173,7 +168,7 @@ static int __diff_tree_sha1(const unsigned char *old, const 
unsigned char *new,
if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
hashcmp(t1.entry.sha1, t2.entry.sha1) ||
(t1.entry.mode != t2.entry.mode))
-   show_path(base, opt, t1, t2);
+   show_path(base, opt, t1, t2);
 
update_tree_entry(t1);
update_tree_entry(t2);
@@ -181,18 +176,17 @@ static int __diff_tree_sha1(const unsigned char *old, 
const unsigned char *new,
 
/* t1  t2 */
else if (cmp  0) {
-   show_path(base, opt, t1, /*t2=*/NULL);
+   show_path(base, opt, t1, /*t2=*/NULL);
update_tree_entry(t1);
}
 
/* t1  t2 */
else {
-   show_path(base, opt, /*t1=*/NULL, t2);
+   show_path(base, opt, /*t1=*/NULL, t2);
update_tree_entry(t2);
}
}
 
-   strbuf_release(base);
free(t2tree);
free(t1tree);
return 0;
@@ -209,7 +203,7 @@ static inline int diff_might_be_rename(void)
!DIFF_FILE_VALID(diff_queued_diff.queue[0]-one);
 }
 
-static void try_to_follow_renames(const unsigned char *old, const unsigned 
char *new, const char *base, struct diff_options *opt)
+static void try_to_follow_renames(const unsigned char *old, const unsigned 
char *new, struct strbuf *base, struct diff_options *opt)
 {
struct diff_options diff_opts;
struct diff_queue_struct *q = diff_queued_diff;
@@ -306,13 +300,19 @@ static void try_to_follow_renames(const unsigned char 
*old, const unsigned char
q-nr = 1;
 }
 
-int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const 
char *base, struct diff_options *opt)
+int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const 
char *base_str, struct diff_options *opt)
 {
+   struct strbuf base;
int retval;
 
-   retval = __diff_tree_sha1(old, new, base, opt);
-   if (!*base  DIFF_OPT_TST(opt, FOLLOW_RENAMES)  
diff_might_be_rename())
-   try_to_follow_renames(old, new, base, opt);
+   strbuf_init(base

Re: [PATCH 1/2] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

[Kirill Smelkov]

On Fri, Feb 14, 2014 at 09:37:00AM -0800, Junio C Hamano wrote:
 Kirill Smelkov k...@mns.spb.ru writes:
 
  Previously diff_tree(), which is now named __diff_tree_sha1(), was
 
 That name with two leading underscores is a rather unfortunate,
 especially for a function that is not a file scope static.  No way
 to rename this to something more sensible?

I agree. In preparatory patches I thought this will go away, but it
stayed. I'll try to come up with something reasonable.


  That impedance mismatch *hurts* *performance* *badly* for generating
  combined diffs - in c839f1bd (combine-diff: optimize combine_diff_path
 
 Please avoid referring to a commit that is not in 'master' by its
 object name.  It can be reworked later and get a different name.

I agree, this makes sense. Is it ok to refer to nearby commits, by say
HEAD~3, if we know we are referring to 3-times previous commit?


  That slowness comes from the fact that currently, while generating
  combined diff, a lot of time is spent computing diff(commit,commit^2)
  just to only then intersect that huge diff to almost small set of files
  from diff(commit,commit^1).
 
 Good observation.
 
 |a|   |b|a  b   -  a ∉ B   -   D(A,B) +=  +aa↓
 |-|   |-|a  b   -  b ∉ A   -   D(A,B) +=  -bb↓
 | |   | |a = b   -  investigate δ(a,b)a↓ b↓
 
 In both the n-parallel and diff-tree, when an entry 'a' is a
 tree, I take this D(A,B) += +a to mean (recursively) adding all
 the paths within 'a' to the result as addition.  Sounds sensible.

Correct.


  D(A,B)
 
  is by definition the same as combined diff
 
  D(A,[B]),
 
  so if we could rework the code for common case and make it be not slower
  for nparent=1 case, usual diff(t1,t2) generation will not be slower, and
  multiparent diff tree-walker would greatly benefit generating
  combine-diff.
 
 OK.

Thanks. My first goal was to demonstrate it is doable - i.e. we could
join two diff tree-walkers into generalized one, and that approach would
be sound and have chances to be accepted.


  What we do is as follows:
 
  1) diff tree-walker __diff_tree_sha1() is internally reworked to be
 a paths generator (new name diff_tree_paths()), with each generated path
 being `struct combine_diff_path` with info for path, new sha1,mode and 
  for
 every parent which sha1,mode it was in it.
 
  2) From that info, we can still generate usual diff queue with
 struct diff_filepairs, via exporting generated
 combine_diff_path, if we know we run for nparent=1 case.
 (see emit_diff() which is now named emit_diff_p0only())
 
 s/p0/first_parent_/; perhaps?

Ok.


  3) In order for diff_can_quit_early(), which checks
 
 DIFF_OPT_TST(opt, HAS_CHANGES))
 
 to work, that exporting have to be happening not in bulk, but
 incrementally, one diff path at a time.
 
 Good thinking.

Yes. This requirement made the diff-paths producer be real generator,
which emits paths incrementally, which is imho, could be a good design
for making the component more reusable.


  Some notes(*):
 
  1) For loops,
 
   i = 0; do { ... } while (++i  nparent);
 
  is used instead of
 
   for (i = 0; i  nparent; ++i)
   ...
 
  because for the former case, the compiler have to emit additional
  prologue code which checks for i = nparent case before entering the
  loop.
 
  As we require nparent must be 0, that additional overhead
  conflicts with the runs not slower for nparent=1 case than before
  goal.
 
 Unfortunate.  I'd rather see us stick to more readable and familiar
 form for maintainability if this were not measurable.

The most effect on performance were avoiding mallocs and reduce register
pressure. This too, was measurable, but of lower impact. If giving away
some part of percent for nparent=1 case is ok, this could be back to
usual for loops.

By the way, I find it a bit unfortunate, we don't have some for-loop
form with post-conditions in C, only do-while. Another observation, is
that it would be good to say to compiler e.g.

assume nparent  0;

and then in e.g.

for (i = 0; i  nparent; i++)
...

the compiler could know it should not do pre-conditions checks before
entering the loop.

I've verified, that such behaviour, at least with gcc, could be achieved
with

if (nparent = 0)
return;

in function prologue - then the compiler deduces, at least with O2, that
after return point, nparent is 0, but this adds slight overhead on each
entry to function, and we have as many calls as there would be
recursions.

To me, the do-while is still readable, but if fors are preferred, I'll
re-measure what it costs and come back.


  2) alloca(), for small arrays, is used for the same reason - if we change
  it to xmalloc()/free() the timings get worse
 
 Do you see any use of it outside compat/?
 
 I thought we specifically avoid alloca() for portability.  Also we
 do not use variable-length-arrays on the stack either, I think.

No, no usage outside

Re: [PATCH 2/2] combine-diff: speed it up, by using multiparent diff tree-walker directly

[Kirill Smelkov]

On Thu, Feb 13, 2014 at 11:55:08AM -0800, Junio C Hamano wrote:
 Kirill Smelkov k...@mns.spb.ru writes:
 
  +   if (need_generic_pathscan) {
  +   /* NOTE generic case also handles --stat, as it computes
  +* diff(sha1,parent_i) for all i to do the job, specifically
  +* for parent0.
  +*/
  +   paths = find_paths_generic(sha1, parents, diffopts);
  +   }
  +   else {
  +   paths = find_paths_multitree(sha1, parents, diffopts);
  +
  +   /* show stat against the first parent even
  +* when doing combined diff.
  +*/
  +   int stat_opt = (opt-output_format 
  +   (DIFF_FORMAT_NUMSTAT|DIFF_FORMAT_DIFFSTAT));
 
 /*
  * We see decl-after-stmt here.
  * Also please have slash-asterisk and asterisk-slash
  * at the beginning and the end of a multi-line comment
  * block on their own line.
  */

Sorry, and thanks for noticing. I usually compile with -Wall, but it
seems it is not enough without explicitly specifying -std=c89.

Comments corrected and the decl-after-stmt fixed, and this time I've
compiled with `-std=c89 -pedantic -Wall -Wextra` to assure no new
warnings are introduced.

Please apply and thanks beforehand,
Kirill

 8 
From: Kirill Smelkov k...@mns.spb.ru
Subject: [PATCH v2 2/2] combine-diff: speed it up, by using multiparent diff
 tree-walker directly

As was recently shown (c839f1bd combine-diff: optimize
combine_diff_path sets intersection), combine-diff runs very slowly. In
that commit we optimized paths sets intersection, but that accounted
only for ~ 25% of the slowness, and as my tracing showed, for linux.git
v3.10..v3.11, for merges a lot of time is spent computing
diff(commit,commit^2) just to only then intersect that huge diff to
almost small set of files from diff(commit,commit^1).

In previous commit, we described the problem in more details, and
reworked the diff tree-walker to be general one - i.e. to work in
multiple parent case too. Now is the time to take advantage of it for
finding paths for combine diff.

The implementation is straightforward - if we know, we can get generated
diff paths directly, and at present that means no diff filtering or
rename/copy detection was requested(*), we can call multiparent tree-walker
directly and get ready paths.

(*) because e.g. at present, all diffcore transformations work on
diff_filepair queues, but in the future, that limitation can be
lifted, if filters would operate directly on combine_diff_paths.

Timings for `git log --raw --no-abbrev --no-renames` without `-c` (git log)
and with `-c` (git log -c) and with `-c --merges` (git log -c --merges)
before and after the patch are as follows:

linux.git v3.10..v3.11

log log -c log -c --merges

before  1.9s16.4s  15.2s
after   1.9s 2.4s   1.1s

The result stayed the same.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---
 combine-diff.c | 88 ++
 diff.c |  1 +
 2 files changed, 84 insertions(+), 5 deletions(-)

Chages since v1:

- fixed declaration-after-statement, and reworked multiline comments to
  start and end with /* and */ on separate lines.

diff --git a/combine-diff.c b/combine-diff.c
index 1732dfd..12764fb 100644
--- a/combine-diff.c
+++ b/combine-diff.c
@@ -1303,7 +1303,7 @@ static const char *path_path(void *obj)
 
 
 /* find set of paths that every parent touches */
-static struct combine_diff_path *find_paths(const unsigned char *sha1,
+static struct combine_diff_path *find_paths_generic(const unsigned char *sha1,
const struct sha1_array *parents, struct diff_options *opt)
 {
struct combine_diff_path *paths = NULL;
@@ -1316,6 +1316,7 @@ static struct combine_diff_path *find_paths(const 
unsigned char *sha1,
/* tell diff_tree to emit paths in sorted (=tree) order */
opt-orderfile = NULL;
 
+   /* D(A,P1...Pn) = D(A,P1) ^ ... ^ D(A,Pn)  (wrt paths) */
for (i = 0; i  num_parent; i++) {
/*
 * show stat against the first parent even when doing
@@ -1346,6 +1347,35 @@ static struct combine_diff_path *find_paths(const 
unsigned char *sha1,
 }
 
 
+/*
+ * find set of paths that everybody touches, assuming diff is run without
+ * rename/copy detection, etc, comparing all trees simultaneously (= faster).
+ */
+static struct combine_diff_path *find_paths_multitree(
+   const unsigned char *sha1, const struct sha1_array *parents,
+   struct diff_options *opt)
+{
+   int i, nparent = parents-nr;
+   const unsigned char **parents_sha1;
+   struct combine_diff_path paths_head;
+   struct strbuf base;
+
+   parents_sha1 = xmalloc(nparent * sizeof(parents_sha1[0]));
+   for (i = 0; i  nparent; i++)
+   parents_sha1[i] = parents-sha1[i];
+
+   /* fake list head, so worker can assume it is non-NULL

Re: [PATCH 1/2] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

[Kirill Smelkov]

On Thu, Feb 13, 2014 at 11:51:19AM -0800, Junio C Hamano wrote:
 Kirill Smelkov k...@mns.spb.ru writes:
 
  +   /* until we go to it next round, .next holds how many bytes we
  +* allocated (for faster realloc - we don't need copying old 
  data).
  +*/
  +   p-next = (struct combine_diff_path *)alloclen;
 
 I am getting this here:
 
 tree-diff.c: In function '__path_appendnew':
 tree-diff.c:140:13: error: cast to pointer from integer of different size 
 [-Werror=int-to-pointer-cast]

Ah, sorry, I only tested on 32 bits, and this indeed could be a valid
warning on systems where sizeof(ptr) != sizeof(int).

In our case, alloclen is small enough that the code should be valid,
and we can avoid the warning, via proper usage of intptr_t.

Please find corrected patch below.

Thanks,
Kirill

 8 
From: Kirill Smelkov k...@mns.spb.ru
Subject: [PATCH v2 1/2] tree-diff: rework diff_tree() to generate diffs for
 multiparent cases as well
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Previously diff_tree(), which is now named __diff_tree_sha1(), was
generating diff_filepair(s) for two trees t1 and t2, and that was
usually used for a commit as t1=HEAD~, and t2=HEAD - i.e. to see changes
a commit introduces.

In Git, however, we have fundamentally built flexibility in that a
commit can have many parents - 1 for a plain commit, 2 for a simple merge,
but also more than 2 for merging several heads at once.

For merges there is a so called combine-diff, which shows diff, a merge
introduces by itself, omitting changes done by any parent. That works
through first finding paths, that are different to all parents, and then
showing generalized diff, with separate columns for +/- for each parent.
The code lives in combine-diff.c .

There is an impedance mismatch, however, in that a commit could
generally have any number of parents, and that while diffing trees, we
divide cases for 2-tree diffs and more-than-2-tree diffs. I mean there
is no special casing for multiple parents commits in e.g.
revision-walker .

That impedance mismatch *hurts* *performance* *badly* for generating
combined diffs - in c839f1bd (combine-diff: optimize combine_diff_path
sets intersection) I've already removed some slowness from it, but from
the timings provided there, it could be seen, that combined diffs still
cost more than an order of magnitude more cpu time, compared to diff for
usual commits, and that would only be an optimistic estimate, if we take
into account that for e.g. linux.git there is only one merge for several
dozens of plain commits.

That slowness comes from the fact that currently, while generating
combined diff, a lot of time is spent computing diff(commit,commit^2)
just to only then intersect that huge diff to almost small set of files
from diff(commit,commit^1).

That's because at present, to compute combine-diff, for first finding
paths, that every parent touches, we use the following combine-diff
property/definition:

D(A,P1...Pn) = D(A,P1) ^ ... ^ D(A,Pn)  (w.r.t. paths)

where

D(A,P1...Pn) is combined diff between commit A, and parents Pi

and

D(A,Pi) is usual two-tree diff Pi..A

So if any of that D(A,Pi) is huge, tracting 1 n-parent combine-diff as n
1-parent diffs and intersecting results will be slow.

And usually, for linux.git and other topic-based workflows, that
D(A,P2) is huge, because, if merge-base of A and P2, is several dozens
of merges (from A, via first parent) below, that D(A,P2) will be diffing
sum of merges from several subsystems to 1 subsystem.

The solution is to avoid computing n 1-parent diffs, and to find
changed-to-all-parents paths via scanning A's and all Pi's trees
simultaneously, at each step comparing their entries, and based on that
comparison, populate paths result, and deduce we could *skip*
*recursing* into subdirectories, if at least for 1 parent, sha1 of that
dir tree is the same as in A. That would save us from doing significant
amount of needless work.

Such approach is very similar to what diff_tree() does, only there we
deal with scanning only 2 trees simultaneously, and for n+1 tree, the
logic is a bit more complex:

D(A,X1...Xn) calculation scheme
---

D(A,X1...Xn) = D(A,X1) ^ ... ^ D(A,Xn)   (regarding resulting paths set)

 D(A,Xj) - diff between A..Xj
 D(A,X1...Xn)- combined diff from A to parents X1,...,Xn

We start from all trees, which are sorted, and compare their entries in
lock-step:

  A X1   Xn
  - --
 |a|   |x1| |xn|
 |-|   |--| ... |--|  i = argmin(x1...xn)
 | |   |  | |  |
 |-|   |--| |--|
 |.|   |. | |. |
  . ..
  . ..

at any time there could be 3 cases:

 1)  a  xi;
 2)  a  xi;
 3)  a = xi.

Schematic deduction of what every case means, and what to do, follows:

1)  a  xi  -  ∀j a ∉ Xj  -  +a ∈ D(A,Xj)  -  D

Re: [PATCH 11/11] tree-diff: reuse base str(buf) memory on sub-tree recursion

[Kirill Smelkov]

On Fri, Feb 07, 2014 at 09:48:52PM +0400, Kirill Smelkov wrote:
 instead of allocating it all the time for every subtree in
 __diff_tree_sha1, let's allocate it once in diff_tree_sha1, and then all
 callee just use it in stacking style, without memory allocations.
 
 This should be faster, and for me this change gives the following
 slight speedups for `git log --raw --no-abbrev --no-renames`
 
 navy.gitlinux.git v3.10..v3.11
 
 before  0.547s  1.791s
 after   0.541s  1.777s
 speedup 1.1%0.8%

The timings above was done with

`git log --raw --no-abbrev --no-renames --format='%H'`
^

Please change them to correct timings:


navy.gitlinux.git v3.10..v3.11

before  0.618s  1.903s
after   0.611s  1.889s
speedup 1.1%0.7%



Thanks,
Kirill
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 0/2] Multiparent diff tree-walker + combine-diff speedup

[Kirill Smelkov]

Here go combine-diff speedup patches in form of first reworking diff
tree-walker to work in general case - when a commit have several parents, not
only one - we are traversing all 1+nparent trees in parallel.

Then we are taking advantage of the new diff tree-walker for speeding up
combine-diff, which for linux.git results in ~14 times speedup.

I understand v1.9.0 is going to be released first, but wanted to finally send
the patches, so that people could start reviewing them.

Please apply on top of ks/tree-diff-more and thanks beforehand,

Kirill



Kirill Smelkov (2):
  tree-diff: rework diff_tree() to generate diffs for multiparent cases as well
  combine-diff: speed it up, by using multiparent diff tree-walker directly

 combine-diff.c |  85 +-
 diff.c |   2 +
 diff.h |  10 ++
 tree-diff.c| 501 +
 4 files changed, 529 insertions(+), 69 deletions(-)

-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 2/2] combine-diff: speed it up, by using multiparent diff tree-walker directly

[Kirill Smelkov]

As was recently shown (c839f1bd combine-diff: optimize
combine_diff_path sets intersection), combine-diff runs very slowly. In
that commit we optimized paths sets intersection, but that accounted
only for ~ 25% of the slowness, and as my tracing showed, for linux.git
v3.10..v3.11, for merges a lot of time is spent computing
diff(commit,commit^2) just to only then intersect that huge diff to
almost small set of files from diff(commit,commit^1).

In previous commit, we described the problem in more details, and
reworked the diff tree-walker to be general one - i.e. to work in
multiple parent case too. Now is the time to take advantage of it for
finding paths for combine diff.

The implementation is straightforward - if we know, we can get generated
diff paths directly, and at present that means no diff filtering or
rename/copy detection was requested(*), we can call multiparent tree-walker
directly and get ready paths.

(*) because e.g. at present, all diffcore transformations work on
diff_filepair queues, but in the future, that limitation can be
lifted, if filters would operate directly on combine_diff_paths.

Timings for `git log --raw --no-abbrev --no-renames` without `-c` (git log)
and with `-c` (git log -c) and with `-c --merges` (git log -c --merges)
before and after the patch are as follows:

linux.git v3.10..v3.11

log log -c log -c --merges

before  1.9s16.4s  15.2s
after   1.9s 2.4s   1.1s

The result stayed the same.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---
 combine-diff.c | 85 ++
 diff.c |  1 +
 2 files changed, 81 insertions(+), 5 deletions(-)

diff --git a/combine-diff.c b/combine-diff.c
index 1732dfd..ddf7495 100644
--- a/combine-diff.c
+++ b/combine-diff.c
@@ -1303,7 +1303,7 @@ static const char *path_path(void *obj)
 
 
 /* find set of paths that every parent touches */
-static struct combine_diff_path *find_paths(const unsigned char *sha1,
+static struct combine_diff_path *find_paths_generic(const unsigned char *sha1,
const struct sha1_array *parents, struct diff_options *opt)
 {
struct combine_diff_path *paths = NULL;
@@ -1316,6 +1316,7 @@ static struct combine_diff_path *find_paths(const 
unsigned char *sha1,
/* tell diff_tree to emit paths in sorted (=tree) order */
opt-orderfile = NULL;
 
+   /* D(A,P1...Pn) = D(A,P1) ^ ... ^ D(A,Pn)  (wrt paths) */
for (i = 0; i  num_parent; i++) {
/*
 * show stat against the first parent even when doing
@@ -1346,6 +1347,35 @@ static struct combine_diff_path *find_paths(const 
unsigned char *sha1,
 }
 
 
+/*
+ * find set of paths that everybody touches, assuming diff is run without
+ * rename/copy detection, etc, comparing all trees simultaneously (= faster).
+ */
+static struct combine_diff_path *find_paths_multitree(
+   const unsigned char *sha1, const struct sha1_array *parents,
+   struct diff_options *opt)
+{
+   int i, nparent = parents-nr;
+   const unsigned char **parents_sha1;
+   struct combine_diff_path paths_head;
+   struct strbuf base;
+
+   parents_sha1 = xmalloc(nparent * sizeof(parents_sha1[0]));
+   for (i = 0; i  nparent; i++)
+   parents_sha1[i] = parents-sha1[i];
+
+   /* fake list head, so worker can assume it is non-NULL */
+   paths_head.next = NULL;
+
+   strbuf_init(base, PATH_MAX);
+   diff_tree_paths(paths_head, sha1, parents_sha1, nparent, base, opt);
+
+   strbuf_release(base);
+   free(parents_sha1);
+   return paths_head.next;
+}
+
+
 void diff_tree_combined(const unsigned char *sha1,
const struct sha1_array *parents,
int dense,
@@ -1355,6 +1385,7 @@ void diff_tree_combined(const unsigned char *sha1,
struct diff_options diffopts;
struct combine_diff_path *p, *paths;
int i, num_paths, needsep, show_log_first, num_parent = parents-nr;
+   int need_generic_pathscan;
 
/* nothing to do, if no parents */
if (!num_parent)
@@ -1377,11 +1408,55 @@ void diff_tree_combined(const unsigned char *sha1,
 
/* find set of paths that everybody touches
 *
-* NOTE find_paths() also handles --stat, as it computes
-* diff(sha1,parent_i) for all i to do the job, specifically
-* for parent0.
+* NOTE
+*
+* Diffcore transformations are bound to diff_filespec and logic
+* comparing two entries - i.e. they do not apply directly to combine
+* diff.
+*
+* If some of such transformations is requested - we launch generic
+* path scanning, which works significantly slower compared to
+* simultaneous all-trees-in-one-go scan in find_paths_multitree().
+*
+* TODO some of the filters could be ported to work on
+* combine_diff_paths - i.e. all

[PATCH 1/2] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

[Kirill Smelkov]

 (merges changes from two parents with both having
separate changes to a file), or an evil one, the map will not be full,
i.e. some valid sha1 would be absent from it.

That case was my initial motivation for combined diffs speedup.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---
 diff.c  |   1 +
 diff.h  |  10 ++
 tree-diff.c | 501 
 3 files changed, 448 insertions(+), 64 deletions(-)

diff --git a/diff.c b/diff.c
index 8e4a6a9..cda4aa8 100644
--- a/diff.c
+++ b/diff.c
@@ -3216,6 +3216,7 @@ void diff_setup(struct diff_options *options)
options-context = diff_context_default;
DIFF_OPT_SET(options, RENAME_EMPTY);
 
+   /* pathchange left =NULL by default */
options-change = diff_change;
options-add_remove = diff_addremove;
options-use_color = diff_use_color_default;
diff --git a/diff.h b/diff.h
index bf834ff..b59e8d6 100644
--- a/diff.h
+++ b/diff.h
@@ -15,6 +15,10 @@ struct diff_filespec;
 struct userdiff_driver;
 struct sha1_array;
 struct commit;
+struct combine_diff_path;
+
+typedef int (*pathchange_fn_t)(struct diff_options *options,
+struct combine_diff_path *path);
 
 typedef void (*change_fn_t)(struct diff_options *options,
 unsigned old_mode, unsigned new_mode,
@@ -157,6 +161,7 @@ struct diff_options {
int close_file;
 
struct pathspec pathspec;
+   pathchange_fn_t pathchange;
change_fn_t change;
add_remove_fn_t add_remove;
diff_format_fn_t format_callback;
@@ -189,6 +194,11 @@ const char *diff_line_prefix(struct diff_options *);
 
 extern const char mime_boundary_leader[];
 
+extern
+struct combine_diff_path *diff_tree_paths(
+   struct combine_diff_path *p, const unsigned char *sha1,
+   const unsigned char **parent_sha1, int nparent,
+   struct strbuf *base, struct diff_options *opt);
 extern int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
struct strbuf *base, struct diff_options *opt);
 extern int diff_tree_sha1(const unsigned char *old, const unsigned char *new,
diff --git a/tree-diff.c b/tree-diff.c
index ab61a0a..92d4087 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -7,6 +7,25 @@
 #include tree.h
 
 /*
+ * internal mode marker, saying a tree entry != entry of tp[imin]
+ * (see __diff_tree_paths for what it means there)
+ *
+ * it *must* not overlap with any valid modes, and we will update/use/emit
+ * entry for diff only with it unset. Only non-overlapping to valid modes is
+ * required, because mode in tree_desc, comes here canonicalized via
+ * canon_mode().
+ *
+ * the definition assumes unsigned is at least 32 bits.
+ */
+#define S_IFXMIN_NEQ   0x8000
+
+
+static struct combine_diff_path *__diff_tree_paths(
+   struct combine_diff_path *p, const unsigned char *sha1,
+   const unsigned char **parents_sha1, int nparent,
+   struct strbuf *base, struct diff_options *opt);
+
+/*
  * Compare two tree entries, taking into account only path/S_ISDIR(mode),
  * but not their sha1's.
  *
@@ -33,72 +52,148 @@ static int tree_entry_pathcmp(struct tree_desc *t1, struct 
tree_desc *t2)
 }
 
 
-/* convert path, t1/t2 - opt-diff_*() callbacks */
-static void emit_diff(struct diff_options *opt, struct strbuf *path,
- struct tree_desc *t1, struct tree_desc *t2)
+/* convert path - opt-diff_*() callbacks
+ *
+ * emits diff to parent0 only.
+ */
+static int emit_diff_p0only(struct diff_options *opt, struct combine_diff_path 
*p)
 {
-   unsigned int mode1 = t1 ? t1-entry.mode : 0;
-   unsigned int mode2 = t2 ? t2-entry.mode : 0;
-
-   if (mode1  mode2) {
-   opt-change(opt, mode1, mode2, t1-entry.sha1, t2-entry.sha1,
-   1, 1, path-buf, 0, 0);
+   struct combine_diff_parent *p0 = p-parent[0];
+   if (p-mode  p0-mode) {
+   opt-change(opt, p0-mode, p-mode, p0-sha1, p-sha1,
+   1, 1, p-path, 0, 0);
}
else {
const unsigned char *sha1;
unsigned int mode;
int addremove;
 
-   if (mode2) {
+   if (p-mode) {
addremove = '+';
-   sha1 = t2-entry.sha1;
-   mode = mode2;
+   sha1 = p-sha1;
+   mode = p-mode;
}
else {
addremove = '-';
-   sha1 = t1-entry.sha1;
-   mode = mode1;
+   sha1 = p0-sha1;
+   mode = p0-mode;
}
 
-   opt-add_remove(opt, addremove, mode, sha1, 1, path-buf, 0);
+   opt-add_remove(opt, addremove, mode, sha1, 1, p-path, 0);
}
+
+   return 0;   /* = no need to keep allocated combine_diff_path */
 }
 
 
-/* new path should be added to diff
+/*
+ * Make a new

Re: [PATCH 00/11] More preparatory work for multiparent tree-walker

[Kirill Smelkov]

On Wed, Feb 12, 2014 at 09:25:51AM -0800, Junio C Hamano wrote:
 Junio C Hamano gits...@pobox.com writes:
 
  Kirill Smelkov k...@mns.spb.ru writes:
 
  Sorry for the confusion. Could you please do the following:
 
  Patches should be applied over to ks/tree-diff-walk
  (74aa4a18). Before applying the patches, please cherry-pick
 
  c90483d9(tree-diff: no need to manually verify that there is no
   mode change for a path)
 
  ef4f0928(tree-diff: no need to pass match to
   skip_uninteresting())
 
  into that branch, and drop them from ks/combine-diff - we'll have one
  branch reworking the diff tree-walker, and the other taking advantage of
  it for combine-diff.
 
  As long as that does not lose changes to tests and clean-ups, I'm
  fine with that direction.  For example, I do not know if you want to
  lose e3f62d12 (diffcore-order: export generic ordering interface,
  2014-01-20), which is part of the combine-diff topic.
 
 Ahh, sorry, I misread the drop as salvage these two and drop the
 rest.  The new series does apply cleanly on a commit in master..pu
 that has both ks/tree-diff-walk and ks/combine-diff, and the latter
 is not yet in 'next' so we are free to reorganize.
 
 Let me flip the latter topic around, also queue these updates and
 push the result out on 'pu'.
 
 Thanks.

Thank you. As we've managed to apply this to pu, I've send the final
speedup patches. Please review them as time permits.

Thanks beforehand,
Kirill
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 02/11] tree-diff: consolidate code for emitting diffs and recursion in one place

[Kirill Smelkov]

On Thu, Feb 13, 2014 at 09:43:27AM -0800, Junio C Hamano wrote:
 Kirill Smelkov k...@mns.spb.ru writes:
 
  +static void show_path(struct strbuf *base, struct diff_options *opt,
  + struct tree_desc *t1, struct tree_desc *t2)
   {
  unsigned mode;
  const char *path;
  -   const unsigned char *sha1 = tree_entry_extract(desc, path, mode);
  -   int pathlen = tree_entry_len(desc-entry);
  +   const unsigned char *sha1;
  +   int pathlen;
  int old_baselen = base-len;
  +   int isdir, recurse = 0, emitthis = 1;
  +
  +   /* at least something has to be valid */
  +   assert(t1 || t2);
  +
  +   if (t2) {
  +   /* path present in resulting tree */
  +   sha1 = tree_entry_extract(t2, path, mode);
  +   pathlen = tree_entry_len(t2-entry);
  +   isdir = S_ISDIR(mode);
  +   }
  +   else {
  +   /* a path was removed - take path from parent. Also take
  +* mode from parent, to decide on recursion.
  +*/
  +   tree_entry_extract(t1, path, mode);
  +   pathlen = tree_entry_len(t1-entry);
  +
  +   isdir = S_ISDIR(mode);
  +   sha1 = NULL;
  +   mode = 0;
  +   }
  +
  +   if (DIFF_OPT_TST(opt, RECURSIVE)  isdir) {
  +   recurse = 1;
  +   emitthis = DIFF_OPT_TST(opt, TREE_IN_RECURSIVE);
  +   }
   
  strbuf_add(base, path, pathlen);
  -   if (DIFF_OPT_TST(opt, RECURSIVE)  S_ISDIR(mode)) {
  -   if (DIFF_OPT_TST(opt, TREE_IN_RECURSIVE))
  -   opt-add_remove(opt, *prefix, mode, sha1, 1, base-buf, 
  0);
   
  +   if (emitthis)
  +   emit_diff(opt, base, t1, t2);
  +
  +   if (recurse) {
  strbuf_addch(base, '/');
  -   diff_tree_sha1(*prefix == '-' ? sha1 : NULL,
  -  *prefix == '+' ? sha1 : NULL, base-buf, opt);
  -   } else
  -   opt-add_remove(opt, prefix[0], mode, sha1, 1, base-buf, 0);
  +   diff_tree_sha1(t1 ? t1-entry.sha1 : NULL,
  +  t2 ? t2-entry.sha1 : NULL, base-buf, opt);
  +   }
 
 
 After this step, sha1 is assigned but never gets used.  Please
 double-check the fix-up I queued in the series before merging it to
 'pu'.

Your fixup is correct - it was my overlook when preparing the patch -
sha1 is needed for later patch (multiparent diff tree-walker), but I've
mistakenly left it here.

The two interesting patches I've sent you today, are already adjusted to
this correction - it is safe to squash the fixup in.

Thanks for noticing,
Kirill
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 00/11] More preparatory work for multiparent tree-walker

[Kirill Smelkov]

On Tue, Feb 11, 2014 at 11:59:02AM -0800, Junio C Hamano wrote:
 Kirill Smelkov k...@mns.spb.ru writes:
 
  Sorry for the confusion. Could you please do the following:
 
  Patches should be applied over to ks/tree-diff-walk
  (74aa4a18). Before applying the patches, please cherry-pick
 
  c90483d9(tree-diff: no need to manually verify that there is no
   mode change for a path)
 
  ef4f0928(tree-diff: no need to pass match to
   skip_uninteresting())
 
  into that branch, and drop them from ks/combine-diff - we'll have one
  branch reworking the diff tree-walker, and the other taking advantage of
  it for combine-diff.
 
 As long as that does not lose changes to tests and clean-ups, I'm
 fine with that direction.  For example, I do not know if you want to
 lose e3f62d12 (diffcore-order: export generic ordering interface,
 2014-01-20), which is part of the combine-diff topic.

Sorry for the confusion again, and please don't worry: we are not going
to lose anything - my only plea here was to transfer two of the patches
to more appropriate topic.

That couple touches tree-diff.c - they were some initial cleanups I've
noticed while working on separate combine-diff tree-walker, which we
decided to drop instead of generalizing diff tree-walker to handle all
cases. Only the cleanups are still relevant and needed as a base for
what I've sent you here.

And as to e3f62d12 (diffcore-order: export generic ordering interface,
2014-01-20) and other patches on ks/diff-c-with-diff-order topic - they
stay as they are - we do not need to rework them as ks/combine-diff
builds on top of the topic and generalizing diff tree-walker is
orthogonal work.


So in short: could you please transform the two tree-diff patches from
ks/combine-diff to ks/tree-diff-walk, then apply sent-here patches to
ks/tree-diff-walk; thats all.


Thanks,
Kirill
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 00/11] More preparatory work for multiparent tree-walker

[Kirill Smelkov]

On Mon, Feb 10, 2014 at 04:28:33PM -0800, Junio C Hamano wrote:
 Kirill Smelkov k...@mns.spb.ru writes:
 
  Here I'm preparing tree-diff.c to be ready for the new tree-walker, so that 
  the
  final change is tractable and looks good and non noisy. Some small speedups
  are gained along the way. The final bits are almost ready, but I don't want 
  to
  release them in a hurry.
 
 No worries.  We are not in a hurry to apply non-regression-fix
 changes during a pre-release feature freeze period anyway.

I see.


 This seems to somehow conflict with other topics and does not
 cleanly apply on top of your other tree-diff topic, by the way.

Sorry for the confusion. Could you please do the following:

Patches should be applied over to ks/tree-diff-walk
(74aa4a18). Before applying the patches, please cherry-pick

c90483d9(tree-diff: no need to manually verify that there is no
 mode change for a path)

ef4f0928(tree-diff: no need to pass match to
 skip_uninteresting())

into that branch, and drop them from ks/combine-diff - we'll have one
branch reworking the diff tree-walker, and the other taking advantage of
it for combine-diff.

Then apply sent-here patches, which, as I've verified this time, should
apply cleanly.

Thanks and sorry again,
Kirill
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 00/11] More preparatory work for multiparent tree-walker

[Kirill Smelkov]

Here I'm preparing tree-diff.c to be ready for the new tree-walker, so that the
final change is tractable and looks good and non noisy. Some small speedups
are gained along the way. The final bits are almost ready, but I don't want to
release them in a hurry.

Please apply and thanks,
Kirill

Kirill Smelkov (11):
  tree-diff: show_tree() is not needed
  tree-diff: consolidate code for emitting diffs and recursion in one place
  tree-diff: don't assume compare_tree_entry() returns -1,0,1
  tree-diff: move all action-taking code out of compare_tree_entry()
  tree-diff: rename compare_tree_entry - tree_entry_pathcmp
  tree-diff: show_path prototype is not needed anymore
  tree-diff: simplify tree_entry_pathcmp
  tree-diff: remove special-case diff-emitting code for empty-tree cases
  tree-diff: rework diff_tree interface to be sha1 based
  tree-diff: no need to call full diff_tree_sha1 from show_path()
  tree-diff: reuse base str(buf) memory on sub-tree recursion

 diff.h  |   4 +-
 tree-diff.c | 270 
 2 files changed, 145 insertions(+), 129 deletions(-)

-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 03/11] tree-diff: don't assume compare_tree_entry() returns -1,0,1

[Kirill Smelkov]

It does, but we'll be reworking it in the next patch after it won't, and
besides it is better to stick to standard
strcmp/memcmp/base_name_compare/etc... convention, where comparison
function returns 0, =0, 0

Regarding performance, comparing for 0, =0, 0 should be a little bit
faster, than switch, because it is just 1 test-without-immediate
instruction and then up to 3 conditional branches, and in switch you
have up to 3 tests with immediate and up to 3 conditional branches.

No worry, that update_tree_entry(t2) is duplicated for =0 and 0 - it
will be good after we'll be adding support for multiparent walker and
will stay that way.

=0 case goes first, because it happens more often in real diffs - i.e.
paths are the same.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---
 tree-diff.c | 22 ++
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 0c8e3fc..c3fbfba 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -181,18 +181,24 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
update_tree_entry(t1);
continue;
}
-   switch (compare_tree_entry(t1, t2, base, opt)) {
-   case -1:
+
+   cmp = compare_tree_entry(t1, t2, base, opt);
+
+   /* t1 = t2 */
+   if (cmp == 0) {
update_tree_entry(t1);
-   continue;
-   case 0:
+   update_tree_entry(t2);
+   }
+
+   /* t1  t2 */
+   else if (cmp  0) {
update_tree_entry(t1);
-   /* Fallthrough */
-   case 1:
+   }
+
+   /* t1  t2 */
+   else {
update_tree_entry(t2);
-   continue;
}
-   die(git diff-tree: internal error);
}
 
strbuf_release(base);
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 10/11] tree-diff: no need to call full diff_tree_sha1 from show_path()

[Kirill Smelkov]

As described in previous commit, when recursing into sub-trees, we can
use lower-level tree walker, since its interface is now sha1 based.

The change is ok, because diff_tree_sha1() only invokes
__diff_tree_sha1(), and also, if base is empty, try_to_follow_renames().
But base is not empty here, as we have added a path and '/' before
recursing.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---
 tree-diff.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index dd6c760..e385ed4 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -116,8 +116,8 @@ static void show_path(struct strbuf *base, struct 
diff_options *opt,
 
if (recurse) {
strbuf_addch(base, '/');
-   diff_tree_sha1(t1 ? t1-entry.sha1 : NULL,
-  t2 ? t2-entry.sha1 : NULL, base-buf, opt);
+   __diff_tree_sha1(t1 ? t1-entry.sha1 : NULL,
+t2 ? t2-entry.sha1 : NULL, base-buf, opt);
}
 
strbuf_setlen(base, old_baselen);
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 09/11] tree-diff: rework diff_tree interface to be sha1 based

[Kirill Smelkov]

In the next commit this will allow to reduce intermediate calls, when
recursing into subtrees - at that stage we know only subtree sha1, and
it is natural for tree walker to start from that phase. For now we do

diff_tree
show_path
diff_tree_sha1
diff_tree
...

and the change will allow to reduce it to

diff_tree
show_path
diff_tree

Also, it will allow to omit allocating strbuf for each subtree, and just
reuse the common strbuf via playing with its len.

The above-mentioned improvements go in the next 2 patches.

The downside is that try_to_follow_renames(), if active, we cause
re-reading of 2 initial trees, which was negligible based on my timings,
and which is outweighed cogently by the upsides.

NOTE To keep with the current interface and semantics, I needed to
rename the function. It will probably be renamed once more later, when
its semantic will change to just generate paths for a diff, instead of
producing it. So the current name is appropriate, but probably temporary.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---
 diff.h  |  4 ++--
 tree-diff.c | 60 
 2 files changed, 30 insertions(+), 34 deletions(-)

diff --git a/diff.h b/diff.h
index a24a767..4994d15 100644
--- a/diff.h
+++ b/diff.h
@@ -189,8 +189,8 @@ const char *diff_line_prefix(struct diff_options *);
 
 extern const char mime_boundary_leader[];
 
-extern int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
-const char *base, struct diff_options *opt);
+extern int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
+   const char *base, struct diff_options *opt);
 extern int diff_tree_sha1(const unsigned char *old, const unsigned char *new,
  const char *base, struct diff_options *opt);
 extern int diff_root_tree_sha1(const unsigned char *new, const char *base,
diff --git a/tree-diff.c b/tree-diff.c
index 7688402..dd6c760 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -139,12 +139,17 @@ static void skip_uninteresting(struct tree_desc *t, 
struct strbuf *base,
}
 }
 
-int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
- const char *base_str, struct diff_options *opt)
+int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
+const char *base_str, struct diff_options *opt)
 {
+   struct tree_desc t1, t2;
+   void *t1tree, *t2tree;
struct strbuf base;
int baselen = strlen(base_str);
 
+   t1tree = fill_tree_descriptor(t1, old);
+   t2tree = fill_tree_descriptor(t2, new);
+
/* Enable recursion indefinitely */
opt-pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
 
@@ -157,39 +162,41 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
if (diff_can_quit_early(opt))
break;
if (opt-pathspec.nr) {
-   skip_uninteresting(t1, base, opt);
-   skip_uninteresting(t2, base, opt);
+   skip_uninteresting(t1, base, opt);
+   skip_uninteresting(t2, base, opt);
}
-   if (!t1-size  !t2-size)
+   if (!t1.size  !t2.size)
break;
 
-   cmp = tree_entry_pathcmp(t1, t2);
+   cmp = tree_entry_pathcmp(t1, t2);
 
/* t1 = t2 */
if (cmp == 0) {
if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
-   hashcmp(t1-entry.sha1, t2-entry.sha1) ||
-   (t1-entry.mode != t2-entry.mode))
-   show_path(base, opt, t1, t2);
+   hashcmp(t1.entry.sha1, t2.entry.sha1) ||
+   (t1.entry.mode != t2.entry.mode))
+   show_path(base, opt, t1, t2);
 
-   update_tree_entry(t1);
-   update_tree_entry(t2);
+   update_tree_entry(t1);
+   update_tree_entry(t2);
}
 
/* t1  t2 */
else if (cmp  0) {
-   show_path(base, opt, t1, /*t2=*/NULL);
-   update_tree_entry(t1);
+   show_path(base, opt, t1, /*t2=*/NULL);
+   update_tree_entry(t1);
}
 
/* t1  t2 */
else {
-   show_path(base, opt, /*t1=*/NULL, t2);
-   update_tree_entry(t2);
+   show_path(base, opt, /*t1=*/NULL, t2);
+   update_tree_entry(t2);
}
}
 
strbuf_release(base);
+   free(t2tree);
+   free(t1tree);
return 0;
 }
 
@@ -204,7 +211,7 @@ static inline int diff_might_be_rename(void

[PATCH 11/11] tree-diff: reuse base str(buf) memory on sub-tree recursion

[Kirill Smelkov]

instead of allocating it all the time for every subtree in
__diff_tree_sha1, let's allocate it once in diff_tree_sha1, and then all
callee just use it in stacking style, without memory allocations.

This should be faster, and for me this change gives the following
slight speedups for `git log --raw --no-abbrev --no-renames`

navy.gitlinux.git v3.10..v3.11

before  0.547s  1.791s
after   0.541s  1.777s
speedup 1.1%0.8%

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---
 diff.h  |  2 +-
 tree-diff.c | 36 ++--
 2 files changed, 19 insertions(+), 19 deletions(-)

diff --git a/diff.h b/diff.h
index 4994d15..14016ce 100644
--- a/diff.h
+++ b/diff.h
@@ -190,7 +190,7 @@ const char *diff_line_prefix(struct diff_options *);
 extern const char mime_boundary_leader[];
 
 extern int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
-   const char *base, struct diff_options *opt);
+   struct strbuf *base, struct diff_options *opt);
 extern int diff_tree_sha1(const unsigned char *old, const unsigned char *new,
  const char *base, struct diff_options *opt);
 extern int diff_root_tree_sha1(const unsigned char *new, const char *base,
diff --git a/tree-diff.c b/tree-diff.c
index e385ed4..2adda04 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -117,7 +117,7 @@ static void show_path(struct strbuf *base, struct 
diff_options *opt,
if (recurse) {
strbuf_addch(base, '/');
__diff_tree_sha1(t1 ? t1-entry.sha1 : NULL,
-t2 ? t2-entry.sha1 : NULL, base-buf, opt);
+t2 ? t2-entry.sha1 : NULL, base, opt);
}
 
strbuf_setlen(base, old_baselen);
@@ -140,12 +140,10 @@ static void skip_uninteresting(struct tree_desc *t, 
struct strbuf *base,
 }
 
 int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
-const char *base_str, struct diff_options *opt)
+struct strbuf *base, struct diff_options *opt)
 {
struct tree_desc t1, t2;
void *t1tree, *t2tree;
-   struct strbuf base;
-   int baselen = strlen(base_str);
 
t1tree = fill_tree_descriptor(t1, old);
t2tree = fill_tree_descriptor(t2, new);
@@ -153,17 +151,14 @@ int __diff_tree_sha1(const unsigned char *old, const 
unsigned char *new,
/* Enable recursion indefinitely */
opt-pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
 
-   strbuf_init(base, PATH_MAX);
-   strbuf_add(base, base_str, baselen);
-
for (;;) {
int cmp;
 
if (diff_can_quit_early(opt))
break;
if (opt-pathspec.nr) {
-   skip_uninteresting(t1, base, opt);
-   skip_uninteresting(t2, base, opt);
+   skip_uninteresting(t1, base, opt);
+   skip_uninteresting(t2, base, opt);
}
if (!t1.size  !t2.size)
break;
@@ -175,7 +170,7 @@ int __diff_tree_sha1(const unsigned char *old, const 
unsigned char *new,
if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
hashcmp(t1.entry.sha1, t2.entry.sha1) ||
(t1.entry.mode != t2.entry.mode))
-   show_path(base, opt, t1, t2);
+   show_path(base, opt, t1, t2);
 
update_tree_entry(t1);
update_tree_entry(t2);
@@ -183,18 +178,17 @@ int __diff_tree_sha1(const unsigned char *old, const 
unsigned char *new,
 
/* t1  t2 */
else if (cmp  0) {
-   show_path(base, opt, t1, /*t2=*/NULL);
+   show_path(base, opt, t1, /*t2=*/NULL);
update_tree_entry(t1);
}
 
/* t1  t2 */
else {
-   show_path(base, opt, /*t1=*/NULL, t2);
+   show_path(base, opt, /*t1=*/NULL, t2);
update_tree_entry(t2);
}
}
 
-   strbuf_release(base);
free(t2tree);
free(t1tree);
return 0;
@@ -211,7 +205,7 @@ static inline int diff_might_be_rename(void)
!DIFF_FILE_VALID(diff_queued_diff.queue[0]-one);
 }
 
-static void try_to_follow_renames(const unsigned char *old, const unsigned 
char *new, const char *base, struct diff_options *opt)
+static void try_to_follow_renames(const unsigned char *old, const unsigned 
char *new, struct strbuf *base, struct diff_options *opt)
 {
struct diff_options diff_opts;
struct diff_queue_struct *q = diff_queued_diff;
@@ -308,13 +302,19 @@ static void try_to_follow_renames(const unsigned char 
*old, const unsigned char
q-nr = 1

[PATCH 08/11] tree-diff: remove special-case diff-emitting code for empty-tree cases

[Kirill Smelkov]

via teaching tree_entry_pathcmp() how to compare empty tree descriptors:

While walking trees, we iterate their entries from lowest to highest in
sort order, so empty tree means all entries were already went over.

If we artificially assign +infinity value to such tree entry, it will
go after all usual entries, and through the usual driver loop we will be
taking the same actions, which were hand-coded for special cases, i.e.

t1 empty, t2 non-empty
pathcmp(+∞, t2) - +1
show_path(/*t1=*/NULL, t2); /* = t1  t2 case in main loop */

t1 non-empty, t2-empty
pathcmp(t1, +∞) - -1
show_path(t1, /*t2=*/NULL); /* = t1  t2 case in main loop */

Right now we never go to when compared tree descriptors are infinity, as
this condition is checked in the loop beginning as finishing criteria,
but will do in the future, when there will be several parents iterated
simultaneously, and some pair of them would run to the end.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---
 tree-diff.c | 21 +
 1 file changed, 9 insertions(+), 12 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 330ca07..7688402 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -12,12 +12,19 @@
  *
  * NOTE files and directories *always* compare differently, even when having
  *  the same name - thanks to base_name_compare().
+ *
+ * NOTE empty (=invalid) descriptor(s) take part in comparison as +infty.
  */
 static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
 {
struct name_entry *e1, *e2;
int cmp;
 
+   if (!t1-size)
+   return t2-size ? +1 /* +∞  c */  : 0 /* +∞ = +∞ */;
+   else if (!t2-size)
+   return -1;  /* c  +∞ */
+
e1 = t1-entry;
e2 = t2-entry;
cmp = base_name_compare(e1-path, tree_entry_len(e1), e1-mode,
@@ -153,18 +160,8 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
skip_uninteresting(t1, base, opt);
skip_uninteresting(t2, base, opt);
}
-   if (!t1-size) {
-   if (!t2-size)
-   break;
-   show_path(base, opt, /*t1=*/NULL, t2);
-   update_tree_entry(t2);
-   continue;
-   }
-   if (!t2-size) {
-   show_path(base, opt, t1, /*t2=*/NULL);
-   update_tree_entry(t1);
-   continue;
-   }
+   if (!t1-size  !t2-size)
+   break;
 
cmp = tree_entry_pathcmp(t1, t2);
 
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 07/11] tree-diff: simplify tree_entry_pathcmp

[Kirill Smelkov]

Since 74aa4a18 (Finally switch over tree descriptors to contain a
pre-parsed entry) we can safely access all tree_desc-entry fields directly
instead of first extracting them through tree_entry_extract.

Use it. The code generated stays the same - only it now visually looks
cleaner.

Signed-off-by: Kirill Smelkov k...@mns.spb.ru
---
 tree-diff.c | 17 ++---
 1 file changed, 6 insertions(+), 11 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 604dc57..330ca07 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -15,18 +15,13 @@
  */
 static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
 {
-   unsigned mode1, mode2;
-   const char *path1, *path2;
-   const unsigned char *sha1, *sha2;
-   int cmp, pathlen1, pathlen2;
+   struct name_entry *e1, *e2;
+   int cmp;
 
-   sha1 = tree_entry_extract(t1, path1, mode1);
-   sha2 = tree_entry_extract(t2, path2, mode2);
-
-   pathlen1 = tree_entry_len(t1-entry);
-   pathlen2 = tree_entry_len(t2-entry);
-
-   cmp = base_name_compare(path1, pathlen1, mode1, path2, pathlen2, mode2);
+   e1 = t1-entry;
+   e2 = t2-entry;
+   cmp = base_name_compare(e1-path, tree_entry_len(e1), e1-mode,
+   e2-path, tree_entry_len(e2), e2-mode);
return cmp;
 }
 
-- 
1.9.rc1.181.g641f458

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html