[PATCH v2 01/12] commit-graph: add 'verify' subcommand

2018-05-11 Thread Derrick Stolee
graph file exists, this is an acceptable state. Do not report any errors. During review, we noticed that a FREE_AND_NULL(graph_name) was placed after a possible 'return', and this pattern was also in graph_read(). Fix that case, too. Signed-off-by: Derrick Stolee --- Documentation/git-co

[PATCH v2 03/12] commit-graph: test that 'verify' finds corruption

2018-05-11 Thread Derrick Stolee
Add test cases to t5318-commit-graph.sh that corrupt the commit-graph file and check that the 'git commit-graph verify' command fails. These tests verify the header and chunk information is checked carefully. Helped-by: Martin Ågren Signed-off-by: Derrick Stolee --- t/t5318-commi

[PATCH v2 00/12] Integrate commit-graph into fsck and gc

2018-05-11 Thread Derrick Stolee
now happens by default in graph_report(). The integration into 'fetch' is dropped (thanks Ævar!). Derrick Stolee (12): commit-graph: add 'verify' subcommand commit-graph: verify file header information commit-graph: test that 'verify' finds corruption

[PATCH v2 11/12] gc: automatically write commit-graph files

2018-05-11 Thread Derrick Stolee
er any non-trivial 'git gc' command. Defaults to false while the commit-graph feature matures. We specifically do not want to turn this on by default until the commit-graph feature is fully integrated with history-modifying features like shallow clones. Signed-off-by: Derrick Stolee --- Docu

[PATCH v2 10/12] commit-graph: add '--reachable' option

2018-05-11 Thread Derrick Stolee
ite --reachable' after performing cleanup of the object database. Signed-off-by: Derrick Stolee --- Documentation/git-commit-graph.txt | 8 ++-- builtin/commit-graph.c | 41 ++ 2 files changed, 43 insertions(+), 6 deletions(-) diff --git

[PATCH v2 08/12] commit-graph: verify commit contents against odb

2018-05-11 Thread Derrick Stolee
e accepted range causes a segmentation fault. Add a new check in insert_parent_or_die() that prevents this fault. Check for that error during the test, both in the typical parents and in the list of parents for octopus merges. Signed-off-by: Derrick Stolee --- commit-graph.c

[PATCH v2 07/12] commit-graph: load a root tree from specific graph

2018-05-11 Thread Derrick Stolee
When lazy-loading a tree for a commit, it will be important to select the tree from a specific struct commit_graph. Create a new method that specifies the commit-graph file and use that in get_commit_tree_in_graph(). Signed-off-by: Derrick Stolee --- commit-graph.c | 10 -- 1 file

[PATCH v2 06/12] commit: force commit to parse from object database

2018-05-11 Thread Derrick Stolee
is explicit in avoiding commits from the commit-graph file. Signed-off-by: Derrick Stolee --- commit.c | 13 + commit.h | 1 + 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/commit.c b/commit.c index 1d28677dfb..7c92350373 100644 --- a/commit.c +++ b/commit.c

[PATCH v2 09/12] fsck: verify commit-graph

2018-05-11 Thread Derrick Stolee
erifying a commit-graph file from the other fsck details. 2. The commit-graph verification requires the commits to be loaded in a specific order to guarantee we parse from the commit-graph file for some objects and from the object database for others. Signed-off-by: Derrick Stolee --

[PATCH v2 12/12] commit-graph: update design document

2018-05-11 Thread Derrick Stolee
The commit-graph feature is now integrated with 'fsck' and 'gc', so remove those items from the "Future Work" section of the commit-graph design document. Also remove the section on lazy-loading trees, as that was completed in an earlier patch series.

[PATCH v2 05/12] commit-graph: verify fanout and lookup table

2018-05-11 Thread Derrick Stolee
ull output matches the exact error we inserted, but since our OID order test triggers incorrect fanout values (with possibly different numbers of output lines) we focus only that the correct error is written in that case. Signed-off-by: Derrick Stolee --- commit-graph.c

[PATCH v2 04/12] commit-graph: parse commit from chosen graph

2018-05-11 Thread Derrick Stolee
Before verifying a commit-graph file against the object database, we need to parse all commits from the given commit-graph file. Create parse_commit_in_graph_one() to target a given struct commit_graph. Signed-off-by: Derrick Stolee --- commit-graph.c | 18 +++--- 1 file changed, 15

Re: [PATCH 4/4] mark_parents_uninteresting(): avoid most allocation

2018-05-14 Thread Derrick Stolee
On 5/11/2018 2:03 PM, Jeff King wrote: Commit 941ba8db57 (Eliminate recursion in setting/clearing marks in commit list, 2012-01-14) used a clever double-loop to avoid allocations for single-parent chains of history. However, it did so only when following parents of parents (which was an uncommon

Re: [PATCH 0/4] a few mark_parents_uninteresting cleanups

2018-05-14 Thread Derrick Stolee
hanged, 50 insertions(+), 40 deletions(-) -Peff This series looks good to me. I found Patch 3 hard to read in the diff, so I just looked at the final result and the new arrangement is very clear about how it should behave. Reviewed-by: Derrick Stolee

Re: [RFC] Other chunks for commit-graph, part 1 - Bloom filters, topo order, etc.

2018-05-14 Thread Derrick Stolee
On 5/12/2018 10:00 AM, Jakub Narebski wrote: Derrick Stolee writes: On 5/4/2018 3:40 PM, Jakub Narebski wrote: With early parts of commit-graph feature (ds/commit-graph and ds/lazy-load-trees) close to being merged into "master", see https://public-inbox.org/git/xmqq4ljtz87g..

Re: [PATCH 4/4] mark_parents_uninteresting(): avoid most allocation

2018-05-14 Thread Derrick Stolee
On 5/14/2018 9:09 AM, Jeff King wrote: On Mon, May 14, 2018 at 08:47:33AM -0400, Derrick Stolee wrote: On 5/11/2018 2:03 PM, Jeff King wrote: Commit 941ba8db57 (Eliminate recursion in setting/clearing marks in commit list, 2012-01-14) used a clever double-loop to avoid allocations for single

Re: [PATCH v2 01/12] commit-graph: add 'verify' subcommand

2018-05-14 Thread Derrick Stolee
On 5/12/2018 9:31 AM, Martin Ågren wrote: On 11 May 2018 at 23:15, Derrick Stolee wrote: graph_name = get_commit_graph_filename(opts.obj_dir); graph = load_commit_graph_one(graph_name); + FREE_AND_NULL(graph_name); if (!graph) die("graph

Re: [PATCH v2 02/12] commit-graph: verify file header information

2018-05-14 Thread Derrick Stolee
On 5/12/2018 9:35 AM, Martin Ågren wrote: +static int verify_commit_graph_error; + +static void graph_report(const char *fmt, ...) +{ + va_list ap; + struct strbuf sb = STRBUF_INIT; + verify_commit_graph_error = 1; + + va_start(ap, fmt); + strbuf_vaddf(&sb, fmt, ap);

Re: [PATCH v2 08/12] commit-graph: verify commit contents against odb

2018-05-14 Thread Derrick Stolee
On 5/12/2018 5:17 PM, Martin Ågren wrote: On 11 May 2018 at 23:15, Derrick Stolee wrote: When running 'git commit-graph verify', compare the contents of the commits that are loaded from the commit-graph file with commits that are loaded directly from the object database. This include

Re: [PATCH v2 14/14] commit.h: delete 'util' field in struct commit

2018-05-14 Thread Derrick Stolee
On 5/14/2018 1:30 PM, Duy Nguyen wrote: On Mon, May 14, 2018 at 6:07 PM, Duy Nguyen wrote: On Mon, May 14, 2018 at 04:52:29PM +0900, Junio C Hamano wrote: Nguyễn Thái Ngọc Duy writes: diff --git a/commit.h b/commit.h index 838f6a6b26..70371e111e 100644 --- a/commit.h +++ b/commit.h @@ -18,

Re: worktrees vs. alternates

2018-05-16 Thread Derrick Stolee
On 5/16/2018 6:33 AM, Ævar Arnfjörð Bjarmason wrote: [big snip] And here's where this isn't at all like "worktree", each of those 100 will have their own "master" branch, and they can all create 100 different branches called "topic" that can be different. This is the biggest difference. You ca

Re: Git log range reverse bug

2018-05-16 Thread Derrick Stolee
Hi Mendi, On 5/16/2018 2:19 PM, Mehdi Zeinali wrote: Git Version: Version: 2.14.2 When reversing a range in git log, it does not start from the expected commit: $ git show 8e11b4a41ec21e47fb0bf8b76e1edba739f57a9b commit 8e11b4a41ec21e47fb0bf8b76e1edba739f57a9b Author: Some Name Date: Mon No

Re: What's cooking in git.git (May 2018, #02; Thu, 17)

2018-05-17 Thread Derrick Stolee
On 5/17/2018 2:01 AM, Junio C Hamano wrote: * ds/generation-numbers (2018-05-02) 11 commits - commit-graph.txt: update design document - merge: check config before loading commits - commit: use generation number in remove_redundant() - commit: add short-circuit to paint_down_to_common()

commit-graph: change in "best" merge-base when ambiguous

2018-05-21 Thread Derrick Stolee
Hello all, While working on the commit-graph feature, I made a test commit that sets core.commitGraph and gc.commitGraph to true by default AND runs 'git commit-graph write --reachable' after each 'git commit' command. This helped me find instances in the test suite where the commit-graph fea

Re: commit-graph: change in "best" merge-base when ambiguous

2018-05-22 Thread Derrick Stolee
On 5/22/2018 1:39 AM, Michael Haggerty wrote: On 05/21/2018 08:10 PM, Derrick Stolee wrote: [...] In the Discussion section of the `git merge-base` docs [1], we have the following:     When the history involves criss-cross merges, there can be more than one best common ancestor for two

[PATCH v3 07/20] commit-graph: verify catches corrupt signature

2018-05-24 Thread Derrick Stolee
lines added by 'test_must_fail' when the test is run verbosely. Then, the output is checked to contain a specific error message. Signed-off-by: Derrick Stolee --- t/t5318-commit-graph.sh | 43 +++ 1 file changed, 43 insertions(+) diff --git a/t/t53

[PATCH v3 03/20] commit-graph: parse commit from chosen graph

2018-05-24 Thread Derrick Stolee
Before verifying a commit-graph file against the object database, we need to parse all commits from the given commit-graph file. Create parse_commit_in_graph_one() to target a given struct commit_graph. Signed-off-by: Derrick Stolee --- commit-graph.c | 18 +++--- 1 file changed, 15

[PATCH v3 04/20] commit: force commit to parse from object database

2018-05-24 Thread Derrick Stolee
is explicit in avoiding commits from the commit-graph file. Signed-off-by: Derrick Stolee --- commit.c | 9 +++-- commit.h | 1 + 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/commit.c b/commit.c index 1d28677dfb..6eaed0174c 100644 --- a/commit.c +++ b/commit.c @@ -392,7

[PATCH v3 00/20] Integrate commit-graph into 'fsck' and 'gc'

2018-05-24 Thread Derrick Stolee
ange worth mentioning: in "commit-graph: add '--reachable' option" I put the ref-iteration into a new external 'write_commit_graph_reachable()' method inside commit-graph.c. This makes the 'gc: automatically write commit-graph files' a simpler change. Thanks, -S

[PATCH v3 06/20] commit-graph: add 'verify' subcommand

2018-05-24 Thread Derrick Stolee
graph file exists, this is an acceptable state. Do not report any errors. Signed-off-by: Derrick Stolee --- Documentation/git-commit-graph.txt | 6 ++ builtin/commit-graph.c | 38 ++ commit-graph.c | 26 ++

[PATCH v3 12/20] commit-graph: verify parent list

2018-05-24 Thread Derrick Stolee
one parsed from the object database. Test these checks for corrupt parents, too many parents, and wrong parents. The octopus merge will be tested in a later commit. Signed-off-by: Derrick Stolee --- commit-graph.c | 25 + t/t5318-commit-graph.sh | 18 +++

[PATCH v3 01/20] commit-graph: UNLEAK before die()

2018-05-24 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c index 37420ae0fd..f0875b8bf3 100644 --- a/builtin/commit-graph.c +++ b/builtin/commit-graph.c @@ -51,8 +51,11 @@ static

[PATCH v3 14/20] commit-graph: verify commit date

2018-05-24 Thread Derrick Stolee
Signed-off-by: Derrick Stolee --- commit-graph.c | 6 ++ t/t5318-commit-graph.sh | 6 ++ 2 files changed, 12 insertions(+) diff --git a/commit-graph.c b/commit-graph.c index ead92460c1..d2b291aca2 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -981,6 +981,12 @@ int

[PATCH v3 11/20] commit-graph: verify root tree OIDs

2018-05-24 Thread Derrick Stolee
and parsed directly from the object database. Add checks for the root tree OID. Signed-off-by: Derrick Stolee --- commit-graph.c | 17 - t/t5318-commit-graph.sh | 7 +++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/commit-graph.c b/commit-gra

[PATCH v3 10/20] commit-graph: verify objects exist

2018-05-24 Thread Derrick Stolee
In the 'verify' subcommand, load commits directly from the object database to ensure they exist. Parse by skipping the commit-graph. Signed-off-by: Derrick Stolee --- commit-graph.c | 20 t/t5318-commit-graph.sh | 7 +++ 2 files changed, 27 insertion

[PATCH v3 19/20] gc: automatically write commit-graph files

2018-05-24 Thread Derrick Stolee
er any non-trivial 'git gc' command. Defaults to false while the commit-graph feature matures. We specifically do not want to turn this on by default until the commit-graph feature is fully integrated with history-modifying features like shallow clones. Signed-off-by: Derrick Stolee --- Docu

[PATCH v3 09/20] commit-graph: verify corrupt OID fanout and lookup

2018-05-24 Thread Derrick Stolee
In the commit-graph file, the OID fanout chunk provides an index into the OID lookup. The 'verify' subcommand should find incorrect values in the fanout. Similarly, the 'verify' subcommand should find out-of-order values in the OID lookup. Signed-off-by: Derrick Stolee

[PATCH v3 16/20] commit-graph: verify contents match checksum

2018-05-24 Thread Derrick Stolee
our tests, as we break the checksum as we modify bytes of the commit-graph file. Signed-off-by: Derrick Stolee --- commit-graph.c | 16 ++-- t/t5318-commit-graph.sh | 6 ++ 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/commit-graph.c b/commit-gra

[PATCH v3 02/20] commit-graph: fix GRAPH_MIN_SIZE

2018-05-24 Thread Derrick Stolee
n the magic constants. Signed-off-by: Derrick Stolee --- commit-graph.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index a8c337dd77..82295f0975 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -33,10 +33,11 @@ #define GRAPH_LAST

[PATCH v3 05/20] commit-graph: load a root tree from specific graph

2018-05-24 Thread Derrick Stolee
When lazy-loading a tree for a commit, it will be important to select the tree from a specific struct commit_graph. Create a new method that specifies the commit-graph file and use that in get_commit_tree_in_graph(). Signed-off-by: Derrick Stolee --- commit-graph.c | 12 +--- 1 file

[PATCH v3 08/20] commit-graph: verify required chunks are present

2018-05-24 Thread Derrick Stolee
The commit-graph file requires the following three chunks: * OID Fanout * OID Lookup * Commit Data If any of these are missing, then the 'verify' subcommand should report a failure. This includes the chunk IDs malformed or the chunk count is truncated. Signed-off-by: Derrick Stolee -

[PATCH v3 20/20] commit-graph: update design document

2018-05-24 Thread Derrick Stolee
The commit-graph feature is now integrated with 'fsck' and 'gc', so remove those items from the "Future Work" section of the commit-graph design document. Also remove the section on lazy-loading trees, as that was completed in an earlier patch series.

[PATCH v3 15/20] commit-graph: test for corrupted octopus edge

2018-05-24 Thread Derrick Stolee
The commit-graph file has an extra chunk to store the parent int-ids for parents beyond the first parent for octopus merges. Our test repo has a single octopus merge that we can manipulate to demonstrate the 'verify' subcommand detects incorrect values in that chunk. Signed-off-by: Derr

[PATCH v3 17/20] fsck: verify commit-graph

2018-05-24 Thread Derrick Stolee
erifying a commit-graph file from the other fsck details. 2. The commit-graph verification requires the commits to be loaded in a specific order to guarantee we parse from the commit-graph file for some objects and from the object database for others. Signed-off-by: Derrick Stolee --

[PATCH v3 18/20] commit-graph: add '--reachable' option

2018-05-24 Thread Derrick Stolee
le() after performing cleanup of the object database. Signed-off-by: Derrick Stolee --- Documentation/git-commit-graph.txt | 8 ++-- builtin/commit-graph.c | 16 commit-graph.c | 32 comm

[PATCH v3 13/20] commit-graph: verify generation number

2018-05-24 Thread Derrick Stolee
, we drop the single-line condition on the output. Signed-off-by: Derrick Stolee --- commit-graph.c | 18 ++ t/t5318-commit-graph.sh | 6 ++ 2 files changed, 24 insertions(+) diff --git a/commit-graph.c b/commit-graph.c index fff22dc0c3..ead92460c1 100644 --- a/commit

Re: [PATCH v2 03/12] commit-graph: test that 'verify' finds corruption

2018-05-24 Thread Derrick Stolee
On 5/21/2018 2:53 PM, Jakub Narebski wrote: +corrupt_data() { + file=$1 + pos=$2 + data="${3:-\0}" + printf "$data" | dd of="$file" bs=1 seek="$pos" conv=notrunc +} First, if we do this that way (and not by adding a test helper), the use of this function should be, I thin

Re: [PATCH v3 01/20] commit-graph: UNLEAK before die()

2018-05-24 Thread Derrick Stolee
On 5/24/2018 6:47 PM, Stefan Beller wrote: On Thu, May 24, 2018 at 9:25 AM, Derrick Stolee wrote: Signed-off-by: Derrick Stolee --- builtin/commit-graph.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c index

Re: [PATCH] t990X: use '.git/objects' as 'deep inside .git' path

2018-05-29 Thread Derrick Stolee
On 5/27/2018 12:49 AM, Michael Haggerty wrote: On Sat, May 26, 2018 at 8:47 AM, Christian Couder wrote: Tests t9902-completion.sh and t9903-bash-prompt.sh each have tests that check what happens when we are "in the '.git' directory" and when we are "deep inside the '.git' directory". To test t

Re: [PATCH v3 03/20] commit-graph: parse commit from chosen graph

2018-05-29 Thread Derrick Stolee
On 5/27/2018 6:23 AM, Jakub Narebski wrote: Derrick Stolee writes: Before verifying a commit-graph file against the object database, we need to parse all commits from the given commit-graph file. Create parse_commit_in_graph_one() to target a given struct commit_graph. If I understand it

Re: [PATCH v3 00/20] Integrate commit-graph into 'fsck' and 'gc'

2018-05-29 Thread Derrick Stolee
On 5/29/2018 12:27 AM, Junio C Hamano wrote: Derrick Stolee writes: Thanks for all the feedback on v2. I've tried to make this round's review a bit easier by splitting up the commits into smaller pieces. Also, the test script now has less boilerplate and uses variables and clear ari

Re: [PATCH v3 07/20] commit-graph: verify catches corrupt signature

2018-05-29 Thread Derrick Stolee
On 5/28/2018 10:05 AM, Jakub Narebski wrote: Derrick Stolee writes: This is the first of several commits that add a test to check that 'git commit-graph verify' catches corruption in the commit-graph file. The first test checks that the command catches an error in the file signatur

Re: [PATCH] commit-graph: fix a sparse 'integer as NULL pointer' warning

2018-05-29 Thread Derrick Stolee
n 5/29/2018 4:01 PM, Ramsay Jones wrote: Signed-off-by: Ramsay Jones --- Hi Derrick, If you need to re-roll your 'ds/commit-graph-fsck' branch (pu@a84e06bc0f), could you please squash this into the relevant patch (commit 80453b4529, "commit-graph: add 'verify' subcommand", 2018-05-24). [No, N

Re: [RFC PATCH 00/35] object-store: lookup_commit

2018-05-29 Thread Derrick Stolee
On 5/29/2018 8:47 PM, Stefan Beller wrote: This applies on the merge of nd/commit-util-to-slab and sb/object-store-grafts, and is available at http://github.com/stefanbeller/ as branch object-store-lookup-commit as the merge has some merge conflicts as well as syntactical conflicts (upload-pack

Re: [PATCH v3 06/20] commit-graph: add 'verify' subcommand

2018-05-30 Thread Derrick Stolee
On 5/27/2018 6:55 PM, Jakub Narebski wrote: Derrick Stolee writes: If the commit-graph file becomes corrupt, we need a way to verify that its contents match the object database. In the manner of 'git fsck' we will implement a 'git commit-graph verify' subcommand to report

Re: [PATCH v3 09/20] commit-graph: verify corrupt OID fanout and lookup

2018-05-30 Thread Derrick Stolee
On 5/30/2018 9:34 AM, Jakub Narebski wrote: Derrick Stolee writes: In the commit-graph file, the OID fanout chunk provides an index into the OID lookup. The 'verify' subcommand should find incorrect values in the fanout. Similarly, the 'verify' subcommand should find ou

Re: [PATCH 03/35] object: add repository argument to lookup_unknown_object

2018-05-30 Thread Derrick Stolee
On 5/29/2018 8:47 PM, Stefan Beller wrote: From: Jonathan Nieder Add a repository argument to allow callers of lookup_unknown_object to be more specific about which repository to handle. This is a small mechanical change; it doesn't change the implementation to handle repositories other than th

Re: [PATCH 04/35] object: add repository argument to parse_object_buffer

2018-05-30 Thread Derrick Stolee
On 5/29/2018 8:47 PM, Stefan Beller wrote: Add a repository argument to allow the callers of parse_object_buffer to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet. As

Re: [PATCH 06/35] blob: add repository argument to lookup_blob

2018-05-30 Thread Derrick Stolee
On 5/29/2018 8:47 PM, Stefan Beller wrote: Add a repository argument to allow the callers of lookup_blob to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet. As with the

Re: [PATCH 27/35] commit-slabs: remove realloc counter outside of slab struct

2018-05-30 Thread Derrick Stolee
On 5/29/2018 8:48 PM, Stefan Beller wrote: The realloc counter is declared outside the struct for the given slabname, which makes it harder for a follow up patch to move the declaration of the struct around as then the counter variable would need special treatment. As the reallocation counter is

Re: [RFC PATCH 00/35] object-store: lookup_commit

2018-05-30 Thread Derrick Stolee
On 5/29/2018 11:18 PM, Stefan Beller wrote: On Tue, May 29, 2018 at 6:05 PM, Derrick Stolee wrote: On 5/29/2018 8:47 PM, Stefan Beller wrote: This applies on the merge of nd/commit-util-to-slab and sb/object-store-grafts, and is available at http://github.com/stefanbeller/ as branch object

Re: [PATCH v3 10/20] commit-graph: verify objects exist

2018-05-31 Thread Derrick Stolee
On 5/30/2018 3:22 PM, Jakub Narebski wrote: Derrick Stolee writes: In the 'verify' subcommand, load commits directly from the object database to ensure they exist. Parse by skipping the commit-graph. All right, before we check that the commit data matches, we need to check th

Re: [PATCH v3 11/20] commit-graph: verify root tree OIDs

2018-05-31 Thread Derrick Stolee
On 5/30/2018 6:24 PM, Jakub Narebski wrote: Derrick Stolee writes: The 'verify' subcommand must compare the commit content parsed from the commit-graph and compare it against the content in the object database. You have "compare" twice in the above sentence. Use

[RFC PATCH 0/6] Fix commit-graph/graft/replace/shallow combo

2018-05-31 Thread Derrick Stolee
with it for a while, I figured I should just put this series up for discussion. Maybe someone with more experience in shallow clones can point out the obvious issues I'm having. Thanks, -Stolee Derrick Stolee (6): DO NOT MERGE: compute commit-graph on every commit DO NOT MERGE: write

[RFC PATCH 5/6] fetch: destroy commit graph on shallow parameters

2018-05-31 Thread Derrick Stolee
The commit-graph feature is not compatible with history-rewriting features such as shallow clones. When running a 'git fetch' with any of the shallow/unshallow options, destroy the commit-graph file and override core.commitGraph to be false. Signed-off-by: Derrick Stolee --- builtin/f

[RFC PATCH 2/6] DO NOT MERGE: write commit-graph on every fetch

2018-05-31 Thread Derrick Stolee
THIS IS ONLY FOR TESTING TO INCREASE TEST COVERAGE Signed-off-by: Derrick Stolee --- builtin/fetch.c | 4 1 file changed, 4 insertions(+) diff --git a/builtin/fetch.c b/builtin/fetch.c index 8ee998ea2e..af896e4b74 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -20,6 +20,7

[RFC PATCH 1/6] DO NOT MERGE: compute commit-graph on every commit

2018-05-31 Thread Derrick Stolee
Also enable core.commitGraph and gc.commitGraph. Helps to test the commit-graph feature with the rest of the test infrastructure. Signed-off-by: Derrick Stolee --- builtin/commit.c | 5 + builtin/gc.c | 2 +- environment.c| 2 +- 3 files changed, 7 insertions(+), 2 deletions

[RFC PATCH 4/6] commit-graph: avoid writing when repo is shallow

2018-05-31 Thread Derrick Stolee
Shallow clones do not interact well with the commit-graph feature for several reasons. Instead of doing the hard thing to fix those interactions, instead prevent reading or writing a commit-graph file for shallow repositories. Signed-off-by: Derrick Stolee --- commit-graph.c | 12

[RFC PATCH 6/6] commit-graph: revert to odb on missing parents

2018-05-31 Thread Derrick Stolee
the commit-graph failed to fill the contents. Then the caller is responsible for filling the rest of the data from a commit buffer. Signed-off-by: Derrick Stolee --- commit-graph.c | 33 ++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/commit-graph.

[RFC PATCH 3/6] commit-graph: enable replace-object and grafts

2018-05-31 Thread Derrick Stolee
, then ignore the commit-graph for that commit and insted use the parents loaded by parsing the commit buffer and comparing against the graft file. Signed-off-by: Derrick Stolee --- builtin/replace.c | 3 +++ commit-graph.c| 20 +++- commit-graph.h| 9 + commit.c

Re: [RFC PATCH 0/6] Fix commit-graph/graft/replace/shallow combo

2018-05-31 Thread Derrick Stolee
On 5/31/2018 2:33 PM, Stefan Beller wrote: On Thu, May 31, 2018 at 10:40 AM, Derrick Stolee wrote: The commit-graph file stores a condensed version of the commit history. This helps speed up several operations involving commit walks. This feature does not work well if those commits "c

ds/generation-numbers (was Re: What's cooking in git.git (Jun 2018, #01; Fri, 1))

2018-06-01 Thread Derrick Stolee
On 6/1/2018 3:21 AM, Junio C Hamano wrote: * ds/commit-graph-lockfile-fix (2018-05-22) 1 commit (merged to 'next' on 2018-05-24 at 3d12a02b0c) + commit-graph: fix UX issue when .lock file exists (this branch is used by ds/commit-graph-fsck; uses ds/generation-numbers.) Update to ds/gen

Re: [RFC PATCH 4/6] commit-graph: avoid writing when repo is shallow

2018-06-01 Thread Derrick Stolee
On 5/31/2018 10:30 PM, Junio C Hamano wrote: Derrick Stolee writes: Shallow clones do not interact well with the commit-graph feature for several reasons. Instead of doing the hard thing to fix those interactions, instead prevent reading or writing a commit-graph file for shallow repositories

Re: t5318-commit-graph.sh breaks travis gettext poison job

2018-06-01 Thread Derrick Stolee
On 6/1/2018 12:17 PM, Duy Nguyen wrote: In case you're not checking travis, [1] reports Test Summary Report --- t5318-commit-graph.sh(Wstat: 256 Tests: 62 Failed: 2) Failed tests: 61-62 Non-zero exit status: 1 This usually means you're grepping

Re: ds/generation-numbers (was Re: What's cooking in git.git (Jun 2018, #01; Fri, 1))

2018-06-01 Thread Derrick Stolee
On 6/1/2018 7:15 PM, Junio C Hamano wrote: Derrick Stolee writes: A recently added "commit-graph" datafile has learned to store pre-computed generation numbers to speed up the decisions to stop history traversal. Will cook in 'next'. On Wednesday, these were m

[PATCH 01/14] graph: add packed graph design document

2018-01-25 Thread Derrick Stolee
Add Documentation/technical/packed-graph.txt with details of the planned packed graph feature, including future plans. Signed-off-by: Derrick Stolee --- Documentation/technical/packed-graph.txt | 185 +++ 1 file changed, 185 insertions(+) create mode 100644

[PATCH 03/14] packed-graph: create git-graph builtin

2018-01-25 Thread Derrick Stolee
Teach Git the 'graph' builtin that will be used for writing and reading packed graph files. The current implementation is mostly empty, except for a check that the core.graph setting is enabled and a '--pack-dir' option. Signed-off-by: Derrick Stolee --- Documentatio

[PATCH 00/14] Serialized Commit Graph

2018-01-25 Thread Derrick Stolee
[2] https://github.com/derrickstolee/git/pull/2 A GitHub pull request containing the latest version of this patch. P.S. I'm sending this patch from my gmail address to avoid Outlook munging the URLs included in the design document. Derrick Stolee (14): graph: add packed graph design docu

[PATCH 04/14] packed-graph: add format document

2018-01-25 Thread Derrick Stolee
mit object IDs. * A 256-entry fanout into that list of OIDs. * A list of metadata for the commits. * A list of "large edges" to enable octopus merges. Signed-off-by: Derrick Stolee --- Documentation/technical/graph-format.txt | 88 1 file changed, 88 insert

[PATCH 02/14] packed-graph: add core.graph setting

2018-01-25 Thread Derrick Stolee
The packed graph feature is controlled by the new core.graph config setting. This defaults to 0, so the feature is opt-in. The intention of core.graph is that a user can always stop checking for or parsing packed graph files if core.graph=0. Signed-off-by: Derrick Stolee --- Documentation

[PATCH 09/14] packed-graph: implement git-graph --clear

2018-01-25 Thread Derrick Stolee
Teach Git to delete the current 'graph_head' file and the packed graph it references. This is a good safety valve if somehow the file is corrupted and needs to be recalculated. Since the packed graph is a summary of contents already in the ODB, it can be regenerated. Signed-off-by: Derr

[PATCH 08/14] graph: implement git-graph --update-head

2018-01-25 Thread Derrick Stolee
It is possible to have multiple packed graph files in a pack directory, but only one is important at a time. Use a 'graph_head' file to point to the important file. Teach git-graph to write 'graph_head' upon writing a new packed graph file. Signed-off-by: Derrick Stolee --

[PATCH 06/14] packed-graph: implement git-graph --write

2018-01-25 Thread Derrick Stolee
Teach git-graph to write graph files. Create new test script to verify this command succeeds without failure. Signed-off-by: Derrick Stolee --- Documentation/git-graph.txt | 26 ++ builtin/graph.c | 37 ++-- t/t5319-graph.sh| 83

[PATCH 14/14] packed-graph: teach git-graph to read commits

2018-01-25 Thread Derrick Stolee
, 700,000+ commits were added to the graph file starting from 'master' in 7-9 seconds, depending on the number of packfiles in the repo (1, 24, or 120). Signed-off-by: Derrick Stolee --- builtin/graph.c | 33 + packed-graph.c | 18 +++--- pack

[PATCH 05/14] packed-graph: implement construct_graph()

2018-01-25 Thread Derrick Stolee
Teach Git to write a packed graph file by checking all packed objects to see if they are commits, then store the file in the given pack directory. Signed-off-by: Derrick Stolee --- Makefile | 1 + packed-graph.c | 375 + packed

[PATCH 07/14] packed-graph: implement git-graph --read

2018-01-25 Thread Derrick Stolee
Teach git-graph to read packed graph files and summarize their contents. Use the --read option to verify the contents of a graph file in the graph tests. Signed-off-by: Derrick Stolee --- Documentation/git-graph.txt | 7 +++ builtin/graph.c | 54 packed-graph.c

[PATCH 10/14] packed-graph: teach git-graph --delete-expired

2018-01-25 Thread Derrick Stolee
Teach git-graph to delete the graph previously referenced by 'graph_head' when writing a new graph file and updating 'graph_head'. This prevents data creep by storing a list of useless graphs. Be careful to not delete the graph if the file did not change. Signed-of

[PATCH 12/14] packed-graph: read only from specific pack-indexes

2018-01-25 Thread Derrick Stolee
Teach git-graph to inspect the objects only in a certain list of pack-indexes within the given pack directory. This allows updating the graph iteratively, since we add all commits stored in a previous packed graph. Signed-off-by: Derrick Stolee --- Documentation/git-graph.txt | 12

[PATCH 13/14] packed-graph: close under reachability

2018-01-25 Thread Derrick Stolee
Teach construct_graph() to walk all parents from the commits discovered in packfiles. This prevents gaps given by loose objects or previously-missed packfiles. Signed-off-by: Derrick Stolee --- packed-graph.c | 26 ++ t/t5319-graph.sh | 14 ++ 2 files

[PATCH 11/14] commit: integrate packed graph with commit parsing

2018-01-25 Thread Derrick Stolee
o-order -1000 | 5.9s | 0.7s | -88% | | branch -vv | 0.42s | 0.27s | -35% | | rev-list --all | 6.4s | 1.0s | -84% | | rev-list --all --objects | 32.6s | 27.6s | -15% | Signed-off-by: Derrick Stolee --- alloc.c | 1 + commit.c

Re: [PATCH 00/14] Serialized Commit Graph

2018-01-25 Thread Derrick Stolee
On 1/25/2018 10:46 AM, Ævar Arnfjörð Bjarmason wrote: On Thu, Jan 25 2018, Derrick Stolee jotted: * 'git log --topo-order -1000' walks all reachable commits to avoid incorrect topological orders, but only needs the commit message for the top 1000 commits. * 'git merge-b

Re: [PATCH 02/14] packed-graph: add core.graph setting

2018-01-25 Thread Derrick Stolee
On 1/25/2018 3:17 PM, Stefan Beller wrote: On Thu, Jan 25, 2018 at 6:02 AM, Derrick Stolee wrote: The packed graph feature is controlled by the new core.graph config setting. This defaults to 0, so the feature is opt-in. The intention of core.graph is that a user can always stop checking for

Re: [PATCH 00/14] Serialized Commit Graph

2018-01-26 Thread Derrick Stolee
On 1/25/2018 6:06 PM, Ævar Arnfjörð Bjarmason wrote: On Thu, Jan 25 2018, Derrick Stolee jotted: Oops! This is my mistake. The correct command should be: git show-ref -s | git graph --write --update-head --stdin-commits Without "--stdin-commits" the command will walk all packed

Re: [PATCH 01/14] graph: add packed graph design document

2018-01-26 Thread Derrick Stolee
On 1/25/2018 3:04 PM, Stefan Beller wrote: On Thu, Jan 25, 2018 at 6:02 AM, Derrick Stolee wrote: Add Documentation/technical/packed-graph.txt with details of the planned packed graph feature, including future plans. Signed-off-by: Derrick Stolee --- Documentation/technical/packed

Re: [PATCH 01/14] graph: add packed graph design document

2018-01-26 Thread Derrick Stolee
On 1/25/2018 4:14 PM, Junio C Hamano wrote: Derrick Stolee writes: Add Documentation/technical/packed-graph.txt with details of the planned packed graph feature, including future plans. Signed-off-by: Derrick Stolee --- Documentation/technical/packed-graph.txt | 185

Re: [PATCH 02/14] packed-graph: add core.graph setting

2018-01-26 Thread Derrick Stolee
On 1/25/2018 4:43 PM, Junio C Hamano wrote: Derrick Stolee writes: The packed graph feature is controlled by the new core.graph config setting. This defaults to 0, so the feature is opt-in. The intention of core.graph is that a user can always stop checking for or parsing packed graph files

Re: [PATCH 03/14] packed-graph: create git-graph builtin

2018-01-26 Thread Derrick Stolee
On 1/25/2018 4:45 PM, Stefan Beller wrote: On Thu, Jan 25, 2018 at 6:02 AM, Derrick Stolee wrote: Teach Git the 'graph' builtin that will be used for writing and reading packed graph files. The current implementation is mostly empty, except for a check that the core.graph setting

Re: [PATCH 03/14] packed-graph: create git-graph builtin

2018-01-26 Thread Derrick Stolee
On 1/25/2018 6:01 PM, Junio C Hamano wrote: Derrick Stolee writes: Teach Git the 'graph' builtin that will be used for writing and reading packed graph files. The current implementation is mostly empty, except for a check that the core.graph setting is enabled and a '--pack-dir

Re: [PATCH 04/14] packed-graph: add format document

2018-01-26 Thread Derrick Stolee
On 1/25/2018 5:06 PM, Junio C Hamano wrote: Derrick Stolee writes: Add document specifying the binary format for packed graphs. This format allows for: * New versions. * New hash functions and hash lengths. * Optional extensions. Basic header information is followed by a binary table of

Re: [PATCH 04/14] packed-graph: add format document

2018-01-26 Thread Derrick Stolee
On 1/25/2018 5:07 PM, Stefan Beller wrote: On Thu, Jan 25, 2018 at 6:02 AM, Derrick Stolee wrote: Add document specifying the binary format for packed graphs. This format allows for: * New versions. * New hash functions and hash lengths. * Optional extensions. Basic header information is

<    9   10   11   12   13   14   15   16   17   18   >